Because of the nature of how Office add-ins are loaded, it's an easy enough mistake to try to name your configuration file as:
<library-name-with-extension>.config or in my case, <executable-name-with-extension>.config.
In fact, the configuration file actually needs to be named WINWORD.exe.config and it should be located in the same location as the as the Word runtime. It's kinda of baffling though, since as far as I understand, Word is not a native .Net application.
Of course, I didn't notice this since the application was designed to run as a standalone executable and as an add-in, so the configuration file worked fine when launched as a standalone, but not so well when launched as an add-in.