The absence or presence of files themselves doesn't really make a difference to how XF runs. If you want to remove add-on content, you need to uninstall the add-on, which would remove all of their data and configuration. (You could potentially just disable them as well, though that can leave some references to them.)
I had edited this thread to be a more general question, but the original post gave more details - essentially an add-on had failed to install correctly, and it was the add-on that modified install functionality. This left me in a situation where no add-ons could be installed or uninstalled, and the failed installation add-on was not even listed in the list of installed add-ons.
Luckily I was able to finally resolve the problem.