The absence or presence of files themselves doesn't really make a difference to how XF runs. If you want to remove add-on content, you need to uninstall the add-on, which would remove all of their data and configuration. (You could potentially just disable them as well, though that can leave some references to them.)