Thats bad behave. Lots of orphans Files are remaining this way....
There should be a possibility with xenforo or the addon that all Files are deleted too.
Had a few very old addons installed and could not find the Download here anymore (archive), so i dont know which files i have to delete manually.
I always keep a local copy of any add-on so once I've uninstalled one, I then look at the files and folders locally that come with the add-on and then remove these from my site. This makes it easy keeping things tidy.