this woulld be hard unless if they have one specific addon folder but some addon use styles, src, data, and so on and let say themehouse and XFA that have addons depending on each other it would be a disaster if you removed those files
I don`t think it will be a good idea to do it automatically. Will be nice if we can have a popup notice, when you uninstall an add-on, who tell you the directories of that add-on location and, if the add-on have common files or not.
I agree, uninstalling and removing files should be done with care and not by some automated process. Maybe once an addon is uninstalled, and admins have verified everything is functioning correctly, then have a button to click to remove the files.