I keep a copy of any add-ons on my computer. When uninstalling I just look at the contents of the add-on on my computer and then delete the corresponding files and folders on the server. They are generally easy to find too as add-ons haven't their own folders in library, js and styles directories.
I agree it would be good to have a simpler uninstallation process.