When a permission is granted "Allow" for an add-on, for example I allow Administrative to use the add-on, if the add-on is uninstalled I would expect the permissions associated with that add-on to be removed, it's not. I can reinstall the add-on and the permissions "Allow" for the user group I granted before is already set.
In various places where there's user created data, we don't necessarily remove it when an add-on is removed. This applies to template customizations or phrases, for example. The same principle applies to add-ons with permissions, though there are also some semi-technical considerations that made it the defacto behavior. As such, I don't think this is something we would change.