Edit: Please ignore, forgot Mod-Status also keeps them in the list! Sorry!
I just moved from VB to XenForo and i tried to remove a custom permission using the steps above.
But for some reason the numeric value resets to "Inherit" after saving, but ONLY when it's Zero.
So at the moment i can't seem to remove the custom users. Could that be a bug or am i missing something?