I have already done that and they still show in in the "User Permissions" section. I can't remove some old user with permission. I think this is a bug.
In VBB I just move them from User Permission groups to User registed. That's all, very simple. I don't why xenforo so complex this? May be it is a bug?