It depends on what they are used for. I use six sets of usergroups. One I call the Default set, which are the two that XenForo provides (Registered, and Unregistered/Unconfirmed). Another I call Staff usergroups, which grant permissions to contributors, moderators and admins. A third group I call Automatic--these are usergroups assigned automatically, using the user promotions system. (If a person reaches a certain number of posts, we unlock all the forum features available to standard registered users. And if they reach 50 posts/30 days, our trading section is opened up to them.) The fourth and fifth groups are for manual use, for either adding [+] or removing [-] permissions. Each one only has one single permissions change. If a person acts up in our Off-Topic group, we have a permission that removes Off-Topic access. If a person abuses the private messaging system, we have another usergroup to remove that privilege. For some members, we allow an extra privilege, such as the ability to add thread prefixes--so another usergroup adds that permission where needed. I even have one usergroup that is used for styling the site owner's username--that is its only purpose. A sixth group, which I no longer use, was for Paid features--in other words, if they subscribed for a year, it unlocked a few bonus features.
On a larger forum I might have three or four dozen usergroups. On smaller, perhaps a dozen. The XF permissions system is very flexible for this type of use.
Staff knows to use ONLY the [+] or [-] usergroups, and to never touch the others if they need to adjust someone's permissions. It makes understanding the system that much easier.