That is why the base settings, and design approach of your forums is so important. Specifically the 'special' forums such as a staff forum should ideally be grouped with the other special forums - and that category set to Private. Then let only the special people see that category.
Again, I urge you to what you can before import, much easier that way.
I just came back to suggest that. I did a test import, and was faced with the same issue. I thought through my strategy a bit more, made many changes on the live vb forum by resetting node permission (personally I think that was an important step, those can build up over the years), reduced number of forums & usergroups to a minimum, re-arranged how the forums were set up.. all kinds of things. Then I repeated.
I believe I did that 3 times and finally made notes of exactly what needed to be done for the final import. It really was not too bad for the final import. I never really used ipb but do own a copy, so I really cannot give input on those details.
Yes, good idea, I think I'll review the number of usergroups I've got - especially if removing a single usergroup would mean 40 less operations for checking permissions after the migration - a lot of time saved ...
A bit of user account and older (not visible any more) forum pruning wouldn't go a miss either.
It's funny on the subject of usergroup permissions that I had to create a new one in my vB forum yesterday, I did it completely differently than normal, I did it the XenForo way, making it as a secondary group and allowing only the specific access I wanted, it was so much easier.
I have to say though, GeeksChat is right, the permissions screen in IPB is brilliant for the basic options, listing all the forums and with buttons to set access per column or row too.
Deans advice is really good here, do what you can in vB first before conversion, I actually intend to strip out most extra usergroups and re-add then in XenForo as this is easier than trying to convert them all. Especially for private/staff forums, where it's easier to use the private option in XF.
Still, at least you only have to do the conversion once!
Normally, say in vB, to do a private forum you have to create permissions which explicitly deny access, this can be tedious with lots of usergroups, with XenForo it's the opposite. You mark it "private" then only usergroups with an "allow" setting have access. I guess it kind of inverts the permissions to make it simpler to do.