To test, suppose you have the following structure:
... +--Forum2 (private)
Forums 1 and 2 are hidden by default, and 2 is a subforum in 1.
You want a user group to see all contents of Forum2, but not of Forum1. So, for Forum1 you leave all permissions at Inherit, but set 'view threads of others' and 'start your own thread' to Revoke. Then for Forum2 you leave all settings at Inherit, and set those two settings to Allow. Then you're done and assign this group to a User (no other no other custom settings or usergroups are assigned).
If you login with that user, the result is as intended: in Forum1 all threads are invisible, but when you enter Forum2 all threads there are visible and open for interaction to you.
But if you view the user via "Test Permissions" (without logging in as him), the contents of Forum1 are entirely visible to you when they shouldn't. This makes me think some administrator permissions seep in.