Okay all the said Moderators have common permissions, so I already have them clubbed in one group. I initially had their permissions set in Usergroup Permissions only. But I later noticed that even if I add them for specific forums only, they get moderating powers for all over the forum basically.
It's an area in which (IMO) XenForo handles pretty badly. There is no easy way to determine or change a given set of permissions that will apply to all Moderators but ONLY when they are within "their" forums. Each instance of Moderator <> Forum has to have these permissions explicitly determined.
At a permissions level, XF makes practically no distinction between a Moderator and a SuperModerator. So, if you have a large team and a large board, you have a lot of work to set it up. And, if you want to alter these forum-specific permissions (eg you install a new add-on) then you have to individually alter settings for each Moderator <> Forum instance.