Yes, you can.
Once you work out which user groups you do and don't want to have permissions, head to
Groups & permissions, Usergroup permissions in the Admin CP.
XFMG: Album Permissions controls the ability for users to create their own albums within categories:
For the user groups where you would like them to have permissions, select
Yes on the specific permissions. For the user groups where you don't want them to have permissions, select
No.
As a side note while you are there, it's worth taking a look at your permissions for
XFMG: Media Permissions, which controls the ability to add media without albums which will likely be allowed for the scenario you described above:
If you wanted to extend things and set things up in more detail, you could define these permissions for user groups on a per category basis via
Media, Media categories, Your Category, Permissions.
Hopefully that helps, and let me know if you have any other questions.