XF 1.1 User Groups cause server timeouts

My site is big (100k+ members), so whenever I try and edit user groups, like try and add or remove users from a specific secondary user group, the database will lock up. The server error log displays no errors. Everything else in the admin control panel is fine, just editing a user's usergroups will cause a server timeout.

Any way to fix these?