Won't fix Out of memory running Usergroup promotions cron job manually

[MGSteve]

As part of my efforts to get around the issues with moving users between usergroups, I thought I'd try using Usergroup Promotions to do it.

I setup the promotion for a set of around 100 users, set the last active timestamp for all 100K users to the current time & manually ran the script.

It bailed out with an out of memory error.

Allowed memory size of 134217728 bytes exhausted (tried to allocate 91 bytes) in /home/xxxxx/xxxx_html/library/Zend/Db/Statement/Mysqli.php on line 304, referer: http://xxxxxx/admin.php?cron/
I guess this is related to the deleting users out of memory bug when it tries to load too many users at once.
I know it'll get marked as a future fix no doubt, but I thought I'd mention it anyway.

That said, unless it loads the details of all 100K users in, I fail to see why it should bail out with only 100 users requiring to be promoted. Its not beyond the realms of possibility that this could happen on a large & busy forum anyway.

 

[Jake Bunce]

set the last active timestamp for all 100K users to the current time & manually ran the script.

That explains it.

library/XenForo/CronEntry/UserGroupPromotion.php

		$users = $userModel->getUsers(array(
			'user_state' => 'valid',
			'is_banned' => 0,
			'last_activity' => array('>', XenForo_Application::$time - 86400)
		), array(
			'join' => XenForo_Model_User::FETCH_USER_OPTION
		));

The task starts by selecting all currently active users (by last_activity). That is 100k rows being returned.

Your approach is one that I have suggested to others (either updating last_activity or removing that requirement from the above code). It works fine except that you have way too many users to process all at once.

You could try making batches out of it by manipulating the last_activity dates. Or you could try increasing the memory_limit. See this post. You might also have to edit the library/XenForo/Application.php file:

		@ini_set('memory_limit', 128 * 1024 * 1024);

 

[Jake Bunce]

Consider also the reason for promoting only recently active users. More than just reducing the size of the task... if a user is inactive then they won't be able to use their promotion because they aren't visiting the forum. Once they visit then they will be promoted within an hour when the task runs.

But I understand there are some situations where it is desirable to mass-promote all users when creating a new promotion.

 

[MGSteve]

Consider also the reason for promoting only recently active users. More than just reducing the size of the task... if a user is inactive then they won't be able to use their promotion because they aren't visiting the forum. Once they visit then they will be promoted within an hour when the task runs.

But I understand there are some situations where it is desirable to mass-promote all users when creating a new promotion.

Yeah, in a normal case it wouldn't be an issue, however in our case we want to tidy up the usergroups.

Although I have got 16GB of ram on that server, it may be better to do the upgrades in batches, like you say - even if the RAM was lifted, its still 100K users it would have to cycle through.

 

[Mike]

As mentioned by Jake, this is more or less expected due to some manual database changes. If you restore your last activity times and temporarily change it to promote people that have been active in, say, the last month, that may be a better approach and still cover the vast majority of people using the forum.

 

[MGSteve]

As mentioned by Jake, this is more or less expected due to some manual database changes. If you restore your last activity times and temporarily change it to promote people that have been active in, say, the last month, that may be a better approach and still cover the vast majority of people using the forum.

I agree Mike, it was slightly self inflicted, but I wasn't sure how the usergroup promotion script worked, I should have looked first & posted second, sorry.

In any case, using promotions is of no use - they don't change the primary usergroup, which in some of our cases we need to do - well specifically for one usergroup, we need to move around 50,000 users from one to the other.

hmm, *thinks out loud* that said if we carry over the moderator approval new post requirement for members under 10 posts, I could convert the xf 'registered' users group to 'on probation' group and then setup a user promotion to move them into the vb 'registered' group when they've made the required number of posts, then we don't have to move them. Hmm, that should actually get around the issue.

Any remaining users (like these 100) have the group we want to remove as a secondary group, so I presume we can simply promote them out of that group & not specify a new secondary group to put them in?

Will the script promote them out of one group (i.e. remove them from that group) even if there isn't a new group selected to promote them into? (makes sense?)

 

[Jake Bunce]

Can you tell me exactly what you are trying to accomplish with promotions? Why do you want to promote users to a different group?

 

[MGSteve]

to move them out of a usergroup that was brought over from vB. I am trying to condense the 30 odd usergroups into a more manageable 6 or so!

 

[Jake Bunce]

If you delete a user group then primary members of that group will be moved to the default Registered group (user_group_id 2).

 

[MGSteve]

If you delete a user group then primary members of that group will be moved to the default Registered group (user_group_id 2).

Yeah, I tried doing that the first time with the usergroup with 50,000 members in it and got an out of memory error there as well! For most of the secondary usergroups though, with a couple of hundred at most, deleting the usergroup should work.

 

[Mike]

He's running into memory/timeout issues with when deleting a user group (because of all the user updates and permission calculations that are needed).

[Edit: ninja'd]

 

[Jake Bunce]

Oh. What you really need is a mass move feature with batch processing.

In reference to your other thread:

http://xenforo.com/community/threads/mass-moving-people-from-one-usergroup-to-another.21156/

I have too many usergroups at the moment and I want to shrink them down when we move to XF, this means either doing the leg work in VB3.8 or in XF.

Can you even change people's usergroup membership en-masse in either system? What's the best way to do it?

I recommend doing the leg work in vB 3.8 since that software has a mass move feature for users and the process is lighter weight since vB doesn't need to calculate permission combinations like XenForo does.

 

[Dean]

I recommend doing the leg work in vB 3.8...

Definitely do as much cleanup and planning in vb3.8 before the import as possible... (not sure if you are live yet, have not been following closely)

 

[MGSteve]

I recommend doing the leg work in vB 3.8 since that software has a mass move feature for users and the process is lighter weight since vB doesn't need to calculate permission combinations like XenForo does.

The problem is that the usergroup permissions are all over the place in XF anyway following the vB import, so chances are, I'll have to go through and reset them all anyway (is there any easy way to reset permissions for all forums in one easy step?)

I'll then just go through and set the permissions on a node by node basis where nodes have specific permission needs (i.e. moderator forums).

I don't know why our permissions are wrong, but I suspect its got something to do with the fact that our default vb registered group isn't actually registered group - the anti spam system moves them out into a new registered group once they've made 10 moderator approved posts.

But the importer doesn't know this and sets permissions for the vb default registered group and as a result anyone in our registered group ends up with zero permissions on all the forums!

But, doing all the moving of users around in vB is an option.. I'll have to try doing that next time, the only problem is having to setup a duplicate of the live vB forum to make all the changes on a test dataset.... which is a PITA.