Fixed Usergroup Promotion Cron Running Out of Memory

[latimer]

Recently I noticed that my automatic usergroup promotions stopped working for no apparent reason. I checked the error logs and the script is running out of memory when it is executed. I have roughly 100K registered accounts, of which about 20K are active within the last 3 days and therefore are considered by the promotion cron.

I saw there is a previous bug report labeled "won't fix" for a similar issue, but this is not for a one time usergroup move. It's been consistently failing every hour for the past few days. Would it be possible to only process X users at a time like the forum total recounter scripts so that it scales a bit better on larger forums?

 

[Rob]

This is definitely a bug imo.

 

[Jake Bunce]

That's a big forum. I can see how this could fail on a very large forum.

I am inclined to call this "not a bug" but I will let the devs decide. IMO the memory usage is justified and is a function of user activity on your forum. If a forum is extremely active like yours then you need to increase your memory_limit.

If the software were to be modified to accommodate such a large number of users within a certain memory_limit then there would have to be more intelligent selection of accounts, but that can potentially be complicated. For example, you could try to implement criteria checking in the selection query but that's just nasty.

I am going to suggest two fixes:

1) Increase your memory_limit in PHP.

Or:

2) Edit this file:

library/XenForo/CronEntry/UserGroupPromotion.php

Decrease the red number:

		$users = $userModel->getUsers(array(
			'user_state' => 'valid',
			'is_banned' => 0,
			'last_activity' => array('>', XenForo_Application::$time - 86400 * 3)
		), array(
			'join' => XenForo_Model_User::FETCH_USER_FULL
		));

86400 * 3 seconds is 3 days. But the cron runs every hour so you can afford to decrease it some. It would be good to leave some leeway in case the cron is delayed by forum inactivity. There can also be interaction with other crons like user upgrades which also run hourly.

 

[Jake Bunce]

Would it be possible to only process X users at a time like the forum total recounter scripts so that it scales a bit better on larger forums?

Do you mean the "Rebuild Board Totals Counter" cron? That doesn't do batch processing to count users. It's just one query but it doesn't have to return entire user records. It just returns a count so it uses almost no memory within PHP. This cannot be applied to your situation.

 

[latimer]

Do you mean the "Rebuild Board Totals Counter" cron? That doesn't do batch processing to count users. It's just one query but it doesn't have to return entire user records. It just returns a count so it uses almost no memory within PHP. This cannot be applied to your situation.

Sorry, I was referring to the rebuild caches page:


I know those are two ways to fix it, but those are just temporary and I'd have to keep changing those values as my board grows. There are boards that are much larger than mine, such as IGN's forum, and in those cases the usergroup promotion cron will easily consume gigabytes of memory per run.

 

[Jake Bunce]

Oh. Those cache rebuilds rely on multiple requests. When a cron is called it's a single request so you can't split it up like that.

I just had an idea... if you run multiple SELECT queries using a LIMIT clause then you can process all of the users in a single execution while reducing the maximum memory usage. That's the theory anyways. I will test this and report back. The code is fairly simple, I just need to debug the memory usage to make sure it's effective.

 

[Jake Bunce]

I just had an idea... if you run multiple SELECT queries using a LIMIT clause then you can process all of the users in a single execution while reducing the maximum memory usage. That's the theory anyways. I will test this and report back. The code is fairly simple, I just need to debug the memory usage to make sure it's effective.

It works.

I have attached a new library/XenForo/CronEntry/UserGroupPromotion.php file to this post. Please try it and let me know if it works for you.

For the devs... this modified file is based on version 1.1.2. I added a limit and offset to the query and put it inside of a for loop to query the user records in batches. At the end of each loop I unset($users) thereby freeing up the memory. So the maximum memory usage during execution is a function of the batch size which I measured at 10MB per 1000 records. If you use this fix in the default software then you just need to set the $batchSize (which I left at 1000). I also want to plug a previous suggestion of mine while you're at it:

Processing only active users makes sense for the automated task under normal circumstances. But I would suggest changing the task to check all users if the task is being run manually. It seems to me that the intention would almost always be to process all users if you are manually running the task.

Because this code change alleviates memory concerns you should now be able to implement this suggestion. The trophy cron should also be updated as it can suffer from memory problems as well.

 

[Rob]

Nice thinking Jake!

 

[latimer]

That works great Jake, thanks! I'm not yet having any problems with my trophy cron, but here's the modified file with Jake's fix if anyone needs it.

 

[digitalpoint]

I was doing some testing with some live data and my dev server kept getting an out of memory error...

I ended up tracing it to the cron task that checks for trophies to be awarded.

The runTrophyCheck() method is evaluating users that have visited the site in the last 24 hours, even though it does that evaluation every hour. There really is no need to evaluate users that last visited the site more than 2 hours ago.

It becomes a huge waste of resources, since ultimately you are checking if someone needs a new trophy 24 times every time they visit the site... and in the case of a TON of daily visitors, you can run out of memory since all the users are loaded into a single array.

A better long-term solution would to put some sort of limit on the getUsers() query and loop multiple times if needed, but a short-term fix would be to just change the 86400 second threshold to 7200 or something.

 

[Jake Bunce]

Threads merged. Same issue as the promotion cron.

A Trophy.php file is provided above with a code fix to reduce the memory usage if it's a problem for you.

 

[digitalpoint]

Yep... looping it like that is what I was talking about for a long-term solution that scales, but I still don't see a point of checking for promotions/trophies that would be due 24 times after a user visited (checking hourly for anyone that visited in the last 24 hours).

 

[CTXMedia]

Anyone know if this was included in 1.1.3?

 

[digitalpoint]

Anyone know if this was included in 1.1.3?

Well, it's an unresolved beg, so...

 

[CTXMedia]

Well, it's an unresolved beg, so...

lol ... I hadn't actually scrolled to the top of the page so didn't see the prefix ...

I'm just messing with trophies and thinking about re-applying them along with associated user titles and as I have 20,000+ users didn't want to have MySQL timeout/fall on its knees (on my poor old server).

 

[digitalpoint]

Just use the Trophy.php attachment in one of the previous posts.

 

[CTXMedia]

Will do - thanks.

 

[ashkir]

Thanks. Trying the new one too. Apparently it doesn't like our registered user group. Large forum.

 

[rdn]

It works.

I have attached a new library/XenForo/CronEntry/UserGroupPromotion.php file to this post. Please try it and let me know if it works for you.

For the devs... this modified file is based on version 1.1.2. I added a limit and offset to the query and put it inside of a for loop to query the user records in batches. At the end of each loop I unset($users) thereby freeing up the memory. So the maximum memory usage during execution is a function of the batch size which I measured at 10MB per 1000 records. If you use this fix in the default software then you just need to set the $batchSize (which I left at 1000). I also want to plug a previous suggestion of mine while you're at it:



Because this code change alleviates memory concerns you should now be able to implement this suggestion. The trophy cron should also be updated as it can suffer from memory problems as well.

Internal server error when I upload this file.

 

[Jake Bunce]

Internal server error when I upload this file.

On what page?

Could be wrong file permissions or owner. That's a generic server error.