Design issue PHP memory limit exceeded during search

[Wildcat Media]

We've had this pop up a few times in the past two months. I had our PHP memory limit set to 160MB; I got the error message saying the "Allowed memory size of xxxxxxxxxx bytes exhausted (tried to allocate xx bytes)". I bumped the memory limit to 200MB and the error went away. Tonight, one of our moderators tot the message, and I bumped memory up to 256MB.

We are running the most recent Enhanced Search add-on for XF 1.2.3, indexing 9.3 million messages / 327,626 discussions / 36,296 members, and have an add-on that lets us index personal conversations as well.

I don't know if the Enhanced Search is the cause of it, but using search certainly seems to trigger it. Is this memory usage going to keep growing quickly like this, or is there something else I should look at? I really do not want to press PHP higher on the memory limit if I don't have to.

 

[Mike]

Memory usage and index size are generally unrelated. However, how many search results are you set to return? The larger this number, the larger maximum amount of memory usage. There's not a lot we can do about that.

Of course, you haven't given the specific line number details in the error message as well.

 

[Wildcat Media]

The error was in ...../library/Zend/Db/Statement/Mysqli.php on line 304.

We've had the limit set to return 1,000 results since we converted a year ago in November, but only in the past three or four weeks did we start seeing the error. I have not done any work on the forum (or changed any of the settings) since at least this past August, as I had a full schedule this semester.

I may bump the results to 500. We've had it at 1,000 for several years, going back to our days with vBulletin (which was using Sphinx).

 

[Mike]

This is really down to the specifics of some searches (that are very obscure in general). Depending on the query, we need to do extra filtering that can't really be done in ES. To give better results, we return a multiplied factor of results to filter on those. Worst case could actually return something like 16 times the number of results for initial filtering.

In some cases, future adjustments to this might theoretically be possible, but in other cases, some constraints will generally not be fittable into ES (as they may cause huge write "amplification" many times per day), though again these tend to be the more obscure criteria.

I would have to recommend dropping the number of search results. When was the last time you went to page 50 of a search?

 

[Wildcat Media]

I admit it's been rare when I've gone through more than a few hundred results--I've been unusually stubborn in those rare instances, and have only been able to work with the most vague of search criteria.

I have gone and lowered the results to 500, but have not yet nudged the memory setting back down to see if the error triggers again. Some staffers have suggested we could even go as low as 250, although I remember some members complaining when I tried that in vB all those years ago (prior to installing Sphinx).

 

[Wildcat Media]

@Mike, do you know a way to monitor PHP memory allocation, perhaps as something I could manually add when I'm in Developer view (with the page stats at the bottom of a forum page)? Not a deal breaker but I'm thinking there must be a way I could do this.

I did run into a member issue. I have set our search results to a maximum of 500. We had a member who didn't visit us for two weeks, and upon returning and looking at recent threads, he noticed that the 500 result limit meant that he could list only about a day's worth of "New Posts". Sure enough, if I go to the New Posts link, 500 posts will only take me back about 20 hours' worth of posts. That is one area where I can see a member may want to go beyond 500 posts. (Although I sure know I wouldn't. )

 

[Mike]

Bear in mind that you can get further back if you do some reading, as read threads won't be selected.

Otherwise, there isn't really a particular way to watch memory allocation specifically. You may be able to do something by looking at the peak usage, but there are various situations where this could peak fairly high and it would be "normal".

 

[Wildcat Media]

Bear in mind that you can get further back if you do some reading, as read threads won't be selected.

I should mention that to the staff. I think some members just read down the list of new posts and cherry-pick those to read which they find interesting but I know others who cling onto every word and want to read everything. Those are ones who could benefit from refreshing New Posts to go further back.

 

[Wildcat Media]

Otherwise, there isn't really a particular way to watch memory allocation specifically. You may be able to do something by looking at the peak usage, but there are various situations where this could peak fairly high and it would be "normal".

Today in particular is a good example of why I wish I had a crystal ball into memory usage.

We've had an odd situation over the past 48 hours. Something as simple as a member going from page two of a thread to page three was causing an out-of-memory error. The error logs really don't show anything useful. I bumped memory for PHP up to 312MB, and still had errors. It is now bumped up to 384MB. We have not had any memory issues since I bumped down the search results returned to 500 total.

An odd side effect was that some stats were not being updated, along with member activity history (we look up a user who posted a few hours earlier, and don't see any activity since Thursday). Also, it is showing we have only a dozen members online, when we usually have 700-800 online during this time of day.

One oddity I noticed was that all of the cron jobs showed a "next run" date of Feb. 28th. I ran one of the crons, Rebuild Board Totals Counter, returned to the cron page, and all but one were updated properly. Daily Cleanup did not run, but I also ran that one manually as well.

My guess? One of the cron jobs might have been triggered by a page load, failed to run due to hitting the memory limit, and perhaps caused all of them to "queue up" without running further.

I have checked the database tables--no errors. I am currently rebuilding all of the counters except for statistics. Current stats: we have almost 1400 online during peak hours, and are nearing 10 million posts.

I would have put this in another thread, but I thought previous history with PHP memory issues might be good to keep all in one place. I'm primarily concerned about 1) what is causing so much memory usage, and 2) having the PHP memory limit up so high (not knowing what ill effects that may have on other parts of the system).

I may have to start disabling add-ons and see if any one of them shows a marked change in memory usage. That leads to one question: when I am running in debug mode, does the memory usage figure at the bottom of the page reflect PHP's memory usage (which causes the errors)? I just wondered if there was some other code I could use to perhaps display whatever memory figure would help me track this down.

Thanks for reading!

 

[Wildcat Media]

@Mike, per my last post, we again had a problem where 1) I was getting numerous out-of-memory errors in PHP when members were doing a search, and 2) our cron jobs stalled. I now have PHP memory set to 500MB, which seems extraordinarily high. This happened Wednesday. Here is one of many error messages, all triggered by searches:

Error Info
ErrorException: Fatal Error: Allowed memory size of 402653184 bytes exhausted (tried to allocate 1024 bytes) - library/Zend/Db/Statement/Mysqli.php:295
Generated By: Unknown Account, Friday at 9:57 AM
Stack Trace
#0 [internal function]: XenForo_Application::handleFatalError()
#1 {main}
Request State
array(3) {
  ["url"] => string(80) "http://forums.stevehoffman.tv/search/12555898/?q=rory+gallagher&o=date&c[node]=2"
  ["_GET"] => array(3) {
    ["q"] => string(14) "rory gallagher"
    ["o"] => string(4) "date"
    ["c"] => array(1) {
      ["node"] => string(1) "2"
    }
  }
  ["_POST"] => array(0) {
  }
}

If it happened only one time, I'd have chalked this up to a fluke. But twice? I really would like to find out what is happening.

My only thought is that one of the add-ons I am using is causing this problem. I have one that extends search to private conversations (which is a must-have for us). Our post count is now just 140,000 posts shy of 10 million, and we're seeing as many as 1,500 users online during peak hours (which will jump substantially when our next website goes public).

Is there anything I can use to monitor PHP memory usage? I am very concerned about continually bumping up the PHP memory limit--it is fixing a symptom, not curing a cause. If I could see memory usage while disabling and re-enabling various add-ons, I might be able to begin figuring this out.

I have not yet upgraded us to 1.3--I have to wait a week and a half until the semester is over (I just don't have the time for it). I don't think that would cure anything, but...who knows?

 

[Wildcat Media]

@Mike: it's happened again. I bumped the PHP memory, and now something is exhausting 524,288,000 bytes of memory. And cron stalled again.

I'll have to open a ticket on this, or maybe repost it as a different bug. It may not be tied specifically to search, in other words.