XF 1.4 Slow rebuilds/reparsing

[nrep]

I'm experiencing an usual problem with XF addon installs/upgrades which has become much more pronounced since switching to a Windows 2012 server. The rebuilding/reparsing stage can easily take over an hour per site, even when the site load is extremely low (the server is plenty powerful - Dual Hex Core Xeon 2620, 32GB Ram, RAID 1 SSDs, etc...).

We're only running 1 style and anywhere from 6-20 addons per site. This problem happens on all XF installations, but the number of addons does seem to make a difference.

Has anyone experienced unusually slow rebuilds or know where I can look in to where the delays are coming from? I've had a look at the MySQL monitor and the load barely moves, but I don't know where else to start searching.

 

[Paul B]

Have you carried out any server tuning?

 

[imthebest]

Do you have anything logged in your slow query log? Have you checked the status/health of your RAID array? How big is your database?

It sounds to me like a hardware problem... probably related to your SSDs or RAID configuration.

 

[nrep]

Yep, the server has been recently optimised and the isn't anything in the slow query log around the time of the rebuilds. I've just logged in to manually verify the RAID status, and that's fine too. Just to double-check performance, I ran PassMark and it performs as expected in the tests.

There are several of XF sites on the server, with DBs ranging from tens of MB to tens of GB. The re-build time doesn't seem to be affected by DB size from what I can tell.

 

[Paul B]

his problem happens on all XF installations, but the number of addons does seem to make a difference.

Rebuilding time is heavily affected by the number of add-ons, styles, and languages.

I have seen some servers take up to an hour to do a rebuild but those are typically related to cheap shared hosting.

Typically, a couple of styles and languages with a handful of add-ons shouldn't take more than a few minutes.

 

[nrep]

Is there any way I can trace what's taking so long?

 

[Mike]

The only tracing that can be done is really via developer facing tools (such as XDebug) and would need someone familiar with the code to analyze.

However, issues such as this almost always point to MySQL configuration issues. While you can do things to speed up PHP initialization/compilation, aside from PHP version, the run time is mostly just down to the language. MySQL has huge tuning variables though. The big ones for XF are discussed here: http://www.percona.com/blog/2013/09/20/innodb-performance-optimization-basics-updated/

Is MySQL running locally? If it's not, you may need to consider if there is any network overhead. The rebuild steps run a very significant number of queries so latency to the DB server would be a significant bottleneck here. Reducing that would help this step (and all XF pages really).

 

[nrep]

Thanks Mike. I've got XDebug installed (but not running), so I'll see what I can capture and see if anything obvious stands out.

MySQL is running locally, with all tables InnoDB (except any marked as memory).

I'm currently deploying a backup image to a local test server so I can try changing things and see if there's any performance improvement. We've got Zend OpCache and WinCache running, so I'll pare things back to eliminate any incompatibility issues.

As soon as I've got any data, I'll report back.

 

[Mike]

This is something you'd likely see on most pages, but you may wish to try connecting to MySQL via "127.0.0.1" instead of "localhost". I have seen a few cases where (I think) localhost attempted to connect via IPv6 creating a delay before it fell back to IPv4 (where MySQL was listening). If it's just rebuilds, that may not be related though.

 

[nrep]

Sorry for the slow reply, I've been taking my time to debug this and narrow down the problem. After much playing around on a test server snapshot, it seems that the problem lies with IIS.

I've installed a basic, non-optimised version of Apache and it is FAR faster at rebuilding than IIS is (both share the same MySQL service).

I've read all of the IIS guides I can find with regards to optimisation, but the settings seem fine from what I can tell. I've experienced a few oddities with IIS (such as random page not found errors, fixed thanks to some code for config.php from Mike).

Is there anyone here familiar with IIS and can point me in the right direction to some monitoring tools that may help diagnose the problem, now that it's narrowed down. GGoogle brings up basic app pool monitoring stuff, which isn't much help.

 

[nrep]

I've been struggling with this issue for a long while now, but was able to ignore it by avoiding updating addons frequently.

However, now that I've upgraded some sites to XF 1.5, I've noticed a huge slowdown when sending bulk e-mails. It then occurred to me that it's not the rebuilding/reparsing stage, but anything that uses the deferred system.

Is there anything that springs to mind in the deferred system that I can disable/test? I get this same problem with zero addons enabled.

 

[Mike]

There should be minimal overhead to the deferred system itself. The email sending process is pretty much identical, though before it would send X number of page, rather than sending as many as it can in a certain period. You could potentially increase the length a deferred process runs via config.php as long as you ensure any timeouts outside of XF aren't triggered (FastCGI timeouts, for example).

 

[nrep]

Thanks @Mike, I'll try extending the length out of curiosity and report back ($config['rebuildMaxExecution'] I guess?). I'm part way through an e-mail send (which I guess will take until tomorrow), so I'll do it once that is finished.

On a 1.4 site that sends out e-mails, I'm doing just under 100 per second. On 1.5 I'm doing about 2 per second.

I've disabled all addons and the problem still occurs, so I'm at a loss. Is there anything I can do to break the bulk e-mail sending out of the deferred system? Or perhaps a way to debug/inspect the deferred system?

 

[Mike]

That sounds like a speed issue with the actual email send. What version of 1.5 are you using, to confirm?

You'd really have to debug by inserting debugging code within the job itself. It's unlikely to be the deferred system itself causing that level of a slow down IMO.

 

[nrep]

We're on 1.5.4

I'm really puzzled as anything using the admin defer system is very slow, not just e-mail sends - updating an addon could take an hour in some instances. On other sites running 1.4.x, I'm using the same Mandrill email API to send e-mails in seconds.

It seems like anything that uses the admin defer system on our server is very slow, anything else is fast. I'm happy to set up a demo board somewhere if that will help show the problem.

 

[Mike]

If you want to do a new install somewhere, I can look at it. I'd need FTP access (and ACP to trigger the rebuild consistently).

 

[nrep]

Thanks Mike, I'll work on setting up a test site where we can replicate this.