1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.4 Slow rebuilds/reparsing

Discussion in 'Troubleshooting and Problems' started by nrep, Apr 30, 2015.

  1. nrep

    nrep Well-Known Member

    I'm experiencing an usual problem with XF addon installs/upgrades which has become much more pronounced since switching to a Windows 2012 server. The rebuilding/reparsing stage can easily take over an hour per site, even when the site load is extremely low (the server is plenty powerful - Dual Hex Core Xeon 2620, 32GB Ram, RAID 1 SSDs, etc...).

    We're only running 1 style and anywhere from 6-20 addons per site. This problem happens on all XF installations, but the number of addons does seem to make a difference.

    Has anyone experienced unusually slow rebuilds or know where I can look in to where the delays are coming from? I've had a look at the MySQL monitor and the load barely moves, but I don't know where else to start searching.
     
  2. Brogan

    Brogan XenForo Moderator Staff Member

    Have you carried out any server tuning?
     
  3. imthebest

    imthebest Formerly Super120

    Do you have anything logged in your slow query log? Have you checked the status/health of your RAID array? How big is your database?

    It sounds to me like a hardware problem... probably related to your SSDs or RAID configuration.
     
  4. nrep

    nrep Well-Known Member

    Yep, the server has been recently optimised and the isn't anything in the slow query log around the time of the rebuilds. I've just logged in to manually verify the RAID status, and that's fine too. Just to double-check performance, I ran PassMark and it performs as expected in the tests.

    There are several of XF sites on the server, with DBs ranging from tens of MB to tens of GB. The re-build time doesn't seem to be affected by DB size from what I can tell.
     
  5. Brogan

    Brogan XenForo Moderator Staff Member

    Rebuilding time is heavily affected by the number of add-ons, styles, and languages.

    I have seen some servers take up to an hour to do a rebuild but those are typically related to cheap shared hosting.

    Typically, a couple of styles and languages with a handful of add-ons shouldn't take more than a few minutes.
     
  6. nrep

    nrep Well-Known Member

    Is there any way I can trace what's taking so long?
     
  7. Mike

    Mike XenForo Developer Staff Member

    The only tracing that can be done is really via developer facing tools (such as XDebug) and would need someone familiar with the code to analyze.

    However, issues such as this almost always point to MySQL configuration issues. While you can do things to speed up PHP initialization/compilation, aside from PHP version, the run time is mostly just down to the language. MySQL has huge tuning variables though. The big ones for XF are discussed here: http://www.percona.com/blog/2013/09/20/innodb-performance-optimization-basics-updated/

    Is MySQL running locally? If it's not, you may need to consider if there is any network overhead. The rebuild steps run a very significant number of queries so latency to the DB server would be a significant bottleneck here. Reducing that would help this step (and all XF pages really).
     
  8. nrep

    nrep Well-Known Member

    Thanks Mike. I've got XDebug installed (but not running), so I'll see what I can capture and see if anything obvious stands out.

    MySQL is running locally, with all tables InnoDB (except any marked as memory).

    I'm currently deploying a backup image to a local test server so I can try changing things and see if there's any performance improvement. We've got Zend OpCache and WinCache running, so I'll pare things back to eliminate any incompatibility issues.

    As soon as I've got any data, I'll report back.
     
  9. Mike

    Mike XenForo Developer Staff Member

    This is something you'd likely see on most pages, but you may wish to try connecting to MySQL via "127.0.0.1" instead of "localhost". I have seen a few cases where (I think) localhost attempted to connect via IPv6 creating a delay before it fell back to IPv4 (where MySQL was listening). If it's just rebuilds, that may not be related though.
     
  10. nrep

    nrep Well-Known Member

    Sorry for the slow reply, I've been taking my time to debug this and narrow down the problem. After much playing around on a test server snapshot, it seems that the problem lies with IIS.

    I've installed a basic, non-optimised version of Apache and it is FAR faster at rebuilding than IIS is (both share the same MySQL service).

    I've read all of the IIS guides I can find with regards to optimisation, but the settings seem fine from what I can tell. I've experienced a few oddities with IIS (such as random page not found errors, fixed thanks to some code for config.php from Mike).

    Is there anyone here familiar with IIS and can point me in the right direction to some monitoring tools that may help diagnose the problem, now that it's narrowed down. GGoogle brings up basic app pool monitoring stuff, which isn't much help.
     
  11. nrep

    nrep Well-Known Member

    I've been struggling with this issue for a long while now, but was able to ignore it by avoiding updating addons frequently.

    However, now that I've upgraded some sites to XF 1.5, I've noticed a huge slowdown when sending bulk e-mails. It then occurred to me that it's not the rebuilding/reparsing stage, but anything that uses the deferred system.

    Is there anything that springs to mind in the deferred system that I can disable/test? I get this same problem with zero addons enabled.
     
  12. Mike

    Mike XenForo Developer Staff Member

    There should be minimal overhead to the deferred system itself. The email sending process is pretty much identical, though before it would send X number of page, rather than sending as many as it can in a certain period. You could potentially increase the length a deferred process runs via config.php as long as you ensure any timeouts outside of XF aren't triggered (FastCGI timeouts, for example).
     
  13. nrep

    nrep Well-Known Member

    Thanks @Mike, I'll try extending the length out of curiosity and report back ($config['rebuildMaxExecution'] I guess?). I'm part way through an e-mail send (which I guess will take until tomorrow), so I'll do it once that is finished.

    On a 1.4 site that sends out e-mails, I'm doing just under 100 per second. On 1.5 I'm doing about 2 per second.

    I've disabled all addons and the problem still occurs, so I'm at a loss. Is there anything I can do to break the bulk e-mail sending out of the deferred system? Or perhaps a way to debug/inspect the deferred system?
     
  14. Mike

    Mike XenForo Developer Staff Member

    That sounds like a speed issue with the actual email send. What version of 1.5 are you using, to confirm?

    You'd really have to debug by inserting debugging code within the job itself. It's unlikely to be the deferred system itself causing that level of a slow down IMO.
     
  15. nrep

    nrep Well-Known Member

    We're on 1.5.4

    I'm really puzzled as anything using the admin defer system is very slow, not just e-mail sends - updating an addon could take an hour in some instances. On other sites running 1.4.x, I'm using the same Mandrill email API to send e-mails in seconds.

    It seems like anything that uses the admin defer system on our server is very slow, anything else is fast. I'm happy to set up a demo board somewhere if that will help show the problem.
     
  16. Mike

    Mike XenForo Developer Staff Member

    If you want to do a new install somewhere, I can look at it. I'd need FTP access (and ACP to trigger the rebuild consistently).
     
  17. nrep

    nrep Well-Known Member

    Thanks Mike, I'll work on setting up a test site where we can replicate this.
     

Share This Page