XF 1.2 Cache Rebuilding Process Takes Ages

[yavuz]

This is happening for quite some. It didn't bother me until I started to receive 500 internal server errors during an add-on installation.

I have 60 add-ons, a compilation of simple and detailed ones (don't know how else to describe them) installed on my board. In the past I have installed and uninstalled a variety of styles and add-ons but currently there are also two styles installed which have two child styles.

Why is the rebuild process taking such a long time, especially when it comes to rebuilding templates?

Is there a way to detect, unused but in the database remaining templates or any other part of junk data that we can maybe flush out?

Thank You.

 

[yavuz]

@yavuz, Addon's installed?

60 addons in total.

 

[yavuz]

@Mike @Jeremy I've been doing some enhancements & performance tweaking and digging into the sites.

Currently I’m trying to find the best configuration for my MySQL server.

But please allow me to expand my findings: There are three XenForo sites on this server two share similar genres, near message count & user account, have same number of styles and languages installed. One of them is very slow in terms of add-on installation & rebuilding process. The same goes when someone wants to mark their forum read, it takes a lot of time.

The other forum in the same niche, is very fast. Add-on installation doesn't take more than 30 seconds most at the time, marking forums read is also fast.

The third website is tad different, has more users & posts and threads. More add-ons installed. However it shares the same fate with the other slow XenForo board.

What I’ve done, beside the optimizing the header, expires, putting it behind CloudFlare, loading content from a cookieless domain, having memcached and ACP configured, I've also disabled the listeners and to find out if that made any difference, it doesn’t.

I’m clueless about why this is happening. I hope there is a way we can determine the cause of this slowness.

@Mike @Jeremy

I couldn't get a response to my latest post therefore bumping this up. Since then I changed to Percona and made my.cnf enhancements which had little affect. Marking threads as read is sill really slow.

 

[Mike]

There isn't really an easy or straightforward way to identify specific areas of slowness vs another board. It basically involves a lot of analysis and time.

Note that things will take different lengths of time depending on various factors. Marking all forums read depends on the number of forums you have, for example.

 

[yavuz]

There isn't really an easy or straightforward way to identify specific areas of slowness vs another board. It basically involves a lot of analysis and time.

Note that things will take different lengths of time depending on various factors. Marking all forums read depends on the number of forums you have, for example.

I glanced at the forum number the "fast forum" has, it's more than the others. I've spend a lot of time looking for the root of the problem but nowhere near finding out. What else should I look for? Any help is appreciated. Thank You.

 

[Mike]

I can't say I have any particular further recommendations. Do double check add-ons.

There's something different if the performance difference is that drastic, but if they're on the same MySQL server it's not that (unless you have more data than fits in the cache and it has to load from disk). The read marking process is rather simple all told -- it really just consists of a loop over forums and then a DB write to mark it as read.

 

[Uniphix]

We have 4,000+ Users, 10+ Styles, 1 languages. Over 42,000 phrases. And it takes about 10 minutes to 15 minutes to rebuild the cache, to me there has to be a faster way to rebuild the cache data.

 

[Uniphix]

Now we have 267,633 phrases, that are not specific to an addon, they are generated by user content that our clients using our system. Trying to rebuild this takes almost 1 hour and 20 minutes now, and really starting to get annoying. There has got to be a way to speed this up. One of the issues I see, is suppose the master language has 137,236 phrases (yes that is right 137,236 phrases). Every time it gets ready to rebuild a set it has to LOAD all 137,236 phrases, before it compiles the list and by then it will have to break out because of the max execution timeout, therefore only being able to 'rebuild' a few phrases each cycle. XenForo has 4000+ phrases alone, my addon that I have is almost at 2,000 phrases. Our system is very unique and very large, it would be nice to talk with the XenForo Developers about improving the things that are taking just a day for me to update our production site.

The biggest problem I see with rebuilding phrases themselves is, 1) It has to PULL all phrases of the language it is working on (not caching or anything of that sort which to me is very overhead on the database itself). What would be nice is having it ONLY fetch a certain amount of phrases per cycle.

In concept

public function compileAllPhrases($maxExecution = 0, $startLanguage = 0, $startPhrase = 0)
    {
        $db = $this->_getDb();

        $languages = $this->_getLanguageModel()->getAllLanguages();
        $languageIds = array_merge(array(0), array_keys($languages));
        sort($languageIds);

        $lastLanguage = 0;
        $startTime = microtime(true);
        $complete = true;

        XenForo_Db::beginTransaction($db);

        foreach ($languageIds AS $languageId)
        {
            if ($languageId < $startLanguage)
            {
                continue;
            }

            $lastLanguage = $languageId;
            $lastPhrase = 0;

            $phrases = $this->getAllPhrasesInLanguage($languageId);
            foreach ($phrases AS $phrase)
            {
                $lastPhrase++;
                if ($languageId == $startLanguage && $lastPhrase < $startPhrase)
                {
                    continue;
                }

                $this->compileNamedPhraseInLanguageTree($phrase['title'], $phrase['language_id']);

                if ($maxExecution && (microtime(true) - $startTime) > $maxExecution)
                {
                    $complete = false;
                    break 2;
                }
            }
        }

        if ($complete)
        {
            $compiledRemove = $db->fetchAll("
                SELECT c.title, c.language_id
                FROM xf_phrase_compiled AS c
                LEFT JOIN xf_phrase_map AS m ON (c.title = m.title AND c.language_id = m.language_id)
                WHERE m.title IS NULL
            ");
            foreach ($compiledRemove AS $remove)
            {
                $db->delete('xf_phrase_compiled',
                    "language_id = " . $db->quote($remove['language_id']) . " AND title = " . $db->quote($remove['title'])
                );
            }

            $this->_getLanguageModel()->rebuildLanguageCaches();
        }

        XenForo_Db::commit($db);

        if ($complete)
        {
            return true;
        }
        else
        {
            return array($lastLanguage, $lastPhrase + 1);
        }
    }

Make it so that your not using PHP to determine what phrase you we're last updating and what not this is crazy and in my case really slows down the addon updates.

Use MySQL LIMIT and OFFSET to determine your updating compiling of the phrases.

 

[Uniphix]

Also doesn't help when it has to cache 21,298 total template files (ie S.0,L.0,account.css.php) for example