Migrating a large forum to XenForo (test migration)

[Marcus]

How did you tell google to search your pages more than usual? The last graph also looks familiar.

 

[Deebs]

How did you tell google to search your pages more than usual? The last graph also looks familiar.

Goto your webmaster control panel, click settings and then you can set the crawl rate.

 

[Mikey]

After further optimisation of MySQL here is the output of my latest XF CLI import (~ 4 million posts) :

View attachment 20012

I never understood this.

vBulletin, you have 4 million posts. Xenforo importer says it's converted just under 300k? So where's the other 3.7million gone? lol. Always put it down to a difference in how each system handles numbers.

deebs
You have over 9M posts yourself and the second poster has 33K posts?

Now that's dedication

 

[CTXMedia]

I never understood this.

vBulletin, you have 4 million posts. Xenforo importer says it's converted just under 300k? So where's the other 3.7million gone? lol. Always put it down to a difference in how each system handles numbers.

I always took the 300k to be thread total - with no actual post count shown by the importer.

 

[Mike]

I always took the 300k to be thread total - with no actual post count shown by the importer.

That's correct.

 

[Deebs]

I never understood this.

vBulletin, you have 4 million posts. Xenforo importer says it's converted just under 300k? So where's the other 3.7million gone? lol. Always put it down to a difference in how each system handles numbers.

Now that's dedication

The XF importer only lists the thread count, not the actual post count. My post count was artifically made high back in the day when post count farming was the fashion

I always took the 300k to be thread total - with no actual post count shown by the importer.

Yup, the posts are not displayed for the import.

 

[Deebs]

Time for another update (this is after I have migrated to XF, ignore the bump at the beginning of the graph :

 

[Andy.N]

I want that flat line

 

[AzzidReign]

Time for another update (this is after I have migrated to XF, ignore the bump at the beginning of the graph :View attachment 20632

That's amazing. I tried looking through your OP but there was so much information. Did you switch servers when you converted or kept the same exact set up?

 

[Deebs]

That's amazing. I tried looking through your OP but there was so much information. Did you switch servers when you converted or kept the same exact set up?

The only thing that I changed (before the migration) was the physical box hosting MySQL. Bear in mind that I do not run a stock MySQL but a Percona distro who specialise in optimising InnoDB (which XF uses).

Though to be honest I always look to better my performance everyday, at the SQL and webserver.

 

[CTXMedia]

I was tweaking my test XF install today and did a webpagespeedtest comparison between it and my IPB install; XF appears to be twice as fast - of course I won't know for sure until I do the full switch over but it looks good.

I've used a second domain to act as an in-house CDN (browsers will download asynchronously) which has helped, and have lots of caching.

Can't wait to get switched over.

 

[kmike]

I see that the attachments take a relatively long time to convert. Assuming the same performance as your process showed, and that it wouldn't degrade with the number of records, it would take ~100 hours to convert our ~4 million attachments. Hmm.

 

[Deebs]

Yesterday I finally decided to implement the following:

• Install memcached 1.4.9 (compiled from source)
• Install Libmemcached support in PHP (libmemcached is a newer library which supports more functionality than the older memcache PHP module)
• Install Igbinary. A much faster serializer/unserializer than the stock PHP one.

After a nanosecond outage I had converted from using APC for variable storage to memcached. Great stuff and everything continued to work.

Tonight I performed the following:

• Upgrade XenForo to 1.1 beta 4.
• Updated all changed templates.
• Implement Igbinary as the session handler for PHP.
• Ensure that Libmemcached has native support for Igbinary.

Again, everything went as smooth as smooth peanut butter Since I switched to memcached yesterday (24 hours ago) the stats for data written to and read from are (read Bytes Written as stuff read from the cache):

Bytes Read 89.9 MBytes
Bytes Written 11.6 GBytes

 

[CTXMedia]

Have you replaced APC with memcached - i.e. stopped using APC?

 

[Deebs]

Not for opcode caching but certainly for variable caching.

 

[CTXMedia]

Ah, okay, that's what I was just double-checking.

 

[Deebs]

Ah, okay, that's what I was just double-checking.

Check out the homepage loading speed: http://forums.freddyshouse.com

I have not really started to optimise the CDN or other stuff yet, I have a mixture of files served from my .net domain, some from the .com. The next few weeks will be about further optimizing the speed.

 

[Luke F]

I've found APC+igbinary > APC > libmemcached+igbinary > memcache in terms of speed for variable storage

 

[CTXMedia]

I have not really started to optimise the CDN or other stuff yet, I have a mixture of files served from my .net domain, some from the .com. The next few weeks will be about further optimizing the speed.

lol ... another speed freak ... <waves>.

I spent months shaving of ms here and there on my IPB board, but it's all worth it as the net effect of all the work combined made for a fast site.

Yours is looking pretty sharp already and loads really well, so further optimisation will be icing on the cake. Have fun.

 

[Deebs]

I've found APC+igbinary > APC > libmemcached+igbinary > memcache in terms of speed for variable storage

Interestingly I installed Igbinary and set PHP to use it as the session serializer, I then decided to use Igbinary as the serializer for APC expecting to see memory footprint reductions, but I didn't see that (well I believe I have got memory reductions looking at the graph but resident memory size of files hasn't changed according to apc.php. Speed wise it is very difficult to tell if there is speedup. One thing I did notice was that my fragmentation is now virtual non-existant as the following graphic shows (normally by this time it is fragmented to hell)