XF 2.0 300k threads, 5M posts

[mazdas247]

I have a vb4 system with 5M posts and 300k threads. Running on a beefy enough system that frontend VM unix load is 1 with 8 VCPU, and db VM unix load is 0.2 with 4 VCPU.

I am a little concerned about a few things:

• iSCSI SAN has 10+ spindles so sequential bursty access is fast, but random access and throughput sucks, limited by GbE
• The live VB system is super performant, but only because I allocate enough RAM for mysql to cache the entire database into memory, so SAN is irrelevant for read (forum is mostly read access, 10:1 guest to member ratio).
• I'm afraid that 5M writes to SAN will take forever. Assuming it took 10ms per post (am I dreaming in colors?) it's still 12 hours!
• I have 60 GB of attachments, because reason, will this ruin my life?
• It took 1 hour to import my 130k users, I hope the user import procedure is more complex than that of posts, or I am in trouble

So... could anyone give me general expectation of how long this could take to import? Hours? Days? Weeks? Fugget-about-it? Do I start taking prozac now?

What can I do to help my odds?

1. Yes I read the comment about co-locating DB on the same machine as XF for import purposes. But I can't really do that, the DB server does DB and the web server does web. But of course I can do that, if I temporarily set up a mysql replica on the web node. Setting up mysql/innodb replicas is not in my top 10 things I enjoy doing, so, would this be worth the pain?
2. What if I allocate crazy ram to my VM, do the import to a DB backed by ramdisk, then when completed replicate it to the DB node?

Looking forward to this migration though, XF 2 looks awesome both front and backend especially when compared to VB4.

 

[fotografik]

We recently migrated from VB4 to XF2, and we had to take a slightly indirect route, see this post – Link.

Long story short – you may need to use the Big Board Importer to perform the same steps as us.

 

[Aayush]

How about provision another DB VM. Run a test migration on that DB.
Make sure everything is in order and then move the database to the original VM/change the database connection in config.

 

[mazdas247]

I would also mention that I'd like to keep read-only time to a minimum, I can probably do a day... but not days.

 

[mazdas247]

BTW @fotografik thanks this is exactly the insight I was looking for...

 

[fotografik]

Most welcome, glad to be able to offer some assistance.

We are hosted on AWS and have specified SSDs for both the web and database instances. During the migration process, we increased the CPU and memory size of the DB instance. We didn't run into any issue reading data off the DB as part of the export process nor any issue writing data back to a new DB for the XF import portion.

Best if you look into the specifics of the Big Board Importer and also engage Slavik's assistance to do the migration.

 

[mazdas247]

@fotografik thanks again. I work at a place that has a DC so we are on a cluster and I can scale up and down as much as needed and I will heed your advice there.

So, I have purchased that "big board importer" and have been playing with it. Am I better off now? Probably. But before we go high fiving each other, let's be clear: it is a glorified 2000 line script which is by far not the best code (I am being nice) I've ever had the misfortune to have to fix. For 150$. Let me be even more clear: I am paying someone for the privilege of fixing their code. This is bizarre. I posted nicely in the relevant thread (it has 56 pages no less!) to advise there are "issues" that I'd be willing to contribute the fixes, should anyone care. It's not free so I feel 150$ buys me some right to harp about it.

That said, I am grateful I don't have to dig down into the innards of XF and VB4 to reverse engineer what is going on. And yes, I am sure a fair amount of time went into this project, but it still feels like a macguyverism just thrown together. But still better off than where I was before, and price is not cheap, but probably still worth it.

Which brings me to my next point: this contraption script is leaps and bounds better than the built-in importer. So much that I'm willing to pay, in addition to put myself through the pain of fixing said script, installing the old XF, exporting, importing and upgrading it. And repeating this process N times until I get it right, preparing for the big day (thank god for docker).

That said there are 56 pages of posts in that importer discussion page and probably many people who shelled out 150$; the point of my rambling is:
XF looks great and feels years ahead of my VB4 setup, but is it mature? How is your performance on 9M posts? Similar to VB? Worse? Better? Thanks if you actually read my whole post.

 

[fotografik]

@fotografik thanks again. I work at a place that has a DC so we are on a cluster and I can scale up and down as much as needed and I will heed your advice there.

.. snip snip ..

That said there are 56 pages of posts in that importer discussion page and probably many people who shelled out 150$; the point of my rambling is:
XF looks great and feels years ahead of my VB4 setup, but is it mature? How is your performance on 9M posts? Similar to VB? Worse? Better? Thanks if you actually read my whole post.

I am not a code monkey by any definition of the word, so I am not qualified to comment on the quality (or lack thereof) of the code. All I know is that it worked for us. Disclosure: I did engage Slavik to do a trial import after I had issues configuring, and potentially he may have fixed the script specific to my situation - I did do a compare between the stock BBI files and the ones on the trial server but don't think there were major differences.

Anyhow, I am glad to have moved over to XenForo as it brought about several major benefits that we are beginning to see in terms of increased member participation and posts even just after one short month. Being a photography forum, the one biggest thing the members have taken advantage of is the ease of attaching images to threads and posts.

We are also saving on the hosting costs as we are now on a smaller instance (and only using 10-15% of CPU at peak at that, investigating if can go smaller but am afraid the combination of either less CPU or less memory may impact performance). I am now going about consolidating some of the other various servers we had implemented (eg a standalone photo gallery server) and will look to save potentially 30-40%.

Maturity of XenForo is definitely there and we deliberately delayed our migration to ensure that some of the key functionality of the VB4 platform was available. For example, we had a custom forms-based posting script developed for our Personal Classifieds sections (man, it was a pain to modify and add to the templates - can you imagine nested IFs that are 12 levels deep?), and the XF1.5 version did not do what we wanted. The XF2 version did, and even better, the add-on developer added some features based on my and another members feedback which improved its functionality tremendously.

There are other Quality Of Life benefits that XenForo brings as well - SEO, easier styling, built-in Enhanced Search (we were using Digital Point's Sphinx plugin on VB4), easier group/node permissions once you get your head around the cascading permissions method. Performance-wise, yeah maybe the site responds that little bit faster now but we were using Nginx and various caching tools (DragonBytes vBOptimise plugin) and performance testing on Webpagetest and Pingdom shows very similar results, so it's a wash in my view.

Having said that, site and page responsiveness on XenForo is exemplary, moving from page to page and refreshing is lightning quick for our site. Images load very quickly (unless it has not been proxy-cached due to SSL), but I put that down to Nginx, php-fpm, fastcgi caching and opcache/memcache all doing their work.

Another proof point on performance - we submitted our sitemap to Google for indexing and the Googlebots have been hitting our site really hard between 3 April and now. There was a week where indexing was going at a rate of 500,000+ pages a day with an average transfer of 10GB data daily. Our site is 95% re-indexed by Google now.

Phew, sorry for the War and Peace post, but I am very happy to share our experiences.

 

[mazdas247]

Indeed Tolstoy would be proud. Thanks again.

 

[mazdas247]

@fotografik FYI, after days of fighting random bugs in the big board importer, I threw it on the ground in addition to the 150$ I threw in the garbage and I focused on the normal importer of XF2. In there I found many inefficiencies I patched, and was able to import all of the data, users, threads, posts, attachments, forums, etc... in 16 hours, which is very acceptable to me.

I am still troubleshooting some stuff, but the data is all essentially there.