Xenforo, bigboard, big topics

[tintolio]

Hello,

I am currently with IPB 3.3 and since I've been having several problems I would like to transfer my board to XenForo.

According to the IPB people, the problem is the huge database we have (20 GB) and some huge topics created in the forum, for example, one topic has 89,991 replies and 16,757,907 views, another 29,233 replies and 345,511 views, and so on.

My community currently has over 400,000 registered users and over 17,000,000 posts. The average of online users per day is 1000 with peaks between 1500 to 2000. I mentioned this because I have seen many forums with larger numbers than mine so I should be fine with this but my main concern is the issue with my topics with a big number of replies. Is this still a big issue if I transfer my board to XenForo?

Big topics really are a problem with Invision. Once you hit over 50k replies, you're on you own. Bug in practise, even just more than 10k replies in one topic bring those slow queries and kill the entire board with it.
We pushed several tickets and waited over several big version jumps, from 3.1 to 3.4, and nothing about this performance issues has ever happen. In fact, they just put even more features an load into topic views, and appears like they don't really get or care about this problem.
Their response and solution was to tell us to delete posts inside those big topics (their moderation tools are not very well polished to do something like that in batch). Or even to delete those threads and start over.

There one in particular that it's VERY important but in terms of newcomers. It drive traffic as hell from google to us.

Can we expect problems like this on xenforo? i mean with really big topics, with more than 50k replies on it?.


Also, how long does the transfer from IPB to XenForo take?

We already have a big tweaked bunch of servers, now on AWS.
We use nginx. Have memcached installed. And a big etc. of tweaks and enhancements over the years.

Thank you in advance.

 

[Digital Doctor]

Big topics really are a problem with Invision. Once you hit over 50k replies, you're on you own.

I've never understood why anyone would want such large topics ?
Might the easiest thing to do be trim those topics and stay with IPB?
Should topics that big get their own subforum ?
Is information even remotely findable in a topic with 50K replies ?

I find IPB slooooow.

I've been on an IPB forum alot lately.
It's just slow everywhere.

The forum stats are:
8089 users are online (in the past 90 minutes)
1449 members, 6600 guests, 40 anonymous users
4,518,157 Total Posts
140,009 Total Members
18,069 Most Online

 

[Dean]

I really do not have the answers, but I went ahead and changed the thread title from 'bogboard' to 'bigboard' because I am fairly sure that is what you meant. If you want the title changed for any reason please click on 'Report' and a moderator can change it for you.

 

[Digital Doctor]

bogboard = bigboard getting bogged down. bogboard.

There is a thread where the bigboard admins hangout.
http://xenforo.com/community/threads/the-big-forum-list.26352/

 

[craigiri]

Tin, do you think this particular problem is just mysql? Or is it a bug of some sort in IPB?

There is a limit somewhere as to what mysql can do......at least without a lot of custom tuning.

If I were you, I'd ask Jake and Shawn (digitalpoint) to weigh in with their opinions. Your site, as far as those threads and overall db, definitely falls into the top 1%...so, frankly, most folks would not have experience with those particular problems!

Facebook had to build some crazy server farms as well as using stuff like FusionIO (very high speed SSD) to get performance. You may be entering the realm of some very custom tuning.

 

[CTXMedia]

http://www.cyclechat.net/threads/post-a-lie-about-the-poster-above.35674/ - 30,313 replies - 331,344 views - how does it seem (speed wise) when you navigate it?

 

[CTXMedia]

http://www.cyclechat.net/threads/post-a-lie-about-the-poster-above.35674/ - 30,313 replies - 331,344 views - how does it seem (speed wise) when you navigate it?

Oh, and this is on a five year old 2 core server with 8GB and several other XF sites running on it.

I expect it'll work much faster when our new 64GB 12 core SSD server arrives in a few weeks.

 

[craigiri]

That seems quite quick, click - but I suspect the 20G database tin has may be slowing his down a bit also.

 

[p4guru]

surprised no forum has native support for splitting large post threads off into a separate thread once they hit a preset admin defined number of posts threshold

so if you set max number of posts per thread as 1000 posts max, at 1001 posts software will automatically close the original thread and add a link to the new thread to continue the conversation

Also it isn't just about size of threads in terms of post replies but concurrency - that is how many visitors are concurrently reading those large post count/reply threads. You could have a thread with 100,000 replies and if only a max of 1 visitor at a time views it the resource requirements wouldn't be anywhere near what it would require if you have 500 concurrent visitors hitting that thread.

 

[tintolio]

surprised no forum has native support for splitting large post threads off into a separate thread once they hit a preset admin defined number of posts threshold

so if you set max number of posts per thread as 1000 posts max, at 1001 posts software will automatically close the original thread and add a link to the new thread to continue the conversation

Also it isn't just about size of threads in terms of post replies but concurrency - that is how many visitors are concurrently reading those large post count/reply threads. You could have a thread with 100,000 replies and if only a max of 1 visitor at a time views it the resource requirements wouldn't be anywhere near what it would require if you have 500 concurrent visitors hitting that thread.

True. We have problems when more than 200 concurrent users (guests and visits) hit the same big thread. We even have spikes of 400 or 500 users viewing some of those big topics.

We ask IPB if we can have a solution like the one you mention, p4guru. Auto-splitting when a topic reach a defined threshold. That would work fine for us if is automatic and the old topic could have some sort of auto-lastpost with the link to the new one (ideally). But nothing happened on 3.2, 3.3 or the new 3.4. So, we are still stuck with this problem on Invision.

We convert from vb3 to IPB3.1 (because of the mess IB put into the whole vB development, insecurity, unhappiness about it, etc). And before we knew about Xenforo :/
But with vB3 we really have no mayor problem with those topics causing slow queries. Not as notorious as IPB has.

 

[Digital Doctor]

Auto-splitting when a topic reach a defined threshold.

?
Lock the thread. Make a new one. Done.

 

[tintolio]

?
Lock the thread. Make a new one. Done.

Digital Doctor, are you a member of IPB's support?.
That was the very same first answer (kind of) we get from them long time ago

Certainly we can accomplish something like that. But with several topics like those per week, it's really something we hope would be fixed programmatically.
It's exhausting doing it by hand, specially when you deal with thousands of posts and users per week.

 

[Slavik]

Hi Tintolio,

In general from my own experiences (and the experiences of others) IPB is much heavier on a servers resources than XenForo.

Simply transfering to the XenForo platform should provide you with an increase in performance.

To answer your questions,

My community currently has over 400,000 registered users and over 17,000,000 posts. The average of online users per day is 1000 with peaks between 1500 to 2000. I mentioned this because I have seen many forums with larger numbers than mine so I should be fine with this but my main concern is the issue with my topics with a big number of replies. Is this still a big issue if I transfer my board to XenForo?

17,000,000 posts is indeed impressive, however there are several large boards such as yours running the XenForo platform, including IGN with 85+ million posts. XenForo will handle your board easily.

Big topics really are a problem with Invision. Once you hit over 50k replies, you're on you own. Bug in practise, even just more than 10k replies in one topic bring those slow queries and kill the entire board with it. We pushed several tickets and waited over several big version jumps, from 3.1 to 3.4, and nothing about this performance issues has ever happen. In fact, they just put even more features an load into topic views, and appears like they don't really get or care about this problem.
Their response and solution was to tell us to delete posts inside those big topics (their moderation tools are not very well polished to do something like that in batch). Or even to delete those threads and start over.

Having support tell you to delete content rather than deal with the issue at hand is not on indeed! However again XenForo seems to handle large topics very easily. As previously posted, Cyclechat has a very large thread which works just fine.

There one in particular that it's VERY important but in terms of newcomers. It drive traffic as hell from google to us.

XenForo uses up to date and modern SEO. It is arguably the best built in SEO of any forum currently available. Using the redirect scripts available, you would also maintain all your search engine rankings.

Can we expect problems like this on xenforo? i mean with really big topics, with more than 50k replies on it?

The software has shown itself to be more than capable, so the answer should be no, but as you are no doubt aware large forums often have a life of their own outside of the smaller forum market. However haivng said that the support available here at XenForo is exceptional, and very knowlegable members can often help resolve any issues, and any tickets you put in we aim to answer as quickly as is possible.

Also, how long does the transfer from IPB to XenForo take?

We already have a big tweaked bunch of servers, now on AWS.
We use nginx. Have memcached installed. And a big etc. of tweaks and enhancements over the years.

Thank you in advance.

The import process varies between boards and the hardware available to you. So we wouldn't be able to give you a definative answer on this. I haven't used AWS myself but I imagine if you can bump up your memory for a short period and increase the importer packet sizes you would be able to get an import done in under 24 hours.

 

[tintolio]

Thank you all for your responses.

I hope soon to have in my community XenForo

Thanks again
regards
Tintolio

 

[digitalpoint]

One thing I would suggest (no matter what platform you are using)... get more memory for your DB server. You can never have too much, and 8GB for MySQL when it has 20GB worth of databases is not a good situation. If you have the ability to slap 64GB or even 128GB into that server, I would.