Thousands of forums?

[Sim]

I run a rapidly growing (and heavily customised) vBulletin v3.8.x site which has a very large number of forum topics - more than 1,000 of them all up. Server performance of this site is quite poor with high loads and pages taking a while to load. Memory usage is also very high.

I have another vB forum which sees quite a bit more traffic, with 5 times as many users and 2 times as many posts and about 50% more monthly page views, but with only a fairly "typical" number of forums (say, 10-15) ... which runs very smoothly with very little load on the server by comparison - so I know that first forum is struggling given the traffic.

vBulletin loads details of all forums into memory with every page load - so having more than 1,000 of them is not really a good idea and (amongst other things), is where I suspect a lot of the load (and memory usage) is coming from.

I'm yet to come up with a more elegant way of managing the volume of posts and threads on this site without having so many forums (essentially, each forum represents a zoo from around the world, and there are hundreds, sometimes thousands, of threads in each forum - each one discussing a photo taken at one of those zoos - at one point we had over 120,000 photos on the site).

I own several XenForo licenses and do intend to migrate several of my sites across, but I haven't had time to spend getting to know the code yet and understanding how it all hangs together.

I'm wondering if anyone has done any testing with large numbers of forums on a XenForo site, or if anyone has enough knowledge of the way XenForo is architected to predict how it might react if we tried creating such a large number of forums?

I'm trying to work out a migration strategy - I either try migrating to XenForo first and then fix the unknown problems I might come across (hence this post - I don't know enough to predict problems yet!) ... or I try re-architecting on vBulletin first before migrating if we know XenForo is likely to struggle on my setup as it is now.

Thoughts?

 

[SchmitzIT]

Sounds like a pickle. We run one forum that has a large number (in my mind, but less than 200) forums/nodes and usergroups, and that at times gives us trouble when we alter permissions in XF.

As for the second part of your post, I would advise you to restructure on vB before doing the migration. Purely so the import into XF will be faster, due to having less to import and relink.

 

[Sim]

Thanks for the comments SchmitzIT.

I managed to minimise the number of forums for the main discussion area of the site by using vBulletin thread prefixes and using one forum per country (80+ forums as opposed to 1,000+). I then hacked the interface a bit to make it easy for users to switch between "country" mode (listing threads from all zoos within that country) and zoo mode (listing threads from a single zoo aka thread prefix). This seems to work pretty well as there are relatively few threads for each zoo - each thread tends to get a lot of posts. We also get the bonus of making it easy to manage threads which are not about a specific zoo, but are country specific.

However, for the photos section, each photo gets its own thread and there are relatively few posts in each thread, but lots of photos, so lots of threads for each zoo. To have them all in one "country" forum doesn't make a lot of sense since photos only ever belong to a single zoo and there would just be a heap of clutter if we tried to include all photos from one country in one forum - it's not uncommon for a user to upload 200+ photos at time from a recent trip.

If there was a reasonable way of categorising the (photo) threads such that we could realistically have them in a single forum per country, we could potentially get away with less than 100 forums.

I think I need to spend some time getting up to speed with XF to understand what might be possible.

To make matters worse, my hosting provider is putting pressure on me to move the site to a dedicated solution due to the load - but I know that the traffic doesn't yet justify the expense, especially compared to my other busier forum which is running fine on another server with the same host.

 

[SchmitzIT]

Just as a heads-up, but at this very moment, XF does not support thread prefixes, but will when 1.1 comes around (at least, if I remember correctly).

Also, at this moment there is no gallery solution for XF yet. Out of curiosity, the pictures, are they stored in the database, or the file system? If for whatever reason you do use the database, it might help a lot if you instead go for the filesystem solution.

 

[Sim]

Yes, I was aware of thread prefix limitations - I was actually hoping to avoid using them if possible, but it still might be the best solution once they do make it in to the product.

The galleries on my vB board use a heavily hacked version of PhotoPost (essentially, I just use the PhotoPost upload and database, but all display is via custom code integrated directly into the vB forums). Photos are all stored in the file system.

 

[SchmitzIT]

That's at least something, then (The filesystem being used for storage that is).

The problem could be that perhaps the code is far from optimized, or simply that the site is constantly trying to load pictures. Are you willing to share the URL to your site, perhaps? It might help narrow things down.

Does the "good performing site" also uses that custom code?