How do you manage large amounts of structured data in XenForo?

[KaranGupta]

As XenForo communities grow, they can end up managing a surprisingly large amount of structured content: threads, posts, user data, custom fields, resources, media, tags, and other metadata.

I’m curious how administrators here approach data management when a community becomes large.

For example:

• How do you keep search responsive as the database grows?
• Do you use Elasticsearch or another search solution?
• How do you handle large media libraries?
• Do you keep additional structured data in XenForo or in a separate application?
• Have you had to connect XenForo with an external system through APIs or webhooks?
• At what point did database performance or data organization become a concern?

I’ve also been looking at how similar problems are handled in other software that manages large structured catalogs. PIM platforms such as Akeneo, Pimcore, Salsify, and UnoPIM, for example, separate structured data, search, media, and integrations rather than putting everything into one layer.

Obviously the use case is different, but some of the architecture questions seem surprisingly similar.

For those running larger XenForo communities, what has your setup evolved into, and what would you change if you were starting again?

 

[briansol]

• Enhanced Search
• All attachment data on AWS S3 or CF R2

 

[Suzanne O]

For me personally i limit the length of chat threads.

For example if a chat thread reaches 20 pages long i lock it and start another thread.

 

[briansol]

For me personally i limit the length of chat threads.

For example if a chat thread reaches 20 pages long i lock it and start another thread.

Totally unnecessary.
one of my big threads. no issues from it.

 

[Suzanne O]

Totally unnecessary.
one of my big threads. no issues from it.
View attachment 346076

How big is your server dude?
We have a smallish server and have to actually make those changes.
I saw you're with AWS.
A mate of mine was with them (unsure if they still with them) but they close threads up as well.

 

[briansol]

5.99 a moth vps from ovh

 

[BrettC]

For me personally i limit the length of chat threads.

For example if a chat thread reaches 20 pages long i lock it and start another thread.

How big is your server dude?
We have a smallish server and have to actually make those changes.
I saw you're with AWS.
A mate of mine was with them (unsure if they still with them) but they close threads up as well.

Back in the day with vBulletin, this was an issue with very large (incredibly large) threads. If memory serves right, per-page post limits had to be reduced.

With XenForo 1.x and 2.x series, I've legitimately never observed a slow down with very large threads.


As for OP...

• How do you keep search responsive as the database grows?
  • Once the generic full-text search for users exceeds an average of 2 to 5 seconds 'wait time', other alternatives such as ES is considered and/or used.
  • This transition tends to take place based on:
    1. Hardware of SQL DB (Disk i/o speed and RAM capacity primarily) - the "throw beefier hardware at the problem" solution,
    2. How many total posts the board has, generally around one million post IDs is when a transition has to take place. A physical limitation kicks in once the FULLTEXT search surpasses sane levels of wait-times.
• Do you use Elasticsearch or another search solution?
  • ES, yes.
• How do you handle large media libraries?
  • AWS S3 + Cloudfront. (Any block storage works really)
• Do you keep additional structured data in XenForo or in a separate application?
  • In what sense? Talking containers? Attached storage?
• Have you had to connect XenForo with an external system through APIs or webhooks?
  • Connected Accounts system. Bridged user auth from a central source.
• At what point did database performance or data organization become a concern?
  • Never had an issue, because I know the limitations (or the general upper-end) of set configurations and the hardware in use.

OPs post sort of reads like a subliminal advertisement for competitor products and services, but i digress.

 

[kurarayypoval]

As XenForo communities grow, they can end up managing a surprisingly large amount of structured content: threads, posts, user data, custom fields, resources, media, tags, and other metadata.

I’m curious how administrators here approach data management when a community becomes large.

For example:

• How do you keep search responsive as the database grows?
• Do you use Elasticsearch or another search solution?
• How do you handle large media libraries?
• Do you keep additional structured data in XenForo or in a separate application?
• Have you had to connect XenForo with an external system through APIs or webhooks?
• At what point did database performance or data organization become a concern?

I’ve also been looking at how similar problems are handled in other software that manages large structured catalogs. PIM platforms such as Akeneo, Pimcore, Salsify, and UnoPIM, for example, separate structured data, search, media, and integrations rather than putting everything into one layer.

Obviously the use case is different, but some of the architecture questions seem surprisingly similar.

For those running larger XenForo communities, what has your setup evolved into, and what would you change if you were starting again?

Interesting comparison with PIM platforms. We’d be curious which became the first bottleneck for larger communities: search, media storage, or database queries and which change made the biggest difference.

Team Kuraray Poval

 

[Anthony Parsons]

• How do you keep search responsive as the database grows?

Elasticsearch does that.

• Do you use Elasticsearch or another search solution?

Yes. You have to. MySQL search is ok for a small amount of data, but you will crash it with large amounts simultaneously searched. If you added graph query engine to MySQL, then you would get it pretty close to to ES or OpenSearch. I use OpenSearch on another project... it performs insanely well. They're a split project to my knowledge, so they both perform equally well.

• How do you handle large media libraries?

If you run an image intensive forum, put all media via an R2 bucket.

• Do you keep additional structured data in XenForo or in a separate application?

Text is the easiest to store.

• Have you had to connect XenForo with an external system through APIs or webhooks?

No.

• At what point did database performance or data organization become a concern?

I don't remember exactly, but it was roughly around that 100k to 200k post mark that things became slow. Whilst cache, Redis, etc, can help immensely, the amount of data, you will hit a limit and want a search purposeful search engine.

 

[Digital Doctor]

Xenforo lacks structured data.

The default content vehicle (unthreaded threads) has essentially no structure.