As designed Thread updates unexpectedly slow as thread increases in size

X

Xon

Well-known member
The following operations are gated by thread reply length, and can take massive amounts of time:
  • Moving a thread between forums
  • Changing the visibility state of a thread
This is because in XenForo_DataWriter_Discussion::_insertIntoSearchIndex, the call to XenForo_DataWriter_Discussion::_getDiscussionMessageIds fetches all the posts (even if it is the minimum set of data), which can result in a lengthy query.
 
This is really expected. These elements need the IDs to queue up the necessary rebuilds. These actions should be fairly uncommon compared to replying to a thread, for example. There are things that should be expected to have linear slowdowns/memory use increases as size increases.

I would note that we don't really consider threads with 10s or 100s of thousands of replies to necessarily be a "normal" use case, though we're aware they do happen, but challenges can be expected when they get quite large.
 
By the way, is there some internal limit for the size of a thread?
Or is there an estimated size when a thread will become slow in response?
 
Due to the way that XF stores the post position in the post table, there shouldn't really be a noticeable performance degradation in terms of inserting a new post or viewing a high numbered page. But, clearly, certain actions (as reported here) can have an impact but they should be rare enough to not be an ongoing concern.
 
This answer is good enough for me.
I was thinking about to split some threads. Looks like i can do other things instead.
 
Mike said:
This is really expected. These elements need the IDs to queue up the necessary rebuilds. These actions should be fairly uncommon compared to replying to a thread, for example. There are things that should be expected to have linear slowdowns/memory use increases as size increases.

I would note that we don't really consider threads with 10s or 100s of thousands of replies to necessarily be a "normal" use case, though we're aware they do happen, but challenges can be expected when they get quite large.
Click to expand...
Fair enough.

About the only solution would be to use a sliding window of updating a smaller batch at a time rather than the entire thread. This would also help prevent page timeouts when doing a manual search re-indexing of threads. Since for example CloudFlare has fairly strict limits for long an idle HTTP connection can stay alive.
 
Xon said:
Since for example CloudFlare has fairly strict limits for long an idle HTTP connection can stay alive.
Click to expand...

Yep, I believe it is around 30 seconds. When I imported using the standard web importer I discovered the issue, and had no option but to temporarily disable CF while the migration was performed (announcing my IP to the world :mad:).
 
Back
Top Bottom