As designed Thread updates unexpectedly slow as thread increases in size

[Xon]

The following operations are gated by thread reply length, and can take massive amounts of time:

• Moving a thread between forums
• Changing the visibility state of a thread

This is because in XenForo_DataWriter_Discussion::_insertIntoSearchIndex, the call to XenForo_DataWriter_Discussion::_getDiscussionMessageIds fetches all the posts (even if it is the minimum set of data), which can result in a lengthy query.

 

[Mike]

This is really expected. These elements need the IDs to queue up the necessary rebuilds. These actions should be fairly uncommon compared to replying to a thread, for example. There are things that should be expected to have linear slowdowns/memory use increases as size increases.

I would note that we don't really consider threads with 10s or 100s of thousands of replies to necessarily be a "normal" use case, though we're aware they do happen, but challenges can be expected when they get quite large.

 

[teletubbi]

By the way, is there some internal limit for the size of a thread?
Or is there an estimated size when a thread will become slow in response?

 

[Chris D]

Due to the way that XF stores the post position in the post table, there shouldn't really be a noticeable performance degradation in terms of inserting a new post or viewing a high numbered page. But, clearly, certain actions (as reported here) can have an impact but they should be rare enough to not be an ongoing concern.

 

[teletubbi]

This answer is good enough for me.
I was thinking about to split some threads. Looks like i can do other things instead.

 

[Xon]

This is really expected. These elements need the IDs to queue up the necessary rebuilds. These actions should be fairly uncommon compared to replying to a thread, for example. There are things that should be expected to have linear slowdowns/memory use increases as size increases.

I would note that we don't really consider threads with 10s or 100s of thousands of replies to necessarily be a "normal" use case, though we're aware they do happen, but challenges can be expected when they get quite large.

Fair enough.

About the only solution would be to use a sliding window of updating a smaller batch at a time rather than the entire thread. This would also help prevent page timeouts when doing a manual search re-indexing of threads. Since for example CloudFlare has fairly strict limits for long an idle HTTP connection can stay alive.

 

[imthebest]

Since for example CloudFlare has fairly strict limits for long an idle HTTP connection can stay alive.

Yep, I believe it is around 30 seconds. When I imported using the standard web importer I discovered the issue, and had no option but to temporarily disable CF while the migration was performed (announcing my IP to the world ).