• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

As designed Thread updates unexpectedly slow as thread increases in size

Xon

Well-known member
#1
The following operations are gated by thread reply length, and can take massive amounts of time:
  • Moving a thread between forums
  • Changing the visibility state of a thread
This is because in XenForo_DataWriter_Discussion::_insertIntoSearchIndex, the call to XenForo_DataWriter_Discussion::_getDiscussionMessageIds fetches all the posts (even if it is the minimum set of data), which can result in a lengthy query.
 

Mike

XenForo developer
Staff member
#2
This is really expected. These elements need the IDs to queue up the necessary rebuilds. These actions should be fairly uncommon compared to replying to a thread, for example. There are things that should be expected to have linear slowdowns/memory use increases as size increases.

I would note that we don't really consider threads with 10s or 100s of thousands of replies to necessarily be a "normal" use case, though we're aware they do happen, but challenges can be expected when they get quite large.
 

teletubbi

Well-known member
#3
By the way, is there some internal limit for the size of a thread?
Or is there an estimated size when a thread will become slow in response?
 

Chris D

XenForo developer
Staff member
#4
Due to the way that XF stores the post position in the post table, there shouldn't really be a noticeable performance degradation in terms of inserting a new post or viewing a high numbered page. But, clearly, certain actions (as reported here) can have an impact but they should be rare enough to not be an ongoing concern.
 

Xon

Well-known member
#6
This is really expected. These elements need the IDs to queue up the necessary rebuilds. These actions should be fairly uncommon compared to replying to a thread, for example. There are things that should be expected to have linear slowdowns/memory use increases as size increases.

I would note that we don't really consider threads with 10s or 100s of thousands of replies to necessarily be a "normal" use case, though we're aware they do happen, but challenges can be expected when they get quite large.
Fair enough.

About the only solution would be to use a sliding window of updating a smaller batch at a time rather than the entire thread. This would also help prevent page timeouts when doing a manual search re-indexing of threads. Since for example CloudFlare has fairly strict limits for long an idle HTTP connection can stay alive.
 

imthebest

Formerly Super120
#7
Since for example CloudFlare has fairly strict limits for long an idle HTTP connection can stay alive.
Yep, I believe it is around 30 seconds. When I imported using the standard web importer I discovered the issue, and had no option but to temporarily disable CF while the migration was performed (announcing my IP to the world :mad:).