1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Fixed Request times out when posting in *extremely* large threads

Discussion in 'Resolved Bug Reports' started by Mike Tougeron, Jan 27, 2012.

  1. Mike Tougeron

    Mike Tougeron Well-Known Member

    I know this is definitely an edge case but I came across it last night and it took me a few hours to research the problem so thought I should report it anyway.

    IGN has an extremely large thread (http://www.ign.com/boards/threads/»...to-feed-some-goddamn-fish-thread-««.77575142/) with 152,798 messages that was imported from our old boards system. Because of the way that XenForo handles the addition of a new message the queries timeout and the new message is not saved.

    I believe, but I'm not positive, that it is because of the following flow:
    Because it's a new message, XenForo_DataWriter_DiscussionMessage::_postSave() calls XenForo_DataWriter_DiscussionMessage::_updateDiscussionPostSave() which then calls XenForo_DataWriter_Discussion::updateCountersAfterMessageSave() which is acting on the thread's datawriter. So later in in XenForo_DataWriter_DiscussionMessage::_postSave() it calls XenForo_DataWriter_DiscussionMessage::_saveDiscussionDataWriter().

    Eventually it ends up calling XenForo_DataWriter_Discussion::_postSave(). This method does $messages = $this->_getMessagesInDiscussionSimple(); and that gets all the messages in the thread. In the case of this topic for IGN that means it tries to load all 152,798 messages from the db. With a warm db cache this query takes about 2 minutes to execute. If the query is executed without the sort and the sorting is done in PHP it only takes about a minute (though that is still unacceptably long).

    I'm not sure what the best solution for this is, but for now I'm going to lock that topic and put a cap in for 10k messages per thread.
    Walter likes this.
  2. Mike

    Mike XenForo Developer Staff Member

    Yeah, this is more or less a known issue right now. I've made changes to improve it (by reducing the data fetched), but there are a fair number of knock-on issues that mean it's not that trivial to change, but I do intend to change it. XF is more or less designed to handle any size thread (though certain rebuilds will be hard with large threads).

    Hopefully 1.2...
    ragtek and Mike Tougeron like this.
  3. Mike Tougeron

    Mike Tougeron Well-Known Member

    Yeah, not a trivial change; that's why I decided to just lock the super large threads for now. :(
  4. digitalpoint

    digitalpoint Well-Known Member

    A semi-related issue is email notifications for a new post in a thread are sent before the user sees their post show up. I have a few large threads where over 1,000 users are subscribed to get email notifications... so waiting for the system to send 1,000 emails before the user can see the reply they posted definitely causes some user experience issues.
    I made a relatively simply addon that extends the XenForo_Mail class and stores emails to be sent in an array rather than actually sending them. Then we do the actual mail sending on __destruct() (the class "unloading"). Seems to work well in my limited testing... If you want to play with it, let me know and I'll send it to you.
    wmtech, Shelley and Walter like this.
  5. Mike Tougeron

    Mike Tougeron Well-Known Member

    Cool, thanks for the offer but that won't be necessary. We're about to switch our mail sending to a custom class next week that'll be out-of-process (sort of) from the posting.
  6. robdog

    robdog Well-Known Member

    Would be nice to get a mail queue system going with priority flags that gets processed during a regular cron event.
  7. thebush

    thebush Member

    Mind of releasing it?
    Free or Paid add-on ?
  8. Mike

    Mike XenForo Developer Staff Member

    I've made some significant changes here, so that the messages are only grabbed when they're actually needed. This doesn't apply to normal replies, but it does potentially apply to deleting or moving a thread. I've adjusted some code such that certain areas can be changed to not need the entire set of messages (if we can derive the values from a query instead).
  9. Mike

    Mike XenForo Developer Staff Member

    Oh yeah, 1.2 has had a mail queue for a while too (though I only stumbled upon the fact that I hadn't finished it 100% until recently). :)
    Teapot and AlexT like this.
  10. AlexT

    AlexT Well-Known Member

    Very cool, Mike. (y) Then, does the issue with replies to large threads still exist?
  11. Mike

    Mike XenForo Developer Staff Member

    There shouldn't be an issue replying to big threads. Less common operations may have overheads that scale in proportion to the length of the thread.
  12. AlexT

    AlexT Well-Known Member

    Ahh, I misread your post. Excellent!!! When 1.2 comes out, it'll be worth making a quick addon to warn mods when they attempt to perform such operations such as deleting (which should never happen) or moving on large threads.

    Thanks too also for the news on the mail queue. It was next on my list of addon todos and I am glad it's already done. ;)

Share This Page