XF 1.4 Changed the deferred.php stuff and now hundreds of server errors

[Stuart Wright]

I had this in our cron:
* * * * * /home/sites/avforums/public_html/deferred.php > /dev/null 2>&1
then installed this addon:
https://xenforo.com/community/resources/goodfornothing-shell-defer.3791/
And changed the cron accordingly and started getting server errors like this one below.
So I reverted back and uninstalled the addon but now I'm still getting the errors.
About 600 errors every half an hour. E.g.

ErrorException: Undefined property: stdClass::$_type - library/XenES/Search/SourceHandler/ElasticSearch.php:870
Generated By: Unknown Account, 10 minutes ago
Stack Trace
#0 /home/sites/avforums/public_html/library/XenES/Search/SourceHandler/ElasticSearch.php(870): XenForo_Application::handlePhpError(8, 'Undefined prope...', '/home/sites/avf...', 870, Array)
#1 /home/sites/avforums/public_html/library/XenES/Model/Elasticsearch.php(264): XenES_Search_SourceHandler_ElasticSearch->isIndexSuccessful(Object(stdClass))
#2 /home/sites/avforums/public_html/library/XenES/Model/Elasticsearch.php(315): XenES_Model_Elasticsearch->reattemptFailedIndex('index', 'post', 21644366, Array, 5)
#3 /home/sites/avforums/public_html/library/XenES/CronEntry/EsReindex.php(19): XenES_Model_Elasticsearch->runFailedReindexer()
#4 [internal function]: XenES_CronEntry_EsReindex::reindex(Array)
#5 /home/sites/avforums/public_html/library/XenForo/Model/Cron.php(356): call_user_func(Array, Array)
#6 /home/sites/avforums/public_html/library/XenForo/Deferred/Cron.php(24): XenForo_Model_Cron->runEntry(Array)
#7 /home/sites/avforums/public_html/library/XenForo/Model/Deferred.php(294): XenForo_Deferred_Cron->execute(Array, Array, 7.9999969005585, '')
#8 /home/sites/avforums/public_html/library/XenForo/Model/Deferred.php(428): XenForo_Model_Deferred->runDeferred(Array, 7.9999969005585, '', false)
#9 /home/sites/avforums/public_html/library/XenForo/Model/Deferred.php(373): XenForo_Model_Deferred->_runInternal(Array, NULL, '', false)
#10 /home/sites/avforums/public_html/deferred.php(23): XenForo_Model_Deferred->run(false)
#11 {main}
Request State
array(3) {
["url"] => string(37) "https://www.avforums.com/deferred.php"
["_GET"] => array(0) {
}
["_POST"] => array(4) {
["_xfRequestUri"] => string(56) "/review/panasonic-pt-at6000e-3d-lcd-projector-review.384"
["_xfNoRedirect"] => string(1) "1"
["_xfToken"] => string(8) "********"
["_xfResponseType"] => string(4) "json"
}
}

Does anyone know what on earth is going on?
Thanks

 

[Daniel Hood]

Are they all the same error?

 

[Stuart Wright]

Are they all the same error?

No they alternate between
Undefined property: stdClass::$_id
and
Undefined property: stdClass::$_type

All on library/XenES/Search/SourceHandler/ElasticSearch.php:870

 

[Daniel Hood]

But they're all connected to ElasticSearch? Interesting.. Might need to restart the ES server (I'm not an expert on it though, so don't do it till someone else confirms). I think XenForo Devs may want to consider adding some Exception catching here.

 

[Stuart Wright]

Even when I remove the cron and restore the standard call in page_container, it's the same. About 600 errors every half an hour.

 

[Xon]

@Stuart Wright Since you ran it as root, make sure you reset file permissions so they are owned by the correct user the website runs as.

The deferred task itself may be screwed up, had have bad data persisted to the xf_deferred table. This may require purging from the DB. You can then restart off the failed indexing per normal.

But looking at that XF Elastic Search code, it is bizarre that it failed there. That is parsing the results from Elastic Search.

Can you check your elastic search instance's logs?

 

[Stuart Wright]

Can you check your elastic search instance's logs?

I can't, but I've asked Tim to.

 

[HWS]

You need to chown all files and folders in /data and /internal_data to your web server user if you ran deferred.php as root to prevent future errors.

Regarding the Elastic Search errors, those are very strange. Never had a similar experience. Maybe you updated Elasticsearch recently and this is not connected with the deferred.php change?

 

[Mike]

There are actually some changes relating to this ES code but IIRC, it is mostly hiding another error so it wouldn't happen in normal execution unless there was another error happening. I would look at the status of your Elasticsearch server to make sure everything is ok there.

 

[Stuart Wright]

Seems we've had a major Elastic implosion! Tim completely removed and reinstalled Elastic and it's taking all day to reindex.

 

[Stuart Wright]

I've been getting the same errors from Elastic again since Monday.
They all started with this different error:

XenForo_Exception: Elasticsearch indexing failed Elasticsearch error: MapperParsingException[failed to parse, document is empty] - library/XenES/Search/SourceHandler/ElasticSearch.php:840

So it seems that this error triggers an issue which breaks Elastic which then gives all the subsequent errors (4,500+ of them).
Any ideas please?

 

[Xon]

It sounds like XF is trying to submit an empty post or similar to Elastic Search.

Do you have a stacktrace for that error? It should help suggest what content type is triggering this.

 

[Stuart Wright]

It sounds like XF is trying to submit an empty post or similar to Elastic Search.

Do you have a stacktrace for that error? It should help suggest what content type is triggering this.

Error Info
XenForo_Exception: Elasticsearch indexing failed Elasticsearch error: MapperParsingException[failed to parse, document is empty] - library/XenES/Search/SourceHandler/ElasticSearch.php:840
Generated By: technomen, Monday at 9:24 AM
Stack Trace
#0 /home/sites/avforums/public_html/library/XenES/Search/SourceHandler/ElasticSearch.php(958): XenES_Search_SourceHandler_ElasticSearch->_logSearchResponseError('MapperParsingEx...', false, 'Elasticsearch i...')
#1 /home/sites/avforums/public_html/library/XenES/Search/SourceHandler/ElasticSearch.php(63): XenES_Search_SourceHandler_ElasticSearch->_triggerFailedIndexError(Object(stdClass), Array, true)
#2 /home/sites/avforums/public_html/library/XenForo/Search/Indexer.php(44): XenES_Search_SourceHandler_ElasticSearch->insertIntoIndex('post', 21671744, '', 'The real pictur...', 1422264264, 702463, 1931834, Array)
#3 /home/sites/avforums/public_html/library/XenForo/Search/DataHandler/Post.php(61): XenForo_Search_Indexer->insertIntoIndex('post', 21671744, '', 'The real pictur...', 1422264264, 702463, 1931834, Array)
#4 /home/sites/avforums/public_html/library/XenForo/Search/DataHandler/Abstract.php(275): XenForo_Search_DataHandler_Post->_insertIntoIndex(Object(XenForo_Search_Indexer), Array, Array)
#5 /home/sites/avforums/public_html/library/XenForo/DataWriter/DiscussionMessage.php(760): XenForo_Search_DataHandler_Abstract->insertIntoIndex(Object(XenForo_Search_Indexer), Array, Array)
#6 /home/sites/avforums/public_html/library/XenForo/DataWriter/DiscussionMessage.php(739): XenForo_DataWriter_DiscussionMessage->_insertOrUpdateSearchIndex()
#7 /home/sites/avforums/public_html/library/XenForo/DataWriter/DiscussionMessage.php(585): XenForo_DataWriter_DiscussionMessage->_indexForSearch()
#8 /home/sites/avforums/public_html/library/XenForo/DataWriter.php(1409): XenForo_DataWriter_DiscussionMessage->_postSave()
#9 /home/sites/avforums/public_html/library/DigitalPointSpy/DataWriter/DiscussionMessage/Post.php(7): XenForo_DataWriter->save()
#10 /home/sites/avforums/public_html/library/XenForo/ControllerPublic/Thread.php(557): DigitalPointSpy_DataWriter_DiscussionMessage_Post->save()
#11 /home/sites/avforums/public_html/library/XenForo/FrontController.php(347): XenForo_ControllerPublic_Thread->actionAddReply()
#12 /home/sites/avforums/public_html/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
#13 /home/sites/avforums/public_html/index.php(13): XenForo_FrontController->run()
#14 {main}
Request State
array(3) {
["url"] => string(87) "https://www.avforums.com/threads/hisense-announce-uled-2-0-4k-uhd-tvs.1931834/add-reply"
["_GET"] => array(1) {
["/threads/hisense-announce-uled-2-0-4k-uhd-tvs_1931834/add-reply"] => string(0) ""
}
["_POST"] => array(9) {
["message"] => string(192) "The real picture is one :
http://www.pcpop.com/doc/1/1067/1067459.shtml
http://www.microsofttranslator.com/...http://www.pcpop.com/doc/1/1067/1067459.shtml
"
["_xfRelativeResolver"] => string(78) "https://www.avforums.com/threads/hisense-announce-uled-2-0-4k-uhd-tvs.1931834/"
["attachment_hash"] => string(32) "4b52a2de5a743b95528bd37ca66db989"
["last_date"] => string(10) "1422215548"
["last_known_date"] => string(10) "1422215548"
["_xfToken"] => string(8) "********"
["_xfRequestUri"] => string(54) "/threads/hisense-announce-uled-2-0-4k-uhd-tvs.1931834/"
["_xfNoRedirect"] => string(1) "1"
["_xfResponseType"] => string(4) "json"
}
}

 

[Mike]

It looks like a post with invalid UTF-8 in it. It seems that Elasticsearch fell over when it received that. I don't see any obvious bad UTF-8 in the message, but it might be caused by an add-on to convert links to page titles (and it not giving valid UTF-8).