XF 2.1 Push timeout

[Snog]

ErrorException: Push notification failure: {"success":false,"endpoint":{},"message":"cURL error 28: Connection timed out after 10045 milliseconds (see http:\/\/curl.haxx.se\/libcurl\/c\/libcurl-errors.html)"} src/XF/Error.php:75

It's happened twice now. Both times with the same user.

I understand this could be a DNS issue at any point in the DNS system, but I don't think it's an error that can always be solved at the host server level.

 

[Snog]

I've found I can reproduce this fairly well.

Enabled push notifications on my desktop.
Sent a PC from a test account and received push notification without error.

Enabled push notifications on my phone while connected to WiFi.
Sent a PC from a test account and received push notification without error (both phone and desktop).

Switched off WiFi on phone and connected by 4g LTE
Sent a PC from a test account and received push notification without error (both phone and desktop).

Disabled push notifications on phone while connected by 4g LTE
Sent a PC from a test account and error occurred.

Any further attempts to send a PC result in the same push error.

   ErrorException: Push notification failure: {"success":false,"endpoint":{},"message":"cURL error 28: Connection timed out after 10041 milliseconds (see http:\/\/curl.haxx.se\/libcurl\/c\/libcurl-errors.html)"} src/XF/Error.php:75

    Generated by: XXXXX Feb 9, 2019 at 7:46 AM

Stack trace

#0 src/XF.php(187): XF\Error->logError('Push notificati...', false)
#1 src/XF/Service/PushNotification.php(243): XF::logError('Push notificati...')
#2 src/XF/Service/PushNotification.php(167): XF\Service\PushNotification->handleResults(Array, Array)
#3 src/XF/Service/PusherTrait.php(87): XF\Service\PushNotification->sendNotifications()
#4 src/XF/Service/Conversation/Notifier.php(130): XF\Service\Conversation\Pusher->push()
#5 src/XF/Service/Conversation/Notifier.php(65): XF\Service\Conversation\Notifier->_sendNotifications('reply', Array, Object(XF\Entity\ConversationMessage))
#6 src/XF/Service/Conversation/Replier.php(179): XF\Service\Conversation\Notifier->notifyReply(Object(XF\Entity\ConversationMessage))
#7 src/XF/Service/Conversation/Replier.php(158): XF\Service\Conversation\Replier->sendNotifications()
#8 src/XF/Service/ValidateAndSavableTrait.php(40): XF\Service\Conversation\Replier->_save()
#9 src/XF/Pub/Controller/Conversation.php(505): XF\Service\Conversation\Replier->save()
#10 src/XF/Mvc/Dispatcher.php(321): XF\Pub\Controller\Conversation->actionAddReply(Object(XF\Mvc\ParameterBag))
#11 src/XF/Mvc/Dispatcher.php(248): XF\Mvc\Dispatcher->dispatchClass('XF:Conversation', 'AddReply', Object(XF\Mvc\RouteMatch), Object(XF\Pub\Controller\Conversation), NULL)
#12 src/XF/Mvc/Dispatcher.php(100): XF\Mvc\Dispatcher->dispatchFromMatch(Object(XF\Mvc\RouteMatch), Object(XF\Pub\Controller\Conversation), NULL)
#13 src/XF/Mvc/Dispatcher.php(50): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#14 src/XF/App.php(2177): XF\Mvc\Dispatcher->run()
#15 src/XF.php(390): XF\App->run()
#16 index.php(20): XF::runApp('XF\\Pub\\App')
#17 {main}

cURL 7.58.0
OpenSSL/1.1.0g

 

[Chris D]

There isn't really any feasible way this can be related to whether an individual device is using WiFi or mobile network; that's not how it works.

Your server initiates a HTTP request with the push endpoint, which is a server hosted by the browser vendor. It is then that server which pushes the messages to the relevant devices.

So, my point is, the WiFi or cellular network of the receiving device is not involved when you're experiencing these timeout errors.

If this is happening regularly, you really need to be investigating it as a network issue in the first instance as there appears to be unexpected latency between your server and the push endpoint(s).

 

[Snog]

I think my point was that if this can be an intermittent error that can be caused by circumstances outside of the control of the host server (browser server being down or slow to respond), perhaps it shouldn't be logged?

 

[Kirby]

We do get these timeouts every know and then as well and I am almost 100% sure thate there are no ntwork issues on our side.

IMHO the push endpoints are sometimes just kinda slow/overloaded.

Maybe XF could do some retries in case of a timeout and if it doesn't get hrough just ignore the error, after all it's just a push notification - not smth. that is overly important.

 

[Snog]

We do get these timeouts every know and then as well and I am almost 100% sure thate there are no ntwork issues on our side.

IMHO the push endpoints are sometimes just kinda slow/overloaded.

Maybe XF could do some retries in case of a timeout and if it doesn't get hrough just ignore the error, after all it's just a push notification - not smth. that is overly important.

Having seen it happen when replying to a conversation, there is a 10 second delay between when a user clicks "Post reply" and the post actually happening / the error being generated . So, a retry might extend that even more unless the retry could be put in the job queue.

 

[Kirby]

Yeah, that's right. In our own push implementation (which was done during XF 2.0 developer preview and never made it into production), we did use background jobs specifically to avoid such slowdowns.

 

[Chris D]

I think my point was that if this can be an intermittent error that can be caused by circumstances outside of the control of the host server (browser server being down or slow to respond), perhaps it shouldn't be logged?

Sure. I don’t disagree.

But you’re suggesting it is happening with some frequency which is not usual. We’ve seen one or two timeouts too, but that’s since we launched the feature several months ago.

The whole point of these logs sometimes doesn’t necessarily have to guarantee there’s an actual actionable thing that needs to be solved. They can be seen as informational. If they do only happen once out of several thousand, then that’s negligible and can be ignored. But if it’s happening frequently then you might not want to ignore them, and might want to invest some time in solving them.

 

[rogerl]

I think it would be good if in the future Xenforo could classify the various types of errors we see in the control panel (eg. severe, warning, informational) and we could filter things based on that.

We have seen quite a few errors like this one and various email error issues with bounce messages, which are certainly useful to have as information, but tend to hide "real" errors as there are nearly always some "errors" logged whenever we visit the admin panel.

 

[Nirjonadda]

I am also getting lof of Server error log about Push notification failure.

 

[upnet]

The whole point of these logs sometimes doesn’t necessarily have to guarantee there’s an actual actionable thing that needs to be solved. They can be seen as informational. If they do only happen once out of several thousand, then that’s negligible and can be ignored. But if it’s happening frequently then you might not want to ignore them, and might want to invest some time in solving them.

At what point is this error of concern? I have read in a few other threads that a few of these errors is normal. That makes it hard to discern at what level we should be concerned. For instance this morning we saw this error 24 times in 7 minutes. Now it's 3 hours later and no other issues. Is that reflective of a system/network problem? Or par for the course?

ErrorException: Push notification failure: {"success":false,"endpoint":{},"message":"cURL error 28: Operation timed out after 20026 milliseconds with 0 bytes received (see http:\/\/curl.haxx.se\/libcurl\/c\/libcurl-errors.html)"} src/XF/Error.php:75
Generated by: Unknown account Nov 22, 2019 at 7:32 AM

 

[Chris D]

Is that reflective of a system/network problem?

Yeah essentially.

If it’s been ok since then it can be ignored.