Push notifications should use a queue to send them asynchronously (like emails)

Steffen

Well-known member
This is something in-between a bug report and a feature suggestion and I feel bad for bringing this up so late in the release process of XenForo 2.1. I just didn't notice this potential issue earlier but I think it could be important especially (but not only) for bigger forums. The issue I see is that when XenForo sends push notifications it does so synchronously within the web request that e.g. saves a reply (in contrast to email notifications, which use an async queue).

The first bottleneck are network requests: The architecture of Web Push requires a separate HTTP request to be sent for every subscriber (to servers run by Google, Mozilla and possibly others in the future). In threads with many participants, this can result in dozens or hundreds of HTTP requests when a reply is saved. The Web Push library used by XenForo sends those requests in parallel but obviously the slowest request determines the required time. Needless to say, these requests can run into a timeout. Even if they don't, in larger threads there could be just too many of them. To somewhat alleviate this problem, XenForo reduces the default timeout of the Web Push library from 30 to 10 seconds.

The second bottleneck is CPU usage: The used Web Push library is quite slow because it does quite a lot of cryptography stuff in PHP. I've recently used it in my own project and my CPU (Core i7-4600U) was only able to create (not send, I commented that code out) about 600 push requests in 10 seconds. Today's server CPUs are maybe twice as fast but in general they just have more cores (which won't help because PHP is single-threaded).

There are three possible issues:
  1. A (possibly very) noticeable delay for the visitor who replies to a thread (up to 10 seconds for network requests + CPU time).
  2. Some push notifications may not be sent if there are too many of them and time runs out (situation: reply to a large thread).
  3. This can DoS on your own forum if all of your possibly few PHP-FPM workers are occupied with waiting for timeouts (situation: many replies to different threads at roughly the same time).
Considering all three issues together, I think it would be great if something could be done about this situation. For email notifications, XenForo uses a queue to avoid the very same issues. I think XenForo should use a queue for push notifications (and send them asynchronously via cronjobs), too.

In case of emails, there even exists a workaround for needing a queue (just run an SMTP server on localhost that forwards mail to your real mail server). Unfortunately, no such workaround exists for push notifications.
 
Last edited:
Upvote 32

Steffen

Well-known member
In the past couple of days, there have been a few timeouts and other errors while sending push notifications. I'd greatly appreciate if push notifications were sent using an asynchronous queue to prevent long delays when saving a post.

Code:
ErrorException: Push notification failure: {"success":false,"endpoint":{},"message":"cURL error 28: Operation timed out after 10000 milliseconds with 0 out of 0 bytes received (see http:\/\/curl.haxx.se\/libcurl\/c\/libcurl-errors.html)"} src/XF/Error.php:75

ErrorException: Push notification failure: {"success":false,"endpoint":{},"message":"cURL error 56: OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104 (see http:\/\/curl.haxx.se\/libcurl\/c\/libcurl-errors.html)"}
 

TMA

Active member
The way we did this with the old web push plugin was to store the messages in a table, and release the oldest messages every minute on a limited basis (adjustable by option, default 500 messages per minute).

After implementing this, big boards no longer had problems.
 

Xon

Well-known member
This is bordering on a bug for very active forums, it makes alerting and notifications surprisingly unreliable :(
 

ge66

Well-known member
No, it shouldn't.

It might be bordering on a bug, but it isn't one. We'll be considering it in a future release.
I hope it is here soon.
I have tried to use push for a couple of months now and have had to turn them off because my users say the site is slower with them on.
 
Top