XF 2.2 Push notification failure: 429 Too Many Requests

[Mouth]

XF 2.2.15

These just started regularly appearing in the last few days ...

Server error log
ErrorException: Push notification failure: Client error: `POST https://wns2-bl2p.notify.windows.com/w/?token=[redacted]` resulted in a `429 Too Many Requests` response src/XF/Error.php:77
Generated by: [user] Mar 20, 2024 at 23:10
Stack trace
#0 src/XF.php(219): XF\Error->logError('Push notificati...', false)
#1 src/XF/Service/PushNotification.php(267): XF::logError('Push notificati...')
#2 src/XF/Service/PushNotification.php(179): XF\Service\PushNotification->handleResults(Object(Generator))
#3 src/XF/Service/PusherTrait.php(194): XF\Service\PushNotification->sendNotifications()
#4 src/XF/Repository/UserAlert.php(165): XF\Service\Alert\Pusher->push()
[...]
#18 src/XF.php(524): XF\App->run()
#19 index.php(20): XF::runApp('XF\\Pub\\App')
#20 {main}
Request state
array(4) {
  ["url"] => string(34) "/posts/4219887/react?reaction_id=3"
  ["referrer"] => bool(false)
  ["_GET"] => array(2) {
    ["/posts/4219887/react"] => string(0) ""
    ["reaction_id"] => string(1) "3"
  }
  ["_POST"] => array(4) {
    ["_xfRequestUri"] => string(36) "/threads/funny-pics.119059/page-5143"
    ["_xfWithData"] => string(1) "1"
    ["_xfToken"] => string(8) "********"
    ["_xfResponseType"] => string(4) "json"
  }
}

 

[rdn]

I have these error logs too, like every few minutes.

 

[Mouth]

I've logged an XF support ticket, and they are investigating.

 

[Mr Lucky]

I am also getting these

 

[markku]

429 Too Many Requests errors - Azure

Troubleshoot why you receive 429 Too Many Requests errors on your Kubernetes clusters.

learn.microsoft.com

I wonder if XF is being rate limited due to exceeding their quota?

 

[Chris D]

It likely is rate limiting of some sort because that's just what 429 actually means but that page isn't anything to do with push notifications.

To be clear there's no concept of XF in total being rate limited, though each site could be.

The endpoint we communicate with in these cases are provided by the browser vendor for the sole purpose of sending notifications to which are then delivered to the browser. I don't think browser vendors other than Microsoft are being as aggressive here. And it kind of defeats the point. I'm not even sure if these limits are documented anywhere.

With 2.3 (a later beta) these should be retry-able so that will help.

 

[KensonPlays]

I'm getting these as well, and my host isn't giving much info on whats going on. Could I get an update when a fix is pushed for this, if possible? Error looks almost 100% the same for me.

 

[Chris D]

There may not be a fix.

This is Microsoft saying that the endpoint they publish for sending push notifications is... receiving too many push notifications.

It seems to be an overzealous limit on their end, and not really anything we can do.

In a later version of XF 2.3, these will be retried automatically (but the errors will still be logged).

They are safe to ignore.

 

[KensonPlays]

There may not be a fix.

This is Microsoft saying that the endpoint they publish for sending push notifications is... receiving too many push notifications.

It seems to be an overzealous limit on their end, and not really anything we can do.

In a later version of XF 2.3, these will be retried automatically (but the errors will still be logged).

They are safe to ignore.

Yea... wish I could have it permanently disabled in my ACP. It gets on my nerves seeing errors logged that I can't disable. lol

 

[Mouth]

They are safe to ignore.

Not really, maybe from a purely error message context.
Users not receiving push notifications when they should be, and expect to, results in regular support/user complaints.

 

[Jeremy P]

They're safe to ignore in the sense that they aren't detrimental to the forum as a whole and there's not much that administrators can do to action them. As above, a later version of 2.3 will retry delivery anyway.

 

[KensonPlays]

They're safe to ignore in the sense that they aren't detrimental to the forum as a whole and there's not much that administrators can do to action them. As above, a later version of 2.3 will retry delivery anyway.

That may be true, but maybe I'll use my adblocker to hide the "error log" element for now so it doesn't bother me. lol

 

[z3r010]

It would be nice to have the option to suppress push errors.

 

[Rhody]

I believe it was said that 2.3 will have an automatic retry feature, which should hopefully eliminate any problems

 

[Paul B]

It won't eliminate the issue but it will mitigate it.

 

[cwe]

I'm also getting these 429 'Too many response' errors logged after upgrading to 2.2.15. I did not use to get them with 2.2.13 and prior XF versions.

 

[Jeremy P]

While the timing is suspect I don't think there are changes in 2.2.14/15 that would affect this. I imagine Microsoft introduced rate limiting around the same time as the release.

 

[vbresults]

Usually, when endpoints like this return a 429 error, in their HTTP response header they also indicate after how much time you should retry the request (it tends to vary by request), and from my experience most of these retries work by 3rd attempt, so the message gets delivered.

This might be an opportunity to implement a lightweight queue package (from packagist) with the option to connect to a bona-fide queue system. But something like this is probably out of the scope of what can be done in 2.3.

 

[Jeremy P]

XF has a job queue already, and push sending has already moved to the queue in 2.3. We just need to set up the job to retry on failure.

 

[Mouth]

I've been doing some testing over the last few days, and I believe no/zero push notifications are getting through to Microsoft browsers.
Appears there's more to this than just rate limiting?

Anyone getting these timeouts and blocks able to test/confirm if any Edge browser push notifications are being received?