XF 2.2 Push notification failure: 429 Too Many Requests

[Mouth]

XF 2.2.15

These just started regularly appearing in the last few days ...

Server error log
ErrorException: Push notification failure: Client error: `POST https://wns2-bl2p.notify.windows.com/w/?token=[redacted]` resulted in a `429 Too Many Requests` response src/XF/Error.php:77
Generated by: [user] Mar 20, 2024 at 23:10
Stack trace
#0 src/XF.php(219): XF\Error->logError('Push notificati...', false)
#1 src/XF/Service/PushNotification.php(267): XF::logError('Push notificati...')
#2 src/XF/Service/PushNotification.php(179): XF\Service\PushNotification->handleResults(Object(Generator))
#3 src/XF/Service/PusherTrait.php(194): XF\Service\PushNotification->sendNotifications()
#4 src/XF/Repository/UserAlert.php(165): XF\Service\Alert\Pusher->push()
[...]
#18 src/XF.php(524): XF\App->run()
#19 index.php(20): XF::runApp('XF\\Pub\\App')
#20 {main}
Request state
array(4) {
  ["url"] => string(34) "/posts/4219887/react?reaction_id=3"
  ["referrer"] => bool(false)
  ["_GET"] => array(2) {
    ["/posts/4219887/react"] => string(0) ""
    ["reaction_id"] => string(1) "3"
  }
  ["_POST"] => array(4) {
    ["_xfRequestUri"] => string(36) "/threads/funny-pics.119059/page-5143"
    ["_xfWithData"] => string(1) "1"
    ["_xfToken"] => string(8) "********"
    ["_xfResponseType"] => string(4) "json"
  }
}

 

[KensonPlays]

I've been doing some testing over the last few days, and I believe no/zero push notifications are getting through to Microsoft browsers.
Appears there's more to this than just rate limiting?

Anyone getting these timeouts and blocks able to test/confirm if any Edge browser push notifications are being received?

I know Edge mobile (on my android devices) works, but I can't confirm desktop right atm.

 

[rogerl]

While the timing is suspect I don't think there are changes in 2.2.14/15 that would affect this. I imagine Microsoft introduced rate limiting around the same time as the release.

I'd agree, we had been on 2.2.15 for several weeks before these errors started.

I now have a log full of several thousand of them though and still arriving at a rapid rate

 

[Rhody]

Yea it's very recent, not because of the version. I think maybe the site that receives them is being attacked or is just overwhelmed.

I'm only seeing 2 or 3 failures per week.

 

[PASS]

Curious if any of you are using the Signup abuse detection and blocking addon.

 

[Mouth]

Curious if any of you are using the Signup abuse detection and blocking addon.

No, but still very much impacted by this issue.

No-one is able to find, or get, information from Microsoft on this throttling?

 

[vbresults]

No, but still very much impacted by this issue.

No-one is able to find, or get, information from Microsoft on this throttling?

I wouldn’t characterize the non-delivery of messages in this instance as throttling, but based on what Jeremy said your problems will be fully solved in 2.3.

 

[Mouth]

but based on what Jeremy said your problems will be fully solved in 2.3.

If the recipient server is having resourcing issues (unlikely) or throttling, then queuing and retries could only exasperate and make the problem worse. Without knowing the cause or trigger, it's not reliable to say that an action will resolve or workaround an issue.

 

[Jeremy P]

It depends on how they've implemented throttling really. They'll be retried with back-off (by default after 5/10/20 minutes on 1/2/3 failures respectively, though we can adjust this further). That doesn't necessarily guarantee delivery, especially if the queue is being filled faster than it can drain, but at that point there's just not a whole lot we can do about it.

 

[Mouth]

but at that point there's just not a whole lot we can do about it.

Provide an Alert to the user that the Push Notification to them failed and could not be delivered, with the reason.

 

[Jeremy P]

I don't think this is the sort of implementation detail we'd want to expose to end-users, especially given there's not a lot they can do about it either. I imagine most would just find it confusing. Ultimately it would be nice if Microsoft provided more details on their rate-limiting, or relaxed it altogether.

Users already receive an alert for the action that was supposed to trigger the push notification anyway.

 

[Mouth]

especially given there's not a lot they can do about it either.

You mean other than i) understand that the site is unable to deliver their subscribed push notification(s) for reason other than site issue, ii) not complain to moderator/admins that functionality doesn't work, thereby lowering trust in the site, and iii) change browser on their device.

 

[Kirby]

Provide an Alert to the user that the Push Notification to them failed and could not be delivered, with the reason.

IMHO this would be a bad idea - in case a push server triggers a lot of failures this could cause more support workload (from users asking what those alerts mean to them) then missed push notifications (which in many cases might not even be noticed by users, at least this is my experiemce so far - we do have quite a few failures every day but zero users asking about missed push notifications).

 

[optrex]

I'm getting them too but on a really small instance with a couple of members, so for it to be hitting a limit isnt very flash.

 

[Suzanne O]

I get these as well.
I also get database errors almost all the time. (I've started another thread on these)

 

[vbresults]

If the recipient server is having resourcing issues (unlikely) or throttling, then queuing and retries could only exasperate and make the problem worse. Without knowing the cause or trigger, it's not reliable to say that an action will resolve or workaround an issue.

The cause of the problem is clearly stated in the error message. I developed a Mobile App with push for XF and separately a PWA with notifications add-on long before it was implemented into the XF core, where I experienced a similar issue until implementing a queue and retry myself, so understand I am talking from experience.

Jeremy said the queue is already implemented and all that is left is to retry. @Jeremy P do not retry with an exponential back-off for 429 or the delivery will be delayed further out than normal, just schedule the next notification delivery based on the retry time response header. The exponential backoff needs to be used for other server or client errors (connect, timeout, 5xx, etc).

Your problem is fixed in 2.3 @Mouth just notify your members through an announcement and wait for the release (also advised to everyone else having the issue). Microsoft is not stopping your messages from going out, just spreading them out over time by forcing retries, but until the retry mechanism is implemented in 2.3, messages will be dropped.

There is nothing unaccounted for in this situation.

 

[M@rc]

Seems to have stopped for now.

 

[optrex]

I’m getting them in 2.3 beta 3

 

[Joe Link]

I haven't received any in a few days (XF2.2).

 

[Chris D]

Yeah we haven't had any for quite a while and we were getting them frequently.

@optrex can you post an example error you're receiving in case it's different?