XF 2.1 Push notification errors

tourmeister

Active member
So when I first upgraded to XF 2.1, I had no issues.

A few days after upgrading, I had to do a full restoration of my server from a backup made right before I did the upgrade.

I did the upgrade again after restoring from the backup and other than these push errors, everything is fine.

Any idea what might be causing it?

Code:
[LIST]
[*]               
[*]ErrorException: Push notification failure: {"success":false,"endpoint":{},"message":"Client error: `POST https:\/\/fcm.googleapis.com\/wp\/c-8MoOQ3Zvw:APA91bF7vO5TyLVSmaXcWgK18c4yOGuHzFBxHQffR0_QQk9Zyh3c6ksIqI6C5-jTKhjXIwJnagwPuKydA6i35rR5rsoLeVvbjepmJUaTFUnSwdZoEQLmo7eRUM5u8RjaqDabBKMLecuv` resulted in a `403 Forbidden` response:\n\n<!DOCTYPE html>\n<html lang=en>\n  <meta charset=utf-8>\n  <meta name=viewport content=\"initial-scale=1, minimum-scale=1,  (truncated...)\n","statusCode":403,"reasonPhrase":"Forbidden","expired":false,"content":{},"headers":{"Content-Length":["1449"],"Content-Type":["text\/html; charset=utf-8"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["SAMEORIGIN"],"X-Xss-Protection":["1; mode=block"],"Date":["Sun, 10 Feb 2019 02:23:31 GMT"],"Alt-Svc":["quic=\":443\"; ma=2592000; v=\"44,43,39\""]}}
[*]              
[*]src/XF/Error.php:75
[*]           
[*]               
[*]Generated by: maximus_flavius
[*]              
[*]Feb 9, 2019 at 8:23 PM
[*]           
[/LIST]
        Stack trace
      
           #0 src/XF.php(187): XF\Error->logError('Push notificati...', false)
#1 src/XF/Service/PushNotification.php(237): XF::logError('Push notificati...')
#2 src/XF/Service/PushNotification.php(167): XF\Service\PushNotification->handleResults(Array, Array)
#3 src/XF/Service/PusherTrait.php(87): XF\Service\PushNotification->sendNotifications()
#4 src/XF/Repository/UserAlert.php(147): XF\Service\Alert\Pusher->push()
#5 src/XF/Repository/UserAlert.php(111): XF\Repository\UserAlert->insertAlert(28214, 22889, 'maximus_flavius', 'post', 1595658, 'insert', Array)
#6 src/XF/Notifier/AbstractNotifier.php(55): XF\Repository\UserAlert->alert(Object(SV\ContactUsThread\XF\Entity\User), 22889, 'maximus_flavius', 'post', 1595658, 'insert', Array)
#7 src/XF/Notifier/Post/AbstractWatch.php(104): XF\Notifier\AbstractNotifier->basicAlert(Object(SV\ContactUsThread\XF\Entity\User), 22889, 'maximus_flavius', 'post', 1595658, 'insert')
#8 src/XF/Service/AbstractNotifier.php(86): XF\Notifier\Post\AbstractWatch->sendAlert(Object(SV\ContactUsThread\XF\Entity\User))
#9 src/XF/Service/AbstractNotifier.php(44): XF\Service\AbstractNotifier->notifyType(Object(XF\Notifier\Post\ThreadWatch), Array, G)
#10 src/XF/Service/AbstractNotifier.php(56): XF\Service\AbstractNotifier->notify(3)
#11 src/XF/Service/Thread/Replier.php(205): XF\Service\AbstractNotifier->notifyAndEnqueue(3)
#12 src/XF/Pub/Controller/Thread.php(330): XF\Service\Thread\Replier->sendNotifications()
#13 src/XF/Pub/Controller/Thread.php(463): XF\Pub\Controller\Thread->finalizeThreadReply(Object(XF\Service\Thread\Replier))
#14 src/XF/Mvc/Dispatcher.php(321): XF\Pub\Controller\Thread->actionAddReply(Object(XF\Mvc\ParameterBag))
#15 src/XF/Mvc/Dispatcher.php(248): XF\Mvc\Dispatcher->dispatchClass('XF:Thread', 'AddReply', Object(XF\Mvc\RouteMatch), Object(Andy\Calendar\XF\Pub\Controller\Thread), NULL)
#16 src/XF/Mvc/Dispatcher.php(100): XF\Mvc\Dispatcher->dispatchFromMatch(Object(XF\Mvc\RouteMatch), Object(Andy\Calendar\XF\Pub\Controller\Thread), NULL)
#17 src/XF/Mvc/Dispatcher.php(50): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#18 src/XF/App.php(2177): XF\Mvc\Dispatcher->run()
#19 src/XF.php(390): XF\App->run()
#20 index.php(20): XF::runApp('XF\\Pub\\App')
#21 {main}

        Request state

   array(4) {
  ["url"] => string(46) "/forums/threads/suzuki-dr-big.119802/add-reply"
  ["referrer"] => string(58) "https://www.twtex.com/forums/threads/suzuki-dr-big.119802/"
  ["_GET"] => array(0) {
  }
  ["_POST"] => array(9) {
    ["message_html"] => string(102) "<p>https://www.twtex.com/forums/threads/is-your-strom-going-to-be-out-of-date.119760/#post-1595478</p>"
    ["attachment_hash"] => string(32) "f00da9418e27c121472116a4959245d2"
    ["attachment_hash_combined"] => string(88) "{"type":"post","context":{"thread_id":119802},"hash":"f00da9418e27c121472116a4959245d2"}"
    ["last_date"] => string(10) "1549765289"
    ["last_known_date"] => string(10) "1549765289"
    ["_xfToken"] => string(8) "********"
    ["_xfRequestUri"] => string(37) "/forums/threads/suzuki-dr-big.119802/"
    ["_xfWithData"] => string(1) "1"
    ["_xfResponseType"] => string(4) "json"
  }
}
 
Not 100% sure but disable this addon and see if you still get the errors,
 
Not 100% sure but disable this addon and see if you still get the errors,
No affect.
 
I think it was the accursed TapaTalk plugin... I totally uninstalled it. Totally deleted the files from the server. Went to their site and re downloaded the plugin, uploaded it to the server and did a reinstall. No more errors.
 
Well... so much for that theory... Reinstalling Tapatalk fixed a different error I was getting, but I am still getting Push errors. Here is the latest after doing the TT reinstall.

Code:
Server error log
    
        
            
[LIST]
[*]                
[*]ErrorException: Push notification failure: {"success":false,"endpoint":{},"message":"Client error: `POST https:\/\/fcm.googleapis.com\/wp\/f7_7eFGqkc8:APA91bE7z0t4xXlZoSy6SEZccTpDxRtuWWlvnD-qM5EicrW8Ulk7Tw54HWUO1uXaQBhLff_0mmuBZ17JF2uytHlmRb7uhUUcdCe0pUXotTw8OXFcgsUd3Qtw3o9LX5JpK0uuU8-GnX2-` resulted in a `403 Forbidden` response:\n\n<!DOCTYPE html>\n<html lang=en>\n  <meta charset=utf-8>\n  <meta name=viewport content=\"initial-scale=1, minimum-scale=1,  (truncated...)\n","statusCode":403,"reasonPhrase":"Forbidden","expired":false,"content":{},"headers":{"Content-Length":["1449"],"Content-Type":["text\/html; charset=utf-8"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["SAMEORIGIN"],"X-Xss-Protection":["1; mode=block"],"Date":["Tue, 12 Feb 2019 02:39:21 GMT"],"Alt-Svc":["quic=\":443\"; ma=2592000; v=\"44,43,39\""]}}
[*]               
[*]src/XF/Error.php:75
[*]            
[/LIST]
           
[LIST]
[*]                
[*]Generated by: Morgstang
[*]               
[*]Feb 11, 2019 at 8:39 PM
[*]            
[/LIST]
        
        Stack trace
       
           #0 src/XF.php(187): XF\Error->logError('Push notificati...', false)
#1 src/XF/Service/PushNotification.php(237): XF::logError('Push notificati...')
#2 src/XF/Service/PushNotification.php(167): XF\Service\PushNotification->handleResults(Array, Array)
#3 src/XF/Service/PusherTrait.php(87): XF\Service\PushNotification->sendNotifications()
#4 src/XF/Repository/UserAlert.php(147): XF\Service\Alert\Pusher->push()
#5 src/XF/Repository/UserAlert.php(111): XF\Repository\UserAlert->insertAlert(5533, 26005, 'Morgstang', 'post', 1595974, 'insert', Array)
#6 src/XF/Notifier/AbstractNotifier.php(55): XF\Repository\UserAlert->alert(Object(XF\Entity\User), 26005, 'Morgstang', 'post', 1595974, 'insert', Array)
#7 src/XF/Notifier/Post/AbstractWatch.php(104): XF\Notifier\AbstractNotifier->basicAlert(Object(XF\Entity\User), 26005, 'Morgstang', 'post', 1595974, 'insert')
#8 src/XF/Service/AbstractNotifier.php(86): XF\Notifier\Post\AbstractWatch->sendAlert(Object(XF\Entity\User))
#9 src/XF/Service/AbstractNotifier.php(44): XF\Service\AbstractNotifier->notifyType(Object(XF\Notifier\Post\ForumWatch), Array, G)
#10 src/XF/Service/AbstractNotifier.php(56): XF\Service\AbstractNotifier->notify(3)
#11 src/XF/Service/Thread/Creator.php(328): XF\Service\AbstractNotifier->notifyAndEnqueue(3)
#12 src/XF/Pub/Controller/Forum.php(593): XF\Service\Thread\Creator->sendNotifications()
#13 src/XF/Pub/Controller/Forum.php(751): XF\Pub\Controller\Forum->finalizeThreadCreate(Object(XF\Service\Thread\Creator))
#14 src/XF/Mvc/Dispatcher.php(321): XF\Pub\Controller\Forum->actionPostThread(Object(XF\Mvc\ParameterBag))
#15 src/XF/Mvc/Dispatcher.php(248): XF\Mvc\Dispatcher->dispatchClass('XF:Forum', 'PostThread', Object(XF\Mvc\RouteMatch), Object(XF\Pub\Controller\Forum), NULL)
#16 src/XF/Mvc/Dispatcher.php(100): XF\Mvc\Dispatcher->dispatchFromMatch(Object(XF\Mvc\RouteMatch), Object(XF\Pub\Controller\Forum), NULL)
#17 src/XF/Mvc/Dispatcher.php(50): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#18 src/XF/App.php(2177): XF\Mvc\Dispatcher->run()
#19 src/XF.php(390): XF\App->run()
#20 index.php(20): XF::runApp('XF\\Pub\\App')
#21 {main}

        

        Request state

        

            array(4) {
  ["url"] => string(45) "/forums/forums/for-sale-wanted.12/post-thread"
  ["referrer"] => string(66) "https://www.twtex.com/forums/forums/for-sale-wanted.12/post-thread"
  ["_GET"] => array(0) {
  }
  ["_POST"] => array(14) {
    ["prefix_id"] => string(1) "0"
    ["title"] => string(39) "For Sale: New Michelin Road 5 190/55/17"
    ["message_html"] => string(53) "<p>Best price i’ve seen is $211...my price $175</p>"
    ["attachment_hash"] => string(32) "b1347959240ba5dcad8e7e6298b1eadb"
    ["attachment_hash_combined"] => string(82) "{"type":"post","context":{"node_id":12},"hash":"b1347959240ba5dcad8e7e6298b1eadb"}"
    ["tags"] => string(0) ""
    ["watch_thread"] => string(1) "1"
    ["watch_thread_email"] => string(1) "1"
    ["_xfSet"] => array(1) {
      ["watch_thread"] => string(1) "1"
    }
    ["poll"] => array(5) {
      ["question"] => string(0) ""
      ["new_responses"] => array(1) {
        [0] => string(0) ""
      }
      ["max_votes_type"] => string(6) "single"
      ["change_vote"] => string(1) "1"
      ["view_results_unvoted"] => string(1) "1"
    }
    ["_xfToken"] => string(8) "********"
    ["_xfRequestUri"] => string(45) "/forums/forums/for-sale-wanted.12/post-thread"
    ["_xfWithData"] => string(1) "1"
    ["_xfResponseType"] => string(4) "json"
  }
}
 
So ALL of the push errors I have been getting have had that link to the googleapis address, but now I have one that is different. I don't know if that helps or not, but here it is,

Code:
Server error log
    
        
            
[LIST]
[*]                
[*]ErrorException: Push notification failure: {"success":false,"endpoint":{},"message":"Client error: `POST https:\/\/updates.push.services.mozilla.com\/wpush\/v2\/gAAAAABcWekCHnnqU1rZwJ4g6RzNzfG43bDYKMczuTve_8Lvgnco34zRG4tO3J3I8AloHNnGGOji6A0pVgeuN1cy13c-zqrJw6d402aiH7U7FvwVpxutPlnG--INaWXGBOEu_4hWTgZ_Gju2bh_l0KUB2nYSb7MUrR86C3_5pUmT2QqOFujrQ1U` resulted in a `401 Unauthorized` response:\n{\"code\": 401, \"errno\": 109, \"error\": \"Unauthorized\", \"more_info\": \"http:\/\/autopush.readthedocs.io\/en\/latest\/http.html#er (truncated...)\n","statusCode":401,"reasonPhrase":"Unauthorized","expired":false,"content":{},"headers":{"Access-Control-Allow-Headers":["content-encoding,encryption,crypto-key,ttl,encryption-key,content-type,authorization"],"Access-Control-Allow-Methods":["POST"],"Access-Control-Allow-Origin":["*"],"Access-Control-Expose-Headers":["location,www-authenticate"],"Content-Type":["application\/json"],"Date":["Tue, 12 Feb 2019 14:55:44 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=31536000;includeSubDomains"],"Content-Length":["199"],"Connection":["keep-alive"]}}
[*]               
[*]src/XF/Error.php:75
[*]            
[/LIST]
           
[LIST]
[*]                
[*]Generated by: kickstand~prophet
[*]               
[*]Feb 12, 2019 at 8:55 AM
[*]            
[/LIST]
        
        Stack trace
       
           #0 src/XF.php(187): XF\Error->logError('Push notificati...', false)
#1 src/XF/Service/PushNotification.php(237): XF::logError('Push notificati...')
#2 src/XF/Service/PushNotification.php(167): XF\Service\PushNotification->handleResults(Array, Array)
#3 src/XF/Service/PusherTrait.php(87): XF\Service\PushNotification->sendNotifications()
#4 src/XF/Repository/UserAlert.php(147): XF\Service\Alert\Pusher->push()
#5 src/XF/Repository/UserAlert.php(111): XF\Repository\UserAlert->insertAlert(28989, 23122, 'kickstand~proph...', 'post', 1596054, 'insert', Array)
#6 src/XF/Notifier/AbstractNotifier.php(55): XF\Repository\UserAlert->alert(Object(XF\Entity\User), 23122, 'kickstand~proph...', 'post', 1596054, 'insert', Array)
#7 src/XF/Notifier/Post/AbstractWatch.php(104): XF\Notifier\AbstractNotifier->basicAlert(Object(XF\Entity\User), 23122, 'kickstand~proph...', 'post', 1596054, 'insert')
#8 src/XF/Service/AbstractNotifier.php(86): XF\Notifier\Post\AbstractWatch->sendAlert(Object(XF\Entity\User))
#9 src/XF/Service/AbstractNotifier.php(44): XF\Service\AbstractNotifier->notifyType(Object(XF\Notifier\Post\ThreadWatch), Array, G)
#10 src/XF/Service/AbstractNotifier.php(56): XF\Service\AbstractNotifier->notify(3)
#11 src/XF/Service/Thread/Replier.php(205): XF\Service\AbstractNotifier->notifyAndEnqueue(3)
#12 src/XF/Pub/Controller/Thread.php(330): XF\Service\Thread\Replier->sendNotifications()
#13 src/XF/Pub/Controller/Thread.php(463): XF\Pub\Controller\Thread->finalizeThreadReply(Object(XF\Service\Thread\Replier))
#14 src/XF/Mvc/Dispatcher.php(321): XF\Pub\Controller\Thread->actionAddReply(Object(XF\Mvc\ParameterBag))
#15 src/XF/Mvc/Dispatcher.php(248): XF\Mvc\Dispatcher->dispatchClass('XF:Thread', 'AddReply', Object(XF\Mvc\RouteMatch), Object(Andy\GPXViewer\XF\Pub\Controller\Thread), NULL)
#16 src/XF/Mvc/Dispatcher.php(100): XF\Mvc\Dispatcher->dispatchFromMatch(Object(XF\Mvc\RouteMatch), Object(Andy\GPXViewer\XF\Pub\Controller\Thread), NULL)
#17 src/XF/Mvc/Dispatcher.php(50): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#18 src/XF/App.php(2177): XF\Mvc\Dispatcher->run()
#19 src/XF.php(390): XF\App->run()
#20 index.php(20): XF::runApp('XF\\Pub\\App')
#21 {main}

        

        Request state

        

            array(4) {
  ["url"] => string(76) "/forums/threads/notice-new-date-february-meet-n-greet-n-eat.119754/add-reply"
  ["referrer"] => string(88) "https://www.twtex.com/forums/threads/notice-new-date-february-meet-n-greet-n-eat.119754/"
  ["_GET"] => array(0) {
  }
  ["_POST"] => array(9) {
    ["message_html"] => string(149) "<p>The weather is looking much better this Sunday. &nbsp;Any others folks feeling like joining in please post up so I can let Pepperbelly's know?</p>"
    ["attachment_hash"] => string(32) "91e57a15889eae08779cd10fc5634bd9"
    ["attachment_hash_combined"] => string(88) "{"type":"post","context":{"thread_id":119754},"hash":"91e57a15889eae08779cd10fc5634bd9"}"
    ["last_date"] => string(10) "1549905551"
    ["last_known_date"] => string(10) "1549905551"
    ["_xfToken"] => string(8) "********"
    ["_xfRequestUri"] => string(67) "/forums/threads/notice-new-date-february-meet-n-greet-n-eat.119754/"
    ["_xfWithData"] => string(1) "1"
    ["_xfResponseType"] => string(4) "json"
  }
}
 
So my File Check comes back with everything being great.

I've not been editing templates or playing with code. Everything is stock other than the add-ons I've installed.

They are the same add-ons I had installed previously when the push notifications were working perfectly and I wasn't getting any errors at all.

I can't figure out what might have changed unless something in the upgrade of XF from 2.012 to 2.1 got messed up when I had to do it the second time after restoring my server from a backup.

A little direction on where to look or what to check would be VERY much appreciated.
 
A few days after upgrading, I had to do a full restoration of my server from a backup made right before I did the upgrade.
I think this might explain it.

In restoring the backup to a previous point, you likely lost part of the original push configuration (I’m assuming you enabled push after upgrading and before restoring the backup).

The specific configuration you may have lost may be the VAPID authentication keys which are specific to your server but authorise your server to send push notifications.

That would certainly explain why these endpoints are saying “403 forbidden”.

Perhaps the easiest thing to do is to first of all toggle the “Enable push notifications” option off and on in the Admin CP.

Then see if that makes any difference.

If the errors still happen then ascertain whether you can unsubscribe and resubscribe to push notifications in your own preferences and whether or not you can actually receive those notifications without error.

If you can receive without error then it confirms my hunch: anyone who enabled push notifications prior to the restore may be causing these errors and the requests to send notifications to them are being rejected.

If that’s the case, it might be worth just starting again from scratch, wiping everyone’s push notification subscriptions (working or not) then asking those users to re-enable.
 
In restoring the backup to a previous point, you likely lost part of the original push configuration (I’m assuming you enabled push after upgrading and before restoring the backup).

A backup was made about 15 minutes before I upgraded to 2.1. After the backup, I enabled Push notifications and everything worked fine. When I had to do the restore, we used the backup made right before the upgrade. And then I just upgraded again. So are you saying there are settings that are stored on the server that might still be in place from before the restoration from the backup?

When the hosting service restored the server, it was from a backup of the entire server.


If that’s the case, it might be worth just starting again from scratch, wiping everyone’s push notification subscriptions (working or not) then asking those users to re-enable.

Do I do this by simply disabling push notifications in the Setup -> Options -> Alerts & Notifications?
 
No joy.

I emptied the subscription table in phpMyAdmin. Then I re-enabled Push notifications in the settings.

Some notifications are working. I am getting them. However, I am still getting the same type of errors.

When I restored from a backup, it was not just a database restore. It was a full restoration of the entire server, so it overwrite my entire file structure, databases, everything. Basically, it was like I had never upgraded to XF 2.1.

The only thing odd was that when I tried to upload the XF2.1 files to do the upgrade, it kept stalling during the FTP transfer. Even once I got that completed, when I ran the upgrade routine, it would tell me one or more files had unexpected contents. So based on Brogan's suggestion in another thread, I went to the XF site and downloaded the XF2.1 upgrade zip file again. I then uploaded that and it never hung during the FTP transfer. I then upgraded from that and it worked perfect as far as I could tell, just like the first time before the restore. However, I've been getting these errors. I have also been having issues with TapaTalk not working properly even though it worked perfect before the restore. (Yes, I hate TT, but over half my users seem to love it, so I can't really do away with it and tell them to get over it ;)).

The daily File check always comes back with zero problems.

Should I just start disabling ALL add-ons to see what happens? They are all the same ones I had installed before when it was working just fine. I have reuploaded and reinstalled all of them from scratch just in case one of them got messed up somehow after the restore/upgrade.
 
If that’s the case, it might be worth just starting again from scratch, wiping everyone’s push notification subscriptions (working or not) then asking those users to re-enable.

This did not work.

Is there ANY other way to reset the PUSH feature? Is there something on my server that needs to be checked?
 
For now, I think you mostly need to disregard the errors. We are making changes in 2.1.1 that should resolve this.

The issue is fundamentally that the restore changed the secret keys that the push system uses. The browser thinks the subscriptions are still valid, so they're getting updated on our system. I've added code for the next release that will detect if the push subscription isn't for the correct key so we can force an unsubscription and to remove the subscription on the server side if it returns an error that indicates the key isn't valid.
 
Top Bottom