XF 1.5 403 Google Crawler Errors

[king8084]

403 Crawler errors have been absolutely killing me and it appears to be something in my xenforo configuration. I've worked with a server admin + with the server/host directly, but both have come up empty from the server-side.. Here's the scoop:

- robots.txt is configured correctly
- pages outside of my main/landing page DO crawl correctly and DO NOT receive 403 server errors from google

• google's webmasters tools (search console > URL inspection) yields a "Failed: Crawl anomaly", which has very little information on it
• google's pagespeed insights yields a "ERRORED_DOCUMENT_REQUEST" error code 403
• redleg's file viewer yields a "403 Forbidden" result
• this appears to also effect my google adsense, where the base URL often (but not always) results in no ad displaying, but the other pages do
• URL is: https://www.sectioneighty.com
• all tests have been ran against the above URL + https://sectioneighty.com, https://www.sectioneighty.com/forum, https://sectioneighty.com/forum, & https://forum.sectioneighty.com (the main URL redirects to this)

////////

robots.txt:

Sitemap: https://forum.sectioneighty.com/sitemap.php

User-agent: *
Disallow: /account/
Disallow: /conversations/
Disallow: /online/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /login/
Disallow: /best-posts/
Disallow: /profile-posts/
Disallow: /misc/style?*
Disallow: /misc/quick-navigation-menu?*
Disallow: /lost-password/
Disallow: /poll/
Disallow: /sportsbook/
Disallow: /search/
Disallow: /private.f175/
Disallow: /admin.php
Disallow: /express/
Disallow: /off-topic.f12/
Disallow: /nsfw.f106/
Allow: /

robots.txt test against base URL:



/////////

URL inspection test:



//////////

Request Indexing:





///////////

Pagespeed Insights:

 

[king8084]

Really hoping for some suggestions here. I'm grasping at straws...

ran a search on redbot.org and it also confirmed a 403 forbidden error:

https://redbot.org/?uri=https%3A%2F%2Fwww.sectioneighty.com

 

[king8084]

update: i FINALLY figured out what the problem is... there's an issue with a custom add-on that's somehow causing the 403 errors. disabling the add-on and/or removing it from my widget framework on my homepage instantly allowed google to pass the live test + all of the above tests.

i've got to reach out to the developer to figure out what's going on... it's a pretty simple mod tbh.

 

[dking]

What add on, you just described exactly what is happening to me..

 

[king8084]

What add on, you just described exactly what is happening to me..

it was a custom add-on, so doubt it's the same. if you're curious, it's basically a re-branding of the watched threads functionality that i call "Reading List" that i incorporated into the widgetframework add-on. not relevant, but it looks like this:




...i'd suggest to just start ripping out the wires one by one and test the page on redbot.org -- the scan will reflect the change as soon as it takes place.

 

[dking]

it was a custom add-on, so doubt it's the same. if you're curious, it's basically a re-branding of the watched threads functionality that i call "Reading List" that i incorporated into the widgetframework add-on. not relevant, but it looks like this:

View attachment 207758


...i'd suggest to just start ripping out the wires one by one and test the page on redbot.org -- the scan will reflect the change as soon as it takes place.

Thank you!