XF 1.5 403 Google Crawler Errors

king8084

Well-known member
403 Crawler errors have been absolutely killing me and it appears to be something in my xenforo configuration. I've worked with a server admin + with the server/host directly, but both have come up empty from the server-side.. Here's the scoop:

- robots.txt is configured correctly
- pages outside of my main/landing page DO crawl correctly and DO NOT receive 403 server errors from google


////////

robots.txt:
Code:
Sitemap: https://forum.sectioneighty.com/sitemap.php

User-agent: *
Disallow: /account/
Disallow: /conversations/
Disallow: /online/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /login/
Disallow: /best-posts/
Disallow: /profile-posts/
Disallow: /misc/style?*
Disallow: /misc/quick-navigation-menu?*
Disallow: /lost-password/
Disallow: /poll/
Disallow: /sportsbook/
Disallow: /search/
Disallow: /private.f175/
Disallow: /admin.php
Disallow: /express/
Disallow: /off-topic.f12/
Disallow: /nsfw.f106/
Allow: /

robots.txt test against base URL:
1563229930277.png



/////////

URL inspection test:
1563229980250.png
1563230004499.png

//////////

Request Indexing:


1563230034089.png


///////////

Pagespeed Insights:

1563230205809.png
 

king8084

Well-known member
update: i FINALLY figured out what the problem is... there's an issue with a custom add-on that's somehow causing the 403 errors. disabling the add-on and/or removing it from my widget framework on my homepage instantly allowed google to pass the live test + all of the above tests.

i've got to reach out to the developer to figure out what's going on... it's a pretty simple mod tbh.
 

king8084

Well-known member
What add on, you just described exactly what is happening to me..
it was a custom add-on, so doubt it's the same. if you're curious, it's basically a re-branding of the watched threads functionality that i call "Reading List" that i incorporated into the widgetframework add-on. not relevant, but it looks like this:

1564894840437.png


...i'd suggest to just start ripping out the wires one by one and test the page on redbot.org -- the scan will reflect the change as soon as it takes place.
 

dking

Member
it was a custom add-on, so doubt it's the same. if you're curious, it's basically a re-branding of the watched threads functionality that i call "Reading List" that i incorporated into the widgetframework add-on. not relevant, but it looks like this:

View attachment 207758


...i'd suggest to just start ripping out the wires one by one and test the page on redbot.org -- the scan will reflect the change as soon as it takes place.

Thank you!
 
Top