XF 1.5 Sitemap Error Warning 403s Threads Behind Logon

[XenBurger]

In order to really hone-in on our quality content in Google rankings, we put our "Off Topic" forum behind the login. There are tens of thousands of posts in that forum, and the XenForo sitemap is still generating them as part of the sitemap. Even though technically anything that is behind a login (not public) shouldn't be in a Google sitemap.

How can we get these out of the sitemap? Another problem that is happening because of this, is that our crawl errors page is constantly full of thousands of 403 errors which I would prefer not to be there, so we can focus on the real issues. They are drown out by these unnecessary 403's.

Google keeps trying to find them because the sitemap has them in there.

 

[Mike]

The sitemap specifically checks that a thread is viewable (as a guest) before it's included, so if you have removed guest viewing permissions from the forum, there's no reason for it to still be included. Have you confirmed whether specific URLs are in newly built sitemaps (as in searching the XML directly)?

The 403s aren't necessarily going to be because they're in the sitemap. Google would already know about the URLs (from when they were public) so it will potentially recheck them and discover that it's now getting a 403. You can likely just clear those errors from webmaster tools directly.