XF 2.0 Google Maps Indexing Issues with Content Guests can't See

[The Dark Wizard]

Hi,

So this is an issue I've been slowly monitoring over the past year as it has only become prevalent after updating to xF2. The issue is simple, Google is trying to index links that Guests can't see and thus reports them as errors against your site. This might include Threads in a Member's only area or profiles who have had their privacy settings locked down by members.

I was under the impression that xF instructed Google to not index things it can't access, so thus I'm reporting this as a bug. I believe these links that Guests can't see are being sent via the sitemaps.

Note: This information is made available to me via the search console (formally webmaster tools).

 

[Chris D]

Honestly, we're going to need more information besides that before we can action anything here.

We definitely do not log any private URLs via the sitemap so I recommend checking first to see whether those URLs you suspect are in the sitemap. Obviously your sitemap may be fairly large but once you have found the thread entries the threads should be in thread ID order. If you can locate something in there that shouldn't be there, then that gives us something to work with.

Otherwise the more likely scenario is that these URLs are being crawled naturally through links elsewhere on your site or in other locations on the web.

It's then also important to note that these "errors" are mostly informational. If a huge majority of your possible URLs are returning a 404 or 403 then, clearly, that's something you need to sit up and take note of. Otherwise if there's a relative "handful" and those links are expected to be 404 or 403 then this is really nothing to worry about, nothing you will be penalised for and something that will resolve itself over time.