[XF 2.0] Google will only index about 30% of XenForo URLs

Agreed and have done so, but they're all mostly very varied. I don't know what it is. Can't quite put my finger on it.
We all (mostly) know that content is king et al, but this is weird. Some of our content is not being indexed and registered even though it's open & without duplicates.
I think they're (Google) hitting forums for some reason possibly? Maybe it's the structure - mobile Vs /... who knows?
Just don't know why. When checking, everything comes back fine. But not indexed.
Grrr..! :D
Duplicate content could very well be an issue though.
I wonder if "abc.com/event/blah blah " would be seen as duplicate content?
I mean is as much as would similar threads under one category, started the same way, would be viewed as duplicate content? My initial thoughts would be yes, but the content would be slightly different wouldn't it?
Hard telling with Google tbh. They are always tweaking their algorithm and I wouldn't put it past them to put forums at a much lower weight as they see forums as non-authoritative where as blog posts where there is proper mark up on the Author, and that author alone could carry a higher weight in the algorithm if seen as an authority in a niche. That fact alone could be a big part of forum's issues.

When looking at the ones not indexed, they are either thin content or OLD threads. Google wants to see content constantly updated so if a thread goes to the wayside, Google will eventually drop that from their index since that content is now "outdated" in their eyes. Which is sad, because a lot of my "outdated" information is actually still current and people are beginning to search for it again (og Xbox modding). The guides don't change and there has been no new development in the market that we haven't already hit on.
 
OP here. I consolidated some old threads (but with tens of thousands, it’s only a tiny fraction), and increased the default posts/page length from 20 to 50. A couple of weeks later, there was a big bumping indexed pages - from about 14,000 pages indexed to over 47,000. How much of it I could attribute to the changes, versus Google finally deciding to index those pages, I don’t know.

There’s no rule discouraging reviving old threads on the site. In fact, it’s somethinng I encourage, depending on the context - if it’s not a question from years ago by a one-and-done poster, bump away. This is one of the benefits of a message board format over Reddit and Facebook, anyhow - good content doesn’t have to sink away to the depths. It also helps keep content from going “stale” in the eyes of Google.
 
Last edited:
OP here. I consolidated some old threads (but with tens of thousands, it’s only a tiny fraction), and increased the default posts/page length from 20 to 50. A couple of weeks later, there was a big bumping indexed pages - from about 14,000 pages indexed to over 47,000. How much of it I could attribute to the changes, versus Google finally deciding to index those pages, I don’t know.

There’s no rule discouraging reviving old threads on the site. In fact, it’s somethinng I encourage, depending on the context - if it’s not a question from years ago by a one-and-done poster, bump away. This is one of the benefits of a message board format over Reddit and Facebook, anyhow - good content doesn’t have to sink away to the depths. It also helps keep content from going “stale” in the eyes of Google.
Sounds similar to what I had suggested earlier...interesting.
 
Google has confirmed that there is a bug in their system that has been de-indexing pages from its index.

John Mueller confirmed the bug in a recent tweet. He also said that it has been fixed. However, some people are still experiencing the issue.

Here's another report on this story: Google: De-indexing issue now fixed, result of "technical issues" - Search Engine Land
Makes you wonder if Google can continue to be allowed such huge and unregulated influence on the livelihoods of small business owners (and corporations too probably)
 
Hello, I have the same trobule in a couple of forums that I have. Both started being SMF in 2007, afterwards I went to Xenforo and I never recovered the traffic, I understand that for various reasons, new url's, change to https and the non-indexing of many url.

Yesterday I read this message on twitter (in Spanish),
To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Where they mentioned a utility to detect that url were not correctly indexed,


After passing it to my forum, https://www.solott.com, I see that 60% of the urls appear as "not-indexed"

I will try to send them again to google with a new sitemap , and see what happens.

Sorry for my translator English, I hope you like the information.FireShot Capture 074 - Copia de Greenlane Indexation Tester _ - https___docs.google.com_sprea...webp
 
Hello, I have the same trobule in a couple of forums that I have. Both started being SMF in 2007, afterwards I went to Xenforo and I never recovered the traffic, I understand that for various reasons, new url's, change to https and the non-indexing of many url.

Yesterday I read this message on twitter (in Spanish),
To view this content we will need your consent to set third party cookies.
For more detailed information, see our cookies page.

Where they mentioned a utility to detect that url were not correctly indexed,


After passing it to my forum, https://www.solott.com, I see that 60% of the urls appear as "not-indexed"

I will try to send them again to google with a new sitemap , and see what happens.

Sorry for my translator English, I hope you like the information.View attachment 199934
This is going to take me a while! lol I have 17 sitemaps where 16 of them have ~50k threads. I'm running through my first sitemap. Likely going to have to duplicate either the whole spreadsheet or the tabs. Thanks for the link. I think a site my size is going to have a tough time with this spreadsheet. Will have to go through our prominent threads and see if they are indexed as opposed to going through and looking at all the threads not indexed.
 
The problem is , these URLs are already in the original sitemap of the forum, and Google does not index them.

Why?

That is the question, because even we now forward them again into an additional non-indexed url sitemap, nothing guarantees that they really index these pages.

Honestly, I do not understand what happens, but I see something has changed in google.
 
The problem is , these URLs are already in the original sitemap of the forum, and Google does not index them.

Why?

That is the question, because even we now forward them again into an additional non-indexed url sitemap, nothing guarantees that they really index these pages.

Honestly, I do not understand what happens, but I see something has changed in google.
Just because you have a sitemap does not mean all those URLs will be indexed. It's never been that way.
 
I'm also having this issue, and I know that many of the pages are good content resource articles that should not be deindexed for being "thin"

e.g. this one:


I can understand chatty meaningless threads not being indexed, but that resource should be surely?
 
Traffic is down on my site too!

How do I see the number of threads indexed now by Google, versus say in last November when I was still using vBulletin and no https?

Thanks so much!

Asher
 
I'm still getting loads of unindexed pages on Google Console that just say:

Submitted URL blocked by robots.txt

Lat Crawled March 2


This is my robots.txt

(disallowing two pages that are duplicated on another site)

Code:
User-agent: *
Disallow: /resources/mixing-saxophone-backing-track-in-audacity.122/
Disallow: /resources/part-2-eq-compression.123/
Allow: /

Sitemap: https://cafesaxophone.com/sitemap.xml
 
Top Bottom