Forum has 250K posts and 35K threads but Google only index 22K pages!

[fionix]

It's strange, in the beginning, Google had indexed about 35,000 pages, but over the last year, this number has dropped to around 22,000 and it's still decreasing!What is going wrong here, and is this normal? In SEMRUSH, the site has high authority and also otherwise, only high-quality links, articles, and videos are constantly being added.

 

[James]

Have you increased posts per page?
It's also possible that XF just serves less pages (no pages for categories for example).
Any forums recently made private nodes?

Not sure it's a worry. You still have 20k listings.

 

[fionix]

I have not done anything of this, Google just started to do so for almost a year ago, so we had Theme House switch theme on our forum, that's the only major impact it had... we had 6 months forth and back discussing with them about other issues with Google, they helped here and there to fix some of the issues, so these Google Vitals are all good now.. but the decreasing of the number of pages is not good.

I wonder how many pages other Xenforo webmasters have indexed by Google ?

 

[Chromaniac]

I mean this is totally a Google thing. Mine is also around 30% indexed as per search console. Except for smallish blogs with limited pages, I don't think I have ever seen sites with large number of pages fully or near fully indexed.

There is an add-on here that would put individual pages of threads in sitemap. It did help showing more indexed pages but even that would likely not get Google to show that entire site is indexed. In the end, I think it is more of a data representation issue. Try searching for content it shows crawled but not indexed and see if it does appear on serp.

 

[Mr Lucky]

I wonder how many pages other Xenforo webmasters have indexed by Google ?

The vast majority are pages with redirects, e.g. a page that is a link to a post plus pages that were linked when I had amp addon installed. Those should be redirected I believe.

• Excluded by no index tag - which I have done manually (e.g. old threads with thin content)
• Alternate page with proper canonical. Again - a good thing - e.g. https://cafesaxophone.com/threads/fancy-one-galassine-super-bass-sax.33737/?utm_source=rss&utm_medium=rss (not sure what that is all about though)
• Crawled currently not indexed - I presume that may just be thin content but would need fixing manually

 

[dethfire]

Google has long communicated they don't index everything. Why would they? Most sites have a double-digit percent of garbage. Google can't physically store it all, so they evaluate what is your best content and index that.

 

[Ntown]

It's a non-issue and there's nothing you can do about it. I learned a long time ago that you can't "encourage" or "help" Google.

 

[Mr Lucky]

Google has long communicated they don't index everything. Why would they?

Yes I think it's called "crawl budget" ie there is just so much of a site's pages they will index and given that each thread is at least a page, forums are relatively huge compared to your average common or garden website.

We know that therefore they do favour good informative content and that "thin content" can be ignored and possibly/arguably can cause the whole site to suffer. It's hard to second guess Google but many people think it's a good idea therefore to noindex staff that is useless. e.g a pointless question that nobody answered, or a forum test area, or T & C etc

It's good that xenForo now allows us to noindex entire forums.

 

[alexm]

Ditto. It's Google. Forward and upward!