Google Search Console Valid Pages Dropping

[Alfuzzy]

When I do a "site:mywebsite.com" in Chrome for my forum (for number of results)...and when I check the "Valid" indexed pages in Google Search Console...these have been slowly dropping for about the last month (when previously these numbers were steady or slowly growing).

My forum has been "normally busy" as usual...with the number of pages viewed/day steady or slightly higher...and have similar number of new threads created/day...or slightly more. Definitely no drops in site traffic or site activity.

Any ideas why these numbers might be slowly/steadily dropping? If additional info is needed...please let me know.

Thanks

 

[djbaxter]

This has been an issue for a while now:

Pages Dropping Out Of Google's Index? More Google Issues With Ranking Fluctuations Reported

This morning I am seeing a number of complaints around pages being dropped from Google's search index. This is coupled with a few of the automated tracking tools showing some ranking turbulence this morning in Google Search.

www.seroundtable.com

Google, as usual, makes changes all the time to their algorithms and to their services like Search Console and Google My Business. And generally speaking, when they make a change or try to fix something, it screws up and creates other issues. One of the reasons is the complexity of their algorithms and services. Another is that different departments are responsible for the different services and they rarely or never talk to one another.

My personal opinion is that Google has become so complex in the past few years that we are going to see more and more of this going forward.

Most of the time, this sort of thing gets fixed eventually... Until the next time one of the Googlers breaks something.

 

[JoyFreak]

Yeah something is definitely going on with Google in the past couple of months or so. I’ve seen my ranking get worse. It’s likely to do the their recent update that rolled out around then. The ability to index your pages via the search console has been disabled due to on-going changes that I believe will go on till March 2021. I hope they fix whatever issue that’s causing this soon.

 

[Alfuzzy]

Thanks guys...helps me feel a little bit better.

@DJ and anyone else with thoughts/suggestions...a refresher:

A couple months ago (mid-September) I posted a "Optimizing Google Crawl Budget" question here at XF...where via suggestions from that thread...I modified my XF site map settings & modified my robots.txt file.

Optimizing Google Crawl Budget

I'm trying to optimize my sites Google crawl budget. My understanding is...items can be added to a sites robots.txt file to prevent Googlebot from crawling/indexing unnecessary items...and thus better utilize the Google crawl budget for the important stuff. Can some of the experts out there...

xenforo.com

Before these modifications valid URL's indexed via Google Search Console for my site were slowly growing. Since that time Google Search Console valid indexed URL's have basically dropped to about 40% of what they were in mid-September. Turns out the article DJbaxter linked above was from September 30, 2020 (shortly after I made my changes).

Does it sound like I unfortunately made the changes to my XF sitemap settings & robots.txt file just before the changes Google made...and the drop in valid indexed pages is due to the changes at Google...and not the changes I made?

Thanks

 

[Ozzy47]

Does it sound like I unfortunately made the changes to my XF sitemap settings & robots.txt file just before the changes Google made...and the drop in valid indexed pages is due to the changes at Google...and not the changes I made?

That is a likely scenario.

 

[Alfuzzy]

Good deal...thanks.

Friggin Google! Lol

 

[djbaxter]

I doubt that any of your changes to robots.txt were responsible. I think it's almost certainly an issue on Google's end, not on your end.

For one thing, you're certainly not alone in having pages dropped from Google Search in recent months.

Here's more:

Your Indexed Pages Are Going Down – 5 Possible Reasons Why

Have you noticed a decrease in the number of indexed pages Google is reporting? Here are a few tips on how to diagnose and fix the issue.

www.searchenginejournal.com

Pages Dropping Out Of Google’s Index With More Google Ranking Fluctuations? – Maxxhost – Web Hosting

maxxhost.net

Pages Dropping Out Of Google’s Index? More Google Issues With Ranking Fluctuations Reported – SEO Focus

seo-focus.com

 

[Alfuzzy]

Thanks for all the great info DJ. I'm just sort of concerned how low will the "Valid Indexed URL's" go before things settle.

For sites that get much of their site traffic from Google search results...if Valid Indexed URL's continue to drop...that means less traffic...and less traffic can lead to less site revenue.

If I do a "site:mywebsite.com" (no quotes) in Chrome...this number is down about 60% since mid-September. And Google Search Console "Valid Indexed URL's...is down about 18% since mid-September. And both values continue to drop.

 

[djbaxter]

Well, for starters when Google experiences issues like this, the reality is that there is nothing at all you can do except wait it out until Google fixes it. I learned a long time ago to set my priorities by asking this question: Is there anything I can do about this issue now, today? If the answer is no (as in the current case), I just move on to the next item on my to-do list and try to put it out of my mind.

Of course, in the case of the site:mywebsite.com search, I vaguely remember an announcement in recent weeks that Google had changed how they serve up results for those search queries. In that case, you can expect the results you get from that search query to be less accurate than what you see in the Search Console (although there have also been several glitches in the Search Console reports recently as well).

Are you seeing these numbers dropping progressively over time now? or is it more or less static after the initial decreases in late September/early November?

 

[djbaxter]

Here's some information about glitches in the Search Console:

Google Confirms Search Console Reporting is Delayed

Google notified site owners that it’s experiencing “longer than usual” delays in Search Console reporting.

www.searchenginejournal.com

Data anomalies in Search Console - Search Console Help

What's up with my graph?On rare occasions, there might be an event in Search Console that could affect your report data. For example, if we change our data aggregation methods or there is a logging er

support.google.com

More Google Search bugs: Top stories indexing snag, coverage report delays

Google had yet another indexing issue and the search coverage report in Search Console is very delayed.

searchengineland.com

 

[Alfuzzy]

Are you seeing these numbers dropping progressively over time now? or is it more or less static after the initial decreases in late September/early November?

Slow steady decrease since late September/Early October (about 60% total decrease now). During this time site traffic/day (Page Views)...new threads created/day...posts/day...have all remained mostly steady. Previous to this time (September)...these numbers were slowly increasing.

I'm guessing normally with these sort of Google modifications...changes are rather immediate & settle out quickly. This happened with the large Google algorithm change back in May (saw a big-sudden decrease in site traffic via Google Analytics). With this Google September change...doesn't seem to be sudden/rapid...but slow-steady decrease.

 

[djbaxter]

I'm guessing normally with these sort of Google modifications...changes are rather immediate & settle out quickly. This happened with the large Google algorithm change back in May (saw a big-sudden decrease in site traffic via Google Analytics). With this Google September change...doesn't seem to be sudden/rapid...but slow-steady decrease.

No not always. There have been some significant bugs in Google My Business which have been ongoing for several months now. And the complaints about the Search Console glitches go back to May 2020 or earlier.

 

[Alfuzzy]

As you mentioned above...if you can't do anything about it...then not much purpose in stressing. Since I wasn't aware of the September Google changes...I assumed my XF sitemap settings & robots.txt file changes were responsible for the drop in numbers (thus my concern).

But now that I know about the Google changes (which are most likely responsible)...now I can move on & concentrate on stuff that I can impact. Thanks

 

[jgaulard]

I am no expert, so please do your own research regarding this issue. I've found that @djbaxter gives excellent advice here, so take what he says seriously. I have something to add though. I was just reading a similar thread to this and I replied there. Please take a look:

Decrease in Google indexed pages

My forum always used to rank pretty well. However, things have changed in the last few months and I'm trying to figure out if it's just me if if it is more widespread. I've experienced a dramatic decrease in the amount of pages indexed by Google (as shown in Google Search Console) and I'm...

xenforo.com

From what I've been noticing, it appears that the more "non-essential" pages that Google crawls (internal 301 redirects, 403 errors, noindex pages), the less Google likes a website. I've allowed Google to crawl all of these types of pages in the past and my page indexation rate has dropped as a result. When I blocked those types of pages in the robots.txt file, the indexation rate oddly rises. I'm not sure if this is a new thing, but something is definitely going on. Take a look at the post I linked to above and let me know your thoughts.