XF 2.3 Old threads not indexed

[Salamanca]

Hi,

I just noticed that old threads are not indexed in Google (searching via google search console)

What could be the reason for this?

Thanks
Alex

 

[Deleted member 184953]

Crawled but not indexed or not crawled at all?

 

[Salamanca]

no crawl and no index.

 

[StarArmy]

Hi,

I just noticed that old threads are not indexed in Google (searching via google search console)

What could be the reason for this?

Thanks
Alex

I've heard that Google and other search engines have a "crawl budget" which means they have a finite amount of time/effort they will spend on a particular website to index it. It could be that this "crawl budget" is exhausted before your forum is completely indexed, and the newer threads are prioritized.

If this is the case (and Google doesn't exactly make clear what's happening, so we have to guess), you could try to exclude non-important pages from index in hopes that the crawl budget will be used in places you consider more important. For example, maybe Google doesn't need to index member profiles or other low-content pages when it could be indexing informative articles or high-impact discussion threads that get you traffic. I hope that helps.

 

[Salamanca]

Wow.
Google gives priority to Reddit, very often with garbage/promotional content and ignores forums like mine with millions of posts over a 20 year period which contains gold information.

 

[Chromaniac]

Google is paying reddit a lot of money to have the right to index their content.

Google Is the Only Search Engine That Works on Reddit Now Thanks to AI Deal

DuckDuckGo, Bing, Mojeek, and other search engines are not returning full Reddit results any more.

www.404media.co

Reddit is blocking every spider through their robots.txt file.

https://www.reddit.com/robots.txt