Google Webmaster Tools Coverage Missing tons of URLs

[PJK]

In GWT, I see this:

Dropbox - File Deleted - Simplify your life

www.dropbox.com

Of those 763k, 432k of them are redirects, likely from when I moved the forum from /forum to /, but htaccess has 301 redirects setup so not sure why it would exclude them. Any ideas?

The forum has 1.3M posts, but only Coverage in GWT shows only 167k as valid, a concerning figure. What is the best way to ensure all 1.3M posts are indexed in Google? Any suggestions are much appreciated.

Thanks!

 

[djbaxter]

That's just not going to happen.

Google is likely never going to index all of the threads on any forum. When you are the forum owner, you may think that all of those threads are of burning interest to Google searchers and that all of them should be indexed, but the simple reality is that even among your forum members a lot of them are of limited or no interest to most members, and of even less interest to people who are not members of your forum.

Google's goal is to index content which is of interest to searchers in general, and more importantly these days to answer questions and provide information to their searchers, and to do that they want to target content that provides the best information and the best answers to questions from all of the millions of websites on the net. Google's goal is not to index all of your content, much of which on any forum is arguably trivial chit-chat.

 

[PJK]

That's just not going to happen.

Google is likely never going to index all of the threads on any forum. When you are the forum owner, you may think that all of those threads are of burning interest to Google searchers and that all of them should be indexed, but the simple reality is that even among your forum members a lot of them are of limited or no interest to most members, and of even less interest to people who are not members of your forum.

Google's goal is to index content which is of interest to searchers in general, and more importantly these days to answer questions and provide information to their searchers, and to do that they want to target content that provides the best information and the best answers to questions from all of the millions of websites on the net. Google's goal is not to index all of your content, much of which on any forum is arguably trivial chit-chat.

Good point. I'd be curious if 10% of content on a forum is typically indexed? Is that normal? The fact that most of the excluded are "Page with Redirect" is a bit concerning too as if the redirects were setup correctly, surely that wouldn't cause them to be excluded and that error wouldn't show?

In the sitemaps section, it shows 100,481 URLs found:

Dropbox - File Deleted - Simplify your life

www.dropbox.com

Can you guys post your numbers so we can compare? 100,481 URLs out of 1.3M is concerning, surely the number should be higher than that. But I could be wrong. Curious to hear how others are doing.

 

[rdn]

Mine which I don't care much.

 

[djbaxter]

Google On Content Not Being Indexed

Googlers are often being asked about why content on specific websites do not rank or are not indexed. In this last case, John Mueller of Google responded to one request saying "We just don't index all content - complaining about it on Twitter won't change that."

Google has been saying they do not index everything for years, Google simply won't index all your pages and it is impossible for Google to index everything, Google said. ....

Sometimes there are good pages we don't index, even. There's never been a guarantee that we index it all. Sometimes giving it time helps.