Indexed/Not indexed (by Google) pages ratio

[Anatoliy]

I have 74k not indexed and 19k indexed. Is it standard ratio or I messed something up?

My biggest not indexed categories are:
Blocked by robots.txt 21k
Crawled - currently not indexed 19k
Page with redirect 13k

I guess from point of Google's view my forum is a dump of garbage if only one out of five pages deserves to be indexed.
What's the scoop?

 

[CedricV]

Pretty normal. Those are usually pages that are not accessible for guests.

 

[Anatoliy]

Pretty normal. Those are usually pages that are not accessible for guests.

Thanks for quick reply!
But my brain tells me, that those links, that lead to pages, that are not accessible for guests, just shouldn't be displayed for guests. )

 

[Old Nick]

Those pages are not displayed to guests, it is the links that lead to them that are. The robot follows a link and realizes that it does not have access to the page.

 

[Anatoliy]

Those pages are not displayed to guests, it is the links that lead to them that are. The robot follows a link and realizes that it does not have access to the page.

Yeah, I got that. I can't get why I would want to show to guests so many locked doors with a sign "you can't enter".

May be a better solution would be to show just those doors that can be entered?

 

[Old Nick]

If you want to open all the doors, open them, it's up to you. But also think about the crawl budget...

 

[CedricV]

Yeah, I got that. I can't get why I would want to show to guests so many locked doors with a sign "you can't enter".

May be a better solution would be to show just those doors that can be entered?

You’d think that, but in reality those links are hardly entered. I wouldn’t worry about it too much.

 

[Anatoliy]

If you want to open all the doors, open them, it's up to you. But also think about the crawl budget...

Probably it's my rusty English.)
If you decided to hide from a crawler some pages, you would probably hide from a crawler links to those pages, right? Why would you decide to show a crowler links that return 403 forbidden error?

Or I'm missing something?

 

[Old Nick]

May be a better solution would be to show just those doors that can be entered?

Technically speaking you can hide something from a visitor simply by not displaying it on the front end but the hidden element (link, image, etc.) will still be in the source code and this is where the bots do their market... So as Cedric says, don't worry too much about it, it's just data in the GSC, the main thing is elsewhere.

 

[Anatoliy]

You’d think that, but in reality those links are hardly entered.

By guests? Or by Google spider? It looks like google enter those links 80% out of all it's visits.

 

[CedricV]

By guests? Or by Google spider? It looks like google enter those links 80% out of all it's visits.

Yes by guests. Spiders find anything. Guests usually don’t.

 

[Anatoliy]

Technically speaking you can hide something from a visitor simply by not displaying it on the front end but the hidden element (link, image, etc.) will still be in the source code

???
sorry, but no.

 

[Old Nick]

I learn every day.
How do you make HTML elements disappear from the source code while displaying them to your members?

 

[Anatoliy]

Yes by guests. Spiders find anything. Guests usually don’t.

Ok, now I understand your thoughts. You mean those links are say somewhere on a user profile page under some tab. Human visitors hardly ever hit them, but bots hit everything regularly.

Absolutely agree with that .

But it doesn't answer my question - why would I want to show that tab to guests?

It's easy to put a conditional "if user" on that tab in a template, but XF developers didn't do that... so I'm trying to figure out - what am I missing )

 

[Anatoliy]

I learn every day.
How do you make HTML elements disappear from the source code while displaying them to your members?

Using conditionals in a template

 

[Old Nick]

Using conditionals in a template

Conditionals impact the front end not the source code.

There is a major difference between a simple visitor and a bot: the visitor sees your page with a browser which displays HTML elements formatted with CSS/JS (to put it simply) but the robot does not see the pages like this, he navigates through the source code...

 

[Chromaniac]

Also focus on submitted pages instead of known pages in Google Search Console. You have no control over known pages many of which are going to be locked to guests and spiders and hence are not supposed to be indexed.

 

[CedricV]

What helps is often just browsing your forum as a guest, looking for errors or things that aren’t supposed to be shown. That is what matters. Like I said, don’t worry too much.

 

[Anatoliy]

Like I said, don’t worry too much.

I'm not worrying at all )
I'm thinking of building an add-on, that will modify templates for guests.
I just don't want to waist a lot of time to find out in the end something, that developers already know, and because of that didn't build template modification that I'm planning to build. )))

 

[Anatoliy]

OK, let's narrow question down to "Blocked by robots.txt 21k".
If I set a rule in my robots.txt that disallows Google to follow 21k links, why do I display them to guests?