How can I fix this unnecessary Google problem?

[Garfield™]

There are currently 10,000 links blocked by Robots.txt, and this number will continue to grow. This is actually why I'm bringing this up, even though I know there's no solution. That's why it's increasing.

The esteemed global giant Google doesn't know that the /misc/style query of a popular software like XenForo is a theme change or inter-theme transition link. We block these links with robots.txt to prevent them from being indexed and our crawl budget from being wasted. Because, again, the esteemed global giant Google, if we didn't block them, might index these URLs instead of the content itself.

Is there a way to remove this query from the "not indexed" or "blocked" section? If I remove this ban from robots.txt, will it seriously harm my crawl budget? Thank you.

https://teknolojiforum.com/robots.txt

You blocked him and told him not to index him, and then you said you were upset because he did that. Even if I didn't block him, he'll still get indexed this time. Because he doesn't know what to index and what not to index.

 

[Kirby]

The fix (for future URLs) is rather simple:
Do not use a link, it's not required anyway.

K

Future fix Thread 'Google tries to crawl links from style variation chooser menu'

Jan 28, 2025

We nofollow the links and noindex the pages. Google doesn't listen or care. There's not a whole lot more that we can do out of the box, but blocking the pages via robots.txt should help.

This does not really help:
If crawling is blocked via robots.txt Google still indexes those pages without crawling even though the links are nofollow.

What could help:
Don't use links for live switching; either omit href in this case or do not use <a> at all.
Live switching is done via JS exclusively, so those links are not required...

• Kirby
• Replies: 13
• Forum: Future-fix bug reports

• 

 

[Garfield™]

The fix (for future URLs) is rather simple:
Do not use a link, it's not required anyway.

K

Future fix Thread 'Google tries to crawl links from style variation chooser menu'

Jan 28, 2025

We nofollow the links and noindex the pages. Google doesn't listen or care. There's not a whole lot more that we can do out of the box, but blocking the pages via robots.txt should help.

This does not really help:
If crawling is blocked via robots.txt Google still indexes those pages without crawling even though the links are nofollow.

What could help:
Don't use links for live switching; either omit href in this case or do not use <a> at all.
Live switching is done via JS exclusively, so those links are not required...

• Kirby
• Replies: 13
• Forum: Future-fix bug reports

• 

I guess in 2.4 this is no longer a link but a js process?

 

[Chromaniac]

I did this on 2.3. Not a solution but get rid of the links.

XF 2.3 Post in thread '3k+ 'Blocked due to other 4xx issue' in Google Search Console'

Aug 29, 2024

Google Search Console > Removals > Add something like this:



Now this is not a permanent solution but it got rid of these URLs from search index for me.

Basically, noindex is more useful than nofollow for telling google to not index content. /misc/style page does seem to include <meta name="robots" content="noindex" />.

• Chromaniac

• 

 

[Garfield™]

I did this on 2.3. Not a solution but get rid of the links.

XF 2.3 Post in thread '3k+ 'Blocked due to other 4xx issue' in Google Search Console'

Aug 29, 2024

Google Search Console > Removals > Add something like this:



Now this is not a permanent solution but it got rid of these URLs from search index for me.

Basically, noindex is more useful than nofollow for telling google to not index content. /misc/style page does seem to include <meta name="robots" content="noindex" />.

• Chromaniac

• 

There are 10k connections in total, can I remove them all in bulk?

 

[Chromaniac]

please see the post. you are basically telling google to ignore all links on the misc/style route. so you do not need to remove links manually.

 

[Garfield™]

please see the post. you are basically telling google to ignore all links on the misc/style route. so you do not need to remove links manually.

This is a temporary removal, not a permanent one.

 

[Chromaniac]

yes. permanent solution it seems would be part of 2.4. other than that i guess you would have to dive in template and remove the code for the style switcher i guess.

maybe @Kirby can post the changes he mentioned in this post