How can I fix this unnecessary Google problem?

[Garfield™]

There are currently 10,000 links blocked by Robots.txt, and this number will continue to grow. This is actually why I'm bringing this up, even though I know there's no solution. That's why it's increasing.

The esteemed global giant Google doesn't know that the /misc/style query of a popular software like XenForo is a theme change or inter-theme transition link. We block these links with robots.txt to prevent them from being indexed and our crawl budget from being wasted. Because, again, the esteemed global giant Google, if we didn't block them, might index these URLs instead of the content itself.

Is there a way to remove this query from the "not indexed" or "blocked" section? If I remove this ban from robots.txt, will it seriously harm my crawl budget? Thank you.

https://teknolojiforum.com/robots.txt

You blocked him and told him not to index him, and then you said you were upset because he did that. Even if I didn't block him, he'll still get indexed this time. Because he doesn't know what to index and what not to index.

 

[Kirby]

The fix (for future URLs) is rather simple:
Do not use a link, it's not required anyway.

K

Future fix Thread 'Google tries to crawl links from style variation chooser menu'

Jan 28, 2025

We nofollow the links and noindex the pages. Google doesn't listen or care. There's not a whole lot more that we can do out of the box, but blocking the pages via robots.txt should help.

This does not really help:
If crawling is blocked via robots.txt Google still indexes those pages without crawling even though the links are nofollow.

What could help:
Don't use links for live switching; either omit href in this case or do not use <a> at all.
Live switching is done via JS exclusively, so those links are not required...

• Kirby
• Replies: 15
• Forum: Future-fix bug reports

• 

 

[Garfield™]

The fix (for future URLs) is rather simple:
Do not use a link, it's not required anyway.

K

Future fix Thread 'Google tries to crawl links from style variation chooser menu'

Jan 28, 2025

We nofollow the links and noindex the pages. Google doesn't listen or care. There's not a whole lot more that we can do out of the box, but blocking the pages via robots.txt should help.

This does not really help:
If crawling is blocked via robots.txt Google still indexes those pages without crawling even though the links are nofollow.

What could help:
Don't use links for live switching; either omit href in this case or do not use <a> at all.
Live switching is done via JS exclusively, so those links are not required...

• Kirby
• Replies: 15
• Forum: Future-fix bug reports

• 

I guess in 2.4 this is no longer a link but a js process?

 

[Chromaniac]

I did this on 2.3. Not a solution but get rid of the links.

XF 2.3 Post in thread '3k+ 'Blocked due to other 4xx issue' in Google Search Console'

Aug 29, 2024

Google Search Console > Removals > Add something like this:



Now this is not a permanent solution but it got rid of these URLs from search index for me.

Basically, noindex is more useful than nofollow for telling google to not index content. /misc/style page does seem to include <meta name="robots" content="noindex" />.

• Chromaniac

• 

 

[Garfield™]

I did this on 2.3. Not a solution but get rid of the links.

XF 2.3 Post in thread '3k+ 'Blocked due to other 4xx issue' in Google Search Console'

Aug 29, 2024

Google Search Console > Removals > Add something like this:



Now this is not a permanent solution but it got rid of these URLs from search index for me.

Basically, noindex is more useful than nofollow for telling google to not index content. /misc/style page does seem to include <meta name="robots" content="noindex" />.

• Chromaniac

• 

There are 10k connections in total, can I remove them all in bulk?

 

[Chromaniac]

please see the post. you are basically telling google to ignore all links on the misc/style route. so you do not need to remove links manually.

 

[Garfield™]

please see the post. you are basically telling google to ignore all links on the misc/style route. so you do not need to remove links manually.

This is a temporary removal, not a permanent one.

 

[Chromaniac]

yes. permanent solution it seems would be part of 2.4. other than that i guess you would have to dive in template and remove the code for the style switcher i guess.

maybe @Kirby can post the changes he mentioned in this post

 

[Tamer]

We use @Kirby ’s excellent add-on to serve the default light theme to guests. Since guests cannot retain style preferences, the variation switcher has no real purpose for them. Because of this, we wrapped the variation button with a guest check:

<xf:if is="$xf.visitor.user_id">
    <!-- Style variation switcher -->
</xf:if>

This removes the switcher entirely from the DOM and Google no longer discovers any new misc/style-variation URLs. That no new URLs will be generated anymore. The remaining ones will gradually drop out, which may take several weeks or even months depending on Google’s recrawl cycle.

Our branding is built around the light theme, so we only allow logged-in users to switch to the dark variation. Given that the transition to 2.4 is expected to be somewhat challenging in terms of timing and technical implications, this approach serves as a reasonable temporary solution for us. At the very least, it prevents additional variation URLs from being generated before we complete the upgrade.

 

[Kirby]

Since guests cannot retain style preferences, the variation switcher has no real purpose for them.

Guests can an will retain their variation choice as long as they don't delete / loose their cookies.

This removes the switcher entirely from the DOM and Google no longer discovers any new misc/style-variation URLs.

I've posted a little template modification to remove the links from the variation chooser menu without completely removing the menu for guests.
IMHO this is a better approach.

K

Future fix Post in thread 'Google tries to crawl links from style variation chooser menu'

Dec 3, 2025

A less invasive option:

style_variation_macros

<xf:macro name="variation_menu_item"
    arg-routeType="public"
    arg-route="{{ link('misc/style-variation') }}"
    arg-variation="!"
    arg-icon="fa-adjust"
    arg-label=""
    arg-selectedVariation="{$xf.visitor.style_variation}"
    arg-live="{{ false }}"
    arg-redirect="{{ null }}">

    <xf:set var="$tag" value="{{ $live ? 'span' : 'a' }}" />

    <{$tag} class="menu-linkRow {{ $variation === $selectedVariation ? 'is-selected' : '' }}"
        {{ !$live ? 'href="' . link_type($routeType, $route, null, {...

• Kirby

• 

 

[Tamer]

Thanks, I appreciate your perspective. Your suggestion is actually quite sensible and I might give it a try later on. Since our brand and color identity are built around the light theme, we simply keep dark mode as a members-only feature and present it as one of the small perks of being logged in. : ) Guests don’t really need extra variations on their first visit, so hiding the switcher keeps things cleaner for us.