XF 2.3 3k+ 'Blocked due to other 4xx issue' in Google Search Console

[Anatoliy]

They appeared after I upgraded to 2.3.2 about a month ago.
All of them are these types:
domain/misc/style-variation?reset=1&t=1724763460%2Cad52e9ce6de9c6b628dd7feaac79357a
or
domain/misc/style-variation?variation=default&t=1724762134%2C9ac39b17ff1a324c9d0afeefa19a8472
or
domain/misc/style-variation?variation=alternate&t=1724762133%2Cea0e99e67f692418bd180af747492fed

Where are those links? Pages are not indexed, so Google doesn't show referring pages.
Why those links are there if they lead to closed doors?
How to fix that?

 

[Brian Jones]

I had something like this a few weeks ago. Check your root for robots.txt and make sure it isn't blocking

 

[Anatoliy]

I had something like this a few weeks ago. Check your root for robots.txt and make sure it isn't blocking

It's not about robots.txt. Those pages return 403 status code.

 

[Anatoliy]

Taking back my question "Where are those links?". They are everywhere. )
Style selector. Here is a code from this page:

<a href="/community/misc/style-variation?reset=1&amp;t=1724876645%2C67071b68455be3f4582b6bfc1b6921b8"
        class="menu-linkRow "
        rel="nofollow"
        data-xf-click="style-variation" data-variation="">

        <i class="fa--xf far fa-adjust "><svg xmlns="http://www.w3.org/2000/svg" role="img" aria-hidden="true" ><use href="/community/data/local/icons/regular.svg?v=1724828734#adjust"></use></svg></i>

      
            System
        
    </a>


        

    <a href="/community/misc/style-variation?variation=default&amp;t=1724876645%2C67071b68455be3f4582b6bfc1b6921b8"
        class="menu-linkRow is-selected"
        rel="nofollow"
        data-xf-click="style-variation" data-variation="default">

        <i class="fa--xf far fa-sun "><svg xmlns="http://www.w3.org/2000/svg" role="img" aria-hidden="true" ><use href="/community/data/local/icons/regular.svg?v=1724828734#sun"></use></svg></i>

        
            Light
        
    </a>


        

    <a href="/community/misc/style-variation?variation=alternate&amp;t=1724876645%2C67071b68455be3f4582b6bfc1b6921b8"
        class="menu-linkRow "
        rel="nofollow"
        data-xf-click="style-variation" data-variation="alternate">

        <i class="fa--xf far fa-moon "><svg xmlns="http://www.w3.org/2000/svg" role="img" aria-hidden="true" ><use href="/community/data/local/icons/regular.svg?v=1724828734#moon"></use></svg></i>

        
            Dark
        
    </a>

I guess rel=nofollow doesn't stop Google from following?

 

[Anatoliy]

So about "how to fix that" question. I guess by hiding that switch for guests.
Is there such an option somewhere in settings? Or it will require a template modification?

 

[Chromaniac]

I had to exclude them in search console. Not sure if there is a better fix.

 

[Anatoliy]

I had to exclude them in search console. Not sure if there is a better fix.

Ho to do that? Please teach me.

 

[Chromaniac]

Google Search Console > Removals > Add something like this:



Now this is not a permanent solution but it got rid of these URLs from search index for me.

Basically, noindex is more useful than nofollow for telling google to not index content. /misc/style page does seem to include <meta name="robots" content="noindex" />.

 

[Anatoliy]

Thanks!

 

[Anatoliy]

So almost month later it's 9500... (

@Old Nick , can this style selector be hidden with display none in extra less?

 

[akok]

Same problem, I asked not to index such links in robots.txt

 

[Jeremy P]

We nofollow the links and noindex the pages. Google doesn't listen or care. There's not a whole lot more that we can do out of the box, but blocking the pages via robots.txt should help.

 

[Anatoliy]

We nofollow the links and noindex the pages. Google doesn't listen or care. There's not a whole lot more that we can do out of the box, but blocking the pages via robots.txt should help.

Thanks.
I created an add-on with a template (PAGE_CONTAINER) modification that removes that style-variation switch for guests.

<xf:if is="!$xf.visitor.user_id">

<xf:else />
$0
</xf:if>

That prevents google from discovering new 'Blocked due to other 4xx issue'. However it already discovered 10k and will revisit them from time to time, will get 403, and those 10k will sit in GSC forever. I added Disallow: /misc/style-variation* in robots.txt. But it doesn't feel like a good solution. As those 10K will be just moved into "Blocked by robots.tx" and sit there forever. Instead I'd like to give a Google a signal that those links are gone and G should not keep them in a list and revisit anymore.

I have an idea, could you please confirm that it will work and I'm not crazy? )

I would add a code event listener (Listen to event:app_complete). In Listener.php I would check if response code is 403, and if uri includes /misc/style-variation, then response would send 410 code.

Please advise.

 

[Jeremy P]

But it doesn't feel like a good solution. As those 10K will be just moved into "Blocked by robots.tx" and sit there forever.

I know this seems like madness to a lot of people, but it's perfectly OK for warnings issued by Google Search Console to sit there forever. Many of them are there to let you know something might be problematic. If they're not actually problematic, then you don't need to fix them. If a page is blocked by robots.txt and Google is letting you know this, you don't need to take further action unless you want it to be crawled and indexed. The warnings are there so you can be sure that you haven't inadvertently excluded some pages that you didn't mean to.

 

[Anatoliy]

I know this seems like madness to a lot of people, but it's perfectly OK for warnings issued by Google Search Console to sit there forever.

I would agree if it would be something like 'Alternate page', but not 4xx errors.

 

[Jeremy P]

That's why I suggested adding them to robots.txt. The heading for these messages is 'Why pages aren’t indexed'. These pages shouldn't be indexed, whether it's because of a 4xx issue or because you've added them to robots.txt. The latter helps ensure Google doesn't waste crawl budget on them, but beyond that there isn't a real problem.



Unless I'm completely misunderstanding and you wanted them to be indexed for some reason...?

 

[Anatoliy]

I'll wait another month, will see what happens.

 

[Alpha1]

I'd still love to see noindex pages automatically blocked in robots.txt

 

[Anatoliy]

I'd still love to see noindex pages automatically blocked in robots.txt

Or at least a warning/notification from XF "if you upgrade to this version and not add to robots.txt this rule, Goggle will find on your domain as many 4xx errors as amount of threads you have".

 

[Mr Lucky]

I'm not too bothered about guests being able to change variations so I have removed the variation switch from page container altogether and put into user account (account_visitor_menu template) which I think is a better place anyway.

But I notice the same thing (Blocked due to other 4xx issue) in regard to misc/cookies, e.g. https://logic-users-group.com/misc/cookies?update=1&reject=1&t=1726926344%2Cf05567d420c75a328a87d3e96240fd8d