XF 2.2 GSC and cookies consent URL

[Old Nick]

Hello, since the January 21th i receive a lot of URL errors in GSC, 3.6K of this kind of URL:
https://xenforo.fr/misc/cookies?update=1&reject=1&t=1674994290,d71df2eb1236ef187440fa26d77167eb
https://xenforo.fr/misc/cookies?update=1&accept=1&t=1675076268,279b19e95b315beba530f32200313e8f

@digitalpoint can you think it could be linked to the Cloudflare add-on ?
@mazzly or your AMP add-on ?

Thanks guys.

 

[digitalpoint]

My guess is that it stems from the fact that XenForo is using CSRF tokens in GET requests and those tokens become invalid before too long. Which means those URLs will throw an error if accessed later (a search engine spider doesn’t instantly spider all links right when it sees them).

IMO, XenForo should change the places where it uses CSRF tokens in GET requests to POST requests (for this and other reasons).

 

[Old Nick]

Thanks,
So the initial "problem" would come from XF but would be revealed by your Cloudflare add-on, did I understand correctly?

It's not very disturbing these errors in the GSC, I also asked for the deletion of URLs starting with https://xenforo.fr/misc/cookies?update. I'm just trying to understand.

 

[digitalpoint]

No, doesn’t have anything to do with Cloudflare add-on. XenForo has a few URLs with expiring CSRF tokens embedded in the URL in a few places (for example the advanced cookie consent pop up). It does that in a vanilla installation. It’s not the “right” way to do it, but it does it that way for some reason.

 

[Old Nick]

Ok, understood. What surprises me is that these errors only appeared in the GSC from January 21, while I installed 2.2.12 and the advanced cookie consent from December 8. Perhaps Google's indexing lag.

 

[digitalpoint]

As an example, you can look at Google's last spidering of the XenForo community page here:

XenForo community

Forum software by XenForo

webcache.googleusercontent.com

If you get the URL on that page that is used to accept cookies, you get an error if you use it:

https://xenforo.com/community/misc/cookies?update=1&accept=1&t=1675858029%2C3266c386fa8755e7f5a6aea7bb8fbf88

...basically the same thing that happens to Google.

You could of course block those URLs with robots.txt since they serve no purpose as far as spidering content, but the "right" thing to do would be for XenForo to switch it's usage of CSRF tokens to be POST requests.

 

[Mr Lucky]

I'm also getting nags from Google Search Console e.g. re page this:

https://logic-users-group.com/misc/...t=1675906055,9ad02247909ecbcca3d19b51a2088921

I'm not quite sure of how robots.txt works with URLS with query strings

What would be the line in robots.txt, would it just be

User-agent: *
Disallow:misc/cookies/

or

User-agent: *
Disallow:misc/cookies

Thanks

 

[SeToY]

What would be the line in robots.txt, would it just be

I added the following to my robots.txt:

User-agent: *
Disallow: /misc/cookies?*

and verified it using a failed URL from the Google Search Console with this tool: https://technicalseo.com/tools/robots-txt/