XF 2.3 Cookies giving me 286K google search console warnings

[MickkD]

Hi All,

not sure if this is a legacy issue or not. but i have 283,000 urls not crawled due to the cookie pages

https://mysite/misc/cookies?update=1&accept=1&t=1722828443%2Cefaeebe9a1693307f38e8af30ffe2f3a

as an example. I searched the forum for "cookies google search console" but it never gave me anything, so im starting a new post.

Thanks
Mick

​

 

[JustinHawk]

not sure if this is a legacy issue or not. but i have 283,000 urls not crawled due to the cookie pages

You should disallow indexing of URL's that starts with /misc/ in robots.txt

Which should resolve your issue!

 

[MickkD]

just added

User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /forums/

User-agent: Amazonbot
Disallow: /threads/*/reply


User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php

Disallow: /misc/cookies?*

Allow: /

should i just disallow the whole of /misc/

thanks for the reply

 

[JustinHawk]

should i just disallow the whole of /misc/

I think you should as there is no such relevant content for google to index!

 

[MickkD]

Perfect thanks,

Lastly, once that's done, can I click the fix button on the console?
Or will it work itself out?

Mick

 

[JustinHawk]

Lastly, once that's done, can I click the fix button on the console?
Or will it work itself out?

I would rather not waste crawl budget on validation, as all new "/misc/" URLs will be ignored by google.

 

[MickkD]

Thanks for all the help, Justin!

Mick