Various robots ignore both the Disallow: /misc/cookies directive in robots.txt and the noindex directive used in the misc_cookies template, and still follow the generated cookie consent links.
One example is the robot identified by XenForo as Meta External Agent, but the same behavior can also be observed with other crawlers and bots.
This results in a significant number of unnecessary requests to /misc/cookies, which XenForo correctly handles and rejects when the required token or session context is missing, typically returning HTTP 400 responses. Example of a log entry:
The issue is that the cookie consent actions are exposed as crawlable URLs in the generated HTML source:
Modern crawlers often follow such links regardless of nofollow, noindex, or robots.txt directives.
It may be worth considering a different implementation of the cookie consent mechanism, for example using non-crawlable actions (POST requests, JavaScript-based handlers, or other approaches that do not expose action URLs as ordinary links in the page source).
If no crawlable links were present in the generated HTML, these bot requests would not occur in the first place, reducing unnecessary PHP execution and request processing for large numbers of crawler visits.
One example is the robot identified by XenForo as Meta External Agent, but the same behavior can also be observed with other crawlers and bots.
This results in a significant number of unnecessary requests to /misc/cookies, which XenForo correctly handles and rejects when the required token or session context is missing, typically returning HTTP 400 responses. Example of a log entry:
Code:
2026-11-10 17:36:25 Error 57.141.22.6 400 GET /misc/cookies?update=1&accept=1&t=1793725691%2C932cc443111b1822e75ef2ebbc42078c HTTP/2.0The issue is that the cookie consent actions are exposed as crawlable URLs in the generated HTML source:
Code:
/misc/cookies?update=1&accept=1&t=...
/misc/cookies?update=1&reject=1&t=...Modern crawlers often follow such links regardless of nofollow, noindex, or robots.txt directives.
It may be worth considering a different implementation of the cookie consent mechanism, for example using non-crawlable actions (POST requests, JavaScript-based handlers, or other approaches that do not expose action URLs as ordinary links in the page source).
If no crawlable links were present in the generated HTML, these bot requests would not occur in the first place, reducing unnecessary PHP execution and request processing for large numbers of crawler visits.
Upvote
0
