Closed Sort by thread title is very expensive for "AI" crawlers to hammer at

X

Xon

Well-known member
Licensed customer
Affected version
2.3.6
I've had a couple cases where clients have been significantly impacted by crawlers hammering at an XF forum view with a non-default sort option. By far the worst is the sort by thread title option.

I'm not sure what the best solution, but for my clients; bespoke disabled custom sorts for guests/robots while allowing it for members has been a significant performance saving.
 
Last edited:
Xon said:
I've had a couple cases where clients have been significantly impacted by crawlers hammering at an XF forum view with a non-default sort option. By far the worst is the sort by thread title option.

I'm not sure what the best solution, but for my clients; bespoke disabled custom sorts for guests/robots while allowing it for members has been a significant performance saving.
Click to expand...
Is this the setting you're mentioning?

Screen Shot 2025-05-13 at 12.06.24 PM.webp
 
The prefix filtering is separate from the sorting choices having surprise expensive options.

But it has the same goal of trying to migrate aggressive crawling from making the site unusable for people
 
Automated triage: Design issue

Thank you for reporting this issue. Sorting by title works as written, but nothing in the database speeds it up, so on a large forum each sorted page sorts every thread in that forum. Any visitor, crawlers included, can use these sort options. Limiting who can use them, or adding an index, would be a design change, not a bug fix.

This is an automated assessment. A member of staff may review and update it.
 
Back
Top Bottom