Rather impossible for
@CTS :
So as far as I know he doesn't have access to .htaccess (pun intended).
Oh, I seem to have skimmed over that when i was catching up on this thread. That unfortunate that such a feature is not offered on their cloud platform, no less the lack of being able to apply more extensible rules via .htaccess or the likes thereof.
This is very interesting! If that's true Anubis seems to be possibly the only tool that is able to filter them out currently. Could you explain a little more how that works? Does this mean a legitimate client will have to wait between 3 and 15 seconds before gaining access to the URL it requested?
It operates in a similar fashion to how Cloudflare's WAF check functions (just without a manual turnstile UI-bit). In the case of Anubis, it does not utilize any captcha, but operates on the basis of a challenge that has to be solved by a clients machine, automatically. There is no user interaction at all, unless the user clicks the show more details button.
You have the full capability to set who gets to be screened via advanced settings such as a cookie verification (e.g.: user is logged in --> bypass the check), how harshly to screen them (I want IP range of 10.0.0.0/24 to have Challenge of 3 and 10.10.0.0/24 a level of 16), what sections of your website is to be screened (I want
/threads/* to be screened by Anubis and nothing else), etc.
And then you can set varying levels of challenge difficulties. Where 1 to 3 is generally easily solvable by most bots and clients, including these AI scrapers - takes milliseconds to complete. Levels of 4 to 5 begin to really slow down clients connecting to the service, but ultimately depends on CPU speed of the client machine. Majority of bots, including these AI/LM Scrapers have yet to get past a level of 5. I've seen very few get past a level of 4 - but it's very uncommon. Once you get into levels of 6 and higher on a challenge, it can generally take up to a minute to solve - which is not ideal for legitimate clients. A level of 16 effectively becomes 'impossible' to solve - excellent to use as a shun-list.
My only worry is if Anubis becomes too widely used, that these AI/LLM Botnets will be configured to wait it out to continue it's ingestion hell.
