[TAC] Bot Arrestor 2.0.12

I've fixed the install issue
array_key_exists() expects parameter 2 to be array

I've also added one extra check to session switchers (not just a javascript check but secret ingredient 1.. this should really put my mind at ease that these are always bots)

On some forums, a user can get redirected from forum/163 to forum/myforum.163

previously I was logging the redirect to the recentVisits, this is not fair on humans, I've now fixed this to allow the Xenforo canonicalizeRequestUrl to be applied before counting recentVisits

fixed the proxy_header bug

re-arranged code for earlier detection

Game Changer: Secret Ingredient 1 & 2

This version takes a giant leap at detecting more bots than ever. However, the method should not be revealed.

I have added an option "Tac Dedos Secret Ingredient 1 & 2"..

Tac Dedos Secret Ingredient 1 Will only detect bots that are
1) Logged out
2) Have no javascript
3) Fake their user_agent to look like normal browsers (but act like bots)
4) Do not do the normal things that browser must and always do

- It renders many of the other TAC Dedos methods almost redundant, since it catches bots much quicker and more reliably.

Turn On Secret Ingredient 1:
The mechanism for this detection should not be published. This avoids scrappers / spammers trying to get around it, it also avoids users trying to copy this mechanism. This mechanism is incredibly effective, it simply distinguishes browsers from bots (so bots that are browsers may still get around this, but they are rare). You may find that mechanism is far more effective at catching scrappers and spam bots after a just a few attempts

If you should have genuine reasons for discussing the mechanisms with me, I may do so in private messages, if there is good reason. Please do not talk about this mechanism publicly. It works extremely well, I don't want bots to figure out ways of bypassing this any time soon!

Turn On Secret Ingredient 2: This is mentioned in more detail within the options

It's annoying when you see that DeDos has logged some bots/scrappers, but they have only been logged with a friendly warning message (and sometimes multiple times, or as a cookieless user). For this reason I have added two new options:

Maximum Friendly Messages
If a user hits the maximum number of friendly warning messages (or more) within the users session, the user will be added to the malicious DeDos cache (and thus locked out)

Cache Cookieless Bots Earlier
Use the new DeDos settings to add cookieless bots to the cache. Cookieless bots are far likelier to be spam bots /scrapers than real users, so with this option ticked, cookieless users will get added to the cache immediately (instead of showing the 1st friendly warning message)

This is still 0 query overhead, and uses cache for previous DOS detection.
For blocked bots there is now the option to synchronise the cache, so their IP address are added to the .htaccess file when added to the cache, and removed when removed from the cache.

Using the htaccess file is obviously lower resources (0 queries instead of using 1 query to obtain the cache)

This is optional, since not everyone will want their htaccess file dynamically updated (a back up is created if it does not already exists)

Blocked IP addresses are added to the bottom of the htaccess file in the following format:

Code:

# Start DeDos Deny Block DO NOT MODIFY #
order allow,deny
deny from 107.150.48.51
deny from 91.200.12.93
deny from 91.200.12.78
deny from 91.200.12.10
deny from 23.20.22.2
deny from 91.200.12.65
deny from 77.173.149.78
deny from 86.176.154.235
allow from all
# End DeDos Deny Block DO NOT MODIFY #

• fixes related to js detection.
• removed xenforo search from detection area (this is never crawled by spam bots / scrappers)

This is still 0 query overhead, and uses cache for previous DOS detection.

This still works out of the box, and you do not need to change any of the default settings. I've added a few options for various forum set ups, but the default is designed to work for all forums.

• I've change the default times again Friendly: 8 Requests in 7 Seconds, Malicious (added to the cache): 9 Requests in 6 Seconds.
• Added a link from the user logs, view log IP address, to your IP info page (http://whatismyipaddress.com/)
• Added a link from the cache logs (locked out IPs) to Google, displaying possible Malicious activity for this IP address
• There are now 3 ACP options to avoid real users getting added to the cache
  • 1) Avoid Cache for Logged In Users: Avoid logged in users ever being added to the cache (regardless of how many times they refresh / visit pages)
  • 2) Avoid All DeDos for Logged In Users: This avoids logged in users ever getting added to the cache, they also never have to see the freindly warning message
  • 3) Automatically Remove JavaScript Users From Cache: This setting is for peace of mind. False positives should never occur. However, if your DeDos settings are too harsh (or areas force users to DOS certain pages), then a user could be added to the cache. If a user is added to the cache, they will be locked out of the site with a 401 Unauthorised message. On the 401 page, this option detects the JavaScript of the user, and if found it removes them from the cache immediately, prevents them from getting re-added, logs the information, then redirects the page (The 401 message will only blink at the user) This allows you to detect false positives from the logs , adjust your settings correspondingly, and avoid real users getting locked out from your site
• I now get the response of threads before applying DeDos checks, this allows page number / page title redirects to occur before a count is added (it's not fair on human to count redirects as a recentVisit)
• I've removed counting Likes, Reports and Quotes from Post so they don't get added to recentVisits. Some users really do click on 10 - 20 likes in a very short time frame, if real users do it, I don't want to catch them. So for Threads and now Posts, only the actionIndex (viewing the actual thread/post page) is added to recentVisits.
• Turn Off Portal DeDos Detection: I've added an option to not apply DeDos to the portal page. Some portals (custom or other) use lazy load. If not designed cautiously, real users will call lots of pages in a very short time (effectively DOS attacking your site). Since the portal page is very customisable and some plugins allow this, I've added an option to turn DOS detection off for the portal page.
• Experimental Option - Cookieless Bots In Cache: (not to be used on busy forums). Some malicious bots (it would seem about 1 in 10 from my logs) do not use cookies, this means there is no session data on your site for that user. If there is no session data, the recentVisits for that user can not be logged in their session. One way around this, is that if a user is cookieless, log their recentVisits in the global cache (and immediately remove it from the global cache if it's > 10 seconds, or the user suddenly has a cookie). The reason this option should not be use on large forums: the very 1st visit a user makes, the user appears to be cookieless, and only creates the cookie when going to the next page. So on busy forums, there will be many 1st time visitors recentVisits logged in the global cache (and only removed after going to the next page / waiting 10 seconds). I recommend not ticking this option if you have more than 100 visitors on line at any one time. I have hard coded it so there will never be more than 50 users with recentVisits added to the global cache to avoid issues even if large forums tick this option.

So we have a fairly robust method of never detecting logged in users, and a method of avoid detecting logged out real users...
(If logged out real users are ever detected by DeDos, they are immediately removed from the cache and the logged information flashes, letting you know you may need to change your settings, or look into that user)

This pretty much just leaves bots.

We already have robust methods to avoid detecting spiders
We look for core know spiders and remove them, then look at the user agent and remove further possible spiders, then confirm that the user_agent does indeed look like a browser... malicious bots almost always disguise their selves to look like modern browsers (and since we protect both logged in and logged out users, we should just be left with malicious bots spam-bots/scrapers)

The cache should be cleared out automatically (locked out IPs), failing to do so allows the cache to build up

- On adding a new entry to the cache, old entries are now checked and removed if older than $timenow - $maliciousCacheTime

- This is an important bug fix

I've added an option in the ACP to completely avoid applying DeDos to logged in users (since this plugin mainly focuses on spam bots / scrappers)

I've also now change the default setting, so instead of upgrading, using un-install -> install new plugin

I've change the friendly default settings to : 9 requests in 8 seconds
and Malicious: 9 requests in 6 seconds

This means users are far less likely to ever bump into the friendly message (and find it harder to bump into the Malicious message)

It does mean it will catch far less bots, however FBHP catches a high % of spam bots, DeDos focuses on catching scappers that would usually use a lot of resources (so just the hard hitting ones)