[TAC] Bot Arrestor

[TAC] Bot Arrestor [Paid] 2.0.12

No permission to buy ($19.00)
Overview Updates (22) Reviews (1) History Discussion
tenants said:
I suspect that they botted, then when back as a real user to see why they couldn't bot ... What I try to do as a priority, is avoid false negatives
Click to expand...

Sure, I would want the same thing - all of my settings are tuned to avoid affecting regular users.
 
tenants updated DeDos - Anti DOS for spam bot/scrapers with a new update entry:

Tac DeDos Antispam - Game Changer Update (secret ingredient 1 & 2)

Game Changer: Secret Ingredient 1 & 2

This version takes a giant leap at detecting more bots than ever. However, the method should not be revealed.

I have added an option "Tac Dedos Secret Ingredient 1 & 2"..

Tac Dedos Secret Ingredient 1 Will only detect bots that are
1) Logged out
2) Have no javascript
3) Fake their user_agent to look like normal browsers (but act like bots)
4) Do not do the normal things that browser must and always do

- It renders many of the other TAC Dedos methods almost redundant, since it catches bots much quicker and more reliably.

Turn On Secret Ingredient 1:
The mechanism for this detection should not be published. This avoids scrappers / spammers trying to get around it, it also avoids users trying to copy this mechanism. This mechanism is incredibly effective, it simply distinguishes browsers from bots (so bots that are browsers may still get around this, but they are rare). You may find that mechanism is far more effective at catching scrappers and spam bots after a just a few attempts

If you should have genuine reasons for discussing the mechanisms with me, I may do so in private messages, if there is good reason. Please do not talk about this mechanism publicly. It works extremely well, I don't want bots to figure out ways of bypassing this any time soon!

Turn On Secret Ingredient 2: This is mentioned in more detail within the options
Click to expand...
 
Ridemonkey said:
Interesting, I turned on the JavaScript detection to avoid false positives and the first one that got deactivated is clearly some kind of bot:

View attachment 81478
Click to expand...

I have just looked at your image again, the red in the logs tells you that DeDos has detected this as a definite bot (as it does in your image), the message tells you : IP added to the cache (this means this bot is locked out and will only get a 401/403 when visiting). If you have ticked the option to update the htaccess, it also means this bot with have a significantly lower impact (lower bandwidth, lower cpu resource)

If you ever have a flashing red row in your logs, it is then that the system is telling you about the false positive (and that your settings are probably too harsh). But don't worry, DeDos would have detected the javascipt of the user and automatically removed the user from the cache. The user... [who is logged out, since there are options to not detect logged in users] may have noticed a 401 message flash up for a split second before being automatically redirected to the page they were viewing - no harm done and they wont get detected for that session ever again . In this case, it is worth thinking about less harsh settings

But, from looking at your logs, it does look to me like that there are many bots hitting you and they are just getting warnings (that''s not good), your setting might be too low (not harsh enough)....

However, there is no need to change them, with secret ingredient 1, I can pretty much reliably detect almost all of these bots, your logs will be flooded with redness (and none that flash) as it picks up so many bots reliably now

Secret Ingredient 1 has a massive impact on the amount of resources bots now take up
 
Ah, I have just seen the "javascript detected, user removed." -- I didn't see it since it wasn't flashing at me in your image :)

I'm going back to my orignal thought

look at the time stamps

When they were detected as a bot, all dos attempts were between 9:28 - 9:29
When they went back to the site (to see why their bot was broken), they were correctly detected as a human (javascript detect) and correctly removed from the cache. This was at 10:51 (more than 1 hour after their bot stopped working)
 
MattW said:
I've started to get a lot of those proxy_header errors as well on my site.
Click to expand...


Sorry its taken so long to get on the case for this

fixed, will release 2.0.01 shortly

It was a simple fix

Code: 
      if($proxyHeadersFound)
        {
          $writer->set('proxy_header', $proxyHeadersFound);
       }
to

Code: 
          $writer->set('proxy_header', $proxyHeadersFound);

I can understand why this error occurs, what I don't understand is why myself or many others never saw it :S
 
Last edited:
Okay, after looking through my access logs, for today and yesterday I've managed to wipe out pretty much all spam bots/scrappers that try to fake their selves to look like browser, with htaccess dynamically updating mostly using secret ingredient 1 & 2. I think that has pretty much nailed it. I'm waiting for distributed bots now, since these are the ones I really found hard to target before

If you find bots (malicious ones) in your access logs that aren't getting killed by dedos, post your sever access logs here and I will look through them (You wouldn't believe the number of hours I've spent looking at these, spotting patterns)
 
@tenants, you are ruthless! Haha. Just updated, 5 mins ago, and have added about 20 bad IP's to the cache with your "secret ingredients".

I have also had what appears to be one false positive with the Same IP Session Switcher. At 9:45, an IP was added to the cache (same IP session switcher, no javascript). At 9:46, javascript detected and IP removed from cache (username logged this time).

Interestingly, 20 minutes later, the IP was again hit with the Same IP Session Switcher (no javascript, not logged in). This time the user didn't log in, and IP was not removed from the cache. Weird.

I'm very appreciative of this addon, and have been using it for months!
 
v2.00.02:
ErrorException: array_key_exists() expects parameter 2 to be array, boolean given - library/Tac/DeDos/ControllerPublic/DeDos.php:56
 
kontrabass said:
@tenants, you are ruthless! Haha. Just updated, 5 mins ago, and have added about 20 bad IP's to the cache with your "secret ingredients".

I have also had what appears to be one false positive with the Same IP Session Switcher. At 9:45, an IP was added to the cache (same IP session switcher, no javascript). At 9:46, javascript detected and IP removed from cache (username logged this time).

Interestingly, 20 minutes later, the IP was again hit with the Same IP Session Switcher (no javascript, not logged in). This time the user didn't log in, and IP was not removed from the cache. Weird.

I'm very appreciative of this addon, and have been using it for months!
Click to expand...

It's pretty incredible how accurate and quick secret ingredient 1 is. I have then been going through the logs and adding host names from the dedos logs to secret ingredient 2, this allows the system to pick the malicious bots up even faster. Now the only things that have a bandwidth / cpu impact are real spiders and humans, my resources have shot down.

The Javascript detection cycle I have seen this myself with another person. This cycle strong implies session restarts (either logging out and in over an over, or closing the browser and opening it over and over)

What we found is that it wasn't actually a false positive, but a bot with javascript enabled (but they only enable it with one of their instances). This is why they get detected, get removed then get detected again

Can you post your logs for when it was caught (click the red row, and paste the info here, including the recent visits, and the do the same for the red flashing row)


Mouth said:
v2.00.02:
ErrorException: array_key_exists() expects parameter 2 to be array, boolean given - library/Tac/DeDos/ControllerPublic/DeDos.php:56
Click to expand...

I'll look into this now, haven't seen it my self
 
Mouth said:
v2.00.02:
ErrorException: array_key_exists() expects parameter 2 to be array, boolean given - library/Tac/DeDos/ControllerPublic/DeDos.php:56
Click to expand...

I've found the issue, it looks like this would only happen if no bots had been detected yet (so 1st install / early on). I've fixed it in v2.00.03


Code: 
if(array_key_exists($stringIP, $dd_known_dos))

to

Code: 
if($dd_known_dos && array_key_exists($stringIP, $dd_known_dos))
 
tenants said:
It's pretty incredible how accurate and quick secret ingredient 1 is. I have then been going through the logs and adding host names from the dedos logs to secret ingredient 2, this allows the system to pick the malicious bots up even faster. Now the only things that have a bandwidth / cpu impact are real spiders and humans, my resources have shot down.
Click to expand...

Overnight, Secret Ingredient 1 and 2 have added 141 IP's to the cache :eek:

tenants said:
Can you post your logs for when it was caught (click the red row, and paste the info here, including the recent visits, and the do the same for the red flashing row)
Click to expand...

Sure. The weird thing here is that the red flashing row (javascript detection) shows a real, logged in legit user.

Red row (added to cache):
Code: 
IP address added to cache: 65.8.151.82
Session User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20100101 Firefox/10.0 (Chrome)
Log Time: Yesterday at 9:43 PM
User: User not logged in
JavaScript Detected: False
Has Cookie: True
Session ID: d8e0b4e05a66de76d87e86febd574ab2
gethostbyaddr(65.8.151.82) = adsl-65-8-151-82.mia.bellsouth.net
Friendly messages in this session: 0
gethostbyname(adsl-65-8-151-82.mia.bellsouth.net) = 65.8.151.82
Proxy Header: No Proxy Found
Recent Visits at Log Time
[unix_timestamp](human_readable_time) => url_location

[1431657835](Yesterday at 9:43 PM) => http://www.talkbass.com/threads/new-project-p-bass-mutt.1152450/
[1431657834](Yesterday at 9:43 PM) => http://www.talkbass.com/threads/new-project-p-bass-mutt.1152450/

Extra Info
a:3:{s:32:"047c5780544be006b15ccddea7e21c3b";a:2:{s:1:"t";i:1431657821;s:1:"i";s:11:"65.8.151.82";}s:32:"4f1a841101a7abf07159e81e60a59410";a:2:{s:1:"t";i:1431657826;s:1:"i";s:11:"65.8.151.82";}s:32:"d8e0b4e05a66de76d87e86febd574ab2";a:2:{s:1:"t";i:1431657835;s:1:"i";s:11:"65.8.151.82";}}

Red flashing row (javascript detected, removed from cache):

Code: 
IP address added to cache: 65.8.151.82
Session User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.152 Safari/537.36
Log Time: Yesterday at 9:44 PM
User: rmvdfor_privacy
JavaScript Detected: True
Has Cookie: True
Session ID: e46aa29ebff411c74ccd67eb3bff1fd7
gethostbyaddr(65.8.151.82) = adsl-65-8-151-82.mia.bellsouth.net
Friendly messages in this session: 0
gethostbyname(adsl-65-8-151-82.mia.bellsouth.net) = 65.8.151.82
Proxy Header: No Proxy Found
Recent Visits at Log Time

 
can you add the recent visits for both too (should be just under the bit you just copy-pasted), staring out the urls is fine, I just need to know what controllers they hit, when and how many times (and if it avoided picking up redirects ... also, if they did actually look like a bot from visits to begin with, recent visits can some times indicate this)
 
This could be 2 users on the same IP, it could also be the same user (one looks more like a bot than the other)

So far what is interesting here is the user agent

When detected as a bot:
Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20100101 Firefox/10.0 (Chrome)

That is Firefox (10!!!!) on Linux... this is fairly strange regardless of anything else

- using a very old browser implies automation ( I personally hate having to update browsers, update webdrivers and re-write things when doing automation with selenium/ant/junit/awt robot, so often I keep my browser at the same older version).

- Using a Linux box is also a little bit indicative (this is usually what I host on / automate from)

- No js is also indicative

- Session switching could just be opening / closing browsers a few times within seconds

When detected as a human (at around the same time):
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.152 Safari/537.36
That is Chrome (current latest version) on Windows 7

This one to me looks like their real browser, and now you know there username

I wonder if they attempted to bot, failed, so logged in to see what happened. I think grabbing the server access logs for this user will be more revealing

Have a chat with this member. I would play it dumb and ask just if they did anything unusual with their browsers (I would be interested in their response). It might be innocent (browsing for Linux FF10 is a bit strange, but not naughty on its own.. no js detection on that browser is even stranger). Assume innocence (but have a good look at the server access logs for this IP). Is that member a well known member?
 
Last edited:
...where is the update man, I went to surrey forum and logged in (just like yesterday & day before for those updates) but the file is not there at the bottom of the 1st post
 
tenants said:
This could be 2 users on the same IP, it could also be the same user (one looks more like a bot than the other)

So far what is interesting here is the user agent

When detected as a bot:
Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20100101 Firefox/10.0 (Chrome)
That is Firefox (10!!!!) on Linux... this is fairly strange regardless of anything else

- using a very old browser implies automation ( I personally hate having to update browsers, update webdrivers and re-write things when doing automation with selenium/ant/junit/awt robot, so often I keep my browser at the same older version).

- Using a Linux box is also a little bit indicative (this is usually what I host on / automate from)

- No js is also indicative

- Session switching could just be opening / closing browsers a few times within seconds

When detected as a human (at around the same time):
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.152 Safari/537.36
That is Chrome (current latest version) on Windows 7

This one to me looks like their real browser, and now you know there username

I wonder if they attempted to bot, failed, so logged in to see what happened. I think grabbing the server access logs for this user will be more revealing

Have a chat with this member. I would play it dumb and ask just if they did anything unusual with their browsers (I would be interested in their response). It might be innocent (browsing for Linux FF10 is a bit strange, but not naughty on its own.. no js detection on that browser is even stranger). Assume innocence (but have a good look at the server access logs for this IP). Is that member a well known member?
Click to expand...

It's a well known member, premium (paid) even. He's contacted us, so I'll let you know what he says. What I copied is the entire log entry (it's empty under recent visits).
 
You must log in or register to reply here.
Back
Top Bottom