Known Bots

Known Bots 6.1.0

No permission to download
Overview Updates (36) Reviews (9) History Discussion
v3.x of this addon has been tested with XF v2.2 (previous versions of the addon should also work)
 
mazzly said:
Code: 
3.124.4.30 - - [17/May/2020:22:09:18 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
3.234.221.231 - - [17/May/2020:22:09:18 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
18.141.137.82 - - [17/May/2020:22:10:11 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
54.198.215.110 - - [17/May/2020:22:10:12 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
46.137.192.104 - - [17/May/2020:22:10:20 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
18.156.198.249 - - [17/May/2020:22:10:23 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
54.211.213.171 - - [17/May/2020:22:11:03 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
54.93.236.46 - - [17/May/2020:22:11:04 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
18.141.137.82 - - [17/May/2020:22:11:10 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
52.221.192.72 - - [17/May/2020:22:11:16 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
35.158.137.29 - - [17/May/2020:22:12:03 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
3.235.79.241 - - [17/May/2020:22:12:07 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
54.251.78.22 - - [17/May/2020:22:12:11 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
52.221.192.72 - - [17/May/2020:22:12:14 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
18.141.174.37 - - [17/May/2020:22:13:08 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
18.156.163.52 - - [17/May/2020:22:13:08 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
13.229.207.209 - - [17/May/2020:22:13:15 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
34.238.137.89 - - [17/May/2020:22:13:23 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
3.229.124.106 - - [17/May/2020:22:14:08 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
54.151.207.202 - - [17/May/2020:22:14:09 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
13.229.207.209 - - [17/May/2020:22:14:14 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
3.123.0.61 - - [17/May/2020:22:14:20 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
18.196.203.61 - - [17/May/2020:22:15:09 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
54.157.246.127 - - [17/May/2020:22:15:10 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
18.141.159.13 - - [17/May/2020:22:15:11 -0500] "GET / HTTP/1.1" 200 15441 "-" "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html X-Middleton/1"
There you go :)
Click to expand...

Forgot this one - sorry. Has been added for v3.2.0 release - I'm expecting quite a few new bots to turn up now that I've got auto-detection enabled on some of my sites, so I plan to release v3.2.0 soonish.
 
PHP: 
            $mail->setContent(
                "New bots detected",
                "<ul>" . PHP_EOL . $botList . PHP_EOL . "</ul>" . PHP_EOL
            );
Can you make «New bots detected» a translatable phrase?
 
@Sim Request for feature:

This may need to be a separate addon but having identified bots you don't need or want having an option to click in the list to auto-ban them, either by user agent or by IP, would be excellent.
 
UH Team said:
PHP: 
            $mail->setContent(
                "New bots detected",
                "<ul>" . PHP_EOL . $botList . PHP_EOL . "</ul>" . PHP_EOL
            );
Can you make «New bots detected» a translatable phrase?
Click to expand...

Sure - but please don't send me emails with translated subject lines - otherwise my mail filters won't work. It's fine if you want to have the emails sent to someone else instead.

I'll add a phrase for the next release.
 
Hi!
Did anyone manage to block MegaIndex in nginx?

I add:

NGINX: 
if ($ http_user_agent ~ * MegaIndex) {
    return 444;
}

but this evil spirits still climb.
 
Next update is going to be huge - I have 6 small sites that are running this release with the generic bot detection and between them there were 10 new bots identified already. Then there's all the others I just got emailed this morning - so many bots I've never heard of before! :cool:
 
pilicanbot/nutch-1.16 (pilican.com; bot@pilican.com) is the only unique one in my logs. rest are all connected to mastodon hosts because i share my content on one of their nodes which gets shared in the fediverse.
 
Sim said:
Next update is going to be huge - I have 6 small sites that are running this release with the generic bot detection and between them there were 10 new bots identified already. Then there's all the others I just got emailed this morning - so many bots I've never heard of before! :cool:
Click to expand...

When I did the spider list on vB, I think I was around 1200 bots. It’s amazing how many are out there.
 
Will the email field accept multiple email addresses? I'd like to send it to myself as well as you without manually forwarding it.
 
Nirjonadda said:
@Sim Please can be update Bots Stats show in one line in the sidebar widgets?
Click to expand...
I was just looking at this - your example was not accurate, since the "Total" includes members and guests only and robots are excluded, which is why I put them on a separate line:
Total: 20 (members: 1, guests: 19)
Robots: 170
Click to expand...

Also - if I include the robot count in the brackets, the default sidebar width means that the line will wrap:
1598937247375.webp

I figured it was more appropriate to list robots on a separate line.
 
Sim updated Known Bots with a new update entry:

v3.2.0 big bot update

Lots of new bots
  • 360spider
  • adsbot
  • adstxtcrawler
  • awariorssbot
  • bitlybot
  • boardreader
  • ccbot
  • cincraw
  • clickagy intelligence bot
  • crawlson
  • crsspxlbot
  • datagnionbot
  • deskyobot
  • df bot
  • dnsresearchbot
  • duckduckgo favicons bot
  • eyeotabot
  • feedlybot
  • foocrawlerbot
  • germcrawler
  • google adwords instant
  • gumgum bot
  • hatena
  • hetrixtools
  • hubpages
  • ias_crawler
  • internet structure research project bot
  • jugendschutzprogramm-crawler...
Click to expand...

Read the rest of this update entry...
 
You must log in or register to reply here.

Similar threads

Scandal
  • Question Question
XF 2.2 How to add new bot on known bots?
Replies
3
Views
643
Max Taxable
Max Taxable
Back
Top Bottom