Known Bots 6.1.0

[Sim]

I've updated the overview page and the FAQ page with more information about v4

 

[Sim]

Currently tracking over 1,300 bot user agents in my knownbots app that I use to generate the bot detection lists.

I do have quite a bit of work to do to catch up on the unprocessed bots though!

 

[Nirjonadda]

@Sim After updated v4 then our UptimeRobot used another Site monitoring name. also does added for this addon bot?

 

[Sim]

@Sim After updated v4 then our UptimeRobot used another Site monitoring name. also does added for this addon bot?

View attachment 289944

Thanks - turns out there are a bunch of bots using uptime in their name that were all being detected as the same bot.

I've refined the search strings for each of these bots so they are detected separately - your site should start correctly reporting UptimeRobot.com bots within the next 24 hours, once the cron task runs to update the definitions.

 

[Nirjonadda]

New bots detected [4.0.0]

• curl/7.64.1
• Mozilla/5.0 zgrab/0.x
• Mozilla/5.0 (Linux; U; Android 13; PHP110 Build/SP1A.210812.016; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/115.0.5790.138 Mobile Safari/537.36 OPR/72.0.2254.67482
• Mozilla/5.0 (Linux; Android 13; PHP110 Build/SP1A.210812.016; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/115.0.5790.138 Mobile Safari/537.36
• Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0

 

[Nirjonadda]

@Sim Server error log:

ErrorException: No data returned from BotFetcher src/XF/Error.php:77

Generated by: Unknown account Aug 24, 2023 at 6:15 PM

Stack trace

#0 src/XF.php(219): XF\Error->logError('No data returne...', false)
#1 src/addons/Hampel/KnownBots/SubContainer/Api.php(46): XF::logError('No data returne...')
#2 src/addons/Hampel/KnownBots/Cron/FetchBots.php(19): Hampel\KnownBots\SubContainer\Api->fetchBots()
#3 src/XF/Job/Cron.php(37): Hampel\KnownBots\Cron\FetchBots::fetchBots(Object(XF\Entity\CronEntry))
#4 src/XF/Job/Manager.php(260): XF\Job\Cron->run(8)
#5 src/XF/Job/Manager.php(202): XF\Job\Manager->runJobInternal(Array, 8)
#6 src/XF/Job/Manager.php(86): XF\Job\Manager->runJobEntry(Array, 8)
#7 job.php(43): XF\Job\Manager->runQueue(false, 8)
#8 {main}

Request state

array(4) {
  ["url"] => string(8) "/job.php"
  ["referrer"] => string(37) "/threads/85073/"
  ["_GET"] => array(0) {
  }
  ["_POST"] => array(0) {
  }
}

 

[Sim]

@Sim Server error log:

ErrorException: No data returned from BotFetcher src/XF/Error.php:77

Generated by: Unknown account Aug 24, 2023 at 6:15 PM

Stack trace

#0 src/XF.php(219): XF\Error->logError('No data returne...', false)
#1 src/addons/Hampel/KnownBots/SubContainer/Api.php(46): XF::logError('No data returne...')
#2 src/addons/Hampel/KnownBots/Cron/FetchBots.php(19): Hampel\KnownBots\SubContainer\Api->fetchBots()
#3 src/XF/Job/Cron.php(37): Hampel\KnownBots\Cron\FetchBots::fetchBots(Object(XF\Entity\CronEntry))
#4 src/XF/Job/Manager.php(260): XF\Job\Cron->run(8)
#5 src/XF/Job/Manager.php(202): XF\Job\Manager->runJobInternal(Array, 8)
#6 src/XF/Job/Manager.php(86): XF\Job\Manager->runJobEntry(Array, 8)
#7 job.php(43): XF\Job\Manager->runQueue(false, 8)
#8 {main}

Request state

array(4) {
  ["url"] => string(8) "/job.php"
  ["referrer"] => string(37) "/threads/85073/"
  ["_GET"] => array(0) {
  }
  ["_POST"] => array(0) {
  }
}

Is that the only error in the logs? Usually that occurs when there is an error with the API call and I would expect there to be another error logged with more information.

It could have been a temporary server issue with the API server - I have been doing a lot of work on it in preparation to release v5 of the KnownBots addon - currently in beta testing on my sites.

 

[gerryvz]

I received that same error, as well, last Thursday.

There was another error, a cURL error, but I don't know if it was related or not.

 

[Kleazy]

After the upgrade....Now what used to be 1000bots became 1000guests.

 

[lazy llama]

After the upgrade....Now what used to be 1000bots became 1000guests.

Will probably sort itself when the cron job runs for the first to pull the bot list, or you can trigger the job manually.

 

[Kleazy]

Will probably sort itself when the cron job runs for the first to pull the bot list, or you can trigger the job manually.

Yes... running the cron did the trick.

 

[Debi]

Many thanks for the updated version 4.0.0

 

[Sim]

I received that same error, as well, last Thursday.

There was another error, a cURL error, but I don't know if it was related or not.

The curl error would most likely have been related to the first error and would have indicated an error connecting to the API server, or an error received while waiting for a response - ie a temporary problem at my end.

I'm working on another new version (v5!) - which is much better at differentiating between valid browsers and potential bots - I'll revisit the error messages caused by API outages and try to make sure they are as helpful as possible.

 

[3rdlaw]

Thank You for this. Just installed today, will take a look tomorrow to see if I catch any!

 

[benFF]

@Sim

The bot list update today took out my server (mine updated an hour ago).

• TypeError: substr(): Argument #1 ($string) must be of type string, array given
• src\XF\Repository\SessionActivity.php:216

(Which relates to $robotKey)

 

[wolfgangm]

Had the same problem, server ran into a problem. Needed urgently to disable Add-on.

Wolfgang

 

[Sim]

I'm really sorry guys - I haven't seen anything like this on my own servers in testing (I have it running on 6 sites).

Can you give me more details - is there a full strack trace in the error logs or anything?

 

[Sim]

Don't worry - I found it, some square brackets crept in to my code, so the output ended up being an array within an array.

I did have tests in place in the addon to check whether the information retrieved from the API was valid - but it literally just checked whether it was an array, which it was, so an array within an array passed my tests

Anyway, I've fixed it and pushed a new version of the API to production - when you're ready, please try reinstalling and test whether it now works.

I'll revisit the safety checks to try and avoid a situation like this again.

 

[wolfgangm]

When enabled:

 

[wolfgangm]

Now disenabled: