Known Bots 6.1.0

[wolfgangm]

Server says that:

 

[Sim]

Uninstall the addon and reinstall it - don't just disable it. The knownbots.json file is invalid and must be replaced - the version supplied with the addon will be valid.

You could also enable it and then force an update - php cmd.php known-bots:fetch -f

 

[benFF]

Uninstall the addon and reinstall it - don't just disable it. The knownbots.json file is invalid and must be replaced - the version supplied with the addon will be valid.

You could also enable it and then force an update - php cmd.php known-bots:fetch -f

This sorted it, thank you.

 

[wolfgangm]

Reinstalled it and works fine again - Thank you!

Wolfgang

 

[Sim]

Sim updated Known Bots with a new update entry:

v4.0.1 - broken API mitigation

This release includes additional sanity checks to prevent bad data returned from the API from breaking the forums.

If any of the data returned by the API is not in the exact format we expect, the entire download is discarded and no changes applied to the forum. An error message will be logged prompting further investigation.

After upgrading to 4.0.1, you should manually force a fetch of new API data by executing the following command from your forum root:

php cmd.php...

Read the rest of this update entry...

 

[Sim]

I've released 4.0.1 to help prevent these issues of site breakage from occurring again - it includes much more rigorous sanity checks on the data returned by the API - if any of the data does not exactly match what we expect, the entire download is discarded and an error logged.

I've also made significant improvements to my unit test code coverage for the application that serves the API, to flag unexpected data being returned from the API.

 

[azzurro]

Thanks for the update, I tried the manual fetch and got this error:

# php cmd.php known-bots:fetch -f

An exception occurred: [ErrorException] [E_WARNING] Undefined array key "HTTP_CF_CONNECTING_IP" in src/config.php on line 19
#0 src/config.php(19): XF::handlePhpError(2, '[E_WARNING] Und...', '/var/www/client...', 19)
#1 src/XF/App.php(168): require('/var/www/client...')
#2 src/XF/Container.php(31): XF\App->XF\{closure}(Object(XF\Container))
#3 src/XF/App.php(3458): XF\Container->offsetGet('config')
#4 src/XF/App.php(2003): XF\App->container('config')
#5 src/XF/Cli/App.php(25): XF\App->setup()
#6 src/XF.php(497): XF\Cli\App->setup(Array)
#7 src/XF/Cli/Runner.php(54): XF::setupApp('XF\\Cli\\App')
#8 cmd.php(15): XF\Cli\Runner->run()
#9 {main}

Probably unique to my case, I have this line in the config for a previous fix for CloudFlare.
$_SERVER['REMOTE_ADDR'] = $_SERVER["HTTP_CF_CONNECTING_IP"];

 

[Sim]

Thanks for the update, I tried the manual fetch and got this error:



Probably unique to my case, I have this line in the config for a previous fix for CloudFlare.
$_SERVER['REMOTE_ADDR'] = $_SERVER["HTTP_CF_CONNECTING_IP"];

Yes, when running from CLI, there are no $_SERVER values - so you shouldn't assume that this array key exists.

If you are running PHP v7.0 or higher, try changing that line to the following:

$_SERVER['REMOTE_ADDR'] = $_SERVER["HTTP_CF_CONNECTING_IP"] ?? '';

If $_SERVER["HTTP_CF_CONNECTING_IP"] isn't defined, then it will simply set the variable to an empty string. None of these variables exist in a CLI environment.

 

[azzurro]

Yes, when running from CLI, there are no $_SERVER values - so you shouldn't assume that this array key exists.

If you are running PHP v7.0 or higher, try changing that line to the following:

$_SERVER['REMOTE_ADDR'] = $_SERVER["HTTP_CF_CONNECTING_IP"] ?? '';

If $_SERVER["HTTP_CF_CONNECTING_IP"] isn't defined, then it will simply set the variable to an empty string. None of these variables exist in a CLI environment.

Thanks @Sim, that worked, I was able to fetch the update.

 

[Sim]

Sim updated Known Bots with a new update entry:

v5.0.0 major rewrite

v5.0.0 is a major rewrite of the core functionality of this addon aimed at improving processing speed, bot detection sophistication and greatly enhancing our ability to identify new bots.

Note that the options have changed - so please check the options after upgrading. More information about each option is provided on the main addon page.

• major rewrite - no longer use "bot|spider|crawl" search strings and false-positive lists to identify possible bots, rely instead on search...

Read the rest of this update entry...

 

[Nirjonadda]

@Sim Please add support show Total Bots in Known Bots List.

 

[Nirjonadda]

@Sim Does Googlebot changed name is Generic Bot? Also Generic Bot name are not clickable or link missing.

 

[Sim]

@Sim Does Googlebot changed name is Generic Bot? Also Generic Bot name are not clickable or link missing.

View attachment 291306

Generic bot means that the user agent string wasn't definitive enough for us to determine whether it was a genuine bot or not - there are a lot of Googlebot "clones" out there and bots which pretend to be Googlebot so they are allowed through firewalls and such.

I use the catch-all term "generic bot" to mean "it's definitely a bot, but I can't really tell which one" - which is also why there's no link - because I don't know what bot it is.

I'd need to see the actual user agent string to work out what has happened here - if it's genuinely a Googlebot, it should be identified as such - so I've probably got the match strings wrong there.

 

[Nirjonadda]

Generic bot means that the user agent string wasn't definitive enough for us to determine whether it was a genuine bot or not - there are a lot of Googlebot "clones" out there and bots which pretend to be Googlebot so they are allowed through firewalls and such.

I use the catch-all term "generic bot" to mean "it's definitely a bot, but I can't really tell which one" - which is also why there's no link - because I don't know what bot it is.

I'd need to see the actual user agent string to work out what has happened here - if it's genuinely a Googlebot, it should be identified as such - so I've probably got the match strings wrong there.

Our all Googlebot are showing Generic bot but before all are Googlebot. Now we haven't anymore Googlebot.

 

[Chromaniac]

hah. i can confirm. i do not see googlebot in robots page. running Known Bots: Fetch New Bots from API didn't fix it!

 

[benFF]

Same here

 

[Sim]

@Nirjonadda @Chromaniac @benFF

I found the culprit - there was a search string matching the google.com/bot string in the URL that valid Google user agents have. I've fixed that now, so if you fetch new bots, you should get the updated definitions and Googlebot should start appearing in the bot list again.

 

[Sim]

Sim updated Known Bots with a new update entry:

v6 major rewrite

v6 is another major rewrite of the core functionality of the addon, aimed at improving the submission process for newly detected useragents and performance improvements.

Important for v4 users: with this release, I am deprecating the v1 API - addon versions v4.x and earlier will continue to function for a while but will then start returning 404 error codes once I turn off the v1 API. Anyone still running KnownBots v4.x should upgrade as soon as possible.

Important for v5...

Read the rest of this update entry...

 

[Sim]

In case anyone doesn't read the update entry, I'll re-post some important information here:

Important for v4 users: with this release, I am deprecating the v1 API - addon versions v4.x and earlier will continue to function for a while but will then start returning 404 error codes once I turn off the v1 API. Anyone still running KnownBots v4.x should upgrade as soon as possible.

Important for v5 users: The v2 API used in addon v5.x for fetching new bots will remain operational, however, I am deprecating the email based submission system in favour of a new API based user agent submission system. After a transition period, the inbound email system will be disabled and any emails sent to the knownbots@hampel.io address will bounce back as undelieverable. Anyone still running KnownBots v5.x should either upgrade, or at least disable the "Email user agents" option in the v5.x addon options.

Important for anyone upgrading to v6: the new submission system in v6 uses an authentication process to ensure only valid submissions occur. After upgrading to v6, to continue submitting new user agents for analysis, you must first configure the authentication system - it is a very simple process - see instructions on the addon page. The options for v6 have changed - you should check them after upgrading.

 

[Sim]

If any XenForo cloud customers have attempted to use their XenForo license validation token to configure the KnownBots API and received an error response, please try again now - I have made a change to the backend system to accomodate cloud validation tokens which are longer than non-cloud tokens.

Note that regular validation tokens are 32 characters long, while cloud validation tokens are 35 characters long and start with cl_

Here is where to find your customer validation token:



... click on that link and you'll get a screen similar to the following:



The validation token is the 32 (or 35) character string similar to what I've highlighted above. This is what you'll need to enter into the addon options for validation.

If you regenerate your token, validation will fail once your 28 day API token expires, at which point if you wish to continue using the KnownBots API, you should change your addon options to use the newly generated token.