Known Bots 6.3.0

This XenForo 2.x addon adds additional definitions for bot detection in sessions.

This allows more accurate indication of the number of users vs robots online at any given time.

New bot definitions are pulled automatically via API and potential new bots detected are sent back to the addon author for analysis for inclusion in the bot definitions.

The definitions shipped with version 6.3.0 identify 1,890 bots - 1,637 of them by a fast plain-text match and a further 67 by regular expression - and carry 623 patterns for recognising a genuine browser.

Statistics from the KnownBots API server management system, custom built to track and identify bots and serve the updates API:

Requirements​

This addon requires XenForo 2.2.0 or higher, and PHP 7.4 or higher.

XenForo 2.2 itself runs on PHP 7.0, so a XenForo 2.2 forum on an older PHP cannot install this version - 6.2.0 is the last release that runs there. PHP 7.3 has been end of life since December 2021, so upgrading PHP is worth doing on its own account.

Usage​

When you look at Current Visitors, you'll see additional robots identified - also look at the "Robots" list on that page http://www.example.com/community/online/?type=robot



We also add the current robot count to the Members online widget and the Online statistics widget (from the current visitor page). This can be disabled via the options.





There is a tool in the admin area for displaying a complete list of currently known bots:



... and another tool which lets you test a user agent string to see if it would be detected as a bot or not:



... and a third tool which shows a list of the last 100 bots detected, along with any unidentified user agents waiting to be sent back for analysis.

How detection works​

A user agent is checked first against the known bot definitions with a fast plain-text match, then against a set of regular expressions for bot strings too complex for a plain match. Logged-in members are assumed to be human once those two checks have run.

For guests, if neither check finds a bot, the user agent is checked against a list of valid browser patterns. With "Store user agents in database" enabled, a user agent not recognised as a real browser is stored for further analysis, and with "Send user agents via API" enabled it is sent to the addon author, so that new bots can be identified - including bots that do not identify themselves with the traditional keywords such as bot, crawl or spider.

New in version 6.3​

Two new command line tools let you check the addon without clicking through the admin control panel, and report things no single screen shows: known-bots:config prints every setting and where it came from, and known-bots:validate proves that detection actually works on your forum. Both are documented under Command Line Interface below. known-bots:validate is built for a monitoring system: it returns a meaningful exit code, and --unattended keeps it from making the one check that reaches the API.

Entries this addon writes to the XenForo server error log now begin with the addon's name, so they can be told apart from every other addon's at a glance.

This version also raises the PHP requirement to 7.4 - see Requirements above.

New in Version 6​

Version 6 was a complete rewrite of previous versions - with bot detection up to 15x faster than the previous method. Bot detection is also more sophisticated, with a secondary regex-based detection system to help identify complex bot strings that can't be matched using the simple text string matches used by default.

If no bots are detected, the addon checks the user agent against a list of valid browser regex strings - and if it is not considered to be a valid user-driven browser, it stores the information in the database temporarily before sending it back to the addon author for further analysis.

Previously, the system only sent back user agents which matched the key words bot|crawl|spider - this detection method is significantly more comprehensive and allows much greater accuracy for detecting new bots that may not identify themselves using the traditional keywords.

Version 6 also moved the sending of new user agents from email to the API, with authentication facilitated by the XenForo license validation system and very easy to configure. Nothing is sent to the addon author by email any more.

Options​

Show robot statistics​

Enable to show robot statistics in the sidebar widgets.

Fetch new bots​

Enable to automatically fetch new bot identifiers from the KnownBots API maintained by the addon author. If you disable this option, you must set up your own system to update the known bots definitions - see the Command Line Interface section for options to assist here.

Store user agents in database and purge after​

Enable to store unknown and bot user agents in the database for further analysis. This should be used in conjunction with the "Send user agents via API" option, which sends unknown user agents back to the KnownBots management system for further analysis and identification of new bots. You may also manually send user agent information to the addon author via the addon discussion thread - use the "List detected bots" tool to show unknown user agents.

With this option enabled, you may also choose how long to retain user agent records before they are automatically purged from the database. This uses a "last seen" mechanism to maintain the list of recently seen bots - only bots not recently seen will be purged. For busy sites, consider lowering this value from the default 90 days, to reduce the size of data stored. Set days to zero to never purge user agent data (not recommended).

Send user agents via API​

Enabling this option will send unidentified user agents back to the addon author for further analysis and identification of new bots.

To configure the API, enter the License validation token for your site, found in the XenForo customer interface. The validation token will be sent to the XenForo customer validation API and if valid, a KnownBots API token will be generated and returned back for subsequent authentication purposes.

With a validated license, the authentication process is automatic. API tokens are regenerated every 28 days and are re-authenticated automatically. Customer details are automatically purged from the KnownBots database after 30 days of inactivity (see privacy details below). Regenerating your license validation token will automatically cause API revalidation to fail and customer details to be purged - unless you re-configure the addon options with the new license validation token.

Email user agents​

This option no longer sends anything to the addon author - that moved to the API in version 6. It remains available so that you can have the stored user agents emailed to an address of your choosing, for your own monitoring of newly detected user agents.

Separate several addresses with commas and each one gets its own email; an entry that is not a valid email address is skipped and reported in the error log. Leave the address blank to use the board's contact email address. This option only has effect if the "Store user agents in database" option is also enabled.

Logging​

Install the Monolog Logging Service addon to log what this addon does. Without it, nothing is logged.

At the info level, which is what a forum would normally run, each unit of work is recorded with its counts: bots fetched from the API, user agents sent to the API, emailed or marked as sent, and old user agents purged. A failure is recorded at error, and an API that is briefly unavailable at warning, so a temporary outage is not reported as a fault.

At the debug level this addon records an entry for every user agent it stores, including the user agent itself. Two things follow from that. On a busy forum it is one entry for every page view by a visitor whose browser is not recognised, which is a large number. And a user agent is information about a visitor, as are the IP address and URL that the logging addon adds to every entry - so debug is for answering a particular question about detection, rather than a level to leave switched on.

No entry at any level contains a credential: neither the License validation token you enter nor the KnownBots API token issued back to your site appears in the log.

Command Line Interface​

There are CLI tools provided which may be helpful for certain use-cases.

Show the configuration​

(New in 6.3) Print every setting this addon uses, with where each one comes from - the options page or config.php - and the state that is not on any one screen in the admin control panel: the bot definitions on the filesystem, the code cache detection reads, the stored user agent counts and the three cron entries. It reads and prints only; it changes nothing and sends nothing.


Settings that hold a credential are reported as set or not set, never printed, so the output is safe to paste into a support thread.

Validate​

(New in 6.3) Check that bot detection actually works on this forum: the definitions are present and current, the code cache was built from them, a known bot is recognised and a real browser is not, the cron entries that keep all of it up to date exist and are active, the email addresses are valid, and the API token is accepted.


Each check reports one of four outcomes. [ ok ] was proved. [warn] works now and will bite later. [fail] cannot do what it is configured to. A blank [ ] did not apply - which is never the same as a pass.

The API token check reaches the KnownBots API with your forum's token, because only the API's answer proves that the token works. Use the --unattended option to skip it, for a deployment or cron gate with nobody watching; every other check still runs. It never revalidates: writing a new token into your options is a change, and this command does not make changes. Use known-bots:check-token --revalidate for that.

The exit code is the part a script reads:

• all checks fine: 0
• warnings only: 0, or 2 with --strict
• any check failed: 1, with --strict and without it

A warning exits 0 by default, so known-bots:validate && next-step carries on past one. Use --strict for a monitor that should hear about a warning. Note that this is not the monitoring plugin convention, where 1 means warning and 2 means critical: here 1 always means a check failed.

Fetch Bots​

Fetch new bots via the API. Use this to run your own cron - does the same task that the supplied XenForo Cron job does.


Use the -f option to force updates, bypassing the "last updated" flag and re-fetching the latest bot definitions.

If using this command via your own cron - be sure to disable the supplied XenForo Cron job titled "Known Bots: Fetch New Bots from API".

Load Bots​

Load bots data from the knownbots.json file already on the filesystem.


The addon looks for the file internal_data/knownbots.json - if you download it manually and place the updated version in this location, you may then execute the above command to load the data without calling the API.

Reprocess User Agents​

Cycle through user agents stored in the database and update the definitions based on the latest bot detection data.


By default, only "unknown" user agents are reprocessed. Use the -a option to reprocess all known and unknown user agents - useful if a bot was misidentified but has been updated in the latest API data.

Test Bots​

Test the supplied user agent string and identify whether it is detected as a bot.


Don't forget to quote user agent strings that contain spaces. For example:


Use the -s option to save your user agent strings to the database for further processing.

Check API Token​

Test if the KnownBots API token is valid.


Use the -r or --revalidate option to automatically attempt to revalidate and generate a new API token if authentication fails.

Send Agents​

Send newly detected user agents to the KnownBots API. Does the same thing the cron job does - but can be run externally via a system cron or manually for testing purposes.

Email Agents​

Email the stored user agents that have not yet been sent to one or more addresses, separated by commas: those given on the command line (quote a list), or else the "Email user agents" option's, or else the board's contact email address. Storing user agents must be enabled. Unlike the cron job it does not mark them as sent, so it can be run repeatedly for testing.

Parse Log Files​

Reads web server log file information and lists all detected bots.


Use the -a or --agents option to simply list all user agents found in the supplied log file - no bot detection occurs.

Command will also accept data from stdin if you specify a hyphen as the file name.

Note that this tool assumes you are using standard "Apache" log file format, which is used by most web servers.

Usage examples:

List all user agents. Invalid (or no) data will be returned if you are not using standard Apache log file format.


List bots:


Examples of reading data from stdin:

Import User Agents​

Reads user agents from a text file or stdin, one per line, and stores them for processing.


Command will accept data from stdin if you specify a hyphen as the file name.

Privacy Statement​

Privacy Summary​

Depending on the options enabled, there is some information collected about sites using this addon by the addon author - but it is only ever used for anonymous analytics and for troubleshooting purposes and never disclosed to third parties, nor is it ever used for marketing or any purposes other than for the operation of this addon.

Options are provided to disable or bypass certain functionality if you remain uncomfortable about using the systems as designed.

Details​

With the "Fetch new bots" option enabled, the addon will automatically send a request to the KnownBots API, downloading an updated list of bot definitions. You may query that API directly at any time to see what is contained in the data returned.

API calls by the addon are made using the standard "untrusted" HTTP client built into XenForo, which means they are forwarded through a proxy server if you have one configured. Standard web server log files on the API server will contain information about requests made, including the IP address of your server (or proxy if used), and your forum name as supplied in the user agent of the XenForo HTTP client. For example:


HTTP server log information is used solely for analytics and troubleshooting purposes and is never made available to third parties.

Enabling the "Send user agents via API" setting requires the submission of a XenForo license validation token, retrieved from your customer account on the XenForo website. This token is intended to be supplied to 3rd party addon authors for the purposes of validating XenForo licenses. In this case, it is being utilised as a simple authentication mechanism. You may regenerate your license validation token at any time via the XenForo customer interface, which will automatically cause authentication against the KnownBots API to fail once the existing API token expires.

API tokens expire every 28 days and are automatically regenerated, provided the XenForo license validation token remains valid. Customer details retrieved from the XenForo license validation API are automatically purged from the database after 30 days of inactivity.

When validating licenses, only the license validation token and forum URL are sent to the KnownBots API. Once validation is complete, only the generated API token and a list of new user agents are sent to the server. Web server logs similar to those described above are generated for all API calls.

Neither token is ever written to your forum's log, and no information is sent to the addon author by email.