[XenCustomize] Bot Shield: Content Protection, Crawler Management & Activity Monitoring 2.3.0

XenCustomize Bot Shield for XenForo 2​

Smart bot protection for XenForo — stop unwanted traffic before it reaches your content.

Bot Shield helps protect your XenForo community from unwanted bots, automated browsing, and abusive guest traffic. It evaluates incoming requests before protected content is delivered, helping reduce unnecessary server processing and resource consumption.

Legitimate visitors can complete an automatic browser verification without solving CAPTCHA puzzles. Once verified, they can continue browsing normally.

With early content protection, verified search engine crawlers, administrator-controlled crawler policies, guest rate limiting, and detailed activity monitoring, Bot Shield gives you practical control over how your community is protected.

Why We Built Bot Shield​

Like many XenForo communities, our own websites have experienced waves of unwanted bots and excessive automated traffic consuming valuable server resources.

We developed Bot Shield to address these problems directly, protecting public content while maintaining support for legitimate visitors, search engines, and administrator-approved crawlers.

Tested on Our Own Communities​

Bot Shield has been developed, refined, and tested on our own XenForo websites under real-world traffic conditions.

After months of active use, continuous improvements, and hands-on testing, Bot Shield has matured into a reliable solution that we also trust to deploy on our official XenCustomize support website.

Key Features​

Early Content Protection
Protects configured public content before XenForo generates or serves protected pages, including content delivered through the guest page cache.

Protection focuses on selected content areas rather than unnecessarily challenging every website request.

Automatic Browser Verification
Visitors complete a lightweight browser check automatically, without CAPTCHA puzzles or manual interaction.

Successful verification remains valid for a configurable period, helping visitors browse without repeated interruptions.

Smart Crawler Recognition
Recognizes crawlers through XenForo's built-in robot identification, including compatible extended crawler definitions.

Manage recognized crawlers individually or create custom rules for additional bots and automated services.

Official Search Engine & Google Service Verification
Bot Shield supports official IP verification for important search engines and Google services, including:

• Google Search: Googlebot, Googlebot Image, and Googlebot Video.
• Google search tools: Google Inspection Tool, GoogleOther and its image/video variants, and Google Storebot.
• Google advertising: Google AdSense / Ad Manager and Google Ads crawlers, including mobile variants.
• Google PageSpeed Insights: Chrome Lighthouse web page audit requests originating from Google's published Google-controlled fetcher networks.
• Bingbot: Official Microsoft Bing crawler verification.

Supported crawlers can be allowed automatically when their source IP matches the appropriate official verification ranges.

Crawler verification information is refreshed automatically, and administrators can review the availability of official verification sources.

Administrator-Controlled Crawler Policies
Choose how individual recognized and custom crawlers are handled using the dedicated Crawler management interface.

Available policies include:

• Default protection: Apply normal protection, including automatic official IP verification for supported crawlers.
• Require browser verification: Require matching crawlers to complete browser verification.
• Allow (User-Agent): Allow selected crawlers without official IP verification.
• Allow verified: Allow supported crawlers only after successful official IP verification.
• Block: Deny matching crawler requests.

Administrators can independently manage supported Google crawler types, including search, advertising, and auditing crawlers.

Restrictive Block and Require browser verification policies take precedence over matching allowances.

You decide which crawlers can access your community.

For example, you can permit Internet Archive, social media preview bots, SEO auditing tools, or other desired crawlers that cannot complete JavaScript verification.

User-Agent allowances are optional and can be impersonated, so they should be enabled only for crawlers you choose to trust.

Custom Crawler Rules
Create additional crawler rules using User-Agent matching when a crawler is not available in XenForo's recognized list.

Apply individual protection policies without modifying XenForo's built-in crawler definitions.

Guest Rate Limiting
Limit protected guest requests from individual IP addresses to help reduce excessive automated traffic.

Configure the permitted request rate according to your community's needs. Explicit User-Agent allowances remain subject to guest rate limiting.

Protected Content Management
Control protection for supported public content from XenForo, XenForo Resource Manager, and XenForo Media Gallery.

The dedicated Admin CP interface lets you:

• Enable or disable protection for individual built-in content controllers.
• Discover public content controllers from compatible installed add-ons.
• Select and manage multiple controllers at once.
• Find controllers using convenient search and filtering controls.
• Customize protection coverage without editing code.

Protection can also be extended to supported third-party content without requiring direct integration from the add-on developer.

Activity Dashboard & Monitoring
Monitor browser challenges, successful verifications, denied requests, rate limiting, and verified crawler activity through a dedicated Admin CP dashboard.

View protection statistics, activity charts, official crawler verification status, and storage information.

Detailed Activity History
Review recorded protection decisions with useful information, including:

• Date and time.
• Visitor IP address.
• Recognized or custom crawler identity, when available.
• Protection action and reason.
• Requested URL.
• Applied crawler policy.
• User-Agent and verification details.

Filter activity by crawler, action, IP address, and reporting period.

Click an IP address to filter its activity, or open a crawler's request-details overlay to inspect the available information. Requested URLs are clickable when valid.

Activity records use configurable retention and storage limits, with repeated decisions sampled to help control database growth.

Current Visitors Integration
Extends XenForo's existing Current visitors page with protection-status badges and intercepted visitor activity.

Authorized staff can identify browser challenges, successful verifications, denied requests, rate-limited visitors, officially verified crawlers, and User-Agent-allowed crawlers.

When available, intercepted activity also displays the content visitors attempted to access.

Automatic Maintenance
Scheduled maintenance refreshes official crawler verification information and removes expired activity and temporary records.

Configurable retention and storage limits help keep Bot Shield's stored data under control without requiring regular manual cleanup.

Administrator Options​

Bot Shield provides the following Admin CP options and management features:

• Enable Bot Shield: Turn protection on or off.
• Browser verification lifetime: Set how long successful verification remains valid.
• Guest requests per minute: Configure the per-IP request limit.
• Verification difficulty: Adjust browser-check difficulty.
• Record activity: Enable or disable protection activity logging.
• Record passed verifications: Optionally include successful verification activity.
• Activity retention: Set how long activity records are retained.
• Maximum activity records: Limit the amount of stored activity history.
• Crawler management: Configure recognized and custom crawler policies.
• Crawler feed management: Review verification source status and refresh official crawler information.
• Protected content: Customize protection for built-in and compatible third-party content.
• Activity monitoring: Review recorded decisions, filter activity, and inspect request details.

Management access is controlled by the Manage XenCustomize Bot Shield administrator permission.

Useful For​

• Reducing unwanted automated traffic on busy XenForo communities.
• Protecting public forums, threads, resources, media, and compatible third-party content.
• Supporting legitimate Google Search, Bing, AdSense, Google Ads, and PageSpeed crawlers.
• Allowing desired crawlers such as Internet Archive and SEO auditing tools.
• Blocking or challenging unwanted bots without restricting every crawler.
• Monitoring suspicious traffic, browser verification, and crawler activity.
• Managing protection and stored activity through native XenForo administrator controls.

Compatibility & Requirements​

• XenForo: 2.3.0 or newer in the XenForo 2.3 series.
• PHP: 8.0 or newer.
• Required add-on: XenCustomize Core Library 1.0.4 or newer.
• Optional integrations: XenForo Resource Manager and XenForo Media Gallery.
• Third-party add-ons: Configurable protection for compatible public content controllers.
• Browser verification: Requires HTTPS, JavaScript, and cookies.

Bot Shield focuses on supported public content while leaving native account operations, user upgrades, checkout, and payment callbacks to XenForo's existing handling.

Automated clients that cannot complete browser verification may require an administrator-configured crawler allowance. Official IP verification is available for supported crawler identities and networks.

License Setup​

After installation, open the add-on options and select the License options tab to enter your license key. Bot Shield includes a Remove footer branding option, allowing administrators to remove the XenCustomize branding at no additional cost.

License verification may communicate with XenCustomize.com when needed, with locally cached verification results. Normal installation and uninstallation do not depend on our server being available. An expired license does not disable normal use of the add-on, but an active valid license is required for official support and access to new downloads or updates released after the license expires.

Screenshots & Feature Documentation​

Explore our feature guides to learn how to configure Bot Shield, manage crawlers, understand browser verification, and monitor protection activity.

1. Getting Started with Bot Shield
2. Managing Protected Content
3. How Browser Verification Works
4. Managing Crawler Policies
5. Official Crawler IP Verification
6. Adding Custom Crawler Rules
7. Guest Rate Limiting and Verification Settings
8. Understanding Bot Shield Activity
9. Dashboard Statistics, Logging and Cleanup
10. Bot Shield on the Current Visitors Page