Known Bots

Known Bots 6.0.3

No permission to download
Attached, if you happen to add more bots to this list or find anything wrong with it I'd appreciate an update :)
Thank you so much for sharing this Fillip, I've attached a basic 'converted' file for @Sim's addon which I created using some regex replacements..

I'd previously done a little work looking into dead and/or updated links but had managed to corrupt the file without realising it and all guests were being identified as a Robot. So over the last 24 hours I've opted for a faster alternative just to implement what you have shared so far. I've just uploaded it to my live site and so far, so good..

Maybe if I have a bit of time I'll eventually go back to it.. :)
 

Attachments

Last edited:
@Sim maybe update this with the new robot definitions added here in the discussion?

May also want to put this image in the "overview" so people can visualize what it does?
1528816181008-png.177730
 
@Sim Please can you add option for support show Robots Count in members online on homepage?

Example in Xenforo 1.5 but Xenforo 2 have been removed this functionality.

ScreenShot01230.webp
 
One more issue, Still are not showing some Robots name.

The robots detection relies on being able to parse the UserAgent string to detect robots. If the bot uses an unknown (or fake) UserAgent, then we won't be able to detect them.

I'd be interested to know what UserAgent those bots in your screenshot are using!
 
The robots detection relies on being able to parse the UserAgent string to detect robots. If the bot uses an unknown (or fake) UserAgent, then we won't be able to detect them.

I'd be interested to know what UserAgent those bots in your screenshot are using!


Also googleusercontent.com bots are not showing Robots name and
64.233.172.244 (google-proxy-64-233-172-244.google.com) not showing Robots name.

ScreenShot01233.png
 
Last edited:
Also googleusercontent.com bots are not showing Robots name and
64.233.172.244 (google-proxy-64-233-172-244.google.com) not showing Robots name.

Again, to be clear - we do not look at IP address or domain name for visitors to the site - only UserAgent.

Perhaps look in your web server logs for entries matching that IP address and identify the UserAgent - let me know what it is so I can evaluate why it's not showing up.
 
@Sim Lot of Known Bots List URL are not working because incorrect URL link with our site link.

Example: https://mysite.com/+http://affiliate-program.amazon.com and https://mysite.com/AlexaToolbar/alxf-2.19 and https://mysite.com/admin.php?tools/known-bots

ScreenShot01235.png
 
@Sim Lot of Known Bots List URL are not working because incorrect URL link with our site link.

Example: https://mysite.com/+http://affiliate-program.amazon.com and https://mysite.com/AlexaToolbar/alxf-2.19 and https://mysite.com/admin.php?tools/known-bots

View attachment 211432

Do you have another addon installed which adds bots to the list? The bots you've listed there are not ones that my addon adds.
 
Do you have another addon installed which adds bots to the list? The bots you've listed there are not ones that my addon adds.

Yes this are added from @DragonByte Tech Security add-on with enabling Extended spider identification option. Does not know how to adding Security add-on bots in Show list of Known Bots tools with incorrect URL link.

ScreenShot01236.webp
 
I suggest you post in that other addon's support thread and request that they fix the problems with their addon if any - I have no control over the data they insert into the list and if they're doing it wrong - that's not something I can fix.
 
I rebuilt the built-in search engine index on XF 2.1.5 and ran into a number of server errors that seem to have been generated by this addon.

Check out line #7

Code:
ErrorException: [E_NOTICE] Undefined index: steps src/XF/Job/AbstractRebuildJob.php:28
Generated by: xxxxxxxx Nov 12, 2019 at 9:43 PM
Stack trace
#0 src/XF/Job/AbstractRebuildJob.php(28): XF::handlePhpError(8, '[E_NOTICE] Unde...', '/home/nginx/dom...', 28, Array)
#1 src/XF/Job/Manager.php(253): XF\Job\AbstractRebuildJob->run(G)
#2 src/XF/Job/Manager.php(195): XF\Job\Manager->runJobInternal(Array, G)
#3 src/XF/Job/Manager.php(111): XF\Job\Manager->runJobEntry(Array, G)
#4 src/XF/Admin/Controller/Tools.php(120): XF\Job\Manager->runByIds(Array, 8)
#5 src/XF/Mvc/Dispatcher.php(350): XF\Admin\Controller\Tools->actionRunJob(Object(XF\Mvc\ParameterBag))
#6 src/XF/Mvc/Dispatcher.php(257): XF\Mvc\Dispatcher->dispatchClass('XF:Tools', 'RunJob', Object(XF\Mvc\RouteMatch), Object(Hampel\KnownBots\XF\Admin\Controller\Tools), NULL)
#7 src/XF/Mvc/Dispatcher.php(113): XF\Mvc\Dispatcher->dispatchFromMatch(Object(XF\Mvc\RouteMatch), Object(Hampel\KnownBots\XF\Admin\Controller\Tools), NULL)
#8 src/XF/Mvc/Dispatcher.php(55): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#9 src/XF/App.php(2184): XF\Mvc\Dispatcher->run()
#10 src/XF.php(391): XF\App->run()
#11 admin.php(13): XF::runApp('XF\\Admin\\App')
#12 {main}
Request state
array(4) {
  ["url"] => string(24) "/admin.php?tools/run-job"
  ["referrer"] => string(62) "https://testing.xxxxxxxxxxxxxxxxxxx.com/admin.php?reactions/angry.6/edit"
  ["_GET"] => array(1) {
    ["tools/run-job"] => string(0) ""
  }
  ["_POST"] => array(5) {
    ["only_id"] => string(3) "378"
    ["_xfRequestUri"] => string(33) "/admin.php?reactions/angry.6/edit"
    ["_xfWithData"] => string(1) "1"
    ["_xfToken"] => string(8) "********"
    ["_xfResponseType"] => string(4) "json"
  }
}
Delete
 
I rebuilt the built-in search engine index on XF 2.1.5 and ran into a number of server errors that seem to have been generated by this addon.

Check out line #7

Code:
ErrorException: [E_NOTICE] Undefined index: steps src/XF/Job/AbstractRebuildJob.php:28
Generated by: xxxxxxxx Nov 12, 2019 at 9:43 PM
Stack trace
#0 src/XF/Job/AbstractRebuildJob.php(28): XF::handlePhpError(8, '[E_NOTICE] Unde...', '/home/nginx/dom...', 28, Array)
#1 src/XF/Job/Manager.php(253): XF\Job\AbstractRebuildJob->run(G)
#2 src/XF/Job/Manager.php(195): XF\Job\Manager->runJobInternal(Array, G)
#3 src/XF/Job/Manager.php(111): XF\Job\Manager->runJobEntry(Array, G)
#4 src/XF/Admin/Controller/Tools.php(120): XF\Job\Manager->runByIds(Array, 8)
#5 src/XF/Mvc/Dispatcher.php(350): XF\Admin\Controller\Tools->actionRunJob(Object(XF\Mvc\ParameterBag))
#6 src/XF/Mvc/Dispatcher.php(257): XF\Mvc\Dispatcher->dispatchClass('XF:Tools', 'RunJob', Object(XF\Mvc\RouteMatch), Object(Hampel\KnownBots\XF\Admin\Controller\Tools), NULL)
#7 src/XF/Mvc/Dispatcher.php(113): XF\Mvc\Dispatcher->dispatchFromMatch(Object(XF\Mvc\RouteMatch), Object(Hampel\KnownBots\XF\Admin\Controller\Tools), NULL)
#8 src/XF/Mvc/Dispatcher.php(55): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#9 src/XF/App.php(2184): XF\Mvc\Dispatcher->run()
#10 src/XF.php(391): XF\App->run()
#11 admin.php(13): XF::runApp('XF\\Admin\\App')
#12 {main}
Request state
array(4) {
  ["url"] => string(24) "/admin.php?tools/run-job"
  ["referrer"] => string(62) "https://testing.xxxxxxxxxxxxxxxxxxx.com/admin.php?reactions/angry.6/edit"
  ["_GET"] => array(1) {
    ["tools/run-job"] => string(0) ""
  }
  ["_POST"] => array(5) {
    ["only_id"] => string(3) "378"
    ["_xfRequestUri"] => string(33) "/admin.php?reactions/angry.6/edit"
    ["_xfWithData"] => string(1) "1"
    ["_xfToken"] => string(8) "********"
    ["_xfResponseType"] => string(4) "json"
  }
}
Delete

Okay - I have been able to reproduce this on my dev server and it seems to be unrelated to my addon (same error occurs with my addon disabled) - I think it is an XF core bug.

Will try and locate the source of the error and report the bug if I can track it down.
 
Back
Top Bottom