Known Bots 6.0.3

[Sim]

FYI - this has already been fixed by the dev team and will most likely be included in the next release of XenForo (probably v2.1.6) - see the following thread for a temporary workaround until then: https://xenforo.com/community/threads/rebuild-reaction-score-server-error.171989/

 

[Sim]

Sim updated Known Bots with a new update entry:

v2.3.0

• added curl as a robot
• show robots online in Members online and Online statistics widgets

Read the rest of this update entry...

 

[Sim]

@Sim Please can you add option for support show Robots Count in members online on homepage?

Example in Xenforo 1.5 but Xenforo 2 have been removed this functionality.

View attachment 211323

Ping @Nirjonadda - v2.3.0 of this addon now shows the current robot count in the Members online and Online statistics widgets:





... thanks for the suggestions.

 

[Wutime]

I noticed the <a href=""> links on the bots doesn't have rel="nofollow"

I can do it myself, but thought maybe it should be added to the addon in a future release?

 

[Sim]

I noticed the <a href=""> links on the bots doesn't have rel="nofollow"

I can do it myself, but thought maybe it should be added to the addon in a future release?

It's actually the core that displays the bots and the links, this addon just supplies additional bot/url definitions.

I've logged a bug report, since they are public they really should be nofollow - https://xenforo.com/community/threads/robots-list-urls-are-not-nofollow.172497/

 

[Wutime]

It's actually the core that displays the bots and the links, this addon just supplies additional bot/url definitions.

I've logged a bug report, since they are public they really should be nofollow - https://xenforo.com/community/threads/robots-list-urls-are-not-nofollow.172497/

I just looked and noticed the same as you - thanks for spearheading it with XF staff. Cheers!

 

[Sim]

Sim updated Known Bots with a new update entry:

v2.4.0 - new bots

New bots:

• 7Siters
• AdScanner
• BrandVerity
• DuckDuckBot
• e.ventures Investment Crawler
• Exabot
• HTTrack
• Just-Crawling
• Pinterestbot
• Re-re Studio
• SemanticScholarBot
• Symfony BrowserKit
• TraceMyFile
• Ubermetrics
• Voyager Bot
• Linux Wget

Read the rest of this update entry...

 

[Xon]

@Sim any chance you can add a flag to disable reporting of bots in the forum stats?

 

[Sim]

@Sim any chance you can add a flag to disable reporting of bots in the forum stats?

Do you mean the change I made in v2.3.0 above where it adds the robot count to the stats widget? I can add a switch to disable that.

 

[Xon]

Do you mean the change I made in v2.3.0 above where it adds the robot count to the stats widget? I can add a switch to disable that.

Yup, a global option or a style property (both are about the same complexity to implement).

I'm aware you can just disable the template modifications but users (ie one of my clients) aren't used to diving into that part of the admincp to disable that sort of feature.

 

[Sim]

Sim updated Known Bots with a new update entry:

v2.5.0

Added an option to disable the display of robot statistics in sidebar widgets (which were added in v2.3.0 of this addon)

Read the rest of this update entry...

 

[Sim]

Yup, a global option or a style property

Done

 

[Xon]

Done

Works great

 

[Nirjonadda]

Total: 15 (members: 1, guests: 12, robots: 2)

Yes, I also like this show in one line.

 

[djbaxter]

It's still missing a lot of poorly behaved Chinese bots like this one:

114.119.162.166 (ecs-114-119-162-166.compute.hwclouds-dns.com)

There are a whole list of these in the114.119.* range and the 159.138.* ramge, among others.

 

[lazy llama]

https://github.com/jfqd/robots.txt has a good additional list, as does https://www.robotstxt.org.

 

[djbaxter]

How would you add bots from a list like that?

 

[Sim]

It's still missing a lot of poorly behaved Chinese bots like this one:

114.119.162.166 (ecs-114-119-162-166.compute.hwclouds-dns.com)

There are a whole list of these in the114.119.* range and the 159.138.* ramge, among others.

What user agent do those bots use?

To be clear: this addon does not and cannot identify robots based on IP address - it only ever considers the user agent. I won't be changing that functionality in this addon.

If you have bots which are masking their user agent, then you should just block them at the web server level - I have a simple deny list in nginx which I add bad IP addresses to.

 

[Max Taxable]

https://github.com/jfqd/robots.txt has a good additional list, as does https://www.robotstxt.org.

Useful ONLY for "noble" crawlers. Completely useless for any crawler that ignores it. robots.txt doesn't in any way block or ban anything.

 

[lazy llama]

Useful ONLY for "noble" crawlers. Completely useless for any crawler that ignores it. robots.txt doesn't in any way block or ban anything.

The lists can be used to block by user agent at the web server. Those two places are just good sources of user agent info.