Known Bots 6.1.0
- Thread starter Sim
- Start date
smallwheels
Well-known member
@Sim Message in the error log out of the blue today:
Stacktrace:
For the record: I do autosubmit new bots to you via the API and have a token for that configured in the options. Has been working flawlessly since configured and the known bots addon has been working even much longer w/o issues.
To do what?
Stacktrace:
Code:
Stack trace
#0 src/addons/Hampel/KnownBots/SubContainer/Api.php(99): Hampel\KnownBots\Api\KnownBots->validate('547981bc319aa73...', 'https://myforumurl...')
#1 src/addons/Hampel/KnownBots/Service/UserAgentSender.php(64): Hampel\KnownBots\SubContainer\Api->validate('547981bc319aa73...')
#2 src/addons/Hampel/KnownBots/Cron/SendAgents.php(71): Hampel\KnownBots\Service\UserAgentSender->sendUserAgents()
#3 src/addons/Hampel/KnownBots/Cron/SendAgents.php(42): Hampel\KnownBots\Cron\SendAgents::sendApi(Array)
#4 src/XF/Job/Cron.php(37): Hampel\KnownBots\Cron\SendAgents::send(Object(XF\Entity\CronEntry))
#5 src/XF/Job/Manager.php(260): XF\Job\Cron->run(8)
#6 src/XF/Job/Manager.php(202): XF\Job\Manager->runJobInternal(Array, 8)
#7 src/XF/Job/Manager.php(86): XF\Job\Manager->runJobEntry(Array, 8)
#8 job.php(43): XF\Job\Manager->runQueue(false, 8)
#9 {main}For the record: I do autosubmit new bots to you via the API and have a token for that configured in the options. Has been working flawlessly since configured and the known bots addon has been working even much longer w/o issues.
To do what?
Sim
Well-known member
Have you regenerated your license validation token in your XenForo customer account by any chance? That would invalidate your tokens for KnownBots.
Either way, just reset your API token - go to Admin CP > Options > Known Bots and click on the "Change" button under API Token. Select "Update" and click "Continue" then copy and paste your License validation token from your XenForo customer account for that domain and Save - this should generate a new API token for KnownBots.
smallwheels
Well-known member
Thanks for the very quick response! Probably yes, but already weeks ago.
Oh, that's good to know. Have done that from time to time and never ran into issues with developers where an older one was stored. A bit unlucky if every time I recreate a license token I need a new API key for Known Bots. But now that I know it I'll sipmly try to avoid recreating the API key. Did not know that a new key would render an existing one useless.
Have done as you recommended and am awaiting the result in form of no error messages.
Sim
Well-known member
The flip side to that (and the reason I built it this way) is that you have more control over the information stored on our server - by regenerating your license token, the information we store on our KnownBots server becomes useless. We automatically purge any invalid tokens because there's no point us keeping them in the database.
It would be nice if that could be controlled in a more fine-grained manner so that you can selectively choose which license validation tokens to invalidate for specific addons rather than an all-or-nothing approach, but it's better than nothing.
smallwheels
Well-known member
Totatally up to your own taste. Many of the bots in the list have a link that gives a little background. It is up to you to decide if you want a certain bot on your website or not.
JoyFreak
Well-known member
Is there like a list somewhere that tells us what ones are good to keep and block perhaps? Will make things easier when reviewing the bots.
ENF
Well-known member
Rule of thumb: most bots aren’t terrible, but there are those will consume your resources. And, there are varying opinions on what’s bad and what’s good. Some bots don’t respect robots.txt and those get a big fat ASN block at the border.
It’s really comes down to your own preference.
smallwheels
Well-known member
As said before: Up to your own taste. Basically the ONLY bot that is really relevant is the Google Indexing bot (not the other Google bots). Probably one would accept indexing bots from other search engines as well (Apple, Bing, various smaller search engines), but again not their other bots that i.e. feed their AI models.
What one probably will tolerate are bots that create link previews (those come from various social networks mainly).
All the other bots are on a scale of mostly harmless, useless, annoying, resource consuming, painful, unwanted or even harmful (content scrapers are the latter i.e.).
But: It is really up to your own taste.
Is use this one. Those 3 seperate is because else they don't listen.
Code:
User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
User-agent: YandexBot
User-agent: omgilibot
User-agent: anthropic-ai
User-agent: meta-externalagent
User-agent: Amazonbot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: Barkrowler
Disallow: /ENF
Well-known member
This one from ByteDance, earned itself a full network block. This one really chewed up some resources before we shut the door fully.
Dragonfruit
Well-known member
Did you upload the "upload" folder, or did you upload the contents of the "upload" folder, that's the difference.
Usually within the "upload" folder you find a "src" folder, "js" depending if the addon uses javascript files.
Look at the root folder of your Xenforo installation, it should match your folders there. There should be a "src", "js", etc.
Upload the contents so 'src' matches 'src'.
If you put a 'uploads' folder into your Xenforo directory, then that's incorrect.
