Bot Management using robots.txt in XFcloud

[FoP]

It took about 2 to 3 weeks to see bytespider begin to comply with the suggested addition to robots txt. They do not visit anymore (so far) and it's been as long as my last post in this thread.

The same happened on our site. A few days after adding Bytespider to our robots.txt file they stopped visiting. Well today they’re back again. Four pages of them. Now what?

 

[FoP]

It is concerning to me that Bytedance/spider are ignoring robots.txt. We may look at a more robust solution for this that we can implement centrally for all customers.

Is there a follow up on this maybe? We’re having tons of Bytespider bots at the moment and no way to stop them. They are ignoring our robots.txt file.

 

[philmckrackon]

Is there a follow up on this maybe? We’re having tons of Bytespider bots at the moment and no way to stop them. They are ignoring our robots.txt file.

Unfortunately this particular spider chooses to ignore the robots.txt file so the only way that works is via .htaccess but in the cloud you do not have access.

Post in thread 'Bot Management using robots.txt in XFcloud'

Jun 6, 2024

I added Bytespider and Bytedance to the Robots.txt file yesterday after Bytespider started showing up on our forum. They ignore the file. At this moment we have 50 robots and 26 are Bytespider so they multiplied. Banning IP addresses also doesn’t work. We too are on Cloud hosting.


Thank you. Will try this.

IMO, best to block by .htaccess
Eddit: Just noticed your on XF cloud and do not have access to the file.

BrowserMatchNoCase "Bytedance" bad_bot
BrowserMatchNoCase "Bytespider" bad_bot
BrowserMatchNoCase "Baiduspider" bad_bot
Order Deny,Allow
Deny from env=bad_bot...

• philmckrackon

 

[FoP]

Unfortunately this particular spider chooses to ignore the robots.txt file so the only way that works is via .htaccess but in the cloud you do not have access.

Yes, and that’s a problem. But @Chris D said:

We may look at a more robust solution for this that we can implement centrally for all customers.

And that’s what I’m hoping for.

 

[FoP]

Is there a way to block an IP range? All IP addresses start with 47.128

They seem most interested in our members’ images.

EDIT: I thought of a workaround. I put the whole damn 47.128 range in a severe discouragement mode. That worked.

 

[CTS]

It took about 2 to 3 weeks to see bytespider begin to comply with the suggested addition to robots txt. They do not visit anymore (so far) and it's been as long as my last post in this thread.

I can report that the quote above still holds true, and so far bytespider is complying and no longer sending bots, still.

 

[FoP]

I can report that the quote above still holds true, and so far bytespider is complying and no longer sending bots, still.

Then you were lucky, I guess. I added Bytespider to our robots.txt file and also modified the Page-container on 6 June. A few days after that they stopped visiting us. Until today, when they suddenly swarmed us. No idea why.

But as I said, I put the IP range in a severe discouragement mode and their numbers are now down. They still visit us but there are less of them now and they are all redirected to our homepage and no longer scraping images.