Bot Management using robots.txt in XFcloud

It took about 2 to 3 weeks to see bytespider begin to comply with the suggested addition to robots txt. They do not visit anymore (so far) and it's been as long as my last post in this thread.
The same happened on our site. A few days after adding Bytespider to our robots.txt file they stopped visiting. Well today they’re back again. Four pages of them. Now what?
 
It is concerning to me that Bytedance/spider are ignoring robots.txt. We may look at a more robust solution for this that we can implement centrally for all customers.
Is there a follow up on this maybe? We’re having tons of Bytespider bots at the moment and no way to stop them. They are ignoring our robots.txt file.
 
Is there a follow up on this maybe? We’re having tons of Bytespider bots at the moment and no way to stop them. They are ignoring our robots.txt file.
Unfortunately this particular spider chooses to ignore the robots.txt file so the only way that works is via .htaccess but in the cloud you do not have access.

 
Is there a way to block an IP range? All IP addresses start with 47.128

They seem most interested in our members’ images.

EDIT: I thought of a workaround. I put the whole damn 47.128 range in a severe discouragement mode. That worked.
 
Last edited:
It took about 2 to 3 weeks to see bytespider begin to comply with the suggested addition to robots txt. They do not visit anymore (so far) and it's been as long as my last post in this thread.
I can report that the quote above still holds true, and so far bytespider is complying and no longer sending bots, still.
 
I can report that the quote above still holds true, and so far bytespider is complying and no longer sending bots, still.
Then you were lucky, I guess. I added Bytespider to our robots.txt file and also modified the Page-container on 6 June. A few days after that they stopped visiting us. Until today, when they suddenly swarmed us. No idea why.

But as I said, I put the IP range in a severe discouragement mode and their numbers are now down. They still visit us but there are less of them now and they are all redirected to our homepage and no longer scraping images.
 
Back
Top Bottom