Bot Management using robots.txt in XFcloud
- Thread starter CTS
- Start date
Is there a follow up on this maybe? We’re having tons of Bytespider bots at the moment and no way to stop them. They are ignoring our robots.txt file.
Unfortunately this particular spider chooses to ignore the robots.txt file so the only way that works is via .htaccess but in the cloud you do not have access.
IMO, best to block by .htaccess
Eddit: Just noticed your on XF cloud and do not have access to the file.
Code:
BrowserMatchNoCase "Bytedance" bad_bot
BrowserMatchNoCase "Bytespider" bad_bot
BrowserMatchNoCase "Baiduspider" bad_bot
Order Deny,Allow
Deny from env=bad_bot...Then you were lucky, I guess. I added Bytespider to our robots.txt file and also modified the Page-container on 6 June. A few days after that they stopped visiting us. Until today, when they suddenly swarmed us. No idea why.
But as I said, I put the IP range in a severe discouragement mode and their numbers are now down. They still visit us but there are less of them now and they are all redirected to our homepage and no longer scraping images.
These are far from being efficient for defending bots. At least, your list has lack of those search engines from Mainland China.
I'm sharing my plan here. This blocks Sogou search engine crawler.
NGINX:
# ========== [MUST BE THE FIRST] NEMESIS AGAINST SOGOU CRAWLER ==========
# 444: Cut TCP, GOING DARK, NO RESPONSE.
if ($http_user_agent ~* "Sogou") {
return 444;
}
# 444 for other crawlers:
if ($http_user_agent ~* "(Sogou|360Spider|Bytespider|YisouSpider|Spider)") {
return 444;
}
# Stop crawlers from touching these paths.
location ~* ^/(misc|data|error|files|install|internal_data|js|library|non_official_resources|src|styles|account|attachments|goto|posts|login|search|whats-new)/ {
if ($http_user_agent ~* (spider|bot|crawl|slurp|Sogou)) {
return 444;
}
try_files $uri $uri/ /index.php?$query_string;
}
# Stop crawlers from touching this php.
location ~* ^/(admin.php) {
if ($http_user_agent ~* (spider|bot|crawl|slurp|Sogou)) {
return 444;
}
try_files $uri $uri/ /index.php?$query_string;
}
Last edited:
I doubt that this will work within robots.txt in XF Cloud (which is the topic this thread is about).
Hum... Sogou crawlers don't give robots.txt a f***. That's the problem.
My plan is to set a defense on the Nginx level. This let crawlers eat 444s but can't stop them from flooding against Nginx.
Last edited:
Robots.txt relies on cooperation and today many bots and crawlers do not cooperate, so robots.txt is useless with them. .htaccess (and probably the Nginx aeqivalent as well) does not need cooperative bots and can do way more. So your idea is clearly the right direction.
But this thread is about the topic of robots.txt on XF cloud, not about .htaccess, which is not configurable on XF Cloud. There are loads of threads around the topic of bots and how to deal with them here on the forum and probably a bunch about .htaccess or limiting access via Nginx as well. Probably you should address your ideas in one of those as there will be more responses than in a thread dedicated to XF Cloud where the toolchain you want to use is not available.
For those, robots.txt is a handy public list of sensitive places on the site!
Answering a two year old thread w/o even looking at the question that was asked
is always the most constructive way of dealing with it.
There are a couple bot’s you should probably allow. Applebot Googlebot twitterbot as they do unfurl on phones
Similar threads
- Question
