robots.txt / Webcrawler

R

Robert9

Well-known member
After preparing my robots.txt some weeks ago i could get rid of some crawlers.
Now there is a new one with 114.119.167.x from Asia, Singapur.

May someone knows this IP/crawler/website?
 
You must log in or register to reply here.

Similar threads

C
Bot Management using robots.txt in XFcloud
Replies
12
Views
452
CTS
C
KhanTastic
  • Question Question
XF 2.2 Does XF has a default robots.txt file?
Replies
4
Views
420
KhanTastic
KhanTastic
A
  • Question Question
XF 2.2 Robots.txt unreadable
Replies
3
Views
248
Black Tiger
Black Tiger
JasonBrody
  • Question Question
XF 2.2 Robots.txt is not accessible or doesn't exist
Replies
10
Views
792
Chromaniac
Chromaniac
bzcomputers
  • Question Question
XF 2.2 Is it safe to disallow all query strings in robots.txt for XenForo 2.x?
Replies
0
Views
475
bzcomputers
bzcomputers
Top Bottom