• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Add-on Paid Task advance robots managment

Codeless

Active member
#1
Hy All xenforo experts,
I need modules which can handle Robots & Crawlers in xenforo forum.
I know robots have default guest permission, but they can't crawl protected forums they don't have permission.
I want a module which can give full permissions to robots to crawl forums for
security reason we can add crawler user agent + IP to whitelist it.
Right now I am facing the same issue with Semrush / Moz / Alexa / GoogleBot / Baidu Crawler / Yandex Bot / and some other important for me to get fast indexing and avoid 404 error in webmaster panel

so I will add + robot agent to put on the safe list like

Crawler name: Googlebot
User agent : Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) or Googlebot/2.1 (+http://www.google.com/bot.html)
IP Address: 64.233.160.0/32 & 66.102.0.0/32 & 66.249.64.0/32 & 72.14.192.0/32 & 74.125.0.0/32

so basically we will be allowing Googlebot from these it with full permissions

I will also need lifetime updates we can decide yearly based
if anyone willing to do it let me know
 

Codeless

Active member
#2
we will add a new group for robots.
so guest will see only login screen and crawlers will see whole forums is opened and can see anything
 

Martok

Well-known member
#5
OK, as long as you are aware that Google may take action if you use cloaking, which will have a negative impact on your site.