Add-on Paid Task advance robots managment

Codeless

Active member
Hy All xenforo experts,
I need modules which can handle Robots & Crawlers in xenforo forum.
I know robots have default guest permission, but they can't crawl protected forums they don't have permission.
I want a module which can give full permissions to robots to crawl forums for
security reason we can add crawler user agent + IP to whitelist it.
Right now I am facing the same issue with Semrush / Moz / Alexa / GoogleBot / Baidu Crawler / Yandex Bot / and some other important for me to get fast indexing and avoid 404 error in webmaster panel

so I will add + robot agent to put on the safe list like

Crawler name: Googlebot
User agent : Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) or Googlebot/2.1 (+http://www.google.com/bot.html)
IP Address: 64.233.160.0/32 & 66.102.0.0/32 & 66.249.64.0/32 & 72.14.192.0/32 & 74.125.0.0/32

so basically we will be allowing Googlebot from these it with full permissions

I will also need lifetime updates we can decide yearly based
if anyone willing to do it let me know
 
we will add a new group for robots.
so guest will see only login screen and crawlers will see whole forums is opened and can see anything
 
OK, as long as you are aware that Google may take action if you use cloaking, which will have a negative impact on your site.
 
Back
Top Bottom