How to Prevent Crawling or Scraping on My xenForo Site?

[VisEntities]

Hey,

I just realized that crawling or scraping my site's pages for info could impact its performance. How can I prevent this from happening? I don't want someone slowing down my site for any reason. Does XenForo have a way to prevent it, or is there something else I should be aware of? I'm a newbie in these subjects, so any advice would be appreciated.

Thanks!

 

[bzcomputers]

Your site is open to the internet, if someone wants to scrape your content there is not a lot you can do. Anyone who is doing it for nefarious reasons can make it look like Google is indexing your site if they want.

There are lots of other things to worry about with a site, worrying about bots crawling it is really low on the list in my opinion. Bots typically don't impact site performance very much even with a large million thread site. Remember, if it wasn't for bots nobody would be coming to your site in the first place.

 

[Suzanne O]

Get this addon from @Xon sign up and abuse and his standard library

 

[bzcomputers]

Get this addon from @Xon sign up and abuse and his standard library

Although that is a good add-on you mentioned, it has nothing to do with bots scraping and crawling a site. It will not help at all for what the original question was asking about.

 

[Suzanne O]

Although that is a good add-on you mentioned, it has nothing to do with bots scraping and crawling a site. It will not help at all for what the original question was asking about.

Wrong. It blocks them.
I use this add on and it does everything to stop them.

 

[bzcomputers]

Wrong. It blocks them.
I use this add on and it does everything to stop them.

I don't want to keep going back and forth with you on this, so this will be my last post on the subject.

Xon's Signup abuse detection and blocking add-on is for reducing spam registrations and posts. It attempts to identify and stop nefarious users and spam bots from registering and posting on a site. It does not stop any bots from crawling or scraping a site's information, which is what the original poster was asking about.

 

[VisEntities]

Your site is open to the internet, if someone wants to scrape your content there is not a lot you can do. Anyone who is doing it for nefarious reasons can make it look like Google is indexing your site if they want.

There are lots of other things to worry about with a site, worrying about bots crawling it is really low on the list in my opinion. Bots typically don't impact site performance very much even with a large million thread site. Remember, if it wasn't for bots nobody would be coming to your site in the first place.

So I shall not worry about it your advice, it's inevitable and part of internet