XF 2.2 Bots - how to deny them

[nogridhome.com]

Bots suck all the information off of your site. "So they can help you"
I have no faith or trust in these people. I would rather deny their access than to be on their search engines.
How to deny all bots forever?

I read several posts and have decided to disable auto build and auto site map under admin, setup, options, xml.

Will this deny them?

Thanks heroes!
V2.2.9

 

[Paul B]

No, that won't stop them.

The only way in XF to prevent bots from scraping content from your site is to remove the view permission for the unregistered user group.

 

[Anatoliy]

Cloudflare. Free plan. Block them by IP,.name, country, etc...

 

[AndyB]

You might like this add-on:

https://xenforo.com/community/resources/robots.5623/

 

[OkanOzturk]

Bots suck all the information off of your site. "So they can help you"
I have no faith or trust in these people. I would rather deny their access than to be on their search engines.
How to deny all bots forever?

I read several posts and have decided to disable auto build and auto site map under admin, setup, options, xml.

Will this deny them?

Thanks heroes!
V2.2.9

Create new user group and remove the view permission and you can use this add-on.

[BS] Robot user group

.features: Setting a user group for search robots Sitemap generation based on search bot permissions

xenforo.com

 

[Blatchy]

This works quite well for me

nginx-ultimate-bad-bot-blocker/README.md at master · mitchellkrogza/nginx-ultimate-bad-bot-blocker

Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail f...

github.com

 

[nogridhome.com]

I may try those blockers. Thanks for that!
Before I saw those posts I found wiki stating that I might be able to turn off some bots by creating a robots.txt file with the following contents:
User-agent: *
Disallow: /

I just did it, see what happens. Probably have to use one of those blockers you hero's above mentioned. Thank!

 

[Blatchy]

I may try those blockers. Thanks for that!
Before I saw those posts I found wiki stating that I might be able to turn off some bots by creating a robots.txt file with the following contents:
User-agent: *
Disallow: /

I just did it, see what happens. Probably have to use one of those blockers you hero's above mentioned. Thank!

Unfortunately the bad bots that are troubling you probably won’t obey the robots.txt file

 

[nogridhome.com]

I may try those blockers. Thanks for that!
Before I saw those posts I found wiki stating that I might be able to turn off some bots by creating a robots.txt file with the following contents:
User-agent: *
Disallow: /

I just did it, see what happens. Probably have to use one of those blockers you hero's above mentioned. Thank!

Already see a Robot: Bing so much for the robots.txt file.

 

[Max Taxable]

Already see a Robot: Bing so much for the robots.txt file.

You'll still see them regardless. Whether they are indexing content or not.

Robots.txt blocks nothing at all. It's just a sign saying "please don't index us."

What's your problem with search engines indexing content? It's how people FIND YOUR SITE.

If you don't want ANYTHING indexed, your answer was given in post #2 of this thread. BUT YOU WILL STILL SEE BOTS in the index of who's visiting. Period.

 

[nogridhome.com]

You'll still see them regardless. Whether they are indexing content or not.

Robots.txt blocks nothing at all. It's just a sign saying "please don't index us."

What's your problem with search engines indexing content? It's how people FIND YOUR SITE.

If you don't want ANYTHING indexed, your answer was given in post #2 of this thread. BUT YOU WILL STILL SEE BOTS in the index of who's visiting. Period.

Thank you friend,
I've noticed that big tech is mentally ill. This is why I don't want them. I have nothing to hide. I would simply have them mind their own business unless I ask for them to scour my content. I would have the people I want to find my site come because I backlinked them there. Not because a mentally ill big tech platform decided to list me or censor me or not. Period.

 

[Max Taxable]

Thank you friend,
I've noticed that big tech is mentally ill. This is why I don't want them. I have nothing to hide. I would simply have them mind their own business unless I ask for them to scour my content. I would have the people I want to find my site come because I backlinked them there. Not because a mentally ill big tech platform decided to list me or censor me or not. Period.

Then follow the advice in post #2.

You will still see bots visiting, but they will have no content to index.

Unfortunately human visitors will have to register to read however.

You can block bots completely by user agent if you want to go that far.

 

[nogridhome.com]

thanks friend, I really didn't want to get that spun up into it.
I will just act like it's not a big deal because it is not a big deal.
However, we have a ministry of truth now here in america so I will probably get serious about it eventually.

 

[Max Taxable]

thanks friend, I really didn't want to get that spun up into it.
I will just act like it's not a big deal because it is not a big deal.
However, we have a ministry of truth now here in america so I will probably get serious about it eventually.

If you have good content you want people to find it. The search engines are a tool for that. But if you want the human public to still be able to read without creating an account but block indexing, you need to block bots by user agent.

You're still going to see bots visiting, regardless. It's just a fact of big tech internet. Seeing them visit doesn't mean they're getting your content.

 

[Muddy Boots]

If your running a VPS with Centminmod then there are some good reads on @eva2000 's site

https://community.centminmod.com/threads/nginx-ultimate-bad-bot-blocker-for-with-centmin-fix-auto-update.19771/

https://community.centminmod.com/threads/blocking-bad-or-aggressive-bots.6433/

 

[stefanstp]

Does the module no longer exist? I can't buy/download it anywhere. Is there an alternative for it now?

 

[Jja]

Does the module no longer exist? I can't buy/download it anywhere. Is there an alternative for it now?

Again, You can use Cloudflare to block robots - You can block them by blocking their user-agent.
You can also block them in your webserver configuration, by blocking their user-agent, but depends on your site/hosting configuration.
In my opinion is always better to block what you want before it connects to your server, so CF is very good option.

 

[stefanstp]

Is Cloudflare finally running properly with xenforo? Because in the past there was a problem that users sometimes could not send contributions and so we had to remove Cloudflare again.

 

[Jja]

It was never XF problem.

 

[digitalpoint]

Is Cloudflare finally running properly with xenforo? Because in the past there was a problem that users sometimes could not send contributions and so we had to remove Cloudflare again.

XenForo has worked perfectly fine with XenForo since 1.0. There are Cloudflare settings that can certainly cause problems with XenForo installations if set incorrectly (zone Security Level and Rocket Loader for example). But nothing has been "fixed" on Cloudflare's side that you couldn't already do (disable certain settings that wouldn't apply to some applications like XenForo). See this thread for some general tips on how to set some Cloudflare settings for XenForo.