XF 2.2 Bots - how to deny them

nogridhome.com

New member
Bots suck all the information off of your site. "So they can help you"
I have no faith or trust in these people. I would rather deny their access than to be on their search engines.
How to deny all bots forever?

I read several posts and have decided to disable auto build and auto site map under admin, setup, options, xml.

Will this deny them?

Thanks heroes!
V2.2.9
 

OkanOzturk

Active member
Bots suck all the information off of your site. "So they can help you"
I have no faith or trust in these people. I would rather deny their access than to be on their search engines.
How to deny all bots forever?

I read several posts and have decided to disable auto build and auto site map under admin, setup, options, xml.

Will this deny them?

Thanks heroes!
V2.2.9
Create new user group and remove the view permission and you can use this add-on.

 

nogridhome.com

New member
I may try those blockers. Thanks for that!
Before I saw those posts I found wiki stating that I might be able to turn off some bots by creating a robots.txt file with the following contents:
User-agent: *
Disallow: /

I just did it, see what happens. Probably have to use one of those blockers you hero's above mentioned. Thank!
 

Blatchy

Well-known member
I may try those blockers. Thanks for that!
Before I saw those posts I found wiki stating that I might be able to turn off some bots by creating a robots.txt file with the following contents:
User-agent: *
Disallow: /

I just did it, see what happens. Probably have to use one of those blockers you hero's above mentioned. Thank!
Unfortunately the bad bots that are troubling you probably won’t obey the robots.txt file
 

nogridhome.com

New member
I may try those blockers. Thanks for that!
Before I saw those posts I found wiki stating that I might be able to turn off some bots by creating a robots.txt file with the following contents:
User-agent: *
Disallow: /

I just did it, see what happens. Probably have to use one of those blockers you hero's above mentioned. Thank!
Already see a Robot: Bing so much for the robots.txt file.
 

Max Taxable

Well-known member
Already see a Robot: Bing so much for the robots.txt file.
You'll still see them regardless. Whether they are indexing content or not.

Robots.txt blocks nothing at all. It's just a sign saying "please don't index us."

What's your problem with search engines indexing content? It's how people FIND YOUR SITE.

If you don't want ANYTHING indexed, your answer was given in post #2 of this thread. BUT YOU WILL STILL SEE BOTS in the index of who's visiting. Period.
 

nogridhome.com

New member
You'll still see them regardless. Whether they are indexing content or not.

Robots.txt blocks nothing at all. It's just a sign saying "please don't index us."

What's your problem with search engines indexing content? It's how people FIND YOUR SITE.

If you don't want ANYTHING indexed, your answer was given in post #2 of this thread. BUT YOU WILL STILL SEE BOTS in the index of who's visiting. Period.
Thank you friend,
I've noticed that big tech is mentally ill. This is why I don't want them. I have nothing to hide. I would simply have them mind their own business unless I ask for them to scour my content. I would have the people I want to find my site come because I backlinked them there. Not because a mentally ill big tech platform decided to list me or censor me or not. Period.
 

Max Taxable

Well-known member
Thank you friend,
I've noticed that big tech is mentally ill. This is why I don't want them. I have nothing to hide. I would simply have them mind their own business unless I ask for them to scour my content. I would have the people I want to find my site come because I backlinked them there. Not because a mentally ill big tech platform decided to list me or censor me or not. Period.
Then follow the advice in post #2.

You will still see bots visiting, but they will have no content to index.

Unfortunately human visitors will have to register to read however.

You can block bots completely by user agent if you want to go that far.
 

nogridhome.com

New member
thanks friend, I really didn't want to get that spun up into it.
I will just act like it's not a big deal because it is not a big deal.
However, we have a ministry of truth now here in america so I will probably get serious about it eventually.
 

Max Taxable

Well-known member
thanks friend, I really didn't want to get that spun up into it.
I will just act like it's not a big deal because it is not a big deal.
However, we have a ministry of truth now here in america so I will probably get serious about it eventually.
If you have good content you want people to find it. The search engines are a tool for that. But if you want the human public to still be able to read without creating an account but block indexing, you need to block bots by user agent.

You're still going to see bots visiting, regardless. It's just a fact of big tech internet. Seeing them visit doesn't mean they're getting your content.
 
Top