XF 1.3 ROBOTS

[Alhidayah]

Hi...

My website is showing over 170 robots online due to which i think server cpu usage reaches high whoch then stops website from opening up and gives error saying the website cannot be displayed, database error.

I have used a plugin in called robots, configured it but its still showing over 170 robots.

Is there anyway i can stop this?

Thank You.

 

[Max Taxable]

170 crawlers is... Nothing. Insignificant. If that tiny amount of traffic is causing problems you have hosting issues.

 

[djbaxter]

Which addon is that? Does the addon actually block the bots? or does it just identify them?

Assuming it just identifies them, you still need to block the worst offenders (some bots are very aggressive and they can eat up bandwidth) using robots.txt exclusions for those that obey robots.txt and .htaccess for those that don't.

 

[Alhidayah]

Which addon is that? Does the addon actually block the bots? or does it just identify them?

Assuming it just identifies them, you still need to block the worst offenders (some bots are very aggressive and they can eat up bandwidth) using robots.txt exclusions for those that obey robots.txt and .htaccess for those that don't.

Hi.

This is the addon

Robots

Premium upgrade: This XF2 add-on along with the entire collection can be purchased for $35.00 USD. Your Premium upgrade will allow you to download as many XF2 add-ons as you like for one year. Please see the entire collection located in the...

xenforo.com

Yes i think its just identify them and not block them.

Yes you are right thats wots happening with my bandwidth, as you can see the attachment on my post theres not alot of members but when i visit my hosting control panel the cpu usage rech sky high and when it reacjes to 100 percent the error show up on website and it dosnt open up.

I dont no how to use robots.txt if you can hrlp me get thru it, i woud appreciate it as i am not so good woth database.

Thank you.

 

[Ozzy47]

Hi.

This is the addon

Robots

Premium upgrade: This XF2 add-on along with the entire collection can be purchased for $35.00 USD. Your Premium upgrade will allow you to download as many XF2 add-ons as you like for one year. Please see the entire collection located in the...

xenforo.com

Yes i think its just identify them and not block them.

Yes you are right thats wots happening with my bandwidth, as you can see the attachment on my post theres not alot of members but when i visit my hosting control panel the cpu usage rech sky high and when it reacjes to 100 percent the error show up on website and it dosnt open up.

I dont no how to use robots.txt if you can hrlp me get thru it, i woud appreciate it as i am not so good woth database.

Thank you.

Can’t be that addon, that is for XF2 not XF1

 

[Alhidayah]

Can’t be that addon, that is for XF2 not XF1

Sorry my apology its this 1

Robots

Description: Disallows robots from indexing your site unless explicitly allowed. (Robots link in Visitors tab) (Example of Robots page) (Example of Options page) (Example of User Group Permissions) Features: All phrases start with...

xenforo.com

 

[djbaxter]

According to the description of that addon, it does block bots, although you'd have to check with @AndyB as to how it does that.

Disallows robots from indexing your site unless explicitly allowed.

But not all bots respect robots.txt. If your forum is hosted on a linux/apache site you can block them in your .htaccess file once you know the user-agent string for the bot. However, it's easy to kill your entire site if you mess up editing .htaccess. Even using the wrong editor can do that. You would be better off asking someone to do it for you.

 

[Ozzy47]

Still is not going to stop them from trying to access the site and eating up server resources.

 

[beerForo]

What is the error you are receiving? Server optimization is recommended. It could be a PHP setting like max connections or you could use a caching solution etc. Because, what happens if you block 170 bots but get 170 real members? Same!

 

[djbaxter]

Still is not going to stop them from trying to access the site and eating up server resources.

No but it may help to discourage them from proliferating. It did make a significant difference on one site I worked on for a while that was being beseiged by Chinese bots.