Whats your robots.txt file look like? - Xenforo

[CTXMedia]

But it looks like they still are. Mostly, google accesses the calendar, but once in a while they still access a thread.

Are you sure? They may be requesting it but can they actually see the page content (200 code response from your web server) or are the requests generating a 400 series error code? (Check your web server logs).

 

[Diana]

Is Google treated as a member of your forum because of some addon?

Are you using AdSense?

No, not using AdSense.

 

[Diana]

Are you sure? They may be requesting it but can they actually see the page content (200 code response from your web server) or are the requests generating a 400 series error code? (Check your web server logs).

I've tried this a couple of ways.

1. Denying permissions to visitors. In this case the bots can still access the threads and I assume see the page.

2. Deactivating the forum. In this case the bots can access threads, but an "error" seems to indicate they can't see the page. Of course this is useless as closing the forum will prevent members from being able to log in.

Still, it's curious the bots can access a thread in both instances.

I've just reactivated and will check the server logs.

 

[Paul B]

1. Denying permissions to visitors. In this case the bots can still access the threads and I assume see the page.

If they don't have permission, they can't access the content.

You can confirm by using incognito mode in your browser and attempting to access it yourself.

 

[Diana]

Thank you Brogan. I'll take your word for it. I just didn't understand how when I check "Members Online" I can see the bot, say google or Baidu, listed with the name of a thread next to it.

Hmm incognito mode, I'll look for that.

 

[Brent W]

Thank you Brogan. I'll take your word for it. I just didn't understand how when I check "Members Online" I can see the bot, say google or Baidu, listed with the name of a thread next to it.

Hmm incognito mode, I'll look for that.

Because they have already known it has existed and are still trying to access it. That can't see anything though.

 

[Diana]

Because they have already known it has existed and are still trying to access it. That can't see anything though.

Ah, that makes sense. Thank you.

 

[Diana]

Another question, related I think.

How do I prevent all these spammers (I don't know if they're bots or human) from hitting up my site, especially my test forum (which is deactivate). There are currently 138 (from China, Russia, Ukraine, Korea, Romania, etc) "viewing" as "guests." My host has shut down my forum several times because all this activity has caused my bandwidth limit to be exceeded. I thought a robot.txt might help, but apparently it hasn't.

This is my robot.txt file in my root. Is there something I need to add to it? Or is there another way to block them?

User-agent: *
Disallow: /forum/find-new/
Disallow: /forum/account/
Disallow: /forum/attachments/
Disallow: /forum/goto/
Disallow: /forum/posts/
Disallow: /forum/login/
Disallow: /forum/events/
Disallow: /forum/profile/
Disallow: /forum/calendar/
Disallow: /forum/misc/
Disallow: /forum/search/
Disallow: /forum/members/
Disallow: /forum/register/
Disallow: /forum/online/
Disallow: /forum/recent-activity/
Disallow: /forum/lost-password/
Disallow: /forum/admin.php
Disallow: /forum/help/
Allow: /

 

[Robru]

How do I prevent all these spammers from hitting up my site, especially my test forum (which is deactivate). There are currently 138 (from China, Russia, Ukraine, Korea, Romania, etc) "viewing" as "guests."

Possibility to stop .ru without stuffing your .htaccess and slowing down your server is to block the whole country with only one line in your .htaccess. Simply add on the top of your .htaccess.

#deny all russian IP's
Order allow,deny
deny from .ru
Allow from all

This will invoke that all IP's with veverse DNS .ru are blocked from accessing your site. Nevertheless on your server logs you will continue to see the requests/visits, but with a forbidden http-header. Not all servers allow to block by country, sou you should try by yourself or ask your provider.

 

[Diana]

Possibility to stop .ru without stuffing your .htaccess and slowing down your server is to block the whole country with only one line in your .htaccess. Simply add on the top of your .htaccess.

#deny all russian IP's
Order allow,deny
deny from .ru
Allow from all

This will invoke that all IP's with veverse DNS .ru are blocked from accessing your site. Nevertheless on your server logs you will continue to see the requests/visits, but with a forbidden http-header. Not all servers allow to block by country, sou you should try by yourself or ask your provider.

Thank you. I'll try that.

 

[Diana]

That seems to have worked. Yaaay!

Thank you Robru.

 

[Diana]

Well now Russia and Romania are still coming around, en mass. Then there's also China.

I just don't understand why they're hounding a site that hasn't even been launched yet. I have another forum and there are none, zero, unwelcome intruders. I really don't want to have to pay for extra bandwidth because of them.

 

[Teapot]

Well now Russia and Romania are still coming around, en mass. Then there's also China.

I just don't understand why they're hounding a site that hasn't even been launched yet. I have another forum and there are none, zero, unwelcome intruders. I really don't want to have to pay for extra bandwidth because of them.

Unless your bandwidth bill is tiny, they shouldn't be causing too much trouble. That being said, if you need to stop them before they hit your server (and use your bandwidth), I'd recommend something like Cloudflare – you can use their firewall to blanket-ban countries or IP ranges, and they'll never get to your origin server at all. Their CDN capability can also help reduce the bandwidth your site uses by caching resources such as images. It takes some custom configuration to make it work effectively (particularly in terms of passing through visitor IPs correctly), but the resources to show you what to do are readily available on this site.

 

[Mr Lucky]

Another question, related I think.

How do I prevent all these spammers (I don't know if they're bots or human) from hitting up my site, especially my test forum (which is deactivate).

Your test forum should be in a protected directory. No one can get in then (apart from Mr Putin maybe ) . It is actually a part of the licence agreement I think that the test forum should be password protected, not just disabled.

 

[Diana]

Unless your bandwidth bill is tiny, they shouldn't be causing too much trouble. That being said, if you need to stop them before they hit your server (and use your bandwidth), I'd recommend something like Cloudflare – you can use their firewall to blanket-ban countries or IP ranges, and they'll never get to your origin server at all. Their CDN capability can also help reduce the bandwidth your site uses by caching resources such as images. It takes some custom configuration to make it work effectively (particularly in terms of passing through visitor IPs correctly), but the resources to show you what to do are readily available on this site.

Thank you. I'll check this out.

 

[Diana]

Your test forum should be in a protected directory. No one can get in then (apart from Mr Putin maybe ) . It is actually a part of the licence agreement I think that the test forum should be password protected, not just disabled.

Thanks, I didn't know about this. I just did a search and found a post by Brogan, which reads as follows:

Just use .htaccess or use usergroup permissions to restrict everything to guests, and disallow registration; you can manually register any support staff who may need to access it.

You can use this guide to password protect a directory: http://xenforo.com/community/thread...d-the-install-directory-using-htaccess.13768/

As restricting access doesn't solve the bandwidth problem, I'll look into password protection.