Be wary of IP banning users...

[Chris D]

We don't make a habit of IP banning users. Mainly because it's too easy to circumvent. But, we occasionally do. I need to speak to the Admin involved to rationalise the reason behind the action, but recently we IP banned a user.

It's not usually a problem though.

So I get alerted to the fact today that our site is pretty much invisible to Google. It's nowhere to be seen in search results.

We find hundreds of thousands of 403 errors on Google Webmaster Tools and we investigate a bit further.

Turns out that one of our IP bans (the aforementioned user) had been spoofing his IP address. The IP address he was spoofing with was Google's.

We had IP banned one of Google's crawlbots.

This has now been rectified. And we will never ban an IP address again.

Be wary

 

[FredC]

What am i missing? I fail to see how that couldnt have been avoided with a little homework before the hammer came down on that particular IP? Sorry serious question. I guess ive always made a habit of fully investigating any IP that comes with a bad apple. But then as a disclaimer my site doesn't get hundreds of registrations a week either. And even though i have a large membership the active user base has never been more then a hundred or two at any given point in its history.. Making those couple extra queries not to troublesome.

 

[Chris D]

I couldn't agree more.

I'm pretty sure we briefed the moderators to not use IP banning or discouragement in the first place which is why I need to work out why that decision was made and ensure we don't make that mistake again.

 

[Biker]

Weary or wary?

 

[Chris D]

Weary or wary?

Doh.

Requested it to be changed, thanks.

 

[Biker]

Although, I get tired (weary) of banning them as well.

 

[AlexT]

Uhm. How would it be possible to spoof a Google IP? You don't just "spoof" an IP address unless you somehow gained access to a Google machine and/or datacenter (which I doubt happened here). You can fake IP addresses in certain http headers, which, however should never be used for anything such as banning (and I doubt xenForo uses them in such a way). So whatever happened, to me it looks more like one of your team members accidentally added the wrong IP to the ban.

 

[TPerry]

Weary or wary?

I guess it could be both?
Being wary to not ban an IP, and weary from banning a wrong one and then having to figure out why Google is getting so many 403 errors.

 

[Mike]

What AlexT said.

Unless you're setting the REMOTE_ADDR to something like HTTP_X_FORWARDED_FOR -- that's actually the reason we don't do anything like that. The proxy IP should only be trusted in very specific scenarios. The REMOTE_ADDR is the actual address that did the TCP handshake so, as far as I understand, it can't be spoofed (unlike UDP traffic, hence things like DNS reflection attacks).

 

[Chris D]

Uhm. How would it be possible to spoof a Google IP? You don't just "spoof" an IP address unless you somehow gained access to a Google machine and/or datacenter (which I doubt happened here). You can fake IP addresses in certain http headers, which, however should never be used for anything such as banning (and I doubt xenForo uses them in such a way). So whatever happened, to me it looks more like one of your team members accidentally added the wrong IP to the ban.

One way or another, it definitely happened. This was not human error.

We IP banned the user. The user had Google's IP address.

Here's a look up of the IP address:

Here's a look at the user's IP addresses:

And here's a ping resolving to the hostname:

 

[AlexT]

My guess would be some addon messing with your user sessions or the xF IP Model.

 

[Biker]

I've seen similar things with spoofed Google headers. For example:

95.211.77.144 # lfd: (sshd) Failed SSH login from 95.211.77.144 (NL/Netherlands/crawl-66-249-71-83.googlebot.com): 5 in the last 300 secs - Thu Nov 17 16:46:32 2011

Chris, have you checked your server logs to see if there's something similar?

 

[erich37]

is it possible to "whitelist" IP's of crawlers?
So that those will not get banned, even if you ban it in ACP ?

 

[Adam Howard]

You don't need to spoof Google. You can use Google as a "poor mans proxy". The same IP address uses for Good Translate is also rotated for other services including Google Bot.

 

[Adam Howard]

You don't need to spoof Google. You can use Google as a "poor mans proxy". The same IP address uses for Good Translate is also rotated for other services including Google Bot.

I didn't clear my cookies or flush out my dns and a few other things I could have done before doing this "demo"

But I'm fairly sure if @Mike checks the IP address on this post, it would match Google's

And I would have to do is this
http://www.google.com/translate?hl=&sl=en&tl=nl&u= Change domain name

You can do the same thing with Yahoo the last I checked.

 

[Adam Howard]

http://xenforo.com/community/threads/be-wary-of-ip-banning-users.50657/#post-541743

^ That's the post, not the one I quoted.

Although it likely failed as I was signed into Google. You can easily do a search to confirm how easy it is to use Google, Bing, and Yahoo as a proxy. Information has been out for years now.

 

[AlexT]

The same IP address uses for Good Translate is also rotated for other services including Google Bot.

Not true.

Matt Cutts September 21, 2006 at 7:08 pm
Jack, that’s a really neat idea about making your own reverse DNS zones. But regarding whois, IncrediBILL makes a good point. IP ranges from whois are great, but that would also include (for example) employees surfing from corporate HQ or fetches from translate.google.com. The method I give is how to verify specifically Googlebots that can fetch for the main web index.

IncrediBILL, I believe this will also apply to “Mediapartners-Google” because those bots can fetch into the same cache crawling proxy.

Jonathan, I have to admit that I’m pretty Blogger illiterate–for now.

alek, this should be an invariant for bots that can fetch for the main web index. I’ll try to do a more in-detail post that talks about some things like translate.google.com that fetch pages, but not from the same IP range as bots that can populate the main web index.

 

[Adam Howard]

Not true.

Disagree based upon actual use.

We banned the IP address used for Google Translate for this very reason. Doing so also banned Google Bot.

Your source is also dated 2006 .... 7 years ago.

 

[AlexT]

I've seen similar things with spoofed Google headers. For example:

95.211.77.144 # lfd: (sshd) Failed SSH login from 95.211.77.144 (NL/Netherlands/crawl-66-249-71-83.googlebot.com): 5 in the last 300 secs - Thu Nov 17 16:46:32 2011

This is the reverse DNS - it resolves an IP address to a domain name. That domain name can be freely chosen, thus faked (not spoofed). To verify that it is not a fake domain name, you'd have to try FCrDNS verification. See here:

http://en.wikipedia.org/wiki/Forward_Confirmed_reverse_DNS

 

[AlexT]

Disagree based upon actual use.

We banned the IP address used for Google Translate for this very reason. Doing so also banned Google Bot.

Your source is also dated 2006 .... 7 years ago.

Sorry for quoting an outdated source (who the heck is Matt Cutts anyways?). You banned "the IP address"? Well, there might be more than one IP address. And Google doesn't publish the IP addresses/ranges for their services (since they can change at any time). And I say it again: Googlebot and Google Translate do not share IP addresses, simple because otherwise their recommended way (rDNS on the IP) of verifying Googlebot wouldn't work.