[DigitalPoint] App for Cloudflare® 1.9.1.1

[digitalpoint]

There is no "contains" operator for user agent rules. If you want to use "contains" for user agents, you can use a firewall rule (more flexible, but also more complicated):

http://dash.cloudflare.com/?to=/:account/:zone/security/waf/custom-rules/new

The rule someone posted above is mostly right... general idea anyway (except ByteDanceSpider isn't a user agent that would match).



Again... not a specific user agent rule, but it also doesn't need to match exactly.

 

[Alvin63]

Thanks. Just checking I've been looking in the right place in the app. I was under Firewall - User agent - to set up a rule.

In settings there is a Bots section - should I be using that instead? Although don't want to block all bots obviously.

 

[digitalpoint]

Thanks. Just checking I've been looking in the right place in the app. I was under Firewall - User agent - to set up a rule.

In settings there is a Bots section - should I be using that instead? Although don't want to block all bots obviously.

User agent rules are exact match. The issue is you don't have access to your web server logs, so you don't actually know what the user agent is (exactly anyway). The link I gave you above will allow you to do user agent blocking with a contains operator.

 

[Alvin63]

There is no "contains" operator for user agent rules. If you want to use "contains" for user agents, you can use a firewall rule (more flexible, but also more complicated):

http://dash.cloudflare.com/?to=/:account/:zone/security/waf/custom-rules/new

The rule someone posted above is mostly right... general idea anyway (except ByteDanceSpider isn't a user agent that would match).

View attachment 323756

Again... not a specific user agent rule, but it also doesn't need to match exactly.

Thanks. That didn't seem to work either apparently. So I tried blocking Singapore. And it's still there! With Singapore next to the name. I think it might be bypassing Cloudflare.

 

[digitalpoint]

Thanks. That didn't seem to work either apparently. So I tried blocking Singapore. And it's still there! With Singapore next to the name. I think it might be bypassing Cloudflare.

It's certainly possible... but without access to web server logs, you are kind of flying blind.

 

[Alvin63]

Thanks. I'll give up for now. That would be concerning though if Bytedance knew the origin IP of the website. I suppose it could have obtained it before I set up the app to proxy unfurling .......... If it is bypassing Cloudflare though, then presumably the only option would be to block it in htaccess? Or range block the IP's in the server?

 

[digitalpoint]

It may not be that it tried to determine your server IP specifically, rather they could just be spidering random IPs.

If you have the ability to block IPs at the web server level, I'd block all IPs other than Cloudflare's because if you are proxying through Cloudflare, it should ONLY be Cloudflare IPs making connections to your server. You can find all Cloudflare's IP ranges here:

IP Ranges

This page is intended to be the definitive source of Cloudflare’s current IP ranges.

www.cloudflare.com

 

[Alvin63]

It may not be that tried to determine your server IP specifically, rather they could just be spidering random IPs.

If you have the ability to block IPs at the web server level, I'd block all IPs other than Cloudflare's because if you are proxying through Cloudflare, it should ONLY be Cloudflare IPs making connections to your server. You can find all Cloudflare's IP ranges here:

IP Ranges

This page is intended to be the definitive source of Cloudflare’s current IP ranges.

www.cloudflare.com

Thanks. I had intended to do that before, but it wasn't possible. The server won't let me whitelist the full IP ranges. Only all the individual Cloudflare IP's and I worked out there were probably thousands! Eg there is a list of 20 or so Cloudflare IP's but each of those is a range with many more included.

 

[digitalpoint]

Thanks. I had intended to do that before, but it wasn't possible. The server won't let me whitelist the full IP ranges. Only all the individual Cloudflare IP's and I worked out there were probably thousands! Eg there is a list of 20 or so Cloudflare IP's but each of those is a range with many more included.

Ya, that definitely won't work. There are 524,285 IPs in each /13 block alone.

Maybe time to think about better hosting? Seems like they aren't that great in all the ways.

 

[Alvin63]

Ya, that definitely won't work. There are 524,285 IPs in each /13 block alone.

Maybe time to think about better hosting? Seems like they aren't that great in all the ways.

I know I went round in circles with this before. The only other plan that would allow it is a VPS hosting plan and that's a bit beyond my technical skills.

 

[Alvin63]

I've sent a request to the server to block ByteDance and Bytespider at server level, which I believe some servers do. But they will probably ignore me

How about the AI Labyrinth option in the addon?

 

[digitalpoint]

I've sent a request to the server to block ByteDance and Bytespider at server level, which I believe some servers do. But they will probably ignore me

How about the AI Labyrinth option in the addon?

Well if they are hitting the server direct, none of the Cloudflare settings will do anything because the traffic isn’t passing through Cloudflare.

 

[Alvin63]

Ok that's blocked in .htaccess now Thanks to @philmckrackon. I still have increased "guests" and just wondering if I should or could set up something in the Cloudflare app to block malicious bots (who may be posing as guests?).

In settings in the Bots section there are these options. Would "Defininitely automated" be a good one to disallow? Or something else?

 

[Alvin63]

Actually "definitely automated" is greyed out. Tried turning on Bot fight mode at the top but likely automated and definitely automated are still greyed out (as is verified bots)

 

[Chromaniac]

Those are for cloudflare paying customers iirc.

 

[Alvin63]

Those are for cloudflare paying customers iirc.

Ah right thank you. So greyed out as not available on my free plan then? So just bot fight mode and AI settings available then? Is it worth turning on bot fight mode?

I've enabled bot fight mode and AI labyrinth. Sound ok? It doesn't seem to have reduced the number of guests but maybe it's a protection against bad ones.

 

[Chromaniac]

Bot fight mode is fine. Al settings are also fine if you don't like ai sites to index your content. Guests numbers are also fine as long as your site is not slowing down due to load. It is too much work to keep a constant eye on who is landing on your website. As long as it works optimally, best to focus on community.

 

[Alvin63]

Thanks. As long as it doesn't block Google as well! Yes I agree - focus on the forum. I'm just tidying some things up

 

[Chromaniac]

It doesn't block major search spiders. I can confirm that.

It did block a telegram bot I am using so I had to manually whitelist it

 

[Alvin63]

I just turned them off again for now. See how it goes My thought had been that Bytedance might be being sneaky and appearing as a guest instead now it was blocked - hence the AI labyrinth thing - apparently that only works for AI bots that ignore robots text. But then I might block some ok ones as well maybe - google probably has an AI bot as well, although it would probably follow robots.txt. But left it off for now.