[DigitalPoint] App for Cloudflare® 1.8.7

[Rusty Snippets]

Yes. I’ve got a Singapore bot that lives on my board. Finally blocked the country, but he still gets through. It’s a Huawei data center location. Always viewing unknown page.

 

[Old Nick]

Hello @digitalpoint, I have IndexNow enabled with Bing, and Cloudflare is sending unwanted URLs, is there a way to prevent this?



Thanks.

 

[digitalpoint]

Hello @digitalpoint, I have IndexNow enabled with Bing, and Cloudflare is sending unwanted URLs, is there a way to prevent this?

View attachment 307140

Thanks.

I don't think so. If you don't want a search engine to index certain things, just use robots.txt to tell them what you don't want indexed.

 

[Old Nick]

Thank you, I just discovered that I hadn't edited my robots.txt file, and it was configured with the /community/ path, which I don't use.

 

[JoyFreak]

I have a question, you might be able to help me. I've set the Cloudflare SSL/TLS to Full (strict) and site's URLs all redirect non-www to www. But for some reason URLs for attachments and .txt files etc. are flexible between www and non-www. For example:

https://www.joyfreak.com/data/avatars/o/0/2.jpg?1720952280
https://joyfreak.com/data/avatars/o/0/2.jpg?1720952280

What is going on here?

 

[Old Nick]

I don't think so. If you don't want a search engine to index certain things, just use robots.txt to tell them what you don't want indexed.

Ok but...

User-agent: *
Disallow: /whats-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /data/
Disallow: /styles/
Disallow: /css.php
Disallow: /admin.php
Allow: /

And

It doesn't seem to have any impact.

 

[digitalpoint]

Are they actually being indexed though?

 

[Old Nick]

Good question, it seems not. So I shouldn't worry about that, right? Thanks.

 

[digitalpoint]

Good question, it seems not. So I shouldn't worry about that, right? Thanks.

I wouldn’t… if I’m being honest, I also wouldn’t worry about using IndexNow though.

 

[Old Nick]

Sorry, I don't know the nuances of Shakespeare's language well, you mean you wouldn't use IndexNow at all?

 

[digitalpoint]

Sorry, I don't know the nuances of Shakespeare's language well, you mean you wouldn't use IndexNow at all?

Correct. I wouldn't (and don't). Not because it's "bad" or anything, just seems rather pointless since it's for Bing only.

 

[dougdirac]

I recently added a 50reqs/10sec rate limiting rule to abate a fairly aggressive scraping bot originating from China that was causing high server load and normal usage lag/delays. This was much easier, and more effective, than playing 'whack-a-mole' with IP address blocking, when it was persisting from changing IP addresses.

What's the cost associated with that though? I understand CF charges for rate limiting.

 

[rdn]

What's the cost associated with that though? I understand CF charges for rate limiting.

It's free now in all CF plans.

 

[Mouth]

It's free now in all CF plans.

... for 1 rule. If you want additional rules, or more trigger criteria, requires paid plan.

 

[digitalpoint]

digitalpoint updated [DigitalPoint] App for Cloudflare® with a new update entry:

Minor update

• Removed workaround to allow non-Duotone icons in admin navigation for XenForo 2.3 (fixed in XF core)
• Added new Cloudflare setting (under Security): Replace insecure JavaScript libraries
• Changed verbiage to be worded better when setting up API token initially

Read the rest of this update entry...

 

[JoyFreak]

The add-on causes an issue with rebuilding daily stats: https://xenforo.com/community/threads/rebuild-daily-statistics.223552/#post-1698129

 

[Old Nick]

Same here: https://xenforo.com/community/threads/rebuild-daily-stats-1970.223975/

 

[JoyFreak]

Deleting Cache Rules in the addon page throws this error:

Client error: `DELETE https://api.cloudflare.com/client/v4/zones/7c8d4fc8153e104bad38abeb9e797c1e/rulesets/1089a421a79542b1a60258c3d79cea88/rules/0615f1ae4ac14274aab9aef14a51567a` resulted in a `400 Bad Request` response: { "result": null, "success": false, "errors": [ { "message": "invalid JSON: '' cannot be a array" } (truncated...) / {"result":null,"success":false,"errors":[{"message":"invalid JSON: '' cannot be a array"}],"messages":null}

 

[digitalpoint]

The add-on causes an issue with rebuilding daily stats: https://xenforo.com/community/threads/rebuild-daily-statistics.223552/#post-1698129

Deleting Cache Rules in the addon page throws this error:

Client error: `DELETE https://api.cloudflare.com/client/v4/zones/7c8d4fc8153e104bad38abeb9e797c1e/rulesets/1089a421a79542b1a60258c3d79cea88/rules/0615f1ae4ac14274aab9aef14a51567a` resulted in a `400 Bad Request` response: { "result": null, "success": false, "errors": [ { "message": "invalid JSON: '' cannot be a array" } (truncated...) / {"result":null,"success":false,"errors":[{"message":"invalid JSON: '' cannot be a array"}],"messages":null}

Probably not a lot that can be done about a full rebuild being slow. There are 11 API calls being made (one for each stat that is logged). Most are run in parallel to speed it up, but let's say it takes 1 second to make the 11 API calls...

Realistically, it will probably be more than 1 second, but even if it was only 1 second, the API calls would cumulatively take more than 6 minutes per year of the rebuild.

The only solution would be to not make the API calls for each day of the rebuild (the easiest way to do that would be to disable the addon before a full rebuild). Certainly could do that automatically and never fetch stats for a full rebuild, but it seems like it probably makes sense to let people get the data if they want even if it's not lightning speed.

Deleting Cache Rules in the addon page throws this error:

Client error: `DELETE https://api.cloudflare.com/client/v4/zones/7c8d4fc8153e104bad38abeb9e797c1e/rulesets/1089a421a79542b1a60258c3d79cea88/rules/0615f1ae4ac14274aab9aef14a51567a` resulted in a `400 Bad Request` response: { "result": null, "success": false, "errors": [ { "message": "invalid JSON: '' cannot be a array" } (truncated...) / {"result":null,"success":false,"errors":[{"message":"invalid JSON: '' cannot be a array"}],"messages":null}

Ya, looks like they changed the API call for cache delete to not want anything in the JSON "body".

Fixed for the next version, but you can also do this if you want the fix now...

In the DigitalPoint\Cloudflare\Api\Cloudflare.php file, change this:

return $this->makeRequest('DELETE', sprintf('zones/%s/rulesets/%s/rules/%s', $zoneId, $rulesetId, $identifier), ['json' => []]);

to this:

return $this->makeRequest('DELETE', sprintf('zones/%s/rulesets/%s/rules/%s', $zoneId, $rulesetId, $identifier));

 

[Old Nick]

Same here: https://xenforo.com/community/threads/rebuild-daily-stats-1970.223975/

What about this bug?
Same answer?
Thanks.