Alvin63's latest activity

  • A
    Alvin63 reacted to puterfixer's post in the thread Crazy amount of guests with Like Like.
    Of course it will pay peanuts... Onto the intermittent error - if it cannot be consistently reproduced, it might just be "transitory...
  • A
    Alvin63 replied to the thread Crazy amount of guests.
    Pay per crawl is private/beta but you can sign up for early access here https://www.cloudflare.com/paypercrawl-signup/
  • A
    Alvin63 replied to the thread Crazy amount of guests.
    Found it - could be worth setting up yes? Except I can't see the option to do it - maybe only for paid Cloudflare accounts ...
    • AI payment 2.webp
  • A
    Alvin63 replied to the thread Crazy amount of guests.
    This looks interesting from a Cloudflare article - I like the idea of telling an AI crawler they need to pay if they want access...
    • AI payment.webp
  • A
    Alvin63 reacted to smallwheels's post in the thread Crazy amount of guests with Like Like.
    I'd assume that these three are not the same bot. As many of the big techs Amazon has more than one bot, serving different purposes -...
  • A
    Alvin63 reacted to Wildcat Media's post in the thread Crazy amount of guests with Like Like.
    It's a connection error between CF and the origin, so no rules/challenges apply. Basically the origin isn't responding as quickly as CF...
  • A
    Alvin63 replied to the thread Crazy amount of guests.
    Could it be an ASN challenge somewhere?
  • A
    Alvin63 replied to the thread Crazy amount of guests.
    I'm also allowing Amazonbot for that reason. Even though it shows up as an AI crawler in Cloudflare.
  • A
    Alvin63 reacted to Wildcat Media's post in the thread Crazy amount of guests with Like Like.
    Regarding Amazon's bots... I whitelist two of Amazon's bots. We are Amazon affiliates and two of their crawlers might be checking to...
  • A
    Alvin63 replied to the thread Crazy amount of guests.
    I've been debating whether or not to block AI crawlers in the cloudflare AI crawling options. Hadn't done anything as numbers are ok...
  • A
    Alvin63 reacted to Anthony Parsons's post in the thread The Meta Bot is causing insane traffic!! with Like Like.
    Yes, Meta is extremely aggressive, and from my stats, the most aggressive of all the big names by far. It's like Meta doesn't cache...
  • A
    Alvin63 replied to the thread The Meta Bot is causing insane traffic!!.
    I have these ASN's blocked for Singapore and Asia which seem to be the main culprits from that part of the world - note I looked up the...
  • A
    Alvin63 replied to the thread Crazy amount of guests.
    I hadn't really investigated the AI Crawl aspect of Cloudflare before. But looking at the statistics, it looks like ClaudeBot is...
    • Crawlers.webp
  • A
    Alvin63 replied to the thread Crazy amount of guests.
    This article mentioned some changes as from 15 September. Presumably shown in your screenshot already. "What changes on September 15...
  • A
    Alvin63 reacted to Levina's post in the thread Crazy amount of guests with Like Like.
    For your robot.txt you can set preferences in Cloudflare - Security > Settings > Bot Traffic > Configure AI bot policies. Not all bots...
Show older items
Back
Top Bottom