In such a case I would do three things:
1. Contact the hoster of this other website to get the site removed due to copyright infriction and content theft.
2. Find out which of your members copied the content and throw them out
3. Implement techical measures to discover early when this happened again
(4. add a paragraph in the TOS that explicitely states an excessive sum that s/o who does that will have to pay to you)
I agree, and we're already working on some of this.
The problem is that most of these solutions are after the fact. We might get one copied site removed, but how do we know there aren't others?
As for banning the account, users create a separate account for the AI to use. By the time we catch it, the content is already copied.
That's why I'm trying to prevent the mass scraping in the first place, rather than deal with the damage afterward.
(Again, I can't protect screenshots, or someone Ctrl+C a whole page and ctrl V, but the option of copying a whole site, with a single message to a WhatsApp bot, is something new)
OMG many replies here with users don't get what it is you are up against.
Check outthis
https://xenforo.com/community/resources/xf-bot-guard.10508/
It works, and hopefully the developer will release a much better more advnaced version soon as he promised a few months ago
This together with advanced cloudflare logs and filtering can stop this at least for now. We have been fighting with it for a 6 months or more.. contacting hosts can be a solution if it is not hosted somewhere where they could care less about disputes DMCA etc.
Thanks! I'm actually already using XF Bot Guard.
I tested it with Tab AI using a regular logged-in account. It opened 10 threads in 8 seconds. Bot Guard recorded all the requests, but they scored only 30, below the challenge threshold of 50.
It was in observe-only mode, but based on those scores, it wouldn't have challenged the requests anyway.
As
@smallwheels pointed out, the issue is that these AI agents use real browsers, pass the checks, and can keep browsing with an existing session.
And unfortunately, I'm on XenForo Cloud using their community.forum domain, so I don't have control over the Cloudflare settings.
Use a DNS firewall, such as Cloudflare. Logged in, logged out, doesn't matter, you can catch the majority of bots with an accurate header rule.
AI replication, does not replicate your site. It attempts to replicate the public facing pages. It cannot walk your server side folders or database. It can though create static HTML pages from what you have publicly facing.
A client was having trouble with their dev, so I showed them how easily they can recreate their website with AI.
Original:
https://www.spaceworks.com.au/
AI 1hr:
https://spaceworks.anthonyparsons.workers.dev/
This was a couple months back, so I think they made some changes since then, but it got it about 98% right and just needed a few tweaks. They were shocked, to say the least, yet relieved, because the dev was withholding their website from them and this changed the negotiation very quickly.
Again, that is only a static replication, which is enough in this case as there is no user interaction with the site.
How do you defend against that? A simple block on Claude would have stopped it, along with any other AI bot running around the web. Just a switch at cloudflare.
I understand it doesn't actually copy the database or server files, but that's not really what concerns me.
In our case, the actual threads and posts were copied word for word onto another site. It doesn't need database access for that, just access to the pages like any regular visitor.
Blocking known AI crawlers makes sense, but would that really stop someone using Tab or another AI agent through a normal Chrome browser, logged in with a regular account?
That's the part I'm struggling to find a solution for. And since I'm on XenForo Cloud with their domain, I can't configure Cloudflare rules myself.
