Obituary: Farewell to robots.txt (1994-2025)

[Rusty Snippets]

It is with deep sorrow that we announce the end of robots.txt, the humble text file that served as the silent guardian of digital civility for thirty years. Born on February 1, 1994, out of necessity when Martijn Koster’s server crashed under a faulty crawler named “Websnarf,” robots.txt passed away in July 2025, not by Cloudflare’s hand, but from the consequences of systematic disregard by AI corporations. Cloudflare’s decision to block AI crawlers by default merely marked the moment when even the last major infrastructure provider abandoned faith in voluntary compliance and moved to technical enforcement. This was a last act of desperation that signaled the end of an era. As with all significant losses, it took time for the full extent of this digital tragedy to be comprehended.

A Life of Silent Service​

robots.txt was born in a time when the internet resembled a small, quiet....................

Obituary: Farewell to robots.txt (1994-2025)

The voluntary compliance protocol that civilized the internet has departed, bids Henning Fries farewell.

www.heise.de

 

[smallwheels]

Brilliant article. Best sentence:

When robots.txt was born, the internet was a place of hope and collaboration. It died in a world where billion-dollar companies systematically steal the work of content creators while claiming to democratize knowledge.

I guess many in this forum are old enough to have seen the whole journey.

 

[Rusty Snippets]

I was latecomer, got my first computer in 1995. But it was still fairly the wild west on the Internet. I miss those days.