yes, I agree with you!
robots.txt works, but not as expected, google bot can ignore it & crawl unwanted areas, when bot finds out duplicate content, it removes tons of links out of index, even 1 block rules can hurt the whole web. I had a web that government were blocking every week, I had to...