Personally, I don't have an issue with bot scraping. I don't care what they use it for. If you put it online, publicly available, then its a free for all. If you let Google have it, you may as well let anyone have it, its public after all. If you want private stuff online, then make it private, no such issues with bot scraping.
Saying that... I don't like the intensity, the abusive manner of scraping where it affects and impacts peoples server resources making human visitors interaction poor, slow loading, crashes, etc. I have demonstrated that you can run a multi-million post forum on about $15 a month server, BUT, greedy bots will make human interaction crap because they don't want to limit themselves. Honestly, Microsoft is one of the biggest offenders, Facebook too. Why does Facebook need to aggressively scrape the web to provide social media services? Why are these companies being so aggressive and pissing people off in the process? Why are people just farming for no real intention, other than to be annoying or try and build some personal library for their own experimentation?
A real simple solution would be a global infrastructure limit on all automated detected activity, 1 visit per domain, per 15 seconds. That would come close to solving the entire internet if handled at the tier level infrastructure, anyone not playing fair gets spanked, even those pesky invaders of tv apps using infrastructure unknowingly to most consumers. Big business out there can fingerprint bots, but everyone is charging big bucks for the good stuff, instead of rolling it out freely to actually stop the problem.
It was like using ProxyCheck, I got the impression that the more I queried them, the traffic kept growing, hitting the next limit, requiring upgrade, more $$$. Maybe I'm paranoid... maybe its all a ploy to start paying money for things you don't need, but someone is forcing it upon the web nonetheless. After seeing Anubis in action from @BrettC on my own test server, I think server companies need to be rolling that stuff out network wide, removing the issue from all customers in one swoop.
Saying that... I don't like the intensity, the abusive manner of scraping where it affects and impacts peoples server resources making human visitors interaction poor, slow loading, crashes, etc. I have demonstrated that you can run a multi-million post forum on about $15 a month server, BUT, greedy bots will make human interaction crap because they don't want to limit themselves. Honestly, Microsoft is one of the biggest offenders, Facebook too. Why does Facebook need to aggressively scrape the web to provide social media services? Why are these companies being so aggressive and pissing people off in the process? Why are people just farming for no real intention, other than to be annoying or try and build some personal library for their own experimentation?
A real simple solution would be a global infrastructure limit on all automated detected activity, 1 visit per domain, per 15 seconds. That would come close to solving the entire internet if handled at the tier level infrastructure, anyone not playing fair gets spanked, even those pesky invaders of tv apps using infrastructure unknowingly to most consumers. Big business out there can fingerprint bots, but everyone is charging big bucks for the good stuff, instead of rolling it out freely to actually stop the problem.
It was like using ProxyCheck, I got the impression that the more I queried them, the traffic kept growing, hitting the next limit, requiring upgrade, more $$$. Maybe I'm paranoid... maybe its all a ploy to start paying money for things you don't need, but someone is forcing it upon the web nonetheless. After seeing Anubis in action from @BrettC on my own test server, I think server companies need to be rolling that stuff out network wide, removing the issue from all customers in one swoop.
Last edited:
