Covert Crawler Descends on Web

Websites get looked at by two different kinds of visitors: the human ones who peer around, look at the graphics, think about the links and click slowly; and the spiders, those automated scanners that come in from search engines like Google, or, more ominously, from malicious attackers, competing businesses and spammers looking for e-mail addresses.

Fortunately, it has always been pretty easy to tell the difference between the two in server logs, and block unwanted or anti-social crawlers. But research presented at the Shmoo Con hacker conference here Friday may change that.

By Quinn Norton

Full Story: http://wired.com/news/technology/0,70016-0.html?tw=wn_tophead_1