Cuil Bots Misbehaving

Posted on September 2nd, 2008 by admin in Domain News | No Comments »

Cuil, the search engine that claimed they will be the new Google, is getting a lot of bad press lately.

The folks at The Admin Zone forums have been talking up a storm about the mis-behaving Cuil bot Twiceler.

Twiceler has been rampaging on one of the sites I administer. It leeched enormous amounts of bandwidth, nearly 2Gb this month until it was blocked. (It visited nearly 70,000 times!). Twicler does not obey normal robot txt commands and can only be blocked by denying access in the .htaccess file. (Be careful with this file, and back up the exisiting code for it).

Twiceler caused major bandwidth leeching on numerous sites one of my sons adminsters. In one case it blocked a site by using up all the bandwidth.

There are even posts on how to block Twiceler from “indexing” your site.

Still, some people are unaware of the problem and are literally inviting Twiceler to index their sites.

I wonder how this will end.