MSNBot - You’re now blocked

Posted in Randomness by Will on January 19, 2007.

Well, MSNBot’s activities havn’t improved any - it’s still a giant bandwidth hog.

I’ve now added MSNBot and the SearchPreview bot to robots.txt with “deny: /” (i.e: everything). December was relatively light, with
“only” 173MB of bandwidth used - by a bot.

However, so far this month, it’s used another 471MB. And I can tell you I haven’t added that much content!

It’s not so much the bandwidth usage that I object to - it’s that they’ve either got an inefficient or greedy bot.