Blocking Web Robots
From MyWiki
MSNBot is inefficient, using excessive amounts of server and network resources (which I am not quantifying here in any objective manner).
To disable it, add a robots.txt file to your httpd DocumentRoot (e.g. /var/www/html), and add the following lines:
User-agent: MSNBot Disallow: /
Notes
- The "User-agent" is not case sensitive
- There will be a delay in when this changes becomes effective, since the crawlers check for this only periodically.
