Keep Bad Robots and Spiders Away
Many so called webbots or web spiders are currently used for many different things on the Internet. Examples include search engines that use them to catalog the Internet, email marketing people that search for email addresses and many more. For a description of such robots check out The Web Robots Faq. (full story)