http://jryj.com my daily saved bookmarks & microblog, links and resources Fri, 02 Jan 2009 15:24:16 +0000 http://wordpress.org/?v=2.7 en hourly 1 http://jryj.com/keep-bad-robots-and-spiders-away-218/ http://jryj.com/keep-bad-robots-and-spiders-away-218/#comments Thu, 14 Aug 2008 13:12:21 +0000 admin http://jryj.com/keep-bad-robots-and-spiders-away-218/ Many so called webbots or web spiders are currently used for many different things on the Internet. Examples include search engines that use them to catalog the Internet, email marketing people that search for email addresses and many more. For a description of such robots check out The Web Robots Faq. (full story)

]]> http://jryj.com/keep-bad-robots-and-spiders-away-218/feed/