# # This little bit of code tries to keep parts # of a site not indexed by spiders. # The spiders are screened by their User-Agent # strings. Instead a default message is sent. # BrowserMatchNoCase http spider=1 BrowserMatchNoCase bot spider=1 BrowserMatchNoCase crawler spider=1 BrowserMatchNoCase spider spider=1 BrowserMatch Jeeves)$ spider=1 BrowserMatch ^Bilbo spider=1 BrowserMatch ^websec spider=1 BrowserMatch ^Scooter spider=1 BrowserMatch ^Mercator spider=1 BrowserMatch ^Java1 spider=1 BrowserMatch ^$ spider=1 RewriteEngine On RewriteCond %{ENV:spider} ^1$ RewriteRule .* /spider.html #Deny from env=spider