Hello Janis!

21 Aug 2010 11:58, Janis Kracht wrote to Sean Dennis:

 JK> with anything.. so it's not like google-bot couldn't do it that way 
 JK> if they wanted to!

time to get a mirror :)

but:

   ----- .htaccess begins -----
RewriteEngine on

# Allow only GET and POST verbs
RewriteCond %{REQUEST_METHOD} !^(GET|POST)$ [NC,OR]

# Ban Typical Vulnerability Scanners and others
# Kick out Script Kiddies
RewriteCond %{HTTP_USER_AGENT} ^(java|curl|wget).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT}
^.*(libwww-perl|curl|wget|python|nikto|wkito|pikto|scan|acunetix).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT}
^.*(winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner).
* [NC,OR]

# Ban Search Engines, Crawlers to your administrative panel
# No reasons to access from bots
# Ultimately Better than the useless robots.txt
# Did google respect robots.txt?
# Try google: intitle:phpMyAdmin intext:"Welcome to phpMyAdmin
*.*.*" intext:"Log in" -wiki -forum -forums -questions
intext:"Cookies must be enabled"
RewriteCond %{HTTP_USER_AGENT}
^.*(AdsBot-Google|ia_archiver|Scooter|Ask.Jeeves|Baiduspider|Exabot|FAST.Enter
prise.Crawler|FAST-WebCrawler|www\.neomo\.de|Gigabot|Mediapartners-Google|Goog
le.Desktop|Feedfetcher-Google|Googlebot|heise-IT-Markt-Crawler|heritrix|ibm.co
m\cs/crawler|ICCrawler|ichiro|MJ12bot|MetagerBot|msnbot-NewsBlogs|msnbot|msnbo
t-media|NG-Search|lucene.apache.org|NutchCVS|OmniExplorer_Bot|online.link.vali
dator|psbot0|Seekbot|Sensis.Web.Crawler|SEO.search.Crawler|Seoma.\[SEO.Crawler
\]|SEOsearch|Snappy|www.urltrends.com|www.tkl.iis.u-tokyo.ac.jp/~crawler|Synoo
Bot|crawleradmin.t-info{at}telekom.de|TurnitinBot|voyager|W3.SiteSearch.Crawler|W
3C-checklink|W3C_Validator|www.WISEnutbot.com|yacybot|Yahoo-MMCrawler|Yahoo\!.
DE.Slurp|Yahoo\!.Slurp|YahooSeeker).* [NC]
RewriteRule .* - [F]

   ----- .htaccess ends -----

if bbbs http server cant do this make it as proxy in apache !
 

 Regards Benny

... there can only be one way of life, and it works :)

--- Msged/LNX 6.2.0 (Linux/2.6.34-gentoo-r6 (i686))