Hi Benny,

>> with anything.. so it's not like google-bot couldn't do it that way
>> if they wanted to!

> time to get a mirror :)

Hehe.. Yep, very good idea as we discussed.  The more mirrors the better..
That way my system won't get so boggled down :)

> but:

> ----- .htaccess begins -----
> RewriteEngine on

> # Allow only GET and POST verbs
> RewriteCond %{REQUEST_METHOD} !^(GET|POST)$ [NC,OR]

> # Ban Typical Vulnerability Scanners and others
> # Kick out Script Kiddies
> RewriteCond %{HTTP_USER_AGENT} ^(java|curl|wget).* [NC,OR]
>RewriteCond %{HTTP_USER_AGENT} ^.*(libwww-perl|curl|wget|python|nikto|wkito|pi
> to|scan|acunetix).* [NC,OR]
>RewriteCond %{HTTP_USER_AGENT} ^.*(winhttp|HTTrack|clshttp|archiver|loader|ema
> l|harvest|extract|grab|miner). * [NC,OR]

> # Ban Search Engines, Crawlers to your administrative panel
> # No reasons to access from bots
> # Ultimately Better than the useless robots.txt
> # Did google respect robots.txt?
> # Try google: intitle:phpMyAdmin intext:"Welcome to phpMyAdmin *.*.*"
> intext:"Log in" -wiki -forum -forums -questions
intext:"Cookies must be
> enabled"
>RewriteCond %{HTTP_USER_AGENT} ^.*(AdsBot-Google|ia_archiver|Scooter|Ask.Jeeve
>|Baiduspider|Exabot|FAST.Enter prise.Crawler|FAST-WebCrawler|www\.neomo\.de|Gi
>abot|Mediapartners-Google|Goog le.Desktop|Feedfetcher-Google|Googlebot|heise-I
>-Markt-Crawler|heritrix|ibm.co m\cs/crawler|ICCrawler|ichiro|MJ12bot|MetagerBo
>|msnbot-NewsBlogs|msnbot|msnbo t-media|NG-Search|lucene.apache.org|NutchCVS|Om
>iExplorer_Bot|online.link.vali dator|psbot0|Seekbot|Sensis.Web.Crawler|SEO.sea
>ch.Crawler|Seoma.\[SEO.Crawler \]|SEOsearch|Snappy|www.urltrends.com|www.tkl.i
>s.u-tokyo.ac.jp/~crawler|Synoo Bot|crawleradmin.t-info{at}telekom.de|TurnitinBot|
>oyager|W3.SiteSearch.Crawler|W 3C-checklink|W3C_Validator|www.WISEnutbot.com|y
> cybot|Yahoo-MMCrawler|Yahoo\!. DE.Slurp|Yahoo\!.Slurp|YahooSeeker).* [NC]
> RewriteRule .* - [F]

> ----- .htaccess ends -----

> if bbbs http server cant do this make it as proxy in apache !

Sure, it can .. :)  I have always used mod-rewrite to ban filehogs from
places like those where I find companies that are selling the files
.

I like the way you used it though :)  Thanks for a good idea.

Take care,
Janis

--- BBBS/LiI v4.01 Flag