Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!watmath!clyde!caip!rutgers!sri-spam!nike!ucbcad!ucbvax!ucbarpa.Berkeley.EDU!fair
From: fair@ucbarpa.Berkeley.EDU (Erik E. Fair)
Newsgroups: net.news,net.micro.mac,net.sources.d
Subject: Re: Backbone automatic news-compression question ...
Message-ID: 
Date: Thu, 2-Oct-86 04:31:06 EDT
Article-I.D.: ucbvax.15889
Posted: Thu Oct  2 04:31:06 1986
Date-Received: Sat, 4-Oct-86 05:28:00 EDT
References:  
Sender: usenet@ucbvax.BERKELEY.EDU
Organization: University of California at Berkeley
Lines: 51
Keywords: question [regarding] compression [of] news [for] transmission
Xref: watmath net.news:5260 net.micro.mac:8074 net.sources.d:550

In article  wcs@ho95e.UUCP (Bill Stewart 1-201-949-0705) writes:
> [deleted]
>
>On the subject of compression for transmission, if a backbone site is
>transmitting a given message to 24 other sites, does it compress each
>outgoing message once, or does it do it 24 times?  This is obviously
>moot on a broadcast network (like Stargate, or ihnp4->rest of IH),
>but could have a major CPU impact on an overloaded machine like
>ucbvax, ihnp4, or allegra.  If I were designing "C News", I'd consider
>storing all articles in compressed form, with special pre-defined
>tokens for the common header items; this would have major disk space
>and transmission savings, and would probably have low CPU impact for
>news reading programs.

If you send a single article to 24 sites, yes, it normally would get
compressed and sent 24 times. When I ran "dual", which had seven
"leaf" nodes feeding off of it at its peak, I set up a "pseudo-site"
named "leaf" that generated one batch file, and through the magic of
UNIX links, I sent a single batch to all seven sites. The key is that
the sites that you do this for *must* be

	1. real leaf nodes (that is, they don't get news from anywhere
		else, and generate very little themselves, because this
		scheme will send *everything* that they send to you,
		back to them (in addition to sending to all the other
		leaves)).

	2. all using the same unbatching scheme (that is, they must
		all accept the same format of batch file as input; you
		can vary the command given to uux on a per-site
		basis).

In order to realize maximum savings from this, I had to hack uux to
take one more argument (not hard, the diff of the mod was distributed
with netnews for both v7 & sV UUCPs): the -l (minus el) argument; it is
equivalent to the "-c" option, except that it attempts to make a hard
UNIX link between the input file given to uux, and the queue data file
before trying an explicit "copy". The effect was seven jobs queued with
UUCP, but only one copy of the data file, which was a big win for
/usr/spool/uucp disk space; as a result, the incremental cost to "dual"
of feeding a new "leaf" node was more modem time, and a little more
disk space for C. and X. files (i.e. not much).

As for keeping the articles compressed on the disk, I don't think that
would win much because articles are stored as single files (collecting
articles in a newsgroup together would mean you couldn't link articles
across newsgroups), and the average size of article is pretty small
(last I checked (in 1984, admittedly a long time ago) it was 1600 bytes)
so that the resultant savings (if any) would not be much.

	Erik E. Fair	ucbvax!fair	fair@ucbarpa.berkeley.edu