Guy Hoelzer  wrote or quoted:
> Tim Tyler at tim{at}tt1lock.org wrote on 9/9/04 8:25 AM:
> > Guy Hoelzer  wrote or quoted:
> >> in article chnea4$30dh$1{at}darwin.ediacara.org, Tim Tyler at
tim{at}tt1lock.org

> >>> Shannon's information has proved to be a remarkably
useful construct -
> >>> AFAICS.
> >> 
> >> It has indeed.  Indeed, I often rely on it myself.  However,
my use of the
> >> Shannon/Weaver information measure (-p_i x log p_i,) and most
usage in the
> >> literature utterly ignores the role of perception because it has no
> >> representation in the equation.  In fact, the equation is
solely about the
> >> degree of structure in the data.
> > 
> > p_i is the probability of symbol i arising at that point.
> 
> I have no evidence to offer regarding the actual frequency with which this
> definition of p_i is used, but I do not use it to reference a
"probability"
> per se, and I am certain that many others use it as I do.  We treat it as a
> frequency, which is objectively estimable from data.

In general it is not a frequency.  Probabilities are different from
frequencies.  The frequency of something is an average the number of
times it occurs over a period of time.  Probabilities can be much more 
context sensitive.

The probability of some event occurring can be a function of all the
preceding events, and can include things such as extrapolation - rather 
than having anything to do with symbol frequencies.

One example: the frequency with which "u" is seen in english text
is one thing - but the probability with which it occurs *if* the
last symbol was a letter "q" is something else entirely.

Another example: what comes next:

1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19...?

Extrapolation and probability can be invoked to answer that - but 
symbol frequencies tell you next to nothing - except perhaps than
what comes next might contain a "1".

Probabilities are not really the same as frequencies - and it is 
probabilities - not frequencies - that are used in Shannon's equation.

Probabilities can be estimated from data - but different agents may
have access to different data.  Also, their estimates may depend on
factors such as their intelligence and experience.

There can be no consensus about probabilities - except perhaps by
God - whose probability estimates are either 1.0 or 0.0 - since
presumably he knows what's coming next.

Probabilities are a reflection of the observer's ingorance about
the state of the system - and their uncertainty about its behaviour.

In general - different observers can have different degrees of ignorance.

> In this way, the sensitivity of p_i to individual differences in 
> perception is minimized.
> This transition in the meaning of p_i happened without much notice for many
> (most? The vast majority?) information scientists because it increased the
> extent of objectivity and measurement agreement among scientists, and
> extended the utility of information theory throughout the physical sciences.

Nothing fitting this description ever happened.

> > Different observers will make different estimates of that -
> > based on their past experiences.
> 
> If that were the case, then it would be useless for someone to publish their
> estimates of p_i, except as comparative indices of perception differences,
> because the correct value would be different for each reader.

Whether comparing notes is useful depends on who is measuring what.

If the observers are in similar positions - and have similar levels of
ignorance about the system - comparing notes might result in a good
match.

However, if the observers have wildly different levels of knowledge about 
the system, comparing notes will result in wildly different estimates of
symbol probabilities.

> > An observer who knows what symbol is coming next (because he
> > has seen a very similar message before) will assign different
> > probabilites to the symbols from an observer who is seeing
> > the message for the first time - and both will assign different
> > probabilities from an observer who is not even familiar with
> > the language the message is written in.
> 
> This is a nice description of the (severe IMHO) limitations of the
> "telegraph"-context-laden version of the theory that Shannon
originally
> devised for his telegraph-company employer.  With all of your protestation
> about my lack of fidelity to Shannon's original context, you haven't
> suggested any reasons why treating p_i as a frequency, rather than a
> probability, is problematic.  Can you think of any such problems? [...]

The ones above?

p_i can only be treated as a frequency, *if* the source is something like 
a RNG - where the probability of symbols being emitted is constant - and 
does not depend on the history of the stream or environmental conditions.  

That is certainly not the general case - and it is not the case with many 
common sorts of message streams either.
 
> If not, then don't you think it is worth considering the more extensive 
> version of the theory?

It isn't "more extensive".
-- 
__________
 |im |yler  http://timtyler.org/  tim{at}tt1lock.org  Remove lock to reply.
---
þ RIMEGate(tm)/RGXPost V1.14 at BBSWORLD * Info{at}bbsworld.com

---
 * RIMEGate(tm)V10.2áÿ* RelayNet(tm) NNTP Gateway * MoonDog BBS
 * RgateImp.MoonDog.BBS at 9/10/04 5:07:16 PM