KH> I was just wondering, I have been fiddling with
KH> Tobruk 0.31 
KH> and started thinking about Dupe checking, and I was wondering
KH> what the best way of checking for Dupe's would be?..

Personally, I think most current dupe checking is FAR
too gung ho, and that results in false dupe detection,
and dropping of messages which aren actually dupes at all.

I think one crucial thing is to do a rigorous check that a particular pair
really are a dupe, once the quick look flags a candidate pair, and if there
is any doubt, DONT deleted the dupe, coz an occasional mistake like that which
keeps a dupe is FAR better than deleting what isnt actually a dupe at all.

KH> The only idea I have come up with it checking the MSGID: kludge

Its the best candidate, but has some real problems. Its still nothing
like universally used, and there is some potential for accidental reuse
of a particular MSGID too, mostly when changing message creation system.

KH> and if it's equal to another messages MSGID: kludge,
KH> then it will not process it..

Thats fine if you ALSO have a check on the other fields on a pair with
the same MSGID. That doesnt need to be that complex in this particular
case, if all the main header fields also match too once you have a pair
with the same MSGID, its pretty safe to assume its a real dupe.

KH> although this may have to be put in a array or something
KH> like that  and memory gets to be a bit
KH> of a problem for that when your say under DOS and tossing
KH> into a base of about what.. 20000 messages plus..

True, its got some awkwiditys like that. And unfortunately dupe
checking is often most useful on the deeper message bases too.
The most obvious approach with the primitive DOS limitations is
to optimise the approach so that a decent sized cache will help.

--- PQWK202