On 2018 Jun 19 22:43:24, you wrote to me:

 >> AFAIK, seenbys and paths are not included in most dupe detection
 >> schemes... other non-changing control lines are fine to be included...
 >> one of the problems comes when some system sort those control lines on
 >> messages they are passing along... we don't see so much of that like we
 >> did at one time ;)

 RS> So some metadata is included in the data that is hashed for dupe
 RS> detection and some is not?

yes...

 RS> Are you sure about that?

yes... in fact, and i don't recall who pointed this out to me back in the
'90s, dbridge does exactly this in a manner of speaking... it takes the
whole message header plus X bytes immediately following the message header
and uses all of that as at least part of the checksum calculation... this
was pointed out to me when i was working on my posting tool and was adding
MSGID support to it...

i was using a library and just letting it do its thing... some of my test
posts were reported as dupes when they clearly weren't... IIRC, they were
detected as dupes because they were posted within the same second... it
turned out that my MSGID was somewhere in the middle of the control lines
at the beginning of the message body and only my dbridge using testers were
seeing this... someone pointed out this thing about dbridge also using X
bytes from the beginning of the message body in addition to the message
header so i moved my posting tool's MSGID to the top of the list and no
more dupes were detected by those dbridge systems...

i don't know what other systems do... there's only a very few that provide
this information... SBBS is one of them... when i was testing Mystic, there
was some discussion about dupe detection as james worked to try to figure
out the best method he liked... i have used fastecho here for decades but i
don't know what data it uses for its checksums... i do know it uses two
checksums, though... i know this because i was being nosy one day and
looking at FE's dupe database file (one for all message areas) with a hex
viewer and noticed that groups of bytes were repeated all throughout the
file... i asked about this and was told i found a bug... basically, FE has
two checksums that it uses for each message and both are supposed to be
stored in the database... what i found was that only one was being used and
written to both fields... toby fixed that problem right quick... i just
don't know what data is used to calculate them...

back in the day, dupe detection formulas were not really shared around...
maybe a couple of developers talking amongst themselves would tell each
other what they were doing but this information was not published where
everyone could find it... it was more or less black majik to a point...

 RS> Anyway, duplicate Message-IDs *should* be caught be any FTN software
 RS> written or updated in the past 20 years.

true... and we still have some systems that don't do MSGID at all so other
methods must be used on them...

)\/(ark

Always Mount a Scratch Monkey
Do you manage your own servers? If you are not running an IDS/IPS yer doin'
it wrong...
... WANTED: assistant to magician in beheading illusion. Blue Cross & salary.
---