FM> Of course it is. Is it possible, in general, to scan a PKT and work out
FM> where to put SOT/EOT? 

No.

FM> If not why not, what stuffs you up from doing that?

The below...

 SOT/EOT rationale - originally written 1995-05-25 by Paul Edwards
 This document is released to the public domain.
 Last update of this line: 1995-06-26
 
FSC-XXXX.000, available for FREQ from 3:711/934, contains the
spec for the SOT/EOT kludges.  But since that is a spec, it
does not contain a lot of the background information that explains 
why these kludges were introduced. 


The problems that SOT/EOT attempts to solve
+++++++++++++++++++++++++++++++++++++++++++

The largest problem is that there is not a clear distinction
between what is user-text and what is control information.  When
FTS-1 was created, there was indeed a clear separation, with a
fixed header (control information), kludge lines (control 
information clearly identifiable by "^A" at the beginning of a
line, and the rest is user-text.  However this was completely
stuffed up by FTS-4 and it's introduction of control lines that
were not distinguishable from user-text, they were just plonked
right in the middle of it.  It is far, far too late to remedy
that gross amateurism, but we can endeavour to help heal the
wounds.  The control lines that introduced the problem are
"AREA", "SEENBY", "* Origin" and
"---".  Also some people have
seen fit to exacerbate the problem with control lines such as
"# Origin".

Other problems are:

1. FTS-4 does not make it clear whether origin lines are
mandatory for echomail.  Ditto for tear lines.

2. FTS-4 does not say whether SEENBY lines will be contiguous.

3. FTS-4 does not say whether there can be other control lines
intermingled with the FTS-4 control lines.

4. FTS-4 does not specify the order of tear, origin, SEENBY etc
generation in the message.

5. FTS-4 does not specify the format of the address in the
origin line.

6. FTS-1 mentions x'8d' when it is totally irrelevant to the
transport layer, and has been interpreted by some people to mean
that x'8d' is allowed to be deleted on in-transit mail.  If it
mentions it at all, it should be "people used to use x'8d' as
a formatting character, it is not required, and this practice
causes problems so do not use it for this purpose in new
products".

7. x'8d' is used for both soft-CRs and as a national character
by various countries, including Russia and Japan, and there is
no way to tell which is which.

8. FTS-1 does not give any direction to let new programmers know
that the Fido date format is the dominant date format, and should
be favoured when creating new messages.

9. FTS-1 has ambiguous wording which some people have interpreted 
to mean that only the PRODUCTS Fido and SEAdog need to conform to
the date spec.

10. FTS-1 does not specify any maximum length for kludge lines.
This means that every man and his dog defines an artibrary 
maximum (and no, it isn't 10 gigabytes) when they write their
program.

11. FTS-4 does not specify a maximum length for the AREA, 
SEENBY, tear and origin lines.

12. The FTSC does not appear to believe that unambiguous specs
should be the desired output of a standards committee, or in
fact, that there should be any output at all from a standards
committee.  

13. FTS-1 says that keywords should have a colon after them, and
then lists examples which don't.  It is unclear what is correct
usage for both those kludges, and other kludges.

14. FTS-4 does not specify whether it is compulsory to have some
text after the "---" of the tearline.

15. FTS-4 does not specify that the address in the origin line 
must be contained in "()".

16. FTS-4 does not mention that the addresses in the SEENBY and
PATH are always 2D, and that the SEENBYs are stripped on a zone
boundary.

17. FTS-1 does not say that all the kludges at the top will be
consecutive, e.g. are you meant to be looking for the INTL
kludge after the user-text?


Signs of sickness
+++++++++++++++++

This lack of a clear distinction between control information and
user text manifests itself whenever you try to use these control
words as part of your user text.  E.g. Golded will hide a line
where it recognizes an AREA kludge, even if it is in the middle
of some text.  When quoting a message from someone else, the
tearline and origin line are included as part of the quote, and
the tearline is changed from "---" to "-+-" in an attempt to stop
OTHER software from truncating the message on the first "---"
that is found!!!  You will notice that if you type "+++" and then
quote that, golded will not touch it.  What's so special about
"---" that they don't want the user to enter it?  After all, it's
just NORMAL ASCII CHARACTERS!  "---" could be the start of some
Ada or SQL comments, especially if you are posting an an Ada or
SQL echo!  

None of this is golded's problem of course, it is merely making 
an effort to help software, because software has so much trouble 
trying to tell whether a "---" is a bit of user text or if it is 
a control line, because there is NO CLEAR DISTINCTION.  Which is
in stark contrast to kludge lines which start with a control
character, that can never be confused with NORMAL ASCII TEXT.
It is also in stark contrast to lines that start "+++", because
there is no confusion over whether that is user-text or a control
line, because there is no control line defined that starts "+++"!

The end user should never have to know what control lines are used
by the transport mechanism - to require them to do so is quite
obviously a sign of shoddy design.  Which is why it is distressing
to see the following comment from the author of MM707.ZIP, a
cooking program!  Some poor guy writes a program to deal with
cooking, and he finds that he has to learn about the FTS specs
in order to make his program "fido friendly"!  What a disgrace!

  "A new export format has been added which eliminates the 
  hyphens in the header and footer; this provides compatibility 
  with BBS systems that could not use hyphens."

I think we should all hang our heads in shame that we make authors
of recipe programs have to learn about mail standards because we
designed a system that is so shoddy that the poor guy can't even
transmit hyphens in peace.  SHEESH! 


The current algorithms
++++++++++++++++++++++

To determine the rogue kludge lines in a message, the best
algorithm we can come up with is this:

AREA - look on the very first line, ie don't skip any ^A kludges
to look for an AREA line, just look at the VERY first character,
and if you find "AREA", then you've got an AREA kludge.  You hope
and pray that this is not a netmail message with no ^A kludges at
all, because if it IS a netmail message with no ^A kludges, then
you will be inspecting USER TEXT to see if the message starts
"AREA".  So you hope and pray that the user hasn't entered some
text that starts off with "AREA".  Someone in NET_DEV reported
that that's exactly what happened on his system.  Regardless, any
algorithm that is in any way dependent on what a user enters, is
a very amateur algorithm.  What we really need is a kludge, ANY
kludge, that will stop the algorithm from having to look at the
user text.  It would be feasible to say that INTL kludge should
always be generated, and this would block off that problem.  
^ASOT also does the job.  BTW, here is a quote from someone...

*********************************************************************
Date:   1995-11-22 13:09:08
From:   tom schlangen
To:     Paul Edwards
Subj:   sot/eot rationale
Attr:   recvd 
Area:   NET_DEV                       

Hi Paul,

* Paul Edwards -> HECTOR SANTOS:

 > So you hope and pray that the user hasn't entered some
 > text that starts off with "AREA".  Someone in NET_DEV reported
 > that that's exactly what happened on his system.

    happened to me, too:

i used a tick/announcing prog to generate _netmails_ to notify some downlinks 
about new files, and not thinking about the implications i made an announcing 
template which began with:

Area:  C program out, but alas, this is the exception, not 
the norm in fidonet today!


Why *some* old fido dogs don't like new tricks
++++++++++++++++++++++++++++++++++++++++++++++

As the inventors of the original "hope and pray" abortion of
kludges without ^A prefixes, some people don't like having the
gaping hole in the transport mechanism being pointed out to 
them.  Presumably they think it makes them look silly for not
having realised for the last x years that telling users not to
enter "---" lest it interfere with mailprocessing software is
the sign of a very poorly design transport mechanism, not the
sign of stupid users.  Besides, if they didn't think of it, it
can't be a very good idea, can it?  I really don't know what
goes through their little minds.  Personally I am trying to 
make the current transport mechanism more robust, while they
are pushing the virtues of "hope and pray".  As they say, "it
takes all sorts".
@EOT:

---