From: Chris Lee 
Yes, TSE 2.5 returns different results than 2.8, but I get 4 for the last 
test, not 3.  Here's the output:
Number of tokens: 3
1 How
2 many
3 tokens?
Number of tokens: 6
1 How
2 many
3
4
5
6 tokens?
Number of tokens: 4
1 How
2 many
3
4 tokens?
-----Original Message-----
From:	Ross_Boyd@tnt.com.au [SMTP:Ross_Boyd@tnt.com.au]
Sent:	Monday, December 15, 1997 2:36 PM
To:	tsepro@semware.com
Subject:	RE: GetToken() Bug??
Hi guys,
I've been following this thread with some interest.
I hate to complicate things, but the fact is, using Pieter's macro in 
TSE 2.5 and 2.8
produces _different_ results.In 2.8 the third test produces 3 rather than 5.
See for yourselves.
For my money, Pieter probably needs to write a custom GetToken() function 
hat
applies the same rules to all delimiters.
Cheers,
Ross

----------
From: chrisant
To: kcarr; tsepro
Subject: RE: GetToken() Bug??
Date: Monday, 15 December 1997 13:46
i think what Kevin means is that for most compilers, anywhere in source
code where you can put one whitespace character, the compiler treats one
whitespace the same as a zillion whitespace: the number is irrelevent.
this is a common practice in general for text files, not just for
programming.
the GetToken command does treat whitespace and non-whitespace characters
differently, to mimic this.  this is a very useful feature.  GetToken
also understand quotes.  if you use GetToken('foo "a b c" bar', ' "', 2)
the result is not 'a', it is 'a b c'.  that is, the double quotes are
stripped, and everything between them is returned.
// chris
-----Original Message-----
From: Kevin A. Carr [mailto:kcarr@treb.com]
Sent: Monday, December 15, 1997 4:30 AM
To: 'tsepro@semware.com'
Subject: RE: GetToken() Bug??
Most compilers, SAL including, will compress any consecutive whitespace
(usually the tab and space) characters into one.  When using GetToken()
where you can specify token delimiters that are not whitespace (i.e. the
';') those are not compressed, because many times consecutive delimiters
like those are use to indicate and empty field.
If you wish to ignore those empty fields, then simply check the string
length of your token, and ignore it if is is 0.
Kevin A. Carr
----------
From:  Pieter de Ruiter[SMTP:ruiter@xs4all.nl]
Sent:  Sunday, December 14, 1997 4:00 PM
To:  tsepro@semware.com
Subject:  GetToken() Bug??
Hi,
Using GetToken() in SAL (TSE 2.5) I detected a unexpected effect.
Can anybody explain the output of this test-macro?
In my opinion the output of each mTestToken() call should be equal (as
the
first call)!
The GetToken() function seem to treat a non-blank character different
from a
blank.
Can't figure out what I'm doing wrong...
------------------------------------------------------------------------
----
------------------
proc mTestToken(string sSearch, string sDelim)
    integer i, nNum = NumTokens(sSearch,sDelim )
    addline("Number of tokens: " + str(nNum))
    for i=1 to nNum
        addline(str(i) + " " + GetToken(sSearch, sDelim,i))
    endfor
end
proc mTest()
    mTestToken("How many    tokens?", " ;")
    mTestToken("How many;;;;tokens?", " ;")
    mTestToken("How many ;  tokens?", " ;")
end
             mTest()
------------------------------------------------------------------------
----
------------------
Output:
-------
Number of tokens: 3
1 How
2 many
3 tokens?
Number of tokens: 6
1 How
2 many
3
4
5
6 tokens?
Number of tokens: 5
1 How
2 many
3
4
5 tokens?
------------------------------------------------------------------------
----
------------------
Thanks
Pieter de Ruiter,
Utrecht, Nederland
---
---------------