From: "Kevin A. Carr" 
Most compilers, SAL including, will compress any consecutive whitespace =
(usually the tab and space) characters into one.  When using GetToken() =
where you can specify token delimiters that are not whitespace (i.e. the =
';') those are not compressed, because many times consecutive delimiters =
like those are use to indicate and empty field.
If you wish to ignore those empty fields, then simply check the string =
length of your token, and ignore it if is is 0.
Kevin A. Carr
----------
From:  Pieter de Ruiter[SMTP:ruiter@xs4all.nl]
Sent:  Sunday, December 14, 1997 4:00 PM
To:  tsepro@semware.com
Subject:  GetToken() Bug??
Hi,
Using GetToken() in SAL (TSE 2.5) I detected a unexpected effect.
Can anybody explain the output of this test-macro?=20
In my opinion the output of each mTestToken() call should be equal (as =
the
first call)!=20
The GetToken() function seem to treat a non-blank character different =
from a
blank.
Can't figure out what I'm doing wrong...
-------------------------------------------------------------------------=
---
------------------
proc mTestToken(string sSearch, string sDelim)
    integer i, nNum =3D NumTokens(sSearch,sDelim )
    addline("Number of tokens: " + str(nNum))
    for i=3D1 to nNum
        addline(str(i) + " " + GetToken(sSearch, sDelim,i))
    endfor
end
proc mTest()
    mTestToken("How many    tokens?", " ;")
    mTestToken("How many;;;;tokens?", " ;")
    mTestToken("How many ;  tokens?", " ;")
end
             mTest()
-------------------------------------------------------------------------=
---
------------------
Output:
-------
Number of tokens: 3
1 How
2 many
3 tokens?
Number of tokens: 6
1 How
2 many
3
4
5
6 tokens?
Number of tokens: 5
1 How
2 many
3
4
5 tokens?
-------------------------------------------------------------------------=
---
------------------
Thanks
Pieter de Ruiter,
Utrecht, Nederland
---
---------------