Original from  IVAN TODOROSKI  to DENIS TONN on 01-04-1999
Original Subject: Some weird addressing...

                         ---------------------------------------

 Between being busy and a mail feed interruption, it's been a while 
since I have looked at these discussions. Apologies for a late (and 
possibly inappropriate) response. 


 IT>>   Hey, what's all this 64 KB allocation stuff?!
 
  DT>  It's not 64K allocation, it's allocation units occuring on 64K
  DT> boundries.
 
IT>   Aaahhhh... OK! This single sentence explained it for me! What was I
IT>   thinking?! So DosAllocMem (in addition to setting up the page tables
IT>   appropriately and returning a single offset in the 512MB descriptor)
IT>   just creates an aditional set of descriptors in the LDT for every
IT>   allocation, which just sit there unused until there is need to make
IT>   them available to a 16 bit process which understands only 16:16
IT>   addresses. Is this what "thunking" is?
IT> 
IT>   And does the word "thunk" has any actual meaning in
English? I don't
IT>   have the dictionary handy to check it out (it went with the fallen
IT>   drive)

 Not quite.. This is "tiling of the LDT", which is used to make 
"thunking" faster/easier. A "Thunk" (in the above
context) is the 
process of changing an address from one form to another, while still 
referencing the same memory location. 

 IT>>   I thought that was gone?
 
  DT>  It is. A 32 bit program uses a "flat" selector with a
base of zero
  DT> and a limit of 512MB (448MB for Warp 3 GA).
 
 IT>>    Do you get a single descritor with a 1000000 byte limit, or a bunch
 IT>>    of small descriptors with 65536 limits?
 
  DT>  The 32 bit program uses 3 "flat" 32 bit selectors (CS,
DS, and ES),
  DT> all with a base of zero and a limit of 512MB. A 32 bit application can
  DT> ignore selectors. It is ONLY when it needs to pass an address to a
  DT> 16:16 piece of code that the tiling of the LDT comes into play...
 
IT>   And here you've fixed another misconception of mine...
IT> 
IT>   When I was grinding through the Intel 386 reference, I saw descriptors
IT>   (with their limits) as the most natural and ideal way to control
IT>   memory access by different processes. If I were to design an OS, I
IT>   would have probably had DosAllocMem create a brand new descriptor in
IT>   the LDT of the calling process for each allocation and return a
IT>   selector to it. This way, if a process hits an address outside the
IT>   descriptor limit, the OS would be immediately notified via the
IT>   exception handler, and if the allocation was less than 1MB I would
IT>   have a byte granularity (I would know IMMEDIATELY when a process
IT>   wanders outside the allocated memory, I would not have to wait for it
IT>   to hit the next 4KB page boudary).
IT> 
IT>   But now I can see why this is the wrong way to go :( If the process
IT>   were to alternately access the memory chunks from different
IT>   allocations, it would cause a lot of reloading of selector registers.
IT> 
IT>   If I understood you correctly, you DON'T use descriptors to control
IT>   memory access in OS/2. You just give three big (512MB limit)
IT>   descriptors to every process, and the selectors in that 32-bit process
IT>   DON'T change during its execution. It only uses OFFSETS to access the
IT>   memory in that 512MB range.
IT> 
IT>   The question is, HOW do you stop the process from accessing the parts
IT>   of that 512 megs that it hasn't allocated yet. The answer is, by
IT>   cleverly managing the page mapping tables. You're doing it on a whole
IT>   different level. DosAllocMem just returns an OFFSET into the already
IT>   given 512 MB descriptor (created by the OS at the initialization of
IT>   the process), and sets up the page tables and directories for the
IT>   process in such a way that any attempt to access any other part of
IT>   that 512MB descriptor would cause a page fault.
IT> 
IT>   Very clever, I must say. You lose some precision that way (invalid
IT>   memory accesses can be detected only when they cross a 4KB boundary),
IT>   but you save an awful lot of descriptors and greatly speed up memory
IT>   access! I would have *never* thought of this...
IT> 
IT>   It is not the descriptors that control and restrict the memory access
IT>   in a single process in OS/2, it is actually the page tables! You are
IT>   using the page mapping mechanism to its fullest extent.

IT>   Ummmm... does this make any sense to you?! :)

 Yep.. It looks like you have it exactly (and much of the 
ramifications). In fact, this is the technique used on the 370/390/etc 
mainframe machines (and has been used LONG before Intel did). I always
did find it an amazing "coincidence" that the 386 prcessor used this 
technique after IBM invested in Intel and made some "technology 
transfers" to Intel.. It's a shame that they divested themselves of 
the investment shortly after the 386 debut.. 

 IT>>   What's going on inside there? :)
 
  DT>  OK.. As you suggest, and example might help.
 
  DT>  Lets say you have a 32 bit app that call DosAllocMem for 132K bytes.
  DT> Alloc returns an address to you of (say) 00050000h where you have 132K
   [...]
 
IT>   Thanks for this thorough example. No amount of explanation can
IT>   substitute a well thought out example!
 
  DT>   Selector 002F    Base 00050000 Limit FFFF \
  DT>   Selector 0037    Base 00060000 Limit FFFF   Maps 132K allocation
  DT>   Selector 003F    Base 00070000 Limit 0FFF /
  DT>   Selector 0047    Base 00080000 Limit FFFF \
  DT>   Selector 004F    Base 00090000 Limit FFFF   Maps 204K allocation
  DT>   Selector 0057    Base 000A0000 Limit FFFF
  DT>   Selector 005F    Base 000B0000 Limit 2FFF /
  DT>   Selector 0067    Base 000C0000 Limit 0FFF   Maps 4K allocation
  DT>   Selector 006F    Base 000D0000 Limit 0FFF   Maps 4K allocation
 
  DT>  Now, *IF* your application needed to convert one of the 32 bit
  DT> addreses to a 16:16 form, it can easily do so. Say that you wanted a
  DT> field at address 000B1234 to be updated by some 16:16 code (an old DLL
  DT> for example). You need to convert this 32 bit address into something
  DT> the 16:16 code can handle. Because of the tiling of the LDT, this is
  DT> very simple. Take the upper 16 bits of the 32 bit address and shift to
  DT> the left by 3 bits (times 8) and (since you are trying to convert this
  DT> into a ring 3 LDT selector) turn on the last 3 bits (plus 7). In this
  DT> example 000B*8+7=005F. That becomes our "selector" part
of the 16:16
  DT> address. The last 16 bits of the 32 bit address become the offset part
  DT> of the 16:16 address, thus 005F:1234 is the address we pass to the
  DT> 16:16 code to use. The LDT entry (and the page tables) prevents the
  DT> 16 bit code from accessing beyond the allocation unit (limit of 2FFF).
 
IT>   I was wondering about that strange choice of LDT selectors
IT>   (37,3F,47,4F etc) when I saw it in that first big message from you on
IT>   this subject, but I finally see the light now... this double
IT>   descriptor coverage scheme is simply ingenious!

 When I use selector values, I tend to give the "full" 16 bit values. 
The bottom 2 bits of a selector value are the privilege level, the 3rd
bit from the bottom tells if the value is referencing the GDT or the 
LDT. So sequential "selectors" in the LDT will always be 07, 0F, 17, 
1F, 27, 2F, etc.. 

IT>   It is very sad indeed that those people making marketing decisions at
IT>   IBM don't appreciate at all the creativity and inovative efforts of
IT>   the programmers that made OS/2... it makes me sick!

 All I can say is that the technology is not being wasted. It may not
be marketed to the segment, or in a product form, that you (and I)
would like to see it directed at, but it is being used. Knowledge of 
the techniques and structure will not be wasted time.. 
 

   Denis       

 All opinions are my very own, IBM has no claim upon them
. 
. 
.
 

 



--- Maximus/2 3.01