Craig Swanson wrote in a message to Mike Bilow:

 MB> The 64 KB problem permeates the file system interface, and pipes go
 MB> through the FS router.  Everything in the whole FS
 MB> chain has to observe the 64 KB limit in case the data
 MB> block is eventually mapped down into a physical DMA
 MB> operation.  There is no appropriate method by which
 MB> the IFS, for example, can know whether the controller
 MB> operating its disk will have a 64 KB DMA limit, nor
 MB> even whether DMA will be used at all.

 MB> I do agree that the pipes subsystem should not fail as
 MB> it does in the presense of greater than 64 KB
 MB> transfers, but I suspect that it would be hard to fix.

 CS> If the only problem was that it couldn't handle transfers
 CS> larger than 64KB, that would not be so bad.  The problem is
 CS> worse, however,because any DosRead() or DosWrite() call on a
 CS> named pipe using a buffer that crosses a 64KB boundary (but
 CS> is not larger than 64KB) will fail.

The problem is quite deep in the operating system.  The file system
interface is a 16-bit model, as is the request packet interface used by
physical device drivers.  When 32-bit code calls DosRead() or DosWrite()
from Ring 3, the address it knows is what we call a "linear" or
"0:32" address in "process" address space.  The file
system router, which is part of the kernel, locks the memory buffer
(preventing it from being swapped out or moved) and converts the address of
the buffer to a 32-bit physical address.  This is reasonable because the
operating system expects DosRead() and DosWrite() to be used in connection
with hardware that does DMA, such as a disk controller.

Unfortunately, you can't do much with a physical address except pass it
back and forth to hardware, so code which is emulating hardware, such as
the pipe subsystem, must reconvert the physical address into a
"virtual" or "16:16" address.  There are only so many
ways to accomplish this, and all of them have trade-offs.  The approach
actually used is to map the buffer with a temporary selector.  This gives
triple addressability to this locked and fixed region of memory: linear,
physical, and virtual.  The temporary selector is destroyed (or at least
potentially invalidated) if the driver using it yields, which makes for
nasty problems when operating over a network.  Like any other 16:16
selector, this temporary selector cannot map more than 64 KB at a time.

While this problem is common to the whole file system interface, it turns
out that there is a massive conspiracy among all of the OS/2 Ring 0 code to
make sure that no one ever actually attempts to read or write more than 64
KB at a time.  Even hard and floppy disk accesses are guaranteed to be no
greater than 64 KB per call into the low-level drivers.  However, disk I/O,
unlike pipes, has no internal organization: each sector has the same
signifcance as any other, differing only by address but never by size. 
With pipes, the kernel is not free to just chunk the data as it sees fit,
since pipes are supposed to be able to distinguish one message from
another. 

 CS> Lifting the 64KB message size limit is not trivially easy
 CS> given that named pipes are supported on multiple platforms
 CS> (not just OS/2) and the pipes can transport data across
 CS> networks.  Perhaps a new \PIPE32 prefix could be added for
 CS> pipes that can handle 32-bit message sizes and the existing
 CS> \PIPE prefix could be left for pipes that can only handle
 CS> 16-bit message sizes?

The solution is to move the pipe subsystem out of the file system, or to
make the file system interface itself 32-bit.
 
-- Mike


---