Original from  Will Honea  to Denis Tonn on 04-25-1998
Original Subject: Beginlibpath?
                         ---------------------------------------
WH> We sure have digressed from the original topic, haven't we?  Excellent
WH> article, Denis.  Now I have a lot better idea of what might be causing
WH> ocassional 'unexpected' disk activity.  Perhaps you could cover one
WH> further topic on swap files for us.
WH> 
WH> Take the normal case where some amount of data is swapped.  Now run
WH> something that will fill the remaing swap space and grow the swapper -
WH> Netscape visiting graphics intensive sites would be one example.  For
WH> the sake of clarity, assume it doubles the swap file and that normal
WH> swapper usage is 10-20% of the initial swap allocation.  On top of
WH> this, start another program which will cause additional swapper usage -
WH> Pronews/2, for example, commits substantial memory and will cause
WH> additional growth.  Close Netscape, leave PN/2 open.  How does the
WH> system go about doing garbage collection if the last program uses
WH> blocks of the swapper in the 'grown' area?  Does the system employ
WH> garbage collection to shrink the swapper or will it wait until the last
WH> program is closed?  Or does it re-position to backfill the swap file as
WH> system dynamics cause further swap activity?
 It does both, I hope the explaination below answers this. 
WH> My gut feeling from watching the swap size is that the system
WH> eventually compacts the swap file but at a very slow rate until
WH> programs with commited swap areas are closed.  I tend to believe that
WH> it is this dynamic that causes a lot of 'my swap file never shrinks'
WH> complaints, especially if the far out swapped data is in fact a block
 It depends on the release and FP level of OS/2. This one area where 
there is always ongoing tuning as the applications and functions of 
the OS are added to or changed. 
WH> never released (leaked) by some program (afinet.sys has been shown to
WH> allocated 16K everytime it is used without releasing it, for example).
 Try getting the latest TCPIP stack (and MPTS) updates. I believe this
was fixed in the 8415 MPTS FP (and subsequent ones). 8423 is the 
current one for Warp 4, I don't offhand remember the current one for 
Warp 3. 
 OK, another "mini-essay" on OS/2 memory management.  
 This is a little divergent from "virtual memory management", which is
what my last note tried to touch on. This is more in the realm of 
"swapper management". A closely related subject, but still separate. 
 To understand this, look at the flow of activity as you overcommit 
the system in the above example. When you load Netscape and need extra
memory the system will swap out the "oldest" dormant pages first. This
will usually NOT be Netscape memory. Swapper will grow to accomodate 
these pages. Now load PN/2 and the swapper will grow again, probably 
swapping out a bunch of Netscape memory (although Netscape is still 
running, so there may be a mix of NS, PN/2, and other pages). This 
sets up a condition where lots of pages exist in *BOTH* RAM and 
swapper (active programs). Keep in mind that this is very efficient 
for two reasons. The pages are already preallocated in swapper and if 
the memory page has not been modified, the system can just "reuse" the 
RAM pages without "saving" it to the swapper first (it can recover 
from the swapper if needed). 
 Now, you stop NS and the "memory" it uses is freed - both the RAM 
pages and the swapper pages. The swapper has not "shrunk" yet, so 
there are a bunch of "holes" in the swapper with empty (free) pages. 
The system will use these free blocks as it runs, but odds are pretty
good that you will still have a bunch of them. If there are a bunch of
"holes" at the end of the swapper file, the system can simply shrink
to the last used block, but it can't shrink below the last one without
reorganizing the usage within the swapper.. This will happen, but on a
low priority basis. Odds are very good that you will be running some
other program (sooner or later) that will end up using the free
"holes" in the swapper - it makes sense to delay a while to see if
this happens (and thereby avoid the reallocation of swap file space).
 This is why it may take a while to shrink the swapper back down to
size. In fact, it may not shrink back to it's original size at all,
there may still be "dormant" pages and swapped pages in the swapper
left over from that first growth when NS loaded. The dormant pages
will never be "reloaded" back to RAM until they are needed and since
we already have a copy of the "active" swappable pages (still in use,
but in both RAM and swapper) it makes sense to leave the copy in the
swapper (remember the efficiencies gained).
 There are lots of frills that can be added to this situation, like
"freeing" duplicate swapper pages when a RAM page is modified and you 
still have lots of free RAM pages to allow further swapper shrinkage
(might just as well reallocate swapper later), and "preswapping" pages
when RAM gets below a certain percentage (the CPU/Disk is not overly 
busy, so might as well get ready for later). Usually these actions are 
not quick actions, but designed to keep the system in a efficient
swapper/ram situation. These frills are esoteric and not needed for a
basic understanding.
 The key thing to remember is that memory is NOT swapped back in until
it is needed, and that pages could exist in both places even when you
have lots of "free RAM".
 The system is designed to be as efficient as possible and swapper
SIZE is not the prime indicator you need to be looking at (it is an
indicator, but only when you understand everything else that is going
on). This is not new technology, but really old stuff that has been 
around (on mainframes) for over 20 years. It is very well understood, 
OS/2's VM/swapper management was designed by these same mainframe 
people. Normal programmers (even mainframe ones) are only vaguely 
aware of it, only systems tuning experts (the good ones) and OS 
Virtual Memory designers are normally aware of what is going on. 
   Denis       
 All opinions are my very own, IBM has no claim upon them
. 
. 
.
 
 
--- Maximus/2 3.01
---------------