Re: hpt zlib
  By: Matt Bedynek to Eric Renfro on Tue Sep 15 2015 11:44 am

 MB> On Tue, 15 Sep 2015 17:08:08 -0400, Eric Renfro wrote:

 ER>> It surprises me how many people tout off ZFS, and don't really
 ER>> understand 
 MB> just
 ER>> what they're suggesting. My BBS runs on a VM running 512MB RAM, I
 ER>> only 
 MB> upped it
 ER>> recently to 1GB to run clamd. ZFS, adding compression to it, memory 
 MB> utilization
 ER>> just for ZFS skyrockets. Add deduplication on top of that, now we're

 MB> talking
 ER>> overkill memory. :)

 MB> ZFS memory utilization depends on a few things. First of all, the
 MB> amount of disk space provisioned, how it is tuned, and the OS chosen.
 MB> It has never ran well under Linux, is known to have issues freeing
 MB> memory--I would never run it there. For the space requirements of a
 MB> modern Fidonet system it is unlikely to be a problem. A 100GB
 MB> virtual machine could provide more than enough in resources to
 MB> accommodate the most rigerous fidonet processing today.

Definitely true. I mean, my BBS as it stands today for my actual BBS with
Synchronet and a couple dozen door games, and all of fidonet, agoranet, soon
stn, linuxnet, possibly others. 20GB total space allocated, though I'm not
carrying all fileecho areas, just the needed minimums and a few others. 

So really yeah you're right, not much needed. 

 MB> If you're provisioning more space and are concerned about performance
 MB> degredation associated with caching a larger storage pool then you
 MB> probably don't need compression anyway. My 100GB virtual machine has
 MB> several hundred thousand messages and the is only at 6% utilization. I
 MB> do not use any compression here nor see the need to in the future.

Heh, I never really /needed/ compression of my message bases themselves. I only
mentioned it that Synchronet's SMB message format has LZH compression support,
because you were asking why I'd need zlib support in hpt and htick, apparently
you were thinking it was related to the message bases themselves, and not the
unpacker/packer at the time. hehe

 ER>> It would be less resource intensive to use message base compression
 ER>> than 
 MB> to use
 ER>> ZFS compression, in this logic, and I could, if I wanted to spend
 ER>> the time 
 MB> to
 ER>> do so, prove that. When it comes to Linux, filesystems, and such, I
 ER>> know a seriously heavy amount of information, as I've been doing
 ER>> Linux, hardcore, since 1992. :)

 MB> Running ZFS on Linux is likely to provide a jaded opinion. You should
 MB> run it on FreeBSD or Solaris. The need for vast amounts of ram were
 MB> only needed in cases where a head node mounted terabytes of storage
 MB> and higher performance was desired.

Also true. I have run ZFS on FreeBSD, and while it is better, it's still....
Not optimal in what I'd be using it for overall. I used to run 2 machines that
housed KVM disk images for my 4 hypervisors that used to run directly off them.
ZFS would provide qcow disk images over NFSv4, or straight iSCSI over ZFS
sub-pooling. The performance of that was "acceptable", however not great. 

My best method was actually using 3 systems to run Ceph, which provided RBD
(Remote Block Disks, iSCSI-like), CephFS (NFS-like), and S3 over a
high-availability cluster. That setup alone dropped my access time for my
self-hosted Drupal webserver instance from 4~4.5 seconds to 1~1.5 seconds. So
that was a huge difference in overal performance. Course, ZFS was running on
Linux, not BSD. 

Goods and bads in everything really. :)

These days I simply run 2 hypervisors with local storage, part of which is
shared over GlusterFS, like my BBS, so I can live migrate the BBS between
hypervisors and keep it up and running, even while users are online. :)

)))[Psi-Jack -//- Decker]

... Some people confuse boredom with security.
--- SBBSecho 2.27-Linux