mgzip

mgzip is a program that makes use of SMP machines and zlib to use as many processors as you have to quickly compress files into gzip compatible format. mgzip only compresses; gunzip must be used to uncompress the resultant file.  I have been able to compress files three times faster than standard gzip on a 4 processor Alpha 4100 using three worker threads.  Currently, mgzip will compile and run on Digital Unix 4.0 and 5.1, Linux with a suitable pthread library, and AIX 4.3 and 5.1L. Porting to other Unices with pthread libraries should be an easy task, and may be as simple as compiling. Reports are that it works on Solaris as well with a few additional thread-related flags.

03/25/2003: I've posted the results of a quick comparison between this project, my SMP lzop patch, stock gzip and lzop on a dual Athlon GNU/Linux machine. Note that this comparison uses a single data file and the results are probably highly data dependent. Your mileage will almost certainly vary.

Read the current README file.

I am releasing this program under the GPL.

mgzip is in no way derived from or related to gzip by Mark Adler and Jean-loup Gailly which is a very fine software indeed.

There is a bug in gzip 1.2.4 that you must fix before reliably uncompressing files made with mgzip. Jean-loup Gailly has made available a patch for version 1.2.4, but your best bet is to update your gzip with a newer version as can be found on alpha.gnu.org.

The source code for mgzip is here. It works reasonably well.

Here is the help message generated from the (almost) current version: 

mgzip, a multi-processor capable .gz file creator. 

mgzip version 1.2a build 110 on Feb 13 2003
Usage: mgzip [-cfp19L] [-t N] [-C N] [file...]  
where -1 is fastest and -9 is best compression (default is -2)
and -t N is the number of worker threads to use (default is -t 2)
If filename is omitted, this program compresses from the standard input to the
standard output. 
More parameters:
  -c writes compressed data to standard out and keeps the original
  -C N where N is a size in bytes will change the default
    chunk size (now 131072)
  -f forces compression to a terminal 
  -p preserves the input file after compression
  -L prints the license and quits. 

Note: this program only compresses files; you must use gzip -d (or gunzip) to 
uncompress the resulting compressed files.  This program is neither derived from
nor directly related to "gzip" by Jean-loup Gailly and Mark Adler, although it
does use the "zlib" compression library by the same authors under the GPL.