diff options
Diffstat (limited to 'Master/tlpkg/tlperl/lib/IO/Compress/FAQ.pod')
-rw-r--r-- | Master/tlpkg/tlperl/lib/IO/Compress/FAQ.pod | 597 |
1 files changed, 0 insertions, 597 deletions
diff --git a/Master/tlpkg/tlperl/lib/IO/Compress/FAQ.pod b/Master/tlpkg/tlperl/lib/IO/Compress/FAQ.pod deleted file mode 100644 index d392ff2cc91..00000000000 --- a/Master/tlpkg/tlperl/lib/IO/Compress/FAQ.pod +++ /dev/null @@ -1,597 +0,0 @@ - -=head1 NAME - -IO::Compress::FAQ -- Frequently Asked Questions about IO::Compress - -=head1 DESCRIPTION - -Common questions answered. - -=head1 GENERAL - -=head2 Compatibility with Unix compress/uncompress. - -Although C<Compress::Zlib> has a pair of functions called C<compress> and -C<uncompress>, they are I<not> related to the Unix programs of the same -name. The C<Compress::Zlib> module is not compatible with Unix -C<compress>. - -If you have the C<uncompress> program available, you can use this to read -compressed files - - open F, "uncompress -c $filename |"; - while (<F>) - { - ... - -Alternatively, if you have the C<gunzip> program available, you can use -this to read compressed files - - open F, "gunzip -c $filename |"; - while (<F>) - { - ... - -and this to write compress files, if you have the C<compress> program -available - - open F, "| compress -c $filename "; - print F "data"; - ... - close F ; - -=head2 Accessing .tar.Z files - -The C<Archive::Tar> module can optionally use C<Compress::Zlib> (via the -C<IO::Zlib> module) to access tar files that have been compressed with -C<gzip>. Unfortunately tar files compressed with the Unix C<compress> -utility cannot be read by C<Compress::Zlib> and so cannot be directly -accessed by C<Archive::Tar>. - -If the C<uncompress> or C<gunzip> programs are available, you can use one -of these workarounds to read C<.tar.Z> files from C<Archive::Tar> - -Firstly with C<uncompress> - - use strict; - use warnings; - use Archive::Tar; - - open F, "uncompress -c $filename |"; - my $tar = Archive::Tar->new(*F); - ... - -and this with C<gunzip> - - use strict; - use warnings; - use Archive::Tar; - - open F, "gunzip -c $filename |"; - my $tar = Archive::Tar->new(*F); - ... - -Similarly, if the C<compress> program is available, you can use this to -write a C<.tar.Z> file - - use strict; - use warnings; - use Archive::Tar; - use IO::File; - - my $fh = new IO::File "| compress -c >$filename"; - my $tar = Archive::Tar->new(); - ... - $tar->write($fh); - $fh->close ; - -=head2 How do I recompress using a different compression? - -This is easier that you might expect if you realise that all the -C<IO::Compress::*> objects are derived from C<IO::File> and that all the -C<IO::Uncompress::*> modules can read from an C<IO::File> filehandle. - -So, for example, say you have a file compressed with gzip that you want to -recompress with bzip2. Here is all that is needed to carry out the -recompression. - - use IO::Uncompress::Gunzip ':all'; - use IO::Compress::Bzip2 ':all'; - - my $gzipFile = "somefile.gz"; - my $bzipFile = "somefile.bz2"; - - my $gunzip = new IO::Uncompress::Gunzip $gzipFile - or die "Cannot gunzip $gzipFile: $GunzipError\n" ; - - bzip2 $gunzip => $bzipFile - or die "Cannot bzip2 to $bzipFile: $Bzip2Error\n" ; - -Note, there is a limitation of this technique. Some compression file -formats store extra information along with the compressed data payload. For -example, gzip can optionally store the original filename and Zip stores a -lot of information about the original file. If the original compressed file -contains any of this extra information, it will not be transferred to the -new compressed file usign the technique above. - -=head1 ZIP - -=head2 What Compression Types do IO::Compress::Zip & IO::Uncompress::Unzip support? - -The following compression formats are supported by C<IO::Compress::Zip> and -C<IO::Uncompress::Unzip> - -=over 5 - -=item * Store (method 0) - -No compression at all. - -=item * Deflate (method 8) - -This is the default compression used when creating a zip file with -C<IO::Compress::Zip>. - -=item * Bzip2 (method 12) - -Only supported if the C<IO-Compress-Bzip2> module is installed. - -=item * Lzma (method 14) - -Only supported if the C<IO-Compress-Lzma> module is installed. - -=back - -=head2 Can I Read/Write Zip files larger the 4 Gig? - -Yes, both the C<IO-Compress-Zip> and C<IO-Uncompress-Unzip> modules -support the zip feature called I<Zip64>. That allows them to read/write -files/buffers larger than 4Gig. - -If you are creating a Zip file using the one-shot interface, and any of the -input files is greater than 4Gig, a zip64 complaint zip file will be -created. - - zip "really-large-file" => "my.zip"; - -Similarly with the one-shot interface, if the input is a buffer larger than -4 Gig, a zip64 complaint zip file will be created. - - zip \$really_large_buffer => "my.zip"; - -The one-shot interface allows you to force the creation of a zip64 zip file -by including the C<Zip64> option. - - zip $filehandle => "my.zip", Zip64 => 1; - -If you want to create a zip64 zip file with the OO interface you must -specify the C<Zip64> option. - - my $zip = new IO::Compress::Zip "whatever", Zip64 => 1; - -When uncompressing with C<IO-Uncompress-Unzip>, it will automatically -detect if the zip file is zip64. - -If you intend to manipulate the Zip64 zip files created with -C<IO-Compress-Zip> using an external zip/unzip, make sure that it supports -Zip64. - -In particular, if you are using Info-Zip you need to have zip version 3.x -or better to update a Zip64 archive and unzip version 6.x to read a zip64 -archive. - -=head2 Zip Resources - -The primary reference for zip files is the "appnote" document available at -L<http://www.pkware.com/documents/casestudies/APPNOTE.TXT> - -An alternatively is the Info-Zip appnote. This is available from -L<ftp://ftp.info-zip.org/pub/infozip/doc/> - -=head1 GZIP - -=head2 Gzip Resources - -The primary reference for gzip files is RFC 1952 -L<http://www.faqs.org/rfcs/rfc1952.html> - -The primary site for gzip is F<http://www.gzip.org>. - -=head1 ZLIB - -=head2 Zlib Resources - -The primary site for the I<zlib> compression library is -F<http://www.zlib.org>. - -=head1 HTTP & NETWORK - -=head2 Apache::GZip Revisited - -Below is a mod_perl Apache compression module, called C<Apache::GZip>, -taken from -F<http://perl.apache.org/docs/tutorials/tips/mod_perl_tricks/mod_perl_tricks.html#On_the_Fly_Compression> - - package Apache::GZip; - #File: Apache::GZip.pm - - use strict vars; - use Apache::Constants ':common'; - use Compress::Zlib; - use IO::File; - use constant GZIP_MAGIC => 0x1f8b; - use constant OS_MAGIC => 0x03; - - sub handler { - my $r = shift; - my ($fh,$gz); - my $file = $r->filename; - return DECLINED unless $fh=IO::File->new($file); - $r->header_out('Content-Encoding'=>'gzip'); - $r->send_http_header; - return OK if $r->header_only; - - tie *STDOUT,'Apache::GZip',$r; - print($_) while <$fh>; - untie *STDOUT; - return OK; - } - - sub TIEHANDLE { - my($class,$r) = @_; - # initialize a deflation stream - my $d = deflateInit(-WindowBits=>-MAX_WBITS()) || return undef; - - # gzip header -- don't ask how I found out - $r->print(pack("nccVcc",GZIP_MAGIC,Z_DEFLATED,0,time(),0,OS_MAGIC)); - - return bless { r => $r, - crc => crc32(undef), - d => $d, - l => 0 - },$class; - } - - sub PRINT { - my $self = shift; - foreach (@_) { - # deflate the data - my $data = $self->{d}->deflate($_); - $self->{r}->print($data); - # keep track of its length and crc - $self->{l} += length($_); - $self->{crc} = crc32($_,$self->{crc}); - } - } - - sub DESTROY { - my $self = shift; - - # flush the output buffers - my $data = $self->{d}->flush; - $self->{r}->print($data); - - # print the CRC and the total length (uncompressed) - $self->{r}->print(pack("LL",@{$self}{qw/crc l/})); - } - - 1; - -Here's the Apache configuration entry you'll need to make use of it. Once -set it will result in everything in the /compressed directory will be -compressed automagically. - - <Location /compressed> - SetHandler perl-script - PerlHandler Apache::GZip - </Location> - -Although at first sight there seems to be quite a lot going on in -C<Apache::GZip>, you could sum up what the code was doing as follows -- -read the contents of the file in C<< $r->filename >>, compress it and write -the compressed data to standard output. That's all. - -This code has to jump through a few hoops to achieve this because - -=over - -=item 1. - -The gzip support in C<Compress::Zlib> version 1.x can only work with a real -filesystem filehandle. The filehandles used by Apache modules are not -associated with the filesystem. - -=item 2. - -That means all the gzip support has to be done by hand - in this case by -creating a tied filehandle to deal with creating the gzip header and -trailer. - -=back - -C<IO::Compress::Gzip> doesn't have that filehandle limitation (this was one -of the reasons for writing it in the first place). So if -C<IO::Compress::Gzip> is used instead of C<Compress::Zlib> the whole tied -filehandle code can be removed. Here is the rewritten code. - - package Apache::GZip; - - use strict vars; - use Apache::Constants ':common'; - use IO::Compress::Gzip; - use IO::File; - - sub handler { - my $r = shift; - my ($fh,$gz); - my $file = $r->filename; - return DECLINED unless $fh=IO::File->new($file); - $r->header_out('Content-Encoding'=>'gzip'); - $r->send_http_header; - return OK if $r->header_only; - - my $gz = new IO::Compress::Gzip '-', Minimal => 1 - or return DECLINED ; - - print $gz $_ while <$fh>; - - return OK; - } - -or even more succinctly, like this, using a one-shot gzip - - package Apache::GZip; - - use strict vars; - use Apache::Constants ':common'; - use IO::Compress::Gzip qw(gzip); - - sub handler { - my $r = shift; - $r->header_out('Content-Encoding'=>'gzip'); - $r->send_http_header; - return OK if $r->header_only; - - gzip $r->filename => '-', Minimal => 1 - or return DECLINED ; - - return OK; - } - - 1; - -The use of one-shot C<gzip> above just reads from C<< $r->filename >> and -writes the compressed data to standard output. - -Note the use of the C<Minimal> option in the code above. When using gzip -for Content-Encoding you should I<always> use this option. In the example -above it will prevent the filename being included in the gzip header and -make the size of the gzip data stream a slight bit smaller. - -=head2 Compressed files and Net::FTP - -The C<Net::FTP> module provides two low-level methods called C<stor> and -C<retr> that both return filehandles. These filehandles can used with the -C<IO::Compress/Uncompress> modules to compress or uncompress files read -from or written to an FTP Server on the fly, without having to create a -temporary file. - -Firstly, here is code that uses C<retr> to uncompressed a file as it is -read from the FTP Server. - - use Net::FTP; - use IO::Uncompress::Gunzip qw(:all); - - my $ftp = new Net::FTP ... - - my $retr_fh = $ftp->retr($compressed_filename); - gunzip $retr_fh => $outFilename, AutoClose => 1 - or die "Cannot uncompress '$compressed_file': $GunzipError\n"; - -and this to compress a file as it is written to the FTP Server - - use Net::FTP; - use IO::Compress::Gzip qw(:all); - - my $stor_fh = $ftp->stor($filename); - gzip "filename" => $stor_fh, AutoClose => 1 - or die "Cannot compress '$filename': $GzipError\n"; - -=head1 MISC - -=head2 Using C<InputLength> to uncompress data embedded in a larger file/buffer. - -A fairly common use-case is where compressed data is embedded in a larger -file/buffer and you want to read both. - -As an example consider the structure of a zip file. This is a well-defined -file format that mixes both compressed and uncompressed sections of data in -a single file. - -For the purposes of this discussion you can think of a zip file as sequence -of compressed data streams, each of which is prefixed by an uncompressed -local header. The local header contains information about the compressed -data stream, including the name of the compressed file and, in particular, -the length of the compressed data stream. - -To illustrate how to use C<InputLength> here is a script that walks a zip -file and prints out how many lines are in each compressed file (if you -intend write code to walking through a zip file for real see -L<IO::Uncompress::Unzip/"Walking through a zip file"> ). Also, although -this example uses the zlib-based compression, the technique can be used by -the other C<IO::Uncompress::*> modules. - - use strict; - use warnings; - - use IO::File; - use IO::Uncompress::RawInflate qw(:all); - - use constant ZIP_LOCAL_HDR_SIG => 0x04034b50; - use constant ZIP_LOCAL_HDR_LENGTH => 30; - - my $file = $ARGV[0] ; - - my $fh = new IO::File "<$file" - or die "Cannot open '$file': $!\n"; - - while (1) - { - my $sig; - my $buffer; - - my $x ; - ($x = $fh->read($buffer, ZIP_LOCAL_HDR_LENGTH)) == ZIP_LOCAL_HDR_LENGTH - or die "Truncated file: $!\n"; - - my $signature = unpack ("V", substr($buffer, 0, 4)); - - last unless $signature == ZIP_LOCAL_HDR_SIG; - - # Read Local Header - my $gpFlag = unpack ("v", substr($buffer, 6, 2)); - my $compressedMethod = unpack ("v", substr($buffer, 8, 2)); - my $compressedLength = unpack ("V", substr($buffer, 18, 4)); - my $uncompressedLength = unpack ("V", substr($buffer, 22, 4)); - my $filename_length = unpack ("v", substr($buffer, 26, 2)); - my $extra_length = unpack ("v", substr($buffer, 28, 2)); - - my $filename ; - $fh->read($filename, $filename_length) == $filename_length - or die "Truncated file\n"; - - $fh->read($buffer, $extra_length) == $extra_length - or die "Truncated file\n"; - - if ($compressedMethod != 8 && $compressedMethod != 0) - { - warn "Skipping file '$filename' - not deflated $compressedMethod\n"; - $fh->read($buffer, $compressedLength) == $compressedLength - or die "Truncated file\n"; - next; - } - - if ($compressedMethod == 0 && $gpFlag & 8 == 8) - { - die "Streamed Stored not supported for '$filename'\n"; - } - - next if $compressedLength == 0; - - # Done reading the Local Header - - my $inf = new IO::Uncompress::RawInflate $fh, - Transparent => 1, - InputLength => $compressedLength - or die "Cannot uncompress $file [$filename]: $RawInflateError\n" ; - - my $line_count = 0; - - while (<$inf>) - { - ++ $line_count; - } - - print "$filename: $line_count\n"; - } - -The majority of the code above is concerned with reading the zip local -header data. The code that I want to focus on is at the bottom. - - while (1) { - - # read local zip header data - # get $filename - # get $compressedLength - - my $inf = new IO::Uncompress::RawInflate $fh, - Transparent => 1, - InputLength => $compressedLength - or die "Cannot uncompress $file [$filename]: $RawInflateError\n" ; - - my $line_count = 0; - - while (<$inf>) - { - ++ $line_count; - } - - print "$filename: $line_count\n"; - } - -The call to C<IO::Uncompress::RawInflate> creates a new filehandle C<$inf> -that can be used to read from the parent filehandle C<$fh>, uncompressing -it as it goes. The use of the C<InputLength> option will guarantee that -I<at most> C<$compressedLength> bytes of compressed data will be read from -the C<$fh> filehandle (The only exception is for an error case like a -truncated file or a corrupt data stream). - -This means that once RawInflate is finished C<$fh> will be left at the -byte directly after the compressed data stream. - -Now consider what the code looks like without C<InputLength> - - while (1) { - - # read local zip header data - # get $filename - # get $compressedLength - - # read all the compressed data into $data - read($fh, $data, $compressedLength); - - my $inf = new IO::Uncompress::RawInflate \$data, - Transparent => 1, - or die "Cannot uncompress $file [$filename]: $RawInflateError\n" ; - - my $line_count = 0; - - while (<$inf>) - { - ++ $line_count; - } - - print "$filename: $line_count\n"; - } - -The difference here is the addition of the temporary variable C<$data>. -This is used to store a copy of the compressed data while it is being -uncompressed. - -If you know that C<$compressedLength> isn't that big then using temporary -storage won't be a problem. But if C<$compressedLength> is very large or -you are writing an application that other people will use, and so have no -idea how big C<$compressedLength> will be, it could be an issue. - -Using C<InputLength> avoids the use of temporary storage and means the -application can cope with large compressed data streams. - -One final point -- obviously C<InputLength> can only be used whenever you -know the length of the compressed data beforehand, like here with a zip -file. - -=head1 SEE ALSO - -L<Compress::Zlib>, L<IO::Compress::Gzip>, L<IO::Uncompress::Gunzip>, L<IO::Compress::Deflate>, L<IO::Uncompress::Inflate>, L<IO::Compress::RawDeflate>, L<IO::Uncompress::RawInflate>, L<IO::Compress::Bzip2>, L<IO::Uncompress::Bunzip2>, L<IO::Compress::Lzma>, L<IO::Uncompress::UnLzma>, L<IO::Compress::Xz>, L<IO::Uncompress::UnXz>, L<IO::Compress::Lzop>, L<IO::Uncompress::UnLzop>, L<IO::Compress::Lzf>, L<IO::Uncompress::UnLzf>, L<IO::Uncompress::AnyInflate>, L<IO::Uncompress::AnyUncompress> - -L<IO::Compress::FAQ|IO::Compress::FAQ> - -L<File::GlobMapper|File::GlobMapper>, L<Archive::Zip|Archive::Zip>, -L<Archive::Tar|Archive::Tar>, -L<IO::Zlib|IO::Zlib> - -=head1 AUTHOR - -This module was written by Paul Marquess, F<pmqs@cpan.org>. - -=head1 MODIFICATION HISTORY - -See the Changes file. - -=head1 COPYRIGHT AND LICENSE - -Copyright (c) 2005-2012 Paul Marquess. All rights reserved. - -This program is free software; you can redistribute it and/or -modify it under the same terms as Perl itself. - |