diff options
Diffstat (limited to 'Build/source/utils/xz-4.999.9beta/doc')
22 files changed, 2617 insertions, 0 deletions
diff --git a/Build/source/utils/xz-4.999.9beta/doc/faq.txt b/Build/source/utils/xz-4.999.9beta/doc/faq.txt new file mode 100644 index 00000000000..48c4ec711b8 --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/faq.txt @@ -0,0 +1,122 @@ + +XZ Utils FAQ +============ + +Q: What do the letters XZ mean? + +A: Nothing. They are just two letters, which come from the file format + suffix .xz. The .xz suffix was selected, because it seemed to be + pretty much unused. It is no deeper meaning. + + +Q: What are LZMA and LZMA2? + +A: LZMA stands for Lempel-Ziv-Markov chain-Algorithm. It is the name + of the compression algorithm designed by Igor Pavlov for 7-Zip. + LZMA is based on LZ77 and range encoding. + + LZMA2 is an updated version of the original LZMA to fix a couple of + practical issues. In context of XZ Utils, LZMA is called LZMA1 to + emphasize that LZMA is not the same thing as LZMA2. LZMA2 is the + primary compression algorithm in the .xz file format. + + +Q: There are many LZMA related projects. How does XZ Utils relate to them? + +A: 7-Zip and LZMA SDK are the original projects. LZMA SDK is roughly + a subset of the 7-Zip source tree. + + p7zip is 7-Zip's command line tools ported to POSIX-like systems. + + LZMA Utils provide a gzip-like lzma tool for POSIX-like systems. + LZMA Utils are based on LZMA SDK. XZ Utils are the successor to + LZMA Utils. + + There are several other projects using LZMA. Most are more or less + based on LZMA SDK. + + +Q: Do XZ Utils support the .7z format? + +A: No. Use 7-Zip (Windows) or p7zip (POSIX-like systems) to handle .7z + files. + + +Q: I have many .tar.7z files. Can I convert them to .tar.xz without + spending hours recompressing the data? + +A: In the "extra" directory, there is a script named 7z2lzma.bash which + is able to convert some .7z files to the .lzma format (not .xz). It + needs the 7za (or 7z) command from p7zip. The script may silently + produce corrupt output if certain assumptions are not met, so + decompress the resulting .lzma file and compare it against the + original before deleting the original file! + + +Q: I have many .lzma files. Can I quickly convert them to the .xz format? + +A: For now, no. Since XZ Utils supports the .lzma format, it's usually + not too bad to keep the old files in the old format. If you want to + do the conversion anyway, you need to decompress the .lzma files and + then recompress to the .xz format. + + Technically, there is a way to make the conversion relatively fast + (roughly twice the time that normal decompression takes). Writing + such a tool would take quite a bit time though, and would probably + be useful to only a few people. If you really want such a conversion + tool, contact Lasse Collin and offer some money. + + +Q: Can I recover parts of a broken .xz file (e.g. corrupted CD-R)? + +A: It may be possible if the file consists of multiple blocks, which + typically is not the case if the file was created in single-threaded + mode. There is no recovery program yet. + + +Q: Is (some part of) XZ Utils patented? + +A: Lasse Collin is not aware of any patents that could affect XZ Utils. + However, due to nature of software patents, it's not possible to + guarantee that XZ Utils isn't affected by any third party patent(s). + + +Q: Where can I find documentation about the file format and algorithms? + +A: The .xz format is documented in xz-file-format.txt. It is a container + format only, and doesn't include descriptions of any non-trivial + filters. + + Documenting LZMA and LZMA2 is planned, but for now, there is no other + documentation that the source code. Before you begin, you should know + the basics of LZ77 and range coding algorithms. LZMA is based on LZ77, + but LZMA is *a lot* more complex. Range coding is used to compress + the final bitstream like Huffman coding is used in Deflate. + + +Q: I cannot find BCJ and BCJ2 filters. Don't they exist in liblzma? + +A: BCJ filter is called "x86" in liblzma. BCJ2 is not included, + because it requires using more than one encoded output stream. + + +Q: How do I build a program that needs liblzmadec (lzmadec.h)? + +A: liblzmadec is part of LZMA Utils. XZ Utils has liblzma, but no + liblzmadec. The code using liblzmadec should be ported to use + liblzma instead. If you cannot or don't want to do that, download + LZMA Utils from <http://tukaani.org/lzma/>. + + +Q: The default build of liblzma is too big. How can I make it smaller? + +A: Give --enable-small to the configure script. Use also appropriate + --enable or --disable options to include only those filter encoders + and decoders and integrity checks that you actually need. Use + CFLAGS=-Os (with GCC) or equivalent to tell your compiler to optimize + for size. See INSTALL for information about configure options. + + If the result is still too big, take a look at XZ Embedded. It is + a separate project, which provides a limited but signinificantly + smaller XZ decoder implementation than XZ Utils. + diff --git a/Build/source/utils/xz-4.999.9beta/doc/history.txt b/Build/source/utils/xz-4.999.9beta/doc/history.txt new file mode 100644 index 00000000000..c97492e81a0 --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/history.txt @@ -0,0 +1,149 @@ + +History of LZMA Utils and XZ Utils +================================== + +Tukaani distribution + + In 2005, there was a small group working on Tukaani distribution, which + was a Slackware fork. One of the project goals was to fit the distro on + a single 700 MiB ISO-9660 image. Using LZMA instead of gzip helped a + lot. Roughly speaking, one could fit data that took 1000 MiB in gzipped + form into 700 MiB with LZMA. Naturally compression ratio varied across + packages, but this was what we got on average. + + Slackware packages have traditionally had .tgz as the filename suffix, + which is an abbreviation of .tar.gz. A logical naming for LZMA + compressed packages was .tlz, being an abbreviation of .tar.lzma. + + At the end of the year 2007, there was no distribution under the + Tukaani project anymore, but development of LZMA Utils was kept going. + Still, there were .tlz packages around, because at least Vector Linux + (a Slackware based distribution) used LZMA for its packages. + + First versions of the modified pkgtools used the LZMA_Alone tool from + Igor Pavlov's LZMA SDK as is. It was fine, because users wouldn't need + to interact with LZMA_Alone directly. But people soon wanted to use + LZMA for other files too, and the interface of LZMA_Alone wasn't + comfortable for those used to gzip and bzip2. + + +First steps of LZMA Utils + + The first version of LZMA Utils (4.22.0) included a shell script called + lzmash. It was wrapper that had gzip-like command line interface. It + used the LZMA_Alone tool from LZMA SDK to do all the real work. zgrep, + zdiff, and related scripts from gzip were adapted work with LZMA and + were part of the first LZMA Utils release too. + + LZMA Utils 4.22.0 included also lzmadec, which was a small (less than + 10 KiB) decoder-only command line tool. It was written on top of the + decoder-only C code found from the LZMA SDK. lzmadec was convenient in + situations where LZMA_Alone (a few hundred KiB) would be too big. + + lzmash and lzmadec were written by Lasse Collin. + + +Second generation + + The lzmash script was an ugly and not very secure hack. The last + version of LZMA Utils to use lzmash was 4.27.1. + + LZMA Utils 4.32.0beta1 introduced a new lzma command line tool written + by Ville Koskinen. It was written in C++, and used the encoder and + decoder from C++ LZMA SDK with little modifications. This tool replaced + both the lzmash script and the LZMA_Alone command line tool in LZMA + Utils. + + Introducing this new tool caused some temporary incompatibilities, + because LZMA_Alone executable was simply named lzma like the new + command line tool, but they had completely different command line + interface. The file format was still the same. + + Lasse wrote liblzmadec, which was a small decoder-only library based + on the C code found from LZMA SDK. liblzmadec had API similar to zlib, + although there were some significant differences, which made it + non-trivial to use it in some applications designed for zlib and + libbzip2. + + The lzmadec command line tool was converted to use liblzmadec. + + Alexandre Sauvé helped converting build system to use GNU Autotools. + This made is easier to test for certain less portable features needed + by the new command line tool. + + Since the new command line tool never got completely finished (for + example, it didn't support LZMA_OPT environment variable), the intent + was to not call 4.32.x stable. Similarly, liblzmadec wasn't polished, + but appeared to work well enough, so some people started using it too. + + Because the development of the third generation of LZMA Utils was + delayed considerably (3-4 years), the 4.32.x branch had to be kept + maintained. It got some bug fixes now and then, and finally it was + decided to call it stable, although most of the missing features were + never added. + + +File format problems + + The file format used by LZMA_Alone was primitive. It was designed for + embedded systems in mind, and thus provided only minimal set of + features. The two biggest problems for non-embedded use were lack of + magic bytes and integrity check. + + Igor and Lasse started developing a new file format with some help + from Ville Koskinen. Also Mark Adler, Mikko Pouru, H. Peter Anvin, + and Lars Wirzenius helped with some minor things at some point of the + development. Designing the new format took quite a long time (actually, + too long time would be more appropriate expression). It was mostly + because Lasse was quite slow at getting things done due to personal + reasons. + + Originally the new format was supposed to use the same .lzma suffix + that was already used by the old file format. Switching to the new + format wouldn't have caused much trouble when the old format wasn't + used by many people. But since the development of the new format took + so long time, the old format got quite popular, and it was decided + that the new file format must use a different suffix. + + It was decided to use .xz as the suffix of the new file format. The + first stable .xz file format specification was finally released in + December 2008. In addition to fixing the most obvious problems of + the old .lzma format, the .xz format added some new features like + support for multiple filters (compression algorithms), filter chaining + (like piping on the command line), and limited random-access reading. + + Currently the primary compression algorithm used in .xz is LZMA2. + It is an extension on top of the original LZMA to fix some practical + problems: LZMA2 adds support for flushing the encoder, uncompressed + chunks, eases stateful decoder implementations, and improves support + for multithreading. Since LZMA2 is better than the original LZMA, the + original LZMA is not supported in .xz. + + +Transition to XZ Utils + + The early versions of XZ Utils were called LZMA Utils. The first + releases were 4.42.0alphas. They dropped the rest of the C++ LZMA SDK. + The code was still directly based on LZMA SDK but ported to C and + converted from callback API to stateful API. Later, Igor Pavlov made + C version of the LZMA encoder too; these ports from C++ to C were + independent in LZMA SDK and LZMA Utils. + + The core of the new LZMA Utils was liblzma, a compression library with + zlib-like API. liblzma supported both the old and new file format. The + gzip-like lzma command line tool was rewritten to use liblzma. + + The new LZMA Utils code base was renamed to XZ Utils when the name + of the new file format had been decided. The liblzma compression + library retained its name though, because changing it would have + caused unnecessary breakage in applications already using the early + liblzma snapshots. + + The xz command line tool can emulate the gzip-like lzma tool by + creating appropriate symlinks (e.g. lzma -> xz). Thus, practically + all scripts using the lzma tool from LZMA Utils will work as is with + XZ Utils (and will keep using the old .lzma format). Still, the .lzma + format is more or less deprecated. XZ Utils will keep supporting it, + but new applications should use the .xz format, and migrating old + applications to .xz is often a good idea too. + diff --git a/Build/source/utils/xz-4.999.9beta/doc/lzma-file-format.txt b/Build/source/utils/xz-4.999.9beta/doc/lzma-file-format.txt new file mode 100644 index 00000000000..21fcb19fdcd --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/lzma-file-format.txt @@ -0,0 +1,166 @@ + +The .lzma File Format +===================== + + 0. Preface + 0.1. Notices and Acknowledgements + 0.2. Changes + 1. File Format + 1.1. Header + 1.1.1. Properties + 1.1.2. Dictionary Size + 1.1.3. Uncompressed Size + 1.2. LZMA Compressed Data + 2. References + + +0. Preface + + This document describes the .lzma file format, which is + sometimes also called LZMA_Alone format. It is a legacy file + format, which is being or has been replaced by the .xz format. + The MIME type of the .lzma format is `application/x-lzma'. + + The most commonly used software to handle .lzma files are + LZMA SDK, LZMA Utils, 7-Zip, and XZ Utils. This document + describes some of the differences between these implementations + and gives hints what subset of the .lzma format is the most + portable. + + +0.1. Notices and Acknowledgements + + This file format was designed by Igor Pavlov for use in + LZMA SDK. This document was written by Lasse Collin + <lasse.collin@tukaani.org> using the documentation found + from the LZMA SDK. + + This document has been put into the public domain. + + +0.2. Changes + + Last modified: 2009-05-01 11:15+0300 + + +1. File Format + + +-+-+-+-+-+-+-+-+-+-+-+-+-+==========================+ + | Header | LZMA Compressed Data | + +-+-+-+-+-+-+-+-+-+-+-+-+-+==========================+ + + The .lzma format file consist of 13-byte Header followed by + the LZMA Compressed Data. + + Unlike the .gz, .bz2, and .xz formats, it is not possible to + concatenate multiple .lzma files as is and expect the + decompression tool to decode the resulting file as if it were + a single .lzma file. + + For example, the command line tools from LZMA Utils and + LZMA SDK silently ignore all the data after the first .lzma + stream. In contrast, the command line tool from XZ Utils + considers the .lzma file to be corrupt if there is data after + the first .lzma stream. + + +1.1. Header + + +------------+----+----+----+----+--+--+--+--+--+--+--+--+ + | Properties | Dictionary Size | Uncompressed Size | + +------------+----+----+----+----+--+--+--+--+--+--+--+--+ + + +1.1.1. Properties + + The Properties field contains three properties. An abbreviation + is given in parentheses, followed by the value range of the + property. The field consists of + + 1) the number of literal context bits (lc, [0, 8]); + 2) the number of literal position bits (lp, [0, 4]); and + 3) the number of position bits (pb, [0, 4]). + + The properties are encoded using the following formula: + + Properties = (pb * 5 + lp) * 9 + lc + + The following C code illustrates a straightforward way to + decode the Properties field: + + uint8_t lc, lp, pb; + uint8_t prop = get_lzma_properties(); + if (prop > (4 * 5 + 4) * 9 + 8) + return LZMA_PROPERTIES_ERROR; + + pb = prop / (9 * 5); + prop -= pb * 9 * 5; + lp = prop / 9; + lc = prop - lp * 9; + + XZ Utils has an additional requirement: lc + lp <= 4. Files + which don't follow this requirement cannot be decompressed + with XZ Utils. Usually this isn't a problem since the most + common lc/lp/pb values are 3/0/2. It is the only lc/lp/pb + combination that the files created by LZMA Utils can have, + but LZMA Utils can decompress files with any lc/lp/pb. + + +1.1.2. Dictionary Size + + Dictionary Size is stored as an unsigned 32-bit little endian + integer. Any 32-bit value is possible, but for maximum + portability, only sizes of 2^n and 2^n + 2^(n-1) should be + used. + + LZMA Utils creates only files with dictionary size 2^n, + 16 <= n <= 25. LZMA Utils can decompress files with any + dictionary size. + + XZ Utils creates and decompresses .lzma files only with + dictionary sizes 2^n and 2^n + 2^(n-1). If some other + dictionary size is specified when compressing, the value + stored in the Dictionary Size field is a rounded up, but the + specified value is still used in the actual compression code. + + +1.1.3. Uncompressed Size + + Uncompressed Size is stored as unsigned 64-bit little endian + integer. A special value of 0xFFFF_FFFF_FFFF_FFFF indicates + that Uncompressed Size is unknown. End of Payload Marker (*) + is used if and only if Uncompressed Size is unknown. + + XZ Utils rejects files whose Uncompressed Size field specifies + a known size that is 256 GiB or more. This is to reject false + positives when trying to guess if the input file is in the + .lzma format. When Uncompressed Size is unknown, there is no + limit for the uncompressed size of the file. + + (*) Some tools use the term End of Stream (EOS) marker + instead of End of Payload Marker. + + +1.2. LZMA Compressed Data + + Detailed description of the format of this field is out of + scope of this document. + + +2. References + + LZMA SDK - The original LZMA implementation + http://7-zip.org/sdk.html + + 7-Zip + http://7-zip.org/ + + LZMA Utils - LZMA adapted to POSIX-like systems + http://tukaani.org/lzma/ + + XZ Utils - The next generation of LZMA Utils + http://tukaani.org/xz/ + + The .xz file format - The successor of the the .lzma format + http://tukaani.org/xz/xz-file-format.txt + diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xz-a4.pdf b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xz-a4.pdf Binary files differnew file mode 100644 index 00000000000..0b6805432e5 --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xz-a4.pdf diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xzdec-a4.pdf b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xzdec-a4.pdf Binary files differnew file mode 100644 index 00000000000..92e52eec78e --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xzdec-a4.pdf diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xzdiff-a4.pdf b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xzdiff-a4.pdf Binary files differnew file mode 100644 index 00000000000..80aad7a9df2 --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xzdiff-a4.pdf diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xzgrep-a4.pdf b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xzgrep-a4.pdf Binary files differnew file mode 100644 index 00000000000..2c928946d92 --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xzgrep-a4.pdf diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xzless-a4.pdf b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xzless-a4.pdf Binary files differnew file mode 100644 index 00000000000..a6374692f4d --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xzless-a4.pdf diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xzmore-a4.pdf b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xzmore-a4.pdf Binary files differnew file mode 100644 index 00000000000..b36a5fdfe6d --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-a4/xzmore-a4.pdf diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xz-letter.pdf b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xz-letter.pdf Binary files differnew file mode 100644 index 00000000000..6df9ab66aff --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xz-letter.pdf diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xzdec-letter.pdf b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xzdec-letter.pdf Binary files differnew file mode 100644 index 00000000000..f056c2ee348 --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xzdec-letter.pdf diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xzdiff-letter.pdf b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xzdiff-letter.pdf Binary files differnew file mode 100644 index 00000000000..39bc0f0a287 --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xzdiff-letter.pdf diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xzgrep-letter.pdf b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xzgrep-letter.pdf Binary files differnew file mode 100644 index 00000000000..677186a1b44 --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xzgrep-letter.pdf diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xzless-letter.pdf b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xzless-letter.pdf Binary files differnew file mode 100644 index 00000000000..dcbd4871228 --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xzless-letter.pdf diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xzmore-letter.pdf b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xzmore-letter.pdf Binary files differnew file mode 100644 index 00000000000..e22857d6894 --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/pdf-letter/xzmore-letter.pdf diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/txt/xz.txt b/Build/source/utils/xz-4.999.9beta/doc/man/txt/xz.txt new file mode 100644 index 00000000000..e3dacde4a11 --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/txt/xz.txt @@ -0,0 +1,786 @@ +XZ(1) XZ Utils XZ(1) + + + +NAME + xz, unxz, xzcat, lzma, unlzma, lzcat - Compress or decompress .xz and + .lzma files + +SYNOPSIS + xz [option]... [file]... + + unxz is equivalent to xz --decompress. + xzcat is equivalent to xz --decompress --stdout. + lzma is equivalent to xz --format=lzma. + unlzma is equivalent to xz --format=lzma --decompress. + lzcat is equivalent to xz --format=lzma --decompress --stdout. + + When writing scripts that need to decompress files, it is recommended + to always use the name xz with appropriate arguments (xz -d or xz -dc) + instead of the names unxz and xzcat. + +DESCRIPTION + xz is a general-purpose data compression tool with command line syntax + similar to gzip(1) and bzip2(1). The native file format is the .xz + format, but also the legacy .lzma format and raw compressed streams + with no container format headers are supported. + + xz compresses or decompresses each file according to the selected oper- + ation mode. If no files are given or file is -, xz reads from standard + input and writes the processed data to standard output. xz will refuse + (display an error and skip the file) to write compressed data to stan- + dard output if it is a terminal. Similarly, xz will refuse to read com- + pressed data from standard input if it is a terminal. + + Unless --stdout is specified, files other than - are written to a new + file whose name is derived from the source file name: + + o When compressing, the suffix of the target file format (.xz or + .lzma) is appended to the source filename to get the target file- + name. + + o When decompressing, the .xz or .lzma suffix is removed from the + filename to get the target filename. xz also recognizes the suf- + fixes .txz and .tlz, and replaces them with the .tar suffix. + + If the target file already exists, an error is displayed and the file + is skipped. + + Unless writing to standard output, xz will display a warning and skip + the file if any of the following applies: + + o File is not a regular file. Symbolic links are not followed, thus + they are never considered to be regular files. + + o File has more than one hardlink. + + o File has setuid, setgid, or sticky bit set. + + o The operation mode is set to compress, and the file already has a + suffix of the target file format (.xz or .txz when compressing to + the .xz format, and .lzma or .tlz when compressing to the .lzma for- + mat). + + o The operation mode is set to decompress, and the file doesn't have a + suffix of any of the supported file formats (.xz, .txz, .lzma, or + .tlz). + + After successfully compressing or decompressing the file, xz copies the + owner, group, permissions, access time, and modification time from the + source file to the target file. If copying the group fails, the permis- + sions are modified so that the target file doesn't become accessible to + users who didn't have permission to access the source file. xz doesn't + support copying other metadata like access control lists or extended + attributes yet. + + Once the target file has been successfully closed, the source file is + removed unless --keep was specified. The source file is never removed + if the output is written to standard output. + + Sending SIGINFO or SIGUSR1 to the xz process makes it print progress + information to standard error. This has only limited use since when + standard error is a terminal, using --verbose will display an automati- + cally updating progress indicator. + + Memory usage + The memory usage of xz varies from a few hundred kilobytes to several + gigabytes depending on the compression settings. The settings used when + compressing a file affect also the memory usage of the decompressor. + Typically the decompressor needs only 5 % to 20 % of the amount of RAM + that the compressor needed when creating the file. Still, the worst- + case memory usage of the decompressor is several gigabytes. + + To prevent uncomfortable surprises caused by huge memory usage, xz has + a built-in memory usage limiter. The default limit is 40 % of total + physical RAM. While operating systems provide ways to limit the memory + usage of processes, relying on it wasn't deemed to be flexible enough. + + When compressing, if the selected compression settings exceed the mem- + ory usage limit, the settings are automatically adjusted downwards and + a notice about this is displayed. As an exception, if the memory usage + limit is exceeded when compressing with --format=raw, an error is dis- + played and xz will exit with exit status 1. + + If source file cannot be decompressed without exceeding the memory + usage limit, an error message is displayed and the file is skipped. + Note that compressed files may contain many blocks, which may have been + compressed with different settings. Typically all blocks will have + roughly the same memory requirements, but it is possible that a block + later in the file will exceed the memory usage limit, and an error + about too low memory usage limit gets displayed after some data has + already been decompressed. + + The absolute value of the active memory usage limit can be seen near + the bottom of the output of --long-help. The default limit can be + overriden with --memory=limit. + +OPTIONS + Integer suffixes and special values + In most places where an integer argument is expected, an optional suf- + fix is supported to easily indicate large integers. There must be no + space between the integer and the suffix. + + k or kB + The integer is multiplied by 1,000 (10^3). For example, 5k or + 5kB equals 5000. + + Ki or KiB + The integer is multiplied by 1,024 (2^10). + + M or MB + The integer is multiplied by 1,000,000 (10^6). + + Mi or MiB + The integer is multiplied by 1,048,576 (2^20). + + G or GB + The integer is multiplied by 1,000,000,000 (10^9). + + Gi or GiB + The integer is multiplied by 1,073,741,824 (2^30). + + A special value max can be used to indicate the maximum integer value + supported by the option. + + Operation mode + If multiple operation mode options are given, the last one takes + effect. + + -z, --compress + Compress. This is the default operation mode when no operation + mode option is specified, and no other operation mode is implied + from the command name (for example, unxz implies --decompress). + + -d, --decompress, --uncompress + Decompress. + + -t, --test + Test the integrity of compressed files. No files are created or + removed. This option is equivalent to --decompress --stdout + except that the decompressed data is discarded instead of being + written to standard output. + + -l, --list + View information about the compressed files. No uncompressed + output is produced, and no files are created or removed. In list + mode, the program cannot read the compressed data from standard + input or from other unseekable sources. + + This feature has not been implemented yet. + + Operation modifiers + -k, --keep + Keep (don't delete) the input files. + + -f, --force + This option has several effects: + + o If the target file already exists, delete it before compress- + ing or decompressing. + + o Compress or decompress even if the input is not a regular + file, has more than one hardlink, or has setuid, setgid, or + sticky bit set. The setuid, setgid, and sticky bits are not + copied to the target file. + + o If combined with --decompress --stdout and xz doesn't recog- + nize the type of the source file, xz will copy the source + file as is to standard output. This allows using xzcat + --force like cat(1) for files that have not been compressed + with xz. Note that in future, xz might support new com- + pressed file formats, which may make xz decompress more types + of files instead of copying them as is to standard output. + --format=format can be used to restrict xz to decompress only + a single file format. + + o Allow writing compressed data to a terminal, and reading com- + pressed data from a terminal. + + -c, --stdout, --to-stdout + Write the compressed or decompressed data to standard output + instead of a file. This implies --keep. + + -S .suf, --suffix=.suf + When compressing, use .suf as the suffix for the target file + instead of .xz or .lzma. If not writing to standard output and + the source file already has the suffix .suf, a warning is dis- + played and the file is skipped. + + When decompressing, recognize also files with the suffix .suf in + addition to files with the .xz, .txz, .lzma, or .tlz suffix. If + the source file has the suffix .suf, the suffix is removed to + get the target filename. + + When compressing or decompressing raw streams (--format=raw), + the suffix must always be specified unless writing to standard + output, because there is no default suffix for raw streams. + + --files[=file] + Read the filenames to process from file; if file is omitted, + filenames are read from standard input. Filenames must be termi- + nated with the newline character. If filenames are given also as + command line arguments, they are processed before the filenames + read from file. + + --files0[=file] + This is identical to --files[=file] except that the filenames + must be terminated with the null character. + + Basic file format and compression options + -F format, --format=format + Specify the file format to compress or decompress: + + o auto: This is the default. When compressing, auto is equiva- + lent to xz. When decompressing, the format of the input file + is autodetected. Note that raw streams (created with --for- + mat=raw) cannot be autodetected. + + o xz: Compress to the .xz file format, or accept only .xz files + when decompressing. + + o lzma or alone: Compress to the legacy .lzma file format, or + accept only .lzma files when decompressing. The alternative + name alone is provided for backwards compatibility with LZMA + Utils. + + o raw: Compress or uncompress a raw stream (no headers). This + is meant for advanced users only. To decode raw streams, you + need to set not only --format=raw but also specify the filter + chain, which would normally be stored in the container format + headers. + + -C check, --check=check + Specify the type of the integrity check, which is calculated + from the uncompressed data. This option has an effect only when + compressing into the .xz format; the .lzma format doesn't sup- + port integrity checks. The integrity check (if any) is verified + when the .xz file is decompressed. + + Supported check types: + + o none: Don't calculate an integrity check at all. This is usu- + ally a bad idea. This can be useful when integrity of the + data is verified by other means anyway. + + o crc32: Calculate CRC32 using the polynomial from IEEE-802.3 + (Ethernet). + + o crc64: Calculate CRC64 using the polynomial from ECMA-182. + This is the default, since it is slightly better than CRC32 + at detecting damaged files and the speed difference is negli- + gible. + + o sha256: Calculate SHA-256. This is somewhat slower than CRC32 + and CRC64. + + Integrity of the .xz headers is always verified with CRC32. It + is not possible to change or disable it. + + -0 ... -9 + Select compression preset. If a preset level is specified multi- + ple times, the last one takes effect. + + The compression preset levels can be categorised roughly into + three categories: + + -0 ... -2 + Fast presets with relatively low memory usage. -1 and -2 + should give compression speed and ratios comparable to + bzip2 -1 and bzip2 -9, respectively. Currently -0 is not + very good (not much faster than -1 but much worse com- + pression). In future, -0 may be indicate some fast algo- + rithm instead of LZMA2. + + -3 ... -5 + Good compression ratio with low to medium memory usage. + These are significantly slower than levels 0-2. + + -6 ... -9 + Excellent compression with medium to high memory usage. + These are also slower than the lower preset levels. The + default is -6. Unless you want to maximize the compres- + sion ratio, you probably don't want a higher preset level + than -7 due to speed and memory usage. + + The exact compression settings (filter chain) used by each pre- + set may vary between xz versions. The settings may also vary + between files being compressed, if xz determines that modified + settings will probably give better compression ratio without + significantly affecting compression time or memory usage. + + Because the settings may vary, the memory usage may vary too. + The following table lists the maximum memory usage of each pre- + set level, which won't be exceeded even in future versions of + xz. + + FIXME: The table below is just a rough idea. + + Preset Compression Decompression + -0 6 MiB 1 MiB + -1 6 MiB 1 MiB + -2 10 MiB 1 MiB + -3 20 MiB 2 MiB + -4 30 MiB 3 MiB + -5 60 MiB 6 MiB + -6 100 MiB 10 MiB + -7 200 MiB 20 MiB + -8 400 MiB 40 MiB + -9 800 MiB 80 MiB + + When compressing, xz automatically adjusts the compression set- + tings downwards if the memory usage limit would be exceeded, so + it is safe to specify a high preset level even on systems that + don't have lots of RAM. + + --fast and --best + These are somewhat misleading aliases for -0 and -9, respec- + tively. These are provided only for backwards compatibility + with LZMA Utils. Avoid using these options. + + Especially the name of --best is misleading, because the defini- + tion of best depends on the input data, and that usually people + don't want the very best compression ratio anyway, because it + would be very slow. + + -e, --extreme + Modify the compression preset (-0 ... -9) so that a little bit + better compression ratio can be achieved without increasing mem- + ory usage of the compressor or decompressor (exception: compres- + sor memory usage may increase a little with presets -0 ... -2). + The downside is that the compression time will increase dramati- + cally (it can easily double). + + -M limit, --memory=limit + Set the memory usage limit. If this option is specied multiple + times, the last one takes effect. The limit can be specified in + multiple ways: + + o The limit can be an absolute value in bytes. Using an integer + suffix like MiB can be useful. Example: --memory=80MiB + + o The limit can be specified as a percentage of physical RAM. + Example: --memory=70% + + o The limit can be reset back to its default value (currently + 40 % of physical RAM) by setting it to 0. + + o The memory usage limiting can be effectively disabled by set- + ting limit to max. This isn't recommended. It's usually bet- + ter to use, for example, --memory=90%. + + The current limit can be seen near the bottom of the output of + the --long-help option. + + -T threads, --threads=threads + Specify the maximum number of worker threads to use. The default + is the number of available CPU cores. You can see the current + value of threads near the end of the output of the --long-help + option. + + The actual number of worker threads can be less than threads if + using more threads would exceed the memory usage limit. In + addition to CPU-intensive worker threads, xz may use a few aux- + iliary threads, which don't use a lot of CPU time. + + Multithreaded compression and decompression are not implemented + yet, so this option has no effect for now. + + Custom compressor filter chains + A custom filter chain allows specifying the compression settings in + detail instead of relying on the settings associated to the preset lev- + els. When a custom filter chain is specified, the compression preset + level options (-0 ... -9 and --extreme) are silently ignored. + + A filter chain is comparable to piping on the UN*X command line. When + compressing, the uncompressed input goes to the first filter, whose + output goes to the next filter (if any). The output of the last filter + gets written to the compressed file. The maximum number of filters in + the chain is four, but typically a filter chain has only one or two + filters. + + Many filters have limitations where they can be in the filter chain: + some filters can work only as the last filter in the chain, some only + as a non-last filter, and some work in any position in the chain. + Depending on the filter, this limitation is either inherent to the fil- + ter design or exists to prevent security issues. + + A custom filter chain is specified by using one or more filter options + in the order they are wanted in the filter chain. That is, the order of + filter options is significant! When decoding raw streams (--for- + mat=raw), the filter chain is specified in the same order as it was + specified when compressing. + + Filters take filter-specific options as a comma-separated list. Extra + commas in options are ignored. Every option has a default value, so you + need to specify only those you want to change. + + --lzma1[=options], --lzma2[=options] + Add LZMA1 or LZMA2 filter to the filter chain. These filter can + be used only as the last filter in the chain. + + LZMA1 is a legacy filter, which is supported almost solely due + to the legacy .lzma file format, which supports only LZMA1. + LZMA2 is an updated version of LZMA1 to fix some practical + issues of LZMA1. The .xz format uses LZMA2, and doesn't support + LZMA1 at all. Compression speed and ratios of LZMA1 and LZMA2 + are practically the same. + + LZMA1 and LZMA2 share the same set of options: + + preset=preset + Reset all LZMA1 or LZMA2 options to preset. Preset con- + sist of an integer, which may be followed by single-let- + ter preset modifiers. The integer can be from 0 to 9, + matching the command line options -0 ... -9. The only + supported modifier is currently e, which matches + --extreme. + + The default preset is 6, from which the default values + for the rest of the LZMA1 or LZMA2 options are taken. + + dict=size + Dictionary (history buffer) size indicates how many bytes + of the recently processed uncompressed data is kept in + memory. One method to reduce size of the uncompressed + data is to store distance-length pairs, which indicate + what data to repeat from the dictionary buffer. The big- + ger the dictionary, the better the compression ratio usu- + ally is, but dictionaries bigger than the uncompressed + data are waste of RAM. + + Typical dictionary size is from 64 KiB to 64 MiB. The + minimum is 4 KiB. The maximum for compression is cur- + rently 1.5 GiB. The decompressor already supports dictio- + naries up to one byte less than 4 GiB, which is the maxi- + mum for LZMA1 and LZMA2 stream formats. + + Dictionary size has the biggest effect on compression + ratio. Dictionary size and match finder together deter- + mine the memory usage of the LZMA1 or LZMA2 encoder. The + same dictionary size is required for decompressing that + was used when compressing, thus the memory usage of the + decoder is determined by the dictionary size used when + compressing. + + lc=lc Specify the number of literal context bits. The minimum + is 0 and the maximum is 4; the default is 3. In addi- + tion, the sum of lc and lp must not exceed 4. + + lp=lp Specify the number of literal position bits. The minimum + is 0 and the maximum is 4; the default is 0. + + pb=pb Specify the number of position bits. The minimum is 0 and + the maximum is 4; the default is 2. + + mode=mode + Compression mode specifies the function used to analyze + the data produced by the match finder. Supported modes + are fast and normal. The default is fast for presets 0-2 + and normal for presets 3-9. + + mf=mf Match finder has a major effect on encoder speed, memory + usage, and compression ratio. Usually Hash Chain match + finders are faster than Binary Tree match finders. Hash + Chains are usually used together with mode=fast and + Binary Trees with mode=normal. The memory usage formulas + are only rough estimates, which are closest to reality + when dict is a power of two. + + hc3 Hash Chain with 2- and 3-byte hashing + Minimum value for nice: 3 + Memory usage: dict * 7.5 (if dict <= 16 MiB); + dict * 5.5 + 64 MiB (if dict > 16 MiB) + + hc4 Hash Chain with 2-, 3-, and 4-byte hashing + Minimum value for nice: 4 + Memory usage: dict * 7.5 + + bt2 Binary Tree with 2-byte hashing + Minimum value for nice: 2 + Memory usage: dict * 9.5 + + bt3 Binary Tree with 2- and 3-byte hashing + Minimum value for nice: 3 + Memory usage: dict * 11.5 (if dict <= 16 MiB); + dict * 9.5 + 64 MiB (if dict > 16 MiB) + + bt4 Binary Tree with 2-, 3-, and 4-byte hashing + Minimum value for nice: 4 + Memory usage: dict * 11.5 + + nice=nice + Specify what is considered to be a nice length for a + match. Once a match of at least nice bytes is found, the + algorithm stops looking for possibly better matches. + + nice can be 2-273 bytes. Higher values tend to give bet- + ter compression ratio at expense of speed. The default + depends on the preset level. + + depth=depth + Specify the maximum search depth in the match finder. The + default is the special value 0, which makes the compres- + sor determine a reasonable depth from mf and nice. + + Using very high values for depth can make the encoder + extremely slow with carefully crafted files. Avoid set- + ting the depth over 1000 unless you are prepared to + interrupt the compression in case it is taking too long. + + When decoding raw streams (--format=raw), LZMA2 needs only the + value of dict. LZMA1 needs also lc, lp, and pb. + + --x86[=options] + + --powerpc[=options] + + --ia64[=options] + + --arm[=options] + + --armthumb[=options] + + --sparc[=options] + Add a branch/call/jump (BCJ) filter to the filter chain. These + filters can be used only as non-last filter in the filter chain. + + A BCJ filter converts relative addresses in the machine code to + their absolute counterparts. This doesn't change the size of the + data, but it increases redundancy, which allows e.g. LZMA2 to + get better compression ratio. + + The BCJ filters are always reversible, so using a BCJ filter for + wrong type of data doesn't cause any data loss. However, apply- + ing a BCJ filter for wrong type of data is a bad idea, because + it tends to make the compression ratio worse. + + Different instruction sets have have different alignment: + + Filter Alignment Notes + x86 1 32-bit and 64-bit x86 + PowerPC 4 Big endian only + ARM 4 Little endian only + ARM-Thumb 2 Little endian only + IA-64 16 Big or little endian + SPARC 4 Big or little endian + + Since the BCJ-filtered data is usually compressed with LZMA2, + the compression ratio may be improved slightly if the LZMA2 + options are set to match the alignment of the selected BCJ fil- + ter. For example, with the IA-64 filter, it's good to set pb=4 + with LZMA2 (2^4=16). The x86 filter is an exception; it's usu- + ally good to stick to LZMA2's default four-byte alignment when + compressing x86 executables. + + All BCJ filters support the same options: + + start=offset + Specify the start offset that is used when converting + between relative and absolute addresses. The offset must + be a multiple of the alignment of the filter (see the ta- + ble above). The default is zero. In practice, the + default is good; specifying a custom offset is almost + never useful. + + Specifying a non-zero start offset is probably useful + only if the executable has multiple sections, and there + are many cross-section jumps or calls. Applying a BCJ + filter separately for each section with proper start off- + set and then compressing the result as a single chunk may + give some improvement in compression ratio compared to + applying the BCJ filter with the default offset for the + whole executable. + + --delta[=options] + Add Delta filter to the filter chain. The Delta filter can be + used only as non-last filter in the filter chain. + + Currently only simple byte-wise delta calculation is supported. + It can be useful when compressing e.g. uncompressed bitmap + images or uncompressed PCM audio. However, special purpose algo- + rithms may give significantly better results than Delta + LZMA2. + This is true especially with audio, which compresses faster and + better e.g. with FLAC. + + Supported options: + + dist=distance + Specify the distance of the delta calculation as bytes. + distance must be 1-256. The default is 1. + + For example, with dist=2 and eight-byte input A1 B1 A2 B3 + A3 B5 A4 B7, the output will be A1 B1 01 02 01 02 01 02. + + Other options + -q, --quiet + Suppress warnings and notices. Specify this twice to suppress + errors too. This option has no effect on the exit status. That + is, even if a warning was suppressed, the exit status to indi- + cate a warning is still used. + + -v, --verbose + Be verbose. If standard error is connected to a terminal, xz + will display a progress indicator. Specifying --verbose twice + will give even more verbose output (useful mostly for debug- + ging). + + -Q, --no-warn + Don't set the exit status to 2 even if a condition worth a warn- + ing was detected. This option doesn't affect the verbosity + level, thus both --quiet and --no-warn have to be used to not + display warnings and to not alter the exit status. + + -h, --help + Display a help message describing the most commonly used + options, and exit successfully. + + -H, --long-help + Display a help message describing all features of xz, and exit + successfully + + -V, --version + Display the version number of xz and liblzma. + +EXIT STATUS + 0 All is good. + + 1 An error occurred. + + 2 Something worth a warning occurred, but no actual errors + occurred. + + Notices (not warnings or errors) printed on standard error don't affect + the exit status. + +ENVIRONMENT + XZ_OPT A space-separated list of options is parsed from XZ_OPT before + parsing the options given on the command line. Note that only + options are parsed from XZ_OPT; all non-options are silently + ignored. Parsing is done with getopt_long(3) which is used also + for the command line arguments. + +LZMA UTILS COMPATIBILITY + The command line syntax of xz is practically a superset of lzma, + unlzma, and lzcat as found from LZMA Utils 4.32.x. In most cases, it is + possible to replace LZMA Utils with XZ Utils without breaking existing + scripts. There are some incompatibilities though, which may sometimes + cause problems. + + Compression preset levels + The numbering of the compression level presets is not identical in xz + and LZMA Utils. The most important difference is how dictionary sizes + are mapped to different presets. Dictionary size is roughly equal to + the decompressor memory usage. + + Level xz LZMA Utils + -1 64 KiB 64 KiB + -2 512 KiB 1 MiB + -3 1 MiB 512 KiB + -4 2 MiB 1 MiB + -5 4 MiB 2 MiB + -6 8 MiB 4 MiB + -7 16 MiB 8 MiB + -8 32 MiB 16 MiB + -9 64 MiB 32 MiB + + The dictionary size differences affect the compressor memory usage too, + but there are some other differences between LZMA Utils and XZ Utils, + which make the difference even bigger: + + Level xz LZMA Utils 4.32.x + -1 2 MiB 2 MiB + -2 5 MiB 12 MiB + -3 13 MiB 12 MiB + -4 25 MiB 16 MiB + -5 48 MiB 26 MiB + -6 94 MiB 45 MiB + -7 186 MiB 83 MiB + -8 370 MiB 159 MiB + -9 674 MiB 311 MiB + + The default preset level in LZMA Utils is -7 while in XZ Utils it is + -6, so both use 8 MiB dictionary by default. + + Streamed vs. non-streamed .lzma files + Uncompressed size of the file can be stored in the .lzma header. LZMA + Utils does that when compressing regular files. The alternative is to + mark that uncompressed size is unknown and use end of payload marker to + indicate where the decompressor should stop. LZMA Utils uses this + method when uncompressed size isn't known, which is the case for exam- + ple in pipes. + + xz supports decompressing .lzma files with or without end of payload + marker, but all .lzma files created by xz will use end of payload + marker and have uncompressed size marked as unknown in the .lzma + header. This may be a problem in some (uncommon) situations. For exam- + ple, a .lzma decompressor in an embedded device might work only with + files that have known uncompressed size. If you hit this problem, you + need to use LZMA Utils or LZMA SDK to create .lzma files with known + uncompressed size. + + Unsupported .lzma files + The .lzma format allows lc values up to 8, and lp values up to 4. LZMA + Utils can decompress files with any lc and lp, but always creates files + with lc=3 and lp=0. Creating files with other lc and lp is possible + with xz and with LZMA SDK. + + The implementation of the LZMA1 filter in liblzma requires that the sum + of lc and lp must not exceed 4. Thus, .lzma files which exceed this + limitation, cannot be decompressed with xz. + + LZMA Utils creates only .lzma files which have dictionary size of 2^n + (a power of 2), but accepts files with any dictionary size. liblzma + accepts only .lzma files which have dictionary size of 2^n or 2^n + + 2^(n-1). This is to decrease false positives when autodetecting .lzma + files. + + These limitations shouldn't be a problem in practice, since practically + all .lzma files have been compressed with settings that liblzma will + accept. + + Trailing garbage + When decompressing, LZMA Utils silently ignore everything after the + first .lzma stream. In most situations, this is a bug. This also means + that LZMA Utils don't support decompressing concatenated .lzma files. + + If there is data left after the first .lzma stream, xz considers the + file to be corrupt. This may break obscure scripts which have assumed + that trailing garbage is ignored. + +NOTES + Compressed output may vary + The exact compressed output produced from the same uncompressed input + file may vary between XZ Utils versions even if compression options are + identical. This is because the encoder can be improved (faster or bet- + ter compression) without affecting the file format. The output can vary + even between different builds of the same XZ Utils version, if differ- + ent build options are used or if the endianness of the hardware is dif- + ferent for different builds. + + The above means that implementing --rsyncable to create rsyncable .xz + files is not going to happen without freezing a part of the encoder + implementation, which can then be used with --rsyncable. + + Embedded .xz decompressors + Embedded .xz decompressor implementations like XZ Embedded don't neces- + sarily support files created with check types other than none and + crc32. Since the default is --check=crc64, you must use --check=none + or --check=crc32 when creating files for embedded systems. + + Outside embedded systems, all .xz format decompressors support all the + check types, or at least are able to decompress the file without veri- + fying the integrity check if the particular check is not supported. + + XZ Embedded supports BCJ filters, but only with the default start off- + set. + +SEE ALSO + xzdec(1), gzip(1), bzip2(1) + + XZ Utils: <http://tukaani.org/xz/> + XZ Embedded: <http://tukaani.org/xz/embedded.html> + LZMA SDK: <http://7-zip.org/sdk.html> + + + +Tukaani 2009-08-27 XZ(1) diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/txt/xzdec.txt b/Build/source/utils/xz-4.999.9beta/doc/man/txt/xzdec.txt new file mode 100644 index 00000000000..ee2b820adad --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/txt/xzdec.txt @@ -0,0 +1,95 @@ +XZDEC(1) XZ Utils XZDEC(1) + + + +NAME + xzdec, lzmadec - Small .xz and .lzma decompressors + +SYNOPSIS + xzdec [option]... [file]... + lzmadec [option]... [file]... + +DESCRIPTION + xzdec is a liblzma-based decompression-only tool for .xz (and only .xz) + files. xzdec is intended to work as a drop-in replacement for xz(1) in + the most common situations where a script has been written to use xz + --decompress --stdout (and possibly a few other commonly used options) + to decompress .xz files. lzmadec is identical to xzdec except that + lzmadec supports .lzma files instead of .xz files. + + To reduce the size of the executable, xzdec doesn't support multi- + threading or localization, and doesn't read options from XZ_OPT envi- + ronment variable. xzdec doesn't support displaying intermediate + progress information: sending SIGINFO to xzdec does nothing, but send- + ing SIGUSR1 terminates the process instead of displaying progress + information. + +OPTIONS + -d, --decompress, --uncompress + Ignored for xz(1) compatibility. xzdec supports only decompres- + sion. + + -k, --keep + Ignored for xz(1) compatibility. xzdec never creates or removes + any files. + + -c, --stdout, --to-stdout + Ignored for xz(1) compatibility. xzdec always writes the decom- + pressed data to standard output. + + -M limit, --memory=limit + Set the memory usage limit. If this option is specified multi- + ple times, the last one takes effect. The limit can be specified + in multiple ways: + + o The limit can be an absolute value in bytes. Using an integer + suffix like MiB can be useful. Example: --memory=80MiB + + o The limit can be specified as a percentage of physical RAM. + Example: --memory=70% + + o The limit can be reset back to its default value (currently + 40 % of physical RAM) by setting it to 0. + + o The memory usage limiting can be effectively disabled by set- + ting limit to max. This isn't recommended. It's usually bet- + ter to use, for example, --memory=90%. + + The current limit can be seen near the bottom of the output of + the --help option. + + -q, --quiet + Specifying this once does nothing since xzdec never displays any + warnings or notices. Specify this twice to suppress errors. + + -Q, --no-warn + Ignored for xz(1) compatibility. xzdec never uses the exit sta- + tus 2. + + -h, --help + Display a help message and exit successfully. + + -V, --version + Display the version number of xzdec and liblzma. + +EXIT STATUS + 0 All was good. + + 1 An error occurred. + + xzdec doesn't have any warning messages like xz(1) has, thus the exit + status 2 is not used by xzdec. + +NOTES + xzdec and lzmadec are not really that small. The size can be reduced + further by dropping features from liblzma at compile time, but that + shouldn't usually be done for executables distributed in typical non- + embedded operating system distributions. If you need a truly small .xz + decompressor, consider using XZ Embedded. + +SEE ALSO + xz(1) + + + +Tukaani 2009-06-04 XZDEC(1) diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/txt/xzdiff.txt b/Build/source/utils/xz-4.999.9beta/doc/man/txt/xzdiff.txt new file mode 100644 index 00000000000..f64568f2ae3 --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/txt/xzdiff.txt @@ -0,0 +1,36 @@ +XZDIFF(1) XZ Utils XZDIFF(1) + + + +NAME + xzcmp, xzdiff, lzcmp, lzdiff - compare compressed files + +SYNOPSIS + xzcmp [cmp_options] file1 [file2] + xzdiff [diff_options] file1 [file2] + lzcmp [cmp_options] file1 [file2] + lzdiff [diff_options] file1 [file2] + +DESCRIPTION + xzcmp and xdiff invoke cmp(1) or diff(1) on files compressed with + xz(1), lzma(1), gzip(1), or bzip2(1). All options specified are passed + directly to cmp or diff. If only one file is specified, then the files + compared are file1 (which must have a suffix of a supported compression + format) and file1 from which the compression format suffix has been + stripped. If two files are specified, then they are uncompressed if + necessary and fed to cmp(1) or diff(1). The exit status from cmp or + diff is preserved. + + The names lzcmp and lzdiff are provided for backward compatibility with + LZMA Utils. + +SEE ALSO + cmp(1), diff(1), xz(1), gzip(1), bzip2(1), zdiff(1) + +BUGS + Messages from the cmp(1) or diff(1) programs refer to temporary file- + names instead of those specified. + + + +Tukaani 2009-07-05 XZDIFF(1) diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/txt/xzgrep.txt b/Build/source/utils/xz-4.999.9beta/doc/man/txt/xzgrep.txt new file mode 100644 index 00000000000..7f665bce295 --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/txt/xzgrep.txt @@ -0,0 +1,39 @@ +XZGREP(1) XZ Utils XZGREP(1) + + + +NAME + xzgrep - search compressed files for a regular expression + +SYNOPSIS + xzgrep [grep_options] [-e] pattern file... + xzegrep ... + xzfgrep ... + lzgrep ... + lzegrep ... + lzfgrep ... + +DESCRIPTION + xzgrep invokes grep(1) on files which may be either uncompressed or + compressed with xz(1), lzma(1), gzip(1), or bzip2(1). All options + specified are passed directly to grep(1). + + If no file is specified, then the standard input is decompressed if + necessary and fed to grep(1). When reading from standard input, + gzip(1) and bzip2(1) compressed files are not supported. + + If xzgrep is invoked as xzegrep or xzfgrep then egrep(1) or fgrep(1) is + used instead of grep(1). The same applies to names lzgrep, lzegrep, + and lzfgrep, which are provided for backward compatibility with LZMA + Utils. + +ENVIRONMENT + GREP If the GREP environment variable is set, xzgrep uses it instead + of grep(1), egrep(1), or fgrep(1). + +SEE ALSO + grep(1), xz(1), gzip(1), bzip2(1), zgrep(1) + + + +Tukaani 2009-07-05 XZGREP(1) diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/txt/xzless.txt b/Build/source/utils/xz-4.999.9beta/doc/man/txt/xzless.txt new file mode 100644 index 00000000000..2f3dc9fe2c4 --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/txt/xzless.txt @@ -0,0 +1,40 @@ +XZLESS(1) XZ Utils XZLESS(1) + + + +NAME + xzless, lzless - view xz or lzma compressed (text) files + +SYNOPSIS + xzless [file...] + lzless [file...] + +DESCRIPTION + xzless is a filter that displays pagefulls of uncompressed text from + compressed file(s) to a terminal. It works on files compressed with + xz(1) or lzma(1). If no files are given, xzless reads from standard + input. + + xzless uses less(1) as its only pager. Unlike xzmore, the choice of + pagers is not alterable by an environment variable. Commands are based + on both more(1) and vi(1), and allow back and forth movement and + searching. See the less(1) manual for more information. + + The command named lzless is provided for backward compatibility with + LZMA Utils. + +ENVIRONMENT + LESSMETACHARS + A list of characters special to the shell. Set by xzless unless + it is already set in the environment. + + LESSOPEN + Set to a command line to invoke the xz(1) decompressor for pre- + processing the input files to less(1). + +SEE ALSO + less(1), xz(1), xzmore(1), zless(1) + + + +Tukaani 2009-07-05 XZLESS(1) diff --git a/Build/source/utils/xz-4.999.9beta/doc/man/txt/xzmore.txt b/Build/source/utils/xz-4.999.9beta/doc/man/txt/xzmore.txt new file mode 100644 index 00000000000..6f6cfe93eea --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/man/txt/xzmore.txt @@ -0,0 +1,34 @@ +XZMORE(1) XZ Utils XZMORE(1) + + + +NAME + xzmore, lzmore - view xz or lzma compressed (text) files + +SYNOPSIS + xzmore [filename ...] + lzmore [filename ...] + +DESCRIPTION + xzmore is a filter which allows examination of xz(1) or lzma(1) com- + pressed text files one screenful at a time on a soft-copy terminal. + + To use a pager other than the default more, set environment variable + PAGER to the name of the desired program. The name lzmore is provided + for backward compatibility with LZMA Utils. + + e or q When the prompt --More--(Next file: file) is printed, this com- + mand causes xzmore to exit. + + s When the prompt --More--(Next file: file) is printed, this com- + mand causes xzmore to skip the next file and continue. + + For list of keyboard commands supported while actually viewing the con- + tent of a file, refer to manual of the pager you use, usually more(1). + +SEE ALSO + more(1), xz(1), xzless(1), zmore(1) + + + +Tukaani 2009-07-05 XZMORE(1) diff --git a/Build/source/utils/xz-4.999.9beta/doc/xz-file-format.txt b/Build/source/utils/xz-4.999.9beta/doc/xz-file-format.txt new file mode 100644 index 00000000000..4ed66506018 --- /dev/null +++ b/Build/source/utils/xz-4.999.9beta/doc/xz-file-format.txt @@ -0,0 +1,1150 @@ + +The .xz File Format +=================== + +Version 1.0.4 (2009-08-27) + + + 0. Preface + 0.1. Notices and Acknowledgements + 0.2. Getting the Latest Version + 0.3. Version History + 1. Conventions + 1.1. Byte and Its Representation + 1.2. Multibyte Integers + 2. Overall Structure of .xz File + 2.1. Stream + 2.1.1. Stream Header + 2.1.1.1. Header Magic Bytes + 2.1.1.2. Stream Flags + 2.1.1.3. CRC32 + 2.1.2. Stream Footer + 2.1.2.1. CRC32 + 2.1.2.2. Backward Size + 2.1.2.3. Stream Flags + 2.1.2.4. Footer Magic Bytes + 2.2. Stream Padding + 3. Block + 3.1. Block Header + 3.1.1. Block Header Size + 3.1.2. Block Flags + 3.1.3. Compressed Size + 3.1.4. Uncompressed Size + 3.1.5. List of Filter Flags + 3.1.6. Header Padding + 3.1.7. CRC32 + 3.2. Compressed Data + 3.3. Block Padding + 3.4. Check + 4. Index + 4.1. Index Indicator + 4.2. Number of Records + 4.3. List of Records + 4.3.1. Unpadded Size + 4.3.2. Uncompressed Size + 4.4. Index Padding + 4.5. CRC32 + 5. Filter Chains + 5.1. Alignment + 5.2. Security + 5.3. Filters + 5.3.1. LZMA2 + 5.3.2. Branch/Call/Jump Filters for Executables + 5.3.3. Delta + 5.3.3.1. Format of the Encoded Output + 5.4. Custom Filter IDs + 5.4.1. Reserved Custom Filter ID Ranges + 6. Cyclic Redundancy Checks + 7. References + + +0. Preface + + This document describes the .xz file format (filename suffix + ".xz", MIME type "application/x-xz"). It is intended that this + this format replace the old .lzma format used by LZMA SDK and + LZMA Utils. + + +0.1. Notices and Acknowledgements + + This file format was designed by Lasse Collin + <lasse.collin@tukaani.org> and Igor Pavlov. + + Special thanks for helping with this document goes to + Ville Koskinen. Thanks for helping with this document goes to + Mark Adler, H. Peter Anvin, Mikko Pouru, and Lars Wirzenius. + + This document has been put into the public domain. + + +0.2. Getting the Latest Version + + The latest official version of this document can be downloaded + from <http://tukaani.org/xz/xz-file-format.txt>. + + Specific versions of this document have a filename + xz-file-format-X.Y.Z.txt where X.Y.Z is the version number. + For example, the version 1.0.0 of this document is available + at <http://tukaani.org/xz/xz-file-format-1.0.0.txt>. + + +0.3. Version History + + Version Date Description + + 1.0.4 2009-08-27 Language improvements in Sections 1.2, + 2.1.1.2, 3.1.1, 3.1.2, and 5.3.1 + + 1.0.3 2009-06-05 Spelling fixes in Sections 5.1 and 5.4 + + 1.0.2 2009-06-04 Typo fixes in Sections 4 and 5.3.1 + + 1.0.1 2009-06-01 Typo fix in Section 0.3 and minor + clarifications to Sections 2, 2.2, + 3.3, 4.4, and 5.3.2 + + 1.0.0 2009-01-14 The first official version + + +1. Conventions + + The key words "MUST", "MUST NOT", "REQUIRED", "SHOULD", + "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in [RFC-2119]. + + Indicating a warning means displaying a message, returning + appropriate exit status, or doing something else to let the + user know that something worth warning occurred. The operation + SHOULD still finish if a warning is indicated. + + Indicating an error means displaying a message, returning + appropriate exit status, or doing something else to let the + user know that something prevented successfully finishing the + operation. The operation MUST be aborted once an error has + been indicated. + + +1.1. Byte and Its Representation + + In this document, byte is always 8 bits. + + A "null byte" has all bits unset. That is, the value of a null + byte is 0x00. + + To represent byte blocks, this document uses notation that + is similar to the notation used in [RFC-1952]: + + +-------+ + | Foo | One byte. + +-------+ + + +---+---+ + | Foo | Two bytes; that is, some of the vertical bars + +---+---+ can be missing. + + +=======+ + | Foo | Zero or more bytes. + +=======+ + + In this document, a boxed byte or a byte sequence declared + using this notation is called "a field". The example field + above would be called "the Foo field" or plain "Foo". + + If there are many fields, they may be split to multiple lines. + This is indicated with an arrow ("--->"): + + +=====+ + | Foo | + +=====+ + + +=====+ + ---> | Bar | + +=====+ + + The above is equivalent to this: + + +=====+=====+ + | Foo | Bar | + +=====+=====+ + + +1.2. Multibyte Integers + + Multibyte integers of static length, such as CRC values, + are stored in little endian byte order (least significant + byte first). + + When smaller values are more likely than bigger values (for + example file sizes), multibyte integers are encoded in a + variable-length representation: + - Numbers in the range [0, 127] are copied as is, and take + one byte of space. + - Bigger numbers will occupy two or more bytes. All but the + last byte of the multibyte representation have the highest + (eighth) bit set. + + For now, the value of the variable-length integers is limited + to 63 bits, which limits the encoded size of the integer to + nine bytes. These limits may be increased in the future if + needed. + + The following C code illustrates encoding and decoding of + variable-length integers. The functions return the number of + bytes occupied by the integer (1-9), or zero on error. + + #include <stddef.h> + #include <inttypes.h> + + size_t + encode(uint8_t buf[static 9], uint64_t num) + { + if (num > UINT64_MAX / 2) + return 0; + + size_t i = 0; + + while (num >= 0x80) { + buf[i++] = (uint8_t)(num) | 0x80; + num >>= 7; + } + + buf[i++] = (uint8_t)(num); + + return i; + } + + size_t + decode(const uint8_t buf[], size_t size_max, uint64_t *num) + { + if (size_max == 0) + return 0; + + if (size_max > 9) + size_max = 9; + + *num = buf[0] & 0x7F; + size_t i = 0; + + while (buf[i++] & 0x80) { + if (i >= size_max || buf[i] == 0x00) + return 0; + + *num |= (uint64_t)(buf[i] & 0x7F) << (i * 7); + } + + return i; + } + + +2. Overall Structure of .xz File + + A standalone .xz files consist of one or more Streams which may + have Stream Padding between or after them: + + +========+================+========+================+ + | Stream | Stream Padding | Stream | Stream Padding | ... + +========+================+========+================+ + + The sizes of Stream and Stream Padding are always multiples + of four bytes, thus the size of every valid .xz file MUST be + a multiple of four bytes. + + While a typical file contains only one Stream and no Stream + Padding, a decoder handling standalone .xz files SHOULD support + files that have more than one Stream or Stream Padding. + + In contrast to standalone .xz files, when the .xz file format + is used as an internal part of some other file format or + communication protocol, it usually is expected that the decoder + stops after the first Stream, and doesn't look for Stream + Padding or possibly other Streams. + + +2.1. Stream + + +-+-+-+-+-+-+-+-+-+-+-+-+=======+=======+ +=======+ + | Stream Header | Block | Block | ... | Block | + +-+-+-+-+-+-+-+-+-+-+-+-+=======+=======+ +=======+ + + +=======+-+-+-+-+-+-+-+-+-+-+-+-+ + ---> | Index | Stream Footer | + +=======+-+-+-+-+-+-+-+-+-+-+-+-+ + + All the above fields have a size that is a multiple of four. If + Stream is used as an internal part of another file format, it + is RECOMMENDED to make the Stream start at an offset that is + a multiple of four bytes. + + Stream Header, Index, and Stream Footer are always present in + a Stream. The maximum size of the Index field is 16 GiB (2^34). + + There are zero or more Blocks. The maximum number of Blocks is + limited only by the maximum size of the Index field. + + Total size of a Stream MUST be less than 8 EiB (2^63 bytes). + The same limit applies to the total amount of uncompressed + data stored in a Stream. + + If an implementation supports handling .xz files with multiple + concatenated Streams, it MAY apply the above limits to the file + as a whole instead of limiting per Stream basis. + + +2.1.1. Stream Header + + +---+---+---+---+---+---+-------+------+--+--+--+--+ + | Header Magic Bytes | Stream Flags | CRC32 | + +---+---+---+---+---+---+-------+------+--+--+--+--+ + + +2.1.1.1. Header Magic Bytes + + The first six (6) bytes of the Stream are so called Header + Magic Bytes. They can be used to identify the file type. + + Using a C array and ASCII: + const uint8_t HEADER_MAGIC[6] + = { 0xFD, '7', 'z', 'X', 'Z', 0x00 }; + + In plain hexadecimal: + FD 37 7A 58 5A 00 + + Notes: + - The first byte (0xFD) was chosen so that the files cannot + be erroneously detected as being in .lzma format, in which + the first byte is in the range [0x00, 0xE0]. + - The sixth byte (0x00) was chosen to prevent applications + from misdetecting the file as a text file. + + If the Header Magic Bytes don't match, the decoder MUST + indicate an error. + + +2.1.1.2. Stream Flags + + The first byte of Stream Flags is always a null byte. In the + future, this byte may be used to indicate a new Stream version + or other Stream properties. + + The second byte of Stream Flags is a bit field: + + Bit(s) Mask Description + 0-3 0x0F Type of Check (see Section 3.4): + ID Size Check name + 0x00 0 bytes None + 0x01 4 bytes CRC32 + 0x02 4 bytes (Reserved) + 0x03 4 bytes (Reserved) + 0x04 8 bytes CRC64 + 0x05 8 bytes (Reserved) + 0x06 8 bytes (Reserved) + 0x07 16 bytes (Reserved) + 0x08 16 bytes (Reserved) + 0x09 16 bytes (Reserved) + 0x0A 32 bytes SHA-256 + 0x0B 32 bytes (Reserved) + 0x0C 32 bytes (Reserved) + 0x0D 64 bytes (Reserved) + 0x0E 64 bytes (Reserved) + 0x0F 64 bytes (Reserved) + 4-7 0xF0 Reserved for future use; MUST be zero for now. + + Implementations SHOULD support at least the Check IDs 0x00 + (None) and 0x01 (CRC32). Supporting other Check IDs is + OPTIONAL. If an unsupported Check is used, the decoder SHOULD + indicate a warning or error. + + If any reserved bit is set, the decoder MUST indicate an error. + It is possible that there is a new field present which the + decoder is not aware of, and can thus parse the Stream Header + incorrectly. + + +2.1.1.3. CRC32 + + The CRC32 is calculated from the Stream Flags field. It is + stored as an unsigned 32-bit little endian integer. If the + calculated value does not match the stored one, the decoder + MUST indicate an error. + + The idea is that Stream Flags would always be two bytes, even + if new features are needed. This way old decoders will be able + to verify the CRC32 calculated from Stream Flags, and thus + distinguish between corrupt files (CRC32 doesn't match) and + files that the decoder doesn't support (CRC32 matches but + Stream Flags has reserved bits set). + + +2.1.2. Stream Footer + + +-+-+-+-+---+---+---+---+-------+------+----------+---------+ + | CRC32 | Backward Size | Stream Flags | Footer Magic Bytes | + +-+-+-+-+---+---+---+---+-------+------+----------+---------+ + + +2.1.2.1. CRC32 + + The CRC32 is calculated from the Backward Size and Stream Flags + fields. It is stored as an unsigned 32-bit little endian + integer. If the calculated value does not match the stored one, + the decoder MUST indicate an error. + + The reason to have the CRC32 field before the Backward Size and + Stream Flags fields is to keep the four-byte fields aligned to + a multiple of four bytes. + + +2.1.2.2. Backward Size + + Backward Size is stored as a 32-bit little endian integer, + which indicates the size of the Index field as multiple of + four bytes, minimum value being four bytes: + + real_backward_size = (stored_backward_size + 1) * 4; + + If the stored value does not match the real size of the Index + field, the decoder MUST indicate an error. + + Using a fixed-size integer to store Backward Size makes + it slightly simpler to parse the Stream Footer when the + application needs to parse the Stream backwards. + + +2.1.2.3. Stream Flags + + This is a copy of the Stream Flags field from the Stream + Header. The information stored to Stream Flags is needed + when parsing the Stream backwards. The decoder MUST compare + the Stream Flags fields in both Stream Header and Stream + Footer, and indicate an error if they are not identical. + + +2.1.2.4. Footer Magic Bytes + + As the last step of the decoding process, the decoder MUST + verify the existence of Footer Magic Bytes. If they don't + match, an error MUST be indicated. + + Using a C array and ASCII: + const uint8_t FOOTER_MAGIC[2] = { 'Y', 'Z' }; + + In hexadecimal: + 59 5A + + The primary reason to have Footer Magic Bytes is to make + it easier to detect incomplete files quickly, without + uncompressing. If the file does not end with Footer Magic Bytes + (excluding Stream Padding described in Section 2.2), it cannot + be undamaged, unless someone has intentionally appended garbage + after the end of the Stream. + + +2.2. Stream Padding + + Only the decoders that support decoding of concatenated Streams + MUST support Stream Padding. + + Stream Padding MUST contain only null bytes. To preserve the + four-byte alignment of consecutive Streams, the size of Stream + Padding MUST be a multiple of four bytes. Empty Stream Padding + is allowed. If these requirements are not met, the decoder MUST + indicate an error. + + Note that non-empty Stream Padding is allowed at the end of the + file; there doesn't need to be a new Stream after non-empty + Stream Padding. This can be convenient in certain situations + [GNU-tar]. + + The possibility of Stream Padding MUST be taken into account + when designing an application that parses Streams backwards, + and the application supports concatenated Streams. + + +3. Block + + +==============+=================+===============+=======+ + | Block Header | Compressed Data | Block Padding | Check | + +==============+=================+===============+=======+ + + +3.1. Block Header + + +-------------------+-------------+=================+ + | Block Header Size | Block Flags | Compressed Size | + +-------------------+-------------+=================+ + + +===================+======================+ + ---> | Uncompressed Size | List of Filter Flags | + +===================+======================+ + + +================+--+--+--+--+ + ---> | Header Padding | CRC32 | + +================+--+--+--+--+ + + +3.1.1. Block Header Size + + This field overlaps with the Index Indicator field (see + Section 4.1). + + This field contains the size of the Block Header field, + including the Block Header Size field itself. Valid values are + in the range [0x01, 0xFF], which indicate the size of the Block + Header as multiples of four bytes, minimum size being eight + bytes: + + real_header_size = (encoded_header_size + 1) * 4; + + If a Block Header bigger than 1024 bytes is needed in the + future, a new field can be added between the Block Header and + Compressed Data fields. The presence of this new field would + be indicated in the Block Header field. + + +3.1.2. Block Flags + + The Block Flags field is a bit field: + + Bit(s) Mask Description + 0-1 0x03 Number of filters (1-4) + 2-5 0x3C Reserved for future use; MUST be zero for now. + 6 0x40 The Compressed Size field is present. + 7 0x80 The Uncompressed Size field is present. + + If any reserved bit is set, the decoder MUST indicate an error. + It is possible that there is a new field present which the + decoder is not aware of, and can thus parse the Block Header + incorrectly. + + +3.1.3. Compressed Size + + This field is present only if the appropriate bit is set in + the Block Flags field (see Section 3.1.2). + + The Compressed Size field contains the size of the Compressed + Data field, which MUST be non-zero. Compressed Size is stored + using the encoding described in Section 1.2. If the Compressed + Size doesn't match the size of the Compressed Data field, the + decoder MUST indicate an error. + + +3.1.4. Uncompressed Size + + This field is present only if the appropriate bit is set in + the Block Flags field (see Section 3.1.2). + + The Uncompressed Size field contains the size of the Block + after uncompressing. Uncompressed Size is stored using the + encoding described in Section 1.2. If the Uncompressed Size + does not match the real uncompressed size, the decoder MUST + indicate an error. + + Storing the Compressed Size and Uncompressed Size fields serves + several purposes: + - The decoder knows how much memory it needs to allocate + for a temporary buffer in multithreaded mode. + - Simple error detection: wrong size indicates a broken file. + - Seeking forwards to a specific location in streamed mode. + + It should be noted that the only reliable way to determine + the real uncompressed size is to uncompress the Block, + because the Block Header and Index fields may contain + (intentionally or unintentionally) invalid information. + + +3.1.5. List of Filter Flags + + +================+================+ +================+ + | Filter 0 Flags | Filter 1 Flags | ... | Filter n Flags | + +================+================+ +================+ + + The number of Filter Flags fields is stored in the Block Flags + field (see Section 3.1.2). + + The format of each Filter Flags field is as follows: + + +===========+====================+===================+ + | Filter ID | Size of Properties | Filter Properties | + +===========+====================+===================+ + + Both Filter ID and Size of Properties are stored using the + encoding described in Section 1.2. Size of Properties indicates + the size of the Filter Properties field as bytes. The list of + officially defined Filter IDs and the formats of their Filter + Properties are described in Section 5.3. + + Filter IDs greater than or equal to 0x4000_0000_0000_0000 + (2^62) are reserved for implementation-specific internal use. + These Filter IDs MUST never be used in List of Filter Flags. + + +3.1.6. Header Padding + + This field contains as many null byte as it is needed to make + the Block Header have the size specified in Block Header Size. + If any of the bytes are not null bytes, the decoder MUST + indicate an error. It is possible that there is a new field + present which the decoder is not aware of, and can thus parse + the Block Header incorrectly. + + +3.1.7. CRC32 + + The CRC32 is calculated over everything in the Block Header + field except the CRC32 field itself. It is stored as an + unsigned 32-bit little endian integer. If the calculated + value does not match the stored one, the decoder MUST indicate + an error. + + By verifying the CRC32 of the Block Header before parsing the + actual contents allows the decoder to distinguish between + corrupt and unsupported files. + + +3.2. Compressed Data + + The format of Compressed Data depends on Block Flags and List + of Filter Flags. Excluding the descriptions of the simplest + filters in Section 5.3, the format of the filter-specific + encoded data is out of scope of this document. + + +3.3. Block Padding + + Block Padding MUST contain 0-3 null bytes to make the size of + the Block a multiple of four bytes. This can be needed when + the size of Compressed Data is not a multiple of four. If any + of the bytes in Block Padding are not null bytes, the decoder + MUST indicate an error. + + +3.4. Check + + The type and size of the Check field depends on which bits + are set in the Stream Flags field (see Section 2.1.1.2). + + The Check, when used, is calculated from the original + uncompressed data. If the calculated Check does not match the + stored one, the decoder MUST indicate an error. If the selected + type of Check is not supported by the decoder, it SHOULD + indicate a warning or error. + + +4. Index + + +-----------------+===================+ + | Index Indicator | Number of Records | + +-----------------+===================+ + + +=================+===============+-+-+-+-+ + ---> | List of Records | Index Padding | CRC32 | + +=================+===============+-+-+-+-+ + + Index serves several purposes. Using it, one can + - verify that all Blocks in a Stream have been processed; + - find out the uncompressed size of a Stream; and + - quickly access the beginning of any Block (random access). + + +4.1. Index Indicator + + This field overlaps with the Block Header Size field (see + Section 3.1.1). The value of Index Indicator is always 0x00. + + +4.2. Number of Records + + This field indicates how many Records there are in the List + of Records field, and thus how many Blocks there are in the + Stream. The value is stored using the encoding described in + Section 1.2. If the decoder has decoded all the Blocks of the + Stream, and then notices that the Number of Records doesn't + match the real number of Blocks, the decoder MUST indicate an + error. + + +4.3. List of Records + + List of Records consists of as many Records as indicated by the + Number of Records field: + + +========+========+ + | Record | Record | ... + +========+========+ + + Each Record contains information about one Block: + + +===============+===================+ + | Unpadded Size | Uncompressed Size | + +===============+===================+ + + If the decoder has decoded all the Blocks of the Stream, it + MUST verify that the contents of the Records match the real + Unpadded Size and Uncompressed Size of the respective Blocks. + + Implementation hint: It is possible to verify the Index with + constant memory usage by calculating for example SHA-256 of + both the real size values and the List of Records, then + comparing the hash values. Implementing this using + non-cryptographic hash like CRC32 SHOULD be avoided unless + small code size is important. + + If the decoder supports random-access reading, it MUST verify + that Unpadded Size and Uncompressed Size of every completely + decoded Block match the sizes stored in the Index. If only + partial Block is decoded, the decoder MUST verify that the + processed sizes don't exceed the sizes stored in the Index. + + +4.3.1. Unpadded Size + + This field indicates the size of the Block excluding the Block + Padding field. That is, Unpadded Size is the size of the Block + Header, Compressed Data, and Check fields. Unpadded Size is + stored using the encoding described in Section 1.2. The value + MUST never be zero; with the current structure of Blocks, the + actual minimum value for Unpadded Size is five. + + Implementation note: Because the size of the Block Padding + field is not included in Unpadded Size, calculating the total + size of a Stream or doing random-access reading requires + calculating the actual size of the Blocks by rounding Unpadded + Sizes up to the next multiple of four. + + The reason to exclude Block Padding from Unpadded Size is to + ease making a raw copy of Compressed Data without Block + Padding. This can be useful, for example, if someone wants + to convert Streams to some other file format quickly. + + +4.3.2. Uncompressed Size + + This field indicates the Uncompressed Size of the respective + Block as bytes. The value is stored using the encoding + described in Section 1.2. + + +4.4. Index Padding + + This field MUST contain 0-3 null bytes to pad the Index to + a multiple of four bytes. If any of the bytes are not null + bytes, the decoder MUST indicate an error. + + +4.5. CRC32 + + The CRC32 is calculated over everything in the Index field + except the CRC32 field itself. The CRC32 is stored as an + unsigned 32-bit little endian integer. If the calculated + value does not match the stored one, the decoder MUST indicate + an error. + + +5. Filter Chains + + The Block Flags field defines how many filters are used. When + more than one filter is used, the filters are chained; that is, + the output of one filter is the input of another filter. The + following figure illustrates the direction of data flow. + + v Uncompressed Data ^ + | Filter 0 | + Encoder | Filter 1 | Decoder + | Filter n | + v Compressed Data ^ + + +5.1. Alignment + + Alignment of uncompressed input data is usually the job of + the application producing the data. For example, to get the + best results, an archiver tool should make sure that all + PowerPC executable files in the archive stream start at + offsets that are multiples of four bytes. + + Some filters, for example LZMA2, can be configured to take + advantage of specified alignment of input data. Note that + taking advantage of aligned input can be beneficial also when + a filter is not the first filter in the chain. For example, + if you compress PowerPC executables, you may want to use the + PowerPC filter and chain that with the LZMA2 filter. Because + not only the input but also the output alignment of the PowerPC + filter is four bytes, it is now beneficial to set LZMA2 + settings so that the LZMA2 encoder can take advantage of its + four-byte-aligned input data. + + The output of the last filter in the chain is stored to the + Compressed Data field, which is is guaranteed to be aligned + to a multiple of four bytes relative to the beginning of the + Stream. This can increase + - speed, if the filtered data is handled multiple bytes at + a time by the filter-specific encoder and decoder, + because accessing aligned data in computer memory is + usually faster; and + - compression ratio, if the output data is later compressed + with an external compression tool. + + +5.2. Security + + If filters would be allowed to be chained freely, it would be + possible to create malicious files, that would be very slow to + decode. Such files could be used to create denial of service + attacks. + + Slow files could occur when multiple filters are chained: + + v Compressed input data + | Filter 1 decoder (last filter) + | Filter 0 decoder (non-last filter) + v Uncompressed output data + + The decoder of the last filter in the chain produces a lot of + output from little input. Another filter in the chain takes the + output of the last filter, and produces very little output + while consuming a lot of input. As a result, a lot of data is + moved inside the filter chain, but the filter chain as a whole + gets very little work done. + + To prevent this kind of slow files, there are restrictions on + how the filters can be chained. These restrictions MUST be + taken into account when designing new filters. + + The maximum number of filters in the chain has been limited to + four, thus there can be at maximum of three non-last filters. + Of these three non-last filters, only two are allowed to change + the size of the data. + + The non-last filters, that change the size of the data, MUST + have a limit how much the decoder can compress the data: the + decoder SHOULD produce at least n bytes of output when the + filter is given 2n bytes of input. This limit is not + absolute, but significant deviations MUST be avoided. + + The above limitations guarantee that if the last filter in the + chain produces 4n bytes of output, the chain as a whole will + produce at least n bytes of output. + + +5.3. Filters + +5.3.1. LZMA2 + + LZMA (Lempel-Ziv-Markov chain-Algorithm) is a general-purpose + compression algorithm with high compression ratio and fast + decompression. LZMA is based on LZ77 and range coding + algorithms. + + LZMA2 is an extension on top of the original LZMA. LZMA2 uses + LZMA internally, but adds support for flushing the encoder, + uncompressed chunks, eases stateful decoder implementations, + and improves support for multithreading. Thus, the plain LZMA + will not be supported in this file format. + + Filter ID: 0x21 + Size of Filter Properties: 1 byte + Changes size of data: Yes + Allow as a non-last filter: No + Allow as the last filter: Yes + + Preferred alignment: + Input data: Adjustable to 1/2/4/8/16 byte(s) + Output data: 1 byte + + The format of the one-byte Filter Properties field is as + follows: + + Bits Mask Description + 0-5 0x3F Dictionary Size + 6-7 0xC0 Reserved for future use; MUST be zero for now. + + Dictionary Size is encoded with one-bit mantissa and five-bit + exponent. The smallest dictionary size is 4 KiB and the biggest + is 4 GiB. + + Raw value Mantissa Exponent Dictionary size + 0 2 11 4 KiB + 1 3 11 6 KiB + 2 2 12 8 KiB + 3 3 12 12 KiB + 4 2 13 16 KiB + 5 3 13 24 KiB + 6 2 14 32 KiB + ... ... ... ... + 35 3 27 768 MiB + 36 2 28 1024 MiB + 37 3 29 1536 MiB + 38 2 30 2048 MiB + 39 3 30 3072 MiB + 40 2 31 4096 MiB - 1 B + + Instead of having a table in the decoder, the dictionary size + can be decoded using the following C code: + + const uint8_t bits = get_dictionary_flags() & 0x3F; + if (bits > 40) + return DICTIONARY_TOO_BIG; // Bigger than 4 GiB + + uint32_t dictionary_size; + if (bits == 40) { + dictionary_size = UINT32_MAX; + } else { + dictionary_size = 2 | (bits & 1); + dictionary_size <<= bits / 2 + 11; + } + + +5.3.2. Branch/Call/Jump Filters for Executables + + These filters convert relative branch, call, and jump + instructions to their absolute counterparts in executable + files. This conversion increases redundancy and thus + compression ratio. + + Size of Filter Properties: 0 or 4 bytes + Changes size of data: No + Allow as a non-last filter: Yes + Allow as the last filter: No + + Below is the list of filters in this category. The alignment + is the same for both input and output data. + + Filter ID Alignment Description + 0x04 1 byte x86 filter (BCJ) + 0x05 4 bytes PowerPC (big endian) filter + 0x06 16 bytes IA64 filter + 0x07 4 bytes ARM (little endian) filter + 0x08 2 bytes ARM Thumb (little endian) filter + 0x09 4 bytes SPARC filter + + If the size of Filter Properties is four bytes, the Filter + Properties field contains the start offset used for address + conversions. It is stored as an unsigned 32-bit little endian + integer. The start offset MUST be a multiple of the alignment + of the filter as listed in the table above; if it isn't, the + decoder MUST indicate an error. If the size of Filter + Properties is zero, the start offset is zero. + + Setting the start offset may be useful if an executable has + multiple sections, and there are many cross-section calls. + Taking advantage of this feature usually requires usage of + the Subblock filter, whose design is not complete yet. + + +5.3.3. Delta + + The Delta filter may increase compression ratio when the value + of the next byte correlates with the value of an earlier byte + at specified distance. + + Filter ID: 0x03 + Size of Filter Properties: 1 byte + Changes size of data: No + Allow as a non-last filter: Yes + Allow as the last filter: No + + Preferred alignment: + Input data: 1 byte + Output data: Same as the original input data + + The Properties byte indicates the delta distance, which can be + 1-256 bytes backwards from the current byte: 0x00 indicates + distance of 1 byte and 0xFF distance of 256 bytes. + + +5.3.3.1. Format of the Encoded Output + + The code below illustrates both encoding and decoding with + the Delta filter. + + // Distance is in the range [1, 256]. + const unsigned int distance = get_properties_byte() + 1; + uint8_t pos = 0; + uint8_t delta[256]; + + memset(delta, 0, sizeof(delta)); + + while (1) { + const int byte = read_byte(); + if (byte == EOF) + break; + + uint8_t tmp = delta[(uint8_t)(distance + pos)]; + if (is_encoder) { + tmp = (uint8_t)(byte) - tmp; + delta[pos] = (uint8_t)(byte); + } else { + tmp = (uint8_t)(byte) + tmp; + delta[pos] = tmp; + } + + write_byte(tmp); + --pos; + } + + +5.4. Custom Filter IDs + + If a developer wants to use custom Filter IDs, he has two + choices. The first choice is to contact Lasse Collin and ask + him to allocate a range of IDs for the developer. + + The second choice is to generate a 40-bit random integer, + which the developer can use as his personal Developer ID. + To minimize the risk of collisions, Developer ID has to be + a randomly generated integer, not manually selected "hex word". + The following command, which works on many free operating + systems, can be used to generate Developer ID: + + dd if=/dev/urandom bs=5 count=1 | hexdump + + The developer can then use his Developer ID to create unique + (well, hopefully unique) Filter IDs. + + Bits Mask Description + 0-15 0x0000_0000_0000_FFFF Filter ID + 16-55 0x00FF_FFFF_FFFF_0000 Developer ID + 56-62 0x3F00_0000_0000_0000 Static prefix: 0x3F + + The resulting 63-bit integer will use 9 bytes of space when + stored using the encoding described in Section 1.2. To get + a shorter ID, see the beginning of this Section how to + request a custom ID range. + + +5.4.1. Reserved Custom Filter ID Ranges + + Range Description + 0x0000_0300 - 0x0000_04FF Reserved to ease .7z compatibility + 0x0002_0000 - 0x0007_FFFF Reserved to ease .7z compatibility + 0x0200_0000 - 0x07FF_FFFF Reserved to ease .7z compatibility + + +6. Cyclic Redundancy Checks + + There are several incompatible variations to calculate CRC32 + and CRC64. For simplicity and clarity, complete examples are + provided to calculate the checks as they are used in this file + format. Implementations MAY use different code as long as it + gives identical results. + + The program below reads data from standard input, calculates + the CRC32 and CRC64 values, and prints the calculated values + as big endian hexadecimal strings to standard output. + + #include <stddef.h> + #include <inttypes.h> + #include <stdio.h> + + uint32_t crc32_table[256]; + uint64_t crc64_table[256]; + + void + init(void) + { + static const uint32_t poly32 = UINT32_C(0xEDB88320); + static const uint64_t poly64 + = UINT64_C(0xC96C5795D7870F42); + + for (size_t i = 0; i < 256; ++i) { + uint32_t crc32 = i; + uint64_t crc64 = i; + + for (size_t j = 0; j < 8; ++j) { + if (crc32 & 1) + crc32 = (crc32 >> 1) ^ poly32; + else + crc32 >>= 1; + + if (crc64 & 1) + crc64 = (crc64 >> 1) ^ poly64; + else + crc64 >>= 1; + } + + crc32_table[i] = crc32; + crc64_table[i] = crc64; + } + } + + uint32_t + crc32(const uint8_t *buf, size_t size, uint32_t crc) + { + crc = ~crc; + for (size_t i = 0; i < size; ++i) + crc = crc32_table[buf[i] ^ (crc & 0xFF)] + ^ (crc >> 8); + return ~crc; + } + + uint64_t + crc64(const uint8_t *buf, size_t size, uint64_t crc) + { + crc = ~crc; + for (size_t i = 0; i < size; ++i) + crc = crc64_table[buf[i] ^ (crc & 0xFF)] + ^ (crc >> 8); + return ~crc; + } + + int + main() + { + init(); + + uint32_t value32 = 0; + uint64_t value64 = 0; + uint64_t total_size = 0; + uint8_t buf[8192]; + + while (1) { + const size_t buf_size + = fread(buf, 1, sizeof(buf), stdin); + if (buf_size == 0) + break; + + total_size += buf_size; + value32 = crc32(buf, buf_size, value32); + value64 = crc64(buf, buf_size, value64); + } + + printf("Bytes: %" PRIu64 "\n", total_size); + printf("CRC-32: 0x%08" PRIX32 "\n", value32); + printf("CRC-64: 0x%016" PRIX64 "\n", value64); + + return 0; + } + + +7. References + + LZMA SDK - The original LZMA implementation + http://7-zip.org/sdk.html + + LZMA Utils - LZMA adapted to POSIX-like systems + http://tukaani.org/lzma/ + + XZ Utils - The next generation of LZMA Utils + http://tukaani.org/xz/ + + [RFC-1952] + GZIP file format specification version 4.3 + http://www.ietf.org/rfc/rfc1952.txt + - Notation of byte boxes in section "2.1. Overall conventions" + + [RFC-2119] + Key words for use in RFCs to Indicate Requirement Levels + http://www.ietf.org/rfc/rfc2119.txt + + [GNU-tar] + GNU tar 1.21 manual + http://www.gnu.org/software/tar/manual/html_node/Blocking-Factor.html + - Node 9.4.2 "Blocking Factor", paragraph that begins + "gzip will complain about trailing garbage" + - Note that this URL points to the latest version of the + manual, and may some day not contain the note which is in + 1.21. For the exact version of the manual, download GNU + tar 1.21: ftp://ftp.gnu.org/pub/gnu/tar/tar-1.21.tar.gz + |