summaryrefslogtreecommitdiff
path: root/Master/tlpkg/doc/articles/guit07/tlinfra-guit07.tex
diff options
context:
space:
mode:
Diffstat (limited to 'Master/tlpkg/doc/articles/guit07/tlinfra-guit07.tex')
-rw-r--r--Master/tlpkg/doc/articles/guit07/tlinfra-guit07.tex595
1 files changed, 595 insertions, 0 deletions
diff --git a/Master/tlpkg/doc/articles/guit07/tlinfra-guit07.tex b/Master/tlpkg/doc/articles/guit07/tlinfra-guit07.tex
new file mode 100644
index 00000000000..f613f60b388
--- /dev/null
+++ b/Master/tlpkg/doc/articles/guit07/tlinfra-guit07.tex
@@ -0,0 +1,595 @@
+%
+% TeX Live's new infrastructure
+% Norbert Preining
+% Article presented on the 4th GuITmeeting, Pisa 13 October 2007
+%
+% Copyright 2007 Norbert Preining
+% You can redistribute and/or modify this document under the terms of the
+% GNU General Public License as published by the Free Software Foundation;
+% either version 2 of the License, or (at your option) any later version.
+%
+\documentclass{arstexnica}
+\usepackage[latin1]{inputenc}
+\usepackage[T1]{fontenc}
+\usepackage{lmodern}
+\usepackage[english]{babel}
+\usepackage{graphicx}
+\usepackage{hyperref}
+\usepackage{fancyvrb}
+\usepackage{url}
+\usepackage{marvosym}
+\usepackage{listings}
+\lstset{frame=lines,basicstyle=\ttfamily,showspaces=true,prebreak={\Righttorque},postbreak={\Lefttorque},breaklines}
+\newcommand{\tl}{\TeX~Live}
+\newcommand{\ctan}{CTAN}
+\newcommand{\tpm}{\texttt{tpm}}
+\newcommand{\tpms}{\tpm{}s}
+\newcommand{\tlpsrc}{\texttt{tlpsrc}}
+\newcommand{\tlpsrcs}{\tlpsrc{}s}
+\newcommand{\tlpobj}{\texttt{tlpobj}}
+\newcommand{\tlpobjs}{\tlpobj{}s}
+\newcommand{\tlpdb}{\texttt{tlpdb}}
+\newcommand{\tlpdbs}{\tlpdb{}s}
+
+\hyphenation{infra-struc-ture}
+
+%\catcode`>=\active
+%\def>#1<{\texttt{#1}}
+\DefineShortVerb{\|}
+\begin{document}
+\begin{article}
+%\selectlanguage{italian}%
+
+\title{\tl's new infrastructure\thanks{%
+ Article presented on the 4th GuITmeeting, Pisa 13 October 2007}}
+
+\author{Norbert Preining}
+\address{Vienna University of Technology\\
+ Wiedner Hauptstr.\ 10\\
+ 1040 Wien, Austria}
+\netaddress{preining@logic.at}
+
+\maketitle
+
+
+\begin{abstract}
+Since the release of \tl~2007 a new infrastructure for \tl\
+distribution and management has been developed. This article presents the
+reasons for this switch, the ideas behind the new infrastructure,
+software developed, and ways to incorporate this new
+infrastructure. We will close with a look at what new features
+this new infrastructure could bring to the \TeX\ (Live) world.
+\end{abstract}
+
+%
+% does not work in normal arstexnika mode, needs standalone, I leave
+% this up to the editors
+%\tableofcontents
+
+\section{Introduction}
+\label{sec:intro}
+
+\tl\ is an easy way to get up and running with \TeX. It provides a
+comprehensive \TeX\ system with binaries for most flavors of Unix,
+including GNU/Linux, and also Windows. It includes all the major
+\TeX-related programs, macro packages, and fonts that are free
+software, including support for many languages around the world.
+
+Since 1996 it tries to bring to all \TeX-users as much material
+from \ctan\ as possible, packaged for `consumption', i.e., for using it
+as a live system from DVD, or installing it on a variety of operating
+systems/architecture combinations.
+
+These two requirements, incorporating as much as possible from \ctan,
+and providing binaries for a wide range of OS/arch combinations, has led
+to a huge number of supporting scripts, in a wide variety of programming
+languages (Perl, shell, XML, C, \ldots). These scripts were (and are)
+used to generate installation media, check consistency, update
+important files (e.g., for the installer), incorporating packages from
+\ctan, just to name a few. There have been many contributors; Sebastian
+Rahtz, Fabrice Popineau, and Karl Berry are the principal authors.
+
+As always with overload volunteers working on big projects, not much
+was documented, programming was done by trial and error (see for
+example the packaging scripts of Debian, one of my own more horrible
+creations). This wouldn't have been reason enough to change things
+this deep in the intestines of \tl, but other illusions and dreams
+have driven us to rebuild from the ground up.
+
+\section{A world full of TPMs}
+\label{sec:tpm}
+
+Up to \tl~2007 everything in \tl\ was organized in \tpms, an
+acronym for `\TeX\ Package Manage[r/ment]'. One \tpm\ described more
+or less one package from \ctan, containing the list of included
+files, title, description, license information, sometimes version
+numbers, additional information necessary for incorporation
+(activation of map files, hyphenation patterns, and formats).
+
+Maintenance of these \tpms\ was mostly done by a Perl module |Tpm.pm|
+written by Fabrice Popineau, and the accompanying mystical program
+|tpm-factory.pl|, a script so full of possibilities that nobody
+besides Fabrice probably ever understood everything that could be done with
+it. Some of the jobs of this |tpm-factory.pl| were
+\begin{itemize}
+\item regeneration of the \tpms, this included some magic in finding
+ the right files
+\item creation of a new \tpm\ for a newly installed package from \ctan
+\item checking the coverage, i.e., checking that every file present in
+ the repository is actually contained in a \tpm.
+\item duplication and dependency checks
+\end{itemize}
+
+Alas, there have been some problems with all the \tpm-business:
+\begin{itemize}
+\item they contained a mixture of generated (file lists) and static
+ information (actions to be carried out);
+\item they were full of duplicate information: version, license,
+ descriptions were taken now and then from the \TeX\ Catalogue, but
+ were typically horribly out of date or otherwise wrong;
+\item we had to generate so-called `lists' files (line-oriented plain
+ text file lists) from the \tpms\ for the installer
+ because the XML syntax of the \tpms\ is not practical to parse from a simple
+ shell script.
+\end{itemize}
+
+\section{Aims of the new infrastructure}
+\label{sec:aims}
+
+So around mid-2006 discussion of a new infrastructure started, but
+unfortunately, due to the usual time constraints of all involved,
+without much concrete outcome except many emails. At least a preliminary
+catalogue of items to be improved sprang into existence:
+
+
+\paragraph{Separation of static from generated content}
+
+As already mentioned, the \tpms\ contained a wild mixture of stuff from
+various sources (the file itself, the \TeX\ Catalogue, the
+repository). The new infrastructure should have some kind of `source'
+files where {\em only} the necessary information is stored, and all other
+content is added at generation time (of whatever). Naturally that
+could have been done on top of the \tpms, too, but starting from
+scratch seemed to be a better option.
+
+\paragraph{Getting rid of lists files}
+
+Using the \tpms\ and some XSLT processing, the lists files were generated
+before release. These lists files (one for each package, about 2000 in all)
+must be read by the installer from the DVD, which caused a lot of
+headache and slow installations.
+%Furthermore, these list files often were forgotten, thus out of date.
+%(during development, but not at the end! i hope :) --karl)
+
+The sole reason for the existence of these lists files was the fact
+that the installer, written in shell, couldn't parse the XML \tpms,
+since XML parsing programs cannot be assumed to be present at all
+installations. The new infrastructure should be based on some format
+that is easily parseable using stuff normally already present, or at
+least necessary for \tl\ anyway.
+
+\paragraph{Single package updates via the web}
+
+Some \TeX\ distributions, notably MiKTeX, already support single package
+updates over the Internet.
+\tl, as big and nice as it is, still lacks this lovely feature. Of course
+it was put high on the priority list to allow for single package
+updates.
+
+\paragraph{Better documentation}
+
+Last, but not least, we hoped that by rewritting the infrastructure
+and documenting it on the way we could provide a more stable foundation
+for more development and improvement, thus attracting more
+contributors. Furthermore, since te\TeX\ development has been stopped,
+\tl\ is taking the place as the \TeX\ system of choice in many
+GNU/Linux and other free distributions, and better documentation would
+only help providing those distributions with some aid for the migration.
+
+\section{New infrastructure -- the basics}
+\label{sec:basics}
+
+There are three type of files: \tlpsrc, \tlpobj, \tlpdb. The
+format of these files are very similar to Debian's Package files: a
+simple list of
+\begin{center}
+ |key value|
+\end{center}
+(without leading spaces). Empty lines at the beginning and end of a
+\tlpsrc\ or \tlpobj\ file are ignored. Also lines starting with \#\ are
+ignored (to allow for comments). Some special cases apply for \tlpobj\
+files and the \tlpdb\ file, see below.
+
+\subsection{\tlpsrc\ file format}
+\label{sec:tlpsrc}
+
+The possible keys and their respective interpretation for the \tlpsrc\
+files are:
+\begin{description}
+\item[name] identifies the package, |value| must consist only of
+ \verb+[-_a-zA-Z0-9]+.
+
+\item[category]
+ identifies the category into which this package belongs. Possible
+ categories are |Collection|, |Scheme|, |TLCore|, |Documentation|,
+ |Package|. There are no particular checks as to whether a
+ |tlpsrc| file called |collection-something| actually belongs to the
+ category |Collection|. Most packages will fall into the |Package|
+ category. These categories were inherited from the \tpm\ world and
+ may be modified at some point; for now, they suffice.
+
+\item[catalogue]
+ identifies the name under which this package can be found in the \TeX\
+ Catalogue. If not present the name of the packages is taken as the
+ Catalogue entry.
+
+\item[shortdesc]
+ gives a one line description of the package. Susequent entries will
+ overwrite the former ones. In \tl\ only used for collections and
+ schemes.
+
+\item[longdesc]
+ gives a long description of the package. Susequent entries are added
+ up to a long text. In \tl\ only used for collections and
+ schemes.
+
+\item[depend]
+ gives the list of dependencies of the package in the form
+ \begin{center}
+ |Category/Name|
+ \end{center}
+ for |value|. All the depend lines contribute to the dependencies of
+ the package.
+
+\item[execute]
+ gives a free form entry of install time jobs to be
+ executed. Currently the following values are understood by
+ the installers:
+ \begin{itemize}
+ \item |execute addMap font.map|\\
+ enables the font map file |font.map| in the |updmap.cfg| file.
+ \item |execute addMixedMap font.map|\\
+ enables the font map file |font.map| for Mixed mode in the
+ |updmap.cfg| file. By the way,
+ the purpose of MixedMap is to help users with printers that render
+ the Type~1 versions of the fonts worse than the (mode-tuned)
+ bitmap-based Type~3 fonts. The entries from MixedMap are not
+ added to psfonts\_pk.map; that's the only difference. It's used for
+ fonts which have both Metafont and (typically) autotraced Type~1
+ instantations.
+
+ \item |execute BuildLanguageDat NN|\\
+ activates all the hyphenation patterns found in
+ \path{Master/texmf/tex/generic/config/language.NN.dat} in the
+ generated |language.dat| file.
+ \item |execute BuildFormat FMTCFG|\\
+ activates all the formats present in
+ \path{Master/texmf/fmtutil/format.FMTCFG.cnf}
+ in the generated |fmtutil.cnf| file.
+ \end{itemize}
+
+{
+\UndefineShortVerb{\|}
+\item[(src|run|doc|bin)pattern pattern]
+ adds a pattern (see below) to the respective list of patterns.
+}
+
+\end{description}
+
+\subsubsection{Patterns}
+\label{sec:patterns}
+
+To automatically generate the file lists in \tlpobjs\ a specific
+pattern language was developed. Patterns can include or exclude files
+or whole sub-directories into the file list of a \tlpobj. Patterns are
+of the form
+\begin{center}
+ |[PREFIX]TYPE PAT|
+\end{center}
+where |PREFIX| can be |+|, |!+|, or |!|, and |TYPE| one of the letters
+|t|, |f|, |d|, |r|. An initial |+| for the pattern indicates that the
+automatically generated pattern should not be cleared (see below),
+while the |!| indicates that the matching files should be excluded.
+
+The meaning of the various pattern types is
+\begin{description}
+\item[f path]
+ includes all files which match |path| where \emph{only} the last
+ component of |path| can contain the usual glob characters * and ?
+ (but no others!).
+\item[d path]
+ includes all the files in and below the directory specified as
+ |path|.
+\item[t word1 ... wordN wordL]
+ includes all the files in and below all directories of the form
+ \begin{center}
+ \path{word1/word2/.../wordN/.../any/dirs/.../wordL/}
+ \end{center}
+ i.e., all words but the last form the prefix of the path, then
+ there can be an arbitrary nesting of directories, followed by
+ |wordL| as another directory.
+\item[r regexp]
+ includes all files matching the Perl regexp \verb+/^regexp$/+
+\end{description}
+
+
+\subsubsection{Autogenerated patterns}
+\label{sec:autogenpat}
+
+In the case that one of the pattern sections is empty or \emph{all}
+the provided patterns have the prefix |+| (e.g., |+f ...|), then the
+following patterns are \emph{automatically} added at expansion time
+(but never written to the textual representation):
+
+\begin{itemize}
+\item for runpatterns of category |Package|
+ \begin{center}
+ |t texmf-dist topdir name|
+ \end{center}
+ where |topdir| is one of: bibtex, context, dvips, fonts, makeindex,
+ metafont, metapost, mft, omega, scripts, tex, vtex. |name| refers to
+ the name of the package as given by the |name| directive.
+
+ For other categories \emph{no} patterns are automatically added to
+ the list of runpatterns.
+\item for docpattern of category |Package|
+ \begin{center}
+ |t texmf-dist doc name|
+ \end{center}
+ and for docpattern of category |Documentation|
+ \begin{center}
+ |t texmf-doc doc name|
+ \end{center}
+\item for srcpattern of category |Package|
+ \begin{center}
+ |t texmf-dist source name|
+ \end{center}
+ and for srcpattern of category |Documentation|
+ \begin{center}
+ |t texmf-doc source name|
+ \end{center}
+\end{itemize}
+|binpatterns| are never automatically added.
+
+In addition some magic tricks have been added to make writing of
+|binpatterns| easier:
+\begin{description}
+\item[arch expansion]
+ In case the string \verb+${ARCH}+ occurs in one |binpattern| it is
+ automatically expanded to the respective architecture.
+\item[bat/exe/dll for win32]
+ For |binpatterns| of the form |f bin/win32/foobar| (i.e., also for a
+ binpattern of the form \verb+f bin/${ARCH}/foobar+) files
+ |foobar.bat|, |foobar.dll|, and |foobar.exe| are also matched.
+\end{description}
+The above two properties allows to capture the binaries for all
+architectures in one binpattern:
+\begin{center}
+ \verb+binpattern f bin/${ARCH}/dvips+
+\end{center}
+will include |bin/win32/dvips.exe| in the runfiles when arch=win32.
+
+Note that the bat/exe expansion \emph{only} works for patterns of the
+|f|-type.
+
+\subsection{\tlpobj\ file format}
+\label{sec:tlpobj}
+
+These files are structurally similar to \tlpsrc\ files; the
+only difference is that the |*pattern| keys are not allowed, their
+place being taken by |*files| keys having a peculiar syntax (see
+below). Furthermore, an additional key |revision| and all keys
+matching |catalogue-*| are allowed.
+The |catalogue-*| keys specify information simply copied from the \TeX\
+Catalogue. We'll discuss the others below.
+
+\subsubsection{The \texttt{revision} entry}
+
+The |revision| key is {\em not} related to the package's version as
+specified by the author. Those versions are far too random, hard to
+extract, and sometimes not even present, to be a reliable basis for a
+packaging system. Instead, we look at all the package's files in the
+\tl\ source repository, which is currently Subversion, and use the
+maximum version number found there. Since Subversion uses simple
+integers for version numbers, this is simple to use in programs, and
+just as important, this whole process of finding the version number can
+be reliably automated.
+
+\subsubsection{The \texttt{*files} entries}
+\label{sec:keyfiles}
+
+While the |tlpsrc| files specify the files to be included using the
+patterns, the |tlpobj| files contain the already expanded file list.
+All the `files' keys have in common that they are followed by a list of
+files (and possibly additional tags) indented by exactly one (1) space. They
+differ only in the first line itself (described below).
+
+
+
+\begin{description}
+\item[srcfiles, runfiles] each of these lines is (or better should be)
+ tagged with |size=NNNN| where |NNNN| is the sum of sizes of the single
+ files (currently in bytes), e.g.,
+ \begin{center}
+ |srcfiles size=NNNNNN|
+ \end{center}
+\item[docfiles] The docfiles line itself is similar to the |srcfiles|
+ and |runfiles| lines above:
+ \begin{center}
+ |docfiles size=NNNNNN|
+ \end{center}
+ But the lines listing the files are allowed to have additional tags
+ |details| and |language|:
+ \begin{center}
+ |file detail="foo bar" language="en"|
+ \end{center}
+ For an example see listing~\ref{achemso}. The reason to add these
+ tags is the hope that someone will write a nice replacement for
+ \path{texdoctk}. Note that these tags' source is the \TeX\ Catalogue,
+ and are in no way obligatory.
+\item[binfiles] Since |binfiles| are different for the different
+ architectures, one \tlpobj\ file can contain |binfiles| lines for
+ different architectures. The architecture is specified on the
+ |binfiles| lines using the |arch=XXX| tag. Thus, |binfiles| lines look like
+ \begin{center}
+ |binfiles arch=XXXX size=NNNNN|
+ \end{center}
+ A more complete example taken from |bin-dvipsk.tlpobj| can be seen
+ in listing~\ref{dvipsk}.
+\end{description}
+
+\begin{lstlisting}[caption={Excerpt from achemso.tlpobj},label=achemso]
+...
+docfiles size=135842
+ texmf-dist/doc/latex/achemso/README details="Package Readme" language="de"
+ texmf-dist/doc/latex/achemso/achemso.pdf details="Package documentation" language="de"
+ ...
+\end{lstlisting}
+
+\begin{lstlisting}[caption={Excerpt from bin-dvipsk.tlpobj},label=dvipsk]
+name bin-dvipsk
+category TLCore
+revision 4427
+docfiles size=959434
+ texmf/doc/dvips/dvips.html
+ ...
+runfiles size=1702468
+ texmf/dvips/base/color.pro
+ ...
+ texmf/scripts/pkfix/pkfix.pl
+binfiles arch=i386-solaris size=329700
+ bin/i386-solaris/afm2tfm
+ bin/i386-solaris/dvips
+ bin/i386-solaris/pkfix
+binfiles arch=win32 size=161280
+ bin/win32/afm2tfm.exe
+ bin/win32/dvips.exe
+ bin/win32/pkfix.exe
+...
+\end{lstlisting}
+
+\subsection{\tlpdb\ file format}
+\label{sec:tlpdb}
+
+A \tlpdb\ file will describe the status of an installation, that is it
+will contain all the installed packages' \tlpobj\ files. Thus,
+the format of a \tlpdb\ file is quite simple. It is the concatenation
+of \tlpobjs\ with (at least) one empty line between different
+\tlpobjs.
+
+
+\section{Programming APIs}
+
+Wrapping these file formats into a programming API represents
+the actual work of realizing the new infrastructure. On the way we had
+to create not only modules for accessing the content of the above
+files in some object-oriented way, we also created some additional
+modules for interfacing to the Subversion repository and the \TeX\
+Catalogue.
+
+Currently, the following modules are available in the \tl\ subversion
+repository:
+\begin{description}
+\item[TeXLive::TLTREE] an object that exhibits the properties of the
+ subversion repository in some structured form. In principle it is
+ the scooping up of |svn status -v| output with post-processing
+ for easier and faster searches.
+\item[TeXLive::TeXCatalogue] a very simple interface to the \TeX\
+ Catalogue, barely providing minimal information. Will be used for
+ enriching the final database.
+\item[TeXLive::TLPSRC] provides access to \tlpsrc\ files, basic
+ functionality like reading in, writing out, and member access
+ functions. In addition, it allows to generate, or expand, a |TLPSRC|
+ object to a |TLPOBJ| object using an instance of |TLTREE|. That is,
+ it takes the patterns specified in the \tlpsrc, and tries to find
+ all files matching these patterns. After this it recomputes the size
+ and returns a |TLPOBJ| object.
+\item[TeXLive::TLPOBJ] provides access to \tlpobj\ files, and entries
+ stored in a a |TLPDB| object. As with the \tlpsrc, basic
+ functionalities are exported, and also a function to create a zip
+ file for the |TLPOBJ| (which later should be used for web updates or
+ building the `inst' CD),
+ and together with an instance of a |TeXCatalogue| object some
+ information can be transferred from the \TeX\ Catalogue to the
+ |TLPOBJ| object.
+\item[TeXLive::TLPDB] provides access to the \tl\ database. These
+ files will serve the central purpose of describing the current
+ status of various media, like an installation on a user computer, or
+ the distribution. Comparing these status descriptions, update
+ programs can deduce what packages should be updated.
+\item[TeXLive::TLPUtils] exports some handy functions used in all the
+ other modules.
+\end{description}
+
+The full API documentation is provided in |pod| format within the perl
+modules. A Perl API document
+is available in the \tl\ subversion repository, too.
+
+In addition to the Perl API a start at a shell (sh, bash, etc.)\ API and
+implementation has been created, but it presently lacks several key features,
+and is not up to date. Jim Hef{}feron has contributed a start at a Python API.
+However, we assume that on all computers Perl will be
+available or will be installed during installation.
+
+\section{Integration into current \tl\ development}
+
+Around these modules several day-in-day-out tasks of the \tl\
+developers have been rewritten using the new infrastructure, the most
+important one being the script |ctan2tl|. This is the script responsible for
+transferring a package from \ctan\ into the right places in
+the \tl\ repository, and preparing it for users.
+
+Other things already converted are automatic update of the
+|texlive.tlpdb|, the file containing the |TLPDB|.
+
+On the other hand not everything has been done, the biggest piece for
+sure is the rewrite of the installer. The old installer, relying on
+the lists files we wanted to get rid of, is still the only one we
+have. Furtunately we can generate lists files from the
+|texlive.tlpdb|, but in the not-so-long run, meaning hopefully for
+\tl~2008, we want to have a new installer.
+
+Other things still missing are coverage and duplication checks, but
+those are quite easy to do using the Unix command world (|grep|,
+|sort|, and |uniq| being the magic incantations).
+
+\section{Closing}
+\label{sec:closing}
+
+Although we are quite confident that the new infrastructure will work
+well, only the reality of the next release will prove us right or
+wrong. Many of the key features of the new infrastructure have been
+designed with the \tpm-system in mind, and some `features' carried
+over will probably proven superfluous. Nothing with the current
+infrastructure is written in stone, and we are open for proposals and
+improvements, keeping in mind that releasing \tl\ should not be made
+more difficult.
+
+That said, we invite contributors and everyone interested to contact
+us at \url{tex-live@tug.org}, or take a look at the following
+resources:
+\begin{itemize}
+\item \path{http://www.tug.org/texlive} -- the main entry point, with
+ links to developers' resources, documentation
+\item \path{http://www.tug.org/svn/texlive/trunk/} -- web view onto the
+ subversion repository;
+\item \path{svn://tug.org/texlive/trunk} -- svn repository, anonymous
+ access, take care when you checking out the repository, it needs
+ several Gigabyte of disk space;
+\item \path{http://www.tug.org/texlive/pkgupdate.html} -- an
+ explanation how updates from \ctan\ to \tl\ are done.
+\end{itemize}
+
+Finally, I want to thank all the contributors to \tl, there are too
+many to mention. In the course of developing this infrastructure the
+discussions with Karl Berry, Pawe\l{} Jackowski, Reinhard Kotucha,
+Siep Kroonenberg, and Jerzy B. Ludwichowski were very helpful.
+
+%\bibliographystyle{arstexnica}
+%\bibliography{atsample}
+
+\end{article}
+
+\end{document}
+
+