path: root/Master/tlpkg/doc/tlinfra-guit07.tex

%
% TeX Live's new infrastructure
% Norbert Preining
% Article presented on the 4th GuITmeeting, Pisa 13 October 2007
%
% Copyright 2007 Norbert Preining
% You can redistribute and/or modify this document under the terms of the 
% GNU General Public License as published by the Free Software Foundation; 
% either version 2 of the License, or (at your option) any later version.
%
\documentclass{arstexnica}
\usepackage[latin1]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{lmodern}
\usepackage[english]{babel}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{fancyvrb}
\usepackage{url}
\usepackage{marvosym}
\usepackage{listings}
\lstset{frame=lines,basicstyle=\ttfamily,showspaces=true,prebreak={\Righttorque},postbreak={\Lefttorque},breaklines}
\newcommand{\tl}{\TeX~Live}
\newcommand{\ctan}{CTAN}
\newcommand{\tpm}{\texttt{tpm}}
\newcommand{\tpms}{\tpm{}s}
\newcommand{\tlpsrc}{\texttt{tlpsrc}}
\newcommand{\tlpsrcs}{\tlpsrc{}s}
\newcommand{\tlpobj}{\texttt{tlpobj}}
\newcommand{\tlpobjs}{\tlpobj{}s}
\newcommand{\tlpdb}{\texttt{tlpdb}}
\newcommand{\tlpdbs}{\tlpdb{}s}

\hyphenation{infra-struc-ture}

%\catcode`>=\active
%\def>#1<{\texttt{#1}}
\DefineShortVerb{\|}
\begin{document}
\begin{article}
%\selectlanguage{italian}%

\title{\tl's new infrastructure\thanks{%
  Article presented on the 4th GuITmeeting, Pisa 13 October 2007}}

\author{Norbert Preining}
\address{Vienna University of Technology\\
	Wiedner Hauptstr.\ 10\\
	1040 Wien, Austria}
\netaddress{preining@logic.at}

\maketitle


\begin{abstract}
Since the release of \tl~2007 a new infrastructure for \tl\
distribution and management has been developed. This article presents the
reasons for this switch, the ideas behind the new infrastructure,
software developed, and ways to incorporate this new
infrastructure. We will close with a look at what new features
this new infrastructure could bring to the \TeX\ (Live) world.
\end{abstract}

%
% does not work in normal arstexnika mode, needs standalone, I leave
% this up to the editors
%\tableofcontents

\section{Introduction}
\label{sec:intro}

\tl\ is an easy way to get up and running with \TeX. It provides a
comprehensive \TeX\ system with binaries for most flavors of Unix,
including GNU/Linux, and also Windows. It includes all the major
\TeX-related programs, macro packages, and fonts that are free
software, including support for many languages around the world. 

Since 1996 it tries to bring to all \TeX-users as much material
from \ctan\ as possible, packaged for `consumption', i.e., for using it
as a live system from DVD, or installing it on a variety of operating
systems/architecture combinations.

These two requirements, incorporating as much as possible from \ctan,
and providing binaries for a wide range of OS/arch combinations, has led
to a huge number of supporting scripts, in a wide variety of programming
languages (Perl, shell, XML, C, \ldots). These scripts were (and are)
used to generate installation media, check consistency, update
important files (e.g., for the installer), incorporating packages from
\ctan, just to name a few. There have been many contributors; Sebastian
Rahtz, Fabrice Popineau, and Karl Berry are the principal authors.

As always with overload volunteers working on big projects, not much
was documented, programming was done by trial and error (see for
example the packaging scripts of Debian, one of my own more horrible
creations). This wouldn't have been reason enough to change things
this deep in the intestines of \tl, but other illusions and dreams
have driven us to rebuild from the ground up.

\section{A world full of TPMs}
\label{sec:tpm}

Up to \tl~2007 everything in \tl\ was organized in \tpms, an
acronym for `\TeX\ Package Manage[r/ment]'. One \tpm\ described more
or less one package from \ctan, containing the list of included
files, title, description, license information, sometimes version
numbers, additional information necessary for incorporation
(activation of map files, hyphenation patterns, and formats).

Maintenance of these \tpms\ was mostly done by a Perl module |Tpm.pm|
written by Fabrice Popineau, and the accompanying mystical program
|tpm-factory.pl|, a script so full of possibilities that nobody
besides Fabrice probably ever understood everything that could be done with
it. Some of the jobs of this |tpm-factory.pl| were
\begin{itemize}
\item regeneration of the \tpms, this included some magic in finding
  the right files
\item creation of a new \tpm\ for a newly installed package from \ctan
\item checking the coverage, i.e., checking that every file present in
  the repository is actually contained in a \tpm.
\item duplication and dependency checks
\end{itemize}

Alas, there have been some problems with all the \tpm-business:
\begin{itemize}
\item they contained a mixture of generated (file lists) and static
  information (actions to be carried out);
\item they were full of duplicate information: version, license,
  descriptions were taken now and then from the \TeX\ Catalogue, but
  were typically horribly out of date or otherwise wrong;
\item we had to generate so-called `lists' files (line-oriented plain
  text file lists) from the \tpms\ for the installer
  because the XML syntax of the \tpms\ is not practical to parse from a simple
  shell script.
\end{itemize}

\section{Aims of the new infrastructure}
\label{sec:aims}

So around mid-2006 discussion of a new infrastructure started, but
unfortunately, due to the usual time constraints of all involved,
without much concrete outcome except many emails. At least a preliminary
catalogue of items to be improved sprang into existence:


\paragraph{Separation of static from generated content}

As already mentioned, the \tpms\ contained a wild mixture of stuff from
various sources (the file itself, the \TeX\ Catalogue, the
repository). The new infrastructure should have some kind of `source'
files where {\em only} the necessary information is stored, and all other
content is added at generation time (of whatever). Naturally that
could have been done on top of the \tpms, too, but starting from
scratch seemed to be a better option.

\paragraph{Getting rid of lists files}

Using the \tpms\ and some XSLT processing, the lists files were generated
before release. These lists files (one for each package, about 2000 in all)
must be read by the installer from the DVD, which caused a lot of
headache and slow installations. 
%Furthermore, these list files often were forgotten, thus out of date.
%(during development, but not at the end! i hope :) --karl)

The sole reason for the existence of these lists files was the fact
that the installer, written in shell, couldn't parse the XML \tpms,
since XML parsing programs cannot be assumed to be present at all
installations. The new infrastructure should be based on some format
that is easily parseable using stuff normally already present, or at
least necessary for \tl\ anyway. 

\paragraph{Single package updates via the web}

Some \TeX\ distributions, notably MiKTeX, already support single package
updates over the Internet.
\tl, as big and nice as it is, still lacks this lovely feature. Of course
it was put high on the priority list to allow for single package
updates. 

\paragraph{Better documentation}

Last, but not least, we hoped that by rewritting the infrastructure
and documenting it on the way we could provide a more stable foundation
for more development and improvement, thus attracting more
contributors. Furthermore, since te\TeX\ development has been stopped,
\tl\ is taking the place as the \TeX\ system of choice in many
GNU/Linux and other free distributions, and better documentation would
only help providing those distributions with some aid for the migration. 

\section{New infrastructure -- the basics}
\label{sec:basics}

There are three type of files: \tlpsrc, \tlpobj, \tlpdb. The
format of these files are very similar to Debian's Package files: a
simple list of 
\begin{center}
  |key    value|
\end{center}
(without leading spaces). Empty lines at the beginning and end of a
\tlpsrc\ or \tlpobj\ file are ignored. Also lines starting with \#\ are
ignored (to allow for comments). Some special cases apply for \tlpobj\
files and the \tlpdb\ file, see below.

\subsection{\tlpsrc\ file format}
\label{sec:tlpsrc}

The possible keys and their respective interpretation for the \tlpsrc\
files are: 
\begin{description}
\item[name] identifies the package, |value| must consist only of
  \verb+[-_a-zA-Z0-9]+.

\item[category]
  identifies the category into which this package belongs. Possible
  categories are |Collection|, |Scheme|, |TLCore|, |Documentation|,
  |Package|. There are no particular checks as to whether a
  |tlpsrc| file called |collection-something| actually belongs to the
  category |Collection|. Most packages will fall into the |Package|
  category.  These categories were inherited from the \tpm\ world and
  may be modified at some point; for now, they suffice.

\item[catalogue]
  identifies the name under which this package can be found in the \TeX\ 
  Catalogue. If not present the name of the packages is taken as the
  Catalogue entry.

\item[shortdesc]
  gives a one line description of the package. Susequent entries will
  overwrite the former ones. In \tl\ only used for collections and
  schemes. 

\item[longdesc]
  gives a long description of the package. Susequent entries are added
  up to a long text. In \tl\ only used for collections and
  schemes. 

\item[depend]
  gives the list of dependencies of the package in the form
  \begin{center}
    |Category/Name|
  \end{center}
  for |value|. All the depend lines contribute to the dependencies of
  the package. 

\item[execute]
  gives a free form entry of install time jobs to be
  executed. Currently the following values are understood by
  the installers: 
  \begin{itemize}
  \item |execute addMap font.map|\\
    enables the font map file |font.map| in the |updmap.cfg| file.
  \item |execute addMixedMap font.map|\\
    enables the font map file |font.map| for Mixed mode in the
    |updmap.cfg| file.  By the way,
    the purpose of MixedMap is to help users with printers that render
    the Type~1 versions of the fonts worse than the (mode-tuned) 
    bitmap-based Type~3 fonts. The entries from MixedMap are not
    added to psfonts\_pk.map; that's the only difference.  It's used for
    fonts which have both Metafont and (typically) autotraced Type~1
    instantations.

  \item |execute BuildLanguageDat NN|\\
    activates all the hyphenation patterns found in
    \path{Master/texmf/tex/generic/config/language.NN.dat} in the
    generated |language.dat| file.
  \item |execute BuildFormat FMTCFG|\\
    activates all the formats present in
    \path{Master/texmf/fmtutil/format.FMTCFG.cnf}
    in the generated |fmtutil.cnf| file.
  \end{itemize}

{
\UndefineShortVerb{\|}
\item[(src|run|doc|bin)pattern pattern]
  adds a pattern (see below) to the respective list of patterns.
}

\end{description}

\subsubsection{Patterns}
\label{sec:patterns}

To automatically generate the file lists in \tlpobjs\ a specific
pattern language was developed. Patterns can include or exclude files
or whole sub-directories into the file list of a \tlpobj. Patterns are
of the form
\begin{center}
  |[PREFIX]TYPE PAT|
\end{center}
where |PREFIX| can be |+|, |!+|, or |!|, and |TYPE| one of the letters
|t|, |f|, |d|, |r|. An initial |+| for the pattern indicates that the
automatically generated pattern should not be cleared (see below),
while the |!| indicates that the matching files should be excluded.

The meaning of the various pattern types is
\begin{description}
\item[f path]
  includes all files which match |path| where \emph{only} the last
  component of |path| can contain the usual glob characters * and ?
  (but no others!).
\item[d path]
  includes all the files in and below the directory specified as
  |path|. 
\item[t word1 ... wordN wordL]
  includes all the files in and below all directories of the form
  \begin{center}
    \path{word1/word2/.../wordN/.../any/dirs/.../wordL/}
  \end{center}
  i.e., all words but the last form the prefix of the path, then
  there can be an arbitrary nesting of directories, followed by
  |wordL| as another directory.
\item[r regexp]
  includes all files matching the Perl regexp \verb+/^regexp$/+
\end{description}


\subsubsection{Autogenerated patterns}
\label{sec:autogenpat}

In the case that one of the pattern sections is empty or \emph{all}
the provided patterns have the prefix |+| (e.g., |+f ...|), then the
following patterns are \emph{automatically} added at expansion time
(but never written to the textual representation):

\begin{itemize}
\item for runpatterns of category |Package|
  \begin{center}
    |t texmf-dist topdir name|
  \end{center}
  where |topdir| is one of: bibtex, context, dvips, fonts, makeindex,
  metafont, metapost, mft, omega, scripts, tex, vtex. |name| refers to
  the name of the package as given by the |name| directive.

  For other categories \emph{no} patterns are automatically added to
  the list of runpatterns.
\item for docpattern of category |Package|
  \begin{center}
    |t texmf-dist doc name|
  \end{center}
  and for docpattern of category |Documentation|
  \begin{center}
    |t texmf-doc doc name|
  \end{center}
\item for srcpattern of category |Package|
  \begin{center}
    |t texmf-dist source name|
  \end{center}
  and for srcpattern of category |Documentation|
  \begin{center}
    |t texmf-doc source name|
  \end{center}
\end{itemize}
|binpatterns| are never automatically added.

In addition some magic tricks have been added to make writing of
|binpatterns| easier:
\begin{description}
\item[arch expansion]
  In case the string \verb+${ARCH}+ occurs in one |binpattern| it is 
  automatically expanded to the respective architecture.
\item[bat/exe/dll for win32]
  For |binpatterns| of the form |f bin/win32/foobar| (i.e., also for a
  binpattern of the form \verb+f bin/${ARCH}/foobar+) files
  |foobar.bat|, |foobar.dll|, and |foobar.exe| are also matched.
\end{description}
The above two properties allows to capture the binaries for all
architectures in one binpattern:
\begin{center}
        \verb+binpattern f bin/${ARCH}/dvips+
\end{center}
will include |bin/win32/dvips.exe| in the runfiles when arch=win32. 

Note that the bat/exe expansion \emph{only} works for patterns of the
|f|-type. 

\subsection{\tlpobj\ file format}
\label{sec:tlpobj}

These files are structurally similar to \tlpsrc\ files; the
only difference is that the |*pattern| keys are not allowed, their
place being taken by |*files| keys having a peculiar syntax (see
below). Furthermore, an additional key |revision| and all keys
matching |catalogue-*| are allowed.  
The |catalogue-*| keys specify information simply copied from the \TeX\
Catalogue.  We'll discuss the others below.

\subsubsection{The \texttt{revision} entry}

The |revision| key is {\em not} related to the package's version as
specified by the author.  Those versions are far too random, hard to
extract, and sometimes not even present, to be a reliable basis for a
packaging system.  Instead, we look at all the package's files in the
\tl\ source repository, which is currently Subversion, and use the
maximum version number found there.  Since Subversion uses simple
integers for version numbers, this is simple to use in programs, and
just as important, this whole process of finding the version number can
be reliably automated.

\subsubsection{The \texttt{*files} entries}
\label{sec:keyfiles}

While the |tlpsrc| files specify the files to be included using the 
patterns, the |tlpobj| files contain the already expanded file list.
All the `files' keys have in common that they are followed by a list of
files (and possibly additional tags) indented by exactly one (1) space. They
differ only in the first line itself (described below).



\begin{description}
\item[srcfiles, runfiles] each of these lines is (or better should be)
  tagged with |size=NNNN| where |NNNN| is the sum of sizes of the single 
  files (currently in bytes), e.g.,
  \begin{center}
    |srcfiles size=NNNNNN|
  \end{center}
\item[docfiles] The docfiles line itself is similar to the |srcfiles|
  and |runfiles| lines above:
  \begin{center}
    |docfiles size=NNNNNN|
  \end{center}
  But the lines listing the files are allowed to have additional tags
  |details| and |language|:
  \begin{center}
    |file detail="foo bar" language="en"|
  \end{center}
  For an example see listing~\ref{achemso}. The reason to add these
  tags is the hope that someone will write a nice replacement for
  \path{texdoctk}. Note that these tags' source is the \TeX\ Catalogue, 
  and are in no way obligatory.
\item[binfiles] Since |binfiles| are different for the different
  architectures, one \tlpobj\ file can contain |binfiles| lines for
  different architectures. The architecture is specified on the
  |binfiles| lines using the |arch=XXX| tag. Thus, |binfiles| lines look like
  \begin{center}
    |binfiles arch=XXXX size=NNNNN|
  \end{center}
  A more complete example taken from |bin-dvipsk.tlpobj| can be seen
  in listing~\ref{dvipsk}.
\end{description}

\begin{lstlisting}[caption={Excerpt from achemso.tlpobj},label=achemso]
...
docfiles size=135842
 texmf-dist/doc/latex/achemso/README details="Package Readme" language="de"
 texmf-dist/doc/latex/achemso/achemso.pdf details="Package documentation" language="de"
 ...
\end{lstlisting}

\begin{lstlisting}[caption={Excerpt from bin-dvipsk.tlpobj},label=dvipsk]
name bin-dvipsk
category TLCore
revision 4427
docfiles size=959434
 texmf/doc/dvips/dvips.html
 ...
runfiles size=1702468
 texmf/dvips/base/color.pro
 ...
 texmf/scripts/pkfix/pkfix.pl
binfiles arch=i386-solaris size=329700
 bin/i386-solaris/afm2tfm
 bin/i386-solaris/dvips
 bin/i386-solaris/pkfix
binfiles arch=win32 size=161280
 bin/win32/afm2tfm.exe
 bin/win32/dvips.exe
 bin/win32/pkfix.exe
...
\end{lstlisting}

\subsection{\tlpdb\ file format}
\label{sec:tlpdb}

A \tlpdb\ file will describe the status of an installation, that is it
will contain all the installed packages' \tlpobj\ files. Thus,
the format of a \tlpdb\ file is quite simple. It is the concatenation
of \tlpobjs\ with (at least) one empty line between different
\tlpobjs. 


\section{Programming APIs}

Wrapping these file formats into a programming API represents
the actual work of realizing the new infrastructure. On the way we had
to create not only modules for accessing the content of the above
files in some object-oriented way, we also created some additional
modules for interfacing to the Subversion repository and the \TeX\
Catalogue. 

Currently, the following modules are available in the \tl\ subversion
repository:
\begin{description}
\item[TeXLive::TLTREE] an object that exhibits the properties of the
  subversion repository in some structured form. In principle it is
  the scooping up of |svn status -v| output with post-processing
  for easier and faster searches.
\item[TeXLive::TeXCatalogue] a very simple interface to the \TeX\
  Catalogue, barely providing minimal information. Will be used for
  enriching the final database.
\item[TeXLive::TLPSRC] provides access to \tlpsrc\ files, basic
  functionality like reading in, writing out, and member access
  functions. In addition, it allows to generate, or expand, a |TLPSRC|
  object to a |TLPOBJ| object using an instance of |TLTREE|. That is,
  it takes the patterns specified in the \tlpsrc, and tries to find
  all files matching these patterns. After this it recomputes the size
  and returns a |TLPOBJ| object.
\item[TeXLive::TLPOBJ] provides access to \tlpobj\ files, and entries
  stored in a a |TLPDB| object. As with the \tlpsrc, basic
  functionalities are exported, and also a function to create a zip
  file for the |TLPOBJ| (which later should be used for web updates or
  building the `inst' CD),
  and together with an instance of a |TeXCatalogue| object some
  information can be transferred from the \TeX\ Catalogue to the
  |TLPOBJ| object.
\item[TeXLive::TLPDB] provides access to the \tl\ database. These
  files will serve the central purpose of describing the current
  status of various media, like an installation on a user computer, or
  the distribution. Comparing these status descriptions, update
  programs can deduce what packages should be updated.
\item[TeXLive::TLPUtils] exports some handy functions used in all the
  other modules.
\end{description}

The full API documentation is provided in |pod| format within the perl
modules. A Perl API document
is available in the \tl\ subversion repository, too.

In addition to the Perl API a start at a shell (sh, bash, etc.)\ API and
implementation has been created, but it presently lacks several key features,
and is not up to date. Jim Hef{}feron has contributed a start at a Python API.
However, we assume that on all computers Perl will be
available or will be installed during installation.

\section{Integration into current \tl\ development}

Around these modules several day-in-day-out tasks of the \tl\
developers have been rewritten using the new infrastructure, the most
important one being the script |ctan2tl|. This is the script responsible for
transferring a package from \ctan\ into the right places in
the \tl\ repository, and preparing it for users.

Other things already converted are automatic update of the
|texlive.tlpdb|, the file containing the |TLPDB|.

On the other hand not everything has been done, the biggest piece for
sure is the rewrite of the installer. The old installer, relying on
the lists files we wanted to get rid of, is still the only one we
have. Furtunately we can generate lists files from the
|texlive.tlpdb|, but in the not-so-long run, meaning hopefully for
\tl~2008, we want to have a new installer.

Other things still missing are coverage and duplication checks, but
those are quite easy to do using the Unix command world (|grep|,
|sort|, and |uniq| being the magic incantations).

\section{Closing}
\label{sec:closing}

Although we are quite confident that the new infrastructure will work
well, only the reality of the next release will prove us right or
wrong. Many of the key features of the new infrastructure have been
designed with the \tpm-system in mind, and some `features' carried
over will probably proven superfluous. Nothing with the current
infrastructure is written in stone, and we are open for proposals and
improvements, keeping in mind that releasing \tl\ should not be made
more difficult. 

That said, we invite contributors and everyone interested to contact
us at \url{tex-live@tug.org}, or take a look at the following
resources: 
\begin{itemize}
\item \path{http://www.tug.org/texlive} -- the main entry point, with
  links to developers' resources, documentation
\item \path{http://www.tug.org/svn/texlive/trunk/} -- web view onto the
  subversion repository;
\item \path{svn://tug.org/texlive/trunk} -- svn repository, anonymous
  access, take care when you checking out the repository, it needs
  several Gigabyte of disk space;
\item \path{http://www.tug.org/texlive/pkgupdate.html} -- an
  explanation how updates from \ctan\ to \tl\ are done.
\end{itemize}

Finally, I want to thank all the contributors to \tl, there are too
many to mention. In the course of developing this infrastructure the
discussions with Karl Berry, Pawe\l{} Jackowski, Reinhard Kotucha,
Siep Kroonenberg, and Jerzy B. Ludwichowski were very helpful. 

%\bibliographystyle{arstexnica}
%\bibliography{atsample}
  
\end{article}

\end{document}