diff options
Diffstat (limited to 'Master/tlpkg')
| -rw-r--r-- | Master/tlpkg/doc/tlinfra-guit07.pdf | bin | 0 -> 302548 bytes | |||
| -rw-r--r-- | Master/tlpkg/doc/tlinfra-guit07.tex | 585 |
2 files changed, 585 insertions, 0 deletions
diff --git a/Master/tlpkg/doc/tlinfra-guit07.pdf b/Master/tlpkg/doc/tlinfra-guit07.pdf Binary files differnew file mode 100644 index 00000000000..8cc595be9ce --- /dev/null +++ b/Master/tlpkg/doc/tlinfra-guit07.pdf diff --git a/Master/tlpkg/doc/tlinfra-guit07.tex b/Master/tlpkg/doc/tlinfra-guit07.tex new file mode 100644 index 00000000000..1b3e88f56c9 --- /dev/null +++ b/Master/tlpkg/doc/tlinfra-guit07.tex @@ -0,0 +1,585 @@ +\documentclass{arstexnica} +\usepackage[latin1]{inputenc} +\usepackage[T1]{fontenc} +\usepackage{lmodern} +\usepackage[english]{babel} +\usepackage{graphicx} +\usepackage{hyperref} +\usepackage{fancyvrb} +\usepackage{url} +\usepackage{marvosym} +\usepackage{listings} +\lstset{frame=lines,basicstyle=\ttfamily,showspaces=true,prebreak={\Righttorque},postbreak={\Lefttorque},breaklines} +\newcommand{\tl}{\TeX~Live} +\newcommand{\ctan}{CTAN} +\newcommand{\tpm}{\texttt{tpm}} +\newcommand{\tpms}{\tpm{}s} +\newcommand{\tlpsrc}{\texttt{tlpsrc}} +\newcommand{\tlpsrcs}{\tlpsrc{}s} +\newcommand{\tlpobj}{\texttt{tlpobj}} +\newcommand{\tlpobjs}{\tlpobj{}s} +\newcommand{\tlpdb}{\texttt{tlpdb}} +\newcommand{\tlpdbs}{\tlpdb{}s} + +\hyphenation{infra-struc-ture} + +%\catcode`>=\active +%\def>#1<{\texttt{#1}} +\DefineShortVerb{\|} +\begin{document} +\begin{article} +%\selectlanguage{italian}% + +\title{\tl's new infrastructure\thanks{% + Article presented on the 4th GuITmeeting, Pisa 13 October 2007}} + +\author{Norbert Preining} +\address{Vienna University of Technology\\ + Wiedner Hauptstr.\ 10\\ + 1040 Wien, Austria} +\netaddress{preining@logic.at} + +\maketitle + + +\begin{abstract} +Since the release of \tl~2007 a new infrastructure for \tl\ +distribution and management has been developed. This article presents the +reasons for this switch, the ideas behind the new infrastructure, +software developed, and ways to incorporate this new +infrastructure. We will close with a look at what new features +this new infrastructure could bring to the \TeX\ (Live) world. +\end{abstract} + +% +% does not work in normal arstexnika mode, needs standalone, I leave +% this up to the editors +%\tableofcontents + +\section{Introduction} +\label{sec:intro} + +\tl\ is an easy way to get up and running with \TeX. It provides a +comprehensive \TeX\ system with binaries for most flavors of Unix, +including GNU/Linux, and also Windows. It includes all the major +\TeX-related programs, macro packages, and fonts that are free +software, including support for many languages around the world. + +Since 1996 it tries to bring to all \TeX-users as much material +from \ctan\ as possible, packaged for `consumption', i.e., for using it +as a live system from DVD, or installing it on a variety of operating +systems/architecture combinations. + +These two requirements, incorporating as much as possible from \ctan, +and providing binaries for a wide range of OS/arch combinations, has led +to a huge number of supporting scripts, in a wide variety of programming +languages (Perl, shell, XML, C, \ldots). These scripts were (and are) +used to generate installation media, check consistency, update +important files (e.g., for the installer), incorporating packages from +\ctan, just to name a few. There have been many contributors; Sebastian +Rahtz, Fabrice Popineau, and Karl Berry are the principal authors. + +As always with overload volunteers working on big projects, not much +was documented, programming was done by trial and error (see for +example the packaging scripts of Debian, one of my own more horrible +creations). This wouldn't have been reason enough to change things +this deep in the intestines of \tl, but other illusions and dreams +have driven us to rebuild from the ground up. + +\section{A world full of TPMs} +\label{sec:tpm} + +Up to \tl~2007 everything in \tl\ was organized in \tpms, an +acronym for `\TeX\ Package Manage[r/ment]'. One \tpm\ described more +or less one package from \ctan, containing the list of included +files, title, description, license information, sometimes version +numbers, additional information necessary for incorporation +(activation of map files, hyphenation patterns, and formats). + +Maintenance of these \tpms\ was mostly done by a Perl module |Tpm.pm| +written by Fabrice Popineau, and the accompanying mystical program +|tpm-factory.pl|, a script so full of possibilities that nobody +besides Fabrice probably ever understood everything that could be done with +it. Some of the jobs of this |tpm-factory.pl| were +\begin{itemize} +\item regeneration of the \tpms, this included some magic in finding + the right files +\item creation of a new \tpm\ for a newly installed package from \ctan +\item checking the coverage, i.e., checking that every file present in + the repository is actually contained in a \tpm. +\item duplication and dependency checks +\end{itemize} + +Alas, there have been some problems with all the \tpm-business: +\begin{itemize} +\item they contained a mixture of generated (file lists) and static + information (actions to be carried out); +\item they were full of duplicate information: version, license, + descriptions were taken now and then from the \TeX\ Catalogue, but + were typically horribly out of date or otherwise wrong; +\item we had to generate so-called `lists' files (line-oriented plain + text file lists) from the \tpms\ for the installer + because the XML syntax of the \tpms\ is not practical to parse from a simple + shell script. +\end{itemize} + +\section{Aims of the new infrastructure} +\label{sec:aims} + +So around mid-2006 discussion of a new infrastructure started, but +unfortunately, due to the usual time constraints of all involved, +without much concrete outcome except many emails. At least a preliminary +catalogue of items to be improved sprang into existence: + + +\paragraph{Separation of static from generated content} + +As already mentioned, the \tpms\ contained a wild mixture of stuff from +various sources (the file itself, the \TeX\ Catalogue, the +repository). The new infrastructure should have some kind of `source' +files where {\em only} the necessary information is stored, and all other +content is added at generation time (of whatever). Naturally that +could have been done on top of the \tpms, too, but starting from +scratch seemed to be a better option. + +\paragraph{Getting rid of lists files} + +Using the \tpms\ and some XSLT processing, the lists files were generated +before release. These lists files (one for each package, about 2000 in all) +must be read by the installer from the DVD, which caused a lot of +headache and slow installations. +%Furthermore, these list files often were forgotten, thus out of date. +%(during development, but not at the end! i hope :) --karl) + +The sole reason for the existence of these lists files was the fact +that the installer, written in shell, couldn't parse the XML \tpms, +since XML parsing programs cannot be assumed to be present at all +installations. The new infrastructure should be based on some format +that is easily parseable using stuff normally already present, or at +least necessary for \tl\ anyway. + +\paragraph{Single package updates via the web} + +Some \TeX\ distributions, notably MiKTeX, already support single package +updates over the Internet. +\tl, as big and nice as it is, still lacks this lovely feature. Of course +it was put high on the priority list to allow for single package +updates. + +\paragraph{Better documentation} + +Last, but not least, we hoped that by rewritting the infrastructure +and documenting it on the way we could provide a more stable foundation +for more development and improvement, thus attracting more +contributors. Furthermore, since te\TeX\ development has been stopped, +\tl\ is taking the place as the \TeX\ system of choice in many +GNU/Linux and other free distributions, and better documentation would +only help providing those distributions with some aid for the migration. + +\section{New infrastructure -- the basics} +\label{sec:basics} + +There are three type of files: \tlpsrc, \tlpobj, \tlpdb. The +format of these files are very similar to Debian's Package files: a +simple list of +\begin{center} + |key value| +\end{center} +(without leading spaces). Empty lines at the beginning and end of a +\tlpsrc\ or \tlpobj\ file are ignored. Also lines starting with \#\ are +ignored (to allow for comments). Some special cases apply for \tlpobj\ +files and the \tlpdb\ file, see below. + +\subsection{\tlpsrc\ file format} +\label{sec:tlpsrc} + +The possible keys and their respective interpretation for the \tlpsrc\ +files are: +\begin{description} +\item[name] identifies the package, |value| must consist only of + \verb+[-_a-zA-Z0-9]+. + +\item[category] + identifies the category into which this package belongs. Possible + categories are |Collection|, |Scheme|, |TLCore|, |Documentation|, + |Package|. There are no particular checks as to whether a + |tlpsrc| file called |collection-something| actually belongs to the + category |Collection|. Most packages will fall into the |Package| + category. These categories were inherited from the \tpm\ world and + may be modified at some point; for now, they suffice. + +\item[catalogue] + identifies the name under which this package can be found in the \TeX\ + Catalogue. If not present the name of the packages is taken as the + Catalogue entry. + +\item[shortdesc] + gives a one line description of the package. Susequent entries will + overwrite the former ones. In \tl\ only used for collections and + schemes. + +\item[longdesc] + gives a long description of the package. Susequent entries are added + up to a long text. In \tl\ only used for collections and + schemes. + +\item[depend] + gives the list of dependencies of the package in the form + \begin{center} + |Category/Name| + \end{center} + for |value|. All the depend lines contribute to the dependencies of + the package. + +\item[execute] + gives a free form entry of install time jobs to be + executed. Currently the following values are understood by + the installers: + \begin{itemize} + \item |execute addMap font.map|\\ + enables the font map file |font.map| in the |updmap.cfg| file. + \item |execute addMixedMap font.map|\\ + enables the font map file |font.map| for Mixed mode in the + |updmap.cfg| file. By the way, + the purpose of MixedMap is to help users with printers that render + the Type~1 versions of the fonts worse than the (mode-tuned) + bitmap-based Type~3 fonts. The entries from MixedMap are not + added to psfonts\_pk.map; that's the only difference. It's used for + fonts which have both Metafont and (typically) autotraced Type~1 + instantations. + + \item |execute BuildLanguageDat NN|\\ + activates all the hyphenation patterns found in + \path{Master/texmf/tex/generic/config/language.NN.dat} in the + generated |language.dat| file. + \item |execute BuildFormat FMTCFG|\\ + activates all the formats present in + \path{Master/texmf/fmtutil/format.FMTCFG.cnf} + in the generated |fmtutil.cnf| file. + \end{itemize} + +{ +\UndefineShortVerb{\|} +\item[(src|run|doc|bin)pattern pattern] + adds a pattern (see below) to the respective list of patterns. +} + +\end{description} + +\subsubsection{Patterns} +\label{sec:patterns} + +To automatically generate the file lists in \tlpobjs\ a specific +pattern language was developed. Patterns can include or exclude files +or whole sub-directories into the file list of a \tlpobj. Patterns are +of the form +\begin{center} + |[PREFIX]TYPE PAT| +\end{center} +where |PREFIX| can be |+|, |!+|, or |!|, and |TYPE| one of the letters +|t|, |f|, |d|, |r|. An initial |+| for the pattern indicates that the +automatically generated pattern should not be cleared (see below), +while the |!| indicates that the matching files should be excluded. + +The meaning of the various pattern types is +\begin{description} +\item[f path] + includes all files which match |path| where \emph{only} the last + component of |path| can contain the usual glob characters * and ? + (but no others!). +\item[d path] + includes all the files in and below the directory specified as + |path|. +\item[t word1 ... wordN wordL] + includes all the files in and below all directories of the form + \begin{center} + \path{word1/word2/.../wordN/.../any/dirs/.../wordL/} + \end{center} + i.e., all words but the last form the prefix of the path, then + there can be an arbitrary nesting of directories, followed by + |wordL| as another directory. +\item[r regexp] + includes all files matching the Perl regexp \verb+/^regexp$/+ +\end{description} + + +\subsubsection{Autogenerated patterns} +\label{sec:autogenpat} + +In the case that one of the pattern sections is empty or \emph{all} +the provided patterns have the prefix |+| (e.g., |+f ...|), then the +following patterns are \emph{automatically} added at expansion time +(but never written to the textual representation): + +\begin{itemize} +\item for runpatterns of category |Package| + \begin{center} + |t texmf-dist topdir name| + \end{center} + where |topdir| is one of: bibtex, context, dvips, fonts, makeindex, + metafont, metapost, mft, omega, scripts, tex, vtex. |name| refers to + the name of the package as given by the |name| directive. + + For other categories \emph{no} patterns are automatically added to + the list of runpatterns. +\item for docpattern of category |Package| + \begin{center} + |t texmf-dist doc name| + \end{center} + and for docpattern of category |Documentation| + \begin{center} + |t texmf-doc doc name| + \end{center} +\item for srcpattern of category |Package| + \begin{center} + |t texmf-dist source name| + \end{center} + and for srcpattern of category |Documentation| + \begin{center} + |t texmf-doc source name| + \end{center} +\end{itemize} +|binpatterns| are never automatically added. + +In addition some magic tricks have been added to make writing of +|binpatterns| easier: +\begin{description} +\item[arch expansion] + In case the string \verb+${ARCH}+ occurs in one |binpattern| it is + automatically expanded to the respective architecture. +\item[bat/exe/dll for win32] + For |binpatterns| of the form |f bin/win32/foobar| (i.e., also for a + binpattern of the form \verb+f bin/${ARCH}/foobar+) files + |foobar.bat|, |foobar.dll|, and |foobar.exe| are also matched. +\end{description} +The above two properties allows to capture the binaries for all +architectures in one binpattern: +\begin{center} + \verb+binpattern f bin/${ARCH}/dvips+ +\end{center} +will include |bin/win32/dvips.exe| in the runfiles when arch=win32. + +Note that the bat/exe expansion \emph{only} works for patterns of the +|f|-type. + +\subsection{\tlpobj\ file format} +\label{sec:tlpobj} + +These files are structurally similar to \tlpsrc\ files; the +only difference is that the |*pattern| keys are not allowed, their +place being taken by |*files| keys having a peculiar syntax (see +below). Furthermore, an additional key |revision| and all keys +matching |catalogue-*| are allowed. +The |catalogue-*| keys specify information simply copied from the \TeX\ +Catalogue. We'll discuss the others below. + +\subsubsection{The \texttt{revision} entry} + +The |revision| key is {\em not} related to the package's version as +specified by the author. Those versions are far too random, hard to +extract, and sometimes not even present, to be a reliable basis for a +packaging system. Instead, we look at all the package's files in the +\tl\ source repository, which is currently Subversion, and use the +maximum version number found there. Since Subversion uses simple +integers for version numbers, this is simple to use in programs, and +just as important, this whole process of finding the version number can +be reliably automated. + +\subsubsection{The \texttt{*files} entries} +\label{sec:keyfiles} + +While the |tlpsrc| files specify the files to be included using the +patterns, the |tlpobj| files contain the already expanded file list. +All the `files' keys have in common that they are followed by a list of +files (and possibly additional tags) indented by exactly one (1) space. They +differ only in the first line itself (described below). + + + +\begin{description} +\item[srcfiles, runfiles] each of these lines is (or better should be) + tagged with |size=NNNN| where |NNNN| is the sum of sizes of the single + files (currently in bytes), e.g., + \begin{center} + |srcfiles size=NNNNNN| + \end{center} +\item[docfiles] The docfiles line itself is similar to the |srcfiles| + and |runfiles| lines above: + \begin{center} + |docfiles size=NNNNNN| + \end{center} + But the lines listing the files are allowed to have additional tags + |details| and |language|: + \begin{center} + |file detail="foo bar" language="en"| + \end{center} + For an example see listing~\ref{achemso}. The reason to add these + tags is the hope that someone will write a nice replacement for + \path{texdoctk}. Note that these tags' source is the \TeX\ Catalogue, + and are in no way obligatory. +\item[binfiles] Since |binfiles| are different for the different + architectures, one \tlpobj\ file can contain |binfiles| lines for + different architectures. The architecture is specified on the + |binfiles| lines using the |arch=XXX| tag. Thus, |binfiles| lines look like + \begin{center} + |binfiles arch=XXXX size=NNNNN| + \end{center} + A more complete example taken from |bin-dvipsk.tlpobj| can be seen + in listing~\ref{dvipsk}. +\end{description} + +\begin{lstlisting}[caption={Excerpt from achemso.tlpobj},label=achemso] +... +docfiles size=135842 + texmf-dist/doc/latex/achemso/README details="Package Readme" language="de" + texmf-dist/doc/latex/achemso/achemso.pdf details="Package documentation" language="de" + ... +\end{lstlisting} + +\begin{lstlisting}[caption={Excerpt from bin-dvipsk.tlpobj},label=dvipsk] +name bin-dvipsk +category TLCore +revision 4427 +docfiles size=959434 + texmf/doc/dvips/dvips.html + ... +runfiles size=1702468 + texmf/dvips/base/color.pro + ... + texmf/scripts/pkfix/pkfix.pl +binfiles arch=i386-solaris size=329700 + bin/i386-solaris/afm2tfm + bin/i386-solaris/dvips + bin/i386-solaris/pkfix +binfiles arch=win32 size=161280 + bin/win32/afm2tfm.exe + bin/win32/dvips.exe + bin/win32/pkfix.exe +... +\end{lstlisting} + +\subsection{\tlpdb\ file format} +\label{sec:tlpdb} + +A \tlpdb\ file will describe the status of an installation, that is it +will contain all the installed packages' \tlpobj\ files. Thus, +the format of a \tlpdb\ file is quite simple. It is the concatenation +of \tlpobjs\ with (at least) one empty line between different +\tlpobjs. + + +\section{Programming APIs} + +Wrapping these file formats into a programming API represents +the actual work of realizing the new infrastructure. On the way we had +to create not only modules for accessing the content of the above +files in some object-oriented way, we also created some additional +modules for interfacing to the Subversion repository and the \TeX\ +Catalogue. + +Currently, the following modules are available in the \tl\ subversion +repository: +\begin{description} +\item[TeXLive::TLTREE] an object that exhibits the properties of the + subversion repository in some structured form. In principle it is + the scooping up of |svn status -v| output with post-processing + for easier and faster searches. +\item[TeXLive::TeXCatalogue] a very simple interface to the \TeX\ + Catalogue, barely providing minimal information. Will be used for + enriching the final database. +\item[TeXLive::TLPSRC] provides access to \tlpsrc\ files, basic + functionality like reading in, writing out, and member access + functions. In addition, it allows to generate, or expand, a |TLPSRC| + object to a |TLPOBJ| object using an instance of |TLTREE|. That is, + it takes the patterns specified in the \tlpsrc, and tries to find + all files matching these patterns. After this it recomputes the size + and returns a |TLPOBJ| object. +\item[TeXLive::TLPOBJ] provides access to \tlpobj\ files, and entries + stored in a a |TLPDB| object. As with the \tlpsrc, basic + functionalities are exported, and also a function to create a zip + file for the |TLPOBJ| (which later should be used for web updates or + building the `inst' CD), + and together with an instance of a |TeXCatalogue| object some + information can be transferred from the \TeX\ Catalogue to the + |TLPOBJ| object. +\item[TeXLive::TLPDB] provides access to the \tl\ database. These + files will serve the central purpose of describing the current + status of various media, like an installation on a user computer, or + the distribution. Comparing these status descriptions, update + programs can deduce what packages should be updated. +\item[TeXLive::TLPUtils] exports some handy functions used in all the + other modules. +\end{description} + +The full API documentation is provided in |pod| format within the perl +modules. A Perl API document +is available in the \tl\ subversion repository, too. + +In addition to the Perl API a start at a shell (sh, bash, etc.)\ API and +implementation has been created, but it presently lacks several key features, +and is not up to date. Jim Hef{}feron has contributed a start at a Python API. +However, we assume that on all computers Perl will be +available or will be installed during installation. + +\section{Integration into current \tl\ development} + +Around these modules several day-in-day-out tasks of the \tl\ +developers have been rewritten using the new infrastructure, the most +important one being the script |ctan2tl|. This is the script responsible for +transferring a package from \ctan\ into the right places in +the \tl\ repository, and preparing it for users. + +Other things already converted are automatic update of the +|texlive.tlpdb|, the file containing the |TLPDB|. + +On the other hand not everything has been done, the biggest piece for +sure is the rewrite of the installer. The old installer, relying on +the lists files we wanted to get rid of, is still the only one we +have. Furtunately we can generate lists files from the +|texlive.tlpdb|, but in the not-so-long run, meaning hopefully for +\tl~2008, we want to have a new installer. + +Other things still missing are coverage and duplication checks, but +those are quite easy to do using the Unix command world (|grep|, +|sort|, and |uniq| being the magic incantations). + +\section{Closing} +\label{sec:closing} + +Although we are quite confident that the new infrastructure will work +well, only the reality of the next release will prove us right or +wrong. Many of the key features of the new infrastructure have been +designed with the \tpm-system in mind, and some `features' carried +over will probably proven superfluous. Nothing with the current +infrastructure is written in stone, and we are open for proposals and +improvements, keeping in mind that releasing \tl\ should not be made +more difficult. + +That said, we invite contributors and everyone interested to contact +us at \url{tex-live@tug.org}, or take a look at the following +resources: +\begin{itemize} +\item \path{http://www.tug.org/texlive} -- the main entry point, with + links to developers' resources, documentation +\item \path{http://www.tug.org/svn/texlive/trunk/} -- web view onto the + subversion repository; +\item \path{svn://tug.org/texlive/trunk} -- svn repository, anonymous + access, take care when you checking out the repository, it needs + several Gigabyte of disk space; +\item \path{http://www.tug.org/texlive/pkgupdate.html} -- an + explanation how updates from \ctan\ to \tl\ are done. +\end{itemize} + +Finally, I want to thank all the contributors to \tl, there are too +many to mention. In the course of developing this infrastructure the +discussions with Karl Berry, Pawe\l{} Jackowski, Reinhard Kotucha, +Siep Kroonenberg, and Jerzy B. Ludwichowski were very helpful. + +%\bibliographystyle{arstexnica} +%\bibliography{atsample} + +\end{article} + +\end{document} + + |