% \iffalse
%% File: source3.dtx Copyright (C) 1990-2009 LaTeX3 project
%%
%% It may be distributed and/or modified under the conditions of the
%% LaTeX Project Public License (LPPL), either version 1.3c of this
%% license or (at your option) any later version.  The latest version
%% of this license is in the file
%%
%%    http://www.latex-project.org/lppl.txt
%%
%% This file is part of the ``expl3 bundle'' (The Work in LPPL)
%% and all files in that bundle must be distributed together.
%%
%% The released version of this bundle is available from CTAN.
%%
%% -----------------------------------------------------------------------
%%
%% The development version of the bundle can be found at
%%
%%    http://www.latex-project.org/svnroot/experimental/trunk/
%%
%% for those people who are interested.
%%
%%%%%%%%%%%
%% NOTE: %%
%%%%%%%%%%%
%%
%%   Snapshots taken from the repository represent work in progress and may
%%   not work or may contain conflicting material!  We therefore ask
%%   people _not_ to put them into distributions, archives, etc. without
%%   prior consultation with the LaTeX Project Team.
%%
%% -----------------------------------------------------------------------
%% \fi

% This document will typeset the LaTeX3 sources as a single document.
% This will produce quite a large file (roughly ??? pages) and may
% take a long time on a slow machine.

\documentclass{l3doc}
\listfiles

\begin{document}

\title{The \LaTeX3 Sources}
\author{\Team}


\pagenumbering{roman}
\maketitle

\begin{abstract}

\parindent=0pt
\parskip=\baselineskip

\noindent This is the reference documentation for the \pkg{expl3} 
programming environment. The \pkg{expl3} modules set up an experimental
naming scheme for \LaTeX\ commands, which allow the \LaTeX\ programmer
to systematically name functions and variables, and specify the argument
types of functions.

The \TeX\ and \eTeX\ primitives are all given a new name according to
these conventions. However, in the main direct use of the primitives is
not required or encouraged: the \pkg{expl3} modules define an
independent low-level \LaTeX3 programming language.

At present, the \pkg{expl3} modules are designed to be loaded on top of
\LaTeXe. In time, a \LaTeX3 format will be produced based on this code.
This allows the code to be used in \LaTeXe\ packages \emph{now} while a
stand-alone \LaTeX3 is developed.

\begin{bfseries}
  While \pkg{expl3} is still experimental, the bundle is now regarded as
  broadly stable. The syntax conventions and functions provided are now
  ready for wider use. There may still be changes to some functions, but
  these will be minor when compared to the scope of \pkg{expl3}.

  New modules will be added to the distributed version of \pkg{expl3} as
  they reach maturity.
\end{bfseries}

\end{abstract}

\clearpage

{\def\\{:}% fix "newlines" in the ToC
\tableofcontents}

\clearpage
\pagenumbering{arabic}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% Each of the following \DocInput lines includes a file with extension
% .dtx. Each of these files may be typeset separately. For instance
%   pdflatex l3box.dtx
% will typeset the source of the LaTeX3 box commands. If you use the
% Makefile, the index will be generated automatically; e.g.,
%   make doc F=l3box
%
% If this file is processed, each of these separate dtx files will be
% contained as a part of a single document.

\makeatletter
\def\partname{Part}
\def\maketitle{\part{\@title}}
\let\thanks\@gobble
\let\DelayPrintIndex\PrintIndex
\let\PrintIndex\@empty
\makeatother

\part{Introduction to \pkg{expl3} and this document}

This document is intended to act as a comprehensive reference manual 
for the \pkg{expl3} language. A general guide to the \LaTeX3 
programming language is found in \href{expl3.pdf}{expl3.pdf}.

\section{Naming functions and variables} 

\LaTeX3 does not use \texttt{@} as a ``letter'' for defining
internal macros.  Instead, the symbols |_| and \texttt{:}
are used in internal macro names to provide structure. The name of
each \emph{function} is divided into logical units using \texttt{_}, 
while \texttt{:} separates the \emph{name} of the function from the 
\emph{argument specifier} (``arg-spec''). This describes the arguments 
expected by the function. In most cases, each argument is represented 
by a single letter. The complete list of arg-spec letters for a function
is referred to as the \emph{signature} of the function.

Each function name starts with the \emph{module} to which it belongs.
Thus apart from a small number of very basic functions, all \pkg{expl3}
function names contain at least one underscore to divide the module
name from the descriptive name of the function. For example, all
functions concerned with comma lists are in module \texttt{clist} and
begin \cs{clist_}.

Every function must include an argument specifier. For functions which
take no arguments, this will be blank and the function name will end
\texttt{:}. Most functions take one or more arguments, and use the
following argument specifiers:
\begin{description}
  \item[\texttt{D}] The \texttt{D} specifier means \emph{do not use}. 
    All of the \TeX\ primitives are initially \cs{let} to a \texttt{D}
    name, and some are then given a second name.  Only the kernel 
    team should use anything with a \texttt{D} specifier!
  \item[\texttt{N} and \texttt{n}] These mean \emph{no manipulation},
    of a single token for \texttt{N} and of a set of tokens given in 
    braces for \texttt{n}. Both pass the argument though exactly as 
    given. Usually, if you use a single token for an \texttt{n} argument, 
    all will be well.
  \item[\texttt{c}] This means \emph{csname}, and indicates that the 
    argument will be turned into a csname before being used. So
    So \cs{foo:c} |{ArgumentOne}| will act in the same way as \cs{foo:N}
    \cs{ArgumentOne}.
  \item[\texttt{V} and \texttt{v}] These mean \emph{value 
    of variable}. The \texttt{V} and \texttt{v} specifiers are used to 
    get the content of a variable without needing to worry about the
    underlying \TeX\ structure containing the data. A \texttt{V} 
    argument will be a single token (similar to \texttt{N}), for example
    \cs{foo:V} \cs{MyVariable}; on the other hand, using \texttt{v} a 
    csname is constructed first, and then the value is recovered, for 
    example \cs{foo:v} |{MyVariable}|.
  \item[\texttt{o}] This means \emph{expansion once}. In general, the
    \texttt{V} and \texttt{v} specifiers are favoured over \texttt{o}
    for recovering stored information. However, \texttt{o} is useful
    for correctly processing information with delimited arguments.
  \item[\texttt{x}] The \texttt{x} specifier stands for \emph{exhaustive 
    expansion}: the plain \TeX\ \cs{edef}.
  \item[\texttt{f}] The \texttt{f} specifier stands for \emph{full 
    expansion}, and in contrast to \emph{x} stops at the first 
    non-expandable item without trying to execute it.
  \item[\texttt{T} and \texttt{F}] For logic tests, there are the branch
    specifiers \texttt{T} (\emph{true}) and \texttt{F} (\emph{false}). 
    Both specifiers treat the input in the same way as \texttt{n} (no 
    change), but make the logic much easier to see.
  \item[\texttt{p}] The letter \texttt{p} indicates \TeX\ 
    \emph{parameters}. Normally this will be used for delimited 
    functions as \pkg{expl3} provides better methods for creating simple
    sequential arguments.    
  \item[\texttt{w}] Finally, there is the \texttt{w} specifier for 
    \emph{weird} arguments. This covers everything else, but mainly 
    applies to delimited values (where the argument must be terminated 
    by some arbitrary string).
\end{description}
Notice that the argument specifier describes how the argument is 
processed prior to being passed to the underlying function. For example,
\cs{foo:c} will take its argument, convert it to a control sequence and
pass it to \cs{foo:N}.

Variables are named in a similar manner to functions, but begin with
a single letter to define the type of variable:
\begin{description}
  \item[\texttt{c}] Constant: global parameters whose value should not
    be changed.
  \item[\texttt{g}] Parameters whose value should only be set globally.
  \item[\texttt{l}] Parameters whose value should only be set locally.
\end{description}
Each variable name is then build up in a similar way to that of a 
function, typically starting with the module\footnote{The module names are
  not used in case of generic scratch registers defined in the data
  type modules, e.g., the
  \texttt{int} module contains some scratch variables called \cs{l_tmpa_int},
  \cs{l_tmpb_int}, and so on. In such a case adding the module name up front
  to denote the module
  and in the back to indicate the type, as in
  \cs{l_int_tmpa_int} would be very unreadable.}  name
and then a descriptive part.
Variables end with a short identifier to show the variable type:
\begin{description}
  \item[\texttt{bool}] Either true or false.
  \item[\texttt{box}] Box register.
  \item[\texttt{clist}] Comma separated list.
  \item[\texttt{dim}] `Rigid' lengths.
  \item[\texttt{int}] Integer-valued count register.
  \item[\texttt{num}] A `fake' integer type using only macros. Useful for
    setting up allocation routines.
  \item[\texttt{prop}] Property list.
  \item[\texttt{skip}] `Rubber' lengths.
  \item[\texttt{seq}] `Sequence': a data-type used to implement lists 
    (with access at both ends) and stacks.
  \item[\texttt{stream}] An input or output stream (for reading from or 
    writing to, respectively).
  \item[\texttt{tl}] Token list variables: placeholder for a token list.
  \item[\texttt{toks}] Token register.
\end{description}

\subsubsection{Terminological inexactitude}

A word of warning. In this document, and others referring to the \pkg{expl3}
programming modules, we often refer to `variables' and `functions' as if
they were actual constructs from a real programming language.
In truth, \TeX\ is a macro processor, and functions are simply macros that
may or mayn't take arguments and expand to their replacement text.
Many of the common variables are \emph{also} macros, and if placed into the
input stream will simply expand to their definition as well~--- a `function'
with no arguments and a `token list variable' are in truth one and the same.
On the other hand, some `variables' are actually registers that must be initialised and their values set and retreived with specific functions.

The conventions of the \pkg{expl3} code are designed to clearly separate the
ideas of `macros that contain data' and `macros that contain code', and a consistent wrapper is applied to all forms of `data' whether they be macros or actually registers.
This means that sometimes we will use phrases like `the function returns a value', when actually we just mean `the macro expands to something'. Similarly, the term `execute' might be used in place of `expand' or it might refer to the more specific case of `processing in \TeX's stomach' (if you are familiar with the \TeX{}book parlance).

If in doubt, please ask; chances are we've been hasty in writing certain definitions and need to be told to tighten up our terminology. 

\section{Documentation conventions}

This document is typeset with the experimental \pkg{l3doc} class; 
several conventions are used to help describe the features of the code.
A number of conventions are used here to make the documentation clearer.

Each group of related functions is given in a box. For a function with
a ``user'' name, this might read:
\begin{function}{
    \ExplSyntaxOn |
    \ExplSyntaxOff
  }
  \begin{syntax}
    "\ExplSyntaxOn" \dots\ "\ExplSyntaxOff"
  \end{syntax}
  The textual description of how the function works would appear here. The 
  syntax of the function is shown in mono-spaced text to the right of 
  the box. In this example, the function takes no arguments and so the 
  name of the function is simply reprinted.
\end{function}

For programming functions, which use \texttt{_} and \texttt{:} in their
name there are a few additional conventions: If two related functions are given with identical names but 
different argument specifiers, these are termed \emph{variants} of each 
other, and the latter functions are printed in grey to show this more 
clearly. They will carry out the same function but will take different
types of argument:
\begin{function}{
    \seq_new:N |
    \seq_new:c
  }
  \begin{syntax}
    "\seq_new:N" <sequence>
  \end{syntax}
  When a number of variants are described, the arguments are usually
  illustrated only for the base function. Here, <sequence> indicates 
  that \cs{seq_new:N} expects the name of a sequence. From the argument
  specifier, \cs{seq_new:c} also expects a sequence name, but as a 
  name rather than as a control sequence. Each argument given in the
  illustration should be described in the following text.
\end{function}

Some functions are fully expandable, which allows it to be used within 
an \texttt{x}-type argument (in plain \TeX\ terms, inside an \cs{edef}).
These fully expandable functions are indicated in the documentation by 
a star:
\begin{function}{
    \cs_to_str:N / (EXP)
  }
  \begin{syntax}
    "\cs_to_str:N" <cs>
  \end{syntax}
  As with other functions, some text should follow which explains how
  the function works. Usually, only the star will indicate that the 
  function is expandable. In this case, the function expects a <cs>, 
  shorthand for a <control sequence>. 
\end{function}

Conditional (\texttt{if}) functions are normally defined in three variants, with
\texttt{T}, \texttt{F} and \texttt{TF} argument specifiers. This allows
them to be used for different `true'/`false' branches, depending on 
which outcome the conditional is being used to test. To indicate this 
without repetition, this information is given in a shortened form:
\begin{function}{
    \xetex_if_engine: / (TF) (EXP)
  }
  \begin{syntax}
    "\xetex_if_engine:TF" <true code> <false code>
  \end{syntax}
  The underlining and italic of \texttt{TF} indicates that 
  \cs{xetex_if_engine:T}, \cs{xetex_if_engine:F} and 
  \cs{xetex_if_engine:TF} are all available. Usually, the illustration 
  will use the \texttt{TF} variant, and so both <true code>
  and <false code> will be shown. The two variant forms \texttt{T} and
  \texttt{F} take only <true code> and <false code>, respectively.
  Here, the star also shows that this function is expandable.
  With some minor exceptions, \emph{all} conditional functions in the 
  \pkg{expl3} modules should be defined in this way.
\end{function}

Variables, constants and so on are described in a similar manner:
\begin{variable}{
    \l_tmpa_tl
  }
  A short piece of text will describe the variable: there is no 
  syntax illustration in this case.
\end{variable}

In some cases, the function is similar to one in \LaTeXe\ or plain \TeX.
In these cases, the text will include an extra `\textbf{\TeX{}hackers 
note}' section:
\begin{function}{
    \token_to_str:N / (EXP)
  }
  \begin{syntax}
    "\token_to_str:N" <token> 
  \end{syntax}
  The normal description text.
  \begin{texnote}
    Detail for the experienced \TeX\ or \LaTeXe\ programmer. In this 
    case, it would point out that this function is the \TeX\ primitive
    \cs{string}.
  \end{texnote}
\end{function}

\DisableImplementation

\DocInput{l3names.dtx}

\DocInput{l3basics.dtx}
\DocInput{l3expan.dtx}
\DocInput{l3prg.dtx}
\DocInput{l3quark.dtx}
\DocInput{l3token.dtx}

\DocInput{l3int.dtx}
\DocInput{l3num.dtx}
\DocInput{l3intexpr.dtx}
\DocInput{l3skip.dtx}

\DocInput{l3tl.dtx}
\DocInput{l3toks.dtx}
\DocInput{l3seq.dtx}
\DocInput{l3clist.dtx}
\DocInput{l3prop.dtx}

\DocInput{l3io.dtx}
\DocInput{l3msg.dtx}
\DocInput{l3box.dtx}
\DocInput{l3xref.dtx}
\DocInput{l3keyval.dtx}
\DocInput{l3keys.dtx}
\DocInput{l3calc.dtx}
\DocInput{l3file.dtx}

% \DocInput{l3precom.dtx}
% \DocInput{l3alloc.dtx}
% \DocInput{l3chk.dtx}

\part{Implementation}
\def\maketitle{}
\EnableImplementation
\DisableDocumentation
\DocInputAgain

%% \DocInput{l3vers.dtx}   % Current version date

\clearpage
\pagestyle{headings}

% Make TeX shut up.
\hbadness=10000
\newcount\hbadness
\hfuzz=\maxdimen

\PrintChanges
\clearpage

\begingroup
\def\endash{--}
\catcode`\-\active
\def-{\futurelet\temp\indexdash}
\def\indexdash{\ifx\temp-\endash\fi}

\DelayPrintIndex
\endgroup

\end{document}