\documentclass {ltugproc} % {ltxguide} \usepackage{shortvrb} \MakeShortVerb{\|} \setlength\hfuzz{10pt} % A couple of \provide.. so document runs with % both ltugproc and ltxguide classes % \providecommand\m[1]{$\langle$\textit{#1}$\rangle$} \providecommand\netaddress{\date} \providecommand\acro[1]{\textsc{\MakeLowercase{#1}}} \begin{document} \title{The \LaTeX3 Programming Language\\ A syntax proposal for \TeX\ macro programming} \author{The \LaTeX3 Project} \netaddress{latex-l@urz.uni-heidelberg.de} \begin{abstract} This paper proposes a new set of programming conventions suitable for implementing large scale \TeX\ programming projects such as \LaTeX. (This syntax is not suitable for either document markup, or as a style specification language.) The main features include a systematic naming scheme for all commands, and a systematic mechanism for controlling the expansion of arguments to functions. The syntax is under consideration as a basis for programming within the \LaTeX3 project. This paper is based on a talk given by David Carlisle but describes the work of several people, principally: Frank Mittelbach, Denys Duchier, Rainer Sch\"opf, Chris Rowley, Michael Downes, Johannes Braams, David Carlisle and Alan Jeffrey. \end{abstract} \maketitle \section{Introduction} This paper describes a \TeX\ based language which is intended to provide a more consistent and rational programming environment for the construction of large scale \TeX\ macro projects such as \LaTeX. Variants of this language have been in use within the \LaTeX3 project since around 1990 but the syntax specification to be outlined here should \emph{not} be considered final. This is an experimental language, and the syntax and command names may (and probably will) change as more experience is gained with using the language in practice. \section{Programming Interface levels for \LaTeX} One may identify several distinct languages that one might want to see in a \TeX\ based system. This paper will \emph{only} be concerned with the last of these three. \begin{description} \item[Document Markup] This language consists of the commands that are to be embedded in the document instance. It is generally accepted that such a language should be essentially \emph{declarative}. One might consider a traditional \TeX\ based markup such as the \LaTeX2 markup as described in \cite{A-W:LLa94}, or alternatively one might consider an \acro{SGML} based markup. One problem with more traditional \TeX\ coding conventions is that the command names and syntax of the \TeX\ primitives are designed to have a `natural' syntax when used directly by the author as document markup. In fact one almost never uses the primitives in this way, rather they are just used to define higher level commands. \item[Designer's Interface] In order to easily translate a (human) designer's design specification into a program that accepts the document instance one would ideally like to have a declarative language that allows the relationships and spacing rules of the various document elements to be easily expressed. As this language is not embedded within the document text, it may take a rather different form to the markup language described above. For \acro{SGML} based systems one may consider the \acro{DSSSL} language as playing this role. For \LaTeX2, then this level was essentially missing in \LaTeX2.09. \LaTeXe\ made some improvements in this area but it is still the case that implementing a design specification in \LaTeX\ requires more `low level' coding than would be desired. \item[Programmer's Interface] This language, the subject of this paper, is the implementation language in which the basic typesetting functionality is implemented, building upon the primitives supplied by \TeX\ (or a successor program). It may also be used to implement the previous two languages `within' \TeX, as in the current \LaTeX\ system. \end{description} \section{Programming language: Main Features} The language outlined in this paper aims to provide a suitable base for coding large (and small) scale projects in \TeX. Its main distinguishing features are the following. \begin{itemize} \item Consistent naming scheme for all functions, including \TeX\ primitives. \item Standard mechanisms for controlling argument expansion. \item Provision of sufficiently rich set of core functions for handling programming constructs such as: sequences, sets, stacks, property lists, etc. \item White space ignored. \end{itemize} \section{Naming Scheme} The name space is divided between \emph{Functions} and \emph{Parameters}. Functions (normally) take arguments and are executed, Parameters are usually passed as arguments to functions. They are not directly executed, but accessed though accessor functions. Functions and parameters associated with a similar function (for example accessing counters, or manipulating lists, etc.) are arranged into \emph{modules}. Before giving the details of the form of the command names, we give a few examples. |\l_tmpa_box| is a local parameter (hence the |l_| prefix) corresponding to a box register. |\g_tmpa_box| is a global parameter (hence the |g_| prefix) corresponding to a box register. |\c_one| is the constant (|c_|) parameter with value one. |\cnt_add:Nn| is the function which adds the value specified by its second argument to the count register specified by its first argument. The different natures of the two arguments are indicated by the |:Nn| suffix. The first argument must be a single token specifying the name of the count parameter. Such single token arguments are denoted |N|. The second argument is a normal \TeX\ `non-delimited argument' which may either be a single token, or a brace delimited token list containing an expression for the value to be added. The |n| denotes such `normal' argument forms. |\cnt_add:cn| is similar to the above, but in this case the the counter is specified in the first argument by a list of tokens that expands to the \emph{name} of the count parameter. These examples should give the basic flavour of the scheme. Parameters are classified into local, global or constant (there are further more technical cases in addition to these three) and this access type is shown by |\l_|, |\g_| or |\c_|. Functions are arranged by \emph{module} (The |cnt| module in these cases) with a descriptive name followed by an indication of the type of argument to be passed. In more detail the specification of the names is as follows. Functions have the following general syntax: \begin{quote} |\|\m{module}|_|\m{description}|:|\m{arg-spec} \end{quote} The programmer can chose an arbitrary \m{module} name (consisting of letters only) a group of functions with related functionality are then all given names prefixed by this module name. The \m{description} is a description of the functionality of the function, and should consist of letters, and possibly |_| characters. \m{arg-spec} describes the type of arguments as will be described below. The syntax of parameters is as follows: \begin{quote} |\|\m{access}|_|\m{module}|_|\m{description}|_|\m{type} \end{quote} \m{access} describes how the parameter can be accessed. The principal access types are \emph{constant}, \emph{local} or \emph{global}. As described below, some special access types relate to \TeX\ primitive parameters. The meanings of \m{module} and \m{description} in the parameter syntax is the same as that for functions. Finally \m{type} should denote the type of parameter, such as |cnt| for count registers, etc. Typical \m{module} names in the kernel include |cnt| for integer count related functions, |seq| for functions relating to sequences, |box|, etc. Normally additional packages adding new functionality would add new modules as needed. The \m{description} is an arbitrary name for the function or parameter, consisting of letters, or the |_| character. Function names always end with an \m{arg-spec} after a final colon. This gives an indication of the types of argument that a function takes, and provides a convenient method of naming similar functions that just differ by their argument forms, as will be explained below. The \m{arg-spec} consists of a (possibly empty) list of characters each denoting one argument that the function takes. It is important to note that `argument' here refers to the conceptual argument of the function. The top level \TeX\ macro that has this name typically has no arguments. This is similar to the existing \LaTeX\ convention where one says that |\section| has an optional argument and a mandatory argument, whereas the \TeX\ macro |\section| actually takes no parameters at the \TeX\ level, it merely calls some standard \LaTeX\ internal functions which look ahead for star forms and optional arguments. The list of possible argument specifiers includes: \begin{description} \item[n] Unexpanded token (or token-list if in braces). In other words this is a standard \TeX\ undelimited macro argument. \item[o] One time expanded token or token-list. In the case of a token list then only the first token in the list is expanded. \item[x] Fully expanded token or token-list. Typically this means that the argument is expanded in the style of |\edef| (|\def:Npx|) before being passed to the function. \item[c] A character string used (after expansion) as a command name. The argument (a token or braced token list) should expand to a sequence of characters which is then used to construct a command name (via |\csname|, |\cs:w|). This command token is passed as the argument to the function. \item[N] A single token. (Unlike |n|, this argument must not be surrounded by braces). A typical example of a command taking an N argument is |\def|, in which the command being defined must be unbraced. \item[O] Single unbraced token that is expanded once and passed (as a braced token list) to the function. \item[X] Single unbraced token that is fully expanded and passed (as a braced token list) to the function. \item[C] A character string used as for |c| arguments but the resulting command token is then expanded (as for |O|) and the result passed as a braced token list to the function. \item[p] A primitive \TeX\ parameter specification. This can be something simple like |#1#2#3| but may be arbitrary delimited argument syntax, such as |#1,#2\q_stop#3|. \item[T, F] These are special cases of |n| arguments, used as the true and false cases in conditional tests. \item[D] `Do not use'. This special case is used for \TeX\ primitives that are only used while bootstrapping the \LaTeX\ kernel. If the \TeX\ primitive needs to be used in other contexts it will be given an alternative name with a more appropriate argument specification. \item[w] `weird' syntax. Used for arguments that take non standard forms, usually delimited arguments that are needed internally to implement certain modules, and also the boolean tests of many of the primitive |\if|\ldots\ tests. \end{description} For parameters, the \m{type} should be from the list of available data types (which include the primitive \TeX\ registers, but also data types built within the system). Possible values for \m{type} include: \begin{description} \item[cnt] Integer valued counter. \item[toks] Token register. \item[box] Box register. \item[fcnt] `Fake' count register. A data type supplied by the kernel to avoid problems with the limited number of available count registers in (standard) \TeX. \end{description} The \m{access} codes that are used in parameter names include \begin{description} \item[c] Constants. \item[l] Parameters that should only be set locally. \item[g] Parameters that should only be set globally. \end{description} \section{Checking Parameter assignments} One of the advantages of having a consistent scheme is that the system can provide more extensive error checking and debugging facilities. For example a function that makes a global assignment can check that it is not passed a local parameter as argument by checking that the name of the command to be assigned starts with |\g_|. Such checking is probably too slow for production runs, but the kernel has hooks built in to allow a format to be made in which all functions perform this kind of check. A typical section of code might look like \begin{verbatim} %<*check> \def_new:Npn \toks_gset:Nn #1 { \chk_global:N #1 \pref_global:D #1 } % %<*!check> \let_new:NN \toks_gset:Nn \pref_global:D % \end{verbatim} The function |\toks_gset:Nn| takes a single token (|N|) specifying a token register, and globally sets it to the value passed in the second argument. So typical use would be \begin{verbatim} \toks_gset \g_xxx_toks {some value} \end{verbatim} In the normal definition, |\toks_gset| can be defined just to be |\let| to |\global|, as the primitive token register does not require any explicit assignment function. This is the |%<*!check>| code above. However the alternative definition first checks that the argument passed as |#1| is a global parameter and raises an error if it is not. It does this by taking apart the command name passed as |#1| and checking that it starts |\g_|. \section{Consistent use of accessor functions} The primitive \TeX\ syntax for register assignments has a very minimal syntax, and apart from box functions there are no explicit functions for assignment or use of the registers. This makes it very difficult to implement alternative data types with a syntax that is at all similar to the syntax for the primitives, and also encourages a coding style that is very error prone. As noted in the example given above, The \LaTeX\ data types are provided with explicit functions for setting and using the parameters even when these have essentially empty definitions. This allows for better error checking as described above, and also allows the construction of alternative data types with a similar interface. For example the `fake counter' data type mentioned previously works at the user level just like the data type based on primitive count registers, internally it does not use count registers though. Typical functions in the |fcnt| module include: \verb|\fcnt_new:N \l_tempa_fcnt|\\ Declare the local parameter |\l_tempa_fcnt| as a fake counter. \verb|\fcnt_add:Nn \l_tempa_fcnt \c_thirty_two|\\ Increment the counter by 32. \section{Expansion Control} Anyone who programs in \TeX\ is used to the problem of arranging that arguments to functions are suitably expanded before the function is called. A couple of real examples copied from \texttt{latex.ltx}: \begin{verbatim} \global \expandafter\expandafter\expandafter \let \expandafter \reserved@a \csname\curr@fontshape\endcsname \expandafter \in@ \csname sym#3\expandafter\endcsname \expandafter{\group@list}% \end{verbatim} The first piece of code is a global |\let|. The token to be defined is obtained by expanding \verb|\reserved@a| one level. The command that it is to be let too is obtained by fully expanding \verb|\curr@fontshape| and then using the tokens produced by that expansion to construct a command name. This results in the mess of interwoven \verb|\expandafter| and \verb|\csname| beloved of all \TeX\ programmers, and code that is essentially unreadable. A similar construction using the conventions outlined here would be \begin{verbatim} \glet:Oc \reserved_a: \l_current_font_shape_tlp \end{verbatim} The command \verb|\glet:Oc| is a global |\let| that expands its argument once, and generates a command name out of its second argument, before making the definition. This produces coe that is far more readable. Similarly the second piece of code above produces a token list by expanding \verb|\group@list| once, and then creates a command name out of `\verb|sym#3|' (this is inside the definition of another fumction). The function |\in@| is called which tests if its first argument occurs in the token list of its second argument. Again it would be much clearer, if the above function \verb|\in@| was called (say) \verb|\test_if_in:nn| (a function taking two normal `\texttt{n}' arguments) and then a variant function was defined with the appropriate argument types and simply called as follows: \begin{verbatim} \test_if_in:co {sym#3} \group_list: \end{verbatim} Note that apart from the lack of \verb|\expandafter| the space after \verb|}| will be silently ignored. For many common functions the kernel will provide functions with a range of argument forms, and similarly it is expected that extension packages providing new functions will make then available in the more common forms. However There will be occasions where it is necessary to construct such a variant form. A consistent mechanism is provided by the kernel to produce functions with any argument type, starting from a function that takes `normal' \TeX\ delimited arguments. Suppose you have a function \verb|\cmd:nnn| that takes two arguments, and you need to construct \verb|\cmd:cnx| a variant form in which the first argument is passed as a \emph{name} of a command, and the third argument must be fully expanded before being passed to \verb|\cmd:nnn|. One simply defines \verb|\cmd:cx| as follows: \begin{verbatim} \def:Nn \cmd:cnx {\exp_args:Ncnx \cmd:nnn} \end{verbatim} The function \verb|\exp_args:Ncnx| takes as its first (\texttt{N}) argument the `base' function, and then grabs the next three arguments from the token stream, acts on the first with \verb|\csname|, and the last with \verb|\edef| and then constructs a call to the base function with suitably transformed arguments. So \begin{verbatim} \cmd:cnx {abc}{pq}{\rst\xyz} \end{verbatim} is equivalent, but emminently more readable, to \begin{verbatim} \edef\temp{\rst\xyz} \expandafter\cmd:nnn \csname abc\expandafter\endcsname \expandafter{% \expandafter p\expandafter q% \expandafter}% \expandafter{\temp} \end{verbatim} A large range of argument processing functions are provided in addition to \verb|\exp_args:Ncnx|. If you need a particular argument combination for which a function is not provided, one may be constructed in a simple way. For example you need to construct \verb|\exp_args:Nxcxcxc| a function that fully expands arguments 1,~3 and~5 of a given function, and produces commands to pass as arguments 2,~4 and~6 using \verb|\csname|. The definition is simply \begin{verbatim} \def:Npn \exp_args:Nxcxcxc {\::x\::c\::x\::c\::x\::c\:::} \end{verbatim} Similar functions, |::o| etc exist for all the other argument types, and they may be strung together in any order, terminated by \verb|\:::| to create a function which processes arguments in the desired way. As hopefully demonstrated, the use of variant forms greatly improves the readability of the code, and experience shows that the longer command names which result from the new syntax do not realy make the process of \emph{writing} the code any harder. \section{The Current Experimental Distribution} The initial implementations of a \TeX\ format using this kind of syntax were made with an unreleased (and non functional) format (which pre-dates \LaTeXe!) The current distribution consists of a subset of the functionality of that format, converted to run as packages on top of \LaTeXe. The intention is to allow experienced \TeX\ programmers to experiment on the system and to comment on the interface. This means that \textbf{\itshape the interface will change}. No part of this system, including the names of any commands should be relied upon as being available in a later release. Please do \emph{experiment} with these packages, but do not use them for documents that you expect to keep unchanged over a long period. In view of the proposed experimental use for this distribution, we currently have only converted a few modules for use with \LaTeXe. These set up the basic conventions, and then implement a few basic programming constructs such as lists and sequences. These are really to give a flavour of the code, and to indicate that the intention is that the kernel provide a sufficiently rich set of programming constructs so that packages may use them and thus more efficiently share code, unlike the situation in the current \LaTeX\ where every large package implements its own version of lists, stacks etc. The current packages are: \begin{description} \item[l3names] Sets up the basic naming scheme, including naming the \TeX\ primitives. If used with the option \texttt{[removeoldnames]} then the old primitive names such as \verb|\box| are \emph{undefined} and thus made available for user definitions. Use of this option might possibly break existing \TeX\ code! \item [l3basics] Some basic definitions that are used by the other packages. \item[l3chk] Functions to check (and make) definitions (comparable to the existing \verb|\newcommand| or \verb|\renewcommand|). \item[l3tlp] Token List Pointers. A basic \LaTeX3 data type for storing token lists. (These are essentially macros with no arguments.) \item[l3expan] The argument expansion module discussed in the previous section. \item[l3quarq] A `quark' is a command that is defined to expand to itself. So it may not be directly used (it would generate an infinite loop) but has many uses as special markers within \LaTeX\ code. \item[l3seq] A module implementing the basic list and stack data types. \item[l3prop] Property lists are the data type for handling key/value assignments. \end{description} The distribution also contains the \TeX\ source for this document, a docstrip install file and two small test files. \begin{thebibliography}{1} \bibitem{A-W:LLa94} Leslie Lamport. \newblock {\em {\LaTeX:} A Document Preparation System}. \newblock Addison-Wesley, Reading, Massachusetts, second edition, 1994. \end{thebibliography} \end{document}