l3 (17jul20)

git-svn-id: svn://tug.org/texlive/trunk@55860 c570f23f-e606-0410-a88d-b1316a301751

1 files changed, 93 insertions, 2 deletions

diff --git a/Master/texmf-dist/doc/latex/l3kernel/l3term-glossary.tex b/Master/texmf-dist/doc/latex/l3kernel/l3term-glossary.tex
index 99464b67c97..90e2c3c768e 100644
--- a/Master/texmf-dist/doc/latex/l3kernel/l3term-glossary.tex
+++ b/Master/texmf-dist/doc/latex/l3kernel/l3term-glossary.tex
@@ -32,7 +32,7 @@ The released version of this bundle is available from CTAN.
         {latex-team@latex-project.org}%
     }%
 }
-\date{Released 2020-06-18}
+\date{Released 2020-07-17}
 
 \newcommand{\TF}{\textit{(TF)}}
 
@@ -53,7 +53,98 @@ beginning of a line.
 
 \section{Structure of tokens}
 
-Copy there the section ``Description of all possible tokens'' from \texttt{l3token}.
+We refer to the documentation of \texttt{l3token} for a complete
+description of all \TeX{} tokens.  We distinguish the meaning of the
+token, which controls the expansion of the token and its effect on
+\TeX{}'s state, and its shape, which is used when comparing token lists
+such as for delimited arguments.  At any given time two tokens of the
+same shape automatically have the same meaning, but the converse does
+not hold, and the meaning associated with a given shape change when
+doing assignments.
+
+Apart from a few exceptions, a token has one of the following shapes.
+\begin{itemize}
+  \item A control sequence, characterized by the sequence of characters
+    that constitute its name: for instance, \cs{use:n} is a five-letter
+    control sequence.
+  \item An active character token, characterized by its character code
+    (between $0$ and $1114111$ for \LuaTeX{} and \XeTeX{} and less for
+    other engines) and category code~$13$.
+  \item A character token such as |A| or |#|, characterized by its
+    character code and category code (one of $1$, $2$, $3$, $4$, $6$,
+    $7$, $8$, $10$, $11$ or~$12$ whose meaning is described below).
+\end{itemize}
+
+The meaning of a (non-active) character token is fixed by its category
+code (and character code) and cannot be changed.  We call these tokens
+\emph{explicit} character tokens.  Category codes that a character token
+can have are listed below by giving a sample output of the \TeX{}
+primitive \tn{meaning}, together with their \LaTeX3 names and most
+common example:
+\begin{itemize}
+  \item[1] begin-group character (|group_begin|, often |{|),
+  \item[2] end-group character (|group_end|, often |}|),
+  \item[3] math shift character (|math_toggle|, often |$|), % $
+  \item[4] alignment tab character (|alignment|, often |&|),
+  \item[6] macro parameter character (|parameter|, often |#|),
+  \item[7] superscript character (|math_superscript|, often |^|),
+  \item[8] subscript character (|math_subscript|, often |_|),
+  \item[10] blank space (|space|, often character code~$32$),
+  \item[11] the letter (|letter|, such as |A|),
+  \item[12] the character (|other|, such as |0|).
+\end{itemize}
+Category code~$13$ (|active|) is discussed below.  Input characters can
+also have several other category codes which do not lead to character
+tokens for later processing: $0$~(|escape|), $5$~(|end_line|),
+$9$~(|ignore|), $14$~(|comment|), and $15$~(|invalid|).
+
+The meaning of a control sequence or active character can be identical
+to that of any character token listed above (with any character code),
+and we call such tokens \emph{implicit} character tokens.  The meaning
+is otherwise in the following list:
+\begin{itemize}
+  \item a macro, used in \LaTeX3 for most functions and some variables
+    (|tl|, |fp|, |seq|, \ldots{}),
+  \item a primitive such as \tn{def} or \tn{topmark}, used in \LaTeX3
+    for some functions,
+  \item a register such as \tn{count}|123|, used in \LaTeX3{} for the
+    implementation of some variables (|int|, |dim|, \ldots{}),
+  \item a constant integer such as \tn{char}|"56| or
+    \tn{mathchar}|"121|, used when defining a constant using
+    \cs{int_const:Nn},
+  \item a font selection command,
+  \item undefined.
+\end{itemize}
+Macros can be \tn{protected} or not, \tn{long} or not (the opposite of
+what \LaTeX3 calls |nopar|), and \tn{outer} or not (unused in \LaTeX3).
+Their \tn{meaning} takes the form
+\begin{quote}
+  \meta{prefix} |macro:|\meta{argument}|->|\meta{replacement}
+\end{quote}
+where \meta{prefix} is among \tn{protected}\tn{long}\tn{outer},
+\meta{argument} describes parameters that the macro expects, such as
+|#1#2#3|, and \meta{replacement} describes how the parameters are
+manipulated, such as~|\int_eval:n{#2+#1*#3}|.  This information can be
+accessed by \cs{cs_prefix_spec:N}, \cs{cs_argument_spec:N},
+\cs{cs_replacement_spec:N}.
+
+When a macro takes an undelimited argument, explicit space characters
+(with character code $32$ and category code $10$) are ignored.  If the
+following token is an explicit character token with category code $1$
+(begin-group) and an arbitrary character code, then \TeX{} scans ahead
+to obtain an equal number of explicit character tokens with category
+code $1$ (begin-group) and $2$ (end-group), and the resulting list of
+tokens (with outer braces removed) becomes the argument.  Otherwise, a
+single token is taken as the argument for the macro: we call such single
+tokens \enquote{N-type}, as they are suitable to be used as an argument
+for a function with the signature~\texttt{:N}.
+
+When a macro takes a delimited argument \TeX{} scans ahead until finding
+the delimiter (outside any pairs of begin-group/end-group explicit
+characters), and the resulting list of tokens (with outer braces
+removed) becomes the argument.  Note that explicit space characters at
+the start of the argument are \emph{not} ignored in this case (and they
+prevent brace-stripping).
 
 \section{Quantities and expressions}