diff options
Diffstat (limited to 'Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex')
-rw-r--r-- | Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex | 265 |
1 files changed, 241 insertions, 24 deletions
diff --git a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex index dfacfd2e610..46da664c6f9 100644 --- a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex +++ b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex @@ -25,7 +25,12 @@ \newcommand\Space{\textit{Space}} \newcommand\CGMS{\textit{C-G-M-S}} \newcommand\CGMnS{\textit{C}, \textit{G}, \textit{M}, and \textit{S}} -\newcommand\macname[1]{\texttt{\char92#1}} +\begingroup +\catcode`|=\active +\gdef|{\textbackslash} +\gdef\xmacaux#1{\ttfamily\char92#1\endgroup} +\endgroup +\gdef\macname{\begingroup\catcode`|=\active \xmacaux} \newcommand\thevbox{\medskip\theverbbox\par\medskip\noindent} \newenvironment{specfig} {\par\bigskip\vbox\bgroup\small\sffamily\centering} @@ -153,7 +158,7 @@ As tokens/groups are read from the input stream, they are with that category of token/group. The \TokCycle{} categories by which the input stream is dissected include \Character{}, - \Group{}, \Macro{}, and \Space{}. + \Group{}, \Macro{}, and \Space{} (alphabetized for easy recall). Catcode-0 tokens are directed to the \Macro{} directive for processing. Catcode-10 tokens are directed to the \Space{} directive. @@ -170,6 +175,8 @@ The handling options of implicit cat-1 and 2 tokens are described later in Valid tokens that are neither catcode 0, 1, 2, nor 10, except where noted, are directed to the \Character{} directive for processing. +\subsection{Provided (built-in) \TokCycle{} macros and environments} + The syntax of the non-\texttt{xpress} macros/environments is\medskip \begin{verbbox}[\vbdelim] @@ -206,6 +213,8 @@ or, alternately, for the \texttt{xpress}-pseudo-environment, \thevbox \begin{sloppypar} +\subsection{Create your own \TokCycle{} environments} + In addition to the above macros/environments, the means is provided to define new \TokCycle{} environments: @@ -311,7 +320,7 @@ Let it suffice for now to say that the default directive settings \section{Commands in the \TokCycle{} directives}\label{s:coms} -The command-line token cycling tools provided in the package are +The document-level token cycling tools provided in the package are listed in section~\ref{s:intro}. For each of those commands and/or pseudo-environments, the user must (explicitly or implicitly) detail a set of directives to @@ -326,7 +335,12 @@ There are, however, several macros provided by the package to assist The recommended way to apply this package is to collect the \TokCycle-transformed results of the input stream in a token register provided by the package, named \macname{cytoks}. -Its contents can then be typeset via \verb|\the\cytoks|. +Its contents can then be typeset via \verb|\the\cytoks|.\footnote{% +If a token-cycle input stream contains no macros, or is to be detokenized, + or if the input-stream tokens are not to be typeset, it may be possible + (though not required) to bypass \macname{cytoks} and typeset the output + directly. +} The macro for appending things to \macname{cytoks}, to be used in the package directives, is \macname{addcytoks}. @@ -365,7 +379,7 @@ The specification of all \TokCycle{} processing directives is structured in such a way that ``\argone{}'' may be directly employed to indicate the current token (or group) under consideration in the input stream. -\subsubsection{Transforming the Input Stream} +\subsubsection{Transforming the input stream} Within the \TokCycle{} \CGMS{} directives, the command \verb|\addcytoks{#1}| is used to echo the token being processed to @@ -449,8 +463,8 @@ However, there are times where you might wish to perform a macro task (for example, one that might add additional tokens to \macname{cytoks}) immediately \textit{outside} of the group, before or after. If you were to add the task macro before or after the \macname{processtoks} - invocation, it will still be performed, by default, \textit{inside} - the \macname{cytoks} group. + invocation (anywhere in the \Group{} directive), it will still be + performed, by default, \textit{inside} the \macname{cytoks} group. The macro \macname{groupedcytoks} allows one to manually specify the grouping duties of the \Group{} directive. @@ -518,6 +532,201 @@ Finally, the special case of $-$1 indicates that the implicit-group Such a special treatment has limited application---for example, when the intent is to detokenize these tokens. +\subsection{Looking ahead at the input stream} + +In the normal mode of \TokCycle{} operation, each token of the input stream + is successively digested and sent to one of four user-defined directives + for processing. +Such an approach works well if there is no interdependency of adjacent tokens + in the input stream. +When such an interdepency exists (for example, the argument associated with + an invoked macro), one might typically use flags that are set or cleared + in one directive that can then be status-checked when the subsequent token + is processed. +While such an approach is wholly valid, it can create a complex web of + flag setting/checking that can span across multiple directives. + +With the release of v1.4 of the package, an alternative approach to handle + such dependencies has been developed: look-ahead features, so that the + occurrence of a particular token or condition can provoke an immediate + examination of the input stream to handle a possible dependency, without + waiting for the next iteration of the token cycle. + +It should be noted that the use of look-ahead is perhaps more + \textit{dangerous} than the traditional use of flags---for if it is not + performed with care, tampering with the input stream can destroy its + integrity. +The commands developed for these situations will now be described. +They are designed to be employed as part of \TokCycle's \Character, \Macro, + and \Space{} directives for checking and/or handling successively linked + tokens in the input stream. +They cannot be used inside the \Group{} directive. + +\subsubsection{\macname{tcpeek}} + +When issued witin a directive as, for example \macname{tcpeek|z}, the next + token from the input stream is \macname{futurelet} into \macname{z}. +This has several implications. +The input stream remains undisturbed, so that the peeked-at token will + still be the token to be digested in the next iteration of the token cycle. +Because it has been \macname{let}, \macname{z} does not \textit{contain} the token + from the input stream in the manner of a \macname{def}, but rather it is a + separate token \textit{possessing the same properties} as the input-stream token. +As such, it is ideal to be used for \macname{ifx} comparisons. +For example, \macname{ifx|bgroup|z} will detect whether a cat-1 group-opening + token is at the head of the input stream, e.g., an opening brace, `\{'; +the comparison \macname{ifx|tcsptoken|z} will detect whether a cat-10 space + is at the head of the input stream. +Such comparisons can be useful in directing the logic of the \TokCycle{} + directive. + +\subsubsection{\macname{tcpop}} + +When issued as, for example \macname{tcpop|z}, the next token(s) from the + input stream is \textit{immediately} digested as an argument and the + tokens are placed into the macro \macname{z}. +This token (or tokens) will no longer be digested as part of the next + iteration within the token cycle, unless the tokens are subsequently + replaced at the head of the input stream (see \macname{tcpush} below). + +Beware that when \TeX{} absorbs an argument, leading white space is lost. +Further, if the argument was enclosed in cat-1,2 braces, the braces are + stripped and the whole group is absorbed as the argument. +In some cases, if the argument needs manipulation, this brace-stripping + behavior may be desired. +However, if the retention of the leading white space and the braces are + desired, one can use manual techniques (possibly involving macros + such as \macname{tcpeek}, \macname{tcpopwhitespace} and + \macname{tcpushgroup}) or one can instead use \macname{tcpopliteral} + (see below) in lieu of \macname{tcpop}. + +If \macname{tcpop} is used nonetheless, to aid in such matters when + cat-10 space is at the head of the input stream, the flag + \macname{spacepoppedtrue} will be set when a \macname{tcpop} is issued. +However, even \macname{ifspacepopped} will be unable to differentiate explicit + ``white'' space that is lost during argument absorption versus implicit + space (e.g., \macname{@sptoken} or active-implicit space) that is not + discarded. +Another option for dealing with leading white space is + \macname{tcpopwhitespace} (see below). + + +\subsubsection{\macname{tcpopliteral}} + +This macro is an alternative to \macname{tcpop}, if the retention of + leading white space and possibly braces are required when popping + an argument from the input stream. +This can be particularly useful if the popped tokens must later be + placed back into the input stream in their original state. +The syntax, \macname{tcpopliteral|z}, pops an argument into \macname{z}, + while retaining possible leading whitespace and brace groups. + +\subsubsection{\macname{tcpopappto}\ and \macname{tcpopliteralappto}} + +When issued as, for example \macname{tcpopappto|z}, the next token(s) from the + input stream is absorbed as an argument and \textit{appended} to the + replacement text of \macname{z}. +The same group/space provisos affecting \macname{tcpop} apply here as well. +The control sequence to be appended, here \macname{z}, may not be undefined + when \macname{tcpopappto} is invoked. +This macro is just a convenient joining of two macros: \macname{tcpop} + and \macname{tcappto\#1from\#2} (see below). + +The same applies for the macro \macname{tcpopliteralappto}, which conveniently + joins \macname{tcpopliteral} and \macname{tcappto\#1from\#2}. + +\subsubsection{\macname{tcpopuntil}} + +This command pops one or more tokens from the input stream in the manner of + a delimited argument. +Thus, when issued as, for example \macname{tcpopuntil~0|z}, tokens from + the input stream are absorbed into \macname{z} until an \texttt{0} token is + reached. +Unlike delimited-argument absorption, however, the delimiter (\texttt{0} in + this example) is also added to the specified control sequence, in this case + \macname{z}. +This construct is very useful for obtaining optional-argument tokens from the + input stream. +Consider the following code, with an input stream of \texttt{[1ex]\{2ex\}}: +\begin{verbatim} + \tcpeek\z + \ifx[\z\tcpopuntil ]\q\else\def\q{}\fi +\end{verbatim} +Since an optional argument is next in the input stream, \macname{q} will + obtain the replacement text \texttt{[1ex]}, and the input stream will now + begin with \texttt{\{2ex\}}. +If an optional argument had not been next in the input stream, then + \macname{q} would be empty and the input stream would still begin + with \texttt{\{2ex\}}. + +\subsubsection{\macname{tcpopwhitespace}} + +When issued as, for example \macname{tcpopwhitespace|z}, leading white space + from the input stream will be absorbed and set in \macname{z}, in the + manner of a \macname{def}. +Unlike the flag \macname{ifspacepopped}, leading implicit space will not be + indicated by this macro---only leading explicit white space. +If no leading white space is present, \macname{z} will be empty following + its invocation and no tokens will have been removed from the input stream. + +\subsubsection{\macname{tcpush}} + +To this point, several commands have been described for reading and/or + extracting tokens from the input stream. +There are also a corresponding commands for placing material at the head + of the input stream, should that need arise. +With an invocation of \macname{tcpush|z}, the replacement text of \macname{z} + will be pushed onto the input stream. +Multiple pushes will be handled in a last-in-first-out fashion. + +Beware that, because argument absorption in \TeX{} will strip the braces of + an absorbed group, care must be taken. If the input stream leads with a + grouped quantity, such as \texttt{\{abc\}}, then \macname{tcpop|z|tcpush|z} + will end up with \texttt{abc} in the input stream, with the braces missing. +\textit{The commands \macname{tcpop} and \macname{tcpush} are not strictly + inverse operations} (see \macname{tcpopliteral} as an alternative). +For this reason, the command \macname{tcpushgroup} is also provided + (see below). + +The command \macname{tcpush} supports an optional argument that functions + in the same manner as that of \macname{addcytoks}, to provide additional + levels of expansion beyond the mere replacement text. +Thus, \macname{tcpush[2]|z} will take the replacement text of \macname{z}, + expand it twice, and push the result onto the input stream. +In a similar way, \macname{tcpush[x]|z} will fully expand the replacement + text of \macname{z} and then place those tokens onto the input stream. +Whereas \macname{addytoks} places tokens into the output stream (the + token list \macname{cytoks}), \macname{tcpush} places tokens at the + head of the input stream. + +\subsubsection{\macname{tcpushgroup}} + +The command \macname{tcpushgroup} functions in a manner similar to + \macname{tcpush}, except that the replacement text of the argument + is placed onto the input stream within a braced (cat-1,2) group. +So if the replacement text of \macname{z} is \texttt{abc}, then the + invocation \macname{tcpushgroup|z} will place \texttt{\{abc\}} onto + the input stream. +As with \macname{tcpush}, and with the identical syntax, the command + \macname{tcpushgroup} supports an optional argument that directs + addition levels of expansion beyond the mere replacement text. + +\subsubsection{\macname{tcappto\#1from\#2}} + +This macro does not touch the input stream, per se. +However, as tokens are popped from the input stream and placed into macros, + this command conveniently allows for their aggregation +Thus, if \macname{q} contains \texttt{abc} and the newly popped \macname{z} + contains \texttt{d}, then \macname{tcappto |q from |z} will append the + contents of \macname{z} to \macname{q}, so that, upon conclusion, + \macname{q} will contain the tokens \texttt{abcd}. +Unexpandable tokens can also be appended directly using this macro, using + the syntax of \macname{tcappto |q from\{123\}}. +Expandable tokens can also be added directly with the use of this macro, + by using a leading \macname{noexpand} or \macname{empty}, as in + \macname{tcappto |q from\{|noexpand|today\}}. + + \subsection{Truncating the input stream} The basic process of \TokCycle{} is one in which an input stream (or argument) @@ -554,7 +763,7 @@ It may be examined within the \CGMS{} directives in order to or for the remainder of the total \TokCycle{} input stream. -\subsubsection{\macname{truncategroup} and \macname{truncategroupiftokis}} +\subsubsection{\macname{truncategroup}\ and \macname{truncategroupiftokis}} \label{s:truncgrp} As to truncating the input stream for the remainder of the \TokCycle{} @@ -596,7 +805,7 @@ Thus, an example to echo macro tokens to the output stream, unless Obviously, checks for particular character tokens would be placed in the \Character{} directive, rather than the \Macro{} directive. -\subsubsection{\macname{truncatecycle} and \macname{truncatecycleiftokis}} +\subsubsection{\macname{truncatecycle}\ and \macname{truncatecycleiftokis}} There are two commands that are in every way analogous to the group-\hspace{0pt}terminating macros of section~\ref{s:truncgrp}. @@ -746,7 +955,7 @@ i)~implicit grouping tokens (e.g., \macname{bgroup} and \macname{egroup}) ii)~implicit space tokens (e.g., \macname{@sptoken}) will be processed by the \Space{} directive. -\subsubsection{Active-Implicit Tokens, Including Spaces} +\subsubsection{Active-implicit tokens, including spaces\label{s:acimsp}} One may occasionally run across a token that is both active and implicit. For example, in the following code, \texttt{Q} is made both active and implicit: @@ -761,18 +970,20 @@ In general, both \macname{ifactivetok} and \macname{ifimplicittok} This is true even in the case of active-implicit catcode-10 spaces, which are always processed through the \Space{} directive. -However, because of the process \TokCycle{} uses in digesting space - tokens, the actual token passed to the \Space{} directive (so-called - \verb|#1|), when an active-implicit space is encountered in the input - stream, is a generic implicit space token named \macname{tc@sptoken}. -However, the catcode-12 version of the active character that - produced it will be, for that moment, retained in a definition +As of \TokCycle{} v1.4, the actual active-implicit space, if + encountered in the input stream, will be passed as \verb|#1| to + the \Space{} directive, \textit{as long as the character code of the + token is still \macname{active}.} +If the character code of that token is no longer active, a generic + implicit space token named \macname{tcsptoken} is instead passed + to the the \Space{} directive as \verb|#1|. +In either case, the catcode-12 version of the active character that + was digested will be, for that moment, retained in a definition named \macname{theactivespace}. This can be useful if detokenization is required of the spaces. Such an example is described in the \texttt{tokcycle-examples} adjunct document. - \subsubsection{Parameter (cat-6) tokens (e.g., \texttt{\#}): \macname{ifcatSIX}} \label{s:cat6} @@ -846,7 +1057,7 @@ Note that unlike those macros requiring three arguments, the lone catcodes of the two input toks \textbf{without expansion} (\texttt{nx} denotes \macname{noexpand}). -\subsection{\TokCycle{} Completion: \macname{aftertokcycle} +\subsection{\TokCycle{} completion: \macname{aftertokcycle}\ and \macname{tcendgroup}} The \TokCycle{} \textit{macros}, upon completion, do nothing. @@ -957,12 +1168,18 @@ In any event, all cat-6 tokens are trapped and flag as \texttt{true} \item A similar warning is to be made for active-implicit spaces as was made for implicit-cat-6 tokens. -In the case of active-implicit spaces, the name of the active character is - momentarily stored in the \macname{theactivespace} macro. -If one wishes to output the active \textit{name} to the \macname{cytoks} token list, - one must make sure that \macname{theactivespace} is expanded once before +In the case of active-implicit spaces, the name (a cat-12 string) of the + active character is momentarily stored in the \macname{theactivespace} macro. +If one wishes to output the active \textit{name} to the \macname{cytoks} token + list, one must make sure that \macname{theactivespace} is expanded once before being added to the token list. +One difference here, however, is that, unlike a cat-6 token, an + active-implicit space (and not a string of it) is actually passed + to the \Space{} directive as \texttt{\#1}. +In all but one special case, the active-implicit \texttt{\#1} will match that + which occurred in the input stream (see section~\ref{s:acimsp}). + \end{itemize} \section*{Acknowledgments} @@ -980,13 +1197,13 @@ The \texttt{tex.stackexchange} site provides a wonderful opportunity to interact with the leading developers and practitioners of \TeX{} and \LaTeX{}. -\vspace{1in} +\vspace{.6in} \section*{Source Code} %\vspace{-0.1in} \Large\texttt{tokcycle.sty} \verbfilenobox[\footnotesize]{tokcycle.sty}% -%\vspace{-0.1in} +\vspace{1.5in} \noindent\texttt{tokcycle.tex} \verbfilenobox[\footnotesize]{tokcycle.tex} |