summaryrefslogtreecommitdiff
path: root/macros/generic/tokcycle/tokcycle-doc.tex
diff options
context:
space:
mode:
Diffstat (limited to 'macros/generic/tokcycle/tokcycle-doc.tex')
-rw-r--r--macros/generic/tokcycle/tokcycle-doc.tex265
1 files changed, 241 insertions, 24 deletions
diff --git a/macros/generic/tokcycle/tokcycle-doc.tex b/macros/generic/tokcycle/tokcycle-doc.tex
index dfacfd2e61..46da664c6f 100644
--- a/macros/generic/tokcycle/tokcycle-doc.tex
+++ b/macros/generic/tokcycle/tokcycle-doc.tex
@@ -25,7 +25,12 @@
\newcommand\Space{\textit{Space}}
\newcommand\CGMS{\textit{C-G-M-S}}
\newcommand\CGMnS{\textit{C}, \textit{G}, \textit{M}, and \textit{S}}
-\newcommand\macname[1]{\texttt{\char92#1}}
+\begingroup
+\catcode`|=\active
+\gdef|{\textbackslash}
+\gdef\xmacaux#1{\ttfamily\char92#1\endgroup}
+\endgroup
+\gdef\macname{\begingroup\catcode`|=\active \xmacaux}
\newcommand\thevbox{\medskip\theverbbox\par\medskip\noindent}
\newenvironment{specfig}
{\par\bigskip\vbox\bgroup\small\sffamily\centering}
@@ -153,7 +158,7 @@ As tokens/groups are read from the input stream, they are
with that category of token/group.
The \TokCycle{} categories by which the input stream
is dissected include \Character{},
- \Group{}, \Macro{}, and \Space{}.
+ \Group{}, \Macro{}, and \Space{} (alphabetized for easy recall).
Catcode-0 tokens are directed to the \Macro{} directive for processing.
Catcode-10 tokens are directed to the \Space{} directive.
@@ -170,6 +175,8 @@ The handling options of implicit cat-1 and 2 tokens are described later in
Valid tokens that are neither catcode 0, 1, 2, nor 10, except where noted,
are directed to the \Character{} directive for processing.
+\subsection{Provided (built-in) \TokCycle{} macros and environments}
+
The syntax of the non-\texttt{xpress} macros/environments is\medskip
\begin{verbbox}[\vbdelim]
@@ -206,6 +213,8 @@ or, alternately, for the \texttt{xpress}-pseudo-environment,
\thevbox
\begin{sloppypar}
+\subsection{Create your own \TokCycle{} environments}
+
In addition to the above macros/environments, the means is
provided to define new \TokCycle{} environments:
@@ -311,7 +320,7 @@ Let it suffice for now to say that the default directive settings
\section{Commands in the \TokCycle{} directives}\label{s:coms}
-The command-line token cycling tools provided in the package are
+The document-level token cycling tools provided in the package are
listed in section~\ref{s:intro}.
For each of those commands and/or pseudo-environments, the
user must (explicitly or implicitly) detail a set of directives to
@@ -326,7 +335,12 @@ There are, however, several macros provided by the package to assist
The recommended way to apply this package is to collect the
\TokCycle-transformed results of the input stream in a token register
provided by the package, named \macname{cytoks}.
-Its contents can then be typeset via \verb|\the\cytoks|.
+Its contents can then be typeset via \verb|\the\cytoks|.\footnote{%
+If a token-cycle input stream contains no macros, or is to be detokenized,
+ or if the input-stream tokens are not to be typeset, it may be possible
+ (though not required) to bypass \macname{cytoks} and typeset the output
+ directly.
+}
The macro for appending things to \macname{cytoks}, to be used in
the package directives, is \macname{addcytoks}.
@@ -365,7 +379,7 @@ The specification of all \TokCycle{} processing directives is structured
in such a way that ``\argone{}'' may be directly employed to indicate
the current token (or group) under consideration in the input stream.
-\subsubsection{Transforming the Input Stream}
+\subsubsection{Transforming the input stream}
Within the \TokCycle{} \CGMS{} directives, the command
\verb|\addcytoks{#1}| is used to echo the token being processed to
@@ -449,8 +463,8 @@ However, there are times where you might wish to perform a macro task
(for example, one that might add additional tokens to \macname{cytoks})
immediately \textit{outside} of the group, before or after.
If you were to add the task macro before or after the \macname{processtoks}
- invocation, it will still be performed, by default, \textit{inside}
- the \macname{cytoks} group.
+ invocation (anywhere in the \Group{} directive), it will still be
+ performed, by default, \textit{inside} the \macname{cytoks} group.
The macro \macname{groupedcytoks} allows one to manually specify the
grouping duties of the \Group{} directive.
@@ -518,6 +532,201 @@ Finally, the special case of $-$1 indicates that the implicit-group
Such a special treatment has limited application---for example,
when the intent is to detokenize these tokens.
+\subsection{Looking ahead at the input stream}
+
+In the normal mode of \TokCycle{} operation, each token of the input stream
+ is successively digested and sent to one of four user-defined directives
+ for processing.
+Such an approach works well if there is no interdependency of adjacent tokens
+ in the input stream.
+When such an interdepency exists (for example, the argument associated with
+ an invoked macro), one might typically use flags that are set or cleared
+ in one directive that can then be status-checked when the subsequent token
+ is processed.
+While such an approach is wholly valid, it can create a complex web of
+ flag setting/checking that can span across multiple directives.
+
+With the release of v1.4 of the package, an alternative approach to handle
+ such dependencies has been developed: look-ahead features, so that the
+ occurrence of a particular token or condition can provoke an immediate
+ examination of the input stream to handle a possible dependency, without
+ waiting for the next iteration of the token cycle.
+
+It should be noted that the use of look-ahead is perhaps more
+ \textit{dangerous} than the traditional use of flags---for if it is not
+ performed with care, tampering with the input stream can destroy its
+ integrity.
+The commands developed for these situations will now be described.
+They are designed to be employed as part of \TokCycle's \Character, \Macro,
+ and \Space{} directives for checking and/or handling successively linked
+ tokens in the input stream.
+They cannot be used inside the \Group{} directive.
+
+\subsubsection{\macname{tcpeek}}
+
+When issued witin a directive as, for example \macname{tcpeek|z}, the next
+ token from the input stream is \macname{futurelet} into \macname{z}.
+This has several implications.
+The input stream remains undisturbed, so that the peeked-at token will
+ still be the token to be digested in the next iteration of the token cycle.
+Because it has been \macname{let}, \macname{z} does not \textit{contain} the token
+ from the input stream in the manner of a \macname{def}, but rather it is a
+ separate token \textit{possessing the same properties} as the input-stream token.
+As such, it is ideal to be used for \macname{ifx} comparisons.
+For example, \macname{ifx|bgroup|z} will detect whether a cat-1 group-opening
+ token is at the head of the input stream, e.g., an opening brace, `\{';
+the comparison \macname{ifx|tcsptoken|z} will detect whether a cat-10 space
+ is at the head of the input stream.
+Such comparisons can be useful in directing the logic of the \TokCycle{}
+ directive.
+
+\subsubsection{\macname{tcpop}}
+
+When issued as, for example \macname{tcpop|z}, the next token(s) from the
+ input stream is \textit{immediately} digested as an argument and the
+ tokens are placed into the macro \macname{z}.
+This token (or tokens) will no longer be digested as part of the next
+ iteration within the token cycle, unless the tokens are subsequently
+ replaced at the head of the input stream (see \macname{tcpush} below).
+
+Beware that when \TeX{} absorbs an argument, leading white space is lost.
+Further, if the argument was enclosed in cat-1,2 braces, the braces are
+ stripped and the whole group is absorbed as the argument.
+In some cases, if the argument needs manipulation, this brace-stripping
+ behavior may be desired.
+However, if the retention of the leading white space and the braces are
+ desired, one can use manual techniques (possibly involving macros
+ such as \macname{tcpeek}, \macname{tcpopwhitespace} and
+ \macname{tcpushgroup}) or one can instead use \macname{tcpopliteral}
+ (see below) in lieu of \macname{tcpop}.
+
+If \macname{tcpop} is used nonetheless, to aid in such matters when
+ cat-10 space is at the head of the input stream, the flag
+ \macname{spacepoppedtrue} will be set when a \macname{tcpop} is issued.
+However, even \macname{ifspacepopped} will be unable to differentiate explicit
+ ``white'' space that is lost during argument absorption versus implicit
+ space (e.g., \macname{@sptoken} or active-implicit space) that is not
+ discarded.
+Another option for dealing with leading white space is
+ \macname{tcpopwhitespace} (see below).
+
+
+\subsubsection{\macname{tcpopliteral}}
+
+This macro is an alternative to \macname{tcpop}, if the retention of
+ leading white space and possibly braces are required when popping
+ an argument from the input stream.
+This can be particularly useful if the popped tokens must later be
+ placed back into the input stream in their original state.
+The syntax, \macname{tcpopliteral|z}, pops an argument into \macname{z},
+ while retaining possible leading whitespace and brace groups.
+
+\subsubsection{\macname{tcpopappto}\ and \macname{tcpopliteralappto}}
+
+When issued as, for example \macname{tcpopappto|z}, the next token(s) from the
+ input stream is absorbed as an argument and \textit{appended} to the
+ replacement text of \macname{z}.
+The same group/space provisos affecting \macname{tcpop} apply here as well.
+The control sequence to be appended, here \macname{z}, may not be undefined
+ when \macname{tcpopappto} is invoked.
+This macro is just a convenient joining of two macros: \macname{tcpop}
+ and \macname{tcappto\#1from\#2} (see below).
+
+The same applies for the macro \macname{tcpopliteralappto}, which conveniently
+ joins \macname{tcpopliteral} and \macname{tcappto\#1from\#2}.
+
+\subsubsection{\macname{tcpopuntil}}
+
+This command pops one or more tokens from the input stream in the manner of
+ a delimited argument.
+Thus, when issued as, for example \macname{tcpopuntil~0|z}, tokens from
+ the input stream are absorbed into \macname{z} until an \texttt{0} token is
+ reached.
+Unlike delimited-argument absorption, however, the delimiter (\texttt{0} in
+ this example) is also added to the specified control sequence, in this case
+ \macname{z}.
+This construct is very useful for obtaining optional-argument tokens from the
+ input stream.
+Consider the following code, with an input stream of \texttt{[1ex]\{2ex\}}:
+\begin{verbatim}
+ \tcpeek\z
+ \ifx[\z\tcpopuntil ]\q\else\def\q{}\fi
+\end{verbatim}
+Since an optional argument is next in the input stream, \macname{q} will
+ obtain the replacement text \texttt{[1ex]}, and the input stream will now
+ begin with \texttt{\{2ex\}}.
+If an optional argument had not been next in the input stream, then
+ \macname{q} would be empty and the input stream would still begin
+ with \texttt{\{2ex\}}.
+
+\subsubsection{\macname{tcpopwhitespace}}
+
+When issued as, for example \macname{tcpopwhitespace|z}, leading white space
+ from the input stream will be absorbed and set in \macname{z}, in the
+ manner of a \macname{def}.
+Unlike the flag \macname{ifspacepopped}, leading implicit space will not be
+ indicated by this macro---only leading explicit white space.
+If no leading white space is present, \macname{z} will be empty following
+ its invocation and no tokens will have been removed from the input stream.
+
+\subsubsection{\macname{tcpush}}
+
+To this point, several commands have been described for reading and/or
+ extracting tokens from the input stream.
+There are also a corresponding commands for placing material at the head
+ of the input stream, should that need arise.
+With an invocation of \macname{tcpush|z}, the replacement text of \macname{z}
+ will be pushed onto the input stream.
+Multiple pushes will be handled in a last-in-first-out fashion.
+
+Beware that, because argument absorption in \TeX{} will strip the braces of
+ an absorbed group, care must be taken. If the input stream leads with a
+ grouped quantity, such as \texttt{\{abc\}}, then \macname{tcpop|z|tcpush|z}
+ will end up with \texttt{abc} in the input stream, with the braces missing.
+\textit{The commands \macname{tcpop} and \macname{tcpush} are not strictly
+ inverse operations} (see \macname{tcpopliteral} as an alternative).
+For this reason, the command \macname{tcpushgroup} is also provided
+ (see below).
+
+The command \macname{tcpush} supports an optional argument that functions
+ in the same manner as that of \macname{addcytoks}, to provide additional
+ levels of expansion beyond the mere replacement text.
+Thus, \macname{tcpush[2]|z} will take the replacement text of \macname{z},
+ expand it twice, and push the result onto the input stream.
+In a similar way, \macname{tcpush[x]|z} will fully expand the replacement
+ text of \macname{z} and then place those tokens onto the input stream.
+Whereas \macname{addytoks} places tokens into the output stream (the
+ token list \macname{cytoks}), \macname{tcpush} places tokens at the
+ head of the input stream.
+
+\subsubsection{\macname{tcpushgroup}}
+
+The command \macname{tcpushgroup} functions in a manner similar to
+ \macname{tcpush}, except that the replacement text of the argument
+ is placed onto the input stream within a braced (cat-1,2) group.
+So if the replacement text of \macname{z} is \texttt{abc}, then the
+ invocation \macname{tcpushgroup|z} will place \texttt{\{abc\}} onto
+ the input stream.
+As with \macname{tcpush}, and with the identical syntax, the command
+ \macname{tcpushgroup} supports an optional argument that directs
+ addition levels of expansion beyond the mere replacement text.
+
+\subsubsection{\macname{tcappto\#1from\#2}}
+
+This macro does not touch the input stream, per se.
+However, as tokens are popped from the input stream and placed into macros,
+ this command conveniently allows for their aggregation
+Thus, if \macname{q} contains \texttt{abc} and the newly popped \macname{z}
+ contains \texttt{d}, then \macname{tcappto |q from |z} will append the
+ contents of \macname{z} to \macname{q}, so that, upon conclusion,
+ \macname{q} will contain the tokens \texttt{abcd}.
+Unexpandable tokens can also be appended directly using this macro, using
+ the syntax of \macname{tcappto |q from\{123\}}.
+Expandable tokens can also be added directly with the use of this macro,
+ by using a leading \macname{noexpand} or \macname{empty}, as in
+ \macname{tcappto |q from\{|noexpand|today\}}.
+
+
\subsection{Truncating the input stream}
The basic process of \TokCycle{} is one in which an input stream (or argument)
@@ -554,7 +763,7 @@ It may be examined within the \CGMS{} directives in order to
or for the remainder of the total \TokCycle{}
input stream.
-\subsubsection{\macname{truncategroup} and \macname{truncategroupiftokis}}
+\subsubsection{\macname{truncategroup}\ and \macname{truncategroupiftokis}}
\label{s:truncgrp}
As to truncating the input stream for the remainder of the \TokCycle{}
@@ -596,7 +805,7 @@ Thus, an example to echo macro tokens to the output stream, unless
Obviously, checks for particular character tokens would be placed in the
\Character{} directive, rather than the \Macro{} directive.
-\subsubsection{\macname{truncatecycle} and \macname{truncatecycleiftokis}}
+\subsubsection{\macname{truncatecycle}\ and \macname{truncatecycleiftokis}}
There are two commands that are in every way analogous to the
group-\hspace{0pt}terminating macros of section~\ref{s:truncgrp}.
@@ -746,7 +955,7 @@ i)~implicit grouping tokens (e.g., \macname{bgroup} and \macname{egroup})
ii)~implicit space tokens (e.g., \macname{@sptoken}) will be processed
by the \Space{} directive.
-\subsubsection{Active-Implicit Tokens, Including Spaces}
+\subsubsection{Active-implicit tokens, including spaces\label{s:acimsp}}
One may occasionally run across a token that is both active and implicit.
For example, in the following code, \texttt{Q} is made both active and implicit:
@@ -761,18 +970,20 @@ In general, both \macname{ifactivetok} and \macname{ifimplicittok}
This is true even in the case of active-implicit catcode-10 spaces, which
are always processed through the \Space{} directive.
-However, because of the process \TokCycle{} uses in digesting space
- tokens, the actual token passed to the \Space{} directive (so-called
- \verb|#1|), when an active-implicit space is encountered in the input
- stream, is a generic implicit space token named \macname{tc@sptoken}.
-However, the catcode-12 version of the active character that
- produced it will be, for that moment, retained in a definition
+As of \TokCycle{} v1.4, the actual active-implicit space, if
+ encountered in the input stream, will be passed as \verb|#1| to
+ the \Space{} directive, \textit{as long as the character code of the
+ token is still \macname{active}.}
+If the character code of that token is no longer active, a generic
+ implicit space token named \macname{tcsptoken} is instead passed
+ to the the \Space{} directive as \verb|#1|.
+In either case, the catcode-12 version of the active character that
+ was digested will be, for that moment, retained in a definition
named \macname{theactivespace}.
This can be useful if detokenization is required of the spaces.
Such an example is described in the \texttt{tokcycle-examples}
adjunct document.
-
\subsubsection{Parameter (cat-6) tokens (e.g., \texttt{\#}): \macname{ifcatSIX}}
\label{s:cat6}
@@ -846,7 +1057,7 @@ Note that unlike those macros requiring three arguments, the lone
catcodes of the two input toks \textbf{without expansion} (\texttt{nx}
denotes \macname{noexpand}).
-\subsection{\TokCycle{} Completion: \macname{aftertokcycle}
+\subsection{\TokCycle{} completion: \macname{aftertokcycle}\
and \macname{tcendgroup}}
The \TokCycle{} \textit{macros}, upon completion, do nothing.
@@ -957,12 +1168,18 @@ In any event, all cat-6 tokens are trapped and flag as \texttt{true}
\item A similar warning is to be made for active-implicit spaces as was
made for implicit-cat-6 tokens.
-In the case of active-implicit spaces, the name of the active character is
- momentarily stored in the \macname{theactivespace} macro.
-If one wishes to output the active \textit{name} to the \macname{cytoks} token list,
- one must make sure that \macname{theactivespace} is expanded once before
+In the case of active-implicit spaces, the name (a cat-12 string) of the
+ active character is momentarily stored in the \macname{theactivespace} macro.
+If one wishes to output the active \textit{name} to the \macname{cytoks} token
+ list, one must make sure that \macname{theactivespace} is expanded once before
being added to the token list.
+One difference here, however, is that, unlike a cat-6 token, an
+ active-implicit space (and not a string of it) is actually passed
+ to the \Space{} directive as \texttt{\#1}.
+In all but one special case, the active-implicit \texttt{\#1} will match that
+ which occurred in the input stream (see section~\ref{s:acimsp}).
+
\end{itemize}
\section*{Acknowledgments}
@@ -980,13 +1197,13 @@ The \texttt{tex.stackexchange} site provides a wonderful
opportunity to interact with the leading developers and
practitioners of \TeX{} and \LaTeX{}.
-\vspace{1in}
+\vspace{.6in}
\section*{Source Code}
%\vspace{-0.1in}
\Large\texttt{tokcycle.sty}
\verbfilenobox[\footnotesize]{tokcycle.sty}%
-%\vspace{-0.1in}
+\vspace{1.5in}
\noindent\texttt{tokcycle.tex}
\verbfilenobox[\footnotesize]{tokcycle.tex}