From fd03eb3aab9edb9b15484a1130509cebe187d8bd Mon Sep 17 00:00:00 2001 From: Norbert Preining Date: Fri, 28 May 2021 03:01:17 +0000 Subject: CTAN sync 202105280301 --- macros/generic/tokcycle/README | 2 +- macros/generic/tokcycle/tokcycle-doc.pdf | Bin 569457 -> 600633 bytes macros/generic/tokcycle/tokcycle-doc.tex | 265 +++++++++- macros/generic/tokcycle/tokcycle-examples.pdf | Bin 568281 -> 604836 bytes macros/generic/tokcycle/tokcycle-examples.tex | 688 +++++++++++++++++++++----- macros/generic/tokcycle/tokcycle.tex | 68 ++- 6 files changed, 878 insertions(+), 145 deletions(-) (limited to 'macros/generic/tokcycle') diff --git a/macros/generic/tokcycle/README b/macros/generic/tokcycle/README index ee682cbc05..f926917d49 100644 --- a/macros/generic/tokcycle/README +++ b/macros/generic/tokcycle/README @@ -5,7 +5,7 @@ Author: Steven B Segletes, Christian Tellechea (contributor) Files : 1) tokcycle.tex (tex/generic/tokcycle) 2) tokcycle.sty (tex/latex/tokcycle) - 3) tokcycle-doc.tex + 3) tokcycle-doc.tex (doc/latex/tokcycle) 4) tokcycle-doc.pdf (doc/latex/tokcycle) 5) tokcycle-examples.tex (doc/latex/tokcycle) 6) tokcycle-examples.pdf (doc/latex/tokcycle) diff --git a/macros/generic/tokcycle/tokcycle-doc.pdf b/macros/generic/tokcycle/tokcycle-doc.pdf index 3e4fe83347..bc646b0ccd 100644 Binary files a/macros/generic/tokcycle/tokcycle-doc.pdf and b/macros/generic/tokcycle/tokcycle-doc.pdf differ diff --git a/macros/generic/tokcycle/tokcycle-doc.tex b/macros/generic/tokcycle/tokcycle-doc.tex index dfacfd2e61..46da664c6f 100644 --- a/macros/generic/tokcycle/tokcycle-doc.tex +++ b/macros/generic/tokcycle/tokcycle-doc.tex @@ -25,7 +25,12 @@ \newcommand\Space{\textit{Space}} \newcommand\CGMS{\textit{C-G-M-S}} \newcommand\CGMnS{\textit{C}, \textit{G}, \textit{M}, and \textit{S}} -\newcommand\macname[1]{\texttt{\char92#1}} +\begingroup +\catcode`|=\active +\gdef|{\textbackslash} +\gdef\xmacaux#1{\ttfamily\char92#1\endgroup} +\endgroup +\gdef\macname{\begingroup\catcode`|=\active \xmacaux} \newcommand\thevbox{\medskip\theverbbox\par\medskip\noindent} \newenvironment{specfig} {\par\bigskip\vbox\bgroup\small\sffamily\centering} @@ -153,7 +158,7 @@ As tokens/groups are read from the input stream, they are with that category of token/group. The \TokCycle{} categories by which the input stream is dissected include \Character{}, - \Group{}, \Macro{}, and \Space{}. + \Group{}, \Macro{}, and \Space{} (alphabetized for easy recall). Catcode-0 tokens are directed to the \Macro{} directive for processing. Catcode-10 tokens are directed to the \Space{} directive. @@ -170,6 +175,8 @@ The handling options of implicit cat-1 and 2 tokens are described later in Valid tokens that are neither catcode 0, 1, 2, nor 10, except where noted, are directed to the \Character{} directive for processing. +\subsection{Provided (built-in) \TokCycle{} macros and environments} + The syntax of the non-\texttt{xpress} macros/environments is\medskip \begin{verbbox}[\vbdelim] @@ -206,6 +213,8 @@ or, alternately, for the \texttt{xpress}-pseudo-environment, \thevbox \begin{sloppypar} +\subsection{Create your own \TokCycle{} environments} + In addition to the above macros/environments, the means is provided to define new \TokCycle{} environments: @@ -311,7 +320,7 @@ Let it suffice for now to say that the default directive settings \section{Commands in the \TokCycle{} directives}\label{s:coms} -The command-line token cycling tools provided in the package are +The document-level token cycling tools provided in the package are listed in section~\ref{s:intro}. For each of those commands and/or pseudo-environments, the user must (explicitly or implicitly) detail a set of directives to @@ -326,7 +335,12 @@ There are, however, several macros provided by the package to assist The recommended way to apply this package is to collect the \TokCycle-transformed results of the input stream in a token register provided by the package, named \macname{cytoks}. -Its contents can then be typeset via \verb|\the\cytoks|. +Its contents can then be typeset via \verb|\the\cytoks|.\footnote{% +If a token-cycle input stream contains no macros, or is to be detokenized, + or if the input-stream tokens are not to be typeset, it may be possible + (though not required) to bypass \macname{cytoks} and typeset the output + directly. +} The macro for appending things to \macname{cytoks}, to be used in the package directives, is \macname{addcytoks}. @@ -365,7 +379,7 @@ The specification of all \TokCycle{} processing directives is structured in such a way that ``\argone{}'' may be directly employed to indicate the current token (or group) under consideration in the input stream. -\subsubsection{Transforming the Input Stream} +\subsubsection{Transforming the input stream} Within the \TokCycle{} \CGMS{} directives, the command \verb|\addcytoks{#1}| is used to echo the token being processed to @@ -449,8 +463,8 @@ However, there are times where you might wish to perform a macro task (for example, one that might add additional tokens to \macname{cytoks}) immediately \textit{outside} of the group, before or after. If you were to add the task macro before or after the \macname{processtoks} - invocation, it will still be performed, by default, \textit{inside} - the \macname{cytoks} group. + invocation (anywhere in the \Group{} directive), it will still be + performed, by default, \textit{inside} the \macname{cytoks} group. The macro \macname{groupedcytoks} allows one to manually specify the grouping duties of the \Group{} directive. @@ -518,6 +532,201 @@ Finally, the special case of $-$1 indicates that the implicit-group Such a special treatment has limited application---for example, when the intent is to detokenize these tokens. +\subsection{Looking ahead at the input stream} + +In the normal mode of \TokCycle{} operation, each token of the input stream + is successively digested and sent to one of four user-defined directives + for processing. +Such an approach works well if there is no interdependency of adjacent tokens + in the input stream. +When such an interdepency exists (for example, the argument associated with + an invoked macro), one might typically use flags that are set or cleared + in one directive that can then be status-checked when the subsequent token + is processed. +While such an approach is wholly valid, it can create a complex web of + flag setting/checking that can span across multiple directives. + +With the release of v1.4 of the package, an alternative approach to handle + such dependencies has been developed: look-ahead features, so that the + occurrence of a particular token or condition can provoke an immediate + examination of the input stream to handle a possible dependency, without + waiting for the next iteration of the token cycle. + +It should be noted that the use of look-ahead is perhaps more + \textit{dangerous} than the traditional use of flags---for if it is not + performed with care, tampering with the input stream can destroy its + integrity. +The commands developed for these situations will now be described. +They are designed to be employed as part of \TokCycle's \Character, \Macro, + and \Space{} directives for checking and/or handling successively linked + tokens in the input stream. +They cannot be used inside the \Group{} directive. + +\subsubsection{\macname{tcpeek}} + +When issued witin a directive as, for example \macname{tcpeek|z}, the next + token from the input stream is \macname{futurelet} into \macname{z}. +This has several implications. +The input stream remains undisturbed, so that the peeked-at token will + still be the token to be digested in the next iteration of the token cycle. +Because it has been \macname{let}, \macname{z} does not \textit{contain} the token + from the input stream in the manner of a \macname{def}, but rather it is a + separate token \textit{possessing the same properties} as the input-stream token. +As such, it is ideal to be used for \macname{ifx} comparisons. +For example, \macname{ifx|bgroup|z} will detect whether a cat-1 group-opening + token is at the head of the input stream, e.g., an opening brace, `\{'; +the comparison \macname{ifx|tcsptoken|z} will detect whether a cat-10 space + is at the head of the input stream. +Such comparisons can be useful in directing the logic of the \TokCycle{} + directive. + +\subsubsection{\macname{tcpop}} + +When issued as, for example \macname{tcpop|z}, the next token(s) from the + input stream is \textit{immediately} digested as an argument and the + tokens are placed into the macro \macname{z}. +This token (or tokens) will no longer be digested as part of the next + iteration within the token cycle, unless the tokens are subsequently + replaced at the head of the input stream (see \macname{tcpush} below). + +Beware that when \TeX{} absorbs an argument, leading white space is lost. +Further, if the argument was enclosed in cat-1,2 braces, the braces are + stripped and the whole group is absorbed as the argument. +In some cases, if the argument needs manipulation, this brace-stripping + behavior may be desired. +However, if the retention of the leading white space and the braces are + desired, one can use manual techniques (possibly involving macros + such as \macname{tcpeek}, \macname{tcpopwhitespace} and + \macname{tcpushgroup}) or one can instead use \macname{tcpopliteral} + (see below) in lieu of \macname{tcpop}. + +If \macname{tcpop} is used nonetheless, to aid in such matters when + cat-10 space is at the head of the input stream, the flag + \macname{spacepoppedtrue} will be set when a \macname{tcpop} is issued. +However, even \macname{ifspacepopped} will be unable to differentiate explicit + ``white'' space that is lost during argument absorption versus implicit + space (e.g., \macname{@sptoken} or active-implicit space) that is not + discarded. +Another option for dealing with leading white space is + \macname{tcpopwhitespace} (see below). + + +\subsubsection{\macname{tcpopliteral}} + +This macro is an alternative to \macname{tcpop}, if the retention of + leading white space and possibly braces are required when popping + an argument from the input stream. +This can be particularly useful if the popped tokens must later be + placed back into the input stream in their original state. +The syntax, \macname{tcpopliteral|z}, pops an argument into \macname{z}, + while retaining possible leading whitespace and brace groups. + +\subsubsection{\macname{tcpopappto}\ and \macname{tcpopliteralappto}} + +When issued as, for example \macname{tcpopappto|z}, the next token(s) from the + input stream is absorbed as an argument and \textit{appended} to the + replacement text of \macname{z}. +The same group/space provisos affecting \macname{tcpop} apply here as well. +The control sequence to be appended, here \macname{z}, may not be undefined + when \macname{tcpopappto} is invoked. +This macro is just a convenient joining of two macros: \macname{tcpop} + and \macname{tcappto\#1from\#2} (see below). + +The same applies for the macro \macname{tcpopliteralappto}, which conveniently + joins \macname{tcpopliteral} and \macname{tcappto\#1from\#2}. + +\subsubsection{\macname{tcpopuntil}} + +This command pops one or more tokens from the input stream in the manner of + a delimited argument. +Thus, when issued as, for example \macname{tcpopuntil~0|z}, tokens from + the input stream are absorbed into \macname{z} until an \texttt{0} token is + reached. +Unlike delimited-argument absorption, however, the delimiter (\texttt{0} in + this example) is also added to the specified control sequence, in this case + \macname{z}. +This construct is very useful for obtaining optional-argument tokens from the + input stream. +Consider the following code, with an input stream of \texttt{[1ex]\{2ex\}}: +\begin{verbatim} + \tcpeek\z + \ifx[\z\tcpopuntil ]\q\else\def\q{}\fi +\end{verbatim} +Since an optional argument is next in the input stream, \macname{q} will + obtain the replacement text \texttt{[1ex]}, and the input stream will now + begin with \texttt{\{2ex\}}. +If an optional argument had not been next in the input stream, then + \macname{q} would be empty and the input stream would still begin + with \texttt{\{2ex\}}. + +\subsubsection{\macname{tcpopwhitespace}} + +When issued as, for example \macname{tcpopwhitespace|z}, leading white space + from the input stream will be absorbed and set in \macname{z}, in the + manner of a \macname{def}. +Unlike the flag \macname{ifspacepopped}, leading implicit space will not be + indicated by this macro---only leading explicit white space. +If no leading white space is present, \macname{z} will be empty following + its invocation and no tokens will have been removed from the input stream. + +\subsubsection{\macname{tcpush}} + +To this point, several commands have been described for reading and/or + extracting tokens from the input stream. +There are also a corresponding commands for placing material at the head + of the input stream, should that need arise. +With an invocation of \macname{tcpush|z}, the replacement text of \macname{z} + will be pushed onto the input stream. +Multiple pushes will be handled in a last-in-first-out fashion. + +Beware that, because argument absorption in \TeX{} will strip the braces of + an absorbed group, care must be taken. If the input stream leads with a + grouped quantity, such as \texttt{\{abc\}}, then \macname{tcpop|z|tcpush|z} + will end up with \texttt{abc} in the input stream, with the braces missing. +\textit{The commands \macname{tcpop} and \macname{tcpush} are not strictly + inverse operations} (see \macname{tcpopliteral} as an alternative). +For this reason, the command \macname{tcpushgroup} is also provided + (see below). + +The command \macname{tcpush} supports an optional argument that functions + in the same manner as that of \macname{addcytoks}, to provide additional + levels of expansion beyond the mere replacement text. +Thus, \macname{tcpush[2]|z} will take the replacement text of \macname{z}, + expand it twice, and push the result onto the input stream. +In a similar way, \macname{tcpush[x]|z} will fully expand the replacement + text of \macname{z} and then place those tokens onto the input stream. +Whereas \macname{addytoks} places tokens into the output stream (the + token list \macname{cytoks}), \macname{tcpush} places tokens at the + head of the input stream. + +\subsubsection{\macname{tcpushgroup}} + +The command \macname{tcpushgroup} functions in a manner similar to + \macname{tcpush}, except that the replacement text of the argument + is placed onto the input stream within a braced (cat-1,2) group. +So if the replacement text of \macname{z} is \texttt{abc}, then the + invocation \macname{tcpushgroup|z} will place \texttt{\{abc\}} onto + the input stream. +As with \macname{tcpush}, and with the identical syntax, the command + \macname{tcpushgroup} supports an optional argument that directs + addition levels of expansion beyond the mere replacement text. + +\subsubsection{\macname{tcappto\#1from\#2}} + +This macro does not touch the input stream, per se. +However, as tokens are popped from the input stream and placed into macros, + this command conveniently allows for their aggregation +Thus, if \macname{q} contains \texttt{abc} and the newly popped \macname{z} + contains \texttt{d}, then \macname{tcappto |q from |z} will append the + contents of \macname{z} to \macname{q}, so that, upon conclusion, + \macname{q} will contain the tokens \texttt{abcd}. +Unexpandable tokens can also be appended directly using this macro, using + the syntax of \macname{tcappto |q from\{123\}}. +Expandable tokens can also be added directly with the use of this macro, + by using a leading \macname{noexpand} or \macname{empty}, as in + \macname{tcappto |q from\{|noexpand|today\}}. + + \subsection{Truncating the input stream} The basic process of \TokCycle{} is one in which an input stream (or argument) @@ -554,7 +763,7 @@ It may be examined within the \CGMS{} directives in order to or for the remainder of the total \TokCycle{} input stream. -\subsubsection{\macname{truncategroup} and \macname{truncategroupiftokis}} +\subsubsection{\macname{truncategroup}\ and \macname{truncategroupiftokis}} \label{s:truncgrp} As to truncating the input stream for the remainder of the \TokCycle{} @@ -596,7 +805,7 @@ Thus, an example to echo macro tokens to the output stream, unless Obviously, checks for particular character tokens would be placed in the \Character{} directive, rather than the \Macro{} directive. -\subsubsection{\macname{truncatecycle} and \macname{truncatecycleiftokis}} +\subsubsection{\macname{truncatecycle}\ and \macname{truncatecycleiftokis}} There are two commands that are in every way analogous to the group-\hspace{0pt}terminating macros of section~\ref{s:truncgrp}. @@ -746,7 +955,7 @@ i)~implicit grouping tokens (e.g., \macname{bgroup} and \macname{egroup}) ii)~implicit space tokens (e.g., \macname{@sptoken}) will be processed by the \Space{} directive. -\subsubsection{Active-Implicit Tokens, Including Spaces} +\subsubsection{Active-implicit tokens, including spaces\label{s:acimsp}} One may occasionally run across a token that is both active and implicit. For example, in the following code, \texttt{Q} is made both active and implicit: @@ -761,18 +970,20 @@ In general, both \macname{ifactivetok} and \macname{ifimplicittok} This is true even in the case of active-implicit catcode-10 spaces, which are always processed through the \Space{} directive. -However, because of the process \TokCycle{} uses in digesting space - tokens, the actual token passed to the \Space{} directive (so-called - \verb|#1|), when an active-implicit space is encountered in the input - stream, is a generic implicit space token named \macname{tc@sptoken}. -However, the catcode-12 version of the active character that - produced it will be, for that moment, retained in a definition +As of \TokCycle{} v1.4, the actual active-implicit space, if + encountered in the input stream, will be passed as \verb|#1| to + the \Space{} directive, \textit{as long as the character code of the + token is still \macname{active}.} +If the character code of that token is no longer active, a generic + implicit space token named \macname{tcsptoken} is instead passed + to the the \Space{} directive as \verb|#1|. +In either case, the catcode-12 version of the active character that + was digested will be, for that moment, retained in a definition named \macname{theactivespace}. This can be useful if detokenization is required of the spaces. Such an example is described in the \texttt{tokcycle-examples} adjunct document. - \subsubsection{Parameter (cat-6) tokens (e.g., \texttt{\#}): \macname{ifcatSIX}} \label{s:cat6} @@ -846,7 +1057,7 @@ Note that unlike those macros requiring three arguments, the lone catcodes of the two input toks \textbf{without expansion} (\texttt{nx} denotes \macname{noexpand}). -\subsection{\TokCycle{} Completion: \macname{aftertokcycle} +\subsection{\TokCycle{} completion: \macname{aftertokcycle}\ and \macname{tcendgroup}} The \TokCycle{} \textit{macros}, upon completion, do nothing. @@ -957,12 +1168,18 @@ In any event, all cat-6 tokens are trapped and flag as \texttt{true} \item A similar warning is to be made for active-implicit spaces as was made for implicit-cat-6 tokens. -In the case of active-implicit spaces, the name of the active character is - momentarily stored in the \macname{theactivespace} macro. -If one wishes to output the active \textit{name} to the \macname{cytoks} token list, - one must make sure that \macname{theactivespace} is expanded once before +In the case of active-implicit spaces, the name (a cat-12 string) of the + active character is momentarily stored in the \macname{theactivespace} macro. +If one wishes to output the active \textit{name} to the \macname{cytoks} token + list, one must make sure that \macname{theactivespace} is expanded once before being added to the token list. +One difference here, however, is that, unlike a cat-6 token, an + active-implicit space (and not a string of it) is actually passed + to the \Space{} directive as \texttt{\#1}. +In all but one special case, the active-implicit \texttt{\#1} will match that + which occurred in the input stream (see section~\ref{s:acimsp}). + \end{itemize} \section*{Acknowledgments} @@ -980,13 +1197,13 @@ The \texttt{tex.stackexchange} site provides a wonderful opportunity to interact with the leading developers and practitioners of \TeX{} and \LaTeX{}. -\vspace{1in} +\vspace{.6in} \section*{Source Code} %\vspace{-0.1in} \Large\texttt{tokcycle.sty} \verbfilenobox[\footnotesize]{tokcycle.sty}% -%\vspace{-0.1in} +\vspace{1.5in} \noindent\texttt{tokcycle.tex} \verbfilenobox[\footnotesize]{tokcycle.tex} diff --git a/macros/generic/tokcycle/tokcycle-examples.pdf b/macros/generic/tokcycle/tokcycle-examples.pdf index b5604b382a..72b188e887 100644 Binary files a/macros/generic/tokcycle/tokcycle-examples.pdf and b/macros/generic/tokcycle/tokcycle-examples.pdf differ diff --git a/macros/generic/tokcycle/tokcycle-examples.tex b/macros/generic/tokcycle/tokcycle-examples.tex index 891741bd4b..cdce49c48d 100644 --- a/macros/generic/tokcycle/tokcycle-examples.tex +++ b/macros/generic/tokcycle/tokcycle-examples.tex @@ -1,7 +1,11 @@ \documentclass[10pt]{article} \usepackage[margin=1.3in]{geometry} \usepackage[T1]{fontenc} -\usepackage{tokcycle,xcolor,manfnt,lmodern} +\usepackage{tokcycle,xcolor,manfnt,lmodern,mathtools} +%%%%TRIAL PACKAGE CHANGES (MAKE SURE EXAMPLE FILE COMPILES CORRECTLY) +\makeatletter +\makeatother +%%%%END TRIAL \colorlet{bred}{red!90!blue} \usepackage{tikz} % MACROS: REDEFINE TO ALLOW \cytoks TO ESCAPE SCOPE OF exampleA @@ -48,7 +52,8 @@ \def\altdecytoks{\par\medskip\begingroup\noindent\macname{cytoks} \ \textit{alt}detokenization:\\\raggedright\footnotesize\ttfamily - \expandafter\altdetokenize\expandafter{\the\cytoks}\normalsize\par\endgroup} + \expandafter\altdetokenize\expandafter{\the\cytoks}\normalsize\par + \endgroup\medskip} \newcommand\TokCycle{\textsf{tokcycle}} @@ -120,21 +125,21 @@ \newcommand\nextcap[1]{% \edef\tmp{#1}% - \tctestifx{-#1}{\def\capnext{T}}{}% + \tctestifx{-#1}{\def\capnext{T}}{}% <- TEST FOR HYPHEN \tctestifcon{\if T\capnext}% - {\tctestifcon{\ifcat\tmp A}% - {\uppercase{#1}\def\capnext{F}}% - {#1}}% - {#1}% + {\tctestifcon{\ifcat\tmp A}% <- TEST FOR NEXT CAT-11 + {\addcytoks{\uppercase{#1}}\def\capnext{F}}% + {\addcytoks{#1}}}% + {\addcytoks{#1}}% } \newcommand\TitleCase[1]{% - \def\capnext{T} + \def\capnext{T}% <- INITIAL ENTRY \tokcycle - {\addcytoks{\nextcap{##1}}} - {\processtoks{##1}} + {\nextcap{##1}} + {\def\capnext{T}\processtoks{##1}}% <- GROUP ENTRY {\addcytoks{##1}} - {\addcytoks{##1\def\capnext{T}}} + {\addcytoks{##1}\def\capnext{T}}% <-CAT-10 SPACE {#1}% \the\cytoks } @@ -201,24 +206,23 @@ \newcommand\nextcmptok[1]{\rotoradd{\rotcytoks{#1}}{\addcytoks{#1}}\testmatch{\tmp}} \newcommand\nextctltok[1]{\stepcounter{runcount}\addcytoks{#1}} -\newcommand\detokenizespacetok[1]{\tctestifcon{\ifimplicittok}% - {\tctestifcon{\ifactivetok}{\theactivespace}{\string#1}}{\detokenize{#1}}} - \newcounter{sigd} \newcounter{trailingz} +\newif\iffounddot +\newif\ifzerosig \newcommand\sigdigits[1]{% \setcounter{sigd}{0}% SIGNIFICANT DIGIT COUNT \setcounter{trailingz}{0}% TRAILING ZERO COUNT - \def\founddot{F}% MADE T WHEN DECIMAL HAS BEEN 1ST LOCATED - \def\zerosig{F}% MADE T WHEN ZERO BECOMES SIGNIFICANT + \founddotfalse% MADE T WHEN DECIMAL HAS BEEN 1ST LOCATED + \zerosigfalse% MADE T WHEN ZERO BECOMES SIGNIFICANT \tokcycle% CYCLE THROOUGH EACH TOKEN {\tctestifx{.##1}% - {\if T\founddot\throwerr\fi\def\founddot{T}}% IF . + {\iffounddot\throwerr\fi\founddottrue}% IF . {\tctestifnum{`##1>`/}% {\tctestifnum{`##1<`:}% {\tctestifnum{##1>0}% - {\stepcounter{sigd}\def\zerosig{T}\setcounter{trailingz}{0}}% IF 1-9 - {\if T\zerosig\stepcounter{sigd}\fi\stepcounter{trailingz}}% IF 0 + {\stepcounter{sigd}\zerosigtrue\setcounter{trailingz}{0}}% IF 1-9 + {\ifzerosig\stepcounter{sigd}\fi\stepcounter{trailingz}}% IF 0 }% {\throwerr}% IF ASCII > `9 }% @@ -229,7 +233,7 @@ {\throwerr}% IF CONTROL SEQUENCE {}% IGNORE SPACES {#1}% THE ARGUMENT - \addtocounter{sigd}{-\thetrailingz}% SUBTRACT TRAILING ZEROS + \iffounddot\else\addtocounter{sigd}{-\thetrailingz}\fi% TRAILING ZEROS \expandafter\ifx\expandafter\relax\detokenize\expandafter{\the\cytoks}% \relax\else\setcounter{sigd}{-1}\fi% CHECK FOR INVALID INPUT \thesigd @@ -269,6 +273,81 @@ {\dumpword\ifrunningcount\else\tcafterenv{\par(Wordcount=\thewordcount, Lettercount=\thelettercount)}\fi} +\newcounter{pointcount} +\newcommand\addtopoints[1]{(\textit{#1 point}% + \ifnum#1=1\relax\else\textit{s}\fi)} +\NewEnviron{pointtracker}[2][points]{% + \par\bigskip\resettokcycle + \setcounter{pointcount}{0}% + \Macrodirective{\addcytoks{##1}\tctestifx{\addtopoints##1}{\sumpoints}{}}% + \def\tmp{\tokencyclexpress{\large\bfseries#2: \thepointcount\ #1}}% + \expandafter\tmp\BODY\endtokencyclexpress +} +\def\sumpoints{\tcpop\Q\addtocounter{pointcount}{\Q}\tcpushgroup{\Q}} + +\newcommand\scalerule[1]{% + \tcpeek\zA + \ifx[\zA\tcpopuntil]\opt\fi% IF PRESENT, POP OPTIONAL ARG + \tcpop\ArgA% POP ARG1 + \tcpop\ArgB% POP ARG2 + \tcpushgroup[x]{#1\dimexpr\ArgB}% PUSH BACK IN REVERSE (LIFO) FASHION + \tcpushgroup[x]{#1\dimexpr\ArgA} + \ifx[\zA\tcpush[x]{\expandafter\scaleopt\opt{#1}}\fi +} +\def\scaleopt[#1]#2{[#2\dimexpr#1]} + +\newcommand\protectruleargs[1]{\ifx\rule#1% + \tcpeek\zA + \ifx[\zA\tcpopuntil]\opt + \addcytoks[1]{\opt}\fi% IF OPTIONAL ARG PRESENT, POP AND SAVE TO \cytoks + \tcpopliteral\ArgA% MUST POP ARG 1 WITH BRACES + \addcytoks[1]{\ArgA}% + \tcpopliteral\ArgB% MUST POP ARG 2 WITH BRACES + \addcytoks[1]{\ArgB}% + \fi +} + +\newif\iffoundstring +\newcommand\findstring[4]{% {CURRENT TOK}{MATCH STRING}{SUCCESS OP}{FAIL CMDS} + \foundstringfalse% DEFAULT CONDITION + \def\poptoks{}% INITIALIZATION. IF MATCH FAILS, MUST TO RESTORE POPPED TOKENS + \expandafter\matchnextchar\ifx#1#2\relax\relax\fi\empty\relax + \iffoundstring\addcytoks{#3{#2}}\else\tcpush{\poptoks}#4\fi +} +\def\matchnextchar#1#2\relax{\tctestifx{#1\empty}{}{% NO MATCH FROM 1ST TOKEN + \tctestifx{\relax#1}{% + \foundstringtrue% NOTHING LEFT TO COMPARE; MATCH FOUND + }{% + \tcpeek\Q% WITHOUT POPPING... + \tctestifx{\Q\bgroup}{}{% PEEK FOR GROUPS OR... + \tctestifx{\Q\tcsptoken}{}{% SPACES THAT INDICATE MATCH FAILURE + \tcpopappto\poptoks% POP TOKEN AND APPEND TO \poptoks + \expandafter\tctestifx\expandafter{\Q#1}{% + \matchnextchar#2\relax% STILL MATCHING; KEEP GOING + }{}% + }% + }% + }% +}} + +\newcommand\foo{$\bullet$} +\newcommand\foocheck[1]{\tcpeek\z\ifx\foo\z\else + \ifx\par\z\else\addcytoks{#1}\fi\fi} +\tokcycleenvironment\dofoo + {\addcytoks{##1}} + {\processtoks{##1}} + {\ifx\par##1\foocheck{##1}\else\addcytoks{##1}\fi} + {\foocheck{##1}} + +\newcommand\coleqchek[1]{\tcpeek\z + \ifx=\z\tcpop\z\addcytoks{\coloneqq}\else\addcytoks{#1}\fi} +\tokcycleenvironment\subcoloneqq +{\tctestifx{:##1}{\coleqchek{##1}}{\addcytoks{##1}}} +{\processtoks{##1}} +{\tctestifcon{\if\detokenize{:}\detokenize{##1}} + {\coleqchek{##1}}{\addcytoks{##1}}} +{\addcytoks{##1}} + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Examples, examples, and more examples} @@ -292,7 +371,8 @@ Apply different directives to Characters (under-dot), Groups \begingroup -\begin{exampleA}[lefthand width=7.8cm]{} +\begin{exampleA}[lefthand width=7.8cm]{Apply different directives to + characters{,} groups{,} macros{,} and spaces} \tokcycle{\addcytoks{\underdot{#1}}} {\addcytoks{\{}\processtoks{#1} \addcytoks{\}}} @@ -307,9 +387,13 @@ Apply different directives to Characters (under-dot), Groups \subsubsection{Escaping text} Text between two successive escape characters is bypassed - by \TokCycle{} and instead echoed to the output register. -Default escape character is $|$. One can change it with + by \TokCycle{} and instead echoed to the output token list + (\macname{cytoks}). +The default escape character is $|$. +One can change the token-cycle escape character by way of the \macname{settcEscapechar} macro. +Below, the character codes are incremented by one; however, the + arguments of \macname{rule} are protected against that. \begingroup @@ -343,6 +427,7 @@ This section concerns the issue of whether the characters of the input output token list (\macname{cytoks}). \medskip +\begin{minipage}[t]{\textwidth} Transform characters (+1 ASCII) via unexpandable macro:\medskip \begin{exampleA}[lefthand width=8.3cm] @@ -361,6 +446,7 @@ Transform characters (+1 ASCII) via unexpandable macro:\medskip Note how the above detokenization shows the actual transformation instructions involving \macname{plusl}. This is done because \macname{plusl} is not expandable. +\end{minipage} \par\noindent\hrulefill\medskip \endgroup @@ -394,7 +480,6 @@ Capitalize vowels (but don't expand the character directive) \altdecytoks -\medskip The above detokenization also shows the transformation macro, \macname{vowelcap}, as part of \macname{cytoks}. But because \macname{vowelcap} is expandable, we did not need to @@ -404,9 +489,16 @@ But because \macname{vowelcap} is expandable, we did not need to \endgroup \end{minipage} +\begin{minipage}[t]{\textwidth} \begingroup Capitalize vowels (expanding the character directive) +\medskip +By using the \texttt{[x]} option of \macname{addcytoks}, the + \macname{vowelcap} is fully expanded prior to being placed in + the \macname{cytoks} token list. + + \begin{exampleA}[lefthand width=8.3cm] {Expanded Character directive} \tokcycle @@ -420,11 +512,12 @@ Capitalize vowels (expanding the character directive) \altdecytoks -\medskip Here, we see no vestiges of \macname{vowelcap} in \macname{cytoks}, but only the transformed result. \endgroup +\end{minipage} +\enlargethispage{12pt} \subsubsection{Unexpanded vs. pre-expanded input stream} \begingroup @@ -446,7 +539,6 @@ Here, we see no vestiges of \macname{vowelcap} in \macname{cytoks}, \altdecytoks -\medskip In the above case, vowel-capping is not applied to the date, because \macname{today} remains unexpanded in the \macname{cytoks} token list. @@ -468,7 +560,6 @@ Note that, when pre-expanding the input stream, one must \altdecytoks -\medskip Here, we see the date subjected to vowel-capping, because the \TokCycle{} input was expanded prior to the token cycle. Thus \macname{today} no longer appears \textit{explicitly} in the @@ -494,7 +585,7 @@ The desired options are to be set prior to the \TokCycle{} invocation. The macro \macname{stripimplicitgroupingcase} can take three possible integer arguments: 0 (default) to automatically place unaltered - implicit group tokens in the output register; 1 to strip + implicit group tokens in the \macname{cytoks} output; 1 to strip implicit group tokens from the output; or $-$1 to instead pass the implicit group tokens to the Character directive (as implicit tokens) for separate processing (typically, when detokenization @@ -547,8 +638,8 @@ Group stripping, brought about by \macname{stripgroupingtrue}, involves removing the grouping braces from around the group. The choice of \macname{processtoks} vs.\@ \macname{addcytoks} affects whether the tokens inside the group are recommitted - to \TokCycle{} for processing, or are merely sent to the output - register in their original unprocessed form. + to \TokCycle{} for processing, or are merely sent to the + \macname{cytoks} token list in their original unprocessed form. \medskip Note that, in these examples, underdots and visible spaces @@ -700,7 +791,6 @@ The final disposition of the \macname{cytoks} token list shows the effects of \altdecytoks -\medskip Without \macname{stripgroupingtrue}, all tasks, A,B,C, and D, would have been contained within the explicit \texttt{\{...\}} group. @@ -785,7 +875,7 @@ In certain applications, it may offer a more pleasingly formatted typesetting of detokenized material. \medskip It is an unusual application of \TokCycle{} in that it does not - actually use the \macname{cytoks} token register to collect + actually use the \macname{cytoks} token list to collect its output. This is only possible because all macros in the input stream are detokenized, rather than executed. @@ -821,29 +911,29 @@ versus \string\detokenize: \\ \subsubsection{Capitalize all words, including compound and - parenthetical words} + parenthetical words\label{s:cap}} \begingroup \begin{exampleC}[]{The \macname{Titlecase} and \texttt{\string\nextcap} macros} \newcommand\TitleCase[1]{% - \def\capnext{T} + \def\capnext{T}% <- INITIAL ENTRY \tokcycle - {\addcytoks{\nextcap{##1}}} - {\processtoks{##1}} + {\nextcap{##1}} + {\def\capnext{T}\processtoks{##1}}% <- GROUP ENTRY {\addcytoks{##1}} - {\addcytoks{##1\def\capnext{T}}} + {\addcytoks{##1}\def\capnext{T}}% <-CAT-10 SPACE {#1}% \the\cytoks } \newcommand\nextcap[1]{% \edef\tmp{#1}% - \tctestifx{-#1}{\def\capnext{T}}{}% + \tctestifx{-#1}{\def\capnext{T}}{}% <- TEST FOR HYPHEN \tctestifcon{\if T\capnext}% - {\tctestifcon{\ifcat\tmp A}% - {\uppercase{#1}\def\capnext{F}}% - {#1}}% - {#1}% + {\tctestifcon{\ifcat\tmp A}% <- TEST FOR NEXT CAT-11 + {\addcytoks{\uppercase{#1}}\def\capnext{F}}% + {\addcytoks{#1}}}% + {\addcytoks{#1}}% } \end{exampleC} @@ -859,6 +949,19 @@ versus \string\detokenize: \\ (\textit{capitalizing} words).} \end{exampleA} +As it is currently set up, a future-capitalization of the next catcode-11 token will be triggered when any of the following is encountered in the input stream: +\begin{itemize} +\item The initial entry into the routine + +\item a catcode-10 space token + +\item a hyphen \texttt{-} + +\item upon entry into a group \texttt{\{...\}} +\end{itemize} + +Obviously, additional tests could be added. + \endgroup \subsubsection{Scaling rule dimensions} @@ -890,7 +993,7 @@ versus \string\detokenize: \\ \endgroup -\subsubsection{String search, including non-regex material} +\subsubsection{String search, including non-regex material\label{s:searchA}} \begingroup @@ -953,8 +1056,8 @@ versus \string\detokenize: \\ \begingroup The problem of counting the significant digits in a number is an interesting - one, in that leading and trailing zeros are not significant, but zeros - in the middle of the number are significant. + one, in that leading zeros are not significant, trailing zeros are sometimes + significant, but zeros in the middle of the number are significant. In a problem like this, the \macname{cytoks} token list is not used to carry a transformation of the input tokens, but only to carry a record of errant tokens from the input (as seen from the helper macro @@ -977,30 +1080,32 @@ The main work of the \macname{sigdigits} macro is to use the counters \begin{exampleC}[]{The \macname{sigdigits} macro for counting significant digits} \newcounter{sigd} \newcounter{trailingz} +\newif\iffounddot +\newif\ifzerosig \newcommand\sigdigits[1]{% \setcounter{sigd}{0}% SIGNIFICANT DIGIT COUNT \setcounter{trailingz}{0}% TRAILING ZERO COUNT - \def\founddot{F}% MADE T WHEN DECIMAL HAS BEEN 1ST LOCATED - \def\zerosig{F}% MADE T WHEN ZERO BECOMES SIGNIFICANT + \founddotfalse% MADE T WHEN DECIMAL HAS BEEN 1ST LOCATED + \zerosigfalse% MADE T WHEN ZERO BECOMES SIGNIFICANT \tokcycle% CYCLE THROOUGH EACH TOKEN - {\tctestifx{.##1}% - {\if T\founddot\throwerr\fi\def\founddot{T}}% IF . - {\tctestifnum{`##1>`/}% - {\tctestifnum{`##1<`:}% - {\tctestifnum{##1>0}% - {\stepcounter{sigd}\def\zerosig{T}\setcounter{trailingz}{0}}% IF 1-9 - {\if T\zerosig\stepcounter{sigd}\fi\stepcounter{trailingz}}% IF 0 - }% - {\throwerr}% IF ASCII > `9 - }% - {\throwerr}% IF ASCII < `0 + {\tctestifx{.##1}% + {\iffounddot\throwerr\fi\founddottrue}% IF . + {\tctestifnum{`##1>`/}% + {\tctestifnum{`##1<`:}% + {\tctestifnum{##1>0}% + {\stepcounter{sigd}\zerosigtrue\setcounter{trailingz}{0}}% IF 1-9 + {\ifzerosig\stepcounter{sigd}\fi\stepcounter{trailingz}}% IF 0 }% - }% APPLY ABOVE LOGIC FOR CHAR TOKENS - {\throwerr}% IF BRACES - {\throwerr}% IF CONTROL SEQUENCE - {}% IGNORE SPACES - {#1}% THE ARGUMENT - \addtocounter{sigd}{-\thetrailingz}% SUBTRACT TRAILING ZEROS + {\throwerr}% IF ASCII > `9 + }% + {\throwerr}% IF ASCII < `0 + }% + }% APPLY ABOVE LOGIC FOR CHAR TOKENS + {\throwerr}% IF BRACES + {\throwerr}% IF CONTROL SEQUENCE + {}% IGNORE SPACES + {#1}% THE ARGUMENT + \iffounddot\else\addtocounter{sigd}{-\thetrailingz}\fi% TRAILING ZEROS \expandafter\ifx\expandafter\relax\detokenize\expandafter{\the\cytoks}% \relax\else\setcounter{sigd}{-1}\fi% CHECK FOR INVALID INPUT \thesigd @@ -1213,6 +1318,11 @@ We provide two modes of output: one that provides an ending summary, and As with other \TokCycle{} environments and macros, portions can be exempted from the count (for example macro arguments) by using the escape delimiters provided. +An even more elaborate version of this environment can be found at +\texttt{https://tex.stackexchange.com/questions/\allowbreak + 44618/dynamically-count-and-return-number-of-words-in-a-section/587567\#587567},% +\hfil in\-cluding a version that runs + in Plain \TeX. \begin{exampleC}[]{The \macname{countem} extended-environment definition and supporting macros} @@ -1326,7 +1436,7 @@ Thus, ``-is loc'' is also added to the output stream. } \begin{exampleB}[text only, righthand width=0.01cm, colback=white]% -{Effects of various truncation directives} +{Example: Effects of various truncation directives} {\footnotesize\ttfamily Input text}: \expandafter\altdetokenize\expandafter {\thetext} @@ -1350,9 +1460,338 @@ For example, a detokenization of the final case reveals this to be \endgroup -\subsection{Advanced topics: implicit tokens and catcode changes} +\subsection{Look-ahead features} + +The whole concept of the \TokCycle{} package is to examine the input stream + one token at a time and use the directives to process the tokens. +Occasionally, however, there are tasks that involve several adjacent tokens. +In the example of section~\ref{s:cap}, for instance, in order to capitalize + words, each space token triggers an action on the character that follows. +To accomplish this without any look-ahead features, the Space directive sets + a flag, \verb|\def\capnext{T}|, in order to communicate with a the Character + directive about a subsequent token. +That is a perfectly valid approach to communicate conditional action. + +\medskip +However, \TokCycle{} also provides look-ahead features, to avoid the necessity + of communicating with future tokens in the cycle by way of flag settings. +These tools allow a directive to directly ``peek'' ahead at the input stream + (without disturbing it) and, if desired, push and pop tokens to and from + the input stream. +The package manual should be consulted for a full list of macros provided to + facilitate look-ahead capability. +But here are some examples to look at. + +\begingroup + +\subsubsection{Count in advance that which follows} + +Specify the point values for questions and let tokcycle perform the cumulative + tallies in advance, using a single-pass compilation. + +\begin{exampleC}[]{The \texttt{pointtracker} environment to count point values + in advance} +\newcounter{pointcount} +\newcommand\addtopoints[1]{(\textit{#1 point}% + \ifnum#1=1\relax\else\textit{s}\fi)} +\NewEnviron{pointtracker}[2][points]{% + \par\bigskip\resettokcycle + \setcounter{pointcount}{0}% + \Macrodirective{\addcytoks{##1}\tctestifx{\addtopoints##1}{\sumpoints}{}}% + \def\tmp{\tokencyclexpress{\large\bfseries#2: \thepointcount\ #1}}% + \expandafter\tmp\BODY\endtokencyclexpress +} +\def\sumpoints{\tcpop\Q\addtocounter{pointcount}{\Q}\tcpushgroup{\Q}} +\end{exampleC} + +\begin{exampleA}[lefthand width=6.2cm]{Demo of the pointtracker environment} +\begin{pointtracker}[total points]{\Large Science Test} +\begin{pointtracker}{Advanced Geology} + \begin{itemize} + \item Count the fingers on your left hand. Are there more than 9? + \addtopoints{1} + \item Draw a rectangle with five edges. \addtopoints{3} + \end{itemize} +\end{pointtracker} +\begin{pointtracker}{Meteorology} + \begin{itemize} + \item How many borgs does it take to brew a Danish beer? + \addtopoints{2} + \item What is the meaning of life? \addtopoints{4} + \end{itemize} +\end{pointtracker} +\end{pointtracker} +\end{exampleA} + +Note that this example has several neat features. +The points for Geology, Meteorology, and the Test total are not specified, + but calculated. +They are calculated based on content that appears later in the environment. +Traditionally, multiple compilations would be required with external data + written to the aux file, but that does not occur here---there is only + a single compilation. +Finally, becaue we are nesting the \TokCycle{} part of \texttt{pointtracker} + inside of a \macname{NewEnviron}, we see that the \texttt{pointtracker} + environment can be nested, without adverse effect. + +\endgroup + +\subsubsection{Modify the arguments of every instance of a particular macro} -\subsubsection{Trap Active Characters (catcode 13)} +\begingroup + +This example, through the use of the \macname{scalerule} macro, shows how + the \macname{rule} arguments, including the optional argument, can be + popped off the input stream, adjusted, and placed back in the stream. + +\begin{exampleC}[]{The \macname{scalerule} macro to scale all + \macname{rule} arguments} +\newcommand\scalerule[1]{% + \tcpeek\zA + \ifx[\zA\tcpopuntil]\opt\fi% IF PRESENT, POP OPTIONAL ARG + \tcpop\ArgA% POP ARG1 + \tcpop\ArgB% POP ARG2 + \tcpushgroup[x]{#1\dimexpr\ArgB}% PUSH BACK IN REVERSE (LIFO) FASHION + \tcpushgroup[x]{#1\dimexpr\ArgA} + \ifx[\zA\tcpush[x]{\expandafter\scaleopt\opt{#1}}\fi +} +\def\scaleopt[#1]#2{[#2\dimexpr#1]} +\end{exampleC} + +\begin{exampleA}[lefthand width=9.2cm]{Scale every \macname{rule} argument} +\resettokcycle +\settcEscapechar{!}% SO AS NOT TO INTERFERE WITH `|' TOKS IN tabular +\Macrodirective{\addcytoks{#1}\ifx\rule#1\scalerule{2}\fi} +\tokencyclexpress +The rules appearing in the following input stream are literally +!\rule[-2pt]{1.5ex}{1.5ex}! and !\rule{2ex}{3pt}!: + +My environment can include all kinds of rules + \rule[-2pt]{1.5ex}{1.5ex} and other macros, such as \today{} and +\begin{tabular}{c|c} +23 & 454\\\hline +x & y +\end{tabular} +but each rule \rule{2ex}{3pt} is grown by a factor of two. +\endtokencyclexpress +\end{exampleA} +\settcEscapechar{|} + +\endgroup + + +\subsubsection{Protect arguments of specified macro from Character + directive modification} + +\begingroup +Tokcycle provides the escape character $|$ to delimit tokens to be protected + from processing by tokcycle directives. +But that requires the input stream to be adjusted to include these delimiters. +Here's an alternative, not requiring input stream adjustment. +The Macro directive protects all \macname{rule} arguments against contamination + by the Character directive. + +\medskip +When a \macname{rule} is encountered, \macname{protectruleargs} is called + to pop the rule arguments from the input stream and place them directly + in the \macname{cytoks} token list, out of reach from the Character directive. + + +\begin{exampleC}[]{The \macname{protectruleargs} macro to prevent Character directive + contamination of arguments} +\newcommand\protectruleargs[1]{\ifx\rule#1% + \tcpeek\zA + \ifx[\zA\tcpopuntil]\opt + \addcytoks[1]{\opt}\fi% IF OPTIONAL ARG PRESENT, POP AND SAVE TO \cytoks + \tcpopliteral\ArgA% MUST POP ARG 1 WITH BRACES + \addcytoks[1]{\ArgA}% + \tcpopliteral\ArgB% MUST POP ARG 2 WITH BRACES + \addcytoks[1]{\ArgB}% + \fi +} +\end{exampleC} + +Here, every character is typeset with a slash suffix. +If this action were to be imposed to \macname{rule} argument characters, + it would destroy their integrity. +Fortunately, when the Macro directive detects a \macname{rule}, it can + take evasive action. + +\begin{exampleA}[lefthand width=9.1cm]{Protect every \macname{rule} argument + from Character directive contamination} +\resettokcycle +\Characterdirective{\addcytoks{#1\textup{/}\allowbreak}} +\Macrodirective{\addcytoks{#1}\protectruleargs{#1}} +\tokencyclexpress +My \textit{environment} modifies each character token, but rule + \rule[-4pt]{1.5ex}{1.5ex} arguments \rule{2ex}{3pt} are protected. +\endtokencyclexpress +\end{exampleA} +\endgroup + + +\begingroup + +\subsubsection{Find a string of character tokens} + +Earlier in this manuscript, in section~\ref{s:searchA}, a token cycle + was developed to search a token stream for a string match. +Here, we do something similar, except we use the new ``look ahead'' features + of \TokCycle{} to achieve it. +In this case, however, the search string cannot cross space or group + boundaries. +The \macname{matchnextchar} macro demonstrates a use of \macname{tcpopappto}, + which is one of the newly introduced \TokCycle{} commands; instead of + popping an argument from the input stream \textit{into} a macro, it pops it + and \textit{appends} it to a macro. + +\begin{exampleC}[]{The \macname{findstring} macro to perform a look{-}ahead string + search} +\newcommand\findstring[4]{% {CURRENT TOK}{MATCH STRING}{SUCCESS OP}{FAIL CMDS} + \foundstringfalse% DEFAULT CONDITION + \def\poptoks{}% INITIALIZATION. IF MATCH FAILS, MUST TO RESTORE POPPED TOKENS + \expandafter\matchnextchar\ifx#1#2\relax\relax\fi\empty\relax + \iffoundstring\addcytoks{#3{#2}}\else\tcpush{\poptoks}#4\fi +} +\def\matchnextchar#1#2\relax{\tctestifx{#1\empty}{}{% NO MATCH FROM 1ST TOKEN + \tctestifx{\relax#1}{% + \foundstringtrue% NOTHING LEFT TO COMPARE; MATCH FOUND + }{% + \tcpeek\Q% WITHOUT POPPING... + \tctestifx{\Q\bgroup}{}{% PEEK FOR GROUPS OR... + \tctestifx{\Q\tcsptoken}{}{% SPACES THAT INDICATE MATCH FAILURE + \tcpopappto\poptoks% POP TOKEN AND APPEND TO \poptoks + \expandafter\tctestifx\expandafter{\Q#1}{% + \matchnextchar#2\relax% STILL MATCHING; KEEP GOING + }{}% + }% + }% + }% +}} +\end{exampleC} + +\begin{exampleA}[lefthand width=8cm]{Search for the string ``red''} +\resettokcycle +\Characterdirective{\findstring{#1}{red}% + {\textcolor{purple}}{\addcytoks{#1}}} +\tokencyclexpress +Can I \textit{find red in the text}, even in the + MIDredDLE of words and groups? +\endtokencyclexpress +\end{exampleA} + +\endgroup + + +\subsubsection{Eliminate \textit{prior} spaces and pars before a particular + macro} + +The goal here is to eliminate spaces and paragraphs that occur + \textit{prior} to the occurrence of the macro \macname{foo}, which + sets a \macname{bullet}. +Looking backwards is not part of the \TeX{} approach. +Nonetheless, the problem can, with a token cycle, be turned on its head, + such that all occurrences of spaces and \macname{par} can be made + to look for a future instance of \macname{foo}. + +\medskip +Here, if a space or \macname{par} is detected, \macname{foocheck} is called. + It peeks at the token that follows the space or \macname{par}. +If the peeked argument is a \macname{par} or \macname{foo} itself, it discards + the current space or \macname{par}. + +\begingroup + +\begin{exampleC}[]{The \macname{dofoo} environment to remove prior spaces and + paragraphs before \macname{foo}} +\newcommand\foo{$\bullet$} +\newcommand\foocheck[1]{\tcpeek\z\ifx\foo\z\else + \ifx\par\z\else\addcytoks{#1}\fi\fi} +\tokcycleenvironment\dofoo + {\addcytoks{##1}} + {\processtoks{##1}} + {\ifx\par##1\foocheck{##1}\else\addcytoks{##1}\fi} + {\foocheck{##1}} +\end{exampleC} + +\begin{exampleA}[lefthand width=7cm]{Remove spaces and paragraphs before + \macname{foo}} +\dofoo +Lorem ipsum dolor sit amet. (No foo) + +Lorem ipsum dolor sit amet. \foo + +Lorem ipsum dolor sit amet.\foo + +Lorem ipsum dolor sit amet. +\foo + +Lorem ipsum dolor sit amet. + +\foo +\enddofoo +\end{exampleA} +\endgroup + + +\subsubsection{Replace := with \macname{coloneqq} (\texttt{mathtools} + package required)} + +In most cases, creating a macro to signify a composite symbol makes the most + sense. +However, sometimes you don't realize that until you've typed in a lot of code + that fails to use such a macro. +In such a situation, \TokCycle{} can help. + +\medskip +In the \texttt{mathtools} package, there is a symbol, \macname{coloneqq}, + a colon followed by an equal sign, in which the colon has been + vertically adjusted to be symmetric with respect to the \texttt{=} sign. +The adjustment is slight, but noticeable. +A token cycle can be set up to look for instances of the \texttt{:} character, + peek ahead to see if it is followed by an \texttt{=} character and, if so, + replace the pair with \macname{coloneqq}. +However, the process is further complicated, if one is a user of +\macname{usepackage[french]\{babel\}}, where the \texttt{:} character is is made + an active character. + +\medskip +Therefore, in this case, not only does \texttt{:} need to be located by + the Character directive (in the normal case), but it also need be searched + for by the Macro directive, in the event that + \macname{usepackage[french]\{babel\}} is being employed. +In the example below, there are two such colons that satisfy the criterion for + replacement. + +\begingroup + +\begin{exampleC}[]{The \macname{coleqchek} macro and the \macname{subcoloneqq} + environment} +\newcommand\coleqchek[1]{\tcpeek\z + \ifx=\z\tcpop\z\addcytoks{\coloneqq}\else\addcytoks{#1}\fi} +\tokcycleenvironment\subcoloneqq +{\tctestifx{:##1}{\coleqchek{##1}}{\addcytoks{##1}}}% <- IF : A CHARACTER +{\processtoks{##1}} +{\tctestifcon{\if\detokenize{:}\detokenize{##1}}% <- IF : AN ACTIVE MACRO + {\coleqchek{##1}}{\addcytoks{##1}}} +{\addcytoks{##1}} +\end{exampleC} + +\begin{exampleA}[lefthand width=6cm]{replace {:=} with \macname{coloneqq}} +\subcoloneqq +\LARGE Compare: $::=$ + +$a := b$ vs + $a \ordinarycolon= b$ vs + $a:b$. +\endsubcoloneqq +\end{exampleA} + +\endgroup + +\section{Advanced topics: implicit tokens and catcode changes} + +\subsection{Trap active characters (catcode 13)} %\begin{minipage}[t]{\textwidth} \begingroup @@ -1408,10 +1847,12 @@ This is a test!!\endtokencyclexpress are made before reaching \TokCycle{} processing. \svt hey are, thus, no longer detected as active, unless \macname{noexpand} is applied before the pre-expansion. -In this example, the `\texttt{!}' that is not \macname{noexpand}\kern.3pted - is converted to a `\texttt{?}' prior to reaching \TokCycle{} - processing (and thus, not detected as \macname{active}). -Likewise, the `\texttt{T}' is pre-converted to a `\texttt{+}': +In this example, the `\texttt{!}' that is \textit{not} subject to + a \macname{noexpand} is converted to a `\texttt{?}' prior to + reaching \TokCycle{} processing (and thus, not detected as + \macname{active}). +However, the `\texttt{T}' is not pre-converted to a `\texttt{+}', + because it is implicit, and not a definition: \catcode`!=\active \def !{?} @@ -1424,7 +1865,13 @@ Likewise, the `\texttt{T}' is pre-converted to a `\texttt{+}': \detokenize\expandafter{\the\cytoks} \end{exampleA} -However, pre-tokenization does not suffer this behavior: +\begingroup +\catcode`T=11 +\catcode`!=12 +However, pre-tokenization does not suffer this behavior. +The first `\texttt{!}' is tokenized as active during + its absorption into \macname{tmp}, as shown in the next example: +\endgroup \begin{exampleA}[lefthand width=7.8cm]{Pre-tokenized input stream does not affect active characters} \def\tmp{This is a test!!} @@ -1473,7 +1920,7 @@ The following example demonstrates this concept, while showing, \end{exampleA} \endgroup -\subsubsection{Trap Catcode 6 (explicit \& implicit) tokens} +\subsection{Trap catcode 6 (explicit \& implicit) tokens} \begingroup @@ -1569,7 +2016,7 @@ In this example, a macro is defined and then executed, subject to \endgroup -\subsubsection{Trap implicit tokens in general} +\subsection{Trap implicit tokens in general} \begingroup @@ -1656,7 +2103,7 @@ Active tokens processed through the character directive are followed with in the input stream as duplicate, e.g. \texttt{\#\#}. -\subsubsection{Changing grouping tokens (catcodes 1,2)} +\subsection{Changing grouping tokens (catcodes 1,2)} \begingroup @@ -1755,7 +2202,7 @@ Bottom line: \endgroup -\subsubsection{Catcode 10 space tokens\label{s:spc}} +\subsection{Catcode 10 space tokens\label{s:spc}} \begingroup @@ -1771,17 +2218,19 @@ However, \macname{ifimplicittok} may still be used to \ddbend -Also, if the implicit space is an active character, \macname{ifactivetok} - is also set, for the user's benefit. -Likewise, \macname{ifactivechar} is also checked for active spaces, to see +If the implicit space is \textit{also} an active character, + \macname{ifactivetok} is also set, for the user's benefit. +Likewise, \macname{ifactivechar} is also checked, to see if the charcode of the active space is still, indeed, active. While \verb|#1| may be used to represent the space in the space directive, - a special technique is required to recover the detokenized name of the + a special technique may be required to recover the detokenized name of the active space character. -In particular, \verb|#1| will, for active spaces, contain a generic - implicit space token, \macname{tc@sptoken}. -However, the name of the associated active character in the input stream - will be defined as a cat-12 token in \macname{theactivespace}. +In particular, if the active token is an implicit space, but the charcode + of that token is no longer active, \verb|#1| will, for this case, + contain a generic implicit space token, \macname{tcsptoken}. +However, for all active space tokens, the name of the associated active + character in the input stream will be defined as a cat-12 token in + \macname{theactivespace}. \medskip Note in the following examples that cat-10 tokens do \textit{not} @@ -1827,15 +2276,9 @@ catcode settings for the space and the underscore. \end{exampleA} The next two examples introduce implicit and active-implicit spaces. -The latter example also demonstrates detokenization of such spaces. - -\medskip -Here is a useful expandable macro for detokenizing space tokens - in the context of \TokCycle{}. -It can process not only explicit cat-10 spaces, but also implicit and - active-implicit spaces. -To add it to the \macname{cytoks} token list, make sure you use the - \verb|[x]| fully expanded option to \macname{addcytoks}. +In the first example, functionally identical to the prior example, + we see that \TokCycle{} can digest and direct such + tokens to the Space directive, here in the form of \macname{z}. %%% \resettokcycle @@ -1853,40 +2296,63 @@ To add it to the \macname{cytoks} token list, make sure you use the \endtokencyclexpress \end{exampleA} -\begin{minipage}[t]{\textwidth} -\begin{exampleC}[]{The \macname{detokenizespacetok} macro for handling - implicit and active-implicit spaces} -\newcommand\detokenizespacetok[1]{\tctestifcon{\ifimplicittok}% - {\tctestifcon{\ifactivetok}{\theactivespace}{\string#1}}{\detokenize{#1}}} -\end{exampleC} +In the second example, an active implicit space token is created, \texttt{Q}, + and stored in a definition, \macname{tmp}. +A \macname{string} is applied by the token cycle at one point to the + space tokens and we see that the proper implicit tokens are decoded, whether + they are active or macro. + +\medskip +The one case deserving special note is the final case of the example, + in which the catcode of \texttt{Q}, the active character, is restored + to a value of 12, before the previously tokenized, active-implicit + \texttt{Q} is passed to the token cycle. +In such a case, where future incantations of \texttt{Q} are no longer active, + then the active-implicit space token that gets passed to the + Space directive (by way of \macname{tmp}) is taken as a generic + active-implicit \macname{tcsptoken}, instead of the original + active-implicit token (\texttt{Q}). -\begin{exampleA}[lefthand width=10cm]{Active Implicit spaces work{,} too} +\medskip +Nonetheless, the original string of the active-implicit token (that is, a + cat-12 \texttt{Q}) is preserved in \macname{theactivespace}. +The reason for this behavior is that the absorption of spaces in \TokCycle{} + is a destructive process, in which the spaces are absorbed as strings, + so that a proper interpretation of their implicitness can be ascertained. +With the character no longer active, it would be more difficult to + recreate an active-implicit version of \textit{that} token to + pass to the Space directive. +Thus, a generic active-implicit space token is, instead, employed. + + +\begin{exampleA}[lefthand width=9.05cm]{Active Implicit spaces work{,} too} \resettokcycle -\Characterdirective{\addcytoks{\underdot{#1}}} \def\:{\let\z= } \: % \catcode`Q=\active % \def\:{\let Q= } \: % \catcode`\_=10 % -\tokencyclexpress -xQx x_x\z{}x% -\endtokencyclexpress +\def\tmp{xQx x_x\z{}x} +\expandafter\tokencyclexpress\tmp\endtokencyclexpress -\resettokcycle \Spacedirective{\addcytoks{\textcolor{cyan}\bgroup}% - \addcytoks[x]{(\detokenizespacetok{#1})}% + \addcytoks[x]{(\string#1\ifactivetok + \ifimplicittok\ifactivechar\else\noexpand + \textcolor{red}{:\theactivespace}\fi\fi\fi)}% \addcytoks{\egroup}} \medskip -\tokencyclexpress -xQx x_x\z{}x% -\endtokencyclexpress +\expandafter\tokencyclexpress\tmp\endtokencyclexpress + +\medskip +\catcode`Q=12 +\expandafter\tokencyclexpress\tmp\endtokencyclexpress + \end{exampleA} -\end{minipage} \endgroup -\subsubsection{Changes to catcode 0} +\subsection{Changes to catcode 0} \begingroup diff --git a/macros/generic/tokcycle/tokcycle.tex b/macros/generic/tokcycle/tokcycle.tex index 9aaec68b72..d2e3a059ca 100644 --- a/macros/generic/tokcycle/tokcycle.tex +++ b/macros/generic/tokcycle/tokcycle.tex @@ -1,7 +1,7 @@ \def\tcname {tokcycle} -\def\tcver {1.3} +\def\tcver {1.4} % -\def\tcdate {2021/03/10} +\def\tcdate {2021/05/27} % % Author : Steven B Segletes, Christian Tellechea (contributor) % Maintainer : Steven B Segletes @@ -52,16 +52,18 @@ \long\def\tc@exsecond#1#2{#2} \long\def\tc@testxifx{\tc@earg\tctestifx} \long\def\test@ifmacro#1{\tctestifcatnx#1\relax} -\def\tc@addtoks#1#2{\toks#1\expandafter{\the\toks#1#2}} +\def\tc@addtoks#1#2{\toks#1\expandafter{\the\toks#1 #2}} \def\add@tcdepth{\advance\tcdepth 1\relax} -\def\sub@tcdepth{\advance\tcdepth-1\relax}\def\tc@resetifs{\activetokfalse\implicittokfalse\tc@implicitgrpfalse - \catSIXfalse\activecharfalse\activetokunexpandablefalse - \def\theactivespace{}} +\def\sub@tcdepth{\advance\tcdepth-1\relax} +\def\tc@resetifs{\activetokfalse\implicittokfalse\tc@implicitgrpfalse\catSIXfalse + \activecharfalse\activetokunexpandablefalse\def\theactivespace{}} \long\def\count@stringtoks#1{\tc@earg\count@toks{\string#1}} \long\def\count@toks#1{\the\numexpr-1\count@@toks#1.\tc@endcnt} \long\def\count@@toks#1#2\tc@endcnt{+1\tc@ifempty{#2}{\relax}{\count@@toks#2\tc@endcnt}} \def\sv@hash{##} +\def\def@popname{\expandafter\def\tc@popname} % EXTERNAL TOOLS +\let\tcsptoken= \tc@sptoken \long\def\tctestifcon#1{#1\expandafter\tc@exfirst\else\expandafter\tc@exsecond\fi} \long\def\tctestifcatnx#1#2{\tctestifcon{\ifcat\noexpand#1\noexpand#2}} \long\def\tctestifx#1{\tctestifcon{\ifx#1}} @@ -73,6 +75,7 @@ \newif\ifactivetok \newif\ifactivechar \newif\ifactivetokunexpandable +\newif\ifspacepopped \newtoks\cytoks \long\def\tokcycleenvironment#1#2#3#4#5{\expandafter\def\expandafter#1% \expandafter{\expandafter\let\csname end\expandafter\tc@gobble @@ -122,7 +125,9 @@ {\implicittoktrue\tctestifcon{\if\expandafter\@firstoftwo\string\\\tc@str}% {\expandafter\@@@spcT\tc@gobble}% IMPLICIT MACRO SPACE {\activetoktrue\@@@@spcT}}}% IMPLICIT ACTIVE SPACE -\def\@@@@spcT#1{\def\theactivespace{#1}\trapactivechar{#1}\@@spcT{\tc@sptoken}} +\def\@@@@spcT#1{\def\theactivespace{#1}\trapactivechar{#1}\ifactivechar\edef\tc@spc + {\scantokens{#1\noexpand}}\else\def\tc@spc{\tcsptoken}\fi\expandafter\@@spcT + \expandafter{\tc@spc}} \def\@@@spcT{\csmk{\expandafter\@@spcT\thecs}} \def\@@spcT#1{\@spcT{#1}\@tokcycle} % GROUP DECODE @@ -161,7 +166,7 @@ \addcytoks\expandafter{\expandafter#1}}} \def\settcGrouping#1{\def\@tmp##1{#1}\tc@defx\@@tmp{\@tmp{\the\cytoks}}% \tc@earg\defineexit@grouped{\@@tmp}} -% FAUX TOKENIZATION OF COMMAND NAME (WHEN cat0 TOKEN HAS BEEN MADE cat12) +% FAUX TOKENIZATION OF COMMAND NAME (WHEN \ AND COMMAND-NAME TOKS ARE NOW cat12) \def\csmk#1{\def\csaftermk{#1}\toks0{}\@csmkA} \def\@csmkA{\futurelet\@tmp\@csmkB} \def\@csmkB{\tctestifx{\@tmp\tc@sptoken}% @@ -184,6 +189,38 @@ %% WILL ALSO TRAP ACTIVE \let TO PRIMITIVES AS IMPLICIT; UNDO LATER IN \can@absorb@@ \def\trapactivetokunexpandable#1{\tctestifcon{\expandafter\if \detokenize{#1}#1}{}{\activetoktrue\activetokunexpandabletrue\implicittoktrue}} +% FEATURES TO LOOK-AHEAD INTO THE INPUT STREAM (INTRODUCED v1.4) +\long\def\tcpeek#1#2\@tokcycle{\def\tc@tmp{\ifx#1\endtokcycraw + \let#1=\empty\fi#2\@tokcycle}\futurelet#1\tc@tmp}%___________________________PEEK_ +\def\tcpopliteral#1{\tcpopwhitespace#1\tcpeek\@tmp\ifx\@tmp\bgroup + \tcpop\@tmp\def\tc@tmp{\tcappto#1from}\expandafter\tc@tmp\expandafter + {\expandafter{\@tmp}}\else\tcpopappto#1\fi} +\def\tcpop{\long\def\tc@@@pop##1{\tctestifx{\endtokcycraw##1}{\def@popname{}% + \tc@tmp\endtokcycraw}{\def@popname{##1}\tc@tmp}}\tc@pop} +\long\def\tcpopuntil#1{\long\def\tc@@@pop##1#1{\def@popname{##1#1}\tc@tmp}\tc@pop} +\long\def\tc@pop#1#2\@tokcycle{\def\tc@popname{#1}\def\tc@tmp + {#2\@tokcycle}\futurelet\tc@futuretok\tc@@pop} +\def\tc@@pop{\tc@trapescape\tc@futuretok{\def@popname{}\tc@tmp}{\tctestifx{% + \endtokcycraw\tc@futuretok}{\def@popname{}\tc@tmp}{\tctestifx{\tc@sptoken% + \tc@futuretok}{\spacepoppedtrue\tc@@@pop}{\spacepoppedfalse\tc@@@pop}}}}%_____POP_ +\def\tcappto#1from#2{% + \expandafter\tc@defx\expandafter#1\expandafter{\expandafter#1#2}} +\def\tcpopliteralappto#1{\tcpopliteral\@tmp\tcappto#1from\@tmp} +\def\tcpopappto#1{\tcpop\@tmp\tcappto#1from\@tmp}%___________________________APPEND_ +\long\def\tcpopwhitespace#1#2\@tokcycle{\def\tc@popname{#1}\def@popname{}% + \def\tc@tmp{#2\@tokcycle}\futurelet\tc@futuretok\tc@popspc} +\def\tc@popspc{\tctestifx{\tc@sptoken\tc@futuretok}{\discern@space}{\tc@tmp}} +\def\discern@space{\begingroup\def\@@spcT##1{\tctestifcon\ifimplicittok{\gdef + \tc@nxt{\tc@tmp##1}}{\gdef\tc@nxt{\def@popname{##1}\tc@tmp}}\endgroup + \tc@nxt}\stringify\@@@@@@spcT}%_____________________________________________SPACE_ +\long\def\@tcpush#1#2\@tokcycle{\def\tc@tmp{#2\@tokcycle}\expandafter\tc@tmp#1} +\long\def\@tcpushgroup#1#2\@tokcycle{\def\tc@tmp{#2\@tokcycle}\expandafter\tc@tmp + \expandafter{#1}} +% ...BORROW \addcytoks OPTIONAL ARGUMENT EXPANSION FEATURE FOR \tcpush[group] +\def\tcpush{\bgroup\long\def\tc@addtotoks##1{\egroup + \@tcpush{##1}}\futurelet\nxttok\addcytoks@A} +\def\tcpushgroup{\bgroup\long\def\tc@addtotoks##1{\egroup + \@tcpushgroup{##1}}\futurelet\nxttok\addcytoks@A}%___________________________PUSH_ % EXPRESS-INTERFACE - ALLOWS TO EXTERNALLY DEFINE DIRECTIVES \def\Characterdirective{\def\@chrT##1} \def\Groupdirective{\long\def\@grpT##1} @@ -247,7 +284,7 @@ v1.2 2020/10/01 in the definition \theactivespace. The flags \implicittok and \activetok are both set true, and the \activechar flag is checked, as well. -v1.3 2021/03/09 +v1.3 2021/03/10 - Introduced \xtokcycleenvironment, similar to \tokcycleenvironment, but allows two additional arguments, defining the setup and trailing code that will be run prior to the invocation and following conclusion of the @@ -283,3 +320,16 @@ v1.3 2021/03/09 \tcdepth associated with the Group directive (also, outer token cycle operates with depth = 0 rather than 1). - Renamed \@defgroupedcytoks to \defineexit@grouped, to better match function. + +v1.4 2021/05/26 + - Concerning the \@@@@spcT macro: previously, active-implicit spaces + were passed to the Spacedirective as \tc@sptoken, with the \string of the + active char passed in \theactivespace. Now, the active-implicit space token + itself is passed to the Spacedirective instead of \tc@sptoken, but only *IF* + the charcode of that character is currently active; otherwise, a generic + implicit space, \tcsptoken is passed. + - Introduced a set of "look ahead" macros: \tcpeek, \tcpop, \tcpopliteral, + \tcpopappto, \tcpopliteralappto, \tcappto#1from#2, \tcpopuntil, + \tcpopwhitespace, \tcpush, and \tcpushgroup. + - Bug fixed in \tc@addtoks definition, in the event that #2 was a number. + -- cgit v1.2.3