diff options
author | Karl Berry <karl@freefriends.org> | 2021-03-10 22:25:42 +0000 |
---|---|---|
committer | Karl Berry <karl@freefriends.org> | 2021-03-10 22:25:42 +0000 |
commit | 7a1fbab95b87f9c733da00b296f5134ca3805060 (patch) | |
tree | 6f2784e002f5f8871f46284c5b52506bab2771bb /Master/texmf-dist | |
parent | 201406d49d5e88b2cbb3bf40a03dbda41d90b0a1 (diff) |
tokcycle (10mar21) (branch)
git-svn-id: svn://tug.org/texlive/branches/branch2020.0@58262 c570f23f-e606-0410-a88d-b1316a301751
Diffstat (limited to 'Master/texmf-dist')
-rw-r--r-- | Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf | bin | 258406 -> 569457 bytes | |||
-rw-r--r-- | Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex | 431 | ||||
-rw-r--r-- | Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf | bin | 261160 -> 568281 bytes | |||
-rw-r--r-- | Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex | 593 | ||||
-rw-r--r-- | Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex | 102 |
5 files changed, 994 insertions, 132 deletions
diff --git a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf Binary files differindex c707af4f154..3e4fe833478 100644 --- a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf +++ b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf diff --git a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex index 612ab5baf29..dfacfd2e610 100644 --- a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex +++ b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex @@ -1,6 +1,6 @@ \documentclass{article} \usepackage[T1]{fontenc} -\usepackage{xcolor,tokcycle,verbatimbox,tabstackengine,lipsum} +\usepackage{xcolor,tokcycle,verbatimbox,tabstackengine,lipsum,lmodern} \colorlet{bred}{red!90!blue} \usepackage{caption} \captionsetup{font={sf, footnotesize},labelfont={sf},textfont={footnotesize}} @@ -26,6 +26,7 @@ \newcommand\CGMS{\textit{C-G-M-S}} \newcommand\CGMnS{\textit{C}, \textit{G}, \textit{M}, and \textit{S}} \newcommand\macname[1]{\texttt{\char92#1}} +\newcommand\thevbox{\medskip\theverbbox\par\medskip\noindent} \newenvironment{specfig} {\par\bigskip\vbox\bgroup\small\sffamily\centering} {\egroup\medskip\par} @@ -46,7 +47,7 @@ {\centering{\LARGE Package \TokCycle{} \Large (v\tcver)\par} Steven B Segletes -{\footnotesize $<$steven.b.segletes.civ@mail.mil$>$} +{\footnotesize $<$SSegletes@verizon.net$>$} \\\small{contributor: Christian Tellechea}\footnote{ {I am extremely grateful to Christian $<$unbonpetit@netc.fr$>$} for his assistance in the development of this package. @@ -111,7 +112,7 @@ The macros operate within the document's current scope, but do not typeset the result automatically. In the case of both macros and pseudo-environments, the transformed result is available for later use, being stored in the package - token register named \macname{cytoks}. + token list named \macname{cytoks}. \item Two macros and one pseudo-environment containing the phrase ``\texttt{xpress}''. @@ -156,12 +157,16 @@ The \TokCycle{} categories by which the input stream Catcode-0 tokens are directed to the \Macro{} directive for processing. Catcode-10 tokens are directed to the \Space{} directive. -When an explicit catcode-1 token is encountered in the \TokCycle{} input +When an explicit catcode-1 token\footnote{ + Throughout this document the terms \textit{catcode-} and \textit{cat-} + are used interchangeably. +} + is encountered in the \TokCycle{} input stream, the contents of the entire group (sans the grouping) are directed to the \Group{} directive for further processing, which may in turn, redirect the individual tokens to the other categories. The handling options of implicit cat-1 and 2 tokens are described later in - this document. + this document (section~\ref{s:impgrp}). Valid tokens that are neither catcode 0, 1, 2, nor 10, except where noted, are directed to the \Character{} directive for processing. @@ -185,20 +190,20 @@ or, alternately, for the pseudo-environment, {<Macro processing directive>} {<Space processing directive>}<token input stream>\endtokencycle \end{verbbox} -\medskip\theverbbox\medskip\par\noindent +\thevbox For the \texttt{xpress} macros, the syntax is \begin{verbbox}[\vbdelim] \tokcyclexpress <<or>> \expandedtokcyclexpress {<token input stream>} \end{verbbox} -\medskip\theverbbox\medskip\par\noindent\medskip +\thevbox or, alternately, for the \texttt{xpress}-pseudo-environment, \begin{verbbox}[\vbdelim] -\tokencycleexpress<token input stream>\endtokencyclexpress +\tokencyclexpress<token input stream>\endtokencyclexpress \end{verbbox} -\theverbbox\medskip +\thevbox \begin{sloppypar} In addition to the above macros/environments, the means is @@ -211,14 +216,55 @@ In addition to the above macros/environments, the means is {<Macro processing directive>} {<Space processing directive>} \end{verbbox} -\medskip\theverbbox\medskip\par\noindent +\thevbox This will then permit simplified invocations of the form \begin{verbbox}[\vbdelim] \environment_name<token input stream>\endenvironment_name \end{verbbox} -\medskip\theverbbox\medskip\par\noindent +\thevbox +More recently, an even more versatile \textit{extended} version has + been made available: +\begin{verbbox}[\vbdelim] +\xtokcycleenvironment\environment_name + {<Character processing directive>} + {<Group-content processing directive>} + {<Macro processing directive>} + {<Space processing directive>} + {<Set-up code>} + {<Close-out code>} +\end{verbbox} +\thevbox +This will also permit simplified invocations of the form + +\begin{verbbox}[\vbdelim] +\environment_name<token input stream>\endenvironment_name +\end{verbbox} +\thevbox +However, with this extended environment definition, additional + set-up and close-out code will be executed prior to and following + the token cycle, within the scope of the environment. +The close-out code, while executed after the token cycle, occurs + before the \macname{cytoks} token list is actually typeset; therefore + the close-out code can both execute macros and/or add tokens + to the imminently-typeset token list. +Any definitions or counters that change as a result of executing + the \CGMS{} directives are, in fact, \textit{past tense} when the + close-out code is executed. + +Finally, the set-up and close-out code, as well as the \CGMS{} directives + can invoke \macname{tcafterenv}, whose argument is executed upon the + exit from the scope of \textit{this} defined environment.\footnote{ +The \macname{tcafterenv} macro is recognized \textit{only} in the + context of an extended \macname{xtokcycleenvironment} definition. +It is not available for direct use in the package's \macname{tokencycle} + pseudo-environment, nor in \macname{tokcycleenvironment} definitions. +} +As with all \TokCycle{} environments, those created by way + of \macname{tokcycleenvironment} and \macname{xtokcycleenvironment}, + while operating in the scope of a group, will preserve the contents + of \macname{cytoks} when exiting their scope. \subsection{Externally specified directives and directive resets} @@ -251,13 +297,15 @@ The default directives at package outset and upon reset are \stripgroupingfalse \stripimplicitgroupingcase{0} \end{verbbox} -\medskip\theverbbox\medskip\noindent - +\thevbox The interpretation of these directives will be explained in the remainder of this document. Let it suffice for now to say that the default directive settings pass through the input stream to the output stream, without - change. + change.\footnote{ + Except, possibly, in the case of catcode-6 tokens, which will be + later addressed in sections~\ref{s:cat6} and \ref{s:whennot}. +} @@ -293,7 +341,7 @@ Its mandatory argument consists of tokens denoting \textit{what} you the letter \texttt{[x]}). When the optional argument is omitted, the specified tokens are appended - \texttt{literally} to the register (without expansion). + \textit{literally} to the register (without expansion). An integer option, call it \textit{n}, takes the the mandatory argument, and expands it $n$-times, and appends the result to \macname{cytoks}. @@ -303,23 +351,59 @@ The \texttt{[x]} option employs the \macname{expanded} primitive to maximally The \texttt{[x]} option will prove useful when the \Character{} or other directives involve a transformation that is fully expandable. Its use will allow the expanded - result of the transformation to be placed in the token register, rather + result of the transformation to be placed in the token list, rather than the unexpanded transformation instructions. \subsubsection{\argone} In the context of the \CGMnS{} processing directives, one may refer to - \argone{} (e.g., in the argument to \macname{addcytoks}). + \argone{} (for example, in the argument to \macname{addcytoks} or + \macname{processtoks}). \TeX{} users know that the first parameter to a \TeX{} macro is denoted as \argone. The specification of all \TokCycle{} processing directives is structured in such a way that ``\argone{}'' may be directly employed to indicate the current token (or group) under consideration in the input stream. +\subsubsection{Transforming the Input Stream} + +Within the \TokCycle{} \CGMS{} directives, the command + \verb|\addcytoks{#1}| is used to echo the token being processed to + the output stream that is being collected in the \macname{cytoks} + token list. +However, one may do more than merely echo the input---one may transform + it. +If the user creates a macro, let's call it \macname{xform}, to take + a single character of input and convert it in some way to something + else, then this macro can be used inside the \Character{} directive + as \verb|\addcytoks{\xform{#1}}|. +In this case, if an `\texttt{e}' were the token under consideration + by the \Character{} directive, `\verb|\xform{e}|' would be added + to the \macname{cytoks} token list. +Since \TokCycle{} operates upon each successive token in the input + stream, this \macname{xform} macro that operates on a single token + can be used to completely transform the \TokCycle{} input stream. + +If the \macname{xform} macro is expandable, reaching its termination in + a known number, $n$, of expansions, \macname{addcytoks} can take + advantage of that with its optional argument, performing the + said number of expansions of \macname{xform} \textit{before} + adding the result to \macname{cytoks}. +If the expandability of \macname{xform} is known, but the number $n$ is + not, the optional argument \verb|[x]| to \macname{addcytoks} may + be used to fully expand the argument (by way of the + \macname{expanded} primitive). +In that case, the resulting \macname{cytoks} result will show + no evidence of the \macname{xform} macro, but only the results + of its transformation. + +The accompanying examples document, \texttt{tokcycle-examples.pdf}, + is full of examples showing specific \TokCycle-induced transformations. + \subsection{\Group{} directive: \macname{ifstripgrouping}, and \macname{processtoks}} The \Group{} directive is unique, in that it is the only directive whose - argument (\texttt{\#1}) may consist of more than a single token. + argument (\argone) may consist of more than a single token. There are two issues to consider when handling the tokens comprising a group: do I retain or remove the grouping (cat-1,2 tokens)? Do I process the group's token content @@ -334,21 +418,85 @@ The alternative, \macname{stripgroupingtrue}, is dangerous in the sense that it will strip the cat-1,2 grouping from the group's tokens, thereby affecting or even breaking code that utilizes such grouping for macro arguments. -Modify \macname{ifstripgrouping} with care. +Apply \macname{stripgroupingtrue} with care. The issue of treating the tokens comprising the content of a group individually or collectively is determined by the choice of macro inside the \Group{} directive. -Within the \Group{} directive, the argument \texttt{\#1} contains the +Within the \Group{} directive, the argument \argone{} contains the collective tokens of the group (absent the outer cat-1,2 grouping). -The default directive \macname{processtoks\{\#1\}} will recommit the +The default directive \macname{processtoks\{\argone\}} will recommit the group's tokens individually to be processed through the token cycle. -In contrast, the command \macname{addcytoks\{\#1\}} in the \Group{} +In contrast, the command \macname{addcytoks\{\argone\}} in the \Group{} directive would directly add the collective group of tokens to the \macname{cytoks} register, without being processed individually by \TokCycle. +\subsubsection{The \macname{groupedcytoks} macro for more sophisticated + group processing} +\label{s:grpcy} + +\begin{sloppypar} +To this point, there have been presented two options for handling grouped + material in the token cycle: process each of the grouped tokens (via + \macname{processtoks}) or echo the grouped tokens (via + \macname{addcytoks}). Obviously, one may also discard the grouped + tokens altogether by employing neither of the above choices in the + context of the \Group{} directive. +\end{sloppypar} + +However, there are times where you might wish to perform a macro task + (for example, one that might add additional tokens to \macname{cytoks}) + immediately \textit{outside} of the group, before or after. +If you were to add the task macro before or after the \macname{processtoks} + invocation, it will still be performed, by default, \textit{inside} + the \macname{cytoks} group. + +The macro \macname{groupedcytoks} allows one to manually specify the + grouping duties of the \Group{} directive. +Thus, the following two group-processing configurations are + functionally identical: the default + +\begin{verbbox}[\vbdelim] + \stripgroupingfalse + \Groupdirective{\processtoks{#1}} +\end{verbbox} +\thevbox +and + +\begin{verbbox}[\vbdelim] + \stripgroupingtrue + \Groupdirective{\groupedcytoks{\processtoks{#1}}} +\end{verbbox} +\thevbox +Therefore, unless the goal is to introduce additional nesting levels + into the input stream, the first rule of using \macname{groupedcytoks} + is to \macname{stripgroupingtrue} before entering the token cycle. +The argument of \macname{groupedcytoks} specifies the tasks that are to + occur inside a \macname{cytoks} grouping. +So, in the following example + +\begin{verbbox}[\vbdelim] + \stripgroupingtrue + \Groupdirective{\taskA + \groupedcytoks{\taskB\processtoks{#1}\taskC}% + \taskD} +\end{verbbox} +\thevbox +the tasks A and D will occur \textit{outside} of the \macname{cytoks} + group. +Tasks B and C occur inside the group, immediately before and after processing + the tokens of the group, respectively. +Thus, the use of \macname{groupedcytoks} in this way permits \TokCycle{} + taskings to occur outside of the grouping applied to the output. + +If \macname{stripgroupingtrue} had been omitted while still using + \macname{groupedcytoks}, tasks A and D would have been in the + explicit group, and tasks B and C, along with the grouped tokens + \argone{}, would have been nested within an additional group. + \subsubsection{Implicit grouping tokens: \macname{stripimplicitgroupingcase}} +\label{s:impgrp} Implicit grouping tokens (e.g., \macname{bgroup} \& \macname{egroup}) can be handled in one of \textit{three} separate ways. @@ -370,6 +518,139 @@ Finally, the special case of $-$1 indicates that the implicit-group Such a special treatment has limited application---for example, when the intent is to detokenize these tokens. +\subsection{Truncating the input stream} + +The basic process of \TokCycle{} is one in which an input stream (or argument) + of tokens is examined and processed based on directives + set up by the user. +Typically, the input stream is processed token-by-token to exhaustion, which + occurs when a defined terminating token is reached. +In section~\ref{s:escape}, we will see how a sequence of tokens can be exempted + from any processing that would otherwise occur in the directive, + and be passed instead directly to the output stream. + +However, in this section, we will examine how \TokCycle{} can be directed + to \textit{dynamically} truncate, that is, discard the remaining + input stream based on the tokens found therein. +Truncation commands may be issued within the \Character, \Macro, + and \Space{} directives. +More will be said about the \Group{} directive in section~\ref{s:truncingrp}. + +Truncation can be made to apply either to the remainder of the current + \TokCycle-nesting level (corresponding to an explicit catcode-1,2 + group in the input stream)\footnote{ +Note that implicit groups ({\ttfamily\macname{bgroup}...\macname{egroup}}) + do not create a new \TokCycle{} nesting level, nor do + {\ttfamily\macname{begingroup}...\macname{endgroup}}. +Only explicit catcode-1,2 groups, in the nature of {\ttfamily\{...\}}, + create a nested \TokCycle{} level. +The current \TokCycle{} nesting level may be obtained from the \TeX{} count + \macname{tcdepth}, which starts at zero upon \TokCycle{} entry. +This count is incremented and decremented as groups are entered and + exited in the input stream. +It may be examined within the \CGMS{} directives in order to + guide decisions. +} + or for the remainder of the total \TokCycle{} + input stream. + +\subsubsection{\macname{truncategroup} and \macname{truncategroupiftokis}} +\label{s:truncgrp} + +As to truncating the input stream for the remainder of the \TokCycle{} + nesting level, the command is simply \macname{truncategroup}. +So, for example, the directive + +\begin{verbbox}[\vbdelim] + \Spacedirective{\truncategroup} +\end{verbbox} +\thevbox +would direct \TokCycle{}, if it ever finds a (catcode-10) space token + in the input stream, to discard the remainder of the tokens within + the group in which that space was initially found. + +More often, however, one would desire to issue the truncation + \textit{conditionally}. +One could use one of the conditional commands of section~\ref{s:misc}, + for example, + +\begin{verbbox}[\vbdelim] +\tctestifcon{<condition>}{\truncategroup}{<code if condition not met>} +\end{verbbox} +\thevbox +Alternatively, if the \textit{condition} is the occurrence of a particular + single token in the input stream, one may use an abbreviated syntax: + +\begin{verbbox}[\vbdelim] +\truncategroupiftokis{<tok>}{<code if condition not met>} +\end{verbbox} +\thevbox +Thus, an example to echo macro tokens to the output stream, unless + a \macname{relax} token is found, in which case terminate the group, + would be + +\begin{verbbox}[\vbdelim] +\Macrodirective{\truncategroupiftokis{\relax}{\addcytoks{#1}}} +\end{verbbox} +\thevbox +Obviously, checks for particular character tokens would be placed in the + \Character{} directive, rather than the \Macro{} directive. + +\subsubsection{\macname{truncatecycle} and \macname{truncatecycleiftokis}} + +There are two commands that are in every way analogous to the + group-\hspace{0pt}terminating macros of section~\ref{s:truncgrp}. +These commands are \macname{truncatecycle} and + \macname{truncatecycleiftokis}. +But in this case, all tokens are discarded to the end of the input + stream, not just the current group (i.e., \TokCycle-nesting level). +One point to note, however, is that open group levels will be + closed by the \macname{truncatecycle} command, so that the tokens, + both those being executed as well as those retained in + \macname{cytoks}, will be group-balanced. + +\subsubsection{Truncating from within the \Group{} Directive} +\label{s:truncingrp} + +The truncate commands described in prior sections may \textit{not} + be executed from the \Group{} directive, as has already been + mentioned. +What does termination even mean in the \macname{Groupdirective} + context? +The \Group{} directive is executed whenever an explicit catcode-1,2 + group is encountered in the input stream. +The argument, \argone, to that directive will be the complete + contents of the group (sans the catcode-1,2 delimiters). + +So, to truncate the group from the outset, not doing anything with + \argone{} will accomplish the desired result. +For example, to truncate all group-level-3 tokens, you could say, + using the conditional commands of section~\ref{s:misc}, + +\begin{verbbox}[\vbdelim] +\Groupdirective{\tctestifnum{\tcdepth=3}{}{\processtoks{#1}}} +\end{verbbox} +\thevbox +There is no need to use an explicit \macname{truncategroup} macro + to accomplish it. + +On the other hand, to truncate the complete token cycle if, for example, + group-level-3 tokens are encountered, you have to set some sort of + flag that is immediately picked up by one of the other directives. +Here is an example of how that can be done: + +\begin{verbbox}[\vbdelim] +\Groupdirective{\tctestifnum{\tcdepth=3}{\processtoks{\truncatenow}}% + {\processtoks{#1}}} +\Macrodirective{\truncatecycleiftokis{\truncatenow}{\addcytoks{#1}}} +\end{verbbox} +\thevbox +Note that, as long as \macname{stripgroupingfalse} is active, the above + examples will result in empty, rather than absent, level-3 groups, + since the group open occurs before the truncation is processed. +If the presence of even an empty group is to be avoided in such a case, + one may employ the \macname{groupedcytoks} techniques described in + section~\ref{s:grpcy}. \subsection{Escaping content from \TokCycle{} processing\label{s:escape}} @@ -474,16 +755,14 @@ For example, in the following code, \texttt{Q} is made both active and implicit: \catcode`Q=\active \let Qx \end{verbbox} -\medskip\theverbbox\medskip\noindent - -\noindent +\thevbox In general, both \macname{ifactivetok} and \macname{ifimplicittok} tests can be performed together to determine such cases. This is true even in the case of active-implicit catcode-10 spaces, which - are always processed through the Space directive. + are always processed through the \Space{} directive. However, because of the process \TokCycle{} uses in digesting space - tokens, the actual token passed to the Space directive (so-called + tokens, the actual token passed to the \Space{} directive (so-called \verb|#1|), when an active-implicit space is encountered in the input stream, is a generic implicit space token named \macname{tc@sptoken}. However, the catcode-12 version of the active character that @@ -495,14 +774,15 @@ Such an example is described in the \texttt{tokcycle-examples} \subsubsection{Parameter (cat-6) tokens (e.g., \texttt{\#}): \macname{ifcatSIX}} +\label{s:cat6} Typically, category-code 6 tokens (like \texttt{\#}) are used to - designate a parameter (e.g., \texttt{\#1}--\texttt{\#9}). + designate a parameter (e.g., \argone--\texttt{\#9}). Since they are unlikely to be used in that capacity inside a tokcycle input stream, the package behavior is to convert them into something cat-12 and set the \textit{if}-condition \macname{catSIXtrue}. In this manner, \macname{ifcatSIX} can be used inside the - Character directive to trap and convert cat-6 tokens into something of + \Character{} directive to trap and convert cat-6 tokens into something of the user's choosing. \begin{sloppypar} @@ -515,8 +795,9 @@ On the other hand, implicit cat-6 macros (e.g., \verb|\let\myhash#|) implicit-macro name. \end{sloppypar} -\subsubsection{Parameters (\texttt{\#1}--\texttt{\#9}): +\subsubsection{Parameters (\argone--\texttt{\#9}): \macname{whennotprocessingparameter}} +\label{s:whennot} While, generally, one would not intend to use parameters in the \TokCycle{} input stream, the package provides, not a flag, @@ -528,17 +809,18 @@ The macro is to be used \textit{within} the \Character{} directive \whennotprocessingparameter#1{<non-parameter-processing-directive>}\end{verbbox} \theverbbox\smallskip -\noindent Here, the \texttt{\#1} doesn't refer to \texttt{\#1} - as it appears in the input stream, but to the sole parameter of +\noindent Here, the \argone{} doesn't refer to \argone{} as + it might appear in the input stream, but to the sole parameter of the \Character{} directive, referring to the current token being processed. With this syntax, when the token under consideration is a - parameter (e.g., \texttt{\#1}--\texttt{\#9}), that parameter is + parameter (e.g., \argone--\texttt{\#9}), that parameter is added to the \macname{cytoks} register. If the token under consideration is not a parameter, the code in the final argument is executed. \subsection{Misc: general \textit{if}-condition tools} +\label{s:misc} \TeX{} comes equipped with a variety of \macname{if...} condition primitives. @@ -547,34 +829,53 @@ When dealing with macros for which the order of expansion is important, of proper expansion and execution. These four restructured \textit{if} macros came in handy writing the package, and may be of use in creating your directives, - without \macname{else} and \macname{fi} getting in the way:\medskip + preventing \macname{else} and \macname{fi} from getting in the + way:\medskip \begin{verbbox}[\vbdelim] \tctestifcon{<TeX-if-condition>}{<true-code>}{<false-code>} \tctestifx{<\ifx-comparison-toks>}{<true-code>}{<false-code>} \tctestifnum{<\ifnum-condition>}{<true-code>}{<false-code>} -\tctestifcatnx<\ifcat\noexpand-comparison-toks>{<true-code>}{<false-code>} +\tctestifcatnx<tok1><tok2>{<true-code>}{<false-code>} \end{verbbox} \theverbbox +\medskip\noindent +Note that unlike those macros requiring three arguments, the lone + exception is \macname{tctestifcatnx}, which requires four, comparing the + catcodes of the two input toks \textbf{without expansion} (\texttt{nx} + denotes \macname{noexpand}). -\subsection{\TokCycle{} macro close-out: \macname{aftertokcycle}} +\subsection{\TokCycle{} Completion: \macname{aftertokcycle} + and \macname{tcendgroup}} -The \TokCycle{} macros, upon completion, do nothing. -Unlike \macname{tokencycle} environments, they don't even +The \TokCycle{} \textit{macros}, upon completion, do nothing. +Unlike \macname{tokencycle} \textit{environments}, they don't even output \macname{the}\macname{cytoks} token register. A command has been provided, \macname{aftertokcycle}, which takes an argument to denote the actions to be executed following - completion of \textit{all} subsequent \TokCycle{} invocations. -It might be as simple as \verb|\aftertokcycle{\the\cytoks}|, to - have the output stream automatically typeset. + completion of \textit{all} subsequent \TokCycle{} \textit{macro} + invocations within the scope of the current group. +It might be specified as simply as \verb|\aftertokcycle{\the\cytoks}|, + so as to have the output stream automatically typeset by + future invocations of the \macname{tokcycle} macro. The meaning of \macname{aftertokcycle} can be reset with \macname{aftertokcycle\{\}}, but is also reset as part of \macname{resettokcycle}. Unlike macros, the \TokCycle{} environments are unaffected by \macname{aftertokcycle}, as they actually set it locally - (within their scope) to accomplish their own purposes. + (within their grouped scope) to accomplish their own purposes. + +Speaking of the \TokCycle{} environments, there is a command employed by + the environments \macname{tokencycle} and \macname{tokencyclexpress} + that is likewise available to the user: \macname{tcendgroup}. +This is a version of \macname{endgroup} that additionally transmits the + current value of the \macname{cytoks} token list \textit{outside of + the group}. +So if a user wishes to confine certain \TokCycle{} activity inside a + group, but wishes to have access to the \macname{cytoks} result afterward, + he may simply wrap the activity in \verb|\begingroup...\tcendgroup|. \subsection{Accommodating catcode-1,2 changes: \macname{settcGrouping}% \label{s:grouping}} @@ -631,17 +932,36 @@ If I can't fix them, I can at least disclaim and declaim them below: be detected), even if the package is not notified. See section~\ref{s:grouping}. -\item Should one need to keep track of the \textit{names} of implicit tokens, - then only one named implicit cat-6 token (e.g., \verb|\let\svhash#|) - may appear in the input - stream (though it can appear multiple times). -There is no limit on explicit cat-6 toks. -This limitation occurs because implicit cat-6 tokens are converted - into the fixed-name implicit macro \macname{implicitsixtok} which - contains the \macname{string} of the most recently used - implicit cat-6 token name. -In any event, all cat-6 tokens are trapped and flag \texttt{true} - the \macname{ifcatSIX} condition. +\item Should one need to keep track of the \textit{names} of implicit + cat-6 tokens (e.g., \verb|\svhash| following \verb|\let\svhash#|), + care must be exercised. +When encountered in the input stream, a cat-12 \macname{string} of + the implicit-cat-6 token \textit{name} is stored in the \macname{implicitsixtok} + macro. +It is the \macname{implicitsixtok} token which is passed to the + \Character{} directive as \verb|#1|, in lieu of the actual cat-6 token. +If there is at most a single implicit-cat-6 named token in the input stream, + then \macname{implicitsixtok} will reflect its name. +However, if there is more than one, \macname{implicitsixtok} will be redefined + on the fly with each occurrence. + +Therefore, should one intend to add more than one such token \textit{name} + to the token list by way of \macname{addcytoks}, \textit{it must be expanded + first} (e.g., \macname{addcytoks[1]\{\argone\}}), in order to capture the + \textit{name} of that particular cat-6 implicit token, rather than the name + of the last one in the input stream. +There is no similar problem for explicit cat-6 tokens. +In any event, all cat-6 tokens are trapped and flag as \texttt{true} + the \macname{ifcatSIX} condition, which is how the user should detect + their presence. + +\item A similar warning is to be made for active-implicit spaces as was + made for implicit-cat-6 tokens. +In the case of active-implicit spaces, the name of the active character is + momentarily stored in the \macname{theactivespace} macro. +If one wishes to output the active \textit{name} to the \macname{cytoks} token list, + one must make sure that \macname{theactivespace} is expanded once before + being added to the token list. \end{itemize} @@ -660,10 +980,13 @@ The \texttt{tex.stackexchange} site provides a wonderful opportunity to interact with the leading developers and practitioners of \TeX{} and \LaTeX{}. +\vspace{1in} \section*{Source Code} +%\vspace{-0.1in} \Large\texttt{tokcycle.sty} -\verbfilenobox[\footnotesize]{tokcycle.sty}\vspace{0.5in} +\verbfilenobox[\footnotesize]{tokcycle.sty}% +%\vspace{-0.1in} \noindent\texttt{tokcycle.tex} \verbfilenobox[\footnotesize]{tokcycle.tex} diff --git a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf Binary files differindex 0c1f565132d..b5604b382ae 100644 --- a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf +++ b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf diff --git a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex index 49fbe6c66ce..891741bd4be 100644 --- a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex +++ b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex @@ -1,7 +1,7 @@ \documentclass[10pt]{article} \usepackage[margin=1.3in]{geometry} \usepackage[T1]{fontenc} -\usepackage{tokcycle,xcolor,manfnt} +\usepackage{tokcycle,xcolor,manfnt,lmodern} \colorlet{bred}{red!90!blue} \usepackage{tikz} % MACROS: REDEFINE TO ALLOW \cytoks TO ESCAPE SCOPE OF exampleA @@ -25,7 +25,6 @@ sidebyside, title=#2,#1} - \newtcblisting{exampleB}[2][]{% colframe=red!50!yellow!50!black, colback=red!50!yellow!5!white,%white, @@ -49,7 +48,8 @@ \def\altdecytoks{\par\medskip\begingroup\noindent\macname{cytoks} \ \textit{alt}detokenization:\\\raggedright\footnotesize\ttfamily - \expandafter\altdetokenize\expandafter{\the\cytoks}\par\endgroup} + \expandafter\altdetokenize\expandafter{\the\cytoks}\normalsize\par\endgroup} + \newcommand\TokCycle{\textsf{tokcycle}} \parindent0pt @@ -78,9 +78,10 @@ \newif\ifmacro \newcommand\altdetokenize[1]{\begingroup\stripgroupingtrue\macrofalse + \stripimplicitgroupingcase{-1}% \tokcycle {\ifmacro\def\tmp{##1}\ifcat\tmp A\else\unskip\allowbreak\fi\macrofalse\fi - \detokenize{##1}} + \detokenize{##1}\ifx##1\bgroup\unskip\fi\ifx##1\egroup\unskip\fi} {\ifmacro\unskip\macrofalse\fi\{\processtoks{##1}\ifmacro\unskip\fi\}\allowbreak} {\tctestifx{\\##1}{\\}{\ifmacro\unskip\allowbreak\fi \allowbreak\detokenize{##1}\macrotrue}} @@ -152,7 +153,7 @@ {\addcytoks{##1}} \newcommand*\tcmapto[2]{\expandafter\def\csname tcmapto#1\endcsname{#2}} \newcommand*\tcremap[1]{\ifcsname tcmapto#1\endcsname - \csname tcmapto#1\endcsname\else#1\fi} + \csname tcmapto#1\expandafter\endcsname\else\expandafter#1\fi} \tcmapto am \tcmapto bf \tcmapto cz \tcmapto de \tcmapto ey \tcmapto fl \tcmapto gx \tcmapto hb \tcmapto ic \tcmapto jn \tcmapto ki \tcmapto lr \tcmapto mh \tcmapto nt \tcmapto ok @@ -160,6 +161,12 @@ \tcmapto uj \tcmapto vp \tcmapto wd \tcmapto xg \tcmapto yu \tcmapto zv +\tokcycleenvironment\greek% NEW TOKCYCLE ENVIRONMENT: +{\addcytoks[4]{\tcremap{##1}}}% HOW TO TREAT CHARACTERS +{\processtoks{##1}}% HOW TO TREAT GROUP CONTENTS +{\addcytoks{##1}}% HOW TO TREAT MACROS +{\addcytoks{##1}}% HOW TO TREAT SPACES + \newcommand\findinstring[2]{\begingroup% \stripgroupingtrue \setcounter{runcount}{0}% @@ -197,6 +204,72 @@ \newcommand\detokenizespacetok[1]{\tctestifcon{\ifimplicittok}% {\tctestifcon{\ifactivetok}{\theactivespace}{\string#1}}{\detokenize{#1}}} +\newcounter{sigd} +\newcounter{trailingz} +\newcommand\sigdigits[1]{% + \setcounter{sigd}{0}% SIGNIFICANT DIGIT COUNT + \setcounter{trailingz}{0}% TRAILING ZERO COUNT + \def\founddot{F}% MADE T WHEN DECIMAL HAS BEEN 1ST LOCATED + \def\zerosig{F}% MADE T WHEN ZERO BECOMES SIGNIFICANT + \tokcycle% CYCLE THROOUGH EACH TOKEN + {\tctestifx{.##1}% + {\if T\founddot\throwerr\fi\def\founddot{T}}% IF . + {\tctestifnum{`##1>`/}% + {\tctestifnum{`##1<`:}% + {\tctestifnum{##1>0}% + {\stepcounter{sigd}\def\zerosig{T}\setcounter{trailingz}{0}}% IF 1-9 + {\if T\zerosig\stepcounter{sigd}\fi\stepcounter{trailingz}}% IF 0 + }% + {\throwerr}% IF ASCII > `9 + }% + {\throwerr}% IF ASCII < `0 + }% + }% APPLY ABOVE LOGIC FOR CHAR TOKENS + {\throwerr}% IF BRACES + {\throwerr}% IF CONTROL SEQUENCE + {}% IGNORE SPACES + {#1}% THE ARGUMENT + \addtocounter{sigd}{-\thetrailingz}% SUBTRACT TRAILING ZEROS + \expandafter\ifx\expandafter\relax\detokenize\expandafter{\the\cytoks}% + \relax\else\setcounter{sigd}{-1}\fi% CHECK FOR INVALID INPUT + \thesigd +} +\newcommand\throwerr{\addcytoks{##1}}% ADD INVALID TOKS TO \cytoks + +\newcounter{Tcount} +\xtokcycleenvironment\capTive% ENVIRONMENT NAME +{\tctestifx{t##1}{\stepcounter{Tcount}\addcytoks{T}}{\addcytoks{##1}}}% CHAR. DIRECTIVE +{\processtoks{##1}}% GROUP DIRECTIVE +{\addcytoks{##1}}% MACRO DIRECTIVE +{\addcytoks{##1}}% SPACE DIRECTIVE +{\setcounter{Tcount}{0}\color{red}\tcafterenv{\ Here}}% INITIALIZATION CODE +{\theTcount{} t's capitalized: \addcytoks{!!}}% CLOSE-OUT CODE + +\newcounter{wordcount} +\newcounter{lettercount} +\newif\ifinword +\newif\ifrunningcount +\newcommand\dumpword{% + \addcytoks[1]{\accumword}% + \tctestifcon\ifinword{\stepcounter{wordcount}% + \ifrunningcount\addcytoks[x]{$^{\thewordcount,\thelettercount}$}\fi}{}% + \inwordfalse\def\accumword{}} +\makeatletter +\newcommand\addletter[1]{% + \tctestifcatnx A#1{\stepcounter{lettercount}\inwordtrue}{\dumpword}% + \tc@defx\accumword{\accumword#1}} +\makeatother + +\xtokcycleenvironment\countem + {\addletter{##1}} + {\processtoks{##1}\dumpword} + {\dumpword\addcytoks{##1}} + {\dumpword\addcytoks{##1}} + {\def\accumword{}\setcounter{wordcount}{0}\setcounter{lettercount}{0}} + {\dumpword\ifrunningcount\else\tcafterenv{\par(Wordcount=\thewordcount, + Lettercount=\thelettercount)}\fi} + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Examples, examples, and more examples} @@ -240,11 +313,13 @@ Default escape character is $|$. One can change it with \begingroup -\begin{exampleC}[]{The unexpandable \macname{plusl} macro} +\begin{exampleC}[]{The unexpandable \macname{plusl} macro to increase + the ASCII value of tokens by 1} \newcommand\plusl[1]{\char\numexpr`#1+1\relax} \end{exampleC} -\begin{exampleA}[lefthand width=5cm]{Escaping text in the input stream} +\begin{exampleA}[lefthand width=5cm]{Escaping text and macro parameters + in the input stream} \tokcycle {\addcytoks{\plusl{#1}}} {\processtoks{#1}} @@ -265,9 +340,9 @@ Default escape character is $|$. One can change it with \begingroup This section concerns the issue of whether the characters of the input stream are transformed before or after being placed in the - output token register (\macname{cytoks}).\medskip + output token list (\macname{cytoks}). -\noindent\begin{minipage}[t]{\textwidth} +\medskip Transform characters (+1 ASCII) via unexpandable macro:\medskip \begin{exampleA}[lefthand width=8.3cm] @@ -283,8 +358,11 @@ Transform characters (+1 ASCII) via unexpandable macro:\medskip \altdecytoks +Note how the above detokenization shows the actual transformation + instructions involving \macname{plusl}. +This is done because \macname{plusl} is not expandable. + \par\noindent\hrulefill\medskip -\end{minipage} \endgroup \noindent\begin{minipage}[t]{\textwidth} @@ -316,6 +394,12 @@ Capitalize vowels (but don't expand the character directive) \altdecytoks +\medskip +The above detokenization also shows the transformation macro, + \macname{vowelcap}, as part of \macname{cytoks}. +But because \macname{vowelcap} is expandable, we did not need to + leave it at this stage, as the next example shows. + \par\noindent\hrulefill\medskip \endgroup \end{minipage} @@ -336,6 +420,9 @@ Capitalize vowels (expanding the character directive) \altdecytoks +\medskip +Here, we see no vestiges of \macname{vowelcap} in \macname{cytoks}, + but only the transformed result. \endgroup \subsubsection{Unexpanded vs. pre-expanded input stream} @@ -359,6 +446,10 @@ Capitalize vowels (expanding the character directive) \altdecytoks +\medskip +In the above case, vowel-capping is not applied to the date, because + \macname{today} remains unexpanded in the \macname{cytoks} token list. + \noindent\hrulefill\strut \end{minipage} @@ -377,6 +468,14 @@ Note that, when pre-expanding the input stream, one must \altdecytoks +\medskip +Here, we see the date subjected to vowel-capping, because the \TokCycle{} + input was expanded prior to the token cycle. +Thus \macname{today} no longer appears \textit{explicitly} in the + \macname{cytoks} token list. +Its pre-expansion into character tokens is now subject to the + \macname{vowelcap}. + \endgroup \subsection{Grouping} @@ -442,7 +541,7 @@ These are embodied in the if-condition \macname{ifstripgrouping} Regardless of which condition is set, the tokens within the explicit group are still passed to the Group directive for processing. - +\medskip Permutations of the following code are used in the subsequent examples. Group stripping, brought about by \macname{stripgroupingtrue}, involves removing the grouping braces from around the group. @@ -451,6 +550,7 @@ The choice of \macname{processtoks} vs.\@ \macname{addcytoks} to \TokCycle{} for processing, or are merely sent to the output register in their original unprocessed form. +\medskip Note that, in these examples, underdots and visible spaces will only appear on characters and spaces that have been directed to the Character and Space directives, respectively. @@ -547,15 +647,73 @@ Note that the content of groups can be altogether eliminated if {1{{3{{5{{7{{9{1{}0}}8}}6}}4}}2}}} \the\cytoks \end{exampleA} +Note, above, that \TokCycle{} does not rely on an argument-based nesting + algorithm that would otherwise be limited to 9 embedded groups. +\endgroup + +\subsubsection{The use of \macname{groupedcytoks} for more sophisticated + group processing} + +\begingroup + +\begin{sloppypar} +By default, \TokCycle{} processes groups such that whatever additions to the + \macname{cytoks} token list you accomplish in the Group directive will be + encased in an explicit grouped scope. +This is the token-cycle behavior with the default \macname{stripgroupingfalse} + declaration. +However, for more sophisticated tasks associated with the processing of + group content, one may override the default (by way of + \macname{stripgroupingtrue}) and manually dictate the groupings associated + with the Group directive, with the use of \macname{groupedcytoks\{...\}}. +\end{sloppypar} + +\medskip In the following example, tasks A and D are performed \textit{outside} + of the explicit grouping otherwise introduced with \macname{groupedcytoks}, + whereas tasks B and C fall inside of that explicit group. +The fact that, in the output, \texttt{d} is not italic confirms the + fact that task D occurs after the conclusion of the explicit \macname{cytoks} + group that encompasses the italic `bxc'. +The fact that none of \texttt{a}, \texttt{b}, \texttt{c}, nor \texttt{d} are + encased in an \macname{fbox} indicates that they were not processed through + the Character directive, but arose from direct additions to \macname{cytoks}, + by way of their tasks. + +\begin{exampleA}[lefthand width=10cm]{Use of the \macname{groupedcytoks} + macro} +\stripgroupingtrue +\Characterdirective{\addcytoks{\fbox{#1}}} +\Groupdirective{\taskA% + \groupedcytoks{\taskB\processtoks{#1}\taskC}% + \taskD} +\def\taskA{\addcytoks{\bgroup\LARGE a}} +\def\taskB{\addcytoks{\itshape b}} +\def\taskC{\addcytoks{c}} +\def\taskD{\addcytoks{d\egroup}} + +\tokcyclexpress{x\textbf{x}x} +\the\cytoks +\end{exampleA} + +The final disposition of the \macname{cytoks} token list shows the effects of + \macname{stripgroupingtrue} and \macname{groupedcytoks} more explicitly: + +\altdecytoks + +\medskip +Without \macname{stripgroupingtrue}, all tasks, A,B,C, and D, would have + been contained within the explicit \texttt{\{...\}} group. \endgroup + \subsection{Direct use of \TokCycle} -\TokCycle{} (in regular or \texttt{xpress} form) may be invoked +The \TokCycle{} macros and pseudo-environments + (in regular or \texttt{xpress} form) may be invoked directly from the document, without - being first encapsulated within a macro or environment. + being first encapsulated within a separate macro or environment. \subsubsection{Modifying counters as part of the Character directive} @@ -563,7 +721,7 @@ Note that the content of groups can be altogether eliminated if \begingroup \begin{exampleA}[lefthand width=8.5cm]{Using a period token (.) to - reset a changing color} + reset a changing color every sentence} \restorecolor \tokencycle {\addcytoks{\bgroup\reducecolor{3}#1\egroup}% @@ -610,6 +768,9 @@ is long.} Now, it is short. Back to regular text. \end{exampleA} +Note that embedded macros such as \macname{textit} and \macname{textbf} + do not adversely affect the capability of the \macname{spaceouttext} + macro. \endgroup @@ -623,7 +784,7 @@ It is \textbf{not} to be confused as a replacement for In certain applications, it may offer a more pleasingly formatted typesetting of detokenized material. -It is an unusual application of \TokCycle{} in that it does not +\medskip It is an unusual application of \TokCycle{} in that it does not actually use the \macname{cytoks} token register to collect its output. This is only possible because all macros in the input stream @@ -634,9 +795,10 @@ This is only possible because all macros in the input stream \begin{exampleC}[]{The \macname{altdetokenize }\ macro} \newif\ifmacro \newcommand\altdetokenize[1]{\begingroup\stripgroupingtrue\macrofalse + \stripimplicitgroupingcase{-1}% \tokcycle {\ifmacro\def\tmp{##1}\ifcat\tmp A\else\unskip\allowbreak\fi\macrofalse\fi - \detokenize{##1}} + \detokenize{##1}\ifx##1\bgroup\unskip\fi\ifx##1\egroup\unskip\fi} {\ifmacro\unskip\macrofalse\fi\{\processtoks{##1}\ifmacro\unskip\fi\}\allowbreak} {\tctestifx{\\##1}{\\}{\ifmacro\unskip\allowbreak\fi \allowbreak\detokenize{##1}\macrotrue}} @@ -652,7 +814,7 @@ This is only possible because all macros in the input stream \string\altdetokenize: \\ \texttt{\altdetokenize{a\mac a \mac2 {\mac}\mac{a\mac\mac}\mac}}! -\string\detokenize: \\ +versus \string\detokenize: \\ \texttt{\detokenize{a\mac a \mac2 {\mac}\mac{a\mac\mac}\mac}}! \end{exampleA} \endgroup @@ -786,12 +948,92 @@ This is only possible because all macros in the input stream \endgroup +\subsubsection{Counting the significant digits in a number} + +\begingroup -\subsection{\TokCycle-based environments} +The problem of counting the significant digits in a number is an interesting + one, in that leading and trailing zeros are not significant, but zeros + in the middle of the number are significant. +In a problem like this, the \macname{cytoks} token list is not used to + carry a transformation of the input tokens, but only to carry a record + of errant tokens from the input (as seen from the helper macro + \macname{throwerr}). +If \macname{the\textbackslash cytoks} is anything other than empty when the + macro completes, the macro will set the significant-digit counter + \texttt{sigd} to a value of -1 to signify an error condition. + +\medskip +The macro expects its input to comprise digits in the 0--9 range, an [optional] + single decimal point, and ignorable spaces. Any other token will throw + an error. + +\medskip +The main work of the \macname{sigdigits} macro is to use the counters + \texttt{sigd} to count significant digits and \texttt{trailingz} to + tabulate the number of trailing-zeros with which to decrement + \texttt{sigd}. + +\begin{exampleC}[]{The \macname{sigdigits} macro for counting significant digits} +\newcounter{sigd} +\newcounter{trailingz} +\newcommand\sigdigits[1]{% + \setcounter{sigd}{0}% SIGNIFICANT DIGIT COUNT + \setcounter{trailingz}{0}% TRAILING ZERO COUNT + \def\founddot{F}% MADE T WHEN DECIMAL HAS BEEN 1ST LOCATED + \def\zerosig{F}% MADE T WHEN ZERO BECOMES SIGNIFICANT + \tokcycle% CYCLE THROOUGH EACH TOKEN + {\tctestifx{.##1}% + {\if T\founddot\throwerr\fi\def\founddot{T}}% IF . + {\tctestifnum{`##1>`/}% + {\tctestifnum{`##1<`:}% + {\tctestifnum{##1>0}% + {\stepcounter{sigd}\def\zerosig{T}\setcounter{trailingz}{0}}% IF 1-9 + {\if T\zerosig\stepcounter{sigd}\fi\stepcounter{trailingz}}% IF 0 + }% + {\throwerr}% IF ASCII > `9 + }% + {\throwerr}% IF ASCII < `0 + }% + }% APPLY ABOVE LOGIC FOR CHAR TOKENS + {\throwerr}% IF BRACES + {\throwerr}% IF CONTROL SEQUENCE + {}% IGNORE SPACES + {#1}% THE ARGUMENT + \addtocounter{sigd}{-\thetrailingz}% SUBTRACT TRAILING ZEROS + \expandafter\ifx\expandafter\relax\detokenize\expandafter{\the\cytoks}% + \relax\else\setcounter{sigd}{-1}\fi% CHECK FOR INVALID INPUT + \thesigd +} +\newcommand\throwerr{\addcytoks{##1}}% ADD INVALID TOKS TO \cytoks +\end{exampleC} + +\begin{exampleA}[lefthand width=8.0cm]{Example use of the \macname{sigdigits} + macro} +1) \sigdigits{12.3}\\ +2) \sigdigits{0.123}\\ +3) \sigdigits{1.2.3}\\ +4) \sigdigits{1.3\today}\\ +5) \sigdigits{321.345 678}\\ +6) \sigdigits{000321.305}\\ +7) \sigdigits{.000 300 345}\\ +8) \sigdigits{0003x1.345}\\ +9) \sigdigits{123000}\\ +A) \sigdigits{123.000}\\ +B) \sigdigits{0003;345} +\end{exampleA} + +\endgroup + + +\subsection{Creating your own \TokCycle-based environments} The \macname{tokcycleenvironment} macro allows users to define their own \TokCycle{} environments. -Here are some examples. +The more recently introduced \macname{xtokcycleenvironment} macro extends + that capability to allow initialization and postprocessing when defining + a named \TokCycle{} environment. +Here are some examples of both types. \subsubsection{``Removing'' spaces, but still breakable/hyphenatable} @@ -805,7 +1047,8 @@ Here are some examples. {\addcytoks{\hspace{.2pt plus .2pt minus .8pt}}}% \end{exampleC} -\begin{exampleA}[lefthand width=6.0cm]{} +\begin{exampleA}[lefthand width=6.0cm]{Example of using the \macname{spaceBgone} + environment} \spaceBgone Here we have a \textit{test} of whether the spaces are removed. @@ -834,7 +1077,7 @@ Here are some examples. {\addcytoks{##1}} \newcommand*\tcmapto[2]{\expandafter\def\csname tcmapto#1\endcsname{#2}} \newcommand*\tcremap[1]{\ifcsname tcmapto#1\endcsname - \csname tcmapto#1\endcsname\else#1\fi} + \csname tcmapto#1\expandafter\endcsname\else\expandafter#1\fi} \tcmapto am \tcmapto bf \tcmapto cz \tcmapto de \tcmapto ey \tcmapto fl \tcmapto gx \tcmapto hb \tcmapto ic \tcmapto jn \tcmapto ki \tcmapto lr \tcmapto mh \tcmapto nt \tcmapto ok @@ -859,11 +1102,259 @@ Because \macname{tcremap} is expandable, the original text is totally absent \endgroup \end{minipage} +\bigskip +\begingroup + +This approach can be used, for example, to temporarily remap Latin into Greek in + math mode. +In the example given below, only several of the Latin letters are remapped + for demonstration purposes, but others can be added. +Also, remember that one can use the \verb+|...|+ delimiters to escape back to + Latin within the environment. + +\begin{exampleC}[]{The \macname{greek} environment with + supporting macros} +\tokcycleenvironment\greek% NEW TOKCYCLE ENVIRONMENT: +{\addcytoks[4]{\tcremap{##1}}}% HOW TO TREAT CHARACTERS +{\processtoks{##1}}% HOW TO TREAT GROUP CONTENTS +{\addcytoks{##1}}% HOW TO TREAT MACROS +{\addcytoks{##1}}% HOW TO TREAT SPACES +\newcommand*\tcmapto[2]{\expandafter\def\csname tcmapto#1\endcsname{#2}} +\newcommand*\tcremap[1]{\ifcsname tcmapto#1\endcsname + \csname tcmapto#1\expandafter\endcsname\else\expandafter#1\fi} +\tcmapto a\alpha \tcmapto b\beta \tcmapto e\epsilon +\tcmapto g\gamma \tcmapto p\pi +\end{exampleC} + +\tcmapto a\alpha \tcmapto b\beta \tcmapto e\epsilon +\tcmapto g\gamma \tcmapto p\pi + +\begin{exampleA}[lefthand width=8.5cm]{Demo of \macname{greek}} +\[ + y = + \greek a^2b\frac{2e}{2 + p^|p| + g}\endgreek + + x +\] +\end{exampleA} + +\endgroup + +\subsubsection{Creating an ``extended'' \TokCycle{} environment} + +\begingroup +The \macname{xtokcycleenvironment} macro extends the capability of + \macname{tokcycleenvironment} by adding two arguments that correspond, + respectively, to initialization code and ``close-out'' code that is run + after the token cycle is complete (but before it is typeset). +Let's say I wanted an environment that capitalizes instances of the letter + `\texttt{t}', but I want it to tell me how many + letter \texttt{t}'s are affected before typesetting the result. +Furthermore, I want it always to set the result in red and add two + exclamation points to the end of the environment output. + +\begin{exampleC}[]{The \macname{capTive} extended-environment definition} +\newcounter{Tcount} +\xtokcycleenvironment\capTive% ENVIRONMENT NAME +{\tctestifx{t##1}{\stepcounter{Tcount}\addcytoks{T}}{\addcytoks{##1}}}% CHAR. DIRECTIVE +{\processtoks{##1}}% GROUP DIRECTIVE +{\addcytoks{##1}}% MACRO DIRECTIVE +{\addcytoks{##1}}% SPACE DIRECTIVE +{\setcounter{Tcount}{0}\color{red}\tcafterenv{\ Here}}% INITIALIZATION CODE +{\theTcount{} t's capitalized: \addcytoks{!!}}% CLOSE-OUT CODE +\end{exampleC} + +\begin{exampleA}[lefthand width=7.9cm]{Demo of \macname{capTive} + environment} +Before \capTive the cytoks \footnotesize token list\endcapTive After +\end{exampleA} + +In the initialization code, the \texttt{Tcount} counter is reset and the color + of what follows is made red. +It also makes use of the specialized \macname{tcafterenv} macro that can + \textbf{only be used in \macname{xtokcycle\-environment} definitions}. +This macro indicates code to execute \textit{after} the \texttt{capTive} + environment scope is exited. +This invocation typesets the word `Here'. +We see that in the output, it occurs at normal size and in black, because + it is no longer subject to the color and fontsize that was left inside + the environment. +The \macname{tcafterenv} macro is unique in that it is an appending, rather + than a defining, macro. +Therefore, multiple invocations of it will append multiple code segments, + each of which will be executed upon environment exit. +It may be invoked in any of the directives, as well as the initialization and + close-out code segments. + +\medskip +For the close-out code, because it is executed after the token cycle is + complete, but before it is typeset, the updated value of \texttt{Tcount} + is available. +Anything typeset directly in the close-out code will occur before the + \macname{cytoks} token-list output is typeset. +Thus, the ``...t's capitalized.'' phrase is typeset before \macname{cytoks} + itself, even as \macname{theTcount} is accurate. +The \macname{addcytoks} occurring in the close-out code is placed at the + end of \macname{cytoks} and thus takes on the \macname{footnotesize} + declaration that is still in force. +\endgroup + +\begingroup +\medskip +Next, we consider a \TokCycle{} extended environment that counts words and + letters. +We try to build some features into the code so that, for example, spaces + and punctuation do not count as letters. +Further, we try to build the logic so that the following cases would + count as two words, not one: ``here...now'', ``Case(number)''. + +\medskip +We provide two modes of output: one that provides an ending summary, and + the other that keeps a running count. +As with other \TokCycle{} environments and macros, portions can be + exempted from the count (for example macro arguments) by using the + escape delimiters provided. + +\begin{exampleC}[]{The \macname{countem} extended-environment definition + and supporting macros} +\newcounter{wordcount} +\newcounter{lettercount} +\newif\ifinword +\newif\ifrunningcount +\newcommand\dumpword{% + \addcytoks[1]{\accumword}% + \tctestifcon\ifinword{\stepcounter{wordcount}% + \ifrunningcount\addcytoks[x]{$^{\thewordcount,\thelettercount}$}\fi}{}% + \inwordfalse\def\accumword{}} +\makeatletter +\newcommand\addletter[1]{% + \tctestifcatnx A#1{\stepcounter{lettercount}\inwordtrue}{\dumpword}% + \tc@defx\accumword{\accumword#1}} +\makeatother + +\xtokcycleenvironment\countem + {\addletter{##1}} + {\processtoks{##1}\dumpword} + {\dumpword\addcytoks{##1}} + {\dumpword\addcytoks{##1}} + {\def\accumword{}\setcounter{wordcount}{0}\setcounter{lettercount}{0}} + {\dumpword\ifrunningcount\else\tcafterenv{\par(Wordcount=\thewordcount, + Lettercount=\thelettercount)}\fi} +\end{exampleC} + +\begin{exampleA}[lefthand width=5.6cm]{Demo of the two modes of + \macname{countem}} +\countem +This is a test. + +But...punctuation does not count. +\endcountem + +\bigskip\runningcounttrue +\countem +This is a test. + +But...punctuation does not count. +\endcountem +\end{exampleA} + +\endgroup + +\subsection{Truncation of input} + +\begingroup + +\TokCycle{} has several facilities for truncating input, either + to the end of the current group or to the end of the input stream. +The basic commands are \macname{truncategroup} and \macname{truncatecycle}, + which can be part of the Character, Macro, or Space directives (see the + package documentation for truncation from the Group directive). + +\medskip +These commands may be issued directly or they may be placed as + one fork of a conditional test. +If the conditional test is the occurrence of a particular token in the + input stream, special conditional shortcuts have been formulated: + \macname{truncategroupiftokis} and \macname{truncatecycleiftokis}. +Both of these commands take two arguments: the first is the comparison + token against which to compare the tokens of the input stream, and the + second is the directive to follow for tokens that fail the + comparison test. +If the input stream token matches the comparitor, the relevant + truncation command is executed. + +\medskip +The description below shows the truncations to a given input text, + resulting from several different forms of truncating directives. +In the first case, the token cycle is to be truncated if a space + is found in the input stream. +Comparing to the input text, we see that the space occurs after the + word ``until'', and that is indeed where the \TokCycle{} output + ceases. + +\medskip +In the second case, the truncation is limited to the current group, + rather than the remainder of the input stream. +Again, the space following ``until'' is found, and the words + ``certain token'' are excised from the input stream until the + end-of-group is located. +The cycle continues successfully with ``-is'' until the next space + is found before ``located''. +Since this space is found in the top grouping level of the input, + the remainder of the input stream is thereafter truncated. + +\medskip +In the third case, the cycle is to be truncated in the event the + letter `a' is found in the input stream. +Examining the input text, the first `a' is in the word ``certain'' and + so the input is truncated after the string ``cert'' is digested. + +\medskip +The final case, calls for group truncation if an `a' is found in the + input stream. +As before, the first `a' is found after ``cert''. +However, in the group case, only the remaining group content, consisting + of ``ain token'', is discarded. +At group end, the cycle successfully resumes until the `a' in + ``located'' is reached. +Thus, ``-is loc'' is also added to the output stream. + +\def\thetext{Continue-\textit{until certain token}-is located.} + +\newcommand\rundemowith[1]{% + {\footnotesize\ttfamily\altdetokenize{#1}}: + \resettokcycle#1\expandafter\tokencyclexpress\thetext\endtokencyclexpress +} + +\begin{exampleB}[text only, righthand width=0.01cm, colback=white]% +{Effects of various truncation directives} +{\footnotesize\ttfamily Input text}: \expandafter\altdetokenize\expandafter + {\thetext} + +\rundemowith{\Spacedirective{\truncatecycle}} + +\rundemowith{\Spacedirective{\truncategroup}} + +\rundemowith{\Characterdirective{\truncatecycleiftokis{a}{\addcytoks{#1}}}} + +\rundemowith{\Characterdirective{\truncategroupiftokis{a}{\addcytoks{#1}}}} +\end{exampleB} + +Note how the macro function, in this case \macname{textit}, is not + disrupted by the truncation. +Even as tokens are truncated from the input stream, group closures + are retained so that the final output is group-balanced. +For example, a detokenization of the final case reveals this to be + the case: + +\altdecytoks + +\endgroup + \subsection{Advanced topics: implicit tokens and catcode changes} \subsubsection{Trap Active Characters (catcode 13)} -\begin{minipage}[t]{\textwidth} +%\begin{minipage}[t]{\textwidth} \begingroup Active characters in the \TokCycle{} input stream are processed in their original form. @@ -886,26 +1377,27 @@ For information on how to process active spaces, please refer to \begin{exampleA}[lefthand width=7.8cm]{Processing active characters} \resettokcycle \tokencyclexpress -This is a test!!\endtokencyclexpress +1. This is a test!!\endtokencyclexpress -\catcode`!=\active -\def !{?} +2. \catcode`!=\active +\def !{?}% \tokencyclexpress This is a test!!\endtokencyclexpress +3. \Characterdirective{\tctestifcon\ifactivetok - {\addcytoks{\fbox{#1-chr}}}{\addcytoks{#1}}} + {\addcytoks{\fbox{#1-chr}}}{\addcytoks{#1}}}% \Macrodirective{\tctestifcon\ifactivetok - {\addcytoks{\fbox{#1-mac}}}{\addcytoks{#1}}} + {\addcytoks{\fbox{#1-mac}}}{\addcytoks{#1}}}% \tokencyclexpress This is a test!!\endtokencyclexpress -\catcode`T=\active -\let T+ +4. \catcode`T=\active +\let T+% \tokencyclexpress This is a test!!\endtokencyclexpress -\detokenize\expandafter{\the\cytoks} +5. \detokenize\expandafter{\the\cytoks} \end{exampleA} %%% @@ -916,15 +1408,16 @@ This is a test!!\endtokencyclexpress are made before reaching \TokCycle{} processing. \svt hey are, thus, no longer detected as active, unless \macname{noexpand} is applied before the pre-expansion. -In this example, the \texttt{!} that is not \macname{noexpand}\kern.3pted - is converted to a \texttt{?} prior to reaching \TokCycle{} - processing (and thus, not detected as \macname{active}): +In this example, the `\texttt{!}' that is not \macname{noexpand}\kern.3pted + is converted to a `\texttt{?}' prior to reaching \TokCycle{} + processing (and thus, not detected as \macname{active}). +Likewise, the `\texttt{T}' is pre-converted to a `\texttt{+}': \catcode`!=\active \def !{?} \catcode`T=\active \let T+ -\begin{exampleA}[lefthand width=7.8cm]{Expanded input stream acts +\begin{exampleA}[lefthand width=7.0cm]{Expanded input stream acts upon active \macname{def}ed characters unless \macname{noexpand} is applied} \expandedtokcyclexpress{This is a test!\noexpand!} \the\cytoks\par @@ -942,7 +1435,7 @@ However, pre-tokenization does not suffer this behavior: %%% \endgroup -\end{minipage} +%\end{minipage} \begingroup \ddbend One aspect of \TeX{} to remember is that catcodes @@ -955,6 +1448,7 @@ But if the cat-13 active definition is changed prior to the execution of the \macname{def}'ed token, the revised token assignment will apply. +\medskip The following example demonstrates this concept, while showing, without changing the input in any way, that \TokCycle{} can properly digest active and implicit grouping (cat-1,2) characters: @@ -996,6 +1490,7 @@ In this manner, \macname{ifcatSIX} can be used inside the Character directive to convert cat-6 tokens into something of the user's choosing. +\medskip As to this cat-12 conversion, explicit cat-6 characters are converted into the same character with cat-12. On the other hand, implicit cat-6 control sequences (e.g., \verb|\let\myhash#|) @@ -1003,6 +1498,7 @@ On the other hand, implicit cat-6 control sequences (e.g., \verb|\let\myhash#|) whose cat-12 substitution text is a \macname{string} of the original implicit-macro name. +\begingroup \resettokcycle \Characterdirective{\ifcatSIX \addcytoks{\fbox{#1}} @@ -1045,6 +1541,7 @@ On the other hand, implicit cat-6 control sequences (e.g., \verb|\let\myhash#|) \catcode`Q=11 \hrulefill +\endgroup \ddbend For what is, perhaps, a rare situation, one can even process input streams that contain cat-6 macro parameters. @@ -1054,10 +1551,6 @@ A package macro, \macname{whennotprocessingparameter\#1\{<}% In this example, a macro is defined and then executed, subject to token replacements brought about by the expandable Character directive. -\Characterdirective{% - \whennotprocessingparameter#1{\addcytoks[x]{\vowelcap{#1}}}} -\aftertokcycle{\global\cytoks\expandafter{\the\cytoks}} -\def\Characterdirective#1{} \begin{exampleA}[lefthand width=7cm]{Preserving parameters (e.g. \texttt{\#1, \#2}) in the \TokCycle{} input stream} \Characterdirective{% @@ -1172,6 +1665,7 @@ Changing grouping tokens (catcodes 1,2) may require something more, In the following examples, pay attention to the detokenized grouping around the argument to \macname{fbox}. +\medskip As we will see, the issues raised here only affect the situation when detokenization of the output stream is required. \medskip @@ -1336,21 +1830,13 @@ The next two examples introduce implicit and active-implicit spaces. The latter example also demonstrates detokenization of such spaces. \medskip -Here is a useful macro for detokenizing space tokens in the context of - \TokCycle{}. +Here is a useful expandable macro for detokenizing space tokens + in the context of \TokCycle{}. It can process not only explicit cat-10 spaces, but also implicit and active-implicit spaces. To add it to the \macname{cytoks} token list, make sure you use the \verb|[x]| fully expanded option to \macname{addcytoks}. -\medskip -\begin{exampleC}[]{The \macname{detokenizespacetok} macro for handling - implicit and active-implicit spaces} -\newcommand\detokenizespacetok[1]{\tctestifcon{\ifimplicittok}% - {\tctestifcon{\ifactivetok}{\theactivespace}{\string#1}}{\detokenize{#1}}} -\end{exampleC} -\medskip - %%% \resettokcycle \Characterdirective{\addcytoks{\underdot{#1}}} @@ -1367,6 +1853,13 @@ To add it to the \macname{cytoks} token list, make sure you use the \endtokencyclexpress \end{exampleA} +\begin{minipage}[t]{\textwidth} +\begin{exampleC}[]{The \macname{detokenizespacetok} macro for handling + implicit and active-implicit spaces} +\newcommand\detokenizespacetok[1]{\tctestifcon{\ifimplicittok}% + {\tctestifcon{\ifactivetok}{\theactivespace}{\string#1}}{\detokenize{#1}}} +\end{exampleC} + \begin{exampleA}[lefthand width=10cm]{Active Implicit spaces work{,} too} \resettokcycle \Characterdirective{\addcytoks{\underdot{#1}}} @@ -1389,7 +1882,7 @@ xQx x_x\z{}x% xQx x_x\z{}x% \endtokencyclexpress \end{exampleA} - +\end{minipage} \endgroup diff --git a/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex b/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex index eeba320682b..9aaec68b728 100644 --- a/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex +++ b/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex @@ -1,7 +1,7 @@ \def\tcname {tokcycle} -\def\tcver {1.2} +\def\tcver {1.3} % -\def\tcdate {2020/10/01} +\def\tcdate {2021/03/10} % % Author : Steven B Segletes, Christian Tellechea (contributor) % Maintainer : Steven B Segletes @@ -22,8 +22,7 @@ \expandafter\the\expandafter\cytoks\expanded{#5}\endtokcycraw} % ENVIRONMENT FORM \long\def\tokencycle#1#2#3#4{\begingroup\let\endtokencycle\endtokcycraw - \aftertokcycle{\the\cytoks\expandafter\endgroup\expandafter\tcenvscope - \expandafter{\the\cytoks}}\tokcycraw{#1}{#2}{#3}{#4}} + \aftertokcycle{\the\cytoks\tcendgroup}\tokcycraw{#1}{#2}{#3}{#4}} % XPRESS-INTERFACE MACRO FORM \long\def\tokcyclexpress#1{\tokcycrawxpress#1\endtokcycraw} % XPRESS-INTERFACE \expanded-ARGUMENT MACRO FORM @@ -31,16 +30,16 @@ \expandafter\tokcycrawxpress\expanded{#1}\endtokcycraw} % XPRESS-INTERFACE ENVIRONMENT FORM \def\tokencyclexpress{\begingroup\let\endtokencyclexpress\endtokcycraw - \aftertokcycle{\the\cytoks\expandafter\endgroup\expandafter\tcenvscope - \expandafter{\the\cytoks}}\tokcycrawxpress} + \aftertokcycle{\the\cytoks\tcendgroup}\tokcycrawxpress} % INITIALIZATION & INTERNAL TOOLS +\def\tcendgroup{\expandafter\endgroup\expandafter\tcenvscope\expandafter{\the\cytoks}} \def\tcenvscope{\cytoks}% CAN SET TO \global\cytoks TO OVERCOME SCOPE LIMITS \edef\restorecatcode{\catcode\number`\@=\number\catcode`\@\relax} \catcode`\@11 \newif\iftc@implicitgrp \newif\if@argnext \newtoks\tc@tok -\newcount\tc@depth +\newcount\tcdepth \def\tc@gobble#1{} \def\tc@deftok#1#2{\let#1= #2\empty} \tc@deftok\tc@sptoken{ } @@ -54,9 +53,8 @@ \long\def\tc@testxifx{\tc@earg\tctestifx} \long\def\test@ifmacro#1{\tctestifcatnx#1\relax} \def\tc@addtoks#1#2{\toks#1\expandafter{\the\toks#1#2}} -\def\addtc@depth{\advance\tc@depth 1} -\def\subtc@depth{\tc@depth=\numexpr\tc@depth-1\relax} -\def\tc@resetifs{\activetokfalse\implicittokfalse\tc@implicitgrpfalse +\def\add@tcdepth{\advance\tcdepth 1\relax} +\def\sub@tcdepth{\advance\tcdepth-1\relax}\def\tc@resetifs{\activetokfalse\implicittokfalse\tc@implicitgrpfalse \catSIXfalse\activecharfalse\activetokunexpandablefalse \def\theactivespace{}} \long\def\count@stringtoks#1{\tc@earg\count@toks{\string#1}} @@ -79,18 +77,29 @@ \long\def\tokcycleenvironment#1#2#3#4#5{\expandafter\def\expandafter#1% \expandafter{\expandafter\let\csname end\expandafter\tc@gobble \string#1\endcsname\endtokcycraw\tokencycle{#2}{#3}{#4}{#5}}} -\long\def\processtoks#1{\addtc@depth\@tokcycle#1\endtokcycraw } +\long\def\xtokcycleenvironment#1#2#3#4#5#6#7{\expandafter\def\expandafter + #1\expandafter{\expandafter\let\csname end\expandafter\tc@gobble + \string#1\endcsname\endtokcycraw\begingroup + \long\def\tcafterenv####1{\tc@defx\tcendgroup{\tcendgroup####1}}% + \aftertokcycle{#7\the\cytoks\tcendgroup}#6\tokcycraw{#2}{#3}{#4}{#5}}} +\long\def\processtoks#1{\@tokcycle#1\endtokcycraw} \def\whennotprocessingparameter#1#2{\tctestifcon\if@argnext{\@argnextfalse\cytoks \expandafter{\the\cytoks###1}}{\tctestifcon\ifcatSIX{\@argnexttrue}{#2}}} +\long\def\truncategroup#1{\tctestifx{\endtokcycraw#1}{#1}{\truncategroup}} +\long\def\truncatecycle#1{\tctestifx{\endtokcycraw#1}{\tctestifnum{\tcdepth=0} + {#1}{\truncatecycle}}{\tctestifx{\sub@tcdepth#1}{#1} + {\tctestifx{\exit@grouped#1}{#1}{}}\truncatecycle}} +\long\def\truncategroupiftokis#1#2{\tctestifx{\tc@next#1}{\truncategroup}{#2}} +\long\def\truncatecycleiftokis#1#2{\tctestifx{\tc@next#1}{\truncatecycle}{#2}} % ESSENTIAL METHOD: STREAMING MACRO WITH TERMINATOR: % \tokcycraw{<Char>}{<Group>}{<Macro>}{<Space>}<input-stream>\endtokcycraw \long\def\tokcycraw#1#2#3#4{\def\@chrT##1{#1}\long\def\@grpT##1{#2}% \long\def\@macT##1{#3}\def\@spcT##1{#4}\tokcycrawxpress} % ENTRY POINT FOR XPRESS METHOD WITHOUT EXPLICIT ARGUMENTS -\def\tokcycrawxpress{\cytoks{}\tc@depth=1\relax\@tokcycle} +\def\tokcycrawxpress{\cytoks{}\tcdepth=0\relax\@tokcycle} % CODE TO EXECUTE AT COMPLETION \long\def\aftertokcycle#1{\def\@aftertokcycle{#1}} -\def\endtokcycraw{\subtc@depth\tctestifnum{\tc@depth=0}{\@aftertokcycle}{}} +\def\endtokcycraw{\tctestifnum{\tcdepth=0}{\@aftertokcycle}{}} % LOOP ENTRY POINT \def\@tokcycle{\tc@resetifs\futurelet\tc@next\detect@CANTabsorb} \def\detect@CANTabsorb{\tctestifx{\tc@next\tc@sptoken}{\stringify\@@@@@@spcT}% @@ -126,8 +135,8 @@ {\begingroup\catcode`#1=\active \xdef\@tmp{\scantokens{#1\noexpand}}\endgroup \expandafter\implicitgrpfork\@tmp\@tokcycle}% ACTIVE CHAR \bgroup }} -\def\@@grpT#1{\tctestifcon\ifstripgrouping{\@grpT{#1}}% - {\groupedcytoks{\@grpT{#1}}}\@tokcycle} +\long\def\@@grpT#1{\add@tcdepth\tctestifcon{\ifstripgrouping}{% + \@grpT{#1}}{\groupedcytoks{\@grpT{#1}}}\sub@tcdepth\@tokcycle} % \ COMMANDS (MACROS AND IMPLICITS) \long\def\backslashcmds#1{% \test@ifmacro#1{\tctestifcon\ifcatSIX{\implicittoktrue\@chrT#1}{\@macT#1}}% @@ -146,12 +155,12 @@ \global\let\implicitsixtok\six@str\tc@tok{\implicitsixtok}}% {\tc@tok\expandafter{\string#1}\tctestifnum{\number\catcode`#1=6}% {}{\activetoktrue\implicittoktrue}}} -% DIRECTIVES FOR HANDLING GROUPS RECURSIVELY; DEFINE tokcycle GROUPING CHARS -\def\@defgroupedcytoks#1{\long\def\groupedcytoks##1{% - \begingroup\cytoks{}##1\expandafter\endgroup\expandafter +% DIRECTIVES FOR HANDLING GROUPED OUTPUT; DEFINE tokcycle GROUPING CHARS +\long\def\groupedcytoks#1{\begingroup\cytoks{}#1\exit@grouped} +\def\defineexit@grouped#1{\def\exit@grouped{\expandafter\endgroup\expandafter \addcytoks\expandafter{\expandafter#1}}} \def\settcGrouping#1{\def\@tmp##1{#1}\tc@defx\@@tmp{\@tmp{\the\cytoks}}% - \tc@earg\@defgroupedcytoks{\@@tmp}} + \tc@earg\defineexit@grouped{\@@tmp}} % FAUX TOKENIZATION OF COMMAND NAME (WHEN cat0 TOKEN HAS BEEN MADE cat12) \def\csmk#1{\def\csaftermk{#1}\toks0{}\@csmkA} \def\@csmkA{\futurelet\@tmp\@csmkB} @@ -208,10 +217,10 @@ \endinput EDIT HISTORY -v1.0 2019/8/21 +v1.0 2019/08/21 - Initial release -v1.1 2019/9/27 +v1.1 2019/09/27 - Introduced \ifactivechar, \ifactivetokunexpandable - Tightened up consistent definition of implicit (to exclude primitives) - Rewrote active token trapping logic, to differentiate between active @@ -230,10 +239,47 @@ v1.12 2020/02/11 - Redefined \tc@defx and \tc@earg to omit #2 as part of definition - Corrected \trapcatSIXb definition to account for revised \tc@earg definition. -v1.2 2020/10/1 - -Added/fixed capability to handle active-implicit spaces. While the - #1 passed to the \Spacedirective, in such a case, is an implicit - space \tc@sptoken, the name of the active character from whence it - originated in the input stream is stored as an explicit cat-12 token - in the definition \theactivespace. The flags \implicittok and \activetok - are both set true, and the \activechar flag is checked, as well.
\ No newline at end of file +v1.2 2020/10/01 + - Added/fixed capability to handle active-implicit spaces. While the + #1 passed to the \Spacedirective, in such a case, is an implicit + space \tc@sptoken, the name of the active character from whence it + originated in the input stream is stored as an explicit cat-12 token + in the definition \theactivespace. The flags \implicittok and \activetok + are both set true, and the \activechar flag is checked, as well. + +v1.3 2021/03/09 + - Introduced \xtokcycleenvironment, similar to \tokcycleenvironment, but + allows two additional arguments, defining the setup and trailing code + that will be run prior to the invocation and following conclusion of the + token cycle. + - Introduced \truncategroup, a directive to discard remaining tokens in the + current token-cycle group and close the group. + - Introduced \truncatecycle, a directive to discard remaining tokens in the + token-cycle input stream, but closing any open groups. + - Introduced \truncategroupiftokis{}{} to conditionally issue a \truncategroup + if the current token under consideration matches the 1st argument. + - Introduced \truncatecycleiftokis{}{} to conditionally issue a \truncatecycle + if the current token under consideration matches the 1st argument. + - Introduced \tcendgroup as a form of \endgroup that saves contents of + \cytoks upon group exit. Gives clarity to definitions of \tokencycle + and \tokencyclexpress environments. Available for general used. + - Fixed bug. Made \@@grpT \long, in the event that \par occurs inside a + group. + - Fixed bug. Added \relax to end of \add@tcdepth definition, the absence + of which had prevented timely update of the \tcdepth count. + - Documentation correction: improved explanation of implicit-cat-6 token + limitations. Likewise, added warning regarding active-implicit spaces. + - Documentation prepared with lmodern font, rather than default cm, + for reasons of better PDF hinting. + - Introduced \exit@grouped to the \groupedcytoks definition, for clarity and + ease of \expandafter. + - Reworked logic so that \tcdepth is associated with nested calls to the + Group directive, rather than with each invocation of \processtoks. + This primarily required redefinition of \@@grpT and its components. + - Changed name of \tc@depth to \tcdepth, as it may be a useful parameter for + users to check the token-cycle nesting depth. Therefore, also renamed + \subtc@depth to \sub@tcdepth, \addtc@depth to \add@tcdepth. + - Excised \sub@tcdepth from \endtokcycraw as part of new logic making + \tcdepth associated with the Group directive (also, outer token cycle + operates with depth = 0 rather than 1). + - Renamed \@defgroupedcytoks to \defineexit@grouped, to better match function. |