diff options
| -rw-r--r-- | Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf | bin | 241795 -> 255336 bytes | |||
| -rw-r--r-- | Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex | 138 | ||||
| -rw-r--r-- | Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf | bin | 244484 -> 260545 bytes | |||
| -rw-r--r-- | Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex | 272 | ||||
| -rw-r--r-- | Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex | 60 |
5 files changed, 359 insertions, 111 deletions
diff --git a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf Binary files differindex c36edd5f642..c35f6412580 100644 --- a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf +++ b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf diff --git a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex index e7131379cf8..882cea40c95 100644 --- a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex +++ b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex @@ -44,21 +44,20 @@ \newcytoks{<new token register>} \end{myverbbox} -{\centering{\LARGE Package \TokCycle{}\par} +{\centering{\LARGE Package \TokCycle{} \Large (v\tcver)\par} Steven B Segletes {\footnotesize $<$steven.b.segletes.civ@mail.mil$>$} -\\\small{contributor\footnote{ -I am extremely grateful to Christian +\\\small{contributor: Christian Tellechea}\footnote{ +{I am extremely grateful to Christian $<$unbonpetit@netc.fr$>$} for his assistance in the development of this package. The \macname{addcytoks} macro was provided by him. -He gave constant reminders on what the token parser should be able +He gave constant reminders on what the parser should be able to achieve, thus motivating me to spend the extra time striving for - a generality of application that would not otherwise have made it - into the package. + a generality of application that would not come naturally to me. I value highly his collegiality and hold his expertise in the highest regard. }% -: Christian Tellechea {\footnotesize$<$unbonpetit@netc.fr$>$}}\\\today\par} +\\\today\par} \begin{quote} The \TokCycle{} package helps one to build tools to process @@ -90,10 +89,12 @@ The package provides a number of options The purpose of the \TokCycle{} package is to provide a tool to assist the user in developing token-processing macros and environments. -The types of processing is limited only by the creativity of the user, +The types of processing are limited only by the creativity of the user, but examples might include letter-case-operations, letter spacing, dimensional manipulation, simple-ciphering, \{group\} manipulation, macro removal, etc. +In one sense, it can be thought of as a streaming editor that operates + on \LaTeX{} input streams. The package can be loaded into both plain \TeX{}, by way of the invocation \verb|\input tokcycle.tex| @@ -134,8 +135,8 @@ Note that there are no \texttt{expanded} environments in \TokCycle, \end{itemize} -The basic approach of all macros/environments is to grab each - successive token (or group) from the input stream, +The basic approach of all \TokCycle{} macros/environments is to + grab each successive token (or group) from the input stream, decide what category it is, and use the currently active processing directives to make any desired transformation of the token (or group). @@ -151,6 +152,18 @@ As tokens/groups are read from the input stream, they are The \TokCycle{} categories by which the input stream is dissected include \Character{}, \Group{}, \Macro{}, and \Space{}. + +Catcode-0 tokens are directed to the \Macro{} directive for processing. +Catcode-10 tokens are directed to the \Space{} directive. +When an explicit catcode-1 token is encountered in the \TokCycle{} input + stream, the contents of the entire group (sans the grouping) are directed + to the \Group{} directive for further processing, which may in turn, + redirect the individual tokens to the other categories. +The handling options of implicit cat-1 and 2 tokens are described later in + this document. +Valid tokens that are neither catcode 0, 1, 2, nor 10, except where noted, + are directed to the \Character{} directive for processing. + The syntax of the non-\texttt{xpress} macros/environments is\medskip \begin{verbbox}[\vbdelim] @@ -195,7 +208,7 @@ In addition to the above macros/environments, the means is {<Character processing directive>} {<Group-content processing directive>} {<Macro processing directive>} - {<Space processing directive>}% + {<Space processing directive>} \end{verbbox} \medskip\theverbbox\medskip\par\noindent This will then permit simplified invocations of the form @@ -241,14 +254,17 @@ The default directives at package outset and upon reset are The interpretation of these directives will be explained in the remainder of this document. +Let it suffice for now to say that the default directive settings + pass through the input stream to the output stream, without + change. \section{Commands in the \TokCycle{} directives}\label{s:coms} -The command-line token cycling tools to be employed in the package were +The command-line token cycling tools provided in the package are listed in section~\ref{s:intro}. -For each of those commands/pseudo-environments provided, the +For each of those commands and/or pseudo-environments, the user must (explicitly or implicitly) detail a set of directives to specify the manner in which the \Character{}, \Group{}, \Macro{}, and \Space{} tokens found in the input stream are to be processed. @@ -310,8 +326,9 @@ Do I process the group's token content Grouping in the output stream is determined by the externally set condition \macname{ifstripgrouping}. -The package default is \macname{stripgroupingfalse}, such that any grouping - that appears in the input stream will be echoed in the output stream. +The package default is \macname{stripgroupingfalse}, such that any + explicit grouping that appears in the input stream will be echoed + in the output stream. The alternative, \macname{stripgroupingtrue}, is dangerous in the sense that it will strip the cat-1,2 grouping from the group's tokens, thereby @@ -356,16 +373,17 @@ Such a special treatment has limited application---for example, \subsection{Escaping content from \TokCycle{} processing\label{s:escape}} There are times you may wish to prevent tokens in the \TokCycle{} - input stream from being operated on by \TokCycle; you just want + input stream from being operated on by \TokCycle. +Rather, you just want the content passed through unchanged to the output; that is, with the intent to have multi-token content bypass the \TokCycle{} directives altogether. \begin{sloppypar} The method developed by the package is to enclose the escaped - content in the input stream with the \TokCycle{} escape character, - initially set to a vertical rule character found on the keyboard: - ``$|\mkern1mu$''. + content in the input stream between a set of \TokCycle{} escape + characters, initially set to a vertical rule character found + on the keyboard: ``$|\mkern1mu$''. The main proviso is that the escaped content cannot straddle a group boundary (the complete group may, however, be escaped). The escape character can be changed with @@ -383,30 +401,67 @@ These \textit{if}-conditions can be examined within the appropriate directive (generally the \Character{} directive), to direct the course of action within the directive. -\subsubsection{Active characters: \macname{ifactivetok}} +\subsubsection{Active characters} -Active (cat-13) characters that occur in the input stream result in the +\paragraph{\macname{ifactivetok}:} + +Active (cat-13) tokens that occur in the input stream result in the flag \macname{ifactivetok} being set \macname{activetoktrue}. -Note that the substitution of the token's active value occurs \textit{after} - \TokCycle{} processing, and so \TokCycle{} only sees it as - \macname{active}, regardless of its later substitution or assignment. +Note that the expansion of the token's active \macname{def} + occurs \textit{after} \TokCycle{} processing. +With active \macname{let}'s, there is no text substitution; however, + the assignment is already active at the time of \TokCycle{} processing. +% The only exception to this rule is with pre-expanded input, \macname{expandedtokcycle[xpress]}. -If an active token's substitution is governed by a \macname{def} (but - not \macname{let}), the text substitution will have occurred before - reaching the token cycle. +If an active token's substitution is governed by a \macname{def}, the + text substitution will have occurred before reaching the token cycle. + +\paragraph{\macname{ifactivetokunexpandable}:} + +This flag is similar to \macname{ifactivetok}, in that a token must + be active for this to be set true. +However, in addition, it is only + true if the active token is \macname{let} to a character or a + primitive, neither of which can be expanded. +Active characters assigned via \macname{def} or else \macname{let} to a + macro will \textit{not} qualify as \macname{activetokunexpandabletrue}. + +\paragraph{\macname{ifactivechar}:} + +This flag, rather than testing the token, tests the character code of + the token, to see if it is set active. +Generally, the token and its character code will be syncronized + in their \textit{activeness}. +However, if a token is tokenized when active, but the corresponding character + code is made non-active in the meantime, prior to the token reaching + \TokCycle{} processing, this flag will be set \macname{activecharfalse}. +A similar discrepency will arise if a token is not active when tokenized, + but the character code is made active in the interim, prior to + \TokCycle{} processing. \subsubsection{Implicit tokens: \macname{ifimplicittok}} -Implicit tokens, those assigned via \macname{let}, are flagged with +Implicit tokens, those assigned via \macname{let} to characters\footnote{% +Some clarification may be needed. +Control sequences and active characters that are \macname{let} + to something other than a cat-0 control sequence will be flagged + as implicit. +If implicit, a token will be processed + through the \Character{} directive (exceptions noted). +On the other hand, if a control sequence or active character is \macname{let} + to a cat-0 control sequence, it will be directed to the \Macro{} + directive for processing, without the implicit flag. +}% + , are flagged with a \texttt{true} setting for \macname{ifimplicittok}. Generally, implicit tokens will be directed to the \Character{} directive for processing. There are, however, two exceptions: -i) implicit grouping tokens (e.g., \macname{bgroup} and \macname{egroup}) +i)~implicit grouping tokens (e.g., \macname{bgroup} and \macname{egroup}) will not appear in any directive unless the \macname{stripimplicitgroupingcase} has been set to $-$1; and -ii) implicit space tokens (e.g., \macname{@sptoken}) will be processed +ii)~implicit space tokens (e.g., \macname{@sptoken}) will be processed by the \Space{} directive. \subsubsection{Parameter (cat-6) tokens (e.g., \texttt{\#}): \macname{ifcatSIX}} @@ -425,13 +480,9 @@ As to the default nature of this conversion (if no special actions are taken), explicit cat-6 characters are converted into the identical character with category code of 12. On the other hand, implicit cat-6 macros (e.g., \verb|\let\myhash#|) - are converted into a fixed-name implicit macro, \macname{implicitsixtok}, + are converted into a fixed-name macro, \macname{implicitsixtok}, whose cat-12 substitution text is a \macname{string} of the original implicit-macro name. - -As previously mentioned, active characters whose active value is - \macname{let} to cat-6 (e.g., \verb|\catcode`X=\active \letX#|) will be - flagged as \macname{activetoktrue}, but not \macname{catSIXtrue}. \end{sloppypar} \subsubsection{Parameters (\texttt{\#1}--\texttt{\#9}): @@ -466,7 +517,7 @@ When dealing with macros for which the order of expansion is important, of proper expansion and execution. These four restructured \textit{if} macros came in handy writing the package, and may be of use in creating your directives, - to prevent \macname{else} and \macname{fi} from getting in the way:\medskip + without \macname{else} and \macname{fi} getting in the way:\medskip \begin{verbbox}[\vbdelim] \tctestifcon{<TeX-if-condition>}{<true-code>}{<false-code>} @@ -495,7 +546,8 @@ Unlike macros, the \TokCycle{} environments are unaffected by \macname{aftertokcycle}, as they actually set it locally (within their scope) to accomplish their own purposes. -\subsection{Accommodating catcode-1,2 changes: \macname{settcGrouping}} +\subsection{Accommodating catcode-1,2 changes: \macname{settcGrouping}% + \label{s:grouping}} In order to avoid making the \TokCycle{} parser overly complex, requiring multiple passes of the input stream, the package defaults to using @@ -518,12 +570,10 @@ If angle-brackets \verb|< >| were to be the \textit{new} invoke \verb|\settcGrouping<<#1>>|. These will then be the grouping tokens in the \TokCycle{} output stream until set to something else. -The grouping tokens are \textit{not} reset as part of - \macname{resettokcycle}. \section{Usage Examples} -See the adjunct file, \texttt{tokcycle-examples.pdf}, for a wide +See the adjunct file, \texttt{tokcycle-examples.pdf}, for an array of \TokCycle{} examples. \section{Summary of known package limitations} @@ -549,9 +599,11 @@ If I can't fix them, I can at least disclaim and declaim them. bracing group; however, grouping will still be handled properly (i.e., cat-1,2 tokens will be detected), even if the package is not notified. +See section~\ref{s:grouping}. \item Should one need to keep track of the \textit{names} of implicit tokens, - then only one named implicit cat-6 token may appear in the input + then only one named implicit cat-6 token (e.g., \verb|\let\svhash#|) + may appear in the input stream (though it can appear multiple times). There is no limit on explicit cat-6 toks. This limitation occurs because implicit cat-6 tokens are converted @@ -581,11 +633,9 @@ The \texttt{tex.stackexchange} site provides a wonderful \section*{Source Code} \Large\texttt{tokcycle.sty} - -\verbfilenobox[\footnotesize]{tokcycle.sty}\bigskip +\verbfilenobox[\footnotesize]{tokcycle.sty}\vspace{2in} \noindent\texttt{tokcycle.tex} - \verbfilenobox[\footnotesize]{tokcycle.tex} diff --git a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf Binary files differindex af6e951390b..b67afe35e45 100644 --- a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf +++ b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf diff --git a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex index b73e89168cf..ec63d48f945 100644 --- a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex +++ b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex @@ -50,20 +50,111 @@ \def\altdecytoks{\par\medskip\begingroup\noindent\macname{cytoks} \ \textit{alt}detokenization:\\\raggedright\footnotesize\ttfamily \expandafter\altdetokenize\expandafter{\the\cytoks}\par\endgroup} -\newcommand\TokCycle[1][t]{\textsf{#1okcycle}} +\newcommand\TokCycle{\textsf{tokcycle}} \parindent0pt \parskip0pt \begin{document} {\centering\LARGE \TokCycle{} Package Examples\par\smallskip -\large Steven B. Segletes\\\today\par} +\large\today\par} \tableofcontents \makeatletter %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - +% ALREADY PLACED IN tokcycle.tex +% +%\def\trapactivetok#1{\tctestifcatnx~#1{\activetoktrue}{\trapactivelet#1}} +% +%% WILL ALSO TRAP ACTIVE \let TO PRIMITIVES AS IMPLICIT; UNDO LATER IN \can@absorb@@ +%\def\trapactivelet#1{\tctestifcon{% +% \expandafter\if\detokenize{#1}#1}{}{\activetoktrue\implicittoktrue}} +% +%\makeatletter +%\long\def\backslashcmds#1{% +% \test@ifmacro#1{\tctestifcon\ifcatSIX{\implicittoktrue\@chrT#1}{\@macT#1}}% +% {\implicittoktrue\trapimplicitegrp#1\tctestifcon{\iftc@implicitgrp}% +% {\ifcase\@implicitgroupingcase\or\addcytoks{#1}\or\@chrT{#1}\fi}% +% {\@chrT#1}}} +% +%\long\def\can@absorb@#1{\tctestifnum{\count@stringtoks{#1}>1}% +% {\tctestifx{\endtokcycraw#1}{#1}{\backslashcmds#1\@tokcycle}}% +% {\trapactivetok#1\tc@trapescape#1{\tc@escapecytoks}{\can@absorb@@#1}}} +% +%\long\def\can@absorb@@#1{\let\@tmp#1\test@ifmacro\@tmp +% {\implicittokfalse\@macT#1}{\@chrT#1}\@tokcycle} +% +%\long\def\count@toks#1{\the\numexpr-1\count@@toks#1.\tc@endcnt} +%\long\def\count@@toks#1#2\tc@endcnt{+1\tc@ifempty{#2}{\relax}{\count@@toks#2\tc@endcnt}} +% +% END ALREADY PLACED IN tokcycle.tex + +% REVISED ACTIVE TOK TRAP +%\newif\ifactivechar +%\newif\ifactivetokunexpandable + +%\def\tc@resetifs{\activetokfalse\implicittokfalse\tc@implicitgrpfalse +% \catSIXfalse\activecharfalse\activetokunexpandablefalse} + +%\long\def\can@absorb@#1{\tctestifnum{\count@stringtoks{#1}>1}% +% {\tctestifx{\endtokcycraw#1}{#1}{\backslashcmds#1\@tokcycle}}% +% {\trapactives#1\tc@trapescape#1{\tc@escapecytoks}{\can@absorb@@#1}}} + +% \activechartrue: \catcode`X=\active AT TIME OF TEST +% \activetoktrue: \catcode`X=\active AT TIME OF X's TOKENIZATION +% \activetokunexpandabletrue: \catcode`X=\active WHEN TOKENIZED AND +% \let XY or \let X\primitive AT TIME OF TEST +% \implicittoktrue \let(X or \X)Y WHERE Y IS NOT cat-0 + +%\def\trapactives#1{\trapactivechar{#1}\trapactivetok{#1}} +%\def\trapactivechar#1{\tctestifnum{\number\catcode`#1=13}{\activechartrue}{}} +%\def\trapactivetok#1{\tctestifcatnx~#1{\activetoktrue}{% +% \trapactivetokunexpandable#1}} +%%% WILL ALSO TRAP ACTIVE \let TO PRIMITIVES AS IMPLICIT; UNDO LATER IN \can@absorb@@ +%\def\trapactivetokunexpandable#1{\tctestifcon{\expandafter\if +% \detokenize{#1}#1}{}{\activetoktrue\activetokunexpandabletrue\implicittoktrue}} +% END REVISED ACTIVE TOK TRAP + +% ACTIVE IMPLICIT CAT-6 TRAP +%\def\trapcatSIXc#1{\tctestifnum{\count@stringtoks{#1}>1}{\tc@defx\six@str{\string#1}% +% \global\let\implicitsixtok\six@str\tc@tok{\implicitsixtok}}% +% {\tc@tok\expandafter{\string#1}\tctestifnum{\number\catcode`#1=6}% +% {}{\activetoktrue\implicittoktrue}}} +% END ACTIVE IMPLICIT CAT-6 TRAP + +% ACTIVE CHAR \bgroup +%\def\@@@grpT#1{\tctestifnum{\number\catcode`#1=1}% +% {\expandafter\@@grpT\expandafter{\iffalse}\fi}% { +% {\implicittoktrue\tc@implicitgrptrue% +% \tctestifnum{`#1=92}% WORKS EVEN IF CAT-0 HAS CHANGED +% {\csmk{\expandafter\backslashcmds\thecs\@tokcycle}}% \bgroup +% {\begingroup\catcode`#1=\active \xdef\@tmp{\scantokens{#1\noexpand}}\endgroup +% \expandafter\@chrT\@tmp\@tokcycle}% ACTIVE CHAR \bgroup +% }% +%} +% END ACTIVE CHAR \bgroup + +%%%%%%%%%%TRAPS ACTIVE-IMPLICIT GROUP TOKS FOR \stripimplicitgroupingcase +%\def\implicitgrpfork#1{\tctestifcon{\iftc@implicitgrp}{\ifcase +% \@implicitgroupingcase\or\addcytoks{#1}\or\@chrT{#1}\fi}{\@chrT#1}} + +%\def\@@@grpT#1{\tctestifnum{\number\catcode`#1=1}% +% {\expandafter\@@grpT\expandafter{\iffalse}\fi}% { +% {\implicittoktrue\tc@implicitgrptrue% +% \tctestifnum{`#1=92}% WORKS EVEN IF CAT-0 HAS CHANGED +% {\csmk{\expandafter\backslashcmds\thecs\@tokcycle}}% \bgroup +% {\begingroup\catcode`#1=\active \xdef\@tmp{\scantokens{#1\noexpand}}\endgroup +% \expandafter\implicitgrpfork\@tmp\@tokcycle}% ACTIVE CHAR \bgroup +% }} + +%\long\def\can@absorb@@#1{\let\@tmp#1\test@ifmacro\@tmp{\implicittokfalse +% \@macT#1}{\trapimplicitegrp#1\implicitgrpfork#1}\@tokcycle} + +%\long\def\backslashcmds#1{% +% \test@ifmacro#1{\tctestifcon\ifcatSIX{\implicittoktrue\@chrT#1}{\@macT#1}}% +% {\implicittoktrue\trapimplicitegrp#1\implicitgrpfork#1}} +%%%%%%%%%%END TRAPS ACTIVE-IMPLICIT GROUP TOKS FOR \stripimplicitgroupingcase %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -227,7 +318,7 @@ Apply different directives to Characters (under-dot), Groups \begingroup -\begin{exampleA}[lefthand width=7.8cm]{Employing \TokCycle{} directives} +\begin{exampleA}[lefthand width=7.8cm]{} \tokcycle{\addcytoks{\underdot{#1}}} {\addcytoks{\{}\processtoks{#1} \addcytoks{\}}} @@ -239,7 +330,7 @@ Apply different directives to Characters (under-dot), Groups \endgroup -\subsubsection{Escaping text\label{s:esc}} +\subsubsection{Escaping text} Text between two successive escape characters is bypassed by \TokCycle{} and instead echoed to the output register. @@ -268,16 +359,15 @@ Default escape character is $|$. One can change it with \endgroup \subsubsection{Unexpandable, unexpanded, and expanded Character - directives\label{s:unex}} + directives} \begingroup -These examples concern the issue of whether the characters of the input +This section concerns the issue of whether the characters of the input stream are transformed before or after being placed in the output token register (\macname{cytoks}).\medskip \noindent\begin{minipage}[t]{\textwidth} -Transform characters (+1 ASCII) via unexpandable macro (see section~% - \ref{s:esc} for \macname{plusl} definition):\medskip +Transform characters (+1 ASCII) via unexpandable macro:\medskip \begin{exampleA}[lefthand width=8.3cm] {Unexpandable Character directive} @@ -353,8 +443,6 @@ Capitalize vowels (expanding the character directive) \Characterdirective{\addcytoks[x]{\vowelcap{#1}}} \noindent\begin{minipage}[t]{\textwidth} -See section~\ref{s:unex} for the \macname{vowelcap} definition. - \begin{exampleA}[lefthand width=5.5cm]{Normal token cycle (input stream not pre-expanded)} \tokcycle @@ -410,10 +498,7 @@ The macro implicit group tokens from the output; or $-$1 to instead pass the implicit group tokens to the Character directive (as implicit tokens) for separate processing (typically, when detokenization - is desired).\smallskip - -In the following example, see section~\ref{s:unex} for the - \macname{vowelcap} definition. + is desired). \begin{exampleA}[lefthand width=6cm]{Using \macname{stripimplicitgroupingcase} to affect treatment of @@ -438,6 +523,11 @@ In the following example, see section~\ref{s:unex} for the \detokenize\expandafter{\the\cytoks} \end{exampleA} +%\tokcyclexpress{Announcement: {\bfseries\bgroup\itshape +% Today \egroup it is} \today, a Wednesday.}\the\cytoks + +%\detokenize\expandafter{\the\cytoks} + \endgroup \subsubsection{Treatment options for explicit groups} @@ -529,7 +619,7 @@ Note that the content of groups can be altogether eliminated if \endgroup -\subsubsection{Group nesting\label{s:nest}} +\subsubsection{Group nesting} \begingroup @@ -562,18 +652,14 @@ Note that the content of groups can be altogether eliminated if \subsection{Direct use of \TokCycle} -While one may find it convenient to encapsulate \TokCycle{} commands - inside of other macros, it should be obvious that this is not a requirement. -\TokCycle[T]{} macros and environments (in regular or \texttt{xpress} - form) may be invoked directly from the document, without +\TokCycle{} (in regular or \texttt{xpress} form) may be invoked + directly from the document, without being first encapsulated within a macro or environment. \subsubsection{Modifying counters as part of the Character directive} \begingroup -In the following example, see section~\ref{s:nest} for the definitions - of \macname{restorecolor} and \macname{reducecolor}. \begin{exampleA}[lefthand width=8.5cm]{Using a period token (.) to reset a changing color} @@ -634,7 +720,7 @@ This macro attempts to give a more natural presentation of It is \textbf{not} to be confused as a replacement for \macname{detokenize}. In certain applications, it may offer a more pleasingly formatted - typesetting of detokenized material.\smallskip + typesetting of detokenized material. It is an unusual application of \TokCycle{} in that it does not actually use the \macname{cytoks} token register to collect @@ -880,14 +966,19 @@ Because \macname{tcremap} is expandable, the original text is totally absent \begingroup Active characters in the \TokCycle{} input stream are processed in their original form. -Their active substitutions only occur \textit{afterwards}, when +Their active substitutions arising from \macname{def}s + only occur \textit{afterwards}, when the \TokCycle{} output is typeset. -They may be identified in the Character directive with - \macname{ifactivetok}. - +They may be identified with the \macname{ifactivetok} test. +If \macname{let} to a character, they may be identified in the Character + directive; +If \macname{let} to a control sequence or defined via \macname{def}, + they may be identified in the Macro directive. \let\svt T \Characterdirective{\tctestifcon\ifactivetok - {\addcytoks{\fbox{#1}}}{\addcytoks{#1}}} + {\addcytoks{\fbox{#1-chr}}}{\addcytoks{#1}}} +\Macrodirective{\tctestifcon\ifactivetok + {\addcytoks{\fbox{#1-mac}}}{\addcytoks{#1}}} \begin{exampleA}[lefthand width=7.8cm]{Processing active characters} \resettokcycle @@ -900,7 +991,9 @@ This is a test!!\endtokencyclexpress This is a test!!\endtokencyclexpress \Characterdirective{\tctestifcon\ifactivetok - {\addcytoks{\fbox{#1}}}{\addcytoks{#1}}} + {\addcytoks{\fbox{#1-chr}}}{\addcytoks{#1}}} +\Macrodirective{\tctestifcon\ifactivetok + {\addcytoks{\fbox{#1-mac}}}{\addcytoks{#1}}} \tokencyclexpress This is a test!!\endtokencyclexpress @@ -913,24 +1006,29 @@ This is a test!!\endtokencyclexpress \end{exampleA} %%% + +\ddbend If the input stream is pre-\textit{expanded}, any active + substitutions that are expandable (i.e., those involving \macname{def} + as well as those \macname{let} to something expandable) + are made before reaching \TokCycle{} processing. +\svt hey are, thus, no longer detected as active, unless + \macname{noexpand} is applied before the pre-expansion. +In this example, the \texttt{!} that is not \macname{noexpand}\kern.3pted + is converted to a \texttt{?} prior to reaching \TokCycle{} + processing (and thus, not detected as \macname{active}): + \catcode`!=\active \def !{?} \catcode`T=\active \let T+ -\ddbend If the input stream is pre-\textit{expanded}, the active - substitutions involving \macname{def} (but not \macname{let}) - are made before reaching {\catcode`T=11 \TokCycle} processing. -Such tokens are no longer detected as active, unless - \macname{noexpand} is applied to the pre-expansion: - \begin{exampleA}[lefthand width=7.8cm]{Expanded input stream acts - upon active characters unless \macname{noexpand} is applied} + upon active \macname{def}ed characters unless \macname{noexpand} is applied} \expandedtokcyclexpress{This is a test!\noexpand!} \the\cytoks\par \detokenize\expandafter{\the\cytoks} \end{exampleA} -However, pre-\textit{tokenization} does not suffer this behavior: +However, pre-tokenization does not suffer this behavior: \begin{exampleA}[lefthand width=7.8cm]{Pre-tokenized input stream does not affect active characters} \def\tmp{This is a test!!} @@ -943,6 +1041,41 @@ However, pre-\textit{tokenization} does not suffer this behavior: \endgroup \end{minipage} +\begingroup +\ddbend One aspect of \TeX{} to remember is that catcodes + are assigned at tokenization; however, for active characters, + the substitution assignment is evaluated only upon execution. +So, if a cat-13 token is placed into a \macname{def}, it will + remain active even if the catcode of that character code is + later changed. +But if the cat-13 active definition is changed prior to the + execution of the \macname{def}'ed token, the revised + token assignment will apply. + +The following example demonstrates this concept, while showing, + whithout changing the input in any way, that \TokCycle{} + can properly digest active and implicit grouping (cat-1,2) characters: + +\begin{exampleA}[lefthand width=8cm]{Active and implicit grouping + tokens digestible by \TokCycle} +\catcode`Y=13 +\catcode`Z=13 +\let Y{ +\let Z} +\let\Y{ +\let\Z} +\def\tmp{\textit YabcZ de\Y\itshape f\Zg}% + +\def Y{\bgroup[NEW]}% APPLIES AT EXECUTION +\catcode`Y=11% DOES NOT AFFECT Y IN \tmp + +\expandafter\tokcyclexpress\expandafter{\tmp} +\the\cytoks + +\detokenize\expandafter{\the\cytoks} +\end{exampleA} +\endgroup + \subsubsection{Trap Catcode 6 (explicit \& implicit) tokens} \begingroup @@ -958,11 +1091,11 @@ Since they are unlikely to be used in that capacity inside a into something cat-12 and set the if-condition \macname{catSIXtrue}. In this manner, \macname{ifcatSIX} can be used inside the Character directive to convert cat-6 tokens into something of - the user's choosing.\smallskip + the user's choosing. As to this cat-12 conversion, explicit cat-6 characters are converted into the same character with cat-12. -On the other hand, implicit cat-6 macros (e.g., \verb|\let\myhash#|) +On the other hand, implicit cat-6 control sequences (e.g., \verb|\let\myhash#|) are converted into a fixed-name macro, \macname{implicitsixtok}, whose cat-12 substitution text is a \macname{string} of the original implicit-macro name. @@ -1018,10 +1151,6 @@ A package macro, \macname{whennotprocessingparameter\#1\{<}% In this example, a macro is defined and then executed, subject to token replacements brought about by the expandable Character directive. -\smallskip -In the following example, see section~\ref{s:unex} for - the \macname{vowelcap} definition. - \Characterdirective{% \whennotprocessingparameter#1{\addcytoks[x]{\vowelcap{#1}}}} \aftertokcycle{\global\cytoks\expandafter{\the\cytoks}} @@ -1048,17 +1177,24 @@ In the following example, see section~\ref{s:unex} for \begingroup -Implicit macros (assigned via \macname{let}) were already - mentioned in the context of cat-6. -However, implicit macros can be of any valid catcode. +Implicit control sequences (assigned via \macname{let} to characters) + were already mentioned in the context of cat-6. +However, implicit control sequences can be of any valid catcode (except + for cat-0, which we instead call macros or primitives). The condition \macname{ifimplicittok} is used to flag such - tokens for special processing. + tokens for special processing, as well as active tokens that + are \macname{let} to anything unexpandable. + +In the next example, implicit, cat-6 and implicit-cat-6 tokens may + all be differentiated, shown here with a multiplicity of + \macname{fbox}es. \fboxsep=1.5pt \begin{exampleA}[lefthand width=8cm]{Implicit\ensuremath{{}={}}% single box\char44 \ cat-6\ensuremath{{}={}}double box\char44 \ % implicit-cat-6\ensuremath{{}={}}triple box} +\catcode`Q=\active \let QN \let\littlet=t \let\littlel=l \let\svhash# @@ -1073,10 +1209,48 @@ The condition \macname{ifimplicittok} is used to flag such This \textit{is a \textbf{big}} \littlet est. -Next pa#agraph ending with implicit cat six +Qext pa#agraph ending with implicit cat six \svhash.\endtokencyclexpress \end{exampleA} +In the following example, we use both control sequences and active characters + in \macname{def} and \macname{let} capacities, to demonstrate how + \TokCycle{} digests things. +Implicit tokens (tokens \macname{let} to characters) are shown in a box, + with both the token name and the implicit value (note that tokens + \macname{let} to macros and primitives are not considered implicit). +Active tokens processed through the character directive are followed with + a \dag, whereas those processed through the macro directive are followed + with a \ddag. + + +\begin{exampleA}[lefthand width=7.5cm]{Non-active vs{.} active \macname{def} + \& \macname{let}} +\Characterdirective{\ifimplicittok + \addcytoks{\fbox{\detokenize{#1}:#1}}% + \else\addcytoks{#1}\fi\ifactivetok + \addcytoks{\rlap{\dag}}\fi\addcytoks{~\,}} +\Macrodirective{\ifimplicittok + \addcytoks{\fbox{\detokenize{#1}}}% + \else\addcytoks{#1}\fi\ifactivetok + \addcytoks{\rlap{\ddag}}\fi + \ifx\par#1\else\addcytoks{~\,}\fi} + +\def\A{a} +\let\B i +\let\C\today +\let\D\relax +\def\E{\relax} +\catcode`V=13 \def V{a} +\catcode`W=13 \let Ww +\catcode`X=13 \let X\today +\catcode`Y=13 \let Y\relax +\catcode`Z=13 \def Z{\relax} +\tokcyclexpress{\A\B\C\D\E ab\par VWXYZab} +\the\cytoks +\end{exampleA} + + \ddbend If the input stream is subject to pre-expansion, one will require \macname{noexpand} for macros where no pre-expansion is desired.\medskip @@ -1093,7 +1267,7 @@ Next pa#agraph ending with implicit cat six Changing grouping tokens (catcodes 1,2) may require something more, if the output stream is to be detokenized. In the following examples, pay attention to the - detokenized grouping around the argument to \macname{fbox}.\smallskip + detokenized grouping around the argument to \macname{fbox}. As we will see, the issues raised here only affect the situation when detokenization of the output stream is required. diff --git a/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex b/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex index d6f1b4003ef..5a33109df65 100644 --- a/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex +++ b/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex @@ -1,7 +1,7 @@ \def\tcname {tokcycle} -\def\tcver {1.0} +\def\tcver {1.1} % -\def\tcdate {2019/8/21} +\def\tcdate {2019/9/27} % % Author : Steven B Segletes, Christian Tellechea (contributor) % Maintainer : Steven B Segletes @@ -52,13 +52,15 @@ \long\def\tc@exfirst#1#2{#1} \long\def\tc@exsecond#1#2{#2} \long\def\tc@testxifx{\tc@earg\tctestifx} +\long\def\test@ifmacro#1{\tctestifcatnx#1\relax} \def\tc@addtoks#1#2{\toks#1\expandafter{\the\toks#1#2}} \def\addtc@depth{\advance\tc@depth 1} \def\subtc@depth{\tc@depth=\numexpr\tc@depth-1\relax} -\def\tc@resetifs{\activetokfalse\implicittokfalse\tc@implicitgrpfalse\catSIXfalse} +\def\tc@resetifs{\activetokfalse\implicittokfalse\tc@implicitgrpfalse + \catSIXfalse\activecharfalse\activetokunexpandablefalse} \long\def\count@stringtoks#1{\tc@earg\count@toks{\string#1}} -\def\count@toks#1{\the\numexpr-1\count@@toks#1.\tc@endcnt} -\def\count@@toks#1#2\tc@endcnt{+1\tc@ifempty{#2}{\relax}{\count@@toks#2\tc@endcnt}} +\long\def\count@toks#1{\the\numexpr-1\count@@toks#1.\tc@endcnt} +\long\def\count@@toks#1#2\tc@endcnt{+1\tc@ifempty{#2}{\relax}{\count@@toks#2\tc@endcnt}} \def\sv@hash{##} % EXTERNAL TOOLS \long\def\tctestifcon#1{#1\expandafter\tc@exfirst\else\expandafter\tc@exsecond\fi} @@ -70,6 +72,8 @@ \newif\ifcatSIX \newif\ifimplicittok \newif\ifactivetok +\newif\ifactivechar +\newif\ifactivetokunexpandable \newtoks\cytoks \long\def\tokcycleenvironment#1#2#3#4#5{\expandafter\def\expandafter#1% \expandafter{\expandafter\let\csname end\expandafter\tc@gobble @@ -89,14 +93,15 @@ % LOOP ENTRY POINT \def\@tokcycle{\tc@resetifs\futurelet\tc@next\detect@CANTabsorb} \def\detect@CANTabsorb{\tctestifx{\tc@next\tc@sptoken}{\stringify\@@@@@spcT}% - {\tctestifx{\tc@next\bgroup}{\stringify\@@@@grpT}% - {\can@absorb}}} + {\tctestifx{\tc@next\bgroup}{\stringify\@@@@grpT}{\can@absorb}}} % NON cat1,10 TOKENS \long\def\can@absorb#1{\tc@tok{#1}\trapcatSIX{#1}\expandafter\can@absorb@ \the\tc@tok} \long\def\can@absorb@#1{\tctestifnum{\count@stringtoks{#1}>1}% {\tctestifx{\endtokcycraw#1}{#1}{\backslashcmds#1\@tokcycle}}% - {\trapactivetok#1\tc@trapescape#1{\tc@escapecytoks}{\@chrT#1\@tokcycle}}} + {\trapactives#1\tc@trapescape#1{\tc@escapecytoks}{\can@absorb@@#1}}} +\long\def\can@absorb@@#1{\let\@tmp#1\test@ifmacro\@tmp{\implicittokfalse + \@macT#1}{\trapimplicitegrp#1\implicitgrpfork#1}\@tokcycle} %CONVERT NEXT (SPACE OR BEGIN-GROUP) TOKEN TO STRING \def\stringify#1{\expandafter#1\string}% #1 WILL BE \@@@@@spcT or \@@@@grpT %SPACE DECODE @@ -110,17 +115,21 @@ \def\@@@@grpT{\futurelet\tc@str\@@@grpT} \def\@@@grpT#1{\tctestifnum{\number\catcode`#1=1}% {\expandafter\@@grpT\expandafter{\iffalse}\fi}% { - {\implicittoktrue\tc@implicitgrptrue - \csmk{\expandafter\backslashcmds\thecs\@tokcycle}}}% \bgroup + {\implicittoktrue\tc@implicitgrptrue% + \tctestifnum{`#1=92}% WORKS EVEN IF CAT-0 HAS CHANGED + {\csmk{\expandafter\backslashcmds\thecs\@tokcycle}}% \bgroup + {\begingroup\catcode`#1=\active \xdef\@tmp{\scantokens{#1\noexpand}}\endgroup + \expandafter\implicitgrpfork\@tmp\@tokcycle}% ACTIVE CHAR \bgroup + }} \def\@@grpT#1{\tctestifcon\ifstripgrouping{\@grpT{#1}}% {\groupedcytoks{\@grpT{#1}}}\@tokcycle} % \ COMMANDS (MACROS AND IMPLICITS) -\long\def\test@ifmacro#1{\tctestifcatnx#1\relax}% REVISED! -\long\def\backslashcmds#1{\test@ifmacro#1{\tctestifcon\ifcatSIX{% - \implicittoktrue\@chrT#1}{\@macT#1}}{\implicittoktrue - \trapimplicitegrp#1\tctestifcon{\iftc@implicitgrp}% - {\ifcase\@implicitgroupingcase\or\addcytoks{#1}\or\@chrT{#1}\fi}% - {\@chrT#1}{\@chrT#1}}} +\long\def\backslashcmds#1{% + \test@ifmacro#1{\tctestifcon\ifcatSIX{\implicittoktrue\@chrT#1}{\@macT#1}}% + {\implicittoktrue\trapimplicitegrp#1\implicitgrpfork#1}} +% FORK BASED ON IMPLICIT GROUP TREATMENT +\def\implicitgrpfork#1{\tctestifcon{\iftc@implicitgrp}{\ifcase + \@implicitgroupingcase\or\addcytoks{#1}\or\@chrT{#1}\fi}{\@chrT#1}} % SET UP ESCAPE MECHANISM \def\settcEscapechar#1{\let\@tcEscapeptr#1% \def\tc@escapecytoks##1#1{\addcytoks{##1}\@tokcycle}} @@ -130,7 +139,8 @@ \def\trapcatSIXb#1{\tc@earg\tctestifcatnx\sv@hash#1{\catSIXtrue\trapcatSIXc#1}{}} \def\trapcatSIXc#1{\tctestifnum{\count@stringtoks{#1}>1}{\tc@defx\six@str{\string#1}% \global\let\implicitsixtok\six@str\tc@tok{\implicitsixtok}}% - {\tc@tok\expandafter{\string#1}}} + {\tc@tok\expandafter{\string#1}\tctestifnum{\number\catcode`#1=6}% + {}{\activetoktrue\implicittoktrue}}} % DIRECTIVES FOR HANDLING GROUPS RECURSIVELY; DEFINE tokcycle GROUPING CHARS \def\@defgroupedcytoks#1{\long\def\groupedcytoks##1{% \begingroup\cytoks{}##1\expandafter\endgroup\expandafter @@ -154,7 +164,12 @@ \def\trapimplicitegrp#1{\tctestifx{#1\egroup}{% \implicittoktrue\tc@implicitgrptrue}{}} % TRAP ACTIVE TOK -\def\trapactivetok#1{\tctestifnum{\number\catcode`#1=13}{\activetoktrue}{}} +\def\trapactives#1{\trapactivechar{#1}\trapactivetok{#1}} +\def\trapactivechar#1{\tctestifnum{\number\catcode`#1=13}{\activechartrue}{}} +\def\trapactivetok#1{\tctestifcatnx~#1{\activetoktrue}{\trapactivetokunexpandable#1}} +%% WILL ALSO TRAP ACTIVE \let TO PRIMITIVES AS IMPLICIT; UNDO LATER IN \can@absorb@@ +\def\trapactivetokunexpandable#1{\tctestifcon{\expandafter\if + \detokenize{#1}#1}{}{\activetoktrue\activetokunexpandabletrue\implicittoktrue}} % EXPRESS-INTERFACE - ALLOWS TO EXTERNALLY DEFINE DIRECTIVES \def\Characterdirective{\def\@chrT##1} \def\Groupdirective{\long\def\@grpT##1} @@ -190,3 +205,12 @@ EDIT HISTORY v1.0 2019/8/21 - Initial release + +v1.1 2019/9/27 + - Introduced \ifactivechar, \ifactivetokunexpandable + - Tightened up consistent definition of implicit (to exclude primitives) + - Rewrote active token trapping logic, to differentiate between active + token vs. active character code, in the event that an earlier tokenized + token no longer shares the current characteristics of the character code + - Added ability to handle active-implicit grouping tokens + - Added ability to handle active-implicit cat-6 tokens
\ No newline at end of file |