tokcycle (27sep19)

git-svn-id: svn://tug.org/texlive/trunk@52191 c570f23f-e606-0410-a88d-b1316a301751

1 files changed, 94 insertions, 44 deletions

diff --git a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex
index e7131379cf8..882cea40c95 100644
--- a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex
+++ b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex
@@ -44,21 +44,20 @@
 \newcytoks{<new token register>}
 \end{myverbbox}
 
-{\centering{\LARGE Package \TokCycle{}\par}
+{\centering{\LARGE Package \TokCycle{} \Large (v\tcver)\par}
 Steven B Segletes
 {\footnotesize $<$steven.b.segletes.civ@mail.mil$>$}
-\\\small{contributor\footnote{
-I am extremely grateful to Christian
+\\\small{contributor: Christian Tellechea}\footnote{
+{I am extremely grateful to Christian $<$unbonpetit@netc.fr$>$}
   for his assistance in the development of this package.
 The \macname{addcytoks} macro was provided by him.
-He gave constant reminders on what the token parser should be able
+He gave constant reminders on what the parser should be able
   to achieve, thus motivating me to spend the extra time striving for
-  a generality of application that would not otherwise have made it
-  into the package.
+  a generality of application that would not come naturally to me.
 I value highly his collegiality and hold his expertise in the highest 
   regard.
 }%
-: Christian Tellechea {\footnotesize$<$unbonpetit@netc.fr$>$}}\\\today\par}
+\\\today\par}
 
 \begin{quote}
 The \TokCycle{} package helps one to build tools to process 
@@ -90,10 +89,12 @@ The package provides a number of options
 The purpose of the \TokCycle{} package is to provide a tool
   to assist the user in developing token-processing macros and
   environments.
-The types of processing is limited only by the creativity of the user,
+The types of processing are limited only by the creativity of the user,
   but examples might include letter-case-operations, letter spacing, 
   dimensional manipulation,
   simple-ciphering, \{group\} manipulation, macro removal, etc.
+In one sense, it can be thought of as a streaming editor that operates
+  on \LaTeX{} input streams.
 
 The package can be loaded into both plain \TeX{}, by way of the 
   invocation \verb|\input tokcycle.tex|
@@ -134,8 +135,8 @@ Note that there are no \texttt{expanded} environments in \TokCycle,
 
 \end{itemize}
 
-The basic approach of all macros/environments is to grab each 
-  successive token (or group) from the input stream, 
+The basic approach of all \TokCycle{} macros/environments is to
+  grab each successive token (or group) from the input stream, 
   decide what category it is, and use the currently active
   processing directives to make any desired transformation of the 
   token (or group).
@@ -151,6 +152,18 @@ As tokens/groups are read from the input stream, they are
 The \TokCycle{} categories by which the input stream
   is dissected include \Character{}, 
   \Group{}, \Macro{}, and \Space{}.
+
+Catcode-0 tokens are directed to the \Macro{} directive for processing.
+Catcode-10 tokens are directed to the \Space{} directive.
+When an explicit catcode-1 token is encountered in the \TokCycle{} input
+  stream, the contents of the entire group (sans the grouping) are directed
+  to the \Group{} directive for further processing, which may in turn,
+  redirect the individual tokens to the other categories.
+The handling options of implicit cat-1 and 2 tokens are described later in
+  this document.
+Valid tokens that are neither catcode 0, 1, 2, nor 10, except where noted,
+  are directed to the \Character{} directive for processing.
+
 The syntax of the non-\texttt{xpress} macros/environments is\medskip
 
 \begin{verbbox}[\vbdelim]
@@ -195,7 +208,7 @@ In addition to the above macros/environments, the means is
   {<Character processing directive>}
   {<Group-content processing directive>}
   {<Macro processing directive>}
-  {<Space processing directive>}%
+  {<Space processing directive>}
 \end{verbbox}
 \medskip\theverbbox\medskip\par\noindent
 This will then permit simplified invocations of the form
@@ -241,14 +254,17 @@ The default directives at package outset and upon reset are
 
 The interpretation of these directives will be explained in the
   remainder of this document.
+Let it suffice for now to say that the default directive settings
+  pass through the input stream to the output stream, without
+  change.
 
 
 
 \section{Commands in the \TokCycle{} directives}\label{s:coms}
 
-The command-line token cycling tools to be employed in the package were 
+The command-line token cycling tools provided in the package are 
   listed in section~\ref{s:intro}.  
-For each of those commands/pseudo-environments provided, the 
+For each of those commands and/or pseudo-environments, the 
   user must (explicitly or implicitly) detail a set of directives to 
   specify the manner in which the \Character{}, \Group{}, \Macro{}, 
   and \Space{} tokens found in the input stream are to be processed.
@@ -310,8 +326,9 @@ Do I process the group's token content
 
 Grouping in the output stream is determined by the externally set 
   condition \macname{ifstripgrouping}.
-The package default is \macname{stripgroupingfalse}, such that any grouping
-  that appears in the input stream will be echoed in the output stream.
+The package default is \macname{stripgroupingfalse}, such that any 
+  explicit grouping that appears in the input stream will be echoed 
+  in the output stream.
 The alternative, \macname{stripgroupingtrue}, is dangerous 
   in the sense that it will strip the cat-1,2 grouping from the 
   group's tokens, thereby 
@@ -356,16 +373,17 @@ Such a special treatment has limited application---for example,
 \subsection{Escaping content from \TokCycle{} processing\label{s:escape}}
 
 There are times you may wish to prevent tokens in the \TokCycle{} 
-  input stream from being operated on by \TokCycle; you just want
+  input stream from being operated on by \TokCycle.
+Rather, you just want
   the content passed through unchanged to the output;  that is,
 with the intent to have multi-token content bypass the 
   \TokCycle{} directives altogether.
 
 \begin{sloppypar}
 The method developed by the package is to enclose the escaped
-  content in the input stream with the \TokCycle{} escape character,
-  initially set to a vertical rule character found on the keyboard:
-  ``$|\mkern1mu$''.
+  content in the input stream between a set of \TokCycle{} escape 
+  characters, initially set to a vertical rule character found 
+  on the keyboard: ``$|\mkern1mu$''.
 The main proviso is that the escaped content cannot straddle a 
   group boundary (the complete group may, however, be escaped).
 The escape character can be changed with
@@ -383,30 +401,67 @@ These \textit{if}-conditions can be examined within the appropriate
   directive (generally the \Character{} directive), to direct the
   course of action within the directive.
 
-\subsubsection{Active characters: \macname{ifactivetok}}
+\subsubsection{Active characters}
 
-Active (cat-13) characters that occur in the input stream result in the
+\paragraph{\macname{ifactivetok}:}
+
+Active (cat-13) tokens that occur in the input stream result in the
   flag \macname{ifactivetok} being set \macname{activetoktrue}.
-Note that the substitution of the token's active value occurs \textit{after}
-  \TokCycle{} processing, and so \TokCycle{} only sees it as 
-  \macname{active}, regardless of its later substitution or assignment.
+Note that the expansion of the token's active \macname{def}
+  occurs \textit{after} \TokCycle{} processing.
+With active \macname{let}'s, there is no text substitution; however,
+  the assignment is already active at the time of \TokCycle{} processing.
+%
 The only exception to this rule is with pre-expanded input,
   \macname{expandedtokcycle[xpress]}.
-If an active token's substitution is governed by a \macname{def} (but 
-  not \macname{let}), the text substitution will have occurred before 
-  reaching the token cycle.
+If an active token's substitution is governed by a \macname{def}, the 
+  text substitution will have occurred before reaching the token cycle.
+
+\paragraph{\macname{ifactivetokunexpandable}:} 
+
+This flag is similar to \macname{ifactivetok}, in that a token must
+  be active for this to be set true.
+However, in addition, it is only
+  true if the active token is \macname{let} to a character or a 
+  primitive, neither of which can be expanded.
+Active characters assigned via \macname{def} or else \macname{let} to a
+  macro will \textit{not} qualify as \macname{activetokunexpandabletrue}.
+
+\paragraph{\macname{ifactivechar}:}
+
+This flag, rather than testing the token, tests the character code of
+  the token, to see if it is set active.  
+Generally, the token and its character code will be syncronized
+  in their \textit{activeness}.
+However, if a token is tokenized when active, but the corresponding character 
+  code is made non-active in the meantime, prior to the token reaching
+  \TokCycle{} processing, this flag will be set \macname{activecharfalse}.
+A similar discrepency will arise if a token is not active when tokenized,
+  but the character code is made active in the interim, prior to
+  \TokCycle{} processing.
 
 \subsubsection{Implicit tokens: \macname{ifimplicittok}}
 
-Implicit tokens, those assigned via \macname{let}, are flagged with
+Implicit tokens, those assigned via \macname{let} to characters\footnote{%
+Some clarification may be needed.
+Control sequences and active characters that are \macname{let} 
+  to something other than a cat-0 control sequence will be flagged 
+  as implicit.
+If implicit, a token will be processed 
+  through the \Character{} directive (exceptions noted).
+On the other hand, if a control sequence or active character is \macname{let} 
+  to a cat-0 control sequence, it will be directed to the \Macro{} 
+  directive for processing, without the implicit flag.
+}%
+  , are flagged with
   a \texttt{true} setting for \macname{ifimplicittok}.
 Generally, implicit tokens will be directed to the \Character{}
   directive for processing.
 There are, however, two exceptions:
-i) implicit grouping tokens (e.g., \macname{bgroup} and \macname{egroup})
+i)~implicit grouping tokens (e.g., \macname{bgroup} and \macname{egroup})
   will not appear in any directive unless the
   \macname{stripimplicitgroupingcase} has been set to $-$1; and
-ii) implicit space tokens (e.g., \macname{@sptoken}) will be processed 
+ii)~implicit space tokens (e.g., \macname{@sptoken}) will be processed 
   by the \Space{} directive.
 
 \subsubsection{Parameter (cat-6) tokens (e.g., \texttt{\#}): \macname{ifcatSIX}}
@@ -425,13 +480,9 @@ As to the default nature of this conversion (if no special actions are
   taken), explicit cat-6 characters are converted
   into the identical character with category code of 12.  
 On the other hand, implicit cat-6 macros (e.g., \verb|\let\myhash#|) 
-  are converted into a fixed-name implicit macro, \macname{implicitsixtok}, 
+  are converted into a fixed-name macro, \macname{implicitsixtok}, 
   whose cat-12 substitution text is a \macname{string} of the original
   implicit-macro name.
-
-As previously mentioned, active characters whose active value is
-  \macname{let} to cat-6 (e.g., \verb|\catcode`X=\active \letX#|) will be
-  flagged as \macname{activetoktrue}, but not \macname{catSIXtrue}.
 \end{sloppypar}
 
 \subsubsection{Parameters (\texttt{\#1}--\texttt{\#9}): 
@@ -466,7 +517,7 @@ When dealing with macros for which the order of expansion is important,
   of proper expansion and execution.
 These four restructured \textit{if} macros came in handy writing the 
   package, and may be of use in creating your directives,
-  to prevent \macname{else} and \macname{fi} from getting in the way:\medskip
+  without \macname{else} and \macname{fi} getting in the way:\medskip
 
 \begin{verbbox}[\vbdelim]
 \tctestifcon{<TeX-if-condition>}{<true-code>}{<false-code>}
@@ -495,7 +546,8 @@ Unlike macros, the \TokCycle{} environments are unaffected
   by \macname{aftertokcycle}, as they actually set it locally 
   (within their scope) to accomplish their own purposes.
 
-\subsection{Accommodating catcode-1,2 changes: \macname{settcGrouping}}
+\subsection{Accommodating catcode-1,2 changes: \macname{settcGrouping}%
+  \label{s:grouping}}
 
 In order to avoid making the \TokCycle{} parser overly complex, requiring
   multiple passes of the input stream, the package defaults to using
@@ -518,12 +570,10 @@ If angle-brackets \verb|< >| were to be the \textit{new}
   invoke \verb|\settcGrouping<<#1>>|.
 These will then be the grouping tokens in the \TokCycle{} output
   stream until set to something else.
-The grouping tokens are \textit{not} reset as part of
-  \macname{resettokcycle}.
 
 \section{Usage Examples}
 
-See the adjunct file, \texttt{tokcycle-examples.pdf}, for a wide 
+See the adjunct file, \texttt{tokcycle-examples.pdf}, for an 
   array of \TokCycle{} examples.
 
 \section{Summary of known package limitations}
@@ -549,9 +599,11 @@ If I can't fix them, I can at least disclaim and declaim them.
   bracing group; however, 
   grouping will still be handled properly (i.e., cat-1,2 tokens will
   be detected), even if the package is not notified.
+See section~\ref{s:grouping}.
 
 \item Should one need to keep track of the \textit{names} of implicit tokens, 
-  then only one named implicit cat-6 token may appear in the input 
+  then only one named implicit cat-6 token (e.g., \verb|\let\svhash#|) 
+  may appear in the input 
   stream (though it can appear multiple times). 
 There is no limit on explicit cat-6 toks.
 This limitation occurs because implicit cat-6 tokens are converted 
@@ -581,11 +633,9 @@ The \texttt{tex.stackexchange} site provides a wonderful
 \section*{Source Code}
 
 \Large\texttt{tokcycle.sty}
-
-\verbfilenobox[\footnotesize]{tokcycle.sty}\bigskip
+\verbfilenobox[\footnotesize]{tokcycle.sty}\vspace{2in}
 
 \noindent\texttt{tokcycle.tex}
-
 \verbfilenobox[\footnotesize]{tokcycle.tex}