tokcycle (27sep19)

git-svn-id: svn://tug.org/texlive/trunk@52191 c570f23f-e606-0410-a88d-b1316a301751

4 files changed, 317 insertions, 93 deletions

diff --git a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf
index c36edd5f642..c35f6412580 100644
--- a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf
+++ b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf
diff --git a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex
index e7131379cf8..882cea40c95 100644
--- a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex
+++ b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex
@@ -44,21 +44,20 @@
 \newcytoks{<new token register>}
 \end{myverbbox}
 
-{\centering{\LARGE Package \TokCycle{}\par}
+{\centering{\LARGE Package \TokCycle{} \Large (v\tcver)\par}
 Steven B Segletes
 {\footnotesize $<$steven.b.segletes.civ@mail.mil$>$}
-\\\small{contributor\footnote{
-I am extremely grateful to Christian
+\\\small{contributor: Christian Tellechea}\footnote{
+{I am extremely grateful to Christian $<$unbonpetit@netc.fr$>$}
   for his assistance in the development of this package.
 The \macname{addcytoks} macro was provided by him.
-He gave constant reminders on what the token parser should be able
+He gave constant reminders on what the parser should be able
   to achieve, thus motivating me to spend the extra time striving for
-  a generality of application that would not otherwise have made it
-  into the package.
+  a generality of application that would not come naturally to me.
 I value highly his collegiality and hold his expertise in the highest 
   regard.
 }%
-: Christian Tellechea {\footnotesize$<$unbonpetit@netc.fr$>$}}\\\today\par}
+\\\today\par}
 
 \begin{quote}
 The \TokCycle{} package helps one to build tools to process 
@@ -90,10 +89,12 @@ The package provides a number of options
 The purpose of the \TokCycle{} package is to provide a tool
   to assist the user in developing token-processing macros and
   environments.
-The types of processing is limited only by the creativity of the user,
+The types of processing are limited only by the creativity of the user,
   but examples might include letter-case-operations, letter spacing, 
   dimensional manipulation,
   simple-ciphering, \{group\} manipulation, macro removal, etc.
+In one sense, it can be thought of as a streaming editor that operates
+  on \LaTeX{} input streams.
 
 The package can be loaded into both plain \TeX{}, by way of the 
   invocation \verb|\input tokcycle.tex|
@@ -134,8 +135,8 @@ Note that there are no \texttt{expanded} environments in \TokCycle,
 
 \end{itemize}
 
-The basic approach of all macros/environments is to grab each 
-  successive token (or group) from the input stream, 
+The basic approach of all \TokCycle{} macros/environments is to
+  grab each successive token (or group) from the input stream, 
   decide what category it is, and use the currently active
   processing directives to make any desired transformation of the 
   token (or group).
@@ -151,6 +152,18 @@ As tokens/groups are read from the input stream, they are
 The \TokCycle{} categories by which the input stream
   is dissected include \Character{}, 
   \Group{}, \Macro{}, and \Space{}.
+
+Catcode-0 tokens are directed to the \Macro{} directive for processing.
+Catcode-10 tokens are directed to the \Space{} directive.
+When an explicit catcode-1 token is encountered in the \TokCycle{} input
+  stream, the contents of the entire group (sans the grouping) are directed
+  to the \Group{} directive for further processing, which may in turn,
+  redirect the individual tokens to the other categories.
+The handling options of implicit cat-1 and 2 tokens are described later in
+  this document.
+Valid tokens that are neither catcode 0, 1, 2, nor 10, except where noted,
+  are directed to the \Character{} directive for processing.
+
 The syntax of the non-\texttt{xpress} macros/environments is\medskip
 
 \begin{verbbox}[\vbdelim]
@@ -195,7 +208,7 @@ In addition to the above macros/environments, the means is
   {<Character processing directive>}
   {<Group-content processing directive>}
   {<Macro processing directive>}
-  {<Space processing directive>}%
+  {<Space processing directive>}
 \end{verbbox}
 \medskip\theverbbox\medskip\par\noindent
 This will then permit simplified invocations of the form
@@ -241,14 +254,17 @@ The default directives at package outset and upon reset are
 
 The interpretation of these directives will be explained in the
   remainder of this document.
+Let it suffice for now to say that the default directive settings
+  pass through the input stream to the output stream, without
+  change.
 
 
 
 \section{Commands in the \TokCycle{} directives}\label{s:coms}
 
-The command-line token cycling tools to be employed in the package were 
+The command-line token cycling tools provided in the package are 
   listed in section~\ref{s:intro}.  
-For each of those commands/pseudo-environments provided, the 
+For each of those commands and/or pseudo-environments, the 
   user must (explicitly or implicitly) detail a set of directives to 
   specify the manner in which the \Character{}, \Group{}, \Macro{}, 
   and \Space{} tokens found in the input stream are to be processed.
@@ -310,8 +326,9 @@ Do I process the group's token content
 
 Grouping in the output stream is determined by the externally set 
   condition \macname{ifstripgrouping}.
-The package default is \macname{stripgroupingfalse}, such that any grouping
-  that appears in the input stream will be echoed in the output stream.
+The package default is \macname{stripgroupingfalse}, such that any 
+  explicit grouping that appears in the input stream will be echoed 
+  in the output stream.
 The alternative, \macname{stripgroupingtrue}, is dangerous 
   in the sense that it will strip the cat-1,2 grouping from the 
   group's tokens, thereby 
@@ -356,16 +373,17 @@ Such a special treatment has limited application---for example,
 \subsection{Escaping content from \TokCycle{} processing\label{s:escape}}
 
 There are times you may wish to prevent tokens in the \TokCycle{} 
-  input stream from being operated on by \TokCycle; you just want
+  input stream from being operated on by \TokCycle.
+Rather, you just want
   the content passed through unchanged to the output;  that is,
 with the intent to have multi-token content bypass the 
   \TokCycle{} directives altogether.
 
 \begin{sloppypar}
 The method developed by the package is to enclose the escaped
-  content in the input stream with the \TokCycle{} escape character,
-  initially set to a vertical rule character found on the keyboard:
-  ``$|\mkern1mu$''.
+  content in the input stream between a set of \TokCycle{} escape 
+  characters, initially set to a vertical rule character found 
+  on the keyboard: ``$|\mkern1mu$''.
 The main proviso is that the escaped content cannot straddle a 
   group boundary (the complete group may, however, be escaped).
 The escape character can be changed with
@@ -383,30 +401,67 @@ These \textit{if}-conditions can be examined within the appropriate
   directive (generally the \Character{} directive), to direct the
   course of action within the directive.
 
-\subsubsection{Active characters: \macname{ifactivetok}}
+\subsubsection{Active characters}
 
-Active (cat-13) characters that occur in the input stream result in the
+\paragraph{\macname{ifactivetok}:}
+
+Active (cat-13) tokens that occur in the input stream result in the
   flag \macname{ifactivetok} being set \macname{activetoktrue}.
-Note that the substitution of the token's active value occurs \textit{after}
-  \TokCycle{} processing, and so \TokCycle{} only sees it as 
-  \macname{active}, regardless of its later substitution or assignment.
+Note that the expansion of the token's active \macname{def}
+  occurs \textit{after} \TokCycle{} processing.
+With active \macname{let}'s, there is no text substitution; however,
+  the assignment is already active at the time of \TokCycle{} processing.
+%
 The only exception to this rule is with pre-expanded input,
   \macname{expandedtokcycle[xpress]}.
-If an active token's substitution is governed by a \macname{def} (but 
-  not \macname{let}), the text substitution will have occurred before 
-  reaching the token cycle.
+If an active token's substitution is governed by a \macname{def}, the 
+  text substitution will have occurred before reaching the token cycle.
+
+\paragraph{\macname{ifactivetokunexpandable}:} 
+
+This flag is similar to \macname{ifactivetok}, in that a token must
+  be active for this to be set true.
+However, in addition, it is only
+  true if the active token is \macname{let} to a character or a 
+  primitive, neither of which can be expanded.
+Active characters assigned via \macname{def} or else \macname{let} to a
+  macro will \textit{not} qualify as \macname{activetokunexpandabletrue}.
+
+\paragraph{\macname{ifactivechar}:}
+
+This flag, rather than testing the token, tests the character code of
+  the token, to see if it is set active.  
+Generally, the token and its character code will be syncronized
+  in their \textit{activeness}.
+However, if a token is tokenized when active, but the corresponding character 
+  code is made non-active in the meantime, prior to the token reaching
+  \TokCycle{} processing, this flag will be set \macname{activecharfalse}.
+A similar discrepency will arise if a token is not active when tokenized,
+  but the character code is made active in the interim, prior to
+  \TokCycle{} processing.
 
 \subsubsection{Implicit tokens: \macname{ifimplicittok}}
 
-Implicit tokens, those assigned via \macname{let}, are flagged with
+Implicit tokens, those assigned via \macname{let} to characters\footnote{%
+Some clarification may be needed.
+Control sequences and active characters that are \macname{let} 
+  to something other than a cat-0 control sequence will be flagged 
+  as implicit.
+If implicit, a token will be processed 
+  through the \Character{} directive (exceptions noted).
+On the other hand, if a control sequence or active character is \macname{let} 
+  to a cat-0 control sequence, it will be directed to the \Macro{} 
+  directive for processing, without the implicit flag.
+}%
+  , are flagged with
   a \texttt{true} setting for \macname{ifimplicittok}.
 Generally, implicit tokens will be directed to the \Character{}
   directive for processing.
 There are, however, two exceptions:
-i) implicit grouping tokens (e.g., \macname{bgroup} and \macname{egroup})
+i)~implicit grouping tokens (e.g., \macname{bgroup} and \macname{egroup})
   will not appear in any directive unless the
   \macname{stripimplicitgroupingcase} has been set to $-$1; and
-ii) implicit space tokens (e.g., \macname{@sptoken}) will be processed 
+ii)~implicit space tokens (e.g., \macname{@sptoken}) will be processed 
   by the \Space{} directive.
 
 \subsubsection{Parameter (cat-6) tokens (e.g., \texttt{\#}): \macname{ifcatSIX}}
@@ -425,13 +480,9 @@ As to the default nature of this conversion (if no special actions are
   taken), explicit cat-6 characters are converted
   into the identical character with category code of 12.  
 On the other hand, implicit cat-6 macros (e.g., \verb|\let\myhash#|) 
-  are converted into a fixed-name implicit macro, \macname{implicitsixtok}, 
+  are converted into a fixed-name macro, \macname{implicitsixtok}, 
   whose cat-12 substitution text is a \macname{string} of the original
   implicit-macro name.
-
-As previously mentioned, active characters whose active value is
-  \macname{let} to cat-6 (e.g., \verb|\catcode`X=\active \letX#|) will be
-  flagged as \macname{activetoktrue}, but not \macname{catSIXtrue}.
 \end{sloppypar}
 
 \subsubsection{Parameters (\texttt{\#1}--\texttt{\#9}): 
@@ -466,7 +517,7 @@ When dealing with macros for which the order of expansion is important,
   of proper expansion and execution.
 These four restructured \textit{if} macros came in handy writing the 
   package, and may be of use in creating your directives,
-  to prevent \macname{else} and \macname{fi} from getting in the way:\medskip
+  without \macname{else} and \macname{fi} getting in the way:\medskip
 
 \begin{verbbox}[\vbdelim]
 \tctestifcon{<TeX-if-condition>}{<true-code>}{<false-code>}
@@ -495,7 +546,8 @@ Unlike macros, the \TokCycle{} environments are unaffected
   by \macname{aftertokcycle}, as they actually set it locally 
   (within their scope) to accomplish their own purposes.
 
-\subsection{Accommodating catcode-1,2 changes: \macname{settcGrouping}}
+\subsection{Accommodating catcode-1,2 changes: \macname{settcGrouping}%
+  \label{s:grouping}}
 
 In order to avoid making the \TokCycle{} parser overly complex, requiring
   multiple passes of the input stream, the package defaults to using
@@ -518,12 +570,10 @@ If angle-brackets \verb|< >| were to be the \textit{new}
   invoke \verb|\settcGrouping<<#1>>|.
 These will then be the grouping tokens in the \TokCycle{} output
   stream until set to something else.
-The grouping tokens are \textit{not} reset as part of
-  \macname{resettokcycle}.
 
 \section{Usage Examples}
 
-See the adjunct file, \texttt{tokcycle-examples.pdf}, for a wide 
+See the adjunct file, \texttt{tokcycle-examples.pdf}, for an 
   array of \TokCycle{} examples.
 
 \section{Summary of known package limitations}
@@ -549,9 +599,11 @@ If I can't fix them, I can at least disclaim and declaim them.
   bracing group; however, 
   grouping will still be handled properly (i.e., cat-1,2 tokens will
   be detected), even if the package is not notified.
+See section~\ref{s:grouping}.
 
 \item Should one need to keep track of the \textit{names} of implicit tokens, 
-  then only one named implicit cat-6 token may appear in the input 
+  then only one named implicit cat-6 token (e.g., \verb|\let\svhash#|) 
+  may appear in the input 
   stream (though it can appear multiple times). 
 There is no limit on explicit cat-6 toks.
 This limitation occurs because implicit cat-6 tokens are converted 
@@ -581,11 +633,9 @@ The \texttt{tex.stackexchange} site provides a wonderful
 \section*{Source Code}
 
 \Large\texttt{tokcycle.sty}
-
-\verbfilenobox[\footnotesize]{tokcycle.sty}\bigskip
+\verbfilenobox[\footnotesize]{tokcycle.sty}\vspace{2in}
 
 \noindent\texttt{tokcycle.tex}
-
 \verbfilenobox[\footnotesize]{tokcycle.tex}
 
 
diff --git a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf
index af6e951390b..b67afe35e45 100644
--- a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf
+++ b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf
diff --git a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex
index b73e89168cf..ec63d48f945 100644
--- a/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex
+++ b/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex
@@ -50,20 +50,111 @@
 \def\altdecytoks{\par\medskip\begingroup\noindent\macname{cytoks}
   \ \textit{alt}detokenization:\\\raggedright\footnotesize\ttfamily
   \expandafter\altdetokenize\expandafter{\the\cytoks}\par\endgroup}
-\newcommand\TokCycle[1][t]{\textsf{#1okcycle}}
+\newcommand\TokCycle{\textsf{tokcycle}}
 
 \parindent0pt
 \parskip0pt
 \begin{document}
 {\centering\LARGE \TokCycle{} Package Examples\par\smallskip
-\large Steven B. Segletes\\\today\par}
+\large\today\par}
 
 \tableofcontents
 
 \makeatletter
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-
+% ALREADY PLACED IN tokcycle.tex
+%
+%\def\trapactivetok#1{\tctestifcatnx~#1{\activetoktrue}{\trapactivelet#1}}
+%
+%% WILL ALSO TRAP ACTIVE \let TO PRIMITIVES AS IMPLICIT; UNDO LATER IN \can@absorb@@
+%\def\trapactivelet#1{\tctestifcon{%
+%  \expandafter\if\detokenize{#1}#1}{}{\activetoktrue\implicittoktrue}}
+%
+%\makeatletter
+%\long\def\backslashcmds#1{%
+%  \test@ifmacro#1{\tctestifcon\ifcatSIX{\implicittoktrue\@chrT#1}{\@macT#1}}%
+%    {\implicittoktrue\trapimplicitegrp#1\tctestifcon{\iftc@implicitgrp}%
+%        {\ifcase\@implicitgroupingcase\or\addcytoks{#1}\or\@chrT{#1}\fi}%
+%        {\@chrT#1}}}
+%
+%\long\def\can@absorb@#1{\tctestifnum{\count@stringtoks{#1}>1}%
+%  {\tctestifx{\endtokcycraw#1}{#1}{\backslashcmds#1\@tokcycle}}%
+%  {\trapactivetok#1\tc@trapescape#1{\tc@escapecytoks}{\can@absorb@@#1}}}
+%
+%\long\def\can@absorb@@#1{\let\@tmp#1\test@ifmacro\@tmp
+%  {\implicittokfalse\@macT#1}{\@chrT#1}\@tokcycle}
+%
+%\long\def\count@toks#1{\the\numexpr-1\count@@toks#1.\tc@endcnt}
+%\long\def\count@@toks#1#2\tc@endcnt{+1\tc@ifempty{#2}{\relax}{\count@@toks#2\tc@endcnt}}
+%
+% END ALREADY PLACED IN tokcycle.tex
+
+% REVISED ACTIVE TOK TRAP
+%\newif\ifactivechar
+%\newif\ifactivetokunexpandable
+
+%\def\tc@resetifs{\activetokfalse\implicittokfalse\tc@implicitgrpfalse
+%  \catSIXfalse\activecharfalse\activetokunexpandablefalse}
+
+%\long\def\can@absorb@#1{\tctestifnum{\count@stringtoks{#1}>1}%
+%  {\tctestifx{\endtokcycraw#1}{#1}{\backslashcmds#1\@tokcycle}}%
+%  {\trapactives#1\tc@trapescape#1{\tc@escapecytoks}{\can@absorb@@#1}}}
+
+% \activechartrue: \catcode`X=\active AT TIME OF TEST
+% \activetoktrue: \catcode`X=\active AT TIME OF X's TOKENIZATION
+% \activetokunexpandabletrue: \catcode`X=\active WHEN TOKENIZED AND
+%     \let XY or \let X\primitive AT TIME OF TEST
+% \implicittoktrue \let(X or \X)Y WHERE Y IS NOT cat-0
+
+%\def\trapactives#1{\trapactivechar{#1}\trapactivetok{#1}}
+%\def\trapactivechar#1{\tctestifnum{\number\catcode`#1=13}{\activechartrue}{}}
+%\def\trapactivetok#1{\tctestifcatnx~#1{\activetoktrue}{%
+%  \trapactivetokunexpandable#1}}
+%%% WILL ALSO TRAP ACTIVE \let TO PRIMITIVES AS IMPLICIT; UNDO LATER IN \can@absorb@@
+%\def\trapactivetokunexpandable#1{\tctestifcon{\expandafter\if
+%  \detokenize{#1}#1}{}{\activetoktrue\activetokunexpandabletrue\implicittoktrue}}
+% END REVISED ACTIVE TOK TRAP
+
+% ACTIVE IMPLICIT CAT-6 TRAP
+%\def\trapcatSIXc#1{\tctestifnum{\count@stringtoks{#1}>1}{\tc@defx\six@str{\string#1}%
+%  \global\let\implicitsixtok\six@str\tc@tok{\implicitsixtok}}%
+%  {\tc@tok\expandafter{\string#1}\tctestifnum{\number\catcode`#1=6}%
+%  {}{\activetoktrue\implicittoktrue}}}
+% END ACTIVE IMPLICIT CAT-6 TRAP
+
+% ACTIVE CHAR \bgroup
+%\def\@@@grpT#1{\tctestifnum{\number\catcode`#1=1}%
+%  {\expandafter\@@grpT\expandafter{\iffalse}\fi}% {
+%  {\implicittoktrue\tc@implicitgrptrue%
+%    \tctestifnum{`#1=92}% WORKS EVEN IF CAT-0 HAS CHANGED
+%    {\csmk{\expandafter\backslashcmds\thecs\@tokcycle}}% \bgroup
+%    {\begingroup\catcode`#1=\active \xdef\@tmp{\scantokens{#1\noexpand}}\endgroup
+%      \expandafter\@chrT\@tmp\@tokcycle}% ACTIVE CHAR \bgroup
+%  }%
+%}
+% END ACTIVE CHAR \bgroup
+
+%%%%%%%%%%TRAPS ACTIVE-IMPLICIT GROUP TOKS FOR \stripimplicitgroupingcase
+%\def\implicitgrpfork#1{\tctestifcon{\iftc@implicitgrp}{\ifcase
+%  \@implicitgroupingcase\or\addcytoks{#1}\or\@chrT{#1}\fi}{\@chrT#1}}
+
+%\def\@@@grpT#1{\tctestifnum{\number\catcode`#1=1}%
+%  {\expandafter\@@grpT\expandafter{\iffalse}\fi}% {
+%  {\implicittoktrue\tc@implicitgrptrue%
+%    \tctestifnum{`#1=92}% WORKS EVEN IF CAT-0 HAS CHANGED
+%    {\csmk{\expandafter\backslashcmds\thecs\@tokcycle}}% \bgroup
+%    {\begingroup\catcode`#1=\active \xdef\@tmp{\scantokens{#1\noexpand}}\endgroup
+%      \expandafter\implicitgrpfork\@tmp\@tokcycle}% ACTIVE CHAR \bgroup
+%  }}
+
+%\long\def\can@absorb@@#1{\let\@tmp#1\test@ifmacro\@tmp{\implicittokfalse
+%  \@macT#1}{\trapimplicitegrp#1\implicitgrpfork#1}\@tokcycle}
+
+%\long\def\backslashcmds#1{%
+%  \test@ifmacro#1{\tctestifcon\ifcatSIX{\implicittoktrue\@chrT#1}{\@macT#1}}%
+%    {\implicittoktrue\trapimplicitegrp#1\implicitgrpfork#1}}
+%%%%%%%%%%END TRAPS ACTIVE-IMPLICIT GROUP TOKS FOR \stripimplicitgroupingcase
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
@@ -227,7 +318,7 @@ Apply different directives to Characters (under-dot), Groups
 
 
 \begingroup
-\begin{exampleA}[lefthand width=7.8cm]{Employing \TokCycle{} directives}
+\begin{exampleA}[lefthand width=7.8cm]{}
 \tokcycle{\addcytoks{\underdot{#1}}}
          {\addcytoks{\{}\processtoks{#1}
           \addcytoks{\}}}
@@ -239,7 +330,7 @@ Apply different directives to Characters (under-dot), Groups
 
 \endgroup
 
-\subsubsection{Escaping text\label{s:esc}}
+\subsubsection{Escaping text}
 
 Text between two successive escape characters is bypassed
   by \TokCycle{} and instead echoed to the output register.
@@ -268,16 +359,15 @@ Default escape character is $|$.  One can change it with
 \endgroup
 
 \subsubsection{Unexpandable, unexpanded, and expanded Character 
-  directives\label{s:unex}}
+  directives}
 
 \begingroup
-These examples concern the issue of whether the characters of the input
+This section concerns the issue of whether the characters of the input
   stream are transformed before or after being placed in the 
   output token register (\macname{cytoks}).\medskip
 
 \noindent\begin{minipage}[t]{\textwidth}
-Transform characters (+1 ASCII) via unexpandable macro (see section~%
-  \ref{s:esc} for \macname{plusl} definition):\medskip
+Transform characters (+1 ASCII) via unexpandable macro:\medskip
 
 \begin{exampleA}[lefthand width=8.3cm]
   {Unexpandable Character directive}
@@ -353,8 +443,6 @@ Capitalize vowels (expanding the character directive)
 \Characterdirective{\addcytoks[x]{\vowelcap{#1}}}
 
 \noindent\begin{minipage}[t]{\textwidth}
-See section~\ref{s:unex} for the \macname{vowelcap} definition.
-
 \begin{exampleA}[lefthand width=5.5cm]{Normal token cycle (input stream
   not pre-expanded)}
 \tokcycle
@@ -410,10 +498,7 @@ The macro
   implicit group tokens from the output; or $-$1 to instead pass the
   implicit group tokens to the Character directive (as implicit
   tokens) for separate processing (typically, when detokenization
-  is desired).\smallskip
-
-In the following example, see section~\ref{s:unex} for the
-  \macname{vowelcap} definition.
+  is desired).
 
 \begin{exampleA}[lefthand width=6cm]{Using 
   \macname{stripimplicitgroupingcase} to affect treatment of 
@@ -438,6 +523,11 @@ In the following example, see section~\ref{s:unex} for the
 \detokenize\expandafter{\the\cytoks}
 \end{exampleA}
 
+%\tokcyclexpress{Announcement: {\bfseries\bgroup\itshape 
+%  Today \egroup it is} \today, a Wednesday.}\the\cytoks
+
+%\detokenize\expandafter{\the\cytoks}
+
 \endgroup
 
 \subsubsection{Treatment options for explicit groups}
@@ -529,7 +619,7 @@ Note that the content of groups can be altogether eliminated if
 
 \endgroup
 
-\subsubsection{Group nesting\label{s:nest}}
+\subsubsection{Group nesting}
 
 \begingroup
 
@@ -562,18 +652,14 @@ Note that the content of groups can be altogether eliminated if
 
 \subsection{Direct use of \TokCycle}
 
-While one may find it convenient to encapsulate \TokCycle{} commands 
-  inside of other macros, it should be obvious that this is not a requirement.
-\TokCycle[T]{} macros and environments (in regular or \texttt{xpress} 
-  form) may be invoked directly from the document, without
+\TokCycle{} (in regular or \texttt{xpress} form) may be invoked 
+  directly from the document, without
   being first encapsulated within a macro or environment.
 
 \subsubsection{Modifying counters as part of the Character
   directive}
 
 \begingroup
-In the following example, see section~\ref{s:nest} for the definitions
-  of \macname{restorecolor} and \macname{reducecolor}.
 
 \begin{exampleA}[lefthand width=8.5cm]{Using a period token (.) to
   reset a changing color}
@@ -634,7 +720,7 @@ This macro attempts to give a more natural presentation of
 It is \textbf{not} to be confused as a replacement for
   \macname{detokenize}.
 In certain applications, it may offer a more pleasingly formatted
-  typesetting of detokenized material.\smallskip
+  typesetting of detokenized material.
 
 It is an unusual application of \TokCycle{} in that it does not
   actually use the \macname{cytoks} token register to collect
@@ -880,14 +966,19 @@ Because \macname{tcremap} is expandable, the original text is totally absent
 \begingroup
 Active characters in the \TokCycle{} input stream are processed in
   their original form.
-Their active substitutions only occur \textit{afterwards}, when
+Their active substitutions arising from \macname{def}s 
+  only occur \textit{afterwards}, when
   the \TokCycle{} output is typeset.
-They may be identified in the Character directive with
-  \macname{ifactivetok}.
-
+They may be identified with the \macname{ifactivetok} test.
+If \macname{let} to a character, they may be identified in the Character
+  directive;
+If \macname{let} to a control sequence or defined via \macname{def},
+  they may be identified in the Macro directive.
 \let\svt T
 \Characterdirective{\tctestifcon\ifactivetok
-  {\addcytoks{\fbox{#1}}}{\addcytoks{#1}}}
+  {\addcytoks{\fbox{#1-chr}}}{\addcytoks{#1}}}
+\Macrodirective{\tctestifcon\ifactivetok
+  {\addcytoks{\fbox{#1-mac}}}{\addcytoks{#1}}}
 
 \begin{exampleA}[lefthand width=7.8cm]{Processing active characters}
 \resettokcycle
@@ -900,7 +991,9 @@ This is a test!!\endtokencyclexpress
 This is a test!!\endtokencyclexpress
 
 \Characterdirective{\tctestifcon\ifactivetok
-  {\addcytoks{\fbox{#1}}}{\addcytoks{#1}}}
+  {\addcytoks{\fbox{#1-chr}}}{\addcytoks{#1}}}
+\Macrodirective{\tctestifcon\ifactivetok
+  {\addcytoks{\fbox{#1-mac}}}{\addcytoks{#1}}}
 \tokencyclexpress 
 This is a test!!\endtokencyclexpress
 
@@ -913,24 +1006,29 @@ This is a test!!\endtokencyclexpress
 \end{exampleA}
 
 %%%
+
+\ddbend If the input stream is pre-\textit{expanded}, any active 
+  substitutions that are expandable (i.e., those involving \macname{def} 
+  as well as those \macname{let} to something expandable) 
+  are made before reaching \TokCycle{} processing.  
+\svt hey are, thus, no longer detected as active, unless 
+  \macname{noexpand} is applied before the pre-expansion.
+In this example, the \texttt{!} that is not \macname{noexpand}\kern.3pted
+  is converted to a \texttt{?} prior to reaching \TokCycle{}
+  processing (and thus, not detected as \macname{active}):
+
 \catcode`!=\active
 \def !{?}
 \catcode`T=\active
 \let T+
-\ddbend If the input stream is pre-\textit{expanded}, the active 
-  substitutions involving \macname{def} (but not \macname{let}) 
-  are made before reaching {\catcode`T=11 \TokCycle} processing.  
-Such tokens are no longer detected as active, unless 
-  \macname{noexpand} is applied to the pre-expansion:
-
 \begin{exampleA}[lefthand width=7.8cm]{Expanded input stream acts 
-  upon active characters unless \macname{noexpand} is applied}
+  upon active \macname{def}ed characters unless \macname{noexpand} is applied}
 \expandedtokcyclexpress{This is a test!\noexpand!}
 \the\cytoks\par
 \detokenize\expandafter{\the\cytoks}
 \end{exampleA}
 
-However, pre-\textit{tokenization} does not suffer this behavior:
+However, pre-tokenization does not suffer this behavior:
 \begin{exampleA}[lefthand width=7.8cm]{Pre-tokenized input stream
   does not affect active characters}
 \def\tmp{This is a test!!}
@@ -943,6 +1041,41 @@ However, pre-\textit{tokenization} does not suffer this behavior:
 \endgroup
 \end{minipage}
 
+\begingroup
+\ddbend One aspect of \TeX{} to remember is that catcodes
+  are assigned at tokenization; however, for active characters,
+  the substitution assignment is evaluated only upon execution.
+So, if a cat-13 token is placed into a \macname{def}, it will
+  remain active even if the catcode of that character code is
+  later changed.
+But if the cat-13 active definition is changed prior to the 
+  execution of the \macname{def}'ed token, the revised
+  token assignment will apply.
+
+The following example demonstrates this concept, while showing,
+  whithout changing the input in any way, that \TokCycle{} 
+  can properly digest active and implicit grouping (cat-1,2) characters:
+
+\begin{exampleA}[lefthand width=8cm]{Active and implicit grouping
+  tokens digestible by \TokCycle}
+\catcode`Y=13
+\catcode`Z=13
+\let Y{
+\let Z}
+\let\Y{
+\let\Z}
+\def\tmp{\textit YabcZ de\Y\itshape f\Zg}%
+
+\def Y{\bgroup[NEW]}% APPLIES AT EXECUTION
+\catcode`Y=11% DOES NOT AFFECT Y IN \tmp
+
+\expandafter\tokcyclexpress\expandafter{\tmp}
+\the\cytoks
+
+\detokenize\expandafter{\the\cytoks}
+\end{exampleA}
+\endgroup
+
 \subsubsection{Trap Catcode 6 (explicit \& implicit) tokens}
 
 \begingroup
@@ -958,11 +1091,11 @@ Since they are unlikely to be used in that capacity inside a
   into something cat-12 and set the if-condition \macname{catSIXtrue}.
 In this manner, \macname{ifcatSIX} can be used inside the
   Character directive to convert cat-6 tokens into something of
-  the user's choosing.\smallskip
+  the user's choosing.
 
 As to this cat-12 conversion, explicit cat-6 characters are converted
   into the same character with cat-12.  
-On the other hand, implicit cat-6 macros (e.g., \verb|\let\myhash#|) 
+On the other hand, implicit cat-6 control sequences (e.g., \verb|\let\myhash#|) 
   are converted into a fixed-name macro, \macname{implicitsixtok}, 
   whose cat-12 substitution text is a \macname{string} of the original
   implicit-macro name.
@@ -1018,10 +1151,6 @@ A package macro, \macname{whennotprocessingparameter\#1\{<}%
 In this example, a macro is defined and then executed, subject to
   token replacements brought about by the expandable Character directive.
 
-\smallskip
-In the following example, see section~\ref{s:unex} for 
- the \macname{vowelcap} definition.
-
 \Characterdirective{%
   \whennotprocessingparameter#1{\addcytoks[x]{\vowelcap{#1}}}}
 \aftertokcycle{\global\cytoks\expandafter{\the\cytoks}}
@@ -1048,17 +1177,24 @@ In the following example, see section~\ref{s:unex} for
 
 \begingroup
 
-Implicit macros (assigned via \macname{let}) were already 
-  mentioned in the context of cat-6.
-However, implicit macros can be of any valid catcode.
+Implicit control sequences (assigned via \macname{let} to characters) 
+  were already mentioned in the context of cat-6.
+However, implicit control sequences can be of any valid catcode (except
+  for cat-0, which we instead call macros or primitives).
 The condition \macname{ifimplicittok} is used to flag such
-  tokens for special processing.
+  tokens for special processing, as well as active tokens that
+  are \macname{let} to anything unexpandable.
+
+In the next example, implicit, cat-6 and implicit-cat-6 tokens may
+  all be differentiated, shown here with a multiplicity of 
+  \macname{fbox}es.
 
 \fboxsep=1.5pt
 
 \begin{exampleA}[lefthand width=8cm]{Implicit\ensuremath{{}={}}%
   single box\char44 \ cat-6\ensuremath{{}={}}double box\char44 \ %
   implicit-cat-6\ensuremath{{}={}}triple box}
+\catcode`Q=\active \let QN
 \let\littlet=t
 \let\littlel=l
 \let\svhash#
@@ -1073,10 +1209,48 @@ The condition \macname{ifimplicittok} is used to flag such
 
 This \textit{is a \textbf{big}} \littlet est.
 
-Next pa#agraph ending with implicit cat six 
+Qext pa#agraph ending with implicit cat six 
   \svhash.\endtokencyclexpress
 \end{exampleA}
 
+In the following example, we use both control sequences and active characters
+ in \macname{def} and \macname{let} capacities, to demonstrate how 
+  \TokCycle{} digests things.
+Implicit tokens (tokens \macname{let} to characters) are shown in a box,
+  with both the token name and the implicit value (note that tokens 
+  \macname{let} to macros and primitives are not considered implicit).
+Active tokens processed through the character directive are followed with
+  a \dag, whereas those processed through the macro directive are followed
+  with a \ddag.
+
+
+\begin{exampleA}[lefthand width=7.5cm]{Non-active vs{.} active \macname{def} 
+  \& \macname{let}}
+\Characterdirective{\ifimplicittok
+  \addcytoks{\fbox{\detokenize{#1}:#1}}%
+  \else\addcytoks{#1}\fi\ifactivetok
+  \addcytoks{\rlap{\dag}}\fi\addcytoks{~\,}}
+\Macrodirective{\ifimplicittok
+  \addcytoks{\fbox{\detokenize{#1}}}%
+  \else\addcytoks{#1}\fi\ifactivetok
+  \addcytoks{\rlap{\ddag}}\fi
+  \ifx\par#1\else\addcytoks{~\,}\fi}
+
+\def\A{a}
+\let\B i
+\let\C\today
+\let\D\relax
+\def\E{\relax}
+\catcode`V=13 \def V{a}
+\catcode`W=13 \let Ww
+\catcode`X=13 \let X\today
+\catcode`Y=13 \let Y\relax
+\catcode`Z=13 \def Z{\relax}
+\tokcyclexpress{\A\B\C\D\E ab\par VWXYZab}
+\the\cytoks
+\end{exampleA}
+
+
 \ddbend If the input stream is subject to pre-expansion, one
   will require \macname{noexpand} for macros where no pre-expansion
   is desired.\medskip
@@ -1093,7 +1267,7 @@ Next pa#agraph ending with implicit cat six
 Changing grouping tokens (catcodes 1,2) may require something more,
   if the output stream is to be detokenized.  
 In the following examples, pay attention to the
-  detokenized grouping around the argument to \macname{fbox}.\smallskip
+  detokenized grouping around the argument to \macname{fbox}.
 
 As we will see, the issues raised here only affect the situation
   when detokenization of the output stream is required.