summaryrefslogtreecommitdiff
path: root/macros/generic/tokcycle/tokcycle-examples.tex
diff options
context:
space:
mode:
authorNorbert Preining <norbert@preining.info>2020-10-02 03:03:29 +0000
committerNorbert Preining <norbert@preining.info>2020-10-02 03:03:29 +0000
commit37f9a57b2f39142fbccb20b758af8e0502499671 (patch)
tree3ad423febb91fa4309d19667a0423ae94e2ecb89 /macros/generic/tokcycle/tokcycle-examples.tex
parentd2a6bb90e56a2836bb59477b8f27248aab9149f5 (diff)
CTAN sync 202010020303
Diffstat (limited to 'macros/generic/tokcycle/tokcycle-examples.tex')
-rw-r--r--macros/generic/tokcycle/tokcycle-examples.tex73
1 files changed, 67 insertions, 6 deletions
diff --git a/macros/generic/tokcycle/tokcycle-examples.tex b/macros/generic/tokcycle/tokcycle-examples.tex
index 0c46129bbd..49fbe6c66c 100644
--- a/macros/generic/tokcycle/tokcycle-examples.tex
+++ b/macros/generic/tokcycle/tokcycle-examples.tex
@@ -194,6 +194,8 @@
\newcommand\nextcmptok[1]{\rotoradd{\rotcytoks{#1}}{\addcytoks{#1}}\testmatch{\tmp}}
\newcommand\nextctltok[1]{\stepcounter{runcount}\addcytoks{#1}}
+\newcommand\detokenizespacetok[1]{\tctestifcon{\ifimplicittok}%
+ {\tctestifcon{\ifactivetok}{\theactivespace}{\string#1}}{\detokenize{#1}}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Examples, examples, and more examples}
@@ -873,6 +875,8 @@ If \macname{let} to a character, they may be identified in the Character
directive;
If \macname{let} to a control sequence or defined via \macname{def},
they may be identified in the Macro directive.
+For information on how to process active spaces, please refer to
+ section~\ref{s:spc}.
\let\svt T
\Characterdirective{\tctestifcon\ifactivetok
{\addcytoks{\fbox{#1-chr}}}{\addcytoks{#1}}}
@@ -1257,20 +1261,35 @@ Bottom line:
\endgroup
-\subsubsection{Catcode 10 space tokens}
+\subsubsection{Catcode 10 space tokens\label{s:spc}}
\begingroup
Here we demonstrate that \TokCycle{} can handle arbitrary redesignation
- of tokens to cat-10, as well as implicit space tokens.
+ of tokens to cat-10, as well as implicit space tokens (both implicit
+ macro spaces and active-implicit character spaces).
\ddbend While it should seem natural, we note that implicit
space tokens are directed to the Space directive rather than
the Character directive.
However, \macname{ifimplicittok} may still be used to
differentiate an explicit space from an implicit one.
-\medskip
+
+\ddbend
+Also, if the implicit space is an active character, \macname{ifactivetok}
+ is also set, for the user's benefit.
+Likewise, \macname{ifactivechar} is also checked for active spaces, to see
+ if the charcode of the active space is still, indeed, active.
+While \verb|#1| may be used to represent the space in the space directive,
+ a special technique is required to recover the detokenized name of the
+ active space character.
+In particular, \verb|#1| will, for active spaces, contain a generic
+ implicit space token, \macname{tc@sptoken}.
+However, the name of the associated active character in the input stream
+ will be defined as a cat-12 token in \macname{theactivespace}.
+
+\medskip
Note in the following examples that cat-10 tokens do \textit{not}
get under-dots.
The next three examples all use the same input, but with different
@@ -1313,19 +1332,61 @@ catcode settings for the space and the underscore.
\fbox{a_c d} b_g\itshape f\upshape\endtokencycle
\end{exampleA}
+The next two examples introduce implicit and active-implicit spaces.
+The latter example also demonstrates detokenization of such spaces.
+
+\medskip
+Here is a useful macro for detokenizing space tokens in the context of
+ \TokCycle{}.
+It can process not only explicit cat-10 spaces, but also implicit and
+ active-implicit spaces.
+To add it to the \macname{cytoks} token list, make sure you use the
+ \verb|[x]| fully expanded option to \macname{addcytoks}.
+
+\medskip
+\begin{exampleC}[]{The \macname{detokenizespacetok} macro for handling
+ implicit and active-implicit spaces}
+\newcommand\detokenizespacetok[1]{\tctestifcon{\ifimplicittok}%
+ {\tctestifcon{\ifactivetok}{\theactivespace}{\string#1}}{\detokenize{#1}}}
+\end{exampleC}
+\medskip
+
%%%
\resettokcycle
\Characterdirective{\addcytoks{\underdot{#1}}}
-\begin{exampleA}[lefthand width=10cm]{Implicit spaces work{,} too}
+\begin{exampleA}[lefthand width=10cm]{Implicit spaces also work}
\resettokcycle
\Characterdirective{\addcytoks{\underdot{#1}}}
-\def\:{\let\mysptoken= } \: %
+\def\:{\let\z= } \: %
\catcode`\_=10 %
\catcode`\ =12 %
\tokencyclexpress
-\fbox{a\mysptoken{}c d} b_g\itshape f\upshape
+\fbox{a\z{}c d} b_g\itshape f\upshape
+\endtokencyclexpress
+\end{exampleA}
+
+\begin{exampleA}[lefthand width=10cm]{Active Implicit spaces work{,} too}
+\resettokcycle
+\Characterdirective{\addcytoks{\underdot{#1}}}
+\def\:{\let\z= } \: %
+\catcode`Q=\active %
+\def\:{\let Q= } \: %
+\catcode`\_=10 %
+
+\tokencyclexpress
+xQx x_x\z{}x%
+\endtokencyclexpress
+
+\resettokcycle
+\Spacedirective{\addcytoks{\textcolor{cyan}\bgroup}%
+ \addcytoks[x]{(\detokenizespacetok{#1})}%
+ \addcytoks{\egroup}}
+
+\medskip
+\tokencyclexpress
+xQx x_x\z{}x%
\endtokencyclexpress
\end{exampleA}