summaryrefslogtreecommitdiff
path: root/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel
diff options
context:
space:
mode:
authorKarl Berry <karl@freefriends.org>2021-03-06 21:58:15 +0000
committerKarl Berry <karl@freefriends.org>2021-03-06 21:58:15 +0000
commit974640d66e61e81cb197ad96fdff7b08343e4c5a (patch)
tree2e1f75f32f312b7f24ba82b4590ae230bcd6f399 /Master/texmf-dist/doc/context/sources/general/manuals/lowlevel
parentb4fa72e61230aca75f7f6fbf988821f71edfb6b2 (diff)
context
git-svn-id: svn://tug.org/texlive/trunk@58167 c570f23f-e606-0410-a88d-b1316a301751
Diffstat (limited to 'Master/texmf-dist/doc/context/sources/general/manuals/lowlevel')
-rw-r--r--Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-boxes.tex18
-rw-r--r--Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-characters.tex256
-rw-r--r--Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-conditionals.tex182
-rw-r--r--Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-expansion.tex40
-rw-r--r--Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-grouping.tex170
-rw-r--r--Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-macros.tex904
-rw-r--r--Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-paragraphs.tex948
-rw-r--r--Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-scope.tex341
-rw-r--r--Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex254
9 files changed, 3087 insertions, 26 deletions
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-boxes.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-boxes.tex
index 986d07b1b61..9a097b8780c 100644
--- a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-boxes.tex
+++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-boxes.tex
@@ -9,9 +9,7 @@
[title=boxes,
color=middlered]
-\startsection[title=Preamble]
-
-\startsubsection[title=Introduction]
+\startsection[title=Introduction]
An average \CONTEXT\ user will not use the low level box primitives but a basic
understanding of how \TEX\ works doesn't hurt. In fact, occasionally using a box
@@ -25,9 +23,9 @@ about all kind of glues, kerns and penalties, just boxes it is.
This explanation will be extended when I feel the need (or users have questions
that can be answered here).
-\stopsubsection
+\stopsection
-\startsubsection[title=Boxes]
+\startsection[title=Boxes]
This paragraph of text is made from lines that contain words that themselves
contain symbolic representations of characters. Each line is wrapped in a so
@@ -68,7 +66,7 @@ other hand wraps a linked list of so called nodes: glyphs, kerns, glue,
penalties, rules, boxes, etc. It is a container with properties like width,
height, depth and shift.
-\stopsubsection
+\stopsection
\stopsection
@@ -469,10 +467,10 @@ a private attribute we define one.
\startbuffer
\newattribute\MyAt
\setbox\scratchbox\hbox attr \MyAt 123 {whatever}
-[\the\boxattr\scratchbox\MyAt]
-\boxattr\scratchbox\MyAt 456
-[\the\boxattr\scratchbox\MyAt]
-[\ifnum\boxattr\scratchbox\MyAt>400 okay\fi]
+[\the\boxattribute\scratchbox\MyAt]
+\boxattribute\scratchbox\MyAt 456
+[\the\boxattribute\scratchbox\MyAt]
+[\ifnum\boxattribute\scratchbox\MyAt>400 okay\fi]
\stopbuffer
\typebuffer[option=TEX]
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-characters.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-characters.tex
new file mode 100644
index 00000000000..ee3ab404442
--- /dev/null
+++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-characters.tex
@@ -0,0 +1,256 @@
+% language=us
+
+\environment lowlevel-style
+
+\startdocument
+ [title=characters,
+ color=middlered]
+
+\startsection[title=Introduction]
+
+This explanation is part of the low level manuals because in practice users will
+not have to deal with these matters in \MKIV\ and even less in \LMTX. You can
+skip to the last section for commands.
+
+\stopsection
+
+\startsection[title=History]
+
+If we travel back in time to when \TEX\ was written we end up in eight bit
+character universe. In fact, the first versions assumed seven bits, but for
+comfortable use with languages other than English that was not sufficient.
+Support for eight bits permits the usage of so called code pages as supported by
+operating systems. Although \ASCII\ input became kind of the standard soon
+afterwards, the engine can be set up for different encodings. This is not only
+true for \TEX, but for many of its companions, like \METAFONT\ and therefore
+\METAPOST. \footnote {This remapping to an internal representation (e.g. ebcdic)
+is not present in \LUATEX\ where we assume \UTF8 to be the input encoding. The
+\METAPOST\ library that comes with \LUATEX\ still has that code but in
+\LUAMETATEX\ it's gone. There one can set up the machinery to be \UTF8 aware
+too.}
+
+Core components of a \TEX\ engine are hyphenation of words, applying
+inter|-|character kerns and build ligatures. In traditional \TEX\ engines those
+processes are interwoven into the par builder but in \LUATEX\ these are separate
+stages. The original approach is the reason that there is a relation between the
+input encoding and the font encoding: the character in the input is the slot used
+in a reference to a glyph. When producing the final result (e.g.\ \PDF) there can
+also be a mapping to an index in a font resource.
+
+\starttyping
+input A [tex ->] font slot A [backend ->] glyph index A
+\stoptyping
+
+The mapping that \TEX\ does is normally one|-|to|-|one but an input character can
+undergo some transformation. For instance a character beyond \ASCII\ 126 can be
+made active and expand to some character number that then becomes the font slot.
+So, it is the expansion (or meaning) of a character that end up as numeric
+reference in the glyph node. Virtual fonts can introduce yet another remapping
+but that's only visible in the backend.
+
+Actually, in \LUATEX\ the same happens but in practice there is no need to go
+active because (at least in \CONTEXT) we assume a \UNICODE\ path so there the
+font slot is the \UNICODE\ got from the \UTF8 input.
+
+In the eight bit universe macro packages (have to) provide all kind of means to
+deal with (in the perspective of English) special characters. For instance, \type
+{\"a} would put a diaeresis on top of the a or even better, refer to a character
+in the encoding that the chosen font provides. Because there are some limitations
+of what can go in an eight bit font, and because in different countries the used
+\TEX\ fonts evolved kind of independent, we ended up with quite some different
+variants of fonts. It was only with the Latin Modern project that this became
+better. Interesting is that when we consider the fact that such a font has often
+also hardly used symbols (like registered or copyright) coming up with an
+encoding vector that covers most (latin based) European languages (scripts) is
+not impossible \footnote {And indeed in the Latin Modern project we came up with
+one but it was already to late for it to become popular.} Special symbols could
+simply go into a dedicated font, also because these are always accessed via a
+macro so who cares about the input. It never happened.
+
+Keep in mind that when \UTF8 is used with eight bit engines, \CONTEXT\ will
+convert sequences of characters into a slot in a font (depending on the font
+encoding used which itself depends on the coverage needed). For this every first
+(possible) byte of a multibyte \UTF\ sequence is an active character, which is no
+big deal because these are outside the \ASCII\ range. Normal \ASCII\ characters
+are single byte \UTF\ sequences and fall through without treatment.
+
+Anyway, in \CONTEXT\ \MKII\ we dealt with this by supporting mixed encodings,
+depending on the (local) language, referencing the relevant font. It permits
+users to enter the text in their preferred input encoding and also get the words
+properly hyphenated. But we can leave these \MKII\ details behind.
+
+\stopsection
+
+\startsection[title=The heritage]
+
+In \MKIV\ we got rid of input and font encodings, although one can still load
+files in a specific code page. \footnote {I'm not sure if users ever depend on an
+input encoding different from \UTF8.} We also kept the means to enter special
+characters, if only because text editors seldom support(ed) a wide range of
+visual editing of those. This is why we still have
+
+\starttyping[option=TEX]
+\"u \^a \v{s} \AE \ij \eacute \oslash
+\stoptyping
+
+and many more. The ones with one character names are rather common in the \TEX\
+community but it is definitely a weird mix of symbols. The next two are kind of
+outdated: in these days you delegate that to the font handler, where turning them
+into \quote {single} character references depends on what the font offers, how it
+is set up with respect to (for instance) ligatures, and even might depend on
+language or script.
+
+The ones with the long names partly are tradition, but as we have a lot of them,
+in \MKII\ they actually serve a purpose. These verbose names are used in the so
+called encoding vectors and are part of the \UTF\ expansion vectors. They are
+also used in labels so that we have a good indication if what goes in there:
+remember that in those times editors often didn't show characters, unless the
+font for display had them, or the operating system somehow provided them from
+another font. These verbose names are used for latin, greek and cyrillic and for
+some other scripts and symbols. They take up quite a bit of hash space and the
+format file. \footnote {In \MKII\ we have an abstract front|-|end with respect to
+encodings and also an abstract backend with respect to supported drivers but both
+approaches no longer make sense today.}
+
+\stopsection
+
+\startsection[title=The \LMTX\ approach]
+
+In the process of tagging all (public) macros in \LMTX\ (which happened in
+2020|-|2021) I wondered if we should keep these one character macros, the
+references to special characters and the verbose ones. When asked on the mailing
+list it became clear that users still expect the short ones to be present, often
+just because old \BIBTEX\ files are used that might need them. However, in \MKIV\
+and \LMTX\ we load \BIBTEX\ files in a way that turn these special character
+references into proper \UTF8 input so it makes a weak argument. Anyway, although
+they could go, for now we keep them because users expect them. However, in \LMTX\
+the implementation is somewhat different now, a bit more efficient in terms of
+hash and memory, potentially a bit less efficient in runtime, but no one will
+notice that.
+
+A new command has been introduced, the very short \type {\chr}.
+
+\startbuffer
+\chr {à} \chr {á} \chr {ä}
+\chr {`a} \chr {'a} \chr {"a}
+\chr {a acute} \chr {a grave} \chr {a umlaut}
+\chr {aacute} \chr {agrave} \chr {aumlaut}
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+In the first line the composed character using two characters, a base and a so
+called mark. Actually, one doesn't have to use \type {\chr} in that case because
+\CONTEXT\ does already collapse characters for you. The second line looks like
+the shortcuts \type {\`}, \type {\'} and \type {\"}. The third and fourth lines
+could eventually replace the more symbolic long names, if we feel the need. Watch
+out: in \UNICODE\ input the marks come {\em after}.
+
+\startlines \getbuffer \stoplines
+
+Currently the repertoire is somewhat limited but it can be easily be extended. It
+all depends on user needs (doing Greek and Cyrillic for instance). The reason why
+we actually save code deep down is that the helpers for this have always been
+there. \footnote {So if needed I can port this approach back to \MKIV, but for
+now we keep it as is because we then have a reference.}
+
+The \type {\"} commands are now just aliases to more verbose and less hackery
+looking macros:
+
+\starttabulate[|||||]
+ \NC \type {\withgrave} \NC \withgrave {a} \NC \type {\`} \NC \`{a} \NC \NR
+ \NC \type {\withacute} \NC \withacute {a} \NC \type {\'} \NC \'{a} \NC \NR
+ \NC \type {\withcircumflex} \NC \withcircumflex {a} \NC \type {\^} \NC \^{a} \NC \NR
+ \NC \type {\withtilde} \NC \withtilde {a} \NC \type {\~} \NC \~{a} \NC \NR
+ \NC \type {\withmacron} \NC \withmacron {a} \NC \type {\=} \NC \={a} \NC \NR
+ \NC \type {\withbreve} \NC \withbreve {e} \NC \type {\u} \NC \u{e} \NC \NR
+ \NC \type {\withdotaccent} \NC \withdot {c} \NC \type {\.} \NC \.{c} \NC \NR
+ \NC \type {\withdiaeresis} \NC \withdieresis {e} \NC \type {\"} \NC \"{e} \NC \NR
+ \NC \type {\withring} \NC \withring {u} \NC \type {\r} \NC \r{u} \NC \NR
+ \NC \type {\withhungarumlaut} \NC \withhungarumlaut{u} \NC \type {\H} \NC \H{u} \NC \NR
+ \NC \type {\withcaron} \NC \withcaron {e} \NC \type {\v} \NC \v{e} \NC \NR
+ \NC \type {\withcedilla} \NC \withcedilla {e} \NC \type {\c} \NC \c{e} \NC \NR
+ \NC \type {\withogonek} \NC \withogonek {e} \NC \type {\k} \NC \k{e} \NC \NR
+\stoptabulate
+
+Not all fonts have these special characters. Most natural is to have them
+available as precomposed single glyphs, but it can be that they are just two
+shapes with the marks anchored to the base. It can even be that the font somehow
+overlays them, assuming (roughly) equal widths. The \type {compose} font feature
+in \CONTEXT\ normally can handle most well.
+
+An occasional ugly rendering doesn't matter that much: better have something than
+nothing. But when it's the main language (script) that needs them you'd better
+look for a font that handles them. When in doubt, in \CONTEXT\ you can enable
+checking:
+
+\starttabulate[|l|l|]
+ \BC command \BC equivalent to \NC \NR
+ \NC \type {\checkmissingcharacters} \NC \type{\enabletrackers[fonts.missing]} \NC \NR
+ \NC \type {\removemissingcharacters} \NC \type{\enabletrackers[fonts.missing=remove]} \NC \NR
+ \NC \type {\replacemissingcharacters} \NC \type{\enabletrackers[fonts.missing=replace]} \NC \NR
+ \NC \type {\handlemissingcharacters} \NC \type{\enabletrackers[fonts.missing={decompose,replace}]} \NC \NR
+\stoptabulate
+
+The decompose variant will try to turn a composed character into its components
+so that at least you get something. If that fails it will inject a replacement
+symbol that stands out so that you can check it. The console also mentions
+missing glyphs. You don't need to enable this by default \footnote {There is some
+overhead involved here.} but you might occasionally do it when you use a font for
+the first time.
+
+In \LMTX\ this mechanism has been upgraded so that replacements follow the shape
+and are actually real characters. The decomposition has not yet been ported back
+to \MKIV.
+
+The full list of commands can be queried when a tracing module is loaded:
+
+\startbuffer
+\usemodule[s][characters-combinations]
+
+\showcharactercombinations
+\stopbuffer
+
+\typebuffer
+
+We get this list:
+
+\getbuffer
+
+Some combinations are special for \CONTEXT\ because \UNICODE\ doesn't specify
+decomposition for all composed characters.
+
+\stopsection
+
+\startsubject[title=Colofon]
+
+\starttabulate
+\NC Author \NC Hans Hagen \NC \NR
+\NC \CONTEXT \NC \contextversion \NC \NR
+\NC \LUAMETATEX \NC \texengineversion \NC \NR
+\NC Support \NC www.pragma-ade.com \NC \NR
+\NC \NC contextgarden.net \NC \NR
+\stoptabulate
+
+\stopsubject
+
+\stopdocument
+
+% on an old machine, so consider them just relative measures
+%
+% mkiv lmtx
+%
+% 0.012 0.009 % faster core code
+% 0.028 0.036 % different io code path
+% 0.055 0.043 % different io code path / faster core code
+% 0.156 0.129 % more efficient resolving
+% 0.153 0.119 % more efficient resolving
+%
+% \ifdefined\withdieresis\else\let\withdieresis\"\fi % for mkiv
+%
+% \setbox0\hpack{\testfeatureonce{100000}{ü}} \par \elapsedtime \par % direct
+% \setbox0\hpack{\testfeatureonce{100000}{ü}} \par \elapsedtime \par % composed (input)
+% \setbox0\hpack{\testfeatureonce{100000}{u{}̈}} \par \elapsedtime \par % overlay
+% \setbox0\hpack{\testfeatureonce{100000}{\withdieresis{u}}} \par \elapsedtime \par % official also \"u
+% \setbox0\hpack{\testfeatureonce{100000}{\" u}} \par \elapsedtime \par % alias of previous
+
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-conditionals.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-conditionals.tex
index ea3c9e1a244..9be2fb4ec6f 100644
--- a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-conditionals.tex
+++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-conditionals.tex
@@ -6,6 +6,8 @@
[title=conditionals,
color=middleblue]
+\pushoverloadmode
+
\startsection[title=Preamble]
\startsubsection[title=Introduction]
@@ -1121,13 +1123,15 @@ This test is like \type {\ifcmpnum} but for dimensions.
\startsubsection[title={\tex{ifchkdim}}]
-This test is like \type {\ifchknum} but for dimensions.
+This test is like \type {\ifchknum} but for dimensions. The last checked value is
+available as \type {\lastchknum}.
\stopsubsection
\startsubsection[title={\tex{ifdimval}}]
-This test is like \type {\ifnumval} but for dimensions.
+This test is like \type {\ifnumval} but for dimensions. The last checked value is
+available as \type {\lastchkdim}
\stopsubsection
@@ -1224,6 +1228,15 @@ defined at the user level or is a build in one. This one might evolve.
\stopsubsection
+\startsubsection[title={\tex{ifarguments}}]
+
+This conditional can be used to check how many arguments were matched. It only
+makes sense when used with macros defined with the \type {\tolerant} prefix
+and|/|or when the sentinel \type {\ignorearguments} after the arguments is used.
+More details can be found in the lowlevel macros manual.
+
+\stopsubsection
+
\startsubsection[title={\tex{orelse}}]
This it not really a test primitive but it does act that way. Say that we have this:
@@ -1254,13 +1267,16 @@ A bit nicer looks this:
\fi
\stoptyping
-We stay at the same level and the only test that cannot be used this way is \type
-{\ifcondition} but that is no real problem. Sometimes a more flat test tree had
-advantages but if you think that it gives better performance then you will be
-disappointed. The fact that we stay at the same level is compensated by a bit
-more parsing, so unless you have millions such cases (or expansions) it might
-make a bit of a difference. As mentioned, I'm a bit sensitive for how code looks so
-that was the main motivation for introducing it.
+% We stay at the same level and the only test that cannot be used this way is \type
+% {\ifcondition} but that is no real problem. Sometimes a more flat test tree had
+
+We stay at the same level. Sometimes a more flat test tree had advantages but if
+you think that it gives better performance then you will be disappointed. The
+fact that we stay at the same level is compensated by a bit more parsing, so
+unless you have millions such cases (or expansions) it might make a bit of a
+difference. As mentioned, I'm a bit sensitive for how code looks so that was the
+main motivation for introducing it. There is a companion \type {\orunless}
+continuation primitive.
A rather neat trick is the definition of \type {\quitcondition}:
@@ -1294,18 +1310,23 @@ reason to go this complex but obscure \TEX\ code attracts some users so \unknown
When you have a macro that has for instance assignments, and when you expand that
macro inside an \type {\edef}, these assignments are not actually expanded but
-tokenized. In \LUATEX\ there is a way to immediately apply these assignments and
-that feature can be used to write a fully expandable user test. For instance:
+tokenized. In \LUAMETATEX\ there is a way to apply these assignments without side
+effects and that feature can be used to write a fully expandable user test. For
+instance:
\startbuffer
\def\truecondition {\iftrue}
\def\falsecondition{\iffalse}
\def\fontwithidhaschar#1#2%
- {\immediateassignment\scratchcounter\numexpr\fontid\font\relax
- \immediateassignment\setfontid\numexpr#1\relax
+ {\beginlocalcontrol
+ \scratchcounter\numexpr\fontid\font\relax
+ \setfontid\numexpr#1\relax
+ \endlocalcontrol
\iffontchar\font\numexpr#2\relax
- \immediateassignment\setfontid\scratchcounter
+ \beginlocalcontrol
+ \setfontid\scratchcounter
+ \endlocalcontrol
\expandafter\truecondition
\else
\expandafter\falsecondition
@@ -1394,6 +1415,125 @@ with \type {\ifcondition} (it has bitten me already a few times).
\stopsection
+\startsection[title=Relaxing]
+
+When \TEX\ scans for a number or dimension it has to check tokens one by one. On
+the case of a number, the scanning stops when there is no digit, in the case of a
+dimension the unit determine the end of scanning. In the case of a number, when a
+token is not a digit that token gets pushed back. When digits are scanned a
+trailing space or \type {\relax} is pushed back. Instead of a number of dimension
+made from digits, periods and units, the scanner also accepts registers, both the
+direct accessors like \type {\count} and \type {\dimen} and those represented by
+one token. Take these definitions:
+
+\startbuffer
+\newdimen\MyDimenA \MyDimenA=1pt \dimen0=\MyDimenA
+\newdimen\MyDimenB \MyDimenB=2pt \dimen2=\MyDimenB
+\stopbuffer
+
+\typebuffer \getbuffer
+
+I will use these to illustrate the side effects of scanning. Watch the spaces
+in the result.
+
+% \startbuffer[a]
+% \testfeatureonce{1000000}{
+% \whatever{1pt}{2pt}%
+% \whatever{1pt}{1pt}%
+% \whatever{\dimen0}{\dimen2}%
+% \whatever{\dimen0}{\dimen0}%
+% \whatever\MyDimenA\MyDimenB
+% \whatever\MyDimenA\MyDimenB
+% } \elapsedtime
+% \stopbuffer
+
+\startbuffer[b]
+\starttabulate[|T|T|]
+\NC \type {\whatever{1pt}{2pt}} \NC \edef\temp{\whatever {1pt}{2pt}}[\meaning\temp] \NC \NR
+\NC \type {\whatever{1pt}{1pt}} \NC \edef\temp{\whatever {1pt}{1pt}}[\meaning\temp] \NC \NR
+\NC \type {\whatever{\dimen0}{\dimen2}} \NC \edef\temp{\whatever{\dimen0}{\dimen2}}[\meaning\temp] \NC \NR
+\NC \type {\whatever{\dimen0}{\dimen0}} \NC \edef\temp{\whatever{\dimen0}{\dimen0}}[\meaning\temp] \NC \NR
+\NC \type {\whatever\MyDimenA\MyDimenB} \NC \edef\temp{\whatever\MyDimenA\MyDimenB}[\meaning\temp] \NC \NR
+\NC \type {\whatever\MyDimenA\MyDimenB} \NC \edef\temp{\whatever\MyDimenA\MyDimenB}[\meaning\temp] \NC \NR
+\stoptabulate
+\stopbuffer
+
+First I show what effect we want to avoid. When second argument contains a number
+(digits) the zero will become part of it so we actually check \type {\dimen00}
+here.
+
+\startbuffer[c]
+\def\whatever#1#2%
+ {\ifdim#1=#20\else1\fi}
+\stopbuffer
+
+\typebuffer[c] \getbuffer[c,b]
+
+The solution is to add a space but watch how that one can end up in the result:
+
+\startbuffer[c]
+\def\whatever#1#2%
+ {\ifdim#1=#2 0\else1\fi}
+\stopbuffer
+
+\typebuffer[c] \getbuffer[c,b]
+
+A variant is using \type {\relax} and this time we get this token retained in
+the output.
+
+\startbuffer[c]
+\def\whatever#1#2%
+ {\ifdim#1=#2\relax0\else1\fi}
+\stopbuffer
+
+\typebuffer[c] \getbuffer[c,b]
+
+A solution that doesn't have side effects of forcing the end of a number (using a
+space or \type {\relax} is one where we use expressions. The added overhead of
+scanning expressions is taken for granted because the effect is what we like:
+
+\startbuffer[c]
+\def\whatever#1#2%
+ {\ifdim\dimexpr#1\relax=\dimexpr#2\relax0\else1\fi}
+\stopbuffer
+
+\typebuffer[c] \getbuffer[c,b]
+
+Just for completeness we show a more obscure trick: this one hides assignments to
+temporary variables. Although performance is okay, it is the least efficient
+one so far.
+
+\ifdefined\beginlocalcontrol
+
+\startbuffer[c]
+\def\whatever#1#2%
+ {\beginlocalcontrol
+ \MyDimenA#1\relax
+ \MyDimenB#2\relax
+ \endlocalcontrol
+ \ifdim\MyDimenA=\MyDimenB0\else1\fi}
+\stopbuffer
+
+\typebuffer[c] \getbuffer[c,b]
+
+\fi
+
+It is kind of a game to come up with alternatives but for sure those involve
+dirty tricks and more tokens (and runtime). The next can be considered a dirty
+trick too: we use a special variant of \type {\relax}. When a number is scanned
+it acts as relax, but otherwise it just is ignored and disappears.
+
+\ifdefined\norelax\else\let\norelax\relax\fi
+
+\startbuffer[c]
+\def\whatever#1#2%
+ {\ifdim#1=#2\norelax0\else1\fi}
+\stopbuffer
+
+\typebuffer[c] \getbuffer[c,b]
+
+\stopsection
+
\startsubject[title=Colofon]
\starttabulate
@@ -1406,4 +1546,18 @@ with \type {\ifcondition} (it has bitten me already a few times).
\stopsubject
+\popoverloadmode
+
\stopdocument
+
+% \def\foo{foo=bar}
+% \def\oof{foo!bar}
+% \scratchtoks{=}
+
+% \ifhasxtoks\scratchtoks{foo!bar} YES\else NOP\fi\par
+% \ifhasxtoks\scratchtoks{foo=bar} YES\else NOP\fi\par
+
+% \showluatokens\foo
+
+% \ifhastoks\scratchtoks\oof YES\else NOP\fi\par
+% \ifhastoks\scratchtoks\foo YES\else NOP\fi\par
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-expansion.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-expansion.tex
index 1e2e00a35ad..15faa802dc4 100644
--- a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-expansion.tex
+++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-expansion.tex
@@ -315,7 +315,7 @@ Although you can never really get back to the original input, you can come prett
close, with:
\startbuffer
-\normaldetokenize{this can $ be anything \bgroup}
+\detokenize{this can $ be anything \bgroup}
\stopbuffer
\typebuffer[option=TEX]
@@ -432,9 +432,45 @@ These two macros now have the meaning:
\startsection[title={\LUAMETATEX\ primitives}]
-{\em todo}
+To be discussed:
+
+\starttyping
+\expand
+\expandtoken
+\localcontrol
+\localcontrolled
+\beginlocalcontrol ... \endlocalcontrol
+\immediate
+\stoptyping
+
+And maybe also here:
+
+\starttyping
+\tokenized : a bonus
+\scantokens : original etex, now using the lua method
+\stoptyping
% \aftergroups
+% \aftergrouped
+
+And:
+
+\starttyping
+ \def\foo{foo}
+\protected\def\oof{oof}
+
+\csname foo\endcsname
+\csname oof\endcsname
+\csname \foo\endcsname
+% \csname \oof\endcsname % error in luametatex
+
+\ifcsname foo\endcsname yes\else nop\fi
+\ifcsname oof\endcsname yes\else nop\fi
+\ifcsname \foo\endcsname yes\else nop\fi
+\ifcsname \oof\endcsname yes\else nop\fi % nop in luametatex
+\stoptyping
+
+\stoptext
\stopsection
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-grouping.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-grouping.tex
new file mode 100644
index 00000000000..eadcca6a9fd
--- /dev/null
+++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-grouping.tex
@@ -0,0 +1,170 @@
+% language=us
+
+\environment lowlevel-style
+
+\startdocument
+ [title=grouping,
+ color=middlecyan]
+
+\startsection[title=Introduction]
+
+This is a rather short explanation. I decided to write it after presenting the
+other topics at the 2019 \CONTEXT\ meeting where there was a question about
+grouping.
+
+\stopsection
+
+\startsection[title=\PASCAL]
+
+In a language like \PASCAL, the language that \TEX\ has been written in, or
+\MODULA, its successor, there is no concept of grouping like in \TEX. But we can
+find keywords that suggests this:
+
+\starttyping
+for i := 1 to 10 do begin ... end
+\stoptyping
+
+This language probably inspired some of the syntax of \TEX\ and \METAPOST. For
+instance an assignment in \METAPOST\ uses \type {:=} too. However, the \type
+{begin} and \type {end} don't really group but define a block of statements. You
+can have local variables in a procedure or function but the block is just a way
+to pack a sequence of statements.
+
+\stopsection
+
+\startsection[title=\TEX]
+
+In \TEX\ macros (or source code) the following can occur:
+
+\starttyping
+\begingroup
+ ...
+\endgroup
+\stoptyping
+
+as well as:
+
+\starttyping
+\bgroup
+ ...
+\egroup
+\stoptyping
+
+Here we really group in the sense that assignments to variables inside a group
+are forgotten afterwards. All assignments are local to the group unless they are
+explicitly done global:
+
+\starttyping
+\scratchcounter=1
+\def\foo{foo}
+\begingroup
+ \scratchcounter=2
+ \global\globalscratchcounter=2
+ \gdef\foo{FOO}
+\endgroup
+\stoptyping
+
+Here \type {\scratchcounter} is still one after the group is left but its global
+counterpart is now two. The \type {\foo} macro is also changed globally.
+
+Although you can use both sets of commands to group, you cannot mix them, so this
+will trigger an error:
+
+\starttyping
+\bgroup
+\endgroup
+\stoptyping
+
+The bottomline is: if you want a value to persist after the group, you need to
+explicitly change its value globally. This makes a lot of sense in the perspective
+of \TEX.
+
+\stopsection
+
+\startsection[title=\METAPOST]
+
+The \METAPOST\ language also has a concept of grouping but in this case it's more like a
+programming language.
+
+\starttyping
+begingroup ;
+ n := 123 ;
+engroup ;
+\stoptyping
+
+In this case the value of \type {n} is 123 after the group is left, unless you do
+this (for numerics there is actually no need to declare them):
+
+\starttyping
+begingroup ;
+ save n ; numeric n ; n := 123 ;
+engroup ;
+\stoptyping
+
+Given the use of \METAPOST\ (read: \METAFONT) this makes a lot of sense: often
+you use macros to simplify code and you do want variables to change. Grouping in
+this language serves other purposes, like hiding what is between these commands
+and let the last expression become the result. In a \type {vardef} grouping is
+implicit.
+
+So, in \METAPOST\ all assignments are global, unless a variable is explicitly
+saved inside a group.
+
+\stopsection
+
+\startsection[title=\LUA]
+
+In \LUA\ all assignments are global unless a variable is defines local:
+
+\starttyping
+local x = 1
+local y = 1
+for i = 1, 10 do
+ local x = 2
+ y = 2
+end
+\stoptyping
+
+Here the value of \type {x} after the loop is still one but \type {y} is now two.
+As in \LUATEX\ we mix \TEX, \METAPOST\ and \LUA\ you can mix up these concepts.
+Another mixup is using \type {:=}, \type {endfor}, \type {fi} in \LUA\ after done
+some \METAPOST\ coding or using \type {end} instead of \type {endfor} in
+\METAPOST\ which can make the library wait for more without triggering an error.
+Proper syntax highlighting in an editor clearly helps.
+
+\stopsection
+
+\startsection[title=\CCODE]
+
+The \LUA\ language is a mix between \PASCAL\ (which is one reason why I like it)
+and \CCODE.
+
+\starttyping
+int x = 1 ;
+int y = 1 ;
+for (i=1; i<=10;i++) {
+ int x = 2 ;
+ y = 2 ;
+}
+\stoptyping
+
+The semicolon is also used in \PASCAL\ but there it is a separator and not a
+statement end, while in \METAPOST\ it does end a statement (expression).
+
+\stopsection
+
+\stopsection
+
+\startsubject[title=Colofon]
+
+\starttabulate
+\NC Author \NC Hans Hagen \NC \NR
+\NC \CONTEXT \NC \contextversion \NC \NR
+\NC \LUAMETATEX \NC \texengineversion \NC \NR
+\NC Support \NC www.pragma-ade.com \NC \NR
+\NC \NC contextgarden.net \NC \NR
+\stoptabulate
+
+\stopsubject
+
+\stopdocument
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-macros.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-macros.tex
new file mode 100644
index 00000000000..ef527732be3
--- /dev/null
+++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-macros.tex
@@ -0,0 +1,904 @@
+% language=us
+
+% Extending the macro argument parser happened stepwise and at each step a bit of
+% \CONTEXT\ code was adapted for testing. At the beginning of October the 20201010
+% version of \LUAMETATEX\ was more of less complete, and I decided to adapt some
+% more and more intrusive too. Of course that resulted in some more files than I
+% had intended so mid October about 100 files were adapted. When this works out
+% well, I'll do some more. In the process many macros got the frozen property so
+% that was also a test and we'll see how that works out (as it can backfire). As
+% usual, here is a musical timestamp: working on this happened when Pineapple Thief
+% released \quotation {Versions of the Truth} which again a magnificent drumming by
+% Gavin Harrison.
+
+
+% \permanent\tolerant\protected\def\xx[#1]#*#;[#2]#:#3% loops .. todo
+
+
+\usemodule[system-tokens]
+
+\environment lowlevel-style
+
+\startdocument
+ [title=macros,
+ color=middleorange]
+
+\startsection[title=Preamble]
+
+This chapter overlaps with other chapters but brings together some extensions to
+the macro definition and expansion parts. As these mechanisms were stepwise
+extended, the other chapters describe intermediate steps in the development.
+
+Now, in spite of the extensions discussed here the main ides is still that we
+have \TEX\ act like before. We keep the charm of the macro language but these
+additions make for easier definitions, but (at least initially) none that could
+not be done before using more code.
+
+\stopsection
+
+\startsection[title=Definitions]
+
+A macro definition normally looks like like this: \footnote {The \type
+{\dontleavehmode} command make the examples stay on one line.}
+
+\startbuffer[definition]
+\def\macro#1#2%
+ {\dontleavehmode\hbox to 6em{\vl\type{#1}\vl\type{#2}\vl\hss}}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+Such a macro can be used as:
+
+\startbuffer[example]
+\macro {1}{2}
+\macro {1} {2} middle space gobbled
+\macro 1 {2} middle space gobbled
+\macro {1} 2 middle space gobbled
+\macro 1 2 middle space gobbled
+\stopbuffer
+
+\typebuffer[example][option=TEX]
+
+We show the result with some comments about how spaces are handled:
+
+\startlines \getbuffer[example] \stoplines
+
+A definition with delimited parameters looks like this:
+
+\startbuffer[definition]
+\def\macro[#1]%
+ {\dontleavehmode\hbox to 6em{\vl\type{#1}\vl\hss}}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+When we use this we get:
+
+\startbuffer[example]
+\macro [1]
+\macro [ 1] leading space kept
+\macro [1 ] trailing space kept
+\macro [ 1 ] both spaces kept
+\stopbuffer
+
+\typebuffer[example][option=TEX]
+
+Again, watch the handling of spaces:
+
+\startlines \getbuffer[example] \stoplines
+
+Just for the record we show a combination:
+
+\startbuffer[definition]
+\def\macro[#1]#2%
+ {\dontleavehmode\hbox to 6em{\vl\type{#1}\vl\type{#2}\vl\hss}}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+With this:
+
+\startbuffer[example]
+\macro [1]{2}
+\macro [1] {2}
+\macro [1] 2
+\stopbuffer
+
+\typebuffer[example][option=TEX]
+
+we can again see the spaces go away:
+
+\startlines \getbuffer[example] \stoplines
+
+A definition with two separately delimited parameters is given next:
+
+\startbuffer[definition]
+\def\macro[#1#2]%
+ {\dontleavehmode\hbox to 6em{\vl\type{#1}\vl\type{#2}\vl\hss}}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+When used:
+
+\startbuffer[example]
+\macro [12]
+\macro [ 12] leading space gobbled
+\macro [12 ] trailing space kept
+\macro [ 12 ] leading space gobbled, trailing space kept
+\macro [1 2] middle space kept
+\macro [ 1 2 ] leading space gobbled, middle and trailing space kept
+\stopbuffer
+
+\typebuffer[example][option=TEX]
+
+We get ourselves:
+
+\startlines \getbuffer[example] \stoplines
+
+These examples demonstrate that the engine does some magic with spaces before
+(and therefore also between multiple) parameters.
+
+We will now go a bit beyond what traditional \TEX\ engines do and enter the
+domain of \LUAMETATEX\ specific parameter specifiers. We start with one that
+deals with this hard coded space behavior:
+
+\startbuffer[definition]
+\def\macro[#^#^]%
+ {\dontleavehmode\hbox to 6em{\vl\type{#1}\vl\type{#2}\vl\hss}}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+The \type {#^} specifier will count the parameter, so here we expect again two
+arguments but the space is kept when parsing for them.
+
+\startbuffer[example]
+\macro [12]
+\macro [ 12]
+\macro [12 ]
+\macro [ 12 ]
+\macro [1 2]
+\macro [ 1 2 ]
+\stopbuffer
+
+\typebuffer[example][option=TEX]
+
+Now keep in mind that we could deal well with all kind of parameter handling in
+\CONTEXT\ for decades, so this is not really something we missed, but it
+complements the to be discussed other ones and it makes sense to have that level
+of control. Also, availability triggers usage. Nevertheless, some day the \type
+{#^} specifier will come in handy.
+
+\startlines \getbuffer[example] \stoplines
+
+We now come back to an earlier example:
+
+\startbuffer[definition]
+\def\macro[#1]%
+ {\dontleavehmode\hbox spread 1em{\vl\type{#1}\vl\hss}}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+When we use this we see that the braces in the second call are removed:
+
+\startbuffer[example]
+\macro [1]
+\macro [{1}]
+\stopbuffer
+
+\typebuffer[example][option=TEX] \getbuffer[example]
+
+This can be prohibited by the \type {#+} specifier, as in:
+
+\startbuffer[definition]
+\def\macro[#+]%
+ {\dontleavehmode\hbox spread 1em{\vl\type{#1}\vl\hss}}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+As we see, the braces are kept:
+
+\startbuffer[example]
+\macro [1]
+\macro [{1}]
+\stopbuffer
+
+\typebuffer[example][option=TEX]
+
+Again, we could easily get around that (for sure intended) side effect but it just makes nicer
+code when we have a feature like this.
+
+\getbuffer[example]
+
+Sometimes you want to grab an argument but are not interested in the results. For this we have
+two specifiers: one that just ignores the argument, and another one that keeps counting but
+discards it, i.e.\ the related parameter is empty.
+
+\startbuffer[definition]
+\def\macro[#1][#0][#3][#-][#4]%
+ {\dontleavehmode\hbox spread 1em
+ {\vl\type{#1}\vl\type{#2}\vl\type{#3}\vl\type{#4}\vl\hss}}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+The second argument is empty and the fourth argument is simply ignored which is why we need
+\type {#4} for the fifth entry.
+
+\startbuffer[example]
+\macro [1][2][3][4][5]
+\stopbuffer
+
+\typebuffer[example][option=TEX]
+
+Here is proof that it works:
+
+\getbuffer[example]
+
+The reasoning behind dropping arguments is that for some cases we get around the
+nine argument limitation, but more important is that we don't construct token
+lists that are not used, which is more memory (and maybe even \CPU\ cache)
+friendly.
+
+Spaces are always kind of special in \TEX, so it will be no surprise that we have
+another specifier that relates to spaces.
+
+\startbuffer[definition]
+\def\macro[#1]#*[#2]%
+ {\dontleavehmode\hbox spread 1em{\vl\type{#1}\vl\type{#2}\vl\hss}}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+This permits usage like the following:
+
+\startbuffer[example]
+\macro [1][2]
+\macro [1] [2]
+\stopbuffer
+
+\typebuffer[example][option=TEX] \getbuffer[example]
+
+Without the optional \quote {grab spaces} specifier the second line would
+possibly throw an error. This because \TEX\ then tries to match \type{][} so the
+\type {] [} in the input is simply added to the first argument and the next
+occurrence of \type {][} will be used. That one can be someplace further in your
+source and if not \TEX\ complains about a premature end of file. But, with the
+\type {#*} option it works out okay (unless of course you don't have that second
+argument \type {[2]}.
+
+Now, you might wonder if there is a way to deal with that second delimited
+argument being optional and of course that can be programmed quite well in
+traditional macro code. In fact, \CONTEXT\ does that a lot because it is set up
+as a parameter driven system with optional arguments. That subsystem has been
+optimized to the max over years and it works quite well and performance wise
+there is very little to gain. However, as soon as you enable tracing you end up
+in an avalanche of expansions and that is no fun.
+
+This time the solution is not in some special specifier but in the way a macro
+gets defined.
+
+\startbuffer[definition]
+\tolerant\def\macro[#1]#*[#2]%
+ {\dontleavehmode\hbox spread 1em{\vl\type{#1}\vl\type{#2}\vl\hss}}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+The magic \type {\tolerant} prefix with delimited arguments and just quits when
+there is no match. So, this is acceptable:
+
+\startbuffer[example]
+\macro [1][2]
+\macro [1] [2]
+\macro [1]
+\macro
+\stopbuffer
+
+\typebuffer[example][option=TEX] \getbuffer[example]
+
+We can check how many arguments have been processed with a dedicated conditional:
+
+\startbuffer[definition]
+\tolerant\def\macro[#1]#*[#2]%
+ {\ifarguments 0\or 1\or 2\or ?\fi: \vl\type{#1}\vl\type{#2}\vl}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+We use this test:
+
+\startbuffer[example]
+\macro [1][2] \macro [1] [2] \macro [1] \macro
+\stopbuffer
+
+\typebuffer[example][option=TEX]
+
+The result is: \inlinebuffer[example]\ which is what we expect because we flush
+inline and there is no change of mode. When the following definition is used in
+display mode, the leading \type {n=} can for instance start a new paragraph and
+when code in \type {\everypar} you can loose the right number when macros get
+expanded before the \type {n} gets injected.
+
+\starttyping[option=TEX]
+\tolerant\def\macro[#1]#*[#2]%
+ {n=\ifarguments 0\or 1\or 2\or ?\fi: \vl\type{#1}\vl\type{#2}\vl}
+\stoptyping
+
+In addition to the \type {\ifarguments} test primitive there is also a related
+internal counter \type {\lastarguments} set that you can consult, so the \type
+{\ifarguments} is actually just a shortcut for \typ {\ifcase \lastarguments}.
+
+We now continue with the argument specifiers and the next two relate to this optional
+grabbing. Consider the next definition:
+
+\startbuffer[definition]
+\tolerant\def\macro#1#*#2%
+ {\dontleavehmode\hbox spread 1em{\vl\type{#1}\vl\type{#2}\vl\hss}}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+With this test:
+
+\startbuffer[example]
+\macro {1} {2}
+\macro {1}
+\macro
+\stopbuffer
+
+\typebuffer[example][option=TEX]
+
+We get:
+
+\getbuffer[example]
+
+This is okay because the last \type {\macro} is a valid (single token) argument. But, we
+can make the braces mandate:
+
+\startbuffer[definition]
+\tolerant\def\macro#=#*#=%
+ {\dontleavehmode\hbox spread 1em{\vl\type{#1}\vl\type{#2}\vl\hss}}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+Here the \type {#=} forces a check for braces, so:
+
+\startbuffer[example]
+\macro {1} {2}
+\macro {1}
+\macro
+\stopbuffer
+
+\typebuffer[example][option=TEX]
+
+gives this:
+
+\getbuffer[example]
+
+However, we do loose these braces and sometimes you don't want that. Of course when you pass the
+results downstream to another macro you can always add them, but it was cheap to add a related
+specifier:
+
+\startbuffer[definition]
+\tolerant\def\macro#_#*#_%
+ {\dontleavehmode\hbox spread 1em{\vl\type{#1}\vl\type{#2}\vl\hss}}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+Again, the magic \type {\tolerant} prefix works will quit scanning when there is
+no match. So:
+
+\startbuffer[example]
+\macro {1} {2}
+\macro {1}
+\macro
+\stopbuffer
+
+\typebuffer[example][option=TEX]
+
+leads to:
+
+\getbuffer[example]
+
+When you're tolerant it can be that you still want to pick up some argument
+later on. This is why we have a continuation option.
+
+\startbuffer[definition]
+\tolerant\def\foo [#1]#*[#2]#:#3{!#1!#2!#3!}
+\tolerant\def\oof[#1]#*[#2]#:(#3)#:#4{!#1!#2!#3!#4!}
+\tolerant\def\ofo [#1]#:(#2)#:#3{!#1!#2!#3!}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+Hopefully the next example demonstrates how it works:
+
+\startbuffer[example]
+\foo{3} \foo[1]{3} \foo[1][2]{3}
+\oof{4} \oof[1]{4} \oof[1][2]{4}
+\oof[1][2](3){4} \oof[1](3){4} \oof(3){4}
+\ofo{3} \ofo[1]{3}
+\ofo[1](2){3} \ofo(2){3}
+\stopbuffer
+
+\typebuffer[example][option=TEX]
+
+As you can see we can have multiple continuations using the \type {#:} directive:
+
+\startlines \getbuffer[example] \stoplines
+
+The last specifier doesn't work well with the \type {\ifarguments} state because
+we no longer know what arguments were skipped. This is why we have another test
+for arguments. A zero value means that the next token is not a parameter
+reference, a value of one means that a parameter has been set and a value of two
+signals an empty parameter. So, it reports the state of the given parameter as
+a kind if \type {\ifcase}.
+
+\startbuffer[definition]
+\def\foo#1#2{ [\ifparameter#1\or(ONE)\fi\ifparameter#2\or(TWO)\fi] }
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+\startbuffer[example]
+\foo{1}{2} \foo{1}{} \foo{}{2} \foo{}{}
+\stopbuffer
+
+Of course the test has to be followed by a valid parameter specifier:
+
+\typebuffer[example][option=TEX]
+
+The previous code gives this:
+
+\getbuffer[example]
+
+A combination check \type {\ifparameters}, again a case, matches the first
+parameter that has a value set.
+
+We could add plenty of specifiers but we need to keep in ind that we're not
+talking of an expression scanner. We need to keep performance in mind, so nesting
+and backtracking are no option. We also have a limited set of useable single
+characters, but here's one that uses a symbol that we had left:
+
+\startbuffer[definition]
+\def\startfoo[#/]#/\stopfoo{ [#1](#2) }
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+\startbuffer[example]
+\startfoo [x ] x \stopfoo
+\startfoo [ x ] x \stopfoo
+\startfoo [ x] x \stopfoo
+\startfoo [ x] \par x \par \par \stopfoo
+\stopbuffer
+
+The slash directive removes leading and trailing so called spacers as well as tokens
+that represent a paragraph end:
+
+\typebuffer[example][option=TEX]
+
+So we get this:
+
+\getbuffer[example]
+
+The next directive, the quitter \type {#;}, is demonstrated with an example. When
+no match has occurred, scanning picks up after this signal, otherwise we just
+quit.
+
+\startbuffer[example]
+\tolerant\def\foo[#1]#;(#2){/#1/#2/}
+
+\foo[1]\quad\foo[2]\quad\foo[3]\par
+\foo(1)\quad\foo(2)\quad\foo(3)\par
+
+\tolerant\def\foo[#1]#;#={/#1/#2/}
+
+\foo[1]\quad\foo[2]\quad\foo[3]\par
+\foo{1}\quad\foo{2}\quad\foo{3}\par
+
+\tolerant\def\foo[#1]#;#2{/#1/#2/}
+
+\foo[1]\quad\foo[2]\quad\foo[3]\par
+\foo{1}\quad\foo{2}\quad\foo{3}\par
+
+\tolerant\def\foo[#1]#;(#2)#;#={/#1/#2/#3/}
+
+\foo[1]\quad\foo[2]\quad\foo[3]\par
+\foo(1)\quad\foo(2)\quad\foo(3)\par
+\foo{1}\quad\foo{2}\quad\foo{3}\par
+\stopbuffer
+
+\typebuffer[example][option=TEX] \startpacked \getbuffer[example] \stoppacked
+
+I have to admit that I don't really need it but it made some macros that I was
+redefining behave better, so there is some self|-|interest here. Anyway, I
+considered some other features, like picking up a detokenized argument but I
+don't expect that to be of much use. In the meantime we ran out of reasonable
+characters, but some day \type {#?} and \type {#!} might show up, or maybe I find
+a use for \type {#<} and \type {#>}. A summary of all this is given here:
+
+\starttabulate[|T|i2l|]
+\FL
+\NC + \NC keep the braces \NC \NR
+\NC - \NC discard and don't count the argument \NC \NR
+\NC / \NC remove leading an trailing spaces and pars \NC \NR
+\NC = \NC braces are mandate \NC \NR
+\NC _ \NC braces are mandate and kept \NC \NR
+\NC ^ \NC keep leading spaces \NC \NR
+\ML
+\NC 1-9 \NC an argument \NC \NR
+\NC 0 \NC discard but count the argument \NC \NR
+\ML
+\NC * \NC ignore spaces \NC \NR
+\NC : \NC pick up scanning here \NC \NR
+\NC ; \NC quit scanning \NC \NR
+\LL
+\stoptabulate
+
+\stopsection
+
+\startsection[title=Runaway arguments]
+
+There is a particular troublesome case left: a runaway argument. The solution is
+not pretty but it's the only way: we need to tell the parser that it can quit.
+
+\startbuffer[definition]
+\tolerant\def\foo[#1=#2]%
+ {\ifarguments 0\or 1\or 2\or 3\or 4\fi:\vl\type{#1}\vl\type{#2}\vl}
+\stopbuffer
+
+\typebuffer[definition][option=TEX] \getbuffer[definition]
+
+\startbuffer[example]
+\dontleavehmode \foo[a=1]
+\dontleavehmode \foo[b=]
+\dontleavehmode \foo[=]
+\dontleavehmode \foo[x]\ignorearguments
+\stopbuffer
+
+The outcome demonstrates that one still has to do some additional checking for sane
+results and there are alternative way to (ab)use this mechanism. It all boils down
+to a clever combination of delimiters and \type {\ignorearguments}.
+
+\typebuffer[example][option=TEX]
+
+All calls are accepted:
+
+\startlines \getbuffer[example] \stoplines
+
+Just in case you wonder about performance: don't expect miracles here. On the one
+hand there is some extra overhead in the engine (when defining macros as well as
+when collecting arguments during a macro call) and maybe using these new features
+can sort of compensate that. As mentioned: the gain is mostly in cleaner macro
+code and less clutter in tracing. And I just want the \CONTEXT\ code to look
+nice: that way users can look in the source to see what happens and not drown in
+all these show|-|off tricks, special characters like underscores, at signs,
+question marks and exclamation marks.
+
+For the record: I normally run tests to see if there are performance side effects
+and as long as processing the test suite that has thousands of files of all kind
+doesn't take more time it's okay. Actually, there is a little gain in \CONTEXT\
+but that is to be expected, but I bet users won't notice it, because it's easily
+offset by some inefficient styling. Of course another gain of loosing some
+indirectness is that error messages point to the macro that the user called for
+and not to some follow up.
+
+\stopsection
+
+\startsection[title=Introspection]
+
+A macro has a meaning. You can serialize that meaning as follows:
+
+\startbuffer[definition]
+\tolerant\protected\def\foo#1[#2]#*[#3]%
+ {(1=#1) (2=#3) (3=#3)}
+
+\meaning\foo
+\stopbuffer
+
+\typebuffer[definition][option=TEX]
+
+The meaning of \type {\foo} comes out as:
+
+\startnarrower \getbuffer[definition] \stopnarrower
+
+When you load the module \type {system-tokens} you can also say:
+
+\startbuffer[example]
+\luatokentable\foo
+\stopbuffer
+
+\typebuffer[example][option=TEX]
+
+This produces a table of tokens specifications:
+
+{\getbuffer[definition]\getbuffer[example]}
+
+A token list is a linked list of tokens. The magic numbers in the first column
+are the token memory pointers. and because macros (and token lists) get recycled
+at some point the available tokens get scattered, which is reflected in the order
+of these numbers. Normally macros defined in the macro package are more sequential
+because they stay around from the start. The second and third row show the so
+called command code and the specifier. The command code groups primitives in
+categories, the specifier is an indicator of what specific action will follow, a
+register number a reference, etc. Users don't need to know these details. This
+macro is a special version of the online variant:
+
+\starttyping[option=TEX]
+\showluatokens\foo
+\stoptyping
+
+That one is always available and shows a similar list on the console. Again, users
+normally don't want to know such details.
+
+\startsection[title=nesting]
+
+You can nest macros, as in:
+
+\startbuffer
+\def\foo#1#2{\def\oof##1{<#1>##1<#2>}}
+\stopbuffer
+
+\typebuffer[option=TEX] \getbuffer
+
+At first sight the duplication of \type {#} looks strange but this is what
+happens. When \TEX\ scans the definition of \type {\foo} it sees two arguments.
+Their specification ends up in the preamble that defines the matching. When the
+body is scanned, the \type {#1} and \type {#2} are turned into a parameter
+reference. In order to make nested macros with arguments possible a \type {#}
+followed by another \type {#} becomes just one \type {#}. Keep in mind that the
+definition of \type {\oof} is delayed till the macro \type {\foo} gets expanded.
+That definition is just stored and the only thing that get's replaced are the two
+references to a macro parameter
+
+\luatokentable\foo
+
+Now, when we look at these details, it might become clear why for instance we
+have \quote {variable} names like \type {#4} and not \type {#whatever} (with or
+without hash). Macros are essentially token lists and token lists can be seen as
+a sequence of numbers. This is not that different from other programming
+environments. When you run into buzzwords like \quote {bytecode} and \quote
+{virtual machines} there is actually nothing special about it: some high level
+programming (using whatever concept, and in the case of \TEX\ it's macros)
+eventually ends up as a sequence of instructions, say bytecodes. Then you need
+some machinery to run over that and act upon those numbers. It's something you
+arrive at naturally when you play with interpreting languages. \footnote {I
+actually did when I wrote an interpreter for some computer assisted learning
+system, think of a kind of interpreted \PASCAL, but later realized that it was a a
+bytecode plus virtual machine thing. I'd just applied what I learned when playing
+with eight bit processors that took bytes, and interpreted opcodes and such.
+There's nothing spectacular about all this and I only realized decades later that
+the buzzwords describes old natural concepts.}
+
+So, internally a \type {#4} is just one token, a operator|-|operand combination
+where the operator is \quotation {grab a parameter} and the operand tells
+\quotation {where to store} it. Using names is of course an option but then one
+has to do more parsing and turn the name into a number \footnote {This is kind of
+what \METAPOST\ does with parameters to macros. The side effect is that in
+reporting you get \type {text0}, \type {expr2} and such reported which doesn't
+make things more clear.}, add additional checking in the macro body, figure out
+some way to retain the name for the purpose of reporting (which then uses more
+token memory or strings). It is simply not worth the trouble, let alone the fact
+that we loose performance, and when \TEX\ showed up those things really mattered.
+
+It is also important to realize that a \type {#} becomes either a preamble token
+(grab an argument) or a reference token (inject the passed tokens into a new
+input level). Therefore the duplication of hash tokens \type {##} that you see in
+macro nested bodies also makes sense: it makes it possible for the parser to
+distinguish between levels. Take:
+
+\starttyping[option=TEX]
+\def\foo#1{\def\oof##1{#1##1#1}}
+\stoptyping
+
+Of course one can think of this:
+
+\starttyping[option=TEX]
+\def\foo#fence{\def\oof#text{#fence#text#fence}}
+\stoptyping
+
+But such names really have to be unique then! Actually \CONTEXT\ does have an
+input method that supports such names, but discussing it here is a bit out of
+scope. Now, imagine that in the above case we use this:
+
+\starttyping[option=TEX]
+\def\foo[#1][#2]{\def\oof##1{#1##1#2}}
+\stoptyping
+
+If you're a bit familiar with the fact that \TEX\ has a model of category codes
+you can imagine that a predictable \quotation {hash followed by a number} is way
+more robust than enforcing the user to ensure that catcodes of \quote {names} are
+in the right category (read: is a bracket part of the name or not). So, say that
+we go completely arbitrary names, we then suddenly needs some escaping, like:
+
+\starttyping[option=TEX]
+\def\foo[#{left}][#{right}]{\def\oof#{text}{#{left}#{text}#{right}}}
+\stoptyping
+
+And, if you ever looked into macro packages, you will notice that they differ in
+the way they assign category codes. Asking users to take that into account when
+defining macros makes not that much sense.
+
+So, before one complains about \TEX\ being obscure (the hash thing), think twice.
+Your demand for simplicity for your coding demand will make coding more
+cumbersome for the complex cases that macro packages have to deal with. It's
+comparable using \TEX\ for input or using (say) mark down. For simple documents
+the later is fine, but when things become complex, you end up with similar
+complexity (or even worse because you lost the enforced detailed structure). So,
+just accept the unavoidable: any language has its peculiar properties (and for
+sure I do know why I dislike some languages for it). The \TEX\ system is not the
+only one where dollars, percent signs, ampersands and hashes have special
+meaning.
+
+\stopsection
+
+\startsection[title=Prefixes]
+
+Traditional \TEX\ has three prefixes that can be used with macros: \type {\global},
+\type {\outer} and \type {\long}. The last two are no|-|op's in \LUAMETATEX\ and
+if you want to know what they do (did) you can look it up in the \TEX book. The
+\ETEX\ extension gave us \type {\protected}.
+
+In \LUAMETATEX\ we have \type {\global}, \type {\protected}, \type {\tolerant}
+and overload related prefixes like \type {\frozen}. A protected macro is one that
+doesn't expand in an expandable context, so for instance inside an \type {\edef}.
+You can force expansion by using the \type {\expand} primitive in front which is
+also something \LUAMETATEX.
+
+% A protected macro can be made expandable by \typ {\unletprotected} and can be
+% protected with \typ {\letprotected}.
+%
+% \startbuffer[example]
+% \def\foo{foo} \edef\oof{oof\foo} 1: \meaning\oof
+% \protected\def\foo{foo} \edef\oof{oof\foo} 2: \meaning\oof
+% \unletprotected \foo \edef\oof{oof\foo} 3: \meaning\oof
+% \stopbuffer
+%
+% \typebuffer[example][option=TEX]
+%
+% \startlines \getbuffer[example] \stoplines
+
+Frozen macros cannot be redefined without some effort. This feature can to some
+extent be used to prevent a user from overloading, but it also makes it harder
+for the macro package itself to redefine on the fly. You can remove the lock with
+\typ {\unletfrozen} and add a lock with \typ {\letfrozen} so in the end users
+still have all the freedoms that \TEX\ normally provides.
+
+\startbuffer[example]
+ \def\foo{foo} 1: \meaning\foo
+ \frozen\def\foo{foo} 2: \meaning\foo
+ \unletfrozen \foo 3: \meaning\foo
+\protected\frozen\def\foo{foo} 4: \meaning\foo
+ \unletfrozen \foo 5: \meaning\foo
+\stopbuffer
+
+\typebuffer[example][option=TEX]
+
+\startlines \overloadmode0 \getbuffer[example] \stoplines
+
+This actually only works when you have set \type {\overloadmode} to a value that
+permits redefining a frozen macro, so for the purpose of this example we set it
+to zero.
+
+A \type {\tolerant} macro is one that will quit scanning arguments when a
+delimiter cannot be matched. We saw examples of that in a previous section.
+
+These prefixes can be chained (in arbitrary order):
+
+\starttyping[option=TEX]
+\frozen\tolerant\protected\global\def\foo[#1]#*[#2]{...}
+\stoptyping
+
+There is actually an additional prefix, \type {\immediate} but that one is there
+as signal for a macro that is defined in and handled by \LUA. This prefix can
+then perform the same function as the one in traditional \TEX, where it is used
+for backend related tasks like \type {\write}.
+
+Now, the question is of course, to what extent will \CONTEXT\ use these new
+features. One important argument in favor of using \type {\tolerant} is that it
+gives (hopefully) better error messages. It also needs less code due to lack of
+indirectness. Using \type {\frozen} adds some safeguards although in some places
+where \CONTEXT\ itself overloads commands, we need to defrost. Adapting the code
+is a tedious process and it can introduce errors due to mistypings, although
+these can easily be fixed. So, it will be used but it will take a while to adapt
+the code base.
+
+One problem with frozen macros is that they don't play nice with for instance
+\type {\futurelet}. Also, there are places in \CONTEXT\ where we actually do
+redefine some core macro that we also want to protect from redefinition by a
+user. One can of course \type {\unletfrozen} such a command first but as a bonus
+we have a prefix \type {\overloaded} that can be used as prefix. So, one can easily
+redefine a frozen macro but it takes a little effort. After all, this feature is
+mainly meant to protect a user for side effects of definitions, and not as final
+blocker. \footnote {As usual adding features like this takes some experimenting
+and we're now at the third variant of the implementation, so we're getting there.
+The fact that we can apply such features in large macro package like \CONTEXT\
+helps figuring out the needs and best approaches.}
+
+A frozen macro can still be overloaded, so what if we want to prevent that? For
+this we have the \type {\permanent} prefix. Internally we also create primitives
+but we don't have a prefix for that. But we do have one for a very special case
+which we demonstrate with an example:
+
+\startbuffer[example]
+\def\FOO % trickery needed to pick up an optional argument
+ {\noalign{\vskip10pt}}
+
+\noaligned\protected\tolerant\def\OOF[#1]%
+ {\noalign{\vskip\iftok{#1}\emptytoks10pt\else#1\fi}}
+
+\starttabulate[|l|l|]
+ \NC test \NC test \NC \NR
+ \NC test \NC test \NC \NR
+ \FOO
+ \NC test \NC test \NC \NR
+ \OOF[30pt]
+ \NC test \NC test \NC \NR
+ \OOF
+ \NC test \NC test \NC \NR
+\stoptabulate
+\stopbuffer
+
+\typebuffer[example][option=TEX]
+
+When \TEX\ scans input (from a file or token list) and starts an alignment, it
+will pick up rows. When a row is finished it will look ahead for a \type
+{\noalign} and it expands the next token. However, when that token is protected,
+the scanner will not see a \type {\noalign} in that macro so it will likely start
+complaining when that next macro does get expanded and produces a \type
+{\noalign} when a cell is built. The \type {\noaligned} prefix flags a macro as
+being one that will do some \type {\noalign} as part of its expansion. This trick
+permits clean macros that pick up arguments. Of course it can be done with
+traditional means but this whole exercise is about making the code look nice.
+
+The table comes out as:
+
+\getbuffer[example]
+
+One can check the flags with \type {\ifflags} which takes a control sequence and
+a number, where valid numbers are:
+
+\starttabulate[|r|lw(8em)|r|lw(8em)|r|lw(8em)|r|lw(8em)|]
+\NC \the\frozenflagcode \NC frozen
+\NC \the\permanentflagcode \NC permanent
+\NC \the\immutableflagcode \NC immutable
+\NC \the\primitiveflagcode \NC primitive \NC \NR
+\NC \the\mutableflagcode \NC mutable
+\NC \the\noalignedflagcode \NC noaligned
+\NC \the\instanceflagcode \NC instance
+\NC \NC \NC \NR
+\stoptabulate
+
+The level of checking is controlled with the \type {\overloadmode} but I'm still
+not sure about how many levels we need there. A zero value disables checking,
+the values 1 and 3 give warnings and the values 2 and 4 trigger an error.
+
+\stopsection
+
+\stopdocument
+
+freezing pitfalls:
+
+- \futurelet : \overloaded needed
+- \let : \overloaded sometimes needed
+
+primitive protection:
+
+\newif\iffoo \footrue \foofalse : problem when we make iftrue and iffalse
+permanent ... they inherit, so we can't let them, we need a not permanent
+alias which is again tricky ... something native?
+
+immutable : still \count000 but we can consider blocking that, for instance
+by \def\count{some error}
+
+\defcsname
+\edefcsname
+\letcsname
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-paragraphs.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-paragraphs.tex
new file mode 100644
index 00000000000..3098333f013
--- /dev/null
+++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-paragraphs.tex
@@ -0,0 +1,948 @@
+% language=us
+
+\environment lowlevel-style
+
+\startdocument
+ [title=paragraphs,
+ color=middlecyan]
+
+\startsection[title=Introduction]
+
+This manual is mostly discussing a few low level wrappers around low level \TEX\
+features. Its writing is triggered by an update to the \METAFUN\ and \LUAMETAFUN\
+manuals where we mess a bit with shapes. It gave a good reason to also cover some
+more paragraph related topics but it might take a while to complete. Remind me if
+you feel that takes too much time.
+
+Because paragraphs and their construction are rather central to \TEX, you can
+imagine that the engine exposes dealing with them. This happens via commands
+(primitives) but only when it's robust. Then there are callbacks, and some
+provide detailed information about what we're dealing with. However, intercepting
+node lists can already be hairy and we do that a lot in \CONTEXT. Intercepting
+and tweaking paragraph properties is even more tricky, which is why we try to
+avoid that in the core. But \unknown\ in the following sections you will see that
+there are actually a couple of mechanism that do so. Often new features like this
+are built in stepwise and enabled locally for a while and when they seem okay
+they get enabled by default. \footnote {For this we have \type
+{\enableexperiments} which one can use in \type {cont-loc.mkxl} or \type
+{cont-exp.mkxl}, files that are loaded runtime when on the system. When you use
+them, make sure they don't interfere; they are not part of the updates, contrary
+to \type {cont-new.mkxl}.}
+
+\stopsection
+
+\startsection[title=Paragraphs]
+
+Before we demonstrate some trickery, let's see what a paragraph is. Normally a
+document source is formatted like this:
+
+\starttyping[option=TEX]
+some text (line 1)
+some text (line 2)
+
+some more test (line 1)
+some more test (line 2)
+\stoptyping
+
+There are two blocks of text here separated by an empty line and they become two
+paragraphs. Unless configured otherwise an empty line is an indication that we
+end a paragraph. You can also explicitly do that:
+
+\starttyping[option=TEX]
+some text (line 1)
+some text (line 2)
+\par
+some more test (line 1)
+some more test (line 2)
+\stoptyping
+
+When \TEX\ starts a paragraph, it actually also does something think of:
+
+\starttyping[option=TEX]
+[\the\everypar]some text (line 1) some text (line 2) \par
+[\the\everypar]some more test (line 1) some more test (line 2) \par
+\stoptyping
+
+or more accurate:
+
+\starttyping[option=TEX]
+[\the\everypar]some text some text \par
+[\the\everypar]some more test some more test \par
+\stoptyping
+
+because the end|-|of|-|line character has become a space. As mentioned,
+an empty line is actually the end of a paragraph. But in \LUAMETATEX\
+we can cheat a bit. If we have this:
+
+\startbuffer
+line 1
+
+line 2
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+We can do this (watch how we need to permit overloading a primitive when we have
+enabled \type {\overloadmode}):
+
+\startbuffer
+\pushoverloadmode
+\def\linepar{\removeunwantedspaces !\ignorespaces}
+\popoverloadmode
+line 1
+
+line 2
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+This comes out as:
+
+\start \getbuffer \stop
+
+I admit that since it got added (as part of some cleanup halfway the overhaul of
+the engine) I never saw a reason to use it, but it is a cheap feature. The \type
+{\linepar} primitive is undefined (\type {\undefined}) by default so no user sees
+it anyway. Just don't use it unless maybe for some pseudo database trickery (I
+considered using it for the database module but it is not needed). In a similar
+fashion, just don't redefine \type {\par}: it's asking for troubles and \quote
+{not done} in \CONTEXT\ anyway.
+
+Back to reality. In \LUATEX\ we get a node list that starts with a so called
+\type {localpar} node and ends with a \type {\parfillskip}. The first node is
+prepended automatically. That list travels through the system: hyphenation,
+applying font properties, break the effectively one line into lines, wrap them
+and add them to a vertical list, etc. Each stage can be intercepted via
+callbacks.
+
+When the paragraph is broken into lines hanging indentation or a so called par
+shape can be applied, and we will see more of that later, here we talk \type
+{\par} and show another \LUAMETATEX\ trick:
+
+\startbuffer
+\def\foo{{\bf test:} \ignorepars}
+
+\foo
+
+line
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+The macro typesets some text and then skips to the next paragraph:
+
+\start \getbuffer \stop
+
+Think of this primitive as being a more powerful variant of \type
+{\ignorespaces}. This leaves one aspect: how do we start a paragraph. Technically
+we need to force \TEX\ into so called horizontal mode. When you look at plain
+\TEX\ documents you will notice commands like \type {\noindent} and \type
+{\indent}. In \CONTEXT\ we have more high level variants, for instance we have
+\type {\noindentation}.
+
+A robust way to make sure that you get in horizontal mode is using \type
+{\dontleavehmode} which is a wink to \type {\leavevmode}, a command that you
+should never use in \CONTEXT, so when you come from plain or \LATEX, it's one of
+the commands you should wipe from your memory.
+
+When \TEX\ starts with a paragraph the \type {\everypar} token list is expanded
+and again this is a primitive you should not mess with yourself unless in very
+controlled situations. If you change its content, you're on your own with respect
+to interferences and side effects.
+
+One of the things that \TEX\ does in injecting the indentation. Even when there
+is none, it gets added, not as skip but as an empty horizontal box of a certain
+width. This is easier on the engine when it constructs the paragraph from the one
+liner: starting with a skip demands a bit more testing in the process (a nice
+trick so to say). However, in \CONTEXT\ we enable the \LUAMETATEX\ feature that
+does use a skip instead of a box. It's part of the normalization that is
+discussed later. Instead of checking for a box with property indent, we check for
+a skip with such property. This is often easier and cleaner.
+
+A bit off topic is the fact that in traditional \TEX\ empty lines or \type {\par}
+primitives can trigger an error. This has to do with the fact that the program
+evolved in a time where paper terminals were used and runtime could be excessive.
+So, in order to catch a possible missing brace, a concept of \type {\long}
+macros, permitting \type {\par} or equivalents in arguments, was introduced as
+well as not permitting them in for instance display math. In \CONTEXT\ \MKII\
+most macros that could be sensitive for this were defined as \type {\long} so
+that users never had to bother about it and probably were not even aware of it.
+Right from the start in \LUATEX\ these error|-|triggers could be disabled which
+of course we enable in \CONTEXT\ and in \LUAMETATEX\ these features have been
+removed altogether. I don't think users will complain about this.
+
+If you want to enforce a newline but not a new paragraph you can use the \type
+{\crlf} command. When used on its own it will produce an empty line. Don't use
+this to create whitespace between lines.
+
+If you want to do something after so called par tokens are seen you can do this:
+
+\startbuffer
+\def\foo{{\bf >>>> }}
+\expandafterpars\foo
+
+this is a new paragraph ...
+
+\expandafterpars\foo
+\par\par\par\par
+this is a new paragraph ...
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+This not to be confused with \type {\everypar} which is a token list that \TEX\
+itself injects before each paragraph (also nested ones).
+
+\getbuffer
+
+This is typically a primitive that will only be used in macros. You can actually
+program it using macros: pickup a token, check and push it back when it's not a
+par equivalent token. The primitive is is just nicer (and easier on the log when
+tracing is enabled).
+
+\stopsection
+
+\startsection[title=Properties]
+
+A paragraph is just a collection of lines that result from one input line that
+got broken. This process of breaking into lines is influenced by quite some
+parameters. In traditional \TEX\ and also in \LUAMETATEX\ by default the values
+that are in effect when the end of the paragraph is met are used. So, when you
+change them in a group and then ends the paragraph after the group, the values
+you've set in the group are not used.
+
+However, in \LUAMETATEX\ we can optionally store them with the paragraph. When
+that happens the values current at the start are frozen. You can still overload
+them but that has to be done explicitly then. The advantage is that grouping no
+longer interferes with the line break algorithm. The magic primitive is \type
+{\snapshotpar} which takes a number made from categories mentioned below:
+
+\starttabulate[|l|l|r|]
+\BC variable \BC category \BC code \NC \NR
+\NC \type {\hsize} \NC hsize \NC 0x\uchexnumbers\frozenhsizecode \NC \NR
+\NC \type {\leftskip} \NC skip \NC 0x\uchexnumbers\frozenskipcode \NC \NR
+\NC \type {\rightskip} \NC skip \NC 0x\uchexnumbers\frozenskipcode \NC \NR
+\NC \type {\hangindent} \NC hang \NC 0x\uchexnumbers\frozenhangcode \NC \NR
+\NC \type {\hangafter} \NC hang \NC 0x\uchexnumbers\frozenhangcode \NC \NR
+\NC \type {\parindent} \NC indent \NC 0x\uchexnumbers\frozenindentcode \NC \NR
+\NC \type {\parfillleftskip} \NC par fill \NC 0x\uchexnumbers\frozenparfillcode \NC \NR
+\NC \type {\parfillrightskip} \NC par fill \NC 0x\uchexnumbers\frozenparfillcode \NC \NR
+\NC \type {\adjustspacing} \NC adjust \NC 0x\uchexnumbers\frozenadjustcode \NC \NR
+\NC \type {\adjustspacingstep} \NC adjust \NC 0x\uchexnumbers\frozenadjustcode \NC \NR
+\NC \type {\adjustspacingshrink} \NC adjust \NC 0x\uchexnumbers\frozenadjustcode \NC \NR
+\NC \type {\adjustspacingstretch} \NC adjust \NC 0x\uchexnumbers\frozenadjustcode \NC \NR
+\NC \type {\protrudechars} \NC protrude \NC 0x\uchexnumbers\frozenprotrudecode \NC \NR
+\NC \type {\pretolerance} \NC tolerance \NC 0x\uchexnumbers\frozentolerancecode \NC \NR
+\NC \type {\tolerance} \NC tolerance \NC 0x\uchexnumbers\frozentolerancecode \NC \NR
+\NC \type {\emergencystretch} \NC stretch \NC 0x\uchexnumbers\frozenstretchcode \NC \NR
+\NC \type {\looseness} \NC looseness \NC 0x\uchexnumbers\frozenloosenesscode \NC \NR
+\NC \type {\lastlinefit} \NC last line \NC 0x\uchexnumbers\frozenlastlinecode \NC \NR
+\NC \type {\linepenalty} \NC line penalty \NC 0x\uchexnumbers\frozenlinepenaltycode \NC \NR
+\NC \type {\interlinepenalty} \NC line penalty \NC 0x\uchexnumbers\frozenlinepenaltycode \NC \NR
+\NC \type {\interlinepenalties} \NC line penalty \NC 0x\uchexnumbers\frozenlinepenaltycode \NC \NR
+\NC \type {\clubpenalty} \NC club penalty \NC 0x\uchexnumbers\frozenclubpenaltycode \NC \NR
+\NC \type {\clubpenalties} \NC club penalty \NC 0x\uchexnumbers\frozenclubpenaltycode \NC \NR
+\NC \type {\widowpenalty} \NC widow penalty \NC 0x\uchexnumbers\frozenwidowpenaltycode \NC \NR
+\NC \type {\widowpenalties} \NC widow penalty \NC 0x\uchexnumbers\frozenwidowpenaltycode \NC \NR
+\NC \type {\displaywidowpenalty} \NC display penalty \NC 0x\uchexnumbers\frozendisplaypenaltycode \NC \NR
+\NC \type {\displaywidowpenalties} \NC display penalty \NC 0x\uchexnumbers\frozendisplaypenaltycode \NC \NR
+\NC \type {\brokenpenalty} \NC broken penalty \NC 0x\uchexnumbers\frozenbrokenpenaltycode \NC \NR
+\NC \type {\adjdemerits} \NC demerits \NC 0x\uchexnumbers\frozendemeritscode \NC \NR
+\NC \type {\doublehyphendemerits} \NC demerits \NC 0x\uchexnumbers\frozendemeritscode \NC \NR
+\NC \type {\finalhyphendemerits} \NC demerits \NC 0x\uchexnumbers\frozendemeritscode \NC \NR
+\NC \type {\parshape} \NC shape \NC 0x\uchexnumbers\frozenshapecode \NC \NR
+\NC \type {\baselineskip} \NC line \NC 0x\uchexnumbers\frozenlinecode \NC \NR
+\NC \type {\lineskip} \NC line \NC 0x\uchexnumbers\frozenlinecode \NC \NR
+\NC \type {\lineskiplimit} \NC line \NC 0x\uchexnumbers\frozenlinecode \NC \NR
+\NC \type {\hyphenationmode} \NC hyphenation \NC 0x\uchexnumbers\frozenhyphenationcode \NC \NR
+\stoptabulate
+
+
+As you can see here, there are more paragraph related parameters than in for
+instance \PDFTEX\ and \LUATEX\ and these are (to be) explained in the
+\LUAMETATEX\ manual. You can imagine that keeping this around with the paragraph
+adds some extra overhead to the machinery but most users won't notice that
+because is is compensated by gains elsewhere.
+
+This is pretty low level and there are a bunch of helpers that support this but
+these are not really user level macros. As with everything \TEX\ you can mess
+around as much as you like, and the code gives plenty of examples but when you do
+this, you're on your own because it can interfere with \CONTEXT\ core
+functionality.
+
+In \LMTX\ taking these snapshots is turned on by default and because it thereby
+fundamentally influences the par builder, users can run into compatibility issues
+but in practice there has been no complaints (and this feature has been in use
+quite a while before this document was written). One reason for users not
+noticing is that one of the big benefits is probably handled by tricks mentioned
+on the mailing list. Imagine that you have this:
+
+\starttyping[option=TEX]
+{\bf watch out:} here is some text
+\stoptyping
+
+In this small example the result will be as expected. But what if something magic
+with the start of a paragraph is done? Like this:
+
+\starttyping[option=TEX]
+\placefigure[left]{A cow!}{\externalfigure[cow.pdf]}
+
+{\bf watch out:} here is some text ... of course much more is needed to
+ get a flow around the figure!
+\stoptyping
+
+The figure will hang at the left side of the paragraph but it is put there when
+the text starts and that happens inside the bold group. It means that the
+properties we set in order to get the shape around the figure are lost as soon as
+we're at \quote{\type {here is some text}} and definitely is wrong when the
+paragraph ends and the par builder has to use them to get the shape right. We get
+text overlapping the figure. A trick to overcome this is:
+
+\starttyping[option=TEX]
+\dontleavehmode {\bf watch out:} here is some text ... of course much
+ more is needed to get a flow around the figure!
+\stoptyping
+
+where the first macro makes sure we already start a paragraph before the group is
+entered (using a \type {\strut} also works). It's not nice and I bet users have
+been bitten by this and by now know the tricks. But, with snapshots such fuzzy
+hacks are not needed any more! The same is true with this:
+
+\starttyping[option=TEX]
+{\leftskip 1em some text \par}
+\stoptyping
+
+where we had to explicitly end the paragraph inside the group in order to retain
+the skip. I suppose that users normally use the high level environments so they
+never had to worry about this. It's also why users probably won't notice that
+this new mechanism has been active for a while. Actually, when you now change a
+parameter inside the paragraph its new value will not be applied (unless you
+prefix it with \type {\frozen} or snapshot it) but no one did that anyway.
+
+\stopsection
+
+\startsection[title=Wrapping up]
+
+In \CONTEXT\ \LMTX\ we have a mechanism to exercise macros (or content) before a
+paragraph ends. This is implemented using the \type {\wrapuppar} primitive. The
+to be wrapped up material is bound to the current paragraph which in order to
+get this done has to be started when this primitive is used.
+
+Although the high level interface has been around for a while it still needs a
+bit more testing (read: use cases are needed). In the few cases where we already
+use it application can be different because again it relates to snapshots. This
+because in the past we had to use tricks that also influenced the user interface
+of some macros (which made them less natural as one would expect). So the
+question is: where do we apply it in old mechanisms and where not.
+
+{\em todo: accumulation, interference, where applied, limitations}
+
+% \vbox {vbox : \wrapuppar{1}test\par x\wrapuppar{2}test}\blank
+% \vtop {vtop : \wrapuppar{1}test\par x\wrapuppar{2}test}\blank
+% \vcenter{vcenter : \wrapuppar{1}test\par x\wrapuppar{2}test}\blank
+% $$x = \vcenter{vcenter : \wrapuppar{1}test\par x\wrapuppar{2}test}$$\blank
+% x\vadjust{vadjust : \wrapuppar{1}test\par x\wrapuppar{2}test}x\blank
+
+\stopsection
+
+\startsection[title=Hanging]
+
+There are two mechanisms for getting a specific paragraph shape: rectangular
+hanging and arbitrary shapes. Both mechanisms work top|-|down. The first
+mechanism uses a combination of \type {\hangafter} and \type {\hangindent}, and
+the second one depends on \type {\parshape}. In this section we discuss the
+rectangular one.
+
+\startbuffer[demo-5]
+\hangafter 4 \hangindent 4cm \samplefile{tufte} \page
+\hangafter -4 \hangindent 4cm \samplefile{tufte} \page
+\hangafter 4 \hangindent -4cm \samplefile{tufte} \page
+\hangafter -4 \hangindent -4cm \samplefile{tufte} \page
+\stopbuffer
+
+\typebuffer[demo-5][option=TEX]
+
+As you can see in \in {figure} [fig:hang], the four cases are driven by the sign
+of the values. If you want to hang into the margin you need to use different
+tricks, like messing with the \type {\leftskip}, \type {\rightskip} or \type
+{\parindent} parameters (which then of course can interfere with other mechanisms
+uses at the same time).
+
+\startplacefigure[title=Hanging indentation,reference=fig:hang]
+\startcombination[nx=2,ny=2]
+ {\typesetbuffer[demo-5][page=1,width=.4\textwidth,frame=on]} {\type{\hangafter +4 \hangindent +4cm}}
+ {\typesetbuffer[demo-5][page=2,width=.4\textwidth,frame=on]} {\type{\hangafter -4 \hangindent +4cm}}
+ {\typesetbuffer[demo-5][page=3,width=.4\textwidth,frame=on]} {\type{\hangafter +4 \hangindent -4cm}}
+ {\typesetbuffer[demo-5][page=4,width=.4\textwidth,frame=on]} {\type{\hangafter -4 \hangindent -4cm}}
+\stopcombination
+\stopplacefigure
+
+\stopsection
+
+\startsection[title=Shapes]
+
+In \CONTEXT\ we don't use \type {\parshape} a lot. It is used in for instance
+side floats but even there not in all cases. It's more meant for special
+applications. This means that in \MKII\ and \MKIV\ we don't have some high level
+interface. However, when \METAFUN\ got upgraded to \LUAMETAFUN, and the manual
+also needed an update, one of the examples in that manual that used shapes also
+got done differently (read: nicer). And that triggered the arrival of a new low
+level shape mechanism.
+
+One important property of the \type {\parshape} mechanism is that it works per
+paragraph. You define a shape in terms of a left margin and width of a line. The
+shape has a fixed number of such pairs and when there is more content, the last
+one is used for the rest of the lines. When the paragraph is finished, the shape
+is forgotten. \footnote {Not discussed here is a variant that might end up in
+\LUAMETATEX\ that works with the progression, i.e.\ takes the height of the
+content so far into account. This is somewhat tricky because for that to work
+vertical skips need to be frozen, which is no real big deal but has to be done
+careful in the code.}
+
+The high level interface is a follow up on the example in the \METAFUN\ manual and
+uses shapes that carry over to the next paragraph. In addition we can cycle over
+a shape. In this interface shapes are defined using keyword. Here are some
+examples:
+
+\starttyping[option=TEX]
+\startparagraphshape[test]
+ left 1mm right 1mm
+ left 5mm right 5mm
+\stopparagraphshape
+\stoptyping
+
+This shape has only two entries so the first line will have a 1mm margin while
+later lines will get 5mm margins. This translates into a \type {\parshape} like:
+
+\starttyping[option=TEX]
+\parshape 2
+ 1mm \dimexpr\hsize-1mm\relax
+ 5mm \dimexpr\hsize-5mm\relax
+\stoptyping
+
+Watch the number \type {2}: it tells how many specification lines follow. As you
+see, we need to calculate the width.
+
+\starttyping[option=TEX]
+\startparagraphshape[test]
+ left 1mm right 1mm
+ left 5mm right 5mm
+ repeat
+\stopparagraphshape
+\stoptyping
+
+This variant will alternate between 1mm and 5mm margins. The repeating feature is
+translated as follows. Maybe at some point I will introduce a few more options.
+
+\starttyping[option=TEX]
+\parshape 2 options 1
+ 1mm \dimexpr\hsize-1mm\relax
+ 5mm \dimexpr\hsize-5mm\relax
+\stoptyping
+
+A shape can have some repetition, and we can save keystrokes by copying the last
+entry. The resulting \type {\parshape} becomes rather long.
+
+\starttyping[option=TEX]
+\startparagraphshape[test]
+ left 1mm right 1mm
+ left 2mm right 2mm
+ left 3mm right 3mm
+ copy 8
+ left 4mm right 4mm
+ left 5mm right 5mm
+ left 5mm hsize 10cm
+\stopparagraphshape
+\stoptyping
+
+Also watch the \type {hsize} keyword: we don't calculate the hsize from the \type
+{left} and \type {right} values but explicitly set it.
+
+\starttyping[option=TEX]
+\startparagraphshape[test]
+ left 1mm right 1mm
+ right 3mm
+ left 5mm right 5mm
+ repeat
+\stopparagraphshape
+\stoptyping
+
+When a \type {right} keywords comes first the \type {left} is assumed to be zero.
+In the examples that follow we will use a couple of definitions:
+
+\startbuffer[setup-0]
+\startparagraphshape[test]
+ both 1mm both 2mm both 3mm both 4mm both 5mm both 6mm
+ both 7mm both 6mm both 5mm both 4mm both 3mm both 2mm
+\stopparagraphshape
+\stopbuffer
+
+\startbuffer[setup-0-repeat]
+\startparagraphshape[test-repeat]
+ both 1mm both 2mm both 3mm both 4mm both 5mm both 6mm
+ both 7mm both 6mm both 5mm both 4mm both 3mm both 2mm
+ repeat
+\stopparagraphshape
+\stopbuffer
+
+\typebuffer[setup-0,setup-0-repeat][option=TEX]
+
+The last one could also be defines as:
+
+\starttyping[option=TEX]
+\startparagraphshape[test-repeat]
+ \rawparagraphshape{test} repeat
+\stopparagraphshape
+\stoptyping
+
+In the previous code we already introduced the \type {repeat} option. This will
+make the shape repeat at the engine level when the shape runs out of specified
+lines. In the application of a shape definition we can specify a \type {method}
+to be used and that determine if the next paragraph will start where we left off
+and discard afterwards (\type {shift}) or that we move the discarded lines up
+front so that we never run out of lines (\type {cycle}). It sounds complicated
+but just keep in mind that \type {repeat} is part of the \type {\parshape} and
+act within a paragraph while \type {shift} and \type {cycle} are applied when a
+new paragraph is started.
+
+\startbuffer[demo-1]
+\startshapedparagraph[list=test]
+ \dorecurse{8}{\showparagraphshape\samplefile{tufte} \par}
+\stopshapedparagraph
+\stopbuffer
+
+\startbuffer[demo-1-repeat]
+\startshapedparagraph[list=test-repeat]
+ \dorecurse{8}{\showparagraphshape\samplefile{tufte} \par}
+\stopshapedparagraph
+\stopbuffer
+
+In \in {figure} [fig:shape:discard] you see the following applied:
+
+\typebuffer[demo-1,demo-1-repeat][option=TEX]
+
+\startplacefigure[title=Discarded shaping,reference=fig:shape:discard]
+\startcombination[nx=2,ny=2]
+ {\typesetbuffer[setup-0,demo-1] [page=1,width=.4\textwidth,frame=on]} {discard, finite shape, page 1}
+ {\typesetbuffer[setup-0,demo-1] [page=2,width=.4\textwidth,frame=on]} {discard, finite shape, page 2}
+ {\typesetbuffer[setup-0,demo-1-repeat][page=1,width=.4\textwidth,frame=on]} {discard, repeat in shape, page 1}
+ {\typesetbuffer[setup-0,demo-1-repeat][page=2,width=.4\textwidth,frame=on]} {discard, repeat in shape, page 2}
+\stopcombination
+\stopplacefigure
+
+In \in {figure} [fig:shape:shift] we use this instead:
+
+\startbuffer[demo-2]
+\startshapedparagraph[list=test,method=shift]
+ \dorecurse{8}{\showparagraphshape\samplefile{tufte} \par}
+\stopshapedparagraph
+\stopbuffer
+
+\startbuffer[demo-2-shift]
+\startshapedparagraph[list=test-repeat,method=shift]
+ \dorecurse{8}{\showparagraphshape\samplefile{tufte} \par}
+\stopshapedparagraph
+\stopbuffer
+
+\typebuffer[demo-2,demo-2-repeat][option=TEX]
+
+\startplacefigure[title=Shifted shaping,reference=fig:shape:shift]
+\startcombination[nx=2,ny=2]
+ {\typesetbuffer[setup-0,demo-2][page=1,width=.4\textwidth,frame=on]} {shift, finite shape, page 1}
+ {\typesetbuffer[setup-0,demo-2][page=2,width=.4\textwidth,frame=on]} {shift, finite shape, page 2}
+ {\typesetbuffer[setup-0-repeat,demo-2-shift][page=1,width=.4\textwidth,frame=on]} {shift, repeat in shape, page 1}
+ {\typesetbuffer[setup-0-repeat,demo-2-shift][page=2,width=.4\textwidth,frame=on]} {shift, repeat in shape, page 2}
+\stopcombination
+\stopplacefigure
+
+Finally, in \in {figure} [fig:shape:cycle] we use:
+
+\startbuffer[demo-3]
+\startshapedparagraph[list=test,method=cycle]
+ \dorecurse{8}{\showparagraphshape\samplefile{tufte} \par}
+\stopshapedparagraph
+\stopbuffer
+
+\startbuffer[demo-3-cycle]
+\startshapedparagraph[list=test-repeat,method=cycle]
+ \dorecurse{8}{\showparagraphshape\samplefile{tufte} \par}
+\stopshapedparagraph
+\stopbuffer
+
+\typebuffer[demo-3,demo-3-repeat][option=TEX]
+
+\startplacefigure[title=Cycled shaping,reference=fig:shape:cycle]
+\startcombination[nx=2,ny=2]
+ {\typesetbuffer[setup-0,demo-3][page=1,width=.4\textwidth,frame=on]} {cycle, finite shape, page 1}
+ {\typesetbuffer[setup-0,demo-3][page=2,width=.4\textwidth,frame=on]} {cycle, finite shape, page 2}
+ {\typesetbuffer[setup-0-repeat,demo-3-cycle][page=1,width=.4\textwidth,frame=on]} {cycle, repeat in shape, page 1}
+ {\typesetbuffer[setup-0-repeat,demo-3-cycle][page=2,width=.4\textwidth,frame=on]} {cycle, repeat in shape, page 2}
+\stopcombination
+\stopplacefigure
+
+These examples are probably too small to see the details but you can run them
+yourself or zoom in on the details. In the margin we show the values used. Here
+is a simple example of (non) poetry. There are other environments that can be
+used instead but this makes a good example anyway.
+
+\startbuffer
+\startparagraphshape[test]
+ left 0em right 0em
+ left 1em right 0em
+ repeat
+\stopparagraphshape
+
+\startshapedparagraph[list=test,method=cycle]
+ verse line 1.1\crlf verse line 2.1\crlf
+ verse line 3.1\crlf verse line 4.1\par
+ verse line 1.2\crlf verse line 2.2\crlf
+ verse line 3.2\crlf verse line 4.2\crlf
+ verse line 5.2\crlf verse line 6.2\par
+\stopshapedparagraph
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+\start \getbuffer \stop
+
+Because the idea for this feature originates in \METAFUN, we will now kick in
+some \METAPOST. The following code creates a shape for a circle. We use a
+2mm offset here:
+
+\startbuffer
+\startuseMPgraphic{circle}
+ path p ; p := fullcircle scaled TextWidth ;
+ build_parshape(p,
+ 2mm, 0, 0,
+ LineHeight, StrutHeight, StrutDepth, StrutHeight
+ ) ;
+\stopuseMPgraphic
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+\start \getbuffer \stop
+
+We plug this into the already described macros:
+
+\startbuffer
+\startshapedparagraph[mp=circle]%
+ \setupalign[verytolerant,stretch,last]%
+ \samplefile{tufte}
+ \samplefile{tufte}
+\stopshapedparagraph
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+And get ourself a circular shape. Watch out, at this moment the shape environment
+does not add grouping so when as in this case you change the alignment it can
+influence the document.
+
+\start \getbuffer \stop
+
+\startbuffer[framed]
+\framed[align=normal,width=\textwidth,offset=2mm,strut=no]\bgroup
+ \getbuffer
+\egroup
+\stopbuffer
+
+Assuming that the shape definition above is in a buffer we can do this:
+
+\typebuffer[option=TEX]
+
+The result is shown in \in {figure} [fig:shape:circle]. Because all action
+happens in the framed environment, we can also use this definition:
+
+\starttyping[option=TEX]
+\startuseMPgraphic{circle}
+ path p ; p := fullcircle scaled \the\dimexpr\framedwidth+\framedoffset*2\relax ;
+ build_parshape(p,
+ \framedoffset, 0, 0,
+ LineHeight, StrutHeight, StrutDepth, StrutHeight
+ ) ;
+ draw p ;
+\stopuseMPgraphic
+\stoptyping
+
+\startplacefigure[title=A framed circular shape,reference=fig:shape:circle]
+ \getbuffer[framed]
+\stopplacefigure
+
+A mechanism like this is often never completely automatic in the sense that you
+need to keep an eye on the results. Depending on user demands more features can
+be added. With weird shapes you might want to set up the alignment to be \type
+{tolerant} and have some \type {stretch}.
+
+The interface described in the \METAFUN\ manual is pretty old, the time stamp of
+the original code is mid 2000, but the principles didn't change. The examples in
+\type {meta-imp-txt.mkxl} can now be written as:
+
+\startuseMPgraphic{test 1}
+ begingroup ;
+ save p ; path p ; p := fullcircle scaled 6cm ;
+ lmt_parshape [
+ path = p,
+ offset = BodyFontSize/2,
+ dx = 0, % default
+ dy = 0, % default
+ lineheight = LineHeight, % default
+ strutheight = StrutHeight, % default
+ strutdepth = StrutDepth, % default
+ topskip = StrutHeight, % default
+ ] ;
+ draw p withpen pencircle scaled 1pt ;
+ endgroup ;
+\stopuseMPgraphic
+
+\startuseMPgraphic{test 2}
+ begingroup ;
+ save p ; path p ; p := fullsquare rotated 45 scaled 5cm ;
+ lmt_parshape [
+ path = p,
+ offset = BodyFontSize/2,
+ trace = true,
+ ] ;
+ draw p withpen pencircle scaled 1pt ;
+ endgroup ;
+\stopuseMPgraphic
+
+\startuseMPgraphic{test 3}
+ begingroup ;
+ save w, h, p ; path p ; w := h := 6cm ;
+ p := (.5w,h) -- ( 0, h) -- (0,0) -- (w,0) &
+ ( w,0) .. (.75w,.5h) .. (w,h) & (w,h) -- cycle ;
+ lmt_parshape [
+ path = p,
+ offset = BodyFontSize/2,
+ ] ;
+ draw p withpen pencircle scaled 1pt ;
+ endgroup ;
+\stopuseMPgraphic
+
+\startuseMPgraphic{test 4}
+ begingroup ;
+ save d, p, q ; path p, q ; d := BodyFontSize/2;
+ vardef shape(expr w, h, o) =
+ (o,o) -- (w-o,o) & (w-o,o) .. (.75w-o,.5h) ..
+ (w-2o,h-o) & (w-2o,h-o) -- (o,h-o) -- cycle
+ enddef ;
+ p := shape(6cm, 6cm, d) ; q := shape(6cm, 6cm, 0) ;
+ lmt_parshape [
+ path = p,
+ offsetpath = q,
+ dx = d,
+ dy = d,
+ trace = true,
+ ] ;
+ draw q withpen pencircle scaled 1pt ;
+ endgroup ;
+\stopuseMPgraphic
+
+\defineoverlay[test 1][\useMPgraphic{test 1}]
+\defineoverlay[test 2][\useMPgraphic{test 2}]
+\defineoverlay[test 3][\useMPgraphic{test 3}]
+\defineoverlay[test 4][\useMPgraphic{test 4}]
+
+\startbuffer
+ \startshapetext[test 1,test 2,test 3,test 4]
+ \setupalign[verytolerant,stretch,normal]%
+ \samplefile{douglas} % Douglas R. Hofstadter
+ \stopshapetext
+ \startcombination[2*2]
+ {\framed[offset=overlay,frame=off,background=test 1]{\getshapetext}}
+ {test 1}
+ {\framed[offset=overlay,frame=off,background=test 2]{\getshapetext}}
+ {test 2}
+ {\framed[offset=overlay,frame=off,background=test 3]{\getshapetext}}
+ {test 3}
+ {\framed[offset=overlay,frame=off,background=test 4]{\getshapetext}}
+ {test 4}
+ \stopcombination
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+In \in {figure} [fig:shapes:chain] we see the result. Watch how for two shapes
+we have enabled tracing. Of course you need to tweak till all fits well but we're
+talking of special situations anyway.
+
+\startplacefigure[Title=Multiple shapes,reference=fig:shapes:chain]
+ \getbuffer
+\stopplacefigure
+
+Here is a bit more extreme example. Again we use a circle:
+
+\startbuffer
+\startuseMPgraphic{circle}
+ lmt_parshape [
+ path = fullcircle scaled 136mm,
+ offset = 2mm,
+ bottomskip = - 1.5LineHeight,
+ ] ;
+\stopuseMPgraphic
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+But we output a longer text:
+
+\startbuffer
+\startshapedparagraph[mp=circle,repeat=yes,method=cycle]%
+ \setupalign[verytolerant,stretch,last]\dontcomplain
+ {\darkred \samplefile{tufte}}\par
+ {\darkgreen \samplefile{tufte}}\par
+ {\darkblue \samplefile{tufte}}\par
+ {\darkcyan \samplefile{tufte}}\par
+ {\darkmagenta \samplefile{tufte}}\par
+\stopshapedparagraph
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+We get a multi|-|page shape:
+
+\start \getbuffer \stop
+
+Compare this with:
+
+\startbuffer
+\startshapedparagraph[mp=circle,repeat=yes,method=cycle]%
+ \setupalign[verytolerant,stretch,last]\dontcomplain
+ {\darkred \samplefile{tufte}}
+ {\darkgreen \samplefile{tufte}}
+ {\darkblue \samplefile{tufte}}
+ {\darkcyan \samplefile{tufte}}
+ {\darkmagenta \samplefile{tufte}}
+\stopshapedparagraph
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+Which gives:
+
+\start \getbuffer \stop
+
+Here the \type {bottomskip} takes care of subtle rounding issues as well as
+discarding the last line in the shape so that we get nicer continuation. There is
+no full automated solution for all you can come up with.
+
+Mixing a \METAPOST\ specification into a regular one is also possible. The next
+example demonstrates this as well as the option to remove some lines from a
+specification:
+
+\starttyping[option=TEX]
+\startparagraphshape[test]
+ left 0em right 0em
+ left 1em right 0em
+ metapost {circle}
+ delete 3
+ metapost {circle,circle,circle}
+ delete 7
+ metapost {circle}
+ repeat
+\stopparagraphshape
+\stoptyping
+
+You can combine a shape with narrowing a paragraph. Watch the \type {absolute}
+keyword in the next code. The result is shown in \in {figure} [fig:shape:skips].
+
+\startbuffer[demo-4]
+\startuseMPgraphic{circle}
+ lmt_parshape [
+ path = fullcircle scaled TextWidth,
+ bottomskip = - 1.5LineHeight,
+ ] ;
+\stopuseMPgraphic
+
+\startparagraphshape[test-1]
+ metapost {circle} repeat
+\stopparagraphshape
+
+\startparagraphshape[test-2]
+ absolute left metapost {circle} repeat
+\stopparagraphshape
+
+\startparagraphshape[test-3]
+ absolute right metapost {circle} repeat
+\stopparagraphshape
+
+\startparagraphshape[test-4]
+ absolute both metapost {circle} repeat
+\stopparagraphshape
+
+\showframe
+
+\startnarrower[4*left,2*right]
+ \startshapedparagraph[list=test-1,repeat=yes,method=repeat]%
+ \setupalign[verytolerant,stretch,last]\dontcomplain
+ \dorecurse{3}{\samplefile{thuan}}
+ \stopshapedparagraph
+ \page
+ \startshapedparagraph[list=test-2,repeat=yes,method=repeat]%
+ \setupalign[verytolerant,stretch,last]\dontcomplain
+ \dorecurse{3}{\samplefile{thuan}}
+ \stopshapedparagraph
+ \page
+ \startshapedparagraph[list=test-3,repeat=yes,method=repeat]%
+ \setupalign[verytolerant,stretch,last]\dontcomplain
+ \dorecurse{3}{\samplefile{thuan}}
+ \stopshapedparagraph
+ \page
+ \startshapedparagraph[list=test-4,repeat=yes,method=repeat]%
+ \setupalign[verytolerant,stretch,last]\dontcomplain
+ \dorecurse{3}{\samplefile{thuan}}
+ \stopshapedparagraph
+\stopnarrower
+\stopbuffer
+
+\typebuffer[demo-4][option=TEX]
+
+\startplacefigure[title=Skip compensation,reference=fig:shape:skips]
+\startcombination[nx=2,ny=2]
+ {\typesetbuffer[demo-4][page=1,width=.4\textwidth,frame=on]} {test 1}
+ {\typesetbuffer[demo-4][page=2,width=.4\textwidth,frame=on]} {test 2, left}
+ {\typesetbuffer[demo-4][page=3,width=.4\textwidth,frame=on]} {test 3, right}
+ {\typesetbuffer[demo-4][page=4,width=.4\textwidth,frame=on]} {test 4, both}
+\stopcombination
+\stopplacefigure
+
+\stopsection
+
+% \startsection[title=Linebreaks]
+\startsection[title=Modes]
+
+% \ruledvbox{1\ifhmode\writestatus{!}{HMODE 1}\fi} % hsize
+% \ruledvbox{\hbox{\strut 2}\ifhmode\writestatus{!}{HMODE 2}\fi} % fit
+% \ruledvbox{\hbox{\strut 3}\hbox{\strut 3}\ifhmode\writestatus{!}{HMODE 3}\fi} % fit
+% \ruledvbox{\hbox{\strut 4}4\ifhmode\writestatus{!}{HMODE 4}\fi} % hsize
+% \ruledvbox{\hbox{\strut 5}5\hbox{\strut 5}\ifhmode\writestatus{!}{HMODE 5}\fi} % hsize
+% \ruledvbox{6\hbox{\strut 6}\ifhmode\writestatus{!}{HMODE 6}\fi} % hsize
+
+{\em todo: some of the side effects of so called modes}
+
+\stopsection
+
+\startsection[title=Normalization]
+
+{\em todo: users don't need to bother about this but it might be interesting anyway}
+
+\stopsection
+
+\stopdocument
+
+\everyhbox \everyvbox : useless unless one resets
+\parattr
+\snapshotpar
+\wrapuppar
+
+% \parfillleftskip
+
+% I rewarded myself after writing a section by watching the video "Final Thing On
+% My Mind", The Pineapple This, Live, 2020, the usual perfect GH performance,
+% wondering if live would turn to normal so that we could go to such concerts once
+% again given successive covids. Writing manuals can do with a distraction.
+%
+% Gavin Harrison: Soundcheck, Drummerworld Jan 27, 2021 ... I wish I could make
+% something called a check into pefect solo. Okay, another section and I'll check
+% out the latest Simon Phillips and other favourite dummer uploads.
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-scope.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-scope.tex
new file mode 100644
index 00000000000..1313de92332
--- /dev/null
+++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-scope.tex
@@ -0,0 +1,341 @@
+% language=us
+
+% \hfil \hss
+% spread
+
+\environment lowlevel-style
+
+\startdocument
+ [title=scope,
+ color=middleblue]
+
+\startsection[title=Introduction]
+
+When I visited the file where register allocations are implemented I wondered to
+what extend it made sense to limit allocation to global instances only. This
+chapter deals with this phenomena.
+
+\stopsection
+
+\startsection[title=Registers]
+
+In \TEX\ definitions can be local or global. Most assignments are local within a
+group. Registers and definitions can be assigned global by using the \type
+{\global} prefix. There are also some properties that are global by design, like
+\type {\prevdepth}. A mixed breed are boxes. When you tweak its dimensions you
+actually tweak the current box, which can be an outer level. Compare:
+
+\starttyping[option=TEX]
+\scratchcounter = 1
+here the counter has value 1
+\begingroup
+ \scratchcounter = 2
+ here the counter has value 2
+\endgroup
+here the counter has value 1
+\stoptyping
+
+with:
+
+\starttyping[option=TEX]
+\setbox\scratchbox=\hbox{}
+here the box has zero width
+\begingroup
+ \wd\scratchbox=10pt
+ here the box is 10pt wide
+\endgroup
+here the box is 10pt wide
+\stoptyping
+
+It all makes sense so a remark like \quotation {Assignments to box dimensions are
+always global} are sort of confusing. Just look at this:
+
+\startbuffer
+\setbox\scratchbox=\hbox to 20pt{}
+here the box is \the\wd\scratchbox\ wide\par
+\begingroup
+ \setbox\scratchbox=\hbox{}
+ here the box is \the\wd\scratchbox\ wide\par
+ \begingroup
+ \wd\scratchbox=15pt
+ here the box is \the\wd\scratchbox\ wide\par
+ \endgroup
+ here the box is \the\wd\scratchbox\ wide\par
+\endgroup
+here the box is \the\wd\scratchbox\ wide\par
+\stopbuffer
+
+\typebuffer[option=TEX] \startlines \getbuffer \stoplines
+
+If you don't think about it, what happens is what you expect. Now watch the next
+variant:
+
+\startbuffer
+\setbox\scratchbox=\hbox to 20pt{}
+here the box is \the\wd\scratchbox\ wide\par
+\begingroup
+ \setbox\scratchbox=\hbox{}
+ here the box is \the\wd\scratchbox\ wide\par
+ \begingroup
+ \global\wd\scratchbox=15pt
+ here the box is \the\wd\scratchbox\ wide\par
+ \endgroup
+ here the box is \the\wd\scratchbox\ wide\par
+\endgroup
+here the box is \the\wd\scratchbox\ wide\par
+\stopbuffer
+
+The \type {\global} is only effective for the current box. It is good to realize
+that when we talk registers, the box register behaves just like any other
+register but the manipulations happen to the current one.
+
+\typebuffer \startlines \getbuffer \stoplines
+
+\startbuffer
+\scratchdimen=20pt
+here the dimension is \the\scratchdimen\par
+\begingroup
+ \scratchdimen=0pt
+ here the dimension is \the\scratchdimen\par
+ \begingroup
+ \global\scratchdimen=15pt
+ here the dimension is \the\scratchdimen\par
+ \endgroup
+ here the dimension is \the\scratchdimen\par
+\endgroup
+here the dimension is \the\scratchdimen\par
+\stopbuffer
+
+\typebuffer[option=TEX] \startlines \getbuffer \stoplines
+
+\stopsection
+
+\startsection[title=Allocation]
+
+The plain \TEX\ format has set some standards and one of them is that registers
+are allocated with \type {\new...} commands. So we can say:
+
+\starttyping[option=TEX]
+\newcount\mycounta
+\newdimen\mydimena
+\stoptyping
+
+These commands take a register from the pool and relate the given name to that
+entry. In \CONTEXT\ we have a bunch of predefined scratch registers for general
+use, like:
+
+\startbuffer
+scratchcounter : \meaningfull\scratchcounter
+scratchcounterone : \meaningfull\scratchcounterone
+scratchcountertwo : \meaningfull\scratchcountertwo
+scratchdimen : \meaningfull\scratchdimen
+scratchdimenone : \meaningfull\scratchdimenone
+scratchdimentwo : \meaningfull\scratchdimentwo
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+The meaning reveals what these are:
+
+\startlines \tttf \getbuffer \stoplines
+
+You can use the numbers directly but that is a bad idea because they can clash!
+In the original \TEX\ engine there are only 256 registers and some are used by
+the engine and the core of a macro package itself, so that leaves a little amount
+for users. The \ETEX\ extension lifted that limitation and bumped to 32K and
+\LUATEX\ upped that to 64K. One could go higher but what makes sense? These
+registers are taking part of the fixed memory slots because that makes nested
+(grouped) usage efficient and access fast. The number you see above is deduced
+from the so called command code (here indicated by \type {\count}) and an index
+encoded in the same token. So, \type {\scratchcounter} takes a single token
+contrary to the verbose \type {\count257} that takes four tokens where the number
+gets parsed every time it is needed. But those are details that a user can
+forget.
+
+As mentioned, commands like \type {\newcount \foo} create a global control
+sequence \type {\foo} referencing a counter. You can locally redefine that
+control sequence unless in \LUAMETATEX\ you have so called overload mode enabled.
+You can do local or global assignments to these registers.
+
+\starttyping[option=TEX]
+\scratchcounter = 123
+\begingroup
+ \scratchcounter = 456
+ \begingroup
+ \global\scratchcounter = 789
+ \endgroup
+\endgroup
+\stoptyping
+
+And in both cases count register 257 is set. When an assignment is global,
+all current values to that register get the same value. Normally this is all
+quite transparent: you get what you ask for. However the drawback is that
+as a user you cannot know what variables are already defined, which means
+that this will fail (that is: it will issue a message):
+
+\starttyping[option=TEX]
+\newcount\scratchcounter
+\stoptyping
+
+as will the second line in:
+
+\starttyping[option=TEX]
+\newcount\myscratchcounter
+\newcount\myscratchcounter
+\stoptyping
+
+In \CONTEXT\ the scratch registers are visible but there are lots of internally
+used ones are protected from the user by more obscure names. So what if you want
+to use your own register names without \CONTEXT\ barking to you about not being
+able to define it. This is why in \LMTX\ (and maybe some day in \MKIV) we now
+have local definitions:
+
+\startbuffer
+\begingroup
+ \newlocaldimen\mydimena \mydimena1\onepoint
+ \newlocaldimen\mydimenb \mydimenb2\onepoint
+ (\the\mydimena,\the\mydimenb)
+ \begingroup
+ \newlocaldimen\mydimena \mydimena3\onepoint
+ \newlocaldimen\mydimenb \mydimenb4\onepoint
+ \newlocaldimen\mydimenc \mydimenc5\onepoint
+ (\the\mydimena,\the\mydimenb,\the\mydimenc)
+ \begingroup
+ \newlocaldimen\mydimena \mydimena6\onepoint
+ \newlocaldimen\mydimenb \mydimenb7\onepoint
+ (\the\mydimena,\the\mydimenb)
+ \endgroup
+ \newlocaldimen\mydimend \mydimend8\onepoint
+ (\the\mydimena,\the\mydimenb,\the\mydimenc,\the\mydimend)
+ \endgroup
+ (\the\mydimena,\the\mydimenb)
+\endgroup
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+The allocated registers get zero values but you can of course set them to any
+value that fits their nature:
+
+\startlines \getbuffer \stoplines
+
+\startbuffer
+\begingroup
+ \setnewlocaldimen\mydimena 1\onepoint
+ \setnewlocaldimen\mydimenb 2\onepoint
+ (\the\mydimena,\the\mydimenb)
+ \begingroup
+ \setnewlocaldimen\mydimena 3\onepoint
+ \setnewlocaldimen\mydimenb 4\onepoint
+ \setnewlocaldimen\mydimenc 5\onepoint
+ (\the\mydimena,\the\mydimenb,\the\mydimenc)
+ \begingroup
+ \setnewlocaldimen\mydimena 6\onepoint
+ \setnewlocaldimen\mydimenb 7\onepoint
+ (\the\mydimena,\the\mydimenb)
+ \endgroup
+ \setnewlocaldimen\mydimend 8\onepoint
+ (\the\mydimena,\the\mydimenb,\the\mydimenc,\the\mydimend)
+ \endgroup
+ (\the\mydimena,\the\mydimenb)
+\endgroup
+\stopbuffer
+
+You can also use the next variant where you also pass the initial value:
+
+\typebuffer[option=TEX]
+
+So, again we get:
+
+\startlines \getbuffer \stoplines
+
+When used in the body of the macro there is of course a little overhead
+involved in the repetitive allocation but normally that can be neglected.
+
+\stopsection
+
+\startsection[title=Files]
+
+When adding these new allocators I also wondered about the read and write
+allocators. We don't use them in \CONTEXT\ but maybe users like them, so let's
+give an example and see what more demands they have:
+
+\starttyping[option=TEX]
+\integerdef\StartHere\numexpr\inputlineno+2\relax
+\starthiding
+SOME LINE 1
+SOME LINE 2
+SOME LINE 3
+SOME LINE 4
+\stophiding
+\integerdef\StopHere\numexpr\inputlineno-2\relax
+\stoptyping
+
+% We need to be in the file!
+
+\integerdef\StartHere\numexpr\inputlineno+2\relax
+\starthiding
+SOME LINE 1
+SOME LINE 2
+SOME LINE 3
+SOME LINE 4
+\stophiding
+\integerdef\StopHere\numexpr\inputlineno-2\relax
+
+\startbuffer
+\begingroup
+ \newlocalread\myreada
+ \immediate\openin\myreada {lowlevel-scope.tex}
+ \dostepwiserecurse{\StopHere}{\StartHere}{-1}{
+ \readline\myreada line #1 to \scratchstring #1 : \scratchstring \par
+ }
+ \blank
+ \dostepwiserecurse{\StartHere}{\StopHere}{1}{
+ \read \myreada line #1 to \scratchstring #1 : \scratchstring \par
+ }
+ \immediate\closein\myreada
+\endgroup
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+Here, instead of hard coded line numbers we used the stored values. The
+optional \type {line} keyword is a \LMTX\ speciality.
+
+\startlines \getbuffer \stoplines
+
+Actually an application can be found in a small (demonstration) module:
+
+\startbuffer
+\usemodule[system-readers]
+\stopbuffer
+
+\typebuffer[option=TEX] \getbuffer
+
+This provides the code for doing this:
+
+\starttyping[option=TEX]
+\startmarkedlines[test]
+SOME LINE 1
+SOME LINE 2
+SOME LINE 3
+\stopmarkedlines
+
+\begingroup
+ \newlocalread\myreada
+ \immediate\openin\myreada {\markedfilename{test}}
+ \dostepwiserecurse{\lastmarkedline{test}}{\firstmarkedline{test}}{-1}{
+ \readline\myreada line #1 to \scratchstring #1 : \scratchstring \par
+ }
+ \immediate\closein\myreada
+\endgroup
+\stoptyping
+
+As you see in these examples, we an locally define a read channel without
+getting a message about it already being defined.
+
+\stopsection
+
+\stopdocument
+
+
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex
new file mode 100644
index 00000000000..4f705e37caa
--- /dev/null
+++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex
@@ -0,0 +1,254 @@
+% language=us
+
+% It took some time to get the right balance of using the overload related features
+% but at some point it started feeling right. Of course it will never be as
+% perfectly timed and integrated as Gavin Harrison performance on "Threatening War"
+% (youtube movie) but that doesn't mean I should not aim for perfection. But as
+% with drumming, it takes practising and that is what I did on a subset of sources
+% when writing the engine code.
+%
+% It tooks a few weeks but November 21 2020 the last of the core files was turned
+% \LMTX, not that the work was done (checking to be done (thanks Wolfgang!), imp
+% files to be checked, many \LUA\ files to be updated) but it's a start. This time
+% the musical timestamp is listening to Nikola Cvetkovic (piano) and history
+% podcasts.
+
+\usemodule[system-tokens]
+
+\environment lowlevel-style
+
+\startdocument
+ [title=security,
+ color=middleorange]
+
+\startsection[title=Preamble]
+
+Here I will discuss a moderate security subsystem of \LUAMETATEX\ and therefore
+\CONTEXT\ \LMTX. This is not about security in the sense of the typesetting
+machinery doing harm to your environment, but more about making sure that a user
+doesn't change the behavior of the macro package in ways that introduce
+interference and thereby unwanted side effect. It's all about protecting macros.
+
+This is all very experimental and we need to adapt the \CONTEXT\ source code to
+this. Actually that will happen a few times because experiments trigger that. It
+might take a few years before the security model is finalized and all files are
+updated accordingly. There are lots of files and macros involved. In the process
+the underlying features in the engine might evolve.
+
+\stopsection
+
+\startsection[title=Flags]
+
+Before we go into the security levels we see what flags can be set. The \TEX\
+language has a couple of so called prefixes that can be used when setting values
+and defining macros. Any engine that has traditional \TEX\ with \ETEX\ extensions
+can do this:
+
+\starttyping[option=TEX]
+ \def\foo{foo}
+\global \def\foo{foo}
+\global\protected\def\foo{foo}
+\stoptyping
+
+And \LUAMETATEX\ adds another one:
+
+\starttyping[option=TEX]
+ \tolerant \def\foo{foo}
+\global\tolerant \def\foo{foo}
+\global\tolerant\protected\def\foo{foo}
+\stoptyping
+
+What these prefixes do is discussed elsewhere. For now is is enough to know that
+the two optional prefixes \type {\protected} and \type {\tolerant} make for four
+distinctive cases of macro calls.
+
+But there are more prefixes:
+
+\starttabulate
+\HL
+\NC \type {frozen} \NC a macro that has to be redefined in a managed way \NC \NR
+\NC \type {permanent} \NC a macro that had better not be redefined \NC \NR
+\NC \type {primitive} \NC a primitive that normally will not be adapted \NC \NR
+\NC \type {immutable} \NC a macro or quantity that cannot be changed, it is a constant \NC \NR
+\NC \type {mutable} \NC a macro that can be changed no matter how well protected it is \NC \NR
+\HL
+\NC \type {instance} \NC a macro marked as (for instance) be generated by an interface \NC \NR
+\HL
+\NC \type {noaligned} \NC the macro becomes acceptable as \type {\noalign} alias \NC \NR
+\HL
+\NC \type {overloaded} \NC when permitted the flags will be adapted \NC \NR
+\NC \type {enforced} \NC all is permitted (but only in zero mode or ini mode) \NC \NR
+\NC \type {aliased} \NC the macro gets the same flags as the original \NC \NR
+\HL
+\stoptabulate
+
+These prefixed set flags to the command at hand which can be a macro but
+basically any control sequence.
+
+To what extent the engine will complain when a property is changed in a way that
+violates the above depends on the parameter \type {\overloadmode}. When this
+parameter is set to zero no checking takes place. More interesting are values
+larger than zero. If that is the case, when a control sequence is flagged as
+mutable, it is always permitted to change. When it is set to immutable one can
+never change it. The other flags determine the kind of checking done. Currently
+the following overload values are used:
+
+\starttabulate[|l|l|c|c|c|c|c|]
+ \NC \NC \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR
+ \NC 1 \NC warning \NC \star \NC \star \NC \star \NC \NC \NC \NR
+ \NC 2 \NC error \NC \star \NC \star \NC \star \NC \NC \NC \NR
+ \NC 3 \NC warning \NC \star \NC \star \NC \star \NC \star \NC \NC \NR
+ \NC 4 \NC error \NC \star \NC \star \NC \star \NC \star \NC \NC \NR
+ \NC 5 \NC warning \NC \star \NC \star \NC \star \NC \star \NC \star \NC \NR
+ \NC 6 \NC error \NC \star \NC \star \NC \star \NC \star \NC \star \NC \NR
+\stoptabulate
+
+The even values (except zero) will abort the run. In \CONTEXT\ we plug in a
+callback that deals with the messages. A value of 255 will freeze this parameter.
+At level five and above the \type {instance} flag is also checked but no drastic
+action takes place. We use this to signal to the user that a specific instance is
+redefined (of course the definition macros can check for that too).
+
+So, how does it work. The following is okay:
+
+\starttyping[option=TEX]
+\def\MacroA{A}
+\def\MacroB{B}
+\let\MyMacro\MacroA
+\let\MyMacro\MacroB
+\stoptyping
+
+The first two macros are ordinary ones, and the last two lines just create an alias. Such
+an alias shares the definition, but when for instance \type {\MacroA} is redefined, its
+new meaning will not be reflected in the alias.
+
+\starttyping[option=TEX]
+\permanent\protected\def\MacroA{A}
+\permanent\protected\def\MacroB{B}
+\let\MyMacro\MacroA
+\let\MyMacro\MacroB
+\stoptyping
+
+This also works, because the \type {\let} will create an alias with the protected
+property but it will not take the \type {permanent} propery along. For that we need
+to say:
+
+\starttyping[option=TEX]
+\permanent\protected\def\MacroA{A}
+\permanent\protected\def\MacroB{B}
+\permanent\let\MyMacro\MacroA
+\permanent\let\MyMacro\MacroB
+\stoptyping
+
+or, when we want to copy all properties:
+
+\starttyping[option=TEX]
+\permanent\protected\def\MacroA{A}
+\permanent\protected\def\MacroB{B}
+\aliased\let\MyMacro\MacroA
+\aliased\let\MyMacro\MacroB
+\stoptyping
+
+However, in \CONTEXT\ we have commands that we like to protect against
+overloading but at the same time have a different meaning depending on the use
+case. An example is the \type {\NC} (next column) command that has a different
+implementation in each of the table mechanisms.
+
+\starttyping[option=TEX]
+\permanent\protected\def\NC_in_table {...}
+\permanent\protected\def\NC_in_tabulate{...}
+\aliased\let\NC\NC_in_table
+\aliased\let\NC\NC_in_tabulate
+\stoptyping
+
+Here the second aliasing of \type {\NC} fails (assuming of course that we enabled
+overload checking). One can argue that grouping can be used but often no grouping
+takes place when we redefine on the fly. Because \type {frozen} is less restrictive
+than \type {primitive} or \type {permanent}, and of course \type {immutable}, the
+next variant works:
+
+\starttyping[option=TEX]
+\frozen\protected\def\NC_in_table {...}
+\frozen\protected\def\NC_in_tabulate{...}
+\overloaded\let\NC\NC_in_table
+\overloaded\let\NC\NC_in_tabulate
+\stoptyping
+
+However, in practice, as we want to keep the overload checking, we have to do:
+
+\starttyping[option=TEX]
+\frozen\protected\def\NC_in_table {...}
+\frozen\protected\def\NC_in_tabulate{...}
+\overloaded\frozen\let\NC\NC_in_table
+\overloaded\frozen\let\NC\NC_in_tabulate
+\stoptyping
+
+or use \type {\aliased}, but there might be conflicting permissions. This is not
+that nice, so there is a kind of dirty trick possible. Consider this:
+
+\starttyping[option=TEX]
+\frozen\protected\def\NC_in_table {...}
+\frozen\protected\def\NC_in_tabulate{...}
+\def\setNCintable {\enforced\let\frozen\let\NC\NC_in_table}
+\def\setNCintabulate{\enforced\let\frozen\let\NC\NC_in_tabulate}
+\stoptyping
+
+When we're in so called \type {initex} mode or when the overload mode is zero,
+the \type {\enforced} prefix is internalized in a way that signals that the
+follow up is not limited by the overload mode and permissions. This definition
+time binding mechanism makes it possible to use \type {permanent} macros that
+users cannot redefine, but existing macros can, unless of course they tweak the
+mode parameter.
+
+Now keep in mind that users can always cheat but that is intentional. If you
+really want to avoid that you can set the overload mode to 255 after which it
+cannot be set any more. However, it can be useful to set the mode to zero (or
+some warning level) when foreign macro packages are used.
+
+\stopsection
+
+\startsection[title=Complications]
+
+One side effect of all this is that all those prefixes can lead to more code. On
+the other hand we save some due to the extended macro argument handling features.
+When you take the size of the format file as reference, in the end we get a
+somewhat smaller file. Every token that you add of remove gives a 8~bytes
+difference. The extra overhead that got added to the engine is compensated by the
+fact that some macro implementations can be more efficient. In the end, in spite
+of these new features and the more extensive testing of flags performance is
+about the same. \footnote {And if you wonder about memory, by compacting the used
+(often scattered) token memory before dumping I manages to save some 512K on the
+format file, so often the loss and gain are somewhere else.}
+
+\stopsection
+
+\startsection[title=Introspection]
+
+In case you want to get some details about the properties of a macro, you can
+check its meaning. The full variant shows all of them.
+
+\startbuffer
+% a macro with two optional arguments with optional spacing in between:
+
+\permanent\tolerant\protected\def\MyFoo[#1]#*[#2]{(#1)(#2)}
+
+\meaningless\MyFoo\par
+\meaning \MyFoo\par
+\meaningfull\MyFoo\par
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+\startpacked \getbuffer \stoppacked
+
+\stopsection
+
+% In \CONTEXT:
+% c! v! s! ??
+% newif newcount ... newconditional etc
+% userinterface (permanent)
+% primitives
+% noaligned
+% frozen is for users
+
+\stopdocument