diff options
author | Karl Berry <karl@freefriends.org> | 2021-03-06 21:58:15 +0000 |
---|---|---|
committer | Karl Berry <karl@freefriends.org> | 2021-03-06 21:58:15 +0000 |
commit | 974640d66e61e81cb197ad96fdff7b08343e4c5a (patch) | |
tree | 2e1f75f32f312b7f24ba82b4590ae230bcd6f399 /Master/texmf-dist/doc/context/sources/general/manuals/lowlevel | |
parent | b4fa72e61230aca75f7f6fbf988821f71edfb6b2 (diff) |
context
git-svn-id: svn://tug.org/texlive/trunk@58167 c570f23f-e606-0410-a88d-b1316a301751
Diffstat (limited to 'Master/texmf-dist/doc/context/sources/general/manuals/lowlevel')
9 files changed, 3087 insertions, 26 deletions
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-boxes.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-boxes.tex index 986d07b1b61..9a097b8780c 100644 --- a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-boxes.tex +++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-boxes.tex @@ -9,9 +9,7 @@ [title=boxes, color=middlered] -\startsection[title=Preamble] - -\startsubsection[title=Introduction] +\startsection[title=Introduction] An average \CONTEXT\ user will not use the low level box primitives but a basic understanding of how \TEX\ works doesn't hurt. In fact, occasionally using a box @@ -25,9 +23,9 @@ about all kind of glues, kerns and penalties, just boxes it is. This explanation will be extended when I feel the need (or users have questions that can be answered here). -\stopsubsection +\stopsection -\startsubsection[title=Boxes] +\startsection[title=Boxes] This paragraph of text is made from lines that contain words that themselves contain symbolic representations of characters. Each line is wrapped in a so @@ -68,7 +66,7 @@ other hand wraps a linked list of so called nodes: glyphs, kerns, glue, penalties, rules, boxes, etc. It is a container with properties like width, height, depth and shift. -\stopsubsection +\stopsection \stopsection @@ -469,10 +467,10 @@ a private attribute we define one. \startbuffer \newattribute\MyAt \setbox\scratchbox\hbox attr \MyAt 123 {whatever} -[\the\boxattr\scratchbox\MyAt] -\boxattr\scratchbox\MyAt 456 -[\the\boxattr\scratchbox\MyAt] -[\ifnum\boxattr\scratchbox\MyAt>400 okay\fi] +[\the\boxattribute\scratchbox\MyAt] +\boxattribute\scratchbox\MyAt 456 +[\the\boxattribute\scratchbox\MyAt] +[\ifnum\boxattribute\scratchbox\MyAt>400 okay\fi] \stopbuffer \typebuffer[option=TEX] diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-characters.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-characters.tex new file mode 100644 index 00000000000..ee3ab404442 --- /dev/null +++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-characters.tex @@ -0,0 +1,256 @@ +% language=us + +\environment lowlevel-style + +\startdocument + [title=characters, + color=middlered] + +\startsection[title=Introduction] + +This explanation is part of the low level manuals because in practice users will +not have to deal with these matters in \MKIV\ and even less in \LMTX. You can +skip to the last section for commands. + +\stopsection + +\startsection[title=History] + +If we travel back in time to when \TEX\ was written we end up in eight bit +character universe. In fact, the first versions assumed seven bits, but for +comfortable use with languages other than English that was not sufficient. +Support for eight bits permits the usage of so called code pages as supported by +operating systems. Although \ASCII\ input became kind of the standard soon +afterwards, the engine can be set up for different encodings. This is not only +true for \TEX, but for many of its companions, like \METAFONT\ and therefore +\METAPOST. \footnote {This remapping to an internal representation (e.g. ebcdic) +is not present in \LUATEX\ where we assume \UTF8 to be the input encoding. The +\METAPOST\ library that comes with \LUATEX\ still has that code but in +\LUAMETATEX\ it's gone. There one can set up the machinery to be \UTF8 aware +too.} + +Core components of a \TEX\ engine are hyphenation of words, applying +inter|-|character kerns and build ligatures. In traditional \TEX\ engines those +processes are interwoven into the par builder but in \LUATEX\ these are separate +stages. The original approach is the reason that there is a relation between the +input encoding and the font encoding: the character in the input is the slot used +in a reference to a glyph. When producing the final result (e.g.\ \PDF) there can +also be a mapping to an index in a font resource. + +\starttyping +input A [tex ->] font slot A [backend ->] glyph index A +\stoptyping + +The mapping that \TEX\ does is normally one|-|to|-|one but an input character can +undergo some transformation. For instance a character beyond \ASCII\ 126 can be +made active and expand to some character number that then becomes the font slot. +So, it is the expansion (or meaning) of a character that end up as numeric +reference in the glyph node. Virtual fonts can introduce yet another remapping +but that's only visible in the backend. + +Actually, in \LUATEX\ the same happens but in practice there is no need to go +active because (at least in \CONTEXT) we assume a \UNICODE\ path so there the +font slot is the \UNICODE\ got from the \UTF8 input. + +In the eight bit universe macro packages (have to) provide all kind of means to +deal with (in the perspective of English) special characters. For instance, \type +{\"a} would put a diaeresis on top of the a or even better, refer to a character +in the encoding that the chosen font provides. Because there are some limitations +of what can go in an eight bit font, and because in different countries the used +\TEX\ fonts evolved kind of independent, we ended up with quite some different +variants of fonts. It was only with the Latin Modern project that this became +better. Interesting is that when we consider the fact that such a font has often +also hardly used symbols (like registered or copyright) coming up with an +encoding vector that covers most (latin based) European languages (scripts) is +not impossible \footnote {And indeed in the Latin Modern project we came up with +one but it was already to late for it to become popular.} Special symbols could +simply go into a dedicated font, also because these are always accessed via a +macro so who cares about the input. It never happened. + +Keep in mind that when \UTF8 is used with eight bit engines, \CONTEXT\ will +convert sequences of characters into a slot in a font (depending on the font +encoding used which itself depends on the coverage needed). For this every first +(possible) byte of a multibyte \UTF\ sequence is an active character, which is no +big deal because these are outside the \ASCII\ range. Normal \ASCII\ characters +are single byte \UTF\ sequences and fall through without treatment. + +Anyway, in \CONTEXT\ \MKII\ we dealt with this by supporting mixed encodings, +depending on the (local) language, referencing the relevant font. It permits +users to enter the text in their preferred input encoding and also get the words +properly hyphenated. But we can leave these \MKII\ details behind. + +\stopsection + +\startsection[title=The heritage] + +In \MKIV\ we got rid of input and font encodings, although one can still load +files in a specific code page. \footnote {I'm not sure if users ever depend on an +input encoding different from \UTF8.} We also kept the means to enter special +characters, if only because text editors seldom support(ed) a wide range of +visual editing of those. This is why we still have + +\starttyping[option=TEX] +\"u \^a \v{s} \AE \ij \eacute \oslash +\stoptyping + +and many more. The ones with one character names are rather common in the \TEX\ +community but it is definitely a weird mix of symbols. The next two are kind of +outdated: in these days you delegate that to the font handler, where turning them +into \quote {single} character references depends on what the font offers, how it +is set up with respect to (for instance) ligatures, and even might depend on +language or script. + +The ones with the long names partly are tradition, but as we have a lot of them, +in \MKII\ they actually serve a purpose. These verbose names are used in the so +called encoding vectors and are part of the \UTF\ expansion vectors. They are +also used in labels so that we have a good indication if what goes in there: +remember that in those times editors often didn't show characters, unless the +font for display had them, or the operating system somehow provided them from +another font. These verbose names are used for latin, greek and cyrillic and for +some other scripts and symbols. They take up quite a bit of hash space and the +format file. \footnote {In \MKII\ we have an abstract front|-|end with respect to +encodings and also an abstract backend with respect to supported drivers but both +approaches no longer make sense today.} + +\stopsection + +\startsection[title=The \LMTX\ approach] + +In the process of tagging all (public) macros in \LMTX\ (which happened in +2020|-|2021) I wondered if we should keep these one character macros, the +references to special characters and the verbose ones. When asked on the mailing +list it became clear that users still expect the short ones to be present, often +just because old \BIBTEX\ files are used that might need them. However, in \MKIV\ +and \LMTX\ we load \BIBTEX\ files in a way that turn these special character +references into proper \UTF8 input so it makes a weak argument. Anyway, although +they could go, for now we keep them because users expect them. However, in \LMTX\ +the implementation is somewhat different now, a bit more efficient in terms of +hash and memory, potentially a bit less efficient in runtime, but no one will +notice that. + +A new command has been introduced, the very short \type {\chr}. + +\startbuffer +\chr {à} \chr {á} \chr {ä} +\chr {`a} \chr {'a} \chr {"a} +\chr {a acute} \chr {a grave} \chr {a umlaut} +\chr {aacute} \chr {agrave} \chr {aumlaut} +\stopbuffer + +\typebuffer[option=TEX] + +In the first line the composed character using two characters, a base and a so +called mark. Actually, one doesn't have to use \type {\chr} in that case because +\CONTEXT\ does already collapse characters for you. The second line looks like +the shortcuts \type {\`}, \type {\'} and \type {\"}. The third and fourth lines +could eventually replace the more symbolic long names, if we feel the need. Watch +out: in \UNICODE\ input the marks come {\em after}. + +\startlines \getbuffer \stoplines + +Currently the repertoire is somewhat limited but it can be easily be extended. It +all depends on user needs (doing Greek and Cyrillic for instance). The reason why +we actually save code deep down is that the helpers for this have always been +there. \footnote {So if needed I can port this approach back to \MKIV, but for +now we keep it as is because we then have a reference.} + +The \type {\"} commands are now just aliases to more verbose and less hackery +looking macros: + +\starttabulate[|||||] + \NC \type {\withgrave} \NC \withgrave {a} \NC \type {\`} \NC \`{a} \NC \NR + \NC \type {\withacute} \NC \withacute {a} \NC \type {\'} \NC \'{a} \NC \NR + \NC \type {\withcircumflex} \NC \withcircumflex {a} \NC \type {\^} \NC \^{a} \NC \NR + \NC \type {\withtilde} \NC \withtilde {a} \NC \type {\~} \NC \~{a} \NC \NR + \NC \type {\withmacron} \NC \withmacron {a} \NC \type {\=} \NC \={a} \NC \NR + \NC \type {\withbreve} \NC \withbreve {e} \NC \type {\u} \NC \u{e} \NC \NR + \NC \type {\withdotaccent} \NC \withdot {c} \NC \type {\.} \NC \.{c} \NC \NR + \NC \type {\withdiaeresis} \NC \withdieresis {e} \NC \type {\"} \NC \"{e} \NC \NR + \NC \type {\withring} \NC \withring {u} \NC \type {\r} \NC \r{u} \NC \NR + \NC \type {\withhungarumlaut} \NC \withhungarumlaut{u} \NC \type {\H} \NC \H{u} \NC \NR + \NC \type {\withcaron} \NC \withcaron {e} \NC \type {\v} \NC \v{e} \NC \NR + \NC \type {\withcedilla} \NC \withcedilla {e} \NC \type {\c} \NC \c{e} \NC \NR + \NC \type {\withogonek} \NC \withogonek {e} \NC \type {\k} \NC \k{e} \NC \NR +\stoptabulate + +Not all fonts have these special characters. Most natural is to have them +available as precomposed single glyphs, but it can be that they are just two +shapes with the marks anchored to the base. It can even be that the font somehow +overlays them, assuming (roughly) equal widths. The \type {compose} font feature +in \CONTEXT\ normally can handle most well. + +An occasional ugly rendering doesn't matter that much: better have something than +nothing. But when it's the main language (script) that needs them you'd better +look for a font that handles them. When in doubt, in \CONTEXT\ you can enable +checking: + +\starttabulate[|l|l|] + \BC command \BC equivalent to \NC \NR + \NC \type {\checkmissingcharacters} \NC \type{\enabletrackers[fonts.missing]} \NC \NR + \NC \type {\removemissingcharacters} \NC \type{\enabletrackers[fonts.missing=remove]} \NC \NR + \NC \type {\replacemissingcharacters} \NC \type{\enabletrackers[fonts.missing=replace]} \NC \NR + \NC \type {\handlemissingcharacters} \NC \type{\enabletrackers[fonts.missing={decompose,replace}]} \NC \NR +\stoptabulate + +The decompose variant will try to turn a composed character into its components +so that at least you get something. If that fails it will inject a replacement +symbol that stands out so that you can check it. The console also mentions +missing glyphs. You don't need to enable this by default \footnote {There is some +overhead involved here.} but you might occasionally do it when you use a font for +the first time. + +In \LMTX\ this mechanism has been upgraded so that replacements follow the shape +and are actually real characters. The decomposition has not yet been ported back +to \MKIV. + +The full list of commands can be queried when a tracing module is loaded: + +\startbuffer +\usemodule[s][characters-combinations] + +\showcharactercombinations +\stopbuffer + +\typebuffer + +We get this list: + +\getbuffer + +Some combinations are special for \CONTEXT\ because \UNICODE\ doesn't specify +decomposition for all composed characters. + +\stopsection + +\startsubject[title=Colofon] + +\starttabulate +\NC Author \NC Hans Hagen \NC \NR +\NC \CONTEXT \NC \contextversion \NC \NR +\NC \LUAMETATEX \NC \texengineversion \NC \NR +\NC Support \NC www.pragma-ade.com \NC \NR +\NC \NC contextgarden.net \NC \NR +\stoptabulate + +\stopsubject + +\stopdocument + +% on an old machine, so consider them just relative measures +% +% mkiv lmtx +% +% 0.012 0.009 % faster core code +% 0.028 0.036 % different io code path +% 0.055 0.043 % different io code path / faster core code +% 0.156 0.129 % more efficient resolving +% 0.153 0.119 % more efficient resolving +% +% \ifdefined\withdieresis\else\let\withdieresis\"\fi % for mkiv +% +% \setbox0\hpack{\testfeatureonce{100000}{ü}} \par \elapsedtime \par % direct +% \setbox0\hpack{\testfeatureonce{100000}{ü}} \par \elapsedtime \par % composed (input) +% \setbox0\hpack{\testfeatureonce{100000}{u{}̈}} \par \elapsedtime \par % overlay +% \setbox0\hpack{\testfeatureonce{100000}{\withdieresis{u}}} \par \elapsedtime \par % official also \"u +% \setbox0\hpack{\testfeatureonce{100000}{\" u}} \par \elapsedtime \par % alias of previous + diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-conditionals.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-conditionals.tex index ea3c9e1a244..9be2fb4ec6f 100644 --- a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-conditionals.tex +++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-conditionals.tex @@ -6,6 +6,8 @@ [title=conditionals, color=middleblue] +\pushoverloadmode + \startsection[title=Preamble] \startsubsection[title=Introduction] @@ -1121,13 +1123,15 @@ This test is like \type {\ifcmpnum} but for dimensions. \startsubsection[title={\tex{ifchkdim}}] -This test is like \type {\ifchknum} but for dimensions. +This test is like \type {\ifchknum} but for dimensions. The last checked value is +available as \type {\lastchknum}. \stopsubsection \startsubsection[title={\tex{ifdimval}}] -This test is like \type {\ifnumval} but for dimensions. +This test is like \type {\ifnumval} but for dimensions. The last checked value is +available as \type {\lastchkdim} \stopsubsection @@ -1224,6 +1228,15 @@ defined at the user level or is a build in one. This one might evolve. \stopsubsection +\startsubsection[title={\tex{ifarguments}}] + +This conditional can be used to check how many arguments were matched. It only +makes sense when used with macros defined with the \type {\tolerant} prefix +and|/|or when the sentinel \type {\ignorearguments} after the arguments is used. +More details can be found in the lowlevel macros manual. + +\stopsubsection + \startsubsection[title={\tex{orelse}}] This it not really a test primitive but it does act that way. Say that we have this: @@ -1254,13 +1267,16 @@ A bit nicer looks this: \fi \stoptyping -We stay at the same level and the only test that cannot be used this way is \type -{\ifcondition} but that is no real problem. Sometimes a more flat test tree had -advantages but if you think that it gives better performance then you will be -disappointed. The fact that we stay at the same level is compensated by a bit -more parsing, so unless you have millions such cases (or expansions) it might -make a bit of a difference. As mentioned, I'm a bit sensitive for how code looks so -that was the main motivation for introducing it. +% We stay at the same level and the only test that cannot be used this way is \type +% {\ifcondition} but that is no real problem. Sometimes a more flat test tree had + +We stay at the same level. Sometimes a more flat test tree had advantages but if +you think that it gives better performance then you will be disappointed. The +fact that we stay at the same level is compensated by a bit more parsing, so +unless you have millions such cases (or expansions) it might make a bit of a +difference. As mentioned, I'm a bit sensitive for how code looks so that was the +main motivation for introducing it. There is a companion \type {\orunless} +continuation primitive. A rather neat trick is the definition of \type {\quitcondition}: @@ -1294,18 +1310,23 @@ reason to go this complex but obscure \TEX\ code attracts some users so \unknown When you have a macro that has for instance assignments, and when you expand that macro inside an \type {\edef}, these assignments are not actually expanded but -tokenized. In \LUATEX\ there is a way to immediately apply these assignments and -that feature can be used to write a fully expandable user test. For instance: +tokenized. In \LUAMETATEX\ there is a way to apply these assignments without side +effects and that feature can be used to write a fully expandable user test. For +instance: \startbuffer \def\truecondition {\iftrue} \def\falsecondition{\iffalse} \def\fontwithidhaschar#1#2% - {\immediateassignment\scratchcounter\numexpr\fontid\font\relax - \immediateassignment\setfontid\numexpr#1\relax + {\beginlocalcontrol + \scratchcounter\numexpr\fontid\font\relax + \setfontid\numexpr#1\relax + \endlocalcontrol \iffontchar\font\numexpr#2\relax - \immediateassignment\setfontid\scratchcounter + \beginlocalcontrol + \setfontid\scratchcounter + \endlocalcontrol \expandafter\truecondition \else \expandafter\falsecondition @@ -1394,6 +1415,125 @@ with \type {\ifcondition} (it has bitten me already a few times). \stopsection +\startsection[title=Relaxing] + +When \TEX\ scans for a number or dimension it has to check tokens one by one. On +the case of a number, the scanning stops when there is no digit, in the case of a +dimension the unit determine the end of scanning. In the case of a number, when a +token is not a digit that token gets pushed back. When digits are scanned a +trailing space or \type {\relax} is pushed back. Instead of a number of dimension +made from digits, periods and units, the scanner also accepts registers, both the +direct accessors like \type {\count} and \type {\dimen} and those represented by +one token. Take these definitions: + +\startbuffer +\newdimen\MyDimenA \MyDimenA=1pt \dimen0=\MyDimenA +\newdimen\MyDimenB \MyDimenB=2pt \dimen2=\MyDimenB +\stopbuffer + +\typebuffer \getbuffer + +I will use these to illustrate the side effects of scanning. Watch the spaces +in the result. + +% \startbuffer[a] +% \testfeatureonce{1000000}{ +% \whatever{1pt}{2pt}% +% \whatever{1pt}{1pt}% +% \whatever{\dimen0}{\dimen2}% +% \whatever{\dimen0}{\dimen0}% +% \whatever\MyDimenA\MyDimenB +% \whatever\MyDimenA\MyDimenB +% } \elapsedtime +% \stopbuffer + +\startbuffer[b] +\starttabulate[|T|T|] +\NC \type {\whatever{1pt}{2pt}} \NC \edef\temp{\whatever {1pt}{2pt}}[\meaning\temp] \NC \NR +\NC \type {\whatever{1pt}{1pt}} \NC \edef\temp{\whatever {1pt}{1pt}}[\meaning\temp] \NC \NR +\NC \type {\whatever{\dimen0}{\dimen2}} \NC \edef\temp{\whatever{\dimen0}{\dimen2}}[\meaning\temp] \NC \NR +\NC \type {\whatever{\dimen0}{\dimen0}} \NC \edef\temp{\whatever{\dimen0}{\dimen0}}[\meaning\temp] \NC \NR +\NC \type {\whatever\MyDimenA\MyDimenB} \NC \edef\temp{\whatever\MyDimenA\MyDimenB}[\meaning\temp] \NC \NR +\NC \type {\whatever\MyDimenA\MyDimenB} \NC \edef\temp{\whatever\MyDimenA\MyDimenB}[\meaning\temp] \NC \NR +\stoptabulate +\stopbuffer + +First I show what effect we want to avoid. When second argument contains a number +(digits) the zero will become part of it so we actually check \type {\dimen00} +here. + +\startbuffer[c] +\def\whatever#1#2% + {\ifdim#1=#20\else1\fi} +\stopbuffer + +\typebuffer[c] \getbuffer[c,b] + +The solution is to add a space but watch how that one can end up in the result: + +\startbuffer[c] +\def\whatever#1#2% + {\ifdim#1=#2 0\else1\fi} +\stopbuffer + +\typebuffer[c] \getbuffer[c,b] + +A variant is using \type {\relax} and this time we get this token retained in +the output. + +\startbuffer[c] +\def\whatever#1#2% + {\ifdim#1=#2\relax0\else1\fi} +\stopbuffer + +\typebuffer[c] \getbuffer[c,b] + +A solution that doesn't have side effects of forcing the end of a number (using a +space or \type {\relax} is one where we use expressions. The added overhead of +scanning expressions is taken for granted because the effect is what we like: + +\startbuffer[c] +\def\whatever#1#2% + {\ifdim\dimexpr#1\relax=\dimexpr#2\relax0\else1\fi} +\stopbuffer + +\typebuffer[c] \getbuffer[c,b] + +Just for completeness we show a more obscure trick: this one hides assignments to +temporary variables. Although performance is okay, it is the least efficient +one so far. + +\ifdefined\beginlocalcontrol + +\startbuffer[c] +\def\whatever#1#2% + {\beginlocalcontrol + \MyDimenA#1\relax + \MyDimenB#2\relax + \endlocalcontrol + \ifdim\MyDimenA=\MyDimenB0\else1\fi} +\stopbuffer + +\typebuffer[c] \getbuffer[c,b] + +\fi + +It is kind of a game to come up with alternatives but for sure those involve +dirty tricks and more tokens (and runtime). The next can be considered a dirty +trick too: we use a special variant of \type {\relax}. When a number is scanned +it acts as relax, but otherwise it just is ignored and disappears. + +\ifdefined\norelax\else\let\norelax\relax\fi + +\startbuffer[c] +\def\whatever#1#2% + {\ifdim#1=#2\norelax0\else1\fi} +\stopbuffer + +\typebuffer[c] \getbuffer[c,b] + +\stopsection + \startsubject[title=Colofon] \starttabulate @@ -1406,4 +1546,18 @@ with \type {\ifcondition} (it has bitten me already a few times). \stopsubject +\popoverloadmode + \stopdocument + +% \def\foo{foo=bar} +% \def\oof{foo!bar} +% \scratchtoks{=} + +% \ifhasxtoks\scratchtoks{foo!bar} YES\else NOP\fi\par +% \ifhasxtoks\scratchtoks{foo=bar} YES\else NOP\fi\par + +% \showluatokens\foo + +% \ifhastoks\scratchtoks\oof YES\else NOP\fi\par +% \ifhastoks\scratchtoks\foo YES\else NOP\fi\par diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-expansion.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-expansion.tex index 1e2e00a35ad..15faa802dc4 100644 --- a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-expansion.tex +++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-expansion.tex @@ -315,7 +315,7 @@ Although you can never really get back to the original input, you can come prett close, with: \startbuffer -\normaldetokenize{this can $ be anything \bgroup} +\detokenize{this can $ be anything \bgroup} \stopbuffer \typebuffer[option=TEX] @@ -432,9 +432,45 @@ These two macros now have the meaning: \startsection[title={\LUAMETATEX\ primitives}] -{\em todo} +To be discussed: + +\starttyping +\expand +\expandtoken +\localcontrol +\localcontrolled +\beginlocalcontrol ... \endlocalcontrol +\immediate +\stoptyping + +And maybe also here: + +\starttyping +\tokenized : a bonus +\scantokens : original etex, now using the lua method +\stoptyping % \aftergroups +% \aftergrouped + +And: + +\starttyping + \def\foo{foo} +\protected\def\oof{oof} + +\csname foo\endcsname +\csname oof\endcsname +\csname \foo\endcsname +% \csname \oof\endcsname % error in luametatex + +\ifcsname foo\endcsname yes\else nop\fi +\ifcsname oof\endcsname yes\else nop\fi +\ifcsname \foo\endcsname yes\else nop\fi +\ifcsname \oof\endcsname yes\else nop\fi % nop in luametatex +\stoptyping + +\stoptext \stopsection diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-grouping.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-grouping.tex new file mode 100644 index 00000000000..eadcca6a9fd --- /dev/null +++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-grouping.tex @@ -0,0 +1,170 @@ +% language=us + +\environment lowlevel-style + +\startdocument + [title=grouping, + color=middlecyan] + +\startsection[title=Introduction] + +This is a rather short explanation. I decided to write it after presenting the +other topics at the 2019 \CONTEXT\ meeting where there was a question about +grouping. + +\stopsection + +\startsection[title=\PASCAL] + +In a language like \PASCAL, the language that \TEX\ has been written in, or +\MODULA, its successor, there is no concept of grouping like in \TEX. But we can +find keywords that suggests this: + +\starttyping +for i := 1 to 10 do begin ... end +\stoptyping + +This language probably inspired some of the syntax of \TEX\ and \METAPOST. For +instance an assignment in \METAPOST\ uses \type {:=} too. However, the \type +{begin} and \type {end} don't really group but define a block of statements. You +can have local variables in a procedure or function but the block is just a way +to pack a sequence of statements. + +\stopsection + +\startsection[title=\TEX] + +In \TEX\ macros (or source code) the following can occur: + +\starttyping +\begingroup + ... +\endgroup +\stoptyping + +as well as: + +\starttyping +\bgroup + ... +\egroup +\stoptyping + +Here we really group in the sense that assignments to variables inside a group +are forgotten afterwards. All assignments are local to the group unless they are +explicitly done global: + +\starttyping +\scratchcounter=1 +\def\foo{foo} +\begingroup + \scratchcounter=2 + \global\globalscratchcounter=2 + \gdef\foo{FOO} +\endgroup +\stoptyping + +Here \type {\scratchcounter} is still one after the group is left but its global +counterpart is now two. The \type {\foo} macro is also changed globally. + +Although you can use both sets of commands to group, you cannot mix them, so this +will trigger an error: + +\starttyping +\bgroup +\endgroup +\stoptyping + +The bottomline is: if you want a value to persist after the group, you need to +explicitly change its value globally. This makes a lot of sense in the perspective +of \TEX. + +\stopsection + +\startsection[title=\METAPOST] + +The \METAPOST\ language also has a concept of grouping but in this case it's more like a +programming language. + +\starttyping +begingroup ; + n := 123 ; +engroup ; +\stoptyping + +In this case the value of \type {n} is 123 after the group is left, unless you do +this (for numerics there is actually no need to declare them): + +\starttyping +begingroup ; + save n ; numeric n ; n := 123 ; +engroup ; +\stoptyping + +Given the use of \METAPOST\ (read: \METAFONT) this makes a lot of sense: often +you use macros to simplify code and you do want variables to change. Grouping in +this language serves other purposes, like hiding what is between these commands +and let the last expression become the result. In a \type {vardef} grouping is +implicit. + +So, in \METAPOST\ all assignments are global, unless a variable is explicitly +saved inside a group. + +\stopsection + +\startsection[title=\LUA] + +In \LUA\ all assignments are global unless a variable is defines local: + +\starttyping +local x = 1 +local y = 1 +for i = 1, 10 do + local x = 2 + y = 2 +end +\stoptyping + +Here the value of \type {x} after the loop is still one but \type {y} is now two. +As in \LUATEX\ we mix \TEX, \METAPOST\ and \LUA\ you can mix up these concepts. +Another mixup is using \type {:=}, \type {endfor}, \type {fi} in \LUA\ after done +some \METAPOST\ coding or using \type {end} instead of \type {endfor} in +\METAPOST\ which can make the library wait for more without triggering an error. +Proper syntax highlighting in an editor clearly helps. + +\stopsection + +\startsection[title=\CCODE] + +The \LUA\ language is a mix between \PASCAL\ (which is one reason why I like it) +and \CCODE. + +\starttyping +int x = 1 ; +int y = 1 ; +for (i=1; i<=10;i++) { + int x = 2 ; + y = 2 ; +} +\stoptyping + +The semicolon is also used in \PASCAL\ but there it is a separator and not a +statement end, while in \METAPOST\ it does end a statement (expression). + +\stopsection + +\stopsection + +\startsubject[title=Colofon] + +\starttabulate +\NC Author \NC Hans Hagen \NC \NR +\NC \CONTEXT \NC \contextversion \NC \NR +\NC \LUAMETATEX \NC \texengineversion \NC \NR +\NC Support \NC www.pragma-ade.com \NC \NR +\NC \NC contextgarden.net \NC \NR +\stoptabulate + +\stopsubject + +\stopdocument diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-macros.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-macros.tex new file mode 100644 index 00000000000..ef527732be3 --- /dev/null +++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-macros.tex @@ -0,0 +1,904 @@ +% language=us + +% Extending the macro argument parser happened stepwise and at each step a bit of +% \CONTEXT\ code was adapted for testing. At the beginning of October the 20201010 +% version of \LUAMETATEX\ was more of less complete, and I decided to adapt some +% more and more intrusive too. Of course that resulted in some more files than I +% had intended so mid October about 100 files were adapted. When this works out +% well, I'll do some more. In the process many macros got the frozen property so +% that was also a test and we'll see how that works out (as it can backfire). As +% usual, here is a musical timestamp: working on this happened when Pineapple Thief +% released \quotation {Versions of the Truth} which again a magnificent drumming by +% Gavin Harrison. + + +% \permanent\tolerant\protected\def\xx[#1]#*#;[#2]#:#3% loops .. todo + + +\usemodule[system-tokens] + +\environment lowlevel-style + +\startdocument + [title=macros, + color=middleorange] + +\startsection[title=Preamble] + +This chapter overlaps with other chapters but brings together some extensions to +the macro definition and expansion parts. As these mechanisms were stepwise +extended, the other chapters describe intermediate steps in the development. + +Now, in spite of the extensions discussed here the main ides is still that we +have \TEX\ act like before. We keep the charm of the macro language but these +additions make for easier definitions, but (at least initially) none that could +not be done before using more code. + +\stopsection + +\startsection[title=Definitions] + +A macro definition normally looks like like this: \footnote {The \type +{\dontleavehmode} command make the examples stay on one line.} + +\startbuffer[definition] +\def\macro#1#2% + {\dontleavehmode\hbox to 6em{\vl\type{#1}\vl\type{#2}\vl\hss}} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +Such a macro can be used as: + +\startbuffer[example] +\macro {1}{2} +\macro {1} {2} middle space gobbled +\macro 1 {2} middle space gobbled +\macro {1} 2 middle space gobbled +\macro 1 2 middle space gobbled +\stopbuffer + +\typebuffer[example][option=TEX] + +We show the result with some comments about how spaces are handled: + +\startlines \getbuffer[example] \stoplines + +A definition with delimited parameters looks like this: + +\startbuffer[definition] +\def\macro[#1]% + {\dontleavehmode\hbox to 6em{\vl\type{#1}\vl\hss}} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +When we use this we get: + +\startbuffer[example] +\macro [1] +\macro [ 1] leading space kept +\macro [1 ] trailing space kept +\macro [ 1 ] both spaces kept +\stopbuffer + +\typebuffer[example][option=TEX] + +Again, watch the handling of spaces: + +\startlines \getbuffer[example] \stoplines + +Just for the record we show a combination: + +\startbuffer[definition] +\def\macro[#1]#2% + {\dontleavehmode\hbox to 6em{\vl\type{#1}\vl\type{#2}\vl\hss}} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +With this: + +\startbuffer[example] +\macro [1]{2} +\macro [1] {2} +\macro [1] 2 +\stopbuffer + +\typebuffer[example][option=TEX] + +we can again see the spaces go away: + +\startlines \getbuffer[example] \stoplines + +A definition with two separately delimited parameters is given next: + +\startbuffer[definition] +\def\macro[#1#2]% + {\dontleavehmode\hbox to 6em{\vl\type{#1}\vl\type{#2}\vl\hss}} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +When used: + +\startbuffer[example] +\macro [12] +\macro [ 12] leading space gobbled +\macro [12 ] trailing space kept +\macro [ 12 ] leading space gobbled, trailing space kept +\macro [1 2] middle space kept +\macro [ 1 2 ] leading space gobbled, middle and trailing space kept +\stopbuffer + +\typebuffer[example][option=TEX] + +We get ourselves: + +\startlines \getbuffer[example] \stoplines + +These examples demonstrate that the engine does some magic with spaces before +(and therefore also between multiple) parameters. + +We will now go a bit beyond what traditional \TEX\ engines do and enter the +domain of \LUAMETATEX\ specific parameter specifiers. We start with one that +deals with this hard coded space behavior: + +\startbuffer[definition] +\def\macro[#^#^]% + {\dontleavehmode\hbox to 6em{\vl\type{#1}\vl\type{#2}\vl\hss}} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +The \type {#^} specifier will count the parameter, so here we expect again two +arguments but the space is kept when parsing for them. + +\startbuffer[example] +\macro [12] +\macro [ 12] +\macro [12 ] +\macro [ 12 ] +\macro [1 2] +\macro [ 1 2 ] +\stopbuffer + +\typebuffer[example][option=TEX] + +Now keep in mind that we could deal well with all kind of parameter handling in +\CONTEXT\ for decades, so this is not really something we missed, but it +complements the to be discussed other ones and it makes sense to have that level +of control. Also, availability triggers usage. Nevertheless, some day the \type +{#^} specifier will come in handy. + +\startlines \getbuffer[example] \stoplines + +We now come back to an earlier example: + +\startbuffer[definition] +\def\macro[#1]% + {\dontleavehmode\hbox spread 1em{\vl\type{#1}\vl\hss}} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +When we use this we see that the braces in the second call are removed: + +\startbuffer[example] +\macro [1] +\macro [{1}] +\stopbuffer + +\typebuffer[example][option=TEX] \getbuffer[example] + +This can be prohibited by the \type {#+} specifier, as in: + +\startbuffer[definition] +\def\macro[#+]% + {\dontleavehmode\hbox spread 1em{\vl\type{#1}\vl\hss}} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +As we see, the braces are kept: + +\startbuffer[example] +\macro [1] +\macro [{1}] +\stopbuffer + +\typebuffer[example][option=TEX] + +Again, we could easily get around that (for sure intended) side effect but it just makes nicer +code when we have a feature like this. + +\getbuffer[example] + +Sometimes you want to grab an argument but are not interested in the results. For this we have +two specifiers: one that just ignores the argument, and another one that keeps counting but +discards it, i.e.\ the related parameter is empty. + +\startbuffer[definition] +\def\macro[#1][#0][#3][#-][#4]% + {\dontleavehmode\hbox spread 1em + {\vl\type{#1}\vl\type{#2}\vl\type{#3}\vl\type{#4}\vl\hss}} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +The second argument is empty and the fourth argument is simply ignored which is why we need +\type {#4} for the fifth entry. + +\startbuffer[example] +\macro [1][2][3][4][5] +\stopbuffer + +\typebuffer[example][option=TEX] + +Here is proof that it works: + +\getbuffer[example] + +The reasoning behind dropping arguments is that for some cases we get around the +nine argument limitation, but more important is that we don't construct token +lists that are not used, which is more memory (and maybe even \CPU\ cache) +friendly. + +Spaces are always kind of special in \TEX, so it will be no surprise that we have +another specifier that relates to spaces. + +\startbuffer[definition] +\def\macro[#1]#*[#2]% + {\dontleavehmode\hbox spread 1em{\vl\type{#1}\vl\type{#2}\vl\hss}} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +This permits usage like the following: + +\startbuffer[example] +\macro [1][2] +\macro [1] [2] +\stopbuffer + +\typebuffer[example][option=TEX] \getbuffer[example] + +Without the optional \quote {grab spaces} specifier the second line would +possibly throw an error. This because \TEX\ then tries to match \type{][} so the +\type {] [} in the input is simply added to the first argument and the next +occurrence of \type {][} will be used. That one can be someplace further in your +source and if not \TEX\ complains about a premature end of file. But, with the +\type {#*} option it works out okay (unless of course you don't have that second +argument \type {[2]}. + +Now, you might wonder if there is a way to deal with that second delimited +argument being optional and of course that can be programmed quite well in +traditional macro code. In fact, \CONTEXT\ does that a lot because it is set up +as a parameter driven system with optional arguments. That subsystem has been +optimized to the max over years and it works quite well and performance wise +there is very little to gain. However, as soon as you enable tracing you end up +in an avalanche of expansions and that is no fun. + +This time the solution is not in some special specifier but in the way a macro +gets defined. + +\startbuffer[definition] +\tolerant\def\macro[#1]#*[#2]% + {\dontleavehmode\hbox spread 1em{\vl\type{#1}\vl\type{#2}\vl\hss}} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +The magic \type {\tolerant} prefix with delimited arguments and just quits when +there is no match. So, this is acceptable: + +\startbuffer[example] +\macro [1][2] +\macro [1] [2] +\macro [1] +\macro +\stopbuffer + +\typebuffer[example][option=TEX] \getbuffer[example] + +We can check how many arguments have been processed with a dedicated conditional: + +\startbuffer[definition] +\tolerant\def\macro[#1]#*[#2]% + {\ifarguments 0\or 1\or 2\or ?\fi: \vl\type{#1}\vl\type{#2}\vl} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +We use this test: + +\startbuffer[example] +\macro [1][2] \macro [1] [2] \macro [1] \macro +\stopbuffer + +\typebuffer[example][option=TEX] + +The result is: \inlinebuffer[example]\ which is what we expect because we flush +inline and there is no change of mode. When the following definition is used in +display mode, the leading \type {n=} can for instance start a new paragraph and +when code in \type {\everypar} you can loose the right number when macros get +expanded before the \type {n} gets injected. + +\starttyping[option=TEX] +\tolerant\def\macro[#1]#*[#2]% + {n=\ifarguments 0\or 1\or 2\or ?\fi: \vl\type{#1}\vl\type{#2}\vl} +\stoptyping + +In addition to the \type {\ifarguments} test primitive there is also a related +internal counter \type {\lastarguments} set that you can consult, so the \type +{\ifarguments} is actually just a shortcut for \typ {\ifcase \lastarguments}. + +We now continue with the argument specifiers and the next two relate to this optional +grabbing. Consider the next definition: + +\startbuffer[definition] +\tolerant\def\macro#1#*#2% + {\dontleavehmode\hbox spread 1em{\vl\type{#1}\vl\type{#2}\vl\hss}} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +With this test: + +\startbuffer[example] +\macro {1} {2} +\macro {1} +\macro +\stopbuffer + +\typebuffer[example][option=TEX] + +We get: + +\getbuffer[example] + +This is okay because the last \type {\macro} is a valid (single token) argument. But, we +can make the braces mandate: + +\startbuffer[definition] +\tolerant\def\macro#=#*#=% + {\dontleavehmode\hbox spread 1em{\vl\type{#1}\vl\type{#2}\vl\hss}} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +Here the \type {#=} forces a check for braces, so: + +\startbuffer[example] +\macro {1} {2} +\macro {1} +\macro +\stopbuffer + +\typebuffer[example][option=TEX] + +gives this: + +\getbuffer[example] + +However, we do loose these braces and sometimes you don't want that. Of course when you pass the +results downstream to another macro you can always add them, but it was cheap to add a related +specifier: + +\startbuffer[definition] +\tolerant\def\macro#_#*#_% + {\dontleavehmode\hbox spread 1em{\vl\type{#1}\vl\type{#2}\vl\hss}} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +Again, the magic \type {\tolerant} prefix works will quit scanning when there is +no match. So: + +\startbuffer[example] +\macro {1} {2} +\macro {1} +\macro +\stopbuffer + +\typebuffer[example][option=TEX] + +leads to: + +\getbuffer[example] + +When you're tolerant it can be that you still want to pick up some argument +later on. This is why we have a continuation option. + +\startbuffer[definition] +\tolerant\def\foo [#1]#*[#2]#:#3{!#1!#2!#3!} +\tolerant\def\oof[#1]#*[#2]#:(#3)#:#4{!#1!#2!#3!#4!} +\tolerant\def\ofo [#1]#:(#2)#:#3{!#1!#2!#3!} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +Hopefully the next example demonstrates how it works: + +\startbuffer[example] +\foo{3} \foo[1]{3} \foo[1][2]{3} +\oof{4} \oof[1]{4} \oof[1][2]{4} +\oof[1][2](3){4} \oof[1](3){4} \oof(3){4} +\ofo{3} \ofo[1]{3} +\ofo[1](2){3} \ofo(2){3} +\stopbuffer + +\typebuffer[example][option=TEX] + +As you can see we can have multiple continuations using the \type {#:} directive: + +\startlines \getbuffer[example] \stoplines + +The last specifier doesn't work well with the \type {\ifarguments} state because +we no longer know what arguments were skipped. This is why we have another test +for arguments. A zero value means that the next token is not a parameter +reference, a value of one means that a parameter has been set and a value of two +signals an empty parameter. So, it reports the state of the given parameter as +a kind if \type {\ifcase}. + +\startbuffer[definition] +\def\foo#1#2{ [\ifparameter#1\or(ONE)\fi\ifparameter#2\or(TWO)\fi] } +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +\startbuffer[example] +\foo{1}{2} \foo{1}{} \foo{}{2} \foo{}{} +\stopbuffer + +Of course the test has to be followed by a valid parameter specifier: + +\typebuffer[example][option=TEX] + +The previous code gives this: + +\getbuffer[example] + +A combination check \type {\ifparameters}, again a case, matches the first +parameter that has a value set. + +We could add plenty of specifiers but we need to keep in ind that we're not +talking of an expression scanner. We need to keep performance in mind, so nesting +and backtracking are no option. We also have a limited set of useable single +characters, but here's one that uses a symbol that we had left: + +\startbuffer[definition] +\def\startfoo[#/]#/\stopfoo{ [#1](#2) } +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +\startbuffer[example] +\startfoo [x ] x \stopfoo +\startfoo [ x ] x \stopfoo +\startfoo [ x] x \stopfoo +\startfoo [ x] \par x \par \par \stopfoo +\stopbuffer + +The slash directive removes leading and trailing so called spacers as well as tokens +that represent a paragraph end: + +\typebuffer[example][option=TEX] + +So we get this: + +\getbuffer[example] + +The next directive, the quitter \type {#;}, is demonstrated with an example. When +no match has occurred, scanning picks up after this signal, otherwise we just +quit. + +\startbuffer[example] +\tolerant\def\foo[#1]#;(#2){/#1/#2/} + +\foo[1]\quad\foo[2]\quad\foo[3]\par +\foo(1)\quad\foo(2)\quad\foo(3)\par + +\tolerant\def\foo[#1]#;#={/#1/#2/} + +\foo[1]\quad\foo[2]\quad\foo[3]\par +\foo{1}\quad\foo{2}\quad\foo{3}\par + +\tolerant\def\foo[#1]#;#2{/#1/#2/} + +\foo[1]\quad\foo[2]\quad\foo[3]\par +\foo{1}\quad\foo{2}\quad\foo{3}\par + +\tolerant\def\foo[#1]#;(#2)#;#={/#1/#2/#3/} + +\foo[1]\quad\foo[2]\quad\foo[3]\par +\foo(1)\quad\foo(2)\quad\foo(3)\par +\foo{1}\quad\foo{2}\quad\foo{3}\par +\stopbuffer + +\typebuffer[example][option=TEX] \startpacked \getbuffer[example] \stoppacked + +I have to admit that I don't really need it but it made some macros that I was +redefining behave better, so there is some self|-|interest here. Anyway, I +considered some other features, like picking up a detokenized argument but I +don't expect that to be of much use. In the meantime we ran out of reasonable +characters, but some day \type {#?} and \type {#!} might show up, or maybe I find +a use for \type {#<} and \type {#>}. A summary of all this is given here: + +\starttabulate[|T|i2l|] +\FL +\NC + \NC keep the braces \NC \NR +\NC - \NC discard and don't count the argument \NC \NR +\NC / \NC remove leading an trailing spaces and pars \NC \NR +\NC = \NC braces are mandate \NC \NR +\NC _ \NC braces are mandate and kept \NC \NR +\NC ^ \NC keep leading spaces \NC \NR +\ML +\NC 1-9 \NC an argument \NC \NR +\NC 0 \NC discard but count the argument \NC \NR +\ML +\NC * \NC ignore spaces \NC \NR +\NC : \NC pick up scanning here \NC \NR +\NC ; \NC quit scanning \NC \NR +\LL +\stoptabulate + +\stopsection + +\startsection[title=Runaway arguments] + +There is a particular troublesome case left: a runaway argument. The solution is +not pretty but it's the only way: we need to tell the parser that it can quit. + +\startbuffer[definition] +\tolerant\def\foo[#1=#2]% + {\ifarguments 0\or 1\or 2\or 3\or 4\fi:\vl\type{#1}\vl\type{#2}\vl} +\stopbuffer + +\typebuffer[definition][option=TEX] \getbuffer[definition] + +\startbuffer[example] +\dontleavehmode \foo[a=1] +\dontleavehmode \foo[b=] +\dontleavehmode \foo[=] +\dontleavehmode \foo[x]\ignorearguments +\stopbuffer + +The outcome demonstrates that one still has to do some additional checking for sane +results and there are alternative way to (ab)use this mechanism. It all boils down +to a clever combination of delimiters and \type {\ignorearguments}. + +\typebuffer[example][option=TEX] + +All calls are accepted: + +\startlines \getbuffer[example] \stoplines + +Just in case you wonder about performance: don't expect miracles here. On the one +hand there is some extra overhead in the engine (when defining macros as well as +when collecting arguments during a macro call) and maybe using these new features +can sort of compensate that. As mentioned: the gain is mostly in cleaner macro +code and less clutter in tracing. And I just want the \CONTEXT\ code to look +nice: that way users can look in the source to see what happens and not drown in +all these show|-|off tricks, special characters like underscores, at signs, +question marks and exclamation marks. + +For the record: I normally run tests to see if there are performance side effects +and as long as processing the test suite that has thousands of files of all kind +doesn't take more time it's okay. Actually, there is a little gain in \CONTEXT\ +but that is to be expected, but I bet users won't notice it, because it's easily +offset by some inefficient styling. Of course another gain of loosing some +indirectness is that error messages point to the macro that the user called for +and not to some follow up. + +\stopsection + +\startsection[title=Introspection] + +A macro has a meaning. You can serialize that meaning as follows: + +\startbuffer[definition] +\tolerant\protected\def\foo#1[#2]#*[#3]% + {(1=#1) (2=#3) (3=#3)} + +\meaning\foo +\stopbuffer + +\typebuffer[definition][option=TEX] + +The meaning of \type {\foo} comes out as: + +\startnarrower \getbuffer[definition] \stopnarrower + +When you load the module \type {system-tokens} you can also say: + +\startbuffer[example] +\luatokentable\foo +\stopbuffer + +\typebuffer[example][option=TEX] + +This produces a table of tokens specifications: + +{\getbuffer[definition]\getbuffer[example]} + +A token list is a linked list of tokens. The magic numbers in the first column +are the token memory pointers. and because macros (and token lists) get recycled +at some point the available tokens get scattered, which is reflected in the order +of these numbers. Normally macros defined in the macro package are more sequential +because they stay around from the start. The second and third row show the so +called command code and the specifier. The command code groups primitives in +categories, the specifier is an indicator of what specific action will follow, a +register number a reference, etc. Users don't need to know these details. This +macro is a special version of the online variant: + +\starttyping[option=TEX] +\showluatokens\foo +\stoptyping + +That one is always available and shows a similar list on the console. Again, users +normally don't want to know such details. + +\startsection[title=nesting] + +You can nest macros, as in: + +\startbuffer +\def\foo#1#2{\def\oof##1{<#1>##1<#2>}} +\stopbuffer + +\typebuffer[option=TEX] \getbuffer + +At first sight the duplication of \type {#} looks strange but this is what +happens. When \TEX\ scans the definition of \type {\foo} it sees two arguments. +Their specification ends up in the preamble that defines the matching. When the +body is scanned, the \type {#1} and \type {#2} are turned into a parameter +reference. In order to make nested macros with arguments possible a \type {#} +followed by another \type {#} becomes just one \type {#}. Keep in mind that the +definition of \type {\oof} is delayed till the macro \type {\foo} gets expanded. +That definition is just stored and the only thing that get's replaced are the two +references to a macro parameter + +\luatokentable\foo + +Now, when we look at these details, it might become clear why for instance we +have \quote {variable} names like \type {#4} and not \type {#whatever} (with or +without hash). Macros are essentially token lists and token lists can be seen as +a sequence of numbers. This is not that different from other programming +environments. When you run into buzzwords like \quote {bytecode} and \quote +{virtual machines} there is actually nothing special about it: some high level +programming (using whatever concept, and in the case of \TEX\ it's macros) +eventually ends up as a sequence of instructions, say bytecodes. Then you need +some machinery to run over that and act upon those numbers. It's something you +arrive at naturally when you play with interpreting languages. \footnote {I +actually did when I wrote an interpreter for some computer assisted learning +system, think of a kind of interpreted \PASCAL, but later realized that it was a a +bytecode plus virtual machine thing. I'd just applied what I learned when playing +with eight bit processors that took bytes, and interpreted opcodes and such. +There's nothing spectacular about all this and I only realized decades later that +the buzzwords describes old natural concepts.} + +So, internally a \type {#4} is just one token, a operator|-|operand combination +where the operator is \quotation {grab a parameter} and the operand tells +\quotation {where to store} it. Using names is of course an option but then one +has to do more parsing and turn the name into a number \footnote {This is kind of +what \METAPOST\ does with parameters to macros. The side effect is that in +reporting you get \type {text0}, \type {expr2} and such reported which doesn't +make things more clear.}, add additional checking in the macro body, figure out +some way to retain the name for the purpose of reporting (which then uses more +token memory or strings). It is simply not worth the trouble, let alone the fact +that we loose performance, and when \TEX\ showed up those things really mattered. + +It is also important to realize that a \type {#} becomes either a preamble token +(grab an argument) or a reference token (inject the passed tokens into a new +input level). Therefore the duplication of hash tokens \type {##} that you see in +macro nested bodies also makes sense: it makes it possible for the parser to +distinguish between levels. Take: + +\starttyping[option=TEX] +\def\foo#1{\def\oof##1{#1##1#1}} +\stoptyping + +Of course one can think of this: + +\starttyping[option=TEX] +\def\foo#fence{\def\oof#text{#fence#text#fence}} +\stoptyping + +But such names really have to be unique then! Actually \CONTEXT\ does have an +input method that supports such names, but discussing it here is a bit out of +scope. Now, imagine that in the above case we use this: + +\starttyping[option=TEX] +\def\foo[#1][#2]{\def\oof##1{#1##1#2}} +\stoptyping + +If you're a bit familiar with the fact that \TEX\ has a model of category codes +you can imagine that a predictable \quotation {hash followed by a number} is way +more robust than enforcing the user to ensure that catcodes of \quote {names} are +in the right category (read: is a bracket part of the name or not). So, say that +we go completely arbitrary names, we then suddenly needs some escaping, like: + +\starttyping[option=TEX] +\def\foo[#{left}][#{right}]{\def\oof#{text}{#{left}#{text}#{right}}} +\stoptyping + +And, if you ever looked into macro packages, you will notice that they differ in +the way they assign category codes. Asking users to take that into account when +defining macros makes not that much sense. + +So, before one complains about \TEX\ being obscure (the hash thing), think twice. +Your demand for simplicity for your coding demand will make coding more +cumbersome for the complex cases that macro packages have to deal with. It's +comparable using \TEX\ for input or using (say) mark down. For simple documents +the later is fine, but when things become complex, you end up with similar +complexity (or even worse because you lost the enforced detailed structure). So, +just accept the unavoidable: any language has its peculiar properties (and for +sure I do know why I dislike some languages for it). The \TEX\ system is not the +only one where dollars, percent signs, ampersands and hashes have special +meaning. + +\stopsection + +\startsection[title=Prefixes] + +Traditional \TEX\ has three prefixes that can be used with macros: \type {\global}, +\type {\outer} and \type {\long}. The last two are no|-|op's in \LUAMETATEX\ and +if you want to know what they do (did) you can look it up in the \TEX book. The +\ETEX\ extension gave us \type {\protected}. + +In \LUAMETATEX\ we have \type {\global}, \type {\protected}, \type {\tolerant} +and overload related prefixes like \type {\frozen}. A protected macro is one that +doesn't expand in an expandable context, so for instance inside an \type {\edef}. +You can force expansion by using the \type {\expand} primitive in front which is +also something \LUAMETATEX. + +% A protected macro can be made expandable by \typ {\unletprotected} and can be +% protected with \typ {\letprotected}. +% +% \startbuffer[example] +% \def\foo{foo} \edef\oof{oof\foo} 1: \meaning\oof +% \protected\def\foo{foo} \edef\oof{oof\foo} 2: \meaning\oof +% \unletprotected \foo \edef\oof{oof\foo} 3: \meaning\oof +% \stopbuffer +% +% \typebuffer[example][option=TEX] +% +% \startlines \getbuffer[example] \stoplines + +Frozen macros cannot be redefined without some effort. This feature can to some +extent be used to prevent a user from overloading, but it also makes it harder +for the macro package itself to redefine on the fly. You can remove the lock with +\typ {\unletfrozen} and add a lock with \typ {\letfrozen} so in the end users +still have all the freedoms that \TEX\ normally provides. + +\startbuffer[example] + \def\foo{foo} 1: \meaning\foo + \frozen\def\foo{foo} 2: \meaning\foo + \unletfrozen \foo 3: \meaning\foo +\protected\frozen\def\foo{foo} 4: \meaning\foo + \unletfrozen \foo 5: \meaning\foo +\stopbuffer + +\typebuffer[example][option=TEX] + +\startlines \overloadmode0 \getbuffer[example] \stoplines + +This actually only works when you have set \type {\overloadmode} to a value that +permits redefining a frozen macro, so for the purpose of this example we set it +to zero. + +A \type {\tolerant} macro is one that will quit scanning arguments when a +delimiter cannot be matched. We saw examples of that in a previous section. + +These prefixes can be chained (in arbitrary order): + +\starttyping[option=TEX] +\frozen\tolerant\protected\global\def\foo[#1]#*[#2]{...} +\stoptyping + +There is actually an additional prefix, \type {\immediate} but that one is there +as signal for a macro that is defined in and handled by \LUA. This prefix can +then perform the same function as the one in traditional \TEX, where it is used +for backend related tasks like \type {\write}. + +Now, the question is of course, to what extent will \CONTEXT\ use these new +features. One important argument in favor of using \type {\tolerant} is that it +gives (hopefully) better error messages. It also needs less code due to lack of +indirectness. Using \type {\frozen} adds some safeguards although in some places +where \CONTEXT\ itself overloads commands, we need to defrost. Adapting the code +is a tedious process and it can introduce errors due to mistypings, although +these can easily be fixed. So, it will be used but it will take a while to adapt +the code base. + +One problem with frozen macros is that they don't play nice with for instance +\type {\futurelet}. Also, there are places in \CONTEXT\ where we actually do +redefine some core macro that we also want to protect from redefinition by a +user. One can of course \type {\unletfrozen} such a command first but as a bonus +we have a prefix \type {\overloaded} that can be used as prefix. So, one can easily +redefine a frozen macro but it takes a little effort. After all, this feature is +mainly meant to protect a user for side effects of definitions, and not as final +blocker. \footnote {As usual adding features like this takes some experimenting +and we're now at the third variant of the implementation, so we're getting there. +The fact that we can apply such features in large macro package like \CONTEXT\ +helps figuring out the needs and best approaches.} + +A frozen macro can still be overloaded, so what if we want to prevent that? For +this we have the \type {\permanent} prefix. Internally we also create primitives +but we don't have a prefix for that. But we do have one for a very special case +which we demonstrate with an example: + +\startbuffer[example] +\def\FOO % trickery needed to pick up an optional argument + {\noalign{\vskip10pt}} + +\noaligned\protected\tolerant\def\OOF[#1]% + {\noalign{\vskip\iftok{#1}\emptytoks10pt\else#1\fi}} + +\starttabulate[|l|l|] + \NC test \NC test \NC \NR + \NC test \NC test \NC \NR + \FOO + \NC test \NC test \NC \NR + \OOF[30pt] + \NC test \NC test \NC \NR + \OOF + \NC test \NC test \NC \NR +\stoptabulate +\stopbuffer + +\typebuffer[example][option=TEX] + +When \TEX\ scans input (from a file or token list) and starts an alignment, it +will pick up rows. When a row is finished it will look ahead for a \type +{\noalign} and it expands the next token. However, when that token is protected, +the scanner will not see a \type {\noalign} in that macro so it will likely start +complaining when that next macro does get expanded and produces a \type +{\noalign} when a cell is built. The \type {\noaligned} prefix flags a macro as +being one that will do some \type {\noalign} as part of its expansion. This trick +permits clean macros that pick up arguments. Of course it can be done with +traditional means but this whole exercise is about making the code look nice. + +The table comes out as: + +\getbuffer[example] + +One can check the flags with \type {\ifflags} which takes a control sequence and +a number, where valid numbers are: + +\starttabulate[|r|lw(8em)|r|lw(8em)|r|lw(8em)|r|lw(8em)|] +\NC \the\frozenflagcode \NC frozen +\NC \the\permanentflagcode \NC permanent +\NC \the\immutableflagcode \NC immutable +\NC \the\primitiveflagcode \NC primitive \NC \NR +\NC \the\mutableflagcode \NC mutable +\NC \the\noalignedflagcode \NC noaligned +\NC \the\instanceflagcode \NC instance +\NC \NC \NC \NR +\stoptabulate + +The level of checking is controlled with the \type {\overloadmode} but I'm still +not sure about how many levels we need there. A zero value disables checking, +the values 1 and 3 give warnings and the values 2 and 4 trigger an error. + +\stopsection + +\stopdocument + +freezing pitfalls: + +- \futurelet : \overloaded needed +- \let : \overloaded sometimes needed + +primitive protection: + +\newif\iffoo \footrue \foofalse : problem when we make iftrue and iffalse +permanent ... they inherit, so we can't let them, we need a not permanent +alias which is again tricky ... something native? + +immutable : still \count000 but we can consider blocking that, for instance +by \def\count{some error} + +\defcsname +\edefcsname +\letcsname diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-paragraphs.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-paragraphs.tex new file mode 100644 index 00000000000..3098333f013 --- /dev/null +++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-paragraphs.tex @@ -0,0 +1,948 @@ +% language=us + +\environment lowlevel-style + +\startdocument + [title=paragraphs, + color=middlecyan] + +\startsection[title=Introduction] + +This manual is mostly discussing a few low level wrappers around low level \TEX\ +features. Its writing is triggered by an update to the \METAFUN\ and \LUAMETAFUN\ +manuals where we mess a bit with shapes. It gave a good reason to also cover some +more paragraph related topics but it might take a while to complete. Remind me if +you feel that takes too much time. + +Because paragraphs and their construction are rather central to \TEX, you can +imagine that the engine exposes dealing with them. This happens via commands +(primitives) but only when it's robust. Then there are callbacks, and some +provide detailed information about what we're dealing with. However, intercepting +node lists can already be hairy and we do that a lot in \CONTEXT. Intercepting +and tweaking paragraph properties is even more tricky, which is why we try to +avoid that in the core. But \unknown\ in the following sections you will see that +there are actually a couple of mechanism that do so. Often new features like this +are built in stepwise and enabled locally for a while and when they seem okay +they get enabled by default. \footnote {For this we have \type +{\enableexperiments} which one can use in \type {cont-loc.mkxl} or \type +{cont-exp.mkxl}, files that are loaded runtime when on the system. When you use +them, make sure they don't interfere; they are not part of the updates, contrary +to \type {cont-new.mkxl}.} + +\stopsection + +\startsection[title=Paragraphs] + +Before we demonstrate some trickery, let's see what a paragraph is. Normally a +document source is formatted like this: + +\starttyping[option=TEX] +some text (line 1) +some text (line 2) + +some more test (line 1) +some more test (line 2) +\stoptyping + +There are two blocks of text here separated by an empty line and they become two +paragraphs. Unless configured otherwise an empty line is an indication that we +end a paragraph. You can also explicitly do that: + +\starttyping[option=TEX] +some text (line 1) +some text (line 2) +\par +some more test (line 1) +some more test (line 2) +\stoptyping + +When \TEX\ starts a paragraph, it actually also does something think of: + +\starttyping[option=TEX] +[\the\everypar]some text (line 1) some text (line 2) \par +[\the\everypar]some more test (line 1) some more test (line 2) \par +\stoptyping + +or more accurate: + +\starttyping[option=TEX] +[\the\everypar]some text some text \par +[\the\everypar]some more test some more test \par +\stoptyping + +because the end|-|of|-|line character has become a space. As mentioned, +an empty line is actually the end of a paragraph. But in \LUAMETATEX\ +we can cheat a bit. If we have this: + +\startbuffer +line 1 + +line 2 +\stopbuffer + +\typebuffer[option=TEX] + +We can do this (watch how we need to permit overloading a primitive when we have +enabled \type {\overloadmode}): + +\startbuffer +\pushoverloadmode +\def\linepar{\removeunwantedspaces !\ignorespaces} +\popoverloadmode +line 1 + +line 2 +\stopbuffer + +\typebuffer[option=TEX] + +This comes out as: + +\start \getbuffer \stop + +I admit that since it got added (as part of some cleanup halfway the overhaul of +the engine) I never saw a reason to use it, but it is a cheap feature. The \type +{\linepar} primitive is undefined (\type {\undefined}) by default so no user sees +it anyway. Just don't use it unless maybe for some pseudo database trickery (I +considered using it for the database module but it is not needed). In a similar +fashion, just don't redefine \type {\par}: it's asking for troubles and \quote +{not done} in \CONTEXT\ anyway. + +Back to reality. In \LUATEX\ we get a node list that starts with a so called +\type {localpar} node and ends with a \type {\parfillskip}. The first node is +prepended automatically. That list travels through the system: hyphenation, +applying font properties, break the effectively one line into lines, wrap them +and add them to a vertical list, etc. Each stage can be intercepted via +callbacks. + +When the paragraph is broken into lines hanging indentation or a so called par +shape can be applied, and we will see more of that later, here we talk \type +{\par} and show another \LUAMETATEX\ trick: + +\startbuffer +\def\foo{{\bf test:} \ignorepars} + +\foo + +line +\stopbuffer + +\typebuffer[option=TEX] + +The macro typesets some text and then skips to the next paragraph: + +\start \getbuffer \stop + +Think of this primitive as being a more powerful variant of \type +{\ignorespaces}. This leaves one aspect: how do we start a paragraph. Technically +we need to force \TEX\ into so called horizontal mode. When you look at plain +\TEX\ documents you will notice commands like \type {\noindent} and \type +{\indent}. In \CONTEXT\ we have more high level variants, for instance we have +\type {\noindentation}. + +A robust way to make sure that you get in horizontal mode is using \type +{\dontleavehmode} which is a wink to \type {\leavevmode}, a command that you +should never use in \CONTEXT, so when you come from plain or \LATEX, it's one of +the commands you should wipe from your memory. + +When \TEX\ starts with a paragraph the \type {\everypar} token list is expanded +and again this is a primitive you should not mess with yourself unless in very +controlled situations. If you change its content, you're on your own with respect +to interferences and side effects. + +One of the things that \TEX\ does in injecting the indentation. Even when there +is none, it gets added, not as skip but as an empty horizontal box of a certain +width. This is easier on the engine when it constructs the paragraph from the one +liner: starting with a skip demands a bit more testing in the process (a nice +trick so to say). However, in \CONTEXT\ we enable the \LUAMETATEX\ feature that +does use a skip instead of a box. It's part of the normalization that is +discussed later. Instead of checking for a box with property indent, we check for +a skip with such property. This is often easier and cleaner. + +A bit off topic is the fact that in traditional \TEX\ empty lines or \type {\par} +primitives can trigger an error. This has to do with the fact that the program +evolved in a time where paper terminals were used and runtime could be excessive. +So, in order to catch a possible missing brace, a concept of \type {\long} +macros, permitting \type {\par} or equivalents in arguments, was introduced as +well as not permitting them in for instance display math. In \CONTEXT\ \MKII\ +most macros that could be sensitive for this were defined as \type {\long} so +that users never had to bother about it and probably were not even aware of it. +Right from the start in \LUATEX\ these error|-|triggers could be disabled which +of course we enable in \CONTEXT\ and in \LUAMETATEX\ these features have been +removed altogether. I don't think users will complain about this. + +If you want to enforce a newline but not a new paragraph you can use the \type +{\crlf} command. When used on its own it will produce an empty line. Don't use +this to create whitespace between lines. + +If you want to do something after so called par tokens are seen you can do this: + +\startbuffer +\def\foo{{\bf >>>> }} +\expandafterpars\foo + +this is a new paragraph ... + +\expandafterpars\foo +\par\par\par\par +this is a new paragraph ... +\stopbuffer + +\typebuffer[option=TEX] + +This not to be confused with \type {\everypar} which is a token list that \TEX\ +itself injects before each paragraph (also nested ones). + +\getbuffer + +This is typically a primitive that will only be used in macros. You can actually +program it using macros: pickup a token, check and push it back when it's not a +par equivalent token. The primitive is is just nicer (and easier on the log when +tracing is enabled). + +\stopsection + +\startsection[title=Properties] + +A paragraph is just a collection of lines that result from one input line that +got broken. This process of breaking into lines is influenced by quite some +parameters. In traditional \TEX\ and also in \LUAMETATEX\ by default the values +that are in effect when the end of the paragraph is met are used. So, when you +change them in a group and then ends the paragraph after the group, the values +you've set in the group are not used. + +However, in \LUAMETATEX\ we can optionally store them with the paragraph. When +that happens the values current at the start are frozen. You can still overload +them but that has to be done explicitly then. The advantage is that grouping no +longer interferes with the line break algorithm. The magic primitive is \type +{\snapshotpar} which takes a number made from categories mentioned below: + +\starttabulate[|l|l|r|] +\BC variable \BC category \BC code \NC \NR +\NC \type {\hsize} \NC hsize \NC 0x\uchexnumbers\frozenhsizecode \NC \NR +\NC \type {\leftskip} \NC skip \NC 0x\uchexnumbers\frozenskipcode \NC \NR +\NC \type {\rightskip} \NC skip \NC 0x\uchexnumbers\frozenskipcode \NC \NR +\NC \type {\hangindent} \NC hang \NC 0x\uchexnumbers\frozenhangcode \NC \NR +\NC \type {\hangafter} \NC hang \NC 0x\uchexnumbers\frozenhangcode \NC \NR +\NC \type {\parindent} \NC indent \NC 0x\uchexnumbers\frozenindentcode \NC \NR +\NC \type {\parfillleftskip} \NC par fill \NC 0x\uchexnumbers\frozenparfillcode \NC \NR +\NC \type {\parfillrightskip} \NC par fill \NC 0x\uchexnumbers\frozenparfillcode \NC \NR +\NC \type {\adjustspacing} \NC adjust \NC 0x\uchexnumbers\frozenadjustcode \NC \NR +\NC \type {\adjustspacingstep} \NC adjust \NC 0x\uchexnumbers\frozenadjustcode \NC \NR +\NC \type {\adjustspacingshrink} \NC adjust \NC 0x\uchexnumbers\frozenadjustcode \NC \NR +\NC \type {\adjustspacingstretch} \NC adjust \NC 0x\uchexnumbers\frozenadjustcode \NC \NR +\NC \type {\protrudechars} \NC protrude \NC 0x\uchexnumbers\frozenprotrudecode \NC \NR +\NC \type {\pretolerance} \NC tolerance \NC 0x\uchexnumbers\frozentolerancecode \NC \NR +\NC \type {\tolerance} \NC tolerance \NC 0x\uchexnumbers\frozentolerancecode \NC \NR +\NC \type {\emergencystretch} \NC stretch \NC 0x\uchexnumbers\frozenstretchcode \NC \NR +\NC \type {\looseness} \NC looseness \NC 0x\uchexnumbers\frozenloosenesscode \NC \NR +\NC \type {\lastlinefit} \NC last line \NC 0x\uchexnumbers\frozenlastlinecode \NC \NR +\NC \type {\linepenalty} \NC line penalty \NC 0x\uchexnumbers\frozenlinepenaltycode \NC \NR +\NC \type {\interlinepenalty} \NC line penalty \NC 0x\uchexnumbers\frozenlinepenaltycode \NC \NR +\NC \type {\interlinepenalties} \NC line penalty \NC 0x\uchexnumbers\frozenlinepenaltycode \NC \NR +\NC \type {\clubpenalty} \NC club penalty \NC 0x\uchexnumbers\frozenclubpenaltycode \NC \NR +\NC \type {\clubpenalties} \NC club penalty \NC 0x\uchexnumbers\frozenclubpenaltycode \NC \NR +\NC \type {\widowpenalty} \NC widow penalty \NC 0x\uchexnumbers\frozenwidowpenaltycode \NC \NR +\NC \type {\widowpenalties} \NC widow penalty \NC 0x\uchexnumbers\frozenwidowpenaltycode \NC \NR +\NC \type {\displaywidowpenalty} \NC display penalty \NC 0x\uchexnumbers\frozendisplaypenaltycode \NC \NR +\NC \type {\displaywidowpenalties} \NC display penalty \NC 0x\uchexnumbers\frozendisplaypenaltycode \NC \NR +\NC \type {\brokenpenalty} \NC broken penalty \NC 0x\uchexnumbers\frozenbrokenpenaltycode \NC \NR +\NC \type {\adjdemerits} \NC demerits \NC 0x\uchexnumbers\frozendemeritscode \NC \NR +\NC \type {\doublehyphendemerits} \NC demerits \NC 0x\uchexnumbers\frozendemeritscode \NC \NR +\NC \type {\finalhyphendemerits} \NC demerits \NC 0x\uchexnumbers\frozendemeritscode \NC \NR +\NC \type {\parshape} \NC shape \NC 0x\uchexnumbers\frozenshapecode \NC \NR +\NC \type {\baselineskip} \NC line \NC 0x\uchexnumbers\frozenlinecode \NC \NR +\NC \type {\lineskip} \NC line \NC 0x\uchexnumbers\frozenlinecode \NC \NR +\NC \type {\lineskiplimit} \NC line \NC 0x\uchexnumbers\frozenlinecode \NC \NR +\NC \type {\hyphenationmode} \NC hyphenation \NC 0x\uchexnumbers\frozenhyphenationcode \NC \NR +\stoptabulate + + +As you can see here, there are more paragraph related parameters than in for +instance \PDFTEX\ and \LUATEX\ and these are (to be) explained in the +\LUAMETATEX\ manual. You can imagine that keeping this around with the paragraph +adds some extra overhead to the machinery but most users won't notice that +because is is compensated by gains elsewhere. + +This is pretty low level and there are a bunch of helpers that support this but +these are not really user level macros. As with everything \TEX\ you can mess +around as much as you like, and the code gives plenty of examples but when you do +this, you're on your own because it can interfere with \CONTEXT\ core +functionality. + +In \LMTX\ taking these snapshots is turned on by default and because it thereby +fundamentally influences the par builder, users can run into compatibility issues +but in practice there has been no complaints (and this feature has been in use +quite a while before this document was written). One reason for users not +noticing is that one of the big benefits is probably handled by tricks mentioned +on the mailing list. Imagine that you have this: + +\starttyping[option=TEX] +{\bf watch out:} here is some text +\stoptyping + +In this small example the result will be as expected. But what if something magic +with the start of a paragraph is done? Like this: + +\starttyping[option=TEX] +\placefigure[left]{A cow!}{\externalfigure[cow.pdf]} + +{\bf watch out:} here is some text ... of course much more is needed to + get a flow around the figure! +\stoptyping + +The figure will hang at the left side of the paragraph but it is put there when +the text starts and that happens inside the bold group. It means that the +properties we set in order to get the shape around the figure are lost as soon as +we're at \quote{\type {here is some text}} and definitely is wrong when the +paragraph ends and the par builder has to use them to get the shape right. We get +text overlapping the figure. A trick to overcome this is: + +\starttyping[option=TEX] +\dontleavehmode {\bf watch out:} here is some text ... of course much + more is needed to get a flow around the figure! +\stoptyping + +where the first macro makes sure we already start a paragraph before the group is +entered (using a \type {\strut} also works). It's not nice and I bet users have +been bitten by this and by now know the tricks. But, with snapshots such fuzzy +hacks are not needed any more! The same is true with this: + +\starttyping[option=TEX] +{\leftskip 1em some text \par} +\stoptyping + +where we had to explicitly end the paragraph inside the group in order to retain +the skip. I suppose that users normally use the high level environments so they +never had to worry about this. It's also why users probably won't notice that +this new mechanism has been active for a while. Actually, when you now change a +parameter inside the paragraph its new value will not be applied (unless you +prefix it with \type {\frozen} or snapshot it) but no one did that anyway. + +\stopsection + +\startsection[title=Wrapping up] + +In \CONTEXT\ \LMTX\ we have a mechanism to exercise macros (or content) before a +paragraph ends. This is implemented using the \type {\wrapuppar} primitive. The +to be wrapped up material is bound to the current paragraph which in order to +get this done has to be started when this primitive is used. + +Although the high level interface has been around for a while it still needs a +bit more testing (read: use cases are needed). In the few cases where we already +use it application can be different because again it relates to snapshots. This +because in the past we had to use tricks that also influenced the user interface +of some macros (which made them less natural as one would expect). So the +question is: where do we apply it in old mechanisms and where not. + +{\em todo: accumulation, interference, where applied, limitations} + +% \vbox {vbox : \wrapuppar{1}test\par x\wrapuppar{2}test}\blank +% \vtop {vtop : \wrapuppar{1}test\par x\wrapuppar{2}test}\blank +% \vcenter{vcenter : \wrapuppar{1}test\par x\wrapuppar{2}test}\blank +% $$x = \vcenter{vcenter : \wrapuppar{1}test\par x\wrapuppar{2}test}$$\blank +% x\vadjust{vadjust : \wrapuppar{1}test\par x\wrapuppar{2}test}x\blank + +\stopsection + +\startsection[title=Hanging] + +There are two mechanisms for getting a specific paragraph shape: rectangular +hanging and arbitrary shapes. Both mechanisms work top|-|down. The first +mechanism uses a combination of \type {\hangafter} and \type {\hangindent}, and +the second one depends on \type {\parshape}. In this section we discuss the +rectangular one. + +\startbuffer[demo-5] +\hangafter 4 \hangindent 4cm \samplefile{tufte} \page +\hangafter -4 \hangindent 4cm \samplefile{tufte} \page +\hangafter 4 \hangindent -4cm \samplefile{tufte} \page +\hangafter -4 \hangindent -4cm \samplefile{tufte} \page +\stopbuffer + +\typebuffer[demo-5][option=TEX] + +As you can see in \in {figure} [fig:hang], the four cases are driven by the sign +of the values. If you want to hang into the margin you need to use different +tricks, like messing with the \type {\leftskip}, \type {\rightskip} or \type +{\parindent} parameters (which then of course can interfere with other mechanisms +uses at the same time). + +\startplacefigure[title=Hanging indentation,reference=fig:hang] +\startcombination[nx=2,ny=2] + {\typesetbuffer[demo-5][page=1,width=.4\textwidth,frame=on]} {\type{\hangafter +4 \hangindent +4cm}} + {\typesetbuffer[demo-5][page=2,width=.4\textwidth,frame=on]} {\type{\hangafter -4 \hangindent +4cm}} + {\typesetbuffer[demo-5][page=3,width=.4\textwidth,frame=on]} {\type{\hangafter +4 \hangindent -4cm}} + {\typesetbuffer[demo-5][page=4,width=.4\textwidth,frame=on]} {\type{\hangafter -4 \hangindent -4cm}} +\stopcombination +\stopplacefigure + +\stopsection + +\startsection[title=Shapes] + +In \CONTEXT\ we don't use \type {\parshape} a lot. It is used in for instance +side floats but even there not in all cases. It's more meant for special +applications. This means that in \MKII\ and \MKIV\ we don't have some high level +interface. However, when \METAFUN\ got upgraded to \LUAMETAFUN, and the manual +also needed an update, one of the examples in that manual that used shapes also +got done differently (read: nicer). And that triggered the arrival of a new low +level shape mechanism. + +One important property of the \type {\parshape} mechanism is that it works per +paragraph. You define a shape in terms of a left margin and width of a line. The +shape has a fixed number of such pairs and when there is more content, the last +one is used for the rest of the lines. When the paragraph is finished, the shape +is forgotten. \footnote {Not discussed here is a variant that might end up in +\LUAMETATEX\ that works with the progression, i.e.\ takes the height of the +content so far into account. This is somewhat tricky because for that to work +vertical skips need to be frozen, which is no real big deal but has to be done +careful in the code.} + +The high level interface is a follow up on the example in the \METAFUN\ manual and +uses shapes that carry over to the next paragraph. In addition we can cycle over +a shape. In this interface shapes are defined using keyword. Here are some +examples: + +\starttyping[option=TEX] +\startparagraphshape[test] + left 1mm right 1mm + left 5mm right 5mm +\stopparagraphshape +\stoptyping + +This shape has only two entries so the first line will have a 1mm margin while +later lines will get 5mm margins. This translates into a \type {\parshape} like: + +\starttyping[option=TEX] +\parshape 2 + 1mm \dimexpr\hsize-1mm\relax + 5mm \dimexpr\hsize-5mm\relax +\stoptyping + +Watch the number \type {2}: it tells how many specification lines follow. As you +see, we need to calculate the width. + +\starttyping[option=TEX] +\startparagraphshape[test] + left 1mm right 1mm + left 5mm right 5mm + repeat +\stopparagraphshape +\stoptyping + +This variant will alternate between 1mm and 5mm margins. The repeating feature is +translated as follows. Maybe at some point I will introduce a few more options. + +\starttyping[option=TEX] +\parshape 2 options 1 + 1mm \dimexpr\hsize-1mm\relax + 5mm \dimexpr\hsize-5mm\relax +\stoptyping + +A shape can have some repetition, and we can save keystrokes by copying the last +entry. The resulting \type {\parshape} becomes rather long. + +\starttyping[option=TEX] +\startparagraphshape[test] + left 1mm right 1mm + left 2mm right 2mm + left 3mm right 3mm + copy 8 + left 4mm right 4mm + left 5mm right 5mm + left 5mm hsize 10cm +\stopparagraphshape +\stoptyping + +Also watch the \type {hsize} keyword: we don't calculate the hsize from the \type +{left} and \type {right} values but explicitly set it. + +\starttyping[option=TEX] +\startparagraphshape[test] + left 1mm right 1mm + right 3mm + left 5mm right 5mm + repeat +\stopparagraphshape +\stoptyping + +When a \type {right} keywords comes first the \type {left} is assumed to be zero. +In the examples that follow we will use a couple of definitions: + +\startbuffer[setup-0] +\startparagraphshape[test] + both 1mm both 2mm both 3mm both 4mm both 5mm both 6mm + both 7mm both 6mm both 5mm both 4mm both 3mm both 2mm +\stopparagraphshape +\stopbuffer + +\startbuffer[setup-0-repeat] +\startparagraphshape[test-repeat] + both 1mm both 2mm both 3mm both 4mm both 5mm both 6mm + both 7mm both 6mm both 5mm both 4mm both 3mm both 2mm + repeat +\stopparagraphshape +\stopbuffer + +\typebuffer[setup-0,setup-0-repeat][option=TEX] + +The last one could also be defines as: + +\starttyping[option=TEX] +\startparagraphshape[test-repeat] + \rawparagraphshape{test} repeat +\stopparagraphshape +\stoptyping + +In the previous code we already introduced the \type {repeat} option. This will +make the shape repeat at the engine level when the shape runs out of specified +lines. In the application of a shape definition we can specify a \type {method} +to be used and that determine if the next paragraph will start where we left off +and discard afterwards (\type {shift}) or that we move the discarded lines up +front so that we never run out of lines (\type {cycle}). It sounds complicated +but just keep in mind that \type {repeat} is part of the \type {\parshape} and +act within a paragraph while \type {shift} and \type {cycle} are applied when a +new paragraph is started. + +\startbuffer[demo-1] +\startshapedparagraph[list=test] + \dorecurse{8}{\showparagraphshape\samplefile{tufte} \par} +\stopshapedparagraph +\stopbuffer + +\startbuffer[demo-1-repeat] +\startshapedparagraph[list=test-repeat] + \dorecurse{8}{\showparagraphshape\samplefile{tufte} \par} +\stopshapedparagraph +\stopbuffer + +In \in {figure} [fig:shape:discard] you see the following applied: + +\typebuffer[demo-1,demo-1-repeat][option=TEX] + +\startplacefigure[title=Discarded shaping,reference=fig:shape:discard] +\startcombination[nx=2,ny=2] + {\typesetbuffer[setup-0,demo-1] [page=1,width=.4\textwidth,frame=on]} {discard, finite shape, page 1} + {\typesetbuffer[setup-0,demo-1] [page=2,width=.4\textwidth,frame=on]} {discard, finite shape, page 2} + {\typesetbuffer[setup-0,demo-1-repeat][page=1,width=.4\textwidth,frame=on]} {discard, repeat in shape, page 1} + {\typesetbuffer[setup-0,demo-1-repeat][page=2,width=.4\textwidth,frame=on]} {discard, repeat in shape, page 2} +\stopcombination +\stopplacefigure + +In \in {figure} [fig:shape:shift] we use this instead: + +\startbuffer[demo-2] +\startshapedparagraph[list=test,method=shift] + \dorecurse{8}{\showparagraphshape\samplefile{tufte} \par} +\stopshapedparagraph +\stopbuffer + +\startbuffer[demo-2-shift] +\startshapedparagraph[list=test-repeat,method=shift] + \dorecurse{8}{\showparagraphshape\samplefile{tufte} \par} +\stopshapedparagraph +\stopbuffer + +\typebuffer[demo-2,demo-2-repeat][option=TEX] + +\startplacefigure[title=Shifted shaping,reference=fig:shape:shift] +\startcombination[nx=2,ny=2] + {\typesetbuffer[setup-0,demo-2][page=1,width=.4\textwidth,frame=on]} {shift, finite shape, page 1} + {\typesetbuffer[setup-0,demo-2][page=2,width=.4\textwidth,frame=on]} {shift, finite shape, page 2} + {\typesetbuffer[setup-0-repeat,demo-2-shift][page=1,width=.4\textwidth,frame=on]} {shift, repeat in shape, page 1} + {\typesetbuffer[setup-0-repeat,demo-2-shift][page=2,width=.4\textwidth,frame=on]} {shift, repeat in shape, page 2} +\stopcombination +\stopplacefigure + +Finally, in \in {figure} [fig:shape:cycle] we use: + +\startbuffer[demo-3] +\startshapedparagraph[list=test,method=cycle] + \dorecurse{8}{\showparagraphshape\samplefile{tufte} \par} +\stopshapedparagraph +\stopbuffer + +\startbuffer[demo-3-cycle] +\startshapedparagraph[list=test-repeat,method=cycle] + \dorecurse{8}{\showparagraphshape\samplefile{tufte} \par} +\stopshapedparagraph +\stopbuffer + +\typebuffer[demo-3,demo-3-repeat][option=TEX] + +\startplacefigure[title=Cycled shaping,reference=fig:shape:cycle] +\startcombination[nx=2,ny=2] + {\typesetbuffer[setup-0,demo-3][page=1,width=.4\textwidth,frame=on]} {cycle, finite shape, page 1} + {\typesetbuffer[setup-0,demo-3][page=2,width=.4\textwidth,frame=on]} {cycle, finite shape, page 2} + {\typesetbuffer[setup-0-repeat,demo-3-cycle][page=1,width=.4\textwidth,frame=on]} {cycle, repeat in shape, page 1} + {\typesetbuffer[setup-0-repeat,demo-3-cycle][page=2,width=.4\textwidth,frame=on]} {cycle, repeat in shape, page 2} +\stopcombination +\stopplacefigure + +These examples are probably too small to see the details but you can run them +yourself or zoom in on the details. In the margin we show the values used. Here +is a simple example of (non) poetry. There are other environments that can be +used instead but this makes a good example anyway. + +\startbuffer +\startparagraphshape[test] + left 0em right 0em + left 1em right 0em + repeat +\stopparagraphshape + +\startshapedparagraph[list=test,method=cycle] + verse line 1.1\crlf verse line 2.1\crlf + verse line 3.1\crlf verse line 4.1\par + verse line 1.2\crlf verse line 2.2\crlf + verse line 3.2\crlf verse line 4.2\crlf + verse line 5.2\crlf verse line 6.2\par +\stopshapedparagraph +\stopbuffer + +\typebuffer[option=TEX] + +\start \getbuffer \stop + +Because the idea for this feature originates in \METAFUN, we will now kick in +some \METAPOST. The following code creates a shape for a circle. We use a +2mm offset here: + +\startbuffer +\startuseMPgraphic{circle} + path p ; p := fullcircle scaled TextWidth ; + build_parshape(p, + 2mm, 0, 0, + LineHeight, StrutHeight, StrutDepth, StrutHeight + ) ; +\stopuseMPgraphic +\stopbuffer + +\typebuffer[option=TEX] + +\start \getbuffer \stop + +We plug this into the already described macros: + +\startbuffer +\startshapedparagraph[mp=circle]% + \setupalign[verytolerant,stretch,last]% + \samplefile{tufte} + \samplefile{tufte} +\stopshapedparagraph +\stopbuffer + +\typebuffer[option=TEX] + +And get ourself a circular shape. Watch out, at this moment the shape environment +does not add grouping so when as in this case you change the alignment it can +influence the document. + +\start \getbuffer \stop + +\startbuffer[framed] +\framed[align=normal,width=\textwidth,offset=2mm,strut=no]\bgroup + \getbuffer +\egroup +\stopbuffer + +Assuming that the shape definition above is in a buffer we can do this: + +\typebuffer[option=TEX] + +The result is shown in \in {figure} [fig:shape:circle]. Because all action +happens in the framed environment, we can also use this definition: + +\starttyping[option=TEX] +\startuseMPgraphic{circle} + path p ; p := fullcircle scaled \the\dimexpr\framedwidth+\framedoffset*2\relax ; + build_parshape(p, + \framedoffset, 0, 0, + LineHeight, StrutHeight, StrutDepth, StrutHeight + ) ; + draw p ; +\stopuseMPgraphic +\stoptyping + +\startplacefigure[title=A framed circular shape,reference=fig:shape:circle] + \getbuffer[framed] +\stopplacefigure + +A mechanism like this is often never completely automatic in the sense that you +need to keep an eye on the results. Depending on user demands more features can +be added. With weird shapes you might want to set up the alignment to be \type +{tolerant} and have some \type {stretch}. + +The interface described in the \METAFUN\ manual is pretty old, the time stamp of +the original code is mid 2000, but the principles didn't change. The examples in +\type {meta-imp-txt.mkxl} can now be written as: + +\startuseMPgraphic{test 1} + begingroup ; + save p ; path p ; p := fullcircle scaled 6cm ; + lmt_parshape [ + path = p, + offset = BodyFontSize/2, + dx = 0, % default + dy = 0, % default + lineheight = LineHeight, % default + strutheight = StrutHeight, % default + strutdepth = StrutDepth, % default + topskip = StrutHeight, % default + ] ; + draw p withpen pencircle scaled 1pt ; + endgroup ; +\stopuseMPgraphic + +\startuseMPgraphic{test 2} + begingroup ; + save p ; path p ; p := fullsquare rotated 45 scaled 5cm ; + lmt_parshape [ + path = p, + offset = BodyFontSize/2, + trace = true, + ] ; + draw p withpen pencircle scaled 1pt ; + endgroup ; +\stopuseMPgraphic + +\startuseMPgraphic{test 3} + begingroup ; + save w, h, p ; path p ; w := h := 6cm ; + p := (.5w,h) -- ( 0, h) -- (0,0) -- (w,0) & + ( w,0) .. (.75w,.5h) .. (w,h) & (w,h) -- cycle ; + lmt_parshape [ + path = p, + offset = BodyFontSize/2, + ] ; + draw p withpen pencircle scaled 1pt ; + endgroup ; +\stopuseMPgraphic + +\startuseMPgraphic{test 4} + begingroup ; + save d, p, q ; path p, q ; d := BodyFontSize/2; + vardef shape(expr w, h, o) = + (o,o) -- (w-o,o) & (w-o,o) .. (.75w-o,.5h) .. + (w-2o,h-o) & (w-2o,h-o) -- (o,h-o) -- cycle + enddef ; + p := shape(6cm, 6cm, d) ; q := shape(6cm, 6cm, 0) ; + lmt_parshape [ + path = p, + offsetpath = q, + dx = d, + dy = d, + trace = true, + ] ; + draw q withpen pencircle scaled 1pt ; + endgroup ; +\stopuseMPgraphic + +\defineoverlay[test 1][\useMPgraphic{test 1}] +\defineoverlay[test 2][\useMPgraphic{test 2}] +\defineoverlay[test 3][\useMPgraphic{test 3}] +\defineoverlay[test 4][\useMPgraphic{test 4}] + +\startbuffer + \startshapetext[test 1,test 2,test 3,test 4] + \setupalign[verytolerant,stretch,normal]% + \samplefile{douglas} % Douglas R. Hofstadter + \stopshapetext + \startcombination[2*2] + {\framed[offset=overlay,frame=off,background=test 1]{\getshapetext}} + {test 1} + {\framed[offset=overlay,frame=off,background=test 2]{\getshapetext}} + {test 2} + {\framed[offset=overlay,frame=off,background=test 3]{\getshapetext}} + {test 3} + {\framed[offset=overlay,frame=off,background=test 4]{\getshapetext}} + {test 4} + \stopcombination +\stopbuffer + +\typebuffer[option=TEX] + +In \in {figure} [fig:shapes:chain] we see the result. Watch how for two shapes +we have enabled tracing. Of course you need to tweak till all fits well but we're +talking of special situations anyway. + +\startplacefigure[Title=Multiple shapes,reference=fig:shapes:chain] + \getbuffer +\stopplacefigure + +Here is a bit more extreme example. Again we use a circle: + +\startbuffer +\startuseMPgraphic{circle} + lmt_parshape [ + path = fullcircle scaled 136mm, + offset = 2mm, + bottomskip = - 1.5LineHeight, + ] ; +\stopuseMPgraphic +\stopbuffer + +\typebuffer[option=TEX] + +But we output a longer text: + +\startbuffer +\startshapedparagraph[mp=circle,repeat=yes,method=cycle]% + \setupalign[verytolerant,stretch,last]\dontcomplain + {\darkred \samplefile{tufte}}\par + {\darkgreen \samplefile{tufte}}\par + {\darkblue \samplefile{tufte}}\par + {\darkcyan \samplefile{tufte}}\par + {\darkmagenta \samplefile{tufte}}\par +\stopshapedparagraph +\stopbuffer + +\typebuffer[option=TEX] + +We get a multi|-|page shape: + +\start \getbuffer \stop + +Compare this with: + +\startbuffer +\startshapedparagraph[mp=circle,repeat=yes,method=cycle]% + \setupalign[verytolerant,stretch,last]\dontcomplain + {\darkred \samplefile{tufte}} + {\darkgreen \samplefile{tufte}} + {\darkblue \samplefile{tufte}} + {\darkcyan \samplefile{tufte}} + {\darkmagenta \samplefile{tufte}} +\stopshapedparagraph +\stopbuffer + +\typebuffer[option=TEX] + +Which gives: + +\start \getbuffer \stop + +Here the \type {bottomskip} takes care of subtle rounding issues as well as +discarding the last line in the shape so that we get nicer continuation. There is +no full automated solution for all you can come up with. + +Mixing a \METAPOST\ specification into a regular one is also possible. The next +example demonstrates this as well as the option to remove some lines from a +specification: + +\starttyping[option=TEX] +\startparagraphshape[test] + left 0em right 0em + left 1em right 0em + metapost {circle} + delete 3 + metapost {circle,circle,circle} + delete 7 + metapost {circle} + repeat +\stopparagraphshape +\stoptyping + +You can combine a shape with narrowing a paragraph. Watch the \type {absolute} +keyword in the next code. The result is shown in \in {figure} [fig:shape:skips]. + +\startbuffer[demo-4] +\startuseMPgraphic{circle} + lmt_parshape [ + path = fullcircle scaled TextWidth, + bottomskip = - 1.5LineHeight, + ] ; +\stopuseMPgraphic + +\startparagraphshape[test-1] + metapost {circle} repeat +\stopparagraphshape + +\startparagraphshape[test-2] + absolute left metapost {circle} repeat +\stopparagraphshape + +\startparagraphshape[test-3] + absolute right metapost {circle} repeat +\stopparagraphshape + +\startparagraphshape[test-4] + absolute both metapost {circle} repeat +\stopparagraphshape + +\showframe + +\startnarrower[4*left,2*right] + \startshapedparagraph[list=test-1,repeat=yes,method=repeat]% + \setupalign[verytolerant,stretch,last]\dontcomplain + \dorecurse{3}{\samplefile{thuan}} + \stopshapedparagraph + \page + \startshapedparagraph[list=test-2,repeat=yes,method=repeat]% + \setupalign[verytolerant,stretch,last]\dontcomplain + \dorecurse{3}{\samplefile{thuan}} + \stopshapedparagraph + \page + \startshapedparagraph[list=test-3,repeat=yes,method=repeat]% + \setupalign[verytolerant,stretch,last]\dontcomplain + \dorecurse{3}{\samplefile{thuan}} + \stopshapedparagraph + \page + \startshapedparagraph[list=test-4,repeat=yes,method=repeat]% + \setupalign[verytolerant,stretch,last]\dontcomplain + \dorecurse{3}{\samplefile{thuan}} + \stopshapedparagraph +\stopnarrower +\stopbuffer + +\typebuffer[demo-4][option=TEX] + +\startplacefigure[title=Skip compensation,reference=fig:shape:skips] +\startcombination[nx=2,ny=2] + {\typesetbuffer[demo-4][page=1,width=.4\textwidth,frame=on]} {test 1} + {\typesetbuffer[demo-4][page=2,width=.4\textwidth,frame=on]} {test 2, left} + {\typesetbuffer[demo-4][page=3,width=.4\textwidth,frame=on]} {test 3, right} + {\typesetbuffer[demo-4][page=4,width=.4\textwidth,frame=on]} {test 4, both} +\stopcombination +\stopplacefigure + +\stopsection + +% \startsection[title=Linebreaks] +\startsection[title=Modes] + +% \ruledvbox{1\ifhmode\writestatus{!}{HMODE 1}\fi} % hsize +% \ruledvbox{\hbox{\strut 2}\ifhmode\writestatus{!}{HMODE 2}\fi} % fit +% \ruledvbox{\hbox{\strut 3}\hbox{\strut 3}\ifhmode\writestatus{!}{HMODE 3}\fi} % fit +% \ruledvbox{\hbox{\strut 4}4\ifhmode\writestatus{!}{HMODE 4}\fi} % hsize +% \ruledvbox{\hbox{\strut 5}5\hbox{\strut 5}\ifhmode\writestatus{!}{HMODE 5}\fi} % hsize +% \ruledvbox{6\hbox{\strut 6}\ifhmode\writestatus{!}{HMODE 6}\fi} % hsize + +{\em todo: some of the side effects of so called modes} + +\stopsection + +\startsection[title=Normalization] + +{\em todo: users don't need to bother about this but it might be interesting anyway} + +\stopsection + +\stopdocument + +\everyhbox \everyvbox : useless unless one resets +\parattr +\snapshotpar +\wrapuppar + +% \parfillleftskip + +% I rewarded myself after writing a section by watching the video "Final Thing On +% My Mind", The Pineapple This, Live, 2020, the usual perfect GH performance, +% wondering if live would turn to normal so that we could go to such concerts once +% again given successive covids. Writing manuals can do with a distraction. +% +% Gavin Harrison: Soundcheck, Drummerworld Jan 27, 2021 ... I wish I could make +% something called a check into pefect solo. Okay, another section and I'll check +% out the latest Simon Phillips and other favourite dummer uploads. diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-scope.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-scope.tex new file mode 100644 index 00000000000..1313de92332 --- /dev/null +++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-scope.tex @@ -0,0 +1,341 @@ +% language=us + +% \hfil \hss +% spread + +\environment lowlevel-style + +\startdocument + [title=scope, + color=middleblue] + +\startsection[title=Introduction] + +When I visited the file where register allocations are implemented I wondered to +what extend it made sense to limit allocation to global instances only. This +chapter deals with this phenomena. + +\stopsection + +\startsection[title=Registers] + +In \TEX\ definitions can be local or global. Most assignments are local within a +group. Registers and definitions can be assigned global by using the \type +{\global} prefix. There are also some properties that are global by design, like +\type {\prevdepth}. A mixed breed are boxes. When you tweak its dimensions you +actually tweak the current box, which can be an outer level. Compare: + +\starttyping[option=TEX] +\scratchcounter = 1 +here the counter has value 1 +\begingroup + \scratchcounter = 2 + here the counter has value 2 +\endgroup +here the counter has value 1 +\stoptyping + +with: + +\starttyping[option=TEX] +\setbox\scratchbox=\hbox{} +here the box has zero width +\begingroup + \wd\scratchbox=10pt + here the box is 10pt wide +\endgroup +here the box is 10pt wide +\stoptyping + +It all makes sense so a remark like \quotation {Assignments to box dimensions are +always global} are sort of confusing. Just look at this: + +\startbuffer +\setbox\scratchbox=\hbox to 20pt{} +here the box is \the\wd\scratchbox\ wide\par +\begingroup + \setbox\scratchbox=\hbox{} + here the box is \the\wd\scratchbox\ wide\par + \begingroup + \wd\scratchbox=15pt + here the box is \the\wd\scratchbox\ wide\par + \endgroup + here the box is \the\wd\scratchbox\ wide\par +\endgroup +here the box is \the\wd\scratchbox\ wide\par +\stopbuffer + +\typebuffer[option=TEX] \startlines \getbuffer \stoplines + +If you don't think about it, what happens is what you expect. Now watch the next +variant: + +\startbuffer +\setbox\scratchbox=\hbox to 20pt{} +here the box is \the\wd\scratchbox\ wide\par +\begingroup + \setbox\scratchbox=\hbox{} + here the box is \the\wd\scratchbox\ wide\par + \begingroup + \global\wd\scratchbox=15pt + here the box is \the\wd\scratchbox\ wide\par + \endgroup + here the box is \the\wd\scratchbox\ wide\par +\endgroup +here the box is \the\wd\scratchbox\ wide\par +\stopbuffer + +The \type {\global} is only effective for the current box. It is good to realize +that when we talk registers, the box register behaves just like any other +register but the manipulations happen to the current one. + +\typebuffer \startlines \getbuffer \stoplines + +\startbuffer +\scratchdimen=20pt +here the dimension is \the\scratchdimen\par +\begingroup + \scratchdimen=0pt + here the dimension is \the\scratchdimen\par + \begingroup + \global\scratchdimen=15pt + here the dimension is \the\scratchdimen\par + \endgroup + here the dimension is \the\scratchdimen\par +\endgroup +here the dimension is \the\scratchdimen\par +\stopbuffer + +\typebuffer[option=TEX] \startlines \getbuffer \stoplines + +\stopsection + +\startsection[title=Allocation] + +The plain \TEX\ format has set some standards and one of them is that registers +are allocated with \type {\new...} commands. So we can say: + +\starttyping[option=TEX] +\newcount\mycounta +\newdimen\mydimena +\stoptyping + +These commands take a register from the pool and relate the given name to that +entry. In \CONTEXT\ we have a bunch of predefined scratch registers for general +use, like: + +\startbuffer +scratchcounter : \meaningfull\scratchcounter +scratchcounterone : \meaningfull\scratchcounterone +scratchcountertwo : \meaningfull\scratchcountertwo +scratchdimen : \meaningfull\scratchdimen +scratchdimenone : \meaningfull\scratchdimenone +scratchdimentwo : \meaningfull\scratchdimentwo +\stopbuffer + +\typebuffer[option=TEX] + +The meaning reveals what these are: + +\startlines \tttf \getbuffer \stoplines + +You can use the numbers directly but that is a bad idea because they can clash! +In the original \TEX\ engine there are only 256 registers and some are used by +the engine and the core of a macro package itself, so that leaves a little amount +for users. The \ETEX\ extension lifted that limitation and bumped to 32K and +\LUATEX\ upped that to 64K. One could go higher but what makes sense? These +registers are taking part of the fixed memory slots because that makes nested +(grouped) usage efficient and access fast. The number you see above is deduced +from the so called command code (here indicated by \type {\count}) and an index +encoded in the same token. So, \type {\scratchcounter} takes a single token +contrary to the verbose \type {\count257} that takes four tokens where the number +gets parsed every time it is needed. But those are details that a user can +forget. + +As mentioned, commands like \type {\newcount \foo} create a global control +sequence \type {\foo} referencing a counter. You can locally redefine that +control sequence unless in \LUAMETATEX\ you have so called overload mode enabled. +You can do local or global assignments to these registers. + +\starttyping[option=TEX] +\scratchcounter = 123 +\begingroup + \scratchcounter = 456 + \begingroup + \global\scratchcounter = 789 + \endgroup +\endgroup +\stoptyping + +And in both cases count register 257 is set. When an assignment is global, +all current values to that register get the same value. Normally this is all +quite transparent: you get what you ask for. However the drawback is that +as a user you cannot know what variables are already defined, which means +that this will fail (that is: it will issue a message): + +\starttyping[option=TEX] +\newcount\scratchcounter +\stoptyping + +as will the second line in: + +\starttyping[option=TEX] +\newcount\myscratchcounter +\newcount\myscratchcounter +\stoptyping + +In \CONTEXT\ the scratch registers are visible but there are lots of internally +used ones are protected from the user by more obscure names. So what if you want +to use your own register names without \CONTEXT\ barking to you about not being +able to define it. This is why in \LMTX\ (and maybe some day in \MKIV) we now +have local definitions: + +\startbuffer +\begingroup + \newlocaldimen\mydimena \mydimena1\onepoint + \newlocaldimen\mydimenb \mydimenb2\onepoint + (\the\mydimena,\the\mydimenb) + \begingroup + \newlocaldimen\mydimena \mydimena3\onepoint + \newlocaldimen\mydimenb \mydimenb4\onepoint + \newlocaldimen\mydimenc \mydimenc5\onepoint + (\the\mydimena,\the\mydimenb,\the\mydimenc) + \begingroup + \newlocaldimen\mydimena \mydimena6\onepoint + \newlocaldimen\mydimenb \mydimenb7\onepoint + (\the\mydimena,\the\mydimenb) + \endgroup + \newlocaldimen\mydimend \mydimend8\onepoint + (\the\mydimena,\the\mydimenb,\the\mydimenc,\the\mydimend) + \endgroup + (\the\mydimena,\the\mydimenb) +\endgroup +\stopbuffer + +\typebuffer[option=TEX] + +The allocated registers get zero values but you can of course set them to any +value that fits their nature: + +\startlines \getbuffer \stoplines + +\startbuffer +\begingroup + \setnewlocaldimen\mydimena 1\onepoint + \setnewlocaldimen\mydimenb 2\onepoint + (\the\mydimena,\the\mydimenb) + \begingroup + \setnewlocaldimen\mydimena 3\onepoint + \setnewlocaldimen\mydimenb 4\onepoint + \setnewlocaldimen\mydimenc 5\onepoint + (\the\mydimena,\the\mydimenb,\the\mydimenc) + \begingroup + \setnewlocaldimen\mydimena 6\onepoint + \setnewlocaldimen\mydimenb 7\onepoint + (\the\mydimena,\the\mydimenb) + \endgroup + \setnewlocaldimen\mydimend 8\onepoint + (\the\mydimena,\the\mydimenb,\the\mydimenc,\the\mydimend) + \endgroup + (\the\mydimena,\the\mydimenb) +\endgroup +\stopbuffer + +You can also use the next variant where you also pass the initial value: + +\typebuffer[option=TEX] + +So, again we get: + +\startlines \getbuffer \stoplines + +When used in the body of the macro there is of course a little overhead +involved in the repetitive allocation but normally that can be neglected. + +\stopsection + +\startsection[title=Files] + +When adding these new allocators I also wondered about the read and write +allocators. We don't use them in \CONTEXT\ but maybe users like them, so let's +give an example and see what more demands they have: + +\starttyping[option=TEX] +\integerdef\StartHere\numexpr\inputlineno+2\relax +\starthiding +SOME LINE 1 +SOME LINE 2 +SOME LINE 3 +SOME LINE 4 +\stophiding +\integerdef\StopHere\numexpr\inputlineno-2\relax +\stoptyping + +% We need to be in the file! + +\integerdef\StartHere\numexpr\inputlineno+2\relax +\starthiding +SOME LINE 1 +SOME LINE 2 +SOME LINE 3 +SOME LINE 4 +\stophiding +\integerdef\StopHere\numexpr\inputlineno-2\relax + +\startbuffer +\begingroup + \newlocalread\myreada + \immediate\openin\myreada {lowlevel-scope.tex} + \dostepwiserecurse{\StopHere}{\StartHere}{-1}{ + \readline\myreada line #1 to \scratchstring #1 : \scratchstring \par + } + \blank + \dostepwiserecurse{\StartHere}{\StopHere}{1}{ + \read \myreada line #1 to \scratchstring #1 : \scratchstring \par + } + \immediate\closein\myreada +\endgroup +\stopbuffer + +\typebuffer[option=TEX] + +Here, instead of hard coded line numbers we used the stored values. The +optional \type {line} keyword is a \LMTX\ speciality. + +\startlines \getbuffer \stoplines + +Actually an application can be found in a small (demonstration) module: + +\startbuffer +\usemodule[system-readers] +\stopbuffer + +\typebuffer[option=TEX] \getbuffer + +This provides the code for doing this: + +\starttyping[option=TEX] +\startmarkedlines[test] +SOME LINE 1 +SOME LINE 2 +SOME LINE 3 +\stopmarkedlines + +\begingroup + \newlocalread\myreada + \immediate\openin\myreada {\markedfilename{test}} + \dostepwiserecurse{\lastmarkedline{test}}{\firstmarkedline{test}}{-1}{ + \readline\myreada line #1 to \scratchstring #1 : \scratchstring \par + } + \immediate\closein\myreada +\endgroup +\stoptyping + +As you see in these examples, we an locally define a read channel without +getting a message about it already being defined. + +\stopsection + +\stopdocument + + diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex new file mode 100644 index 00000000000..4f705e37caa --- /dev/null +++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex @@ -0,0 +1,254 @@ +% language=us + +% It took some time to get the right balance of using the overload related features +% but at some point it started feeling right. Of course it will never be as +% perfectly timed and integrated as Gavin Harrison performance on "Threatening War" +% (youtube movie) but that doesn't mean I should not aim for perfection. But as +% with drumming, it takes practising and that is what I did on a subset of sources +% when writing the engine code. +% +% It tooks a few weeks but November 21 2020 the last of the core files was turned +% \LMTX, not that the work was done (checking to be done (thanks Wolfgang!), imp +% files to be checked, many \LUA\ files to be updated) but it's a start. This time +% the musical timestamp is listening to Nikola Cvetkovic (piano) and history +% podcasts. + +\usemodule[system-tokens] + +\environment lowlevel-style + +\startdocument + [title=security, + color=middleorange] + +\startsection[title=Preamble] + +Here I will discuss a moderate security subsystem of \LUAMETATEX\ and therefore +\CONTEXT\ \LMTX. This is not about security in the sense of the typesetting +machinery doing harm to your environment, but more about making sure that a user +doesn't change the behavior of the macro package in ways that introduce +interference and thereby unwanted side effect. It's all about protecting macros. + +This is all very experimental and we need to adapt the \CONTEXT\ source code to +this. Actually that will happen a few times because experiments trigger that. It +might take a few years before the security model is finalized and all files are +updated accordingly. There are lots of files and macros involved. In the process +the underlying features in the engine might evolve. + +\stopsection + +\startsection[title=Flags] + +Before we go into the security levels we see what flags can be set. The \TEX\ +language has a couple of so called prefixes that can be used when setting values +and defining macros. Any engine that has traditional \TEX\ with \ETEX\ extensions +can do this: + +\starttyping[option=TEX] + \def\foo{foo} +\global \def\foo{foo} +\global\protected\def\foo{foo} +\stoptyping + +And \LUAMETATEX\ adds another one: + +\starttyping[option=TEX] + \tolerant \def\foo{foo} +\global\tolerant \def\foo{foo} +\global\tolerant\protected\def\foo{foo} +\stoptyping + +What these prefixes do is discussed elsewhere. For now is is enough to know that +the two optional prefixes \type {\protected} and \type {\tolerant} make for four +distinctive cases of macro calls. + +But there are more prefixes: + +\starttabulate +\HL +\NC \type {frozen} \NC a macro that has to be redefined in a managed way \NC \NR +\NC \type {permanent} \NC a macro that had better not be redefined \NC \NR +\NC \type {primitive} \NC a primitive that normally will not be adapted \NC \NR +\NC \type {immutable} \NC a macro or quantity that cannot be changed, it is a constant \NC \NR +\NC \type {mutable} \NC a macro that can be changed no matter how well protected it is \NC \NR +\HL +\NC \type {instance} \NC a macro marked as (for instance) be generated by an interface \NC \NR +\HL +\NC \type {noaligned} \NC the macro becomes acceptable as \type {\noalign} alias \NC \NR +\HL +\NC \type {overloaded} \NC when permitted the flags will be adapted \NC \NR +\NC \type {enforced} \NC all is permitted (but only in zero mode or ini mode) \NC \NR +\NC \type {aliased} \NC the macro gets the same flags as the original \NC \NR +\HL +\stoptabulate + +These prefixed set flags to the command at hand which can be a macro but +basically any control sequence. + +To what extent the engine will complain when a property is changed in a way that +violates the above depends on the parameter \type {\overloadmode}. When this +parameter is set to zero no checking takes place. More interesting are values +larger than zero. If that is the case, when a control sequence is flagged as +mutable, it is always permitted to change. When it is set to immutable one can +never change it. The other flags determine the kind of checking done. Currently +the following overload values are used: + +\starttabulate[|l|l|c|c|c|c|c|] + \NC \NC \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR + \NC 1 \NC warning \NC \star \NC \star \NC \star \NC \NC \NC \NR + \NC 2 \NC error \NC \star \NC \star \NC \star \NC \NC \NC \NR + \NC 3 \NC warning \NC \star \NC \star \NC \star \NC \star \NC \NC \NR + \NC 4 \NC error \NC \star \NC \star \NC \star \NC \star \NC \NC \NR + \NC 5 \NC warning \NC \star \NC \star \NC \star \NC \star \NC \star \NC \NR + \NC 6 \NC error \NC \star \NC \star \NC \star \NC \star \NC \star \NC \NR +\stoptabulate + +The even values (except zero) will abort the run. In \CONTEXT\ we plug in a +callback that deals with the messages. A value of 255 will freeze this parameter. +At level five and above the \type {instance} flag is also checked but no drastic +action takes place. We use this to signal to the user that a specific instance is +redefined (of course the definition macros can check for that too). + +So, how does it work. The following is okay: + +\starttyping[option=TEX] +\def\MacroA{A} +\def\MacroB{B} +\let\MyMacro\MacroA +\let\MyMacro\MacroB +\stoptyping + +The first two macros are ordinary ones, and the last two lines just create an alias. Such +an alias shares the definition, but when for instance \type {\MacroA} is redefined, its +new meaning will not be reflected in the alias. + +\starttyping[option=TEX] +\permanent\protected\def\MacroA{A} +\permanent\protected\def\MacroB{B} +\let\MyMacro\MacroA +\let\MyMacro\MacroB +\stoptyping + +This also works, because the \type {\let} will create an alias with the protected +property but it will not take the \type {permanent} propery along. For that we need +to say: + +\starttyping[option=TEX] +\permanent\protected\def\MacroA{A} +\permanent\protected\def\MacroB{B} +\permanent\let\MyMacro\MacroA +\permanent\let\MyMacro\MacroB +\stoptyping + +or, when we want to copy all properties: + +\starttyping[option=TEX] +\permanent\protected\def\MacroA{A} +\permanent\protected\def\MacroB{B} +\aliased\let\MyMacro\MacroA +\aliased\let\MyMacro\MacroB +\stoptyping + +However, in \CONTEXT\ we have commands that we like to protect against +overloading but at the same time have a different meaning depending on the use +case. An example is the \type {\NC} (next column) command that has a different +implementation in each of the table mechanisms. + +\starttyping[option=TEX] +\permanent\protected\def\NC_in_table {...} +\permanent\protected\def\NC_in_tabulate{...} +\aliased\let\NC\NC_in_table +\aliased\let\NC\NC_in_tabulate +\stoptyping + +Here the second aliasing of \type {\NC} fails (assuming of course that we enabled +overload checking). One can argue that grouping can be used but often no grouping +takes place when we redefine on the fly. Because \type {frozen} is less restrictive +than \type {primitive} or \type {permanent}, and of course \type {immutable}, the +next variant works: + +\starttyping[option=TEX] +\frozen\protected\def\NC_in_table {...} +\frozen\protected\def\NC_in_tabulate{...} +\overloaded\let\NC\NC_in_table +\overloaded\let\NC\NC_in_tabulate +\stoptyping + +However, in practice, as we want to keep the overload checking, we have to do: + +\starttyping[option=TEX] +\frozen\protected\def\NC_in_table {...} +\frozen\protected\def\NC_in_tabulate{...} +\overloaded\frozen\let\NC\NC_in_table +\overloaded\frozen\let\NC\NC_in_tabulate +\stoptyping + +or use \type {\aliased}, but there might be conflicting permissions. This is not +that nice, so there is a kind of dirty trick possible. Consider this: + +\starttyping[option=TEX] +\frozen\protected\def\NC_in_table {...} +\frozen\protected\def\NC_in_tabulate{...} +\def\setNCintable {\enforced\let\frozen\let\NC\NC_in_table} +\def\setNCintabulate{\enforced\let\frozen\let\NC\NC_in_tabulate} +\stoptyping + +When we're in so called \type {initex} mode or when the overload mode is zero, +the \type {\enforced} prefix is internalized in a way that signals that the +follow up is not limited by the overload mode and permissions. This definition +time binding mechanism makes it possible to use \type {permanent} macros that +users cannot redefine, but existing macros can, unless of course they tweak the +mode parameter. + +Now keep in mind that users can always cheat but that is intentional. If you +really want to avoid that you can set the overload mode to 255 after which it +cannot be set any more. However, it can be useful to set the mode to zero (or +some warning level) when foreign macro packages are used. + +\stopsection + +\startsection[title=Complications] + +One side effect of all this is that all those prefixes can lead to more code. On +the other hand we save some due to the extended macro argument handling features. +When you take the size of the format file as reference, in the end we get a +somewhat smaller file. Every token that you add of remove gives a 8~bytes +difference. The extra overhead that got added to the engine is compensated by the +fact that some macro implementations can be more efficient. In the end, in spite +of these new features and the more extensive testing of flags performance is +about the same. \footnote {And if you wonder about memory, by compacting the used +(often scattered) token memory before dumping I manages to save some 512K on the +format file, so often the loss and gain are somewhere else.} + +\stopsection + +\startsection[title=Introspection] + +In case you want to get some details about the properties of a macro, you can +check its meaning. The full variant shows all of them. + +\startbuffer +% a macro with two optional arguments with optional spacing in between: + +\permanent\tolerant\protected\def\MyFoo[#1]#*[#2]{(#1)(#2)} + +\meaningless\MyFoo\par +\meaning \MyFoo\par +\meaningfull\MyFoo\par +\stopbuffer + +\typebuffer[option=TEX] + +\startpacked \getbuffer \stoppacked + +\stopsection + +% In \CONTEXT: +% c! v! s! ?? +% newif newcount ... newconditional etc +% userinterface (permanent) +% primitives +% noaligned +% frozen is for users + +\stopdocument |