diff options
Diffstat (limited to 'Master/texmf-dist/doc/context/sources/general/manuals/evenmore/evenmore-keywords.tex')
-rw-r--r-- | Master/texmf-dist/doc/context/sources/general/manuals/evenmore/evenmore-keywords.tex | 395 |
1 files changed, 395 insertions, 0 deletions
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/evenmore/evenmore-keywords.tex b/Master/texmf-dist/doc/context/sources/general/manuals/evenmore/evenmore-keywords.tex new file mode 100644 index 00000000000..1ffacf0ab51 --- /dev/null +++ b/Master/texmf-dist/doc/context/sources/general/manuals/evenmore/evenmore-keywords.tex @@ -0,0 +1,395 @@ +% language=us + +% Talking of keywords: Jacob Collier, Count The People is definitely an example +% of showing keywords and no way that the fonts used there are done by tex: +% +% https://www.youtube.com/watch?v=icplHV25fqs + +\environment evenmore-style + +\startcomponent evenmore-keywords + +\startchapter[title=Keywords] + +Some primitives in \TEX\ can take one or more optional keywords and|/|or keywords +followed by one or more values. In traditional \TEX\ it concerns a handful of +primitives, in \PDFTEX\ there are plenty of backend|-|related primitives, +\LUATEX\ introduced optional keywords to some math constructs and attributes to +boxes, and \LUAMETATEX\ adds some more too. The keyword scanner in \TEX\ is +rather special. Keywords are used in cases like: + +\starttyping +\hbox spread 10cm {...} +\advance\scratchcounter by 10 +\vrule width 3cm height 1ex +\stoptyping + +Sometimes there are multiple keywords, as with rules, in which case you can +imagine a case like: + +\starttyping +\vrule width 3cm depth 1ex width 10cm depth 0ex height 1ex\relax +\stoptyping + +Here we add a \type {\relax} to end the scanning. If we don't do that and the +rule specification is followed by arbitrary (read:\ unpredictable) text, the next +word might be a valid keyword and when followed by a dimension (unlikely) it will +happily be read as a directive, or when not followed by a dimension an error +message will show up. Sometimes the scanning is more restricted, as with glue +where the optional \type {plus} and \type {minus} are to come in that order, but +when missing, again a word from the text can be picked up if one doesn't +explicitly end with a \type {\relax} or some other token. + +\starttyping +\scratchskip = 10pt plus 10pt minus 10pt % okay +\scratchskip = 10pt plus 10pt % okay +\scratchskip = 10pt minus 10pt % okay +\scratchskip = 10pt minus 10pt plus 10pt % typesets "plus 10pt" +\scratchskip = 10pt plus whatever % an error +\stoptyping + +The scanner is case insensitive, so the following specifications are all valid: + +\starttyping +\hbox To 10cm {To} +\hbox TO 10cm {TO} +\hbox tO 10cm {tO} +\hbox to 10cm {to} +\stoptyping + +It happens that keywords are always simple English words so the engine uses a +cheap check deep down, just offsetting to uppercase, but of course that will not +work for arbitrary \UTF-8\ (as used in \LUATEX) and it's also unrelated to the +upper- and lowercase codes as \TEX\ knows them. + +The above lines scan for the keyword \type {to} and after that for a dimension. +While keyword scanning is case tolerant, dimension scanning is period tolerant: + +\starttyping +\hbox to 10cm {10cm} +\hbox to 10.0cm {10.0cm} +\hbox to .0cm {.0cm} +\hbox to .cm {.cm} +\hbox to 10.cm {10.cm} +\stoptyping + +These are all valid and according to the specification; even the single period is +okay, although it looks funny. It would not be hard to intercept that but I guess +that when \TEX\ was written anything that could harm performance was taken into +account. One can even argue for cases like: + +\starttyping +\hbox to \first.\second cm {.cm} +\stoptyping + +Here \type {\first} and|/|or \type {\second} can be empty. Most users won't +notice these side effects of scanning numbers anyway. + +The reason for writing up any discussion of keywords is the following. Optional +keyword scanning is kind of costly, not so much now, but more so decades ago +(which led to some interesting optimizations, as we'll see). For instance, in the +first line below, there is no keyword. The scanner sees a \type {1} and it not +being a keyword, pushes that character back in the input. + +\starttyping +\advance\scratchcounter 10 +\advance\scratchcounter by 10 +\stoptyping + +In the case of: + +\starttyping +\scratchskip 10pt plux +\stoptyping + +it has to push back the four scanned tokens \type {plux}. Now, in the engine +there are lots of cases where lookahead happens and when a condition is not +satisfied, the just|-|read token is pushed back. Incidentally, when picking up +the next token triggered some expansion, it's not the original next token that +gets pushed back, but the first token seen after the expansion. Pushing back +tokens is not that inefficient, although it involves allocating a token and +pushing and popping input stacks (we're talking of a mix of reading from file, +token memory, \LUA\ prints, etc.)\ but it always takes a little time and memory. +In \LUATEX\ there are more keywords for boxes, and there we have loops too: in a +box specification one or more optional attributes are scanned before the optional +\type {to} or \type {spread}, so again there can be push back when no more \type +{attr} are seen. + +\starttyping +\hbox attr 1 98 attr 2 99 to 1cm{...} +\stoptyping + +In \LUAMETATEX\ there is even more optional keyword scanning, but we leave that +for now and just show one example: + +\starttyping +\hbox spread 10em {\hss + \hbox orientation 0 yoffset 1mm to 2em {up}\hss + \hbox to 2em {here}\hss + \hbox orientation 0 xoffset -1mm to 2em {down}\hss +} +\stoptyping + +Although one cannot mess too much with these low|-|level scanners there was room +for some optimization, so the penalty we pay for more keyword scanning in +\LUAMETATEX\ is not that high. (I try to compensate when adding features that +have a possible performance hit with some gain elsewhere.) + +It will be no surprise that there can be interesting side effects to keyword +scanning. For instance, using the two character keyword \type {by} in an \type +{\advance} can be more efficient because nothing needs to be pushed back. The +same is true for the sometimes optional equal: + +\starttyping +\scratchskip = 10pt +\stoptyping + +Similar impacts on efficiency can be found in the way the end of a number is +seen, basically anything not resolving to a number (or digit). (For these, assume +a following token will terminate the number if needed; we're focusing on the +spaces here.) + +\starttyping +\scratchcounter 10% space not seen, ends \cs +\scratchcounter =10% no push back of optional = +\scratchcounter = 10% extra optional space gobble +\scratchcounter = 10 % efficient ending of number scanning +\scratchcounter = 10\relax % depending on engine less efficient +\stoptyping + +In the above examples scanning the number involves: skipping over spaces, +checking for an optional equal, skipping over spaces, scanning for a sign, +checking for an optional octal or hexadecimal trigger (single or double quote +character), scanning the number till a non|-|digit is seen. In the case of +dimensions there is fraction scanning as well as unit scanning too. + +In any case, the equal is optional and kind of a keyword. Having an equal can be +more efficient then not having one, again due to push back in case of no equal +being seen, In the process spaces have been skipped, so add to the overhead the +scanning for optional spaces. In \LUAMETATEX\ all that has been optimized a bit. +By the way, in dimension scanning \type {pt} is actually a keyword and as there +are several dimensions possible quite some push back can happen there, but we +scan for the most likely candidates first. + +All that said, we're now ready for a surprise. The keyword scanner gets a string +that it will test for, say, \type {to} in case of a box specification. It then +will fetch tokens from whatever provides the input. A token encodes a so|-|called +command and a character and can be related to a control sequence. For instance, +the character \type {t} becomes a letter command with related value \number`t. +So, we have three properties: the command code, the character code and the +control sequence code. Now, instead of checking if the command code is a letter +or other character (two checks) a fast check happens for the control sequence +code being zero. If that is the case, the character code is compared. In practice +that works out well because the characters that make up a keyword are in the +range \number"41--\number"5A\ and \number"61--\number"7A, and all other character +codes are either below that (the ones that relate to primitives where the +character code is actually a subcommand of a limited range) or much larger +numbers that, for instance, indicate an entry in some array, where the first +useful index is above the mentioned ranges. + +The surprise is in the fact that there is no checking for letters or other +characters, so this is why the following code will work too: \footnote {No longer +in \LUAMETATEX\ where we do a bit more robust check.} + +\starttyping +\catcode `O= 1 \hbox tO 10cm {...} % { begingroup +\catcode `O= 2 \hbox tO 10cm {...} % } endgroup +\catcode `O= 3 \hbox tO 10cm {...} % $ mathshift +\catcode `O= 4 \hbox tO 10cm {...} % & alignment +\catcode `O= 6 \hbox tO 10cm {...} % # parameter +\catcode `O= 7 \hbox tO 10cm {...} % ^ superscript +\catcode `O= 8 \hbox tO 10cm {...} % _ subscript +\catcode `O=11 \hbox tO 10cm {...} % letter +\catcode `O=12 \hbox tO 10cm {...} % other +\stoptyping + +In the first line, if we changed the catcode of \type {T} (instead of \type {O}), +it gives an error because \TEX\ sees a begin group character (category code 1) +and starts the group, but as a second character in a keyword (\type {O}) it's +okay because \TEX\ will not look at the category code. + +Of course only the cases \type {11} and \type {12} make sense in practice. +Messing with the category codes of regular letters this way will definitely give +problems with processing normal text. In a case like: + +\starttyping +{\catcode `o=3 \hbox to 10cm {oeps}} % $ mathshift {oeps} +{\catcode `O=3 \hbox to 10cm {Oeps}} % $ mathshift {$eps} +\stoptyping + +we have several issues: the primitive control sequence \type {\hbox} has an \type +{o} so \TEX\ will stop after \type {\hb} which can be undefined or a valid macro +and what happens next is hard to predict. Using uppercase will work but then the +content of the box is bad because there the \type {O} enters math. Now consider: + +\starttyping +{\catcode `O=3 \hbox tO 10cm {Oeps Oeps}} % {$eps $eps} +\stoptyping + +This will work because there are now two \type {O}'s in the box, so we have +balanced inline math triggers. But how does one explain that to a user? (Who +probably doesn't understand where an error message comes from in the first +place.) Anyway, this kind of tolerance is still not pretty, so in \LUAMETATEX\ we +now check for the command code and stick to letters and other characters. On +today's machines (and even on my by now ancient workhorse) the performance hit +can be neglected. + +In fact, by intercepting the weird cases we also avoid an unnecessary case check +when we fall through the zero control sequence test. Of course that also means +that the above mentioned category code trickery doesn't work any more: only +letters and other characters are now valid in keyword scanning. Now, it can be +that some macro programmer actually used those side effects but apart from some +macro hacker being hurt because no longer mastering those details can be showed +off, it is users that we care more for, don't we? + +To be sure, the abovementioned performance of keyword and equal scanning is not +that relevant in practice. But for the record, here are some timings on a laptop +with a i7-3849\cap{QM} processor using \MINGW\ binaries on a 64-bit \MSWINDOWS\ +10 system. The times are the averages of five times a million such assignments +and advancements. + +\starttabulate[|l|c|c|c|] +\FL +\NC one million times \NC terminal \NC \LUAMETATEX\ \NC \LUATEX \NC \NR +\ML +\NC \type {\advance\scratchcounter 1} \NC space \NC 0.068 \NC 0.085 \NC \NR +\NC \type {\advance\scratchcounter 1} \NC \type {\relax} \NC 0.135 \NC 0.149 \NC \NR +\NC \type {\advance\scratchcounter by 1} \NC space \NC 0.087 \NC 0.099 \NC \NR +\NC \type {\advance\scratchcounter by 1} \NC \type {\relax} \NC 0.155 \NC 0.161 \NC \NR +\NC \type {\scratchcounter 1} \NC space \NC 0.057 \NC 0.096 \NC \NR +\NC \type {\scratchcounter 1} \NC \type {\relax} \NC 0.125 \NC 0.151 \NC \NR +\NC \type {\scratchcounter=1} \NC space \NC 0.063 \NC 0.080 \NC \NR +\NC \type {\scratchcounter=1} \NC \type {\relax} \NC 0.131 \NC 0.138 \NC \NR +\LL +\stoptabulate + +We differentiate here between using a space as terminal or a \type {\relax}. The +latter is a bit less efficient because more code is involved in resolving the +meaning of the control sequence (which eventually boils down to nothing) but +nevertheless, these are not timings that one can lose sleep over, especially when +the rest of a decent \TEX\ run is taken into account. And yes, \LUAMETATEX\ +(\LMTX) is a bit faster here than \LUATEX, but I would be disappointed if that +weren't the case. + +% luametatex: + +% \luaexpr{(0.068+0.070+0.069+0.067+0.068)/5} 0.068\crlf +% \luaexpr{(0.137+0.132+0.136+0.137+0.134)/5} 0.135\crlf +% \luaexpr{(0.085+0.088+0.084+0.089+0.087)/5} 0.087\crlf +% \luaexpr{(0.145+0.160+0.158+0.156+0.154)/5} 0.155\crlf +% \luaexpr{(0.060+0.055+0.059+0.055+0.056)/5} 0.057\crlf +% \luaexpr{(0.118+0.127+0.128+0.122+0.130)/5} 0.125\crlf +% \luaexpr{(0.063+0.062+0.067+0.061+0.063)/5} 0.063\crlf +% \luaexpr{(0.127+0.128+0.133+0.128+0.140)/5} 0.131\crlf + +% luatex: + +% \luaexpr{(0.087+0.090+0.083+0.081+0.086)/5} 0.085\crlf +% \luaexpr{(0.150+0.151+0.146+0.154+0.145)/5} 0.149\crlf +% \luaexpr{(0.100+0.092+0.113+0.094+0.098)/5} 0.099\crlf +% \luaexpr{(0.162+0.165+0.161+0.160+0.157)/5} 0.161\crlf +% \luaexpr{(0.093+0.101+0.086+0.100+0.098)/5} 0.096\crlf +% \luaexpr{(0.147+0.151+0.160+0.144+0.151)/5} 0.151\crlf +% \luaexpr{(0.076+0.085+0.088+0.073+0.078)/5} 0.080\crlf +% \luaexpr{(0.136+0.138+0.142+0.135+0.140)/5} 0.138\crlf + +After the \CONTEXT\ 2020 meeting I entered another round of staring at the code. +One of the decision made at that meeting was to drop the \type {nd} and \type +{nc} units as they were never official. That made me (again) wonder of that bit +of the code could be done nicer as there is a mix of scanning units like \type +{pt}, \type {bp} and \type {cm}, fillers like \type {fi} and \type {fill}, pseudo +units like \type {ex} and \type {em}, special interception of \type {mu}, as well +as the \type {plus} and \type {minus} parsing for glue. That code was already +redone a bit so that here was less push back of tokens which had the side effect +of dimension scanning being some 50\% faster than in \LUATEX. + +The same is true for scanning rule specs and scanning the box properties. In the +later case part of the optimization came from not checking properties that +already had been set, or only scanning them when for instance the \type +{orientation} flag had been set (a new option in \LUAMETATEX\ with an additional +four offset and move parameters). Also, some options, like the target dimensions, +came after scanning the new ones. Again, this was quite a bit faster than in +\LUATEX, not that it is noticeable on a normal run. All is mixed with skipping +spacers and relax tokens plus quitting at a brace. + +Similar mixed scanning happens in some of the (new) math command, but these are +less critical. Actually there some new commands had to be used because for +instance \type {\over} takes any character as valid argument and keywords would +definitely be incompatible there. + +Anyway, I started wondering if some could be done differently and finally decided +to use a method that I already played with years ago. The main reason for not +using it was that I wanted to remain compatible with the way traditional \TEX\ +scans. However, as we have many more keyword we already are no longer compatible +in that department and the alternative implementation makes the code look nicer +and has the benefit of being (more than) twice as fast. And when I run into +issues in \CONTEXT\ I should just fix sloppy code. + +The compatibility issue is not really a problem when you consider the following +cases. + +\starttyping +\hbox reverse attr 123 456 orientation 4 xoffset 10pt spread 10cm { } +\hrule xoffset 10pt width 10cm depth 3mm +\hskip 3pt plus 2pt minus 1pt +\stoptyping + +In the original approach these three case each have their own special side +effects. In the case of a \type {\hbox} the scanning stops at a relax or left +brace. An unknown keyword gives an error. So, there is no real benefit in pushing +back tokens here. The order matters: the \type {spread} or \type {to} comes last. + +In the case of a \type {\hrule} the scanning stops when the keyword is not one of +the known. This has the side effect that when such a rule definition is hidden in +a macro and followed by for instance \type {width} without unit one gets an error +and when a unit is given the rule can come out different than expected and the +text is gone. For that reason a rule specification like this is often closed by +\type {\relax} (any command that doesn't expand to a keyword works too). Here +keywords can occur multiple times. As we have additional keyword a lookahead +becomes even more an issue (not that \type {xoffset}) is a likely candidate. + +The last example is special in a different way: order matters in the sense that a +\type {minus} specifier can follow a \type {plus} but not the reverse. And only +one \type {plus} and \type {minus} can be given. Again one can best finish this +specification by a something that doesn't look like a keyword, so often one will +see a \type {\relax}. + +The advantage of the new method is that the order doesn't matter any more and +that using a keyword multiple times overloads earlier settings. And this is +consistent for all commands that used keywords (with a few exceptions in math +where keywords drive later parsing and for font definitions where we need to be +compatible. We give a slightly better error message: we mention the expected +keyword. Another side effect is that any characters that is a legal start of a +known keyword will trigger further parsing and issue an error message when it +fails. Indeed, \LUAMETATEX\ has no mercy. + +In practice the mentioned special effects mean that a macro package will not run +into trouble with boxes because unknown keywords make it crash and that rules and +glue is terminated in a way that prevents lookahead. The new method kind of +assumes this and one can argue that when something breaks one has to fix the +macro code. Macro writers know that one cannot predict what users come up with +and that users also don't look into the macros and therefore they take +precautions. Also, a more rigorous parsing results in hopefully a better message. + +And yes, when I ran the test suite there was indeed a case where I had to add a +\type {\relax}, but I can live with that. As long as users don't notice it. + +Now, one of the interesting properties of the slightly different scanning is +that we can do this: + +\starttyping +\hbox to 4cm attr 123 456 reverse to 3cm {...} +\stoptyping + +So, we have a less strict order and we can overload arguments too. We'll see how +this will be applied in \CONTEXT. + +\stopchapter + +\stopcomponent + +% another nice example: \the is expanded so we get the old value + +% \scratchskip = 10pt plus 1fill \the\scratchskip % old value +% \scratchskip = 10pt plus 1fill [\the\scratchskip] % new value +% \scratchskip = 10pt plus 1fi l l [\the\scratchskip] % also works |