path: root/Master/texmf-dist/doc/context/sources/general/manuals/luametatex/luametatex-enhancements.tex

% language=uk

\environment luametatex-style

\startcomponent luametatex-enhancements

\startchapter[reference=enhancements,title={Basic \TEX\ enhancements}]

\startsection[title={Introduction}]

\startsubsection[title={Primitive behaviour}]

From day one, \LUATEX\ has offered extra features compared to the superset of
\PDFTEX, which includes \ETEX, and \ALEPH. This has not been limited to the
possibility to execute \LUA\ code via \prm {directlua}, but \LUATEX\ also adds
functionality via new \TEX|-|side primitives or extensions to existing ones. The
same is true for \LUAMETATEX. Some primitives have \type {luatex} in their name
and there will be no \type {luametatex} variants. This is because we consider
\LUAMETATEX\ to be \LUATEX 2\high{+}.

Contrary to the \LUATEX\ engine \LUAMETATEX\ enables all its primitives. You can
clone (a selection of) primitives with a different prefix, like this:

\starttyping
\directlua { tex.enableprimitives('normal',tex.extraprimitives()) }
\stoptyping

The \type {extraprimitives} function returns the whole list or a subset,
specified by one or more keywords \type {core}, \type {tex}, \type {etex} or
\type {luatex}. When you clone all primitives you can also do this:

\starttyping
\directlua { tex.enableprimitives('normal',true) }
\stoptyping

But be aware that the curly braces may not have the proper \prm {catcode}
assigned to them at this early time (giving a \quote {Missing number} error), so
it may be needed to put these assignments before the above line:

\starttyping
\catcode `\{ = 1
\catcode `\} = 2
\stoptyping

More fine|-|grained primitives control is possible and you can look up the
details in \in {section} [luaprimitives]. There are only three kinds of
primitives: \type {tex}, \type {etex} and \type {luatex} but a future version
might drop this and no longer make that distinction as it no longer serves a
purpose apart from the fact that it reveals some history.

\stopsubsection

\startsubsection[title={Version information}]

\startsubsubsection[title={\lpr {luatexbanner}, \lpr {luatexversion} and \lpr {luatexrevision}}]

\topicindex{version}
\topicindex{banner}

There are three primitives to test the version of \LUATEX\ (and \LUAMETATEX):

\unexpanded\def\VersionHack#1% otherwise different luatex and luajittex runs
  {\ctxlua{%
     local banner = "\luatexbanner"
     local banner = string.match(banner,"(.+)\letterpercent(") or banner
     context(string.gsub(banner ,"jit",""))%
  }}

\starttabulate[|l|l|pl|]
\DB primitive             \BC value
                          \BC explanation \NC \NR
\TB
\NC \lpr {luatexbanner}   \NC \VersionHack{\luatexbanner}
                          \NC the banner reported on the console \NC \NR
\NC \lpr {luatexversion}  \NC \the\luatexversion
                          \NC major and minor number combined \NC \NR
\NC \lpr {luatexrevision} \NC \the\luatexrevision
                          \NC the revision number \NC \NR
\LL
\stoptabulate

A version is defined as follows:

\startitemize
\startitem
    The major version is the integer result of \lpr {luatexversion} divided by
    100. The primitive is an \quote {internal variable}, so you may need to prefix
    its use with \prm {the} or \prm {number} depending on the context.
\stopitem
\startitem
    The minor version is a number running from 0 upto 99.
\stopitem
\startitem
    The revision is reported by \lpr {luatexrevision}. Contrary to other engines
    in \LUAMETATEX\ is also a number so one needs to prefix it with \prm {the} or
    \prm {number}. \footnote {In the past it always was good to prefix the
    revision with \prm {number} anyway, just to play safe, although there have
    for instance been times that \PDFTEX\ had funny revision indicators that at
    some point ended up as letters due to the internal conversions.}
\stopitem
\startitem
    The full version number consists of the major version (\type {X}), minor
    version (\type {YY}) and revision (\type {ZZ}), separated by dots, so \type
    {X.YY.ZZ}.
\stopitem
\stopitemize

\stopsubsubsection

The \LUAMETATEX\ version number starts at~2 in order to prevent a clash with
\LUATEX, and the version commands are the same. This is a way to indicate that
these projects are related.

\stopsubsection

\stopsection

\startsection[title={\UNICODE\ text support}]

\startsubsection[title={Extended ranges}]

\topicindex{\UNICODE}

Text input and output is now considered to be \UNICODE\ text, so input characters
can use the full range of \UNICODE\ ($2^{20}+2^{16}-1 = \hbox{0x10FFFF}$). Later
chapters will talk of characters and glyphs. Although these are not
interchangeable, they are closely related. During typesetting, a character is
always converted to a suitable graphic representation of that character in a
specific font. However, while processing a list of to|-|be|-|typeset nodes, its
contents may still be seen as a character. Inside the engine there is no clear
separation between the two concepts. Because the subtype of a glyph node can be
changed in \LUA\ it is up to the user. Subtypes larger than 255 indicate that
font processing has happened.

A few primitives are affected by this, all in a similar fashion: each of them has
to accommodate for a larger range of acceptable numbers. For instance, \prm
{char} now accepts values between~0 and $1{,}114{,}111$. This should not be a
problem for well|-|behaved input files, but it could create incompatibilities for
input that would have generated an error when processed by older \TEX|-|based
engines. The affected commands with an altered initial (left of the equal sign)
or secondary (right of the equal sign) value are: \prm {char}, \prm {lccode},
\prm {uccode}, \lpr {hjcode}, \prm {catcode}, \prm {sfcode}, \lpr {efcode}, \lpr
{lpcode}, \lpr {rpcode}, \prm {chardef}.

As far as the core engine is concerned, all input and output to text files is
\UTF-8 encoded. Input files can be pre|-|processed using the \type {reader}
callback. This will be explained in \in {section} [iocallback]. Normalization of
the \UNICODE\ input is on purpose not built|-|in and can be handled by a macro
package during callback processing. We have made some practical choices and the
user has to live with those.

Output in byte|-|sized chunks can be achieved by using characters just outside of
the valid \UNICODE\ range, starting at the value $1{,}114{,}112$ (0x110000). When
the time comes to print a character $c>=1{,}114{,}112$, \LUATEX\ will actually
print the single byte corresponding to $c$ minus 1{,}114{,}112.

Contrary to other \TEX\ engines, the output to the terminal is as|-|is so there
is no escaping with \type {^^}. We operate in a \UTF\ universe. Because we
operate in a \CCODE\ universum, zero characters are special but because we also
live in a \UNICODE\ galaxy that is no real problem.

\stopsubsection

\startsubsection[title={\lpr {Uchar}}]

\topicindex{\UNICODE}

The expandable command \lpr {Uchar} reads a number between~0 and $1{,}114{,}111$
and expands to the associated \UNICODE\ character.

\stopsubsection

\startsubsection[title={Extended tables}]

All traditional \TEX\ and \ETEX\ registers can be 16-bit numbers. The affected
commands are:

\startfourcolumns
\startlines
\prm {count}
\prm {dimen}
\prm {skip}
\prm {muskip}
\prm {marks}
\prm {toks}
\prm {countdef}
\prm {dimendef}
\prm {skipdef}
\prm {muskipdef}
\prm {toksdef}
\prm {insert}
\prm {box}
\prm {unhbox}
\prm {unvbox}
\prm {copy}
\prm {unhcopy}
\prm {unvcopy}
\prm {wd}
\prm {ht}
\prm {dp}
\prm {setbox}
\prm {vsplit}
\stoplines
\stopfourcolumns

Fonts are loaded via \LUA\ and a minimal amount of information is kept at the
\TEX\ end. Sharing resources is up to the loaders. The engine doesn't really care
about what a character (or glyph) number represents (a \UNICODE\ or index) as it
only is interested in dimensions.

In \TEX\ the number of registers is 256 and \ETEX\ bumped that to 32K. One reason
for a fixed number is that these registers are fast ways to store data and
therefore are part of the main lookup table (used for data and pointers to data
as well as save and restore housekeeping). In \LUATEX\ the number was bumped to
64K but one can argue that less would also do. In order to keep the default
memory footprint reasonable, in \LUAMETATEX\ the number of languages, fonts and
marks is limited. The size of some tables can be limited by configuration
settings, so they can start out small and grow till configured maximum which is
smaller than the absolute maximum. The following table shows all kind of defaults
as reported by \typ {status.getconstants()}.

\startluacode
    context.starttabulate { "|T|r|" }
    for k, v in table.sortedhash(status.getconstants()) do
        context.NC() context(k) context.NC() context(v) context.NC() context.NR()
    end
    context.stoptabulate()
\stopluacode

Because we have additional ways to store integers, dimensions and glue, we might
actually decide to decrease the maximum of the registers: if 64K is not enough,
and you work around it, then likely 32K might do as well. Also, we have \LUA\ to
store massive amounts of data. One can argue that saving some 1.5MB memory (when
we go halfway) is not worth the effort in a time when you have to close a browser
in order to free the gigabytes it consumes, but there is no reason not to be lean
and mean: a more conservative approach to start with creates headroom for going
wild later.

\stopsubsection

\stopsection

\startsection[title={Attributes}]

\startsubsection[title={Nodes}]

\topicindex {nodes}

When \TEX\ reads input it will interpret the stream according to the properties
of the characters. Some signal a macro name and trigger expansion, others open
and close groups, trigger math mode, etc. What's left over becomes the typeset
text. Internally we get a linked list of nodes. Characters become \nod {glyph}
nodes that have for instance a \type {font} and \type {char} property and \typ
{\kern 10pt} becomes a \nod {kern} node with a \type {width} property. Spaces are
alien to \TEX\ as they are turned into \nod {glue} nodes. So, a simple paragraph
is mostly a mix of sequences of \nod {glyph} nodes (words) and \nod {glue} nodes
(spaces). A node can have a subtype so that it can be recognized as for instance
a space related glue.

The sequences of characters at some point are extended with \nod {disc} nodes
that relate to hyphenation. After that font logic can be applied and we get a
list where some characters can be replaced, for instance multiple characters can
become one ligature, and font kerns can be injected. This is driven by the
font properties.

Boxes (like \prm {hbox} and \prm {vbox}) become \nod {hlist} or \nod {vlist}
nodes with \type {width}, \type {height}, \type {depth} and \type {shift}
properties and a pointer \type {list} to its actual content. Boxes can be
constructed explicitly or can be the result of subprocesses. For instance, when
lines are broken into paragraphs, the lines are a linked list of \nod {hlist}
nodes, possibly with glue and penalties in between.

Internally nodes have a number. This number is actually an index in the memory
used to store nodes.

So, to summarize: all that you enter as content eventually becomes a node, often
as part of a (nested) list structure. They have a relative small memory footprint
and carry only the minimal amount of information needed. In traditional \TEX\ a
character node only held the font and slot number, in \LUATEX\ we also store some
language related information, the expansion factor, etc. Now that we have access
to these nodes from \LUA\ it makes sense to be able to carry more information
with a node and this is where attributes kick in.

It is important to keep in mind that there are situations where nodes get created
in the current context. For instance, when \TEX\ builds a paragraph or page or
constructs math formulas, it does add nodes and giving these the current
attributes makes no sense and can even give weird side effects. In these cases,
the attributes are inherited from neighbouring nodes.

\stopsubsection

\startsubsection[title={Attribute registers}]

\topicindex {attributes}

Attributes are a completely new concept in \LUATEX. Syntactically, they behave a
lot like counters: attributes obey \TEX's nesting stack and can be used after
\prm {the} etc.\ just like the normal \prm {count} registers.

\startsyntax
\attribute <16-bit number> <optional equals> <32-bit number>!crlf
\attributedef <csname> <optional equals> <16-bit number>
\stopsyntax

Conceptually, an attribute is either \quote {set} or \quote {unset}. Unset
attributes have a special negative value to indicate that they are unset, that
value is the lowest legal value: \type {-"7FFFFFFF} in hexadecimal, a.k.a.
$-2147483647$ in decimal. It follows that the value \type {-"7FFFFFFF} cannot be
used as a legal attribute value, but you {\it can\/} assign \type {-"7FFFFFFF} to
\quote {unset} an attribute. All attributes start out in this \quote {unset}
state in \INITEX.

Attributes can be used as extra counter values, but their usefulness comes mostly
from the fact that the numbers and values of all \quote {set} attributes are
attached to all nodes created in their scope. These can then be queried from any
\LUA\ code that deals with node processing. Further information about how to use
attributes for node list processing from \LUA\ is given in~\in {chapter}[nodes].

Attributes are stored in a sorted (sparse) linked list that are shared when
possible. This permits efficient testing and updating. You can define many
thousands of attributes but normally such a large number makes no sense and is
also not that efficient because each node carries a (possibly shared) link to a
list of currently set attributes. But they are a convenient extension and one of
the first extensions we implemented in \LUATEX.

In \LUAMETATEX\ we try to minimize the memory footprint and creation of these
attribute lists more aggressive sharing them. This feature is still somewhat
experimental.

\stopsubsection

\startsubsection[title={Box attributes}]

\topicindex {attributes}
\topicindex {boxes}
\topicindex {vcentering}

Nodes typically receive the list of attributes that is in effect when they are
created. This moment can be quite asynchronous. For example: in paragraph
building, the individual line boxes are created after the \prm {par} command has
been processed, so they will receive the list of attributes that is in effect
then, not the attributes that were in effect in, say, the first or third line of
the paragraph.

Similar situations happen in \LUATEX\ regularly. A few of the more obvious
problematic cases are dealt with: the attributes for nodes that are created
during hyphenation, kerning and ligaturing borrow their attributes from their
surrounding glyphs, and it is possible to influence box attributes directly.

When you assemble a box in a register, the attributes of the nodes contained in
the box are unchanged when such a box is placed, unboxed, or copied. In this
respect attributes act the same as characters that have been converted to
references to glyphs in fonts. For instance, when you use attributes to implement
color support, each node carries information about its eventual color. In that
case, unless you implement mechanisms that deal with it, applying a color to
already boxed material will have no effect. Keep in mind that this
incompatibility is mostly due to the fact that separate specials and literals are
a more unnatural approach to colors than attributes.

It is possible to fine|-|tune the list of attributes that are applied to a \type
{hbox}, \type {vbox} or \type {vtop} by the use of the keyword \type {attr}. The
\type {attr} keyword(s) should come before a \type {to} or \type {spread}, if
that is also specified. An example is:

\startbuffer[tex]
\attribute997=123
\attribute998=456
\setbox0=\hbox {Hello}
\setbox2=\hbox attr 999 = 789 attr 998 = -"7FFFFFFF{Hello}
\stopbuffer

\startbuffer[lua]
  for b=0,2,2 do
    for a=997, 999 do
      tex.sprint("box ", b, " : attr ",a," : ",tostring(tex.box[b]     [a]))
      tex.sprint("\\quad\\quad")
      tex.sprint("list ",b, " : attr ",a," : ",tostring(tex.box[b].list[a]))
      tex.sprint("\\par")
    end
  end
\stopbuffer

\typebuffer[tex]

Box 0 now has attributes 997 and 998 set while box 2 has attributes 997 and 999
set while the nodes inside that box will all have attributes 997 and 998 set.
Assigning the maximum negative value causes an attribute to be ignored.

To give you an idea of what this means at the \LUA\ end, take the following
code:

\typebuffer[lua]

Later we will see that you can access properties of a node. The boxes here are so
called \nod {hlist} nodes that have a field \type {list} that points to the
content. Because the attributes are a list themselves you can access them by
indexing the node (here we do that with \type {[a]}). Running this snippet gives:

\start
    \getbuffer[tex]
    \startpacked \tt
        \ctxluabuffer[lua]
    \stoppacked
\stop

Because some values are not set we need to apply the \type {tostring} function
here so that we get the word \type {nil}.

A special kind of box is \prm {vcenter}. This one also can have attributes. When
one or more are set these plus the currently set attributes are bound to the
resulting box. In regular \TEX\ these centered boxes are only permitted in math
mode, but in \LUAMETATEX\ there is no error message and the box the height and
depth are equally divided. Of course in text mode there is no math axis related
offset applied.

It is possible to change or add to the attributes assigned to a box:

\starttyping
\boxattr 0 123 456
\stoptyping

\stopsubsection

\stopsection

\startsection[title={\LUA\ related primitives}]

\startsubsection[title={\prm {directlua}}]

In order to merge \LUA\ code with \TEX\ input, a few new primitives are needed.
The primitive \prm {directlua} is used to execute \LUA\ code immediately. The
syntax is

\startsyntax
\directlua <general text>
\stopsyntax

The \syntax {<general text>} is expanded fully, and then fed into the \LUA\
interpreter. After reading and expansion has been applied to the \syntax
{<general text>}, the resulting token list is converted to a string as if it was
displayed using \type {\the\toks}. On the \LUA\ side, each \prm {directlua} block
is treated as a separate chunk. In such a chunk you can use the \type {local}
directive to keep your variables from interfering with those used by the macro
package.

The conversion to and from a token list means that you normally can not use \LUA\
line comments (starting with \type {--}) within the argument. As there typically
will be only one \quote {line} the first line comment will run on until the end
of the input. You will either need to use \TEX|-|style line comments (starting
with \%), or change the \TEX\ category codes locally. Another possibility is to
say:

\starttyping
\begingroup
\endlinechar=10
\directlua ...
\endgroup
\stoptyping

Then \LUA\ line comments can be used, since \TEX\ does not replace line endings
with spaces. Of course such an approach depends on the macro package that you
use.

The \prm {directlua} command is expandable. Since it passes \LUA\ code to the
\LUA\ interpreter its expansion from the \TEX\ viewpoint is usually empty.
However, there are some \LUA\ functions that produce material to be read by \TEX,
the so called print functions. The most simple use of these is \type
{tex.print(<string> s)}. The characters of the string \type {s} will be placed on
the \TEX\ input buffer, that is, \quote {before \TEX's eyes} to be read by \TEX\
immediately. For example:

\startbuffer
\count10=20
a\directlua{tex.print(tex.count[10]+5)}b
\stopbuffer

\typebuffer

expands to

\getbuffer

Here is another example:

\startbuffer
$\pi = \directlua{tex.print(math.pi)}$
\stopbuffer

\typebuffer

will result in

\getbuffer

Note that the expansion of \prm {directlua} is a sequence of characters, not of
tokens, contrary to all \TEX\ commands. So formally speaking its expansion is
null, but it collects material in a new level on the input stack to be
immediately read by \TEX\ after the \LUA\ call as finished. It is a bit like
\ETEX's \prm {scantokens}, which now uses the same mechanism. For a description
of print functions look at \in {section} [sec:luaprint].

Because the \syntax {<general text>} is a chunk, the normal \LUA\ error handling
is triggered if there is a problem in the included code. The \LUA\ error messages
should be clear enough, but the contextual information is often suboptimal
because it can come from deep down, and \TEX\ has no knowledge about what you do
in \LUA. Often, you will only see the line number of the right brace at the end
of the code.

While on the subject of errors: some of the things you can do inside \LUA\ code
can break up \LUAMETATEX\ pretty bad. If you are not careful while working with
the node list interface, you may even end up with errors or even crashes from
within the \TEX\ portion of the executable.

\stopsubsection

\startsubsection[title={\lpr {luaescapestring}}]

\topicindex {escaping}

This primitive converts a \TEX\ token sequence so that it can be safely used as
the contents of a \LUA\ string: embedded backslashes, double and single quotes,
and newlines and carriage returns are escaped. This is done by prepending an
extra token consisting of a backslash with category code~12, and for the line
endings, converting them to \type {n} and \type {r} respectively. The token
sequence is fully expanded.

\startsyntax
\luaescapestring <general text>
\stopsyntax

Most often, this command is not actually the best way to deal with the
differences between \TEX\ and \LUA. In very short bits of \LUA\ code it is often
not needed, and for longer stretches of \LUA\ code it is easier to keep the code
in a separate file and load it using \LUA's \type {dofile}:

\starttyping
\directlua { dofile("mysetups.lua") }
\stoptyping

\stopsubsection

\startsubsection[title={\lpr {luafunction}, \lpr {luafunctioncall} and \lpr {luadef}}]

The \prm {directlua} commands involves tokenization of its argument (after
picking up an optional name or number specification). The tokenlist is then
converted into a string and given to \LUA\ to turn into a function that is
called. The overhead is rather small but when you have millions of calls it can
have some impact. For this reason there is a variant call available: \lpr
{luafunction}. This command is used as follows:

\starttyping
\directlua {
    local t = lua.get_functions_table()
    t[1] = function() tex.print("!") end
    t[2] = function() tex.print("?") end
}

\luafunction1
\luafunction2
\stoptyping

Of course the functions can also be defined in a separate file. There is no limit
on the number of functions apart from normal \LUA\ limitations. Of course there
is the limitation of no arguments but that would involve parsing and thereby give
no gain. The function, when called in fact gets one argument, being the index, so
in the following example the number \type {8} gets typeset.

\starttyping
\directlua {
    local t = lua.get_functions_table()
    t[8] = function(slot) tex.print(slot) end
}
\stoptyping

The \lpr {luafunctioncall} primitive does the same but is unexpandable, for
instance in an \prm {edef}. In addition \LUATEX\ provides a definer:

\starttyping
                 \luadef\MyFunctionA 1
          \global\luadef\MyFunctionB 2
\protected\global\luadef\MyFunctionC 3
\stoptyping

You should really use these commands with care. Some references get stored in
tokens and assume that the function is available when that token expands. On the
other hand, as we have tested this functionality in relative complex situations
normal usage should not give problems.

{\em It makes sense to delegate the implementation of the primitives to \LUA.}

\stopsubsection

\startsubsection[title={\lpr {luabytecode} and \lpr {luabytecodecall}}]

Analogue to the function callers discussed in the previous section we have byte
code callers. Again the call variant is unexpandable.

\starttyping
\directlua {
    lua.bytecode[9998] = function(s)
        tex.sprint(s*token.scan_int())
    end
    lua.bytecode[5555] = function(s)
        tex.sprint(s*token.scan_dimen())
    end
}
\stoptyping

This works with:

\starttyping
\luabytecode    9998 5  \luabytecode    5555 5sp
\luabytecodecall9998 5  \luabytecodecall5555 5sp
\stoptyping

The variable \type {s} in the code is the number of the byte code register that
can be used for diagnostic purposes. The advantage of bytecode registers over
function calls is that they are stored in the format (but without upvalues).

{\em It makes sense to delegate the implementation of the primitives to \LUA.}

\stopsubsection

\stopsection

\startsection[title={Catcode tables}]

\startsubsection[title={Catcodes}]

\topicindex {catcodes}

Catcode tables are a new feature that allows you to switch to a predefined
catcode regime in a single statement. You can have lots of different tables, but
if you need a dozen you might wonder what you're doing. This subsystem is
backward compatible: if you never use the following commands, your document will
not notice any difference in behaviour compared to traditional \TEX. The contents
of each catcode table is independent from any other catcode table, and its
contents is stored and retrieved from the format file.

\stopsubsection

\startsubsection[title={\lpr {catcodetable}}]

The primitive \lpr {catcodetable} switches to a different catcode table. Such a
table has to be previously created using one of the two primitives below, or it
has to be zero. Table zero is initialized by \INITEX.

\startsyntax
\catcodetable <15-bit number>
\stopsyntax

\stopsubsection

\startsubsection[title={\lpr {initcatcodetable}}]

\startsyntax
\initcatcodetable <15-bit number>
\stopsyntax

The primitive \lpr {initcatcodetable} creates a new table with catcodes
identical to those defined by \INITEX. The new catcode table is allocated
globally: it will not go away after the current group has ended. If the supplied
number is identical to the currently active table, an error is raised. The
initial values are:

\starttabulate[|c|c|l|l|]
\DB catcode \BC character               \BC equivalent \BC category          \NC \NR
\TB
\NC  0 \NC \tttf \letterbackslash       \NC         \NC \type {escape}       \NC \NR
\NC  5 \NC \tttf \letterhat\letterhat M \NC return  \NC \type {car_ret}      \NC \NR
\NC  9 \NC \tttf \letterhat\letterhat @ \NC null    \NC \type {ignore}       \NC \NR
\NC 10 \NC \tttf <space>                \NC space   \NC \type {spacer}       \NC \NR
\NC 11 \NC {\tttf a} \endash\ {\tttf z} \NC         \NC \type {letter}       \NC \NR
\NC 11 \NC {\tttf A} \endash\ {\tttf Z} \NC         \NC \type {letter}       \NC \NR
\NC 12 \NC everything else              \NC         \NC \type {other}        \NC \NR
\NC 14 \NC \tttf \letterpercent         \NC         \NC \type {comment}      \NC \NR
\NC 15 \NC \tttf \letterhat\letterhat ? \NC delete  \NC \type {invalid_char} \NC \NR
\LL
\stoptabulate

\stopsubsection

\startsubsection[title={\lpr {savecatcodetable}}]

\startsyntax
\savecatcodetable <15-bit number>
\stopsyntax

\lpr {savecatcodetable} copies the current set of catcodes to a new table with
the requested number. The definitions in this new table are all treated as if
they were made in the outermost level. Again, the new table is allocated globally:
it will not go away after the current group has ended. If the supplied number is
the currently active table, an error is raised.

\stopsubsection

\stopsection

\startsection[title={Tokens, commands and strings}]

\startsubsection[title={\lpr {scantextokens} and \lpr {tokenized}}]

\topicindex {tokens+scanning}

The syntax of \lpr {scantextokens} is identical to \prm {scantokens}. This
primitive is a slightly adapted version of \ETEX's \prm {scantokens}. The
differences are:

\startitemize
\startitem
    The last (and usually only) line does not have a \prm {endlinechar} appended.
\stopitem
\startitem
    \lpr {scantextokens} never raises an EOF error, and it does not execute
    \prm {everyeof} tokens.
\stopitem
\startitem
    There are no \quote {\unknown\ while end of file \unknown} error tests
    executed. This allows the expansion to end on a different grouping level or
    while a conditional is still incomplete.
\stopitem
\stopitemize

The implementation in \LUAMETATEX\ is different in the sense that it uses the same
methods as printing from \LUA\ to \TEX\ does. Therefore, in addition to the two
commands we also have this expandable command:

\startsyntax
\tokenized {...}
\tokenized catcodetable <number> {...}
\stopsyntax

The \ETEX\ command \type {\tracingscantokens} has been dropped in the process as
that was interwoven with the old code.

\stopsubsection

\startsubsection[title={\lpr {toksapp}, \lpr {tokspre}, \lpr {etoksapp}, \lpr {etokspre},
\lpr {gtoksapp}, \lpr {gtokspre}, \lpr {xtoksapp},  \lpr {xtokspre}}]

Instead of:

\starttyping
\toks0\expandafter{\the\toks0 foo}
\stoptyping

you can use:

\starttyping
\etoksapp0{foo}
\stoptyping

The \type {pre} variants prepend instead of append, and the \type {e} variants
expand the passed general text. The \type {g} and \type {x} variants are global.

\stopsubsection

\startsubsection[title={\prm {csstring}, \lpr {begincsname} and \lpr {lastnamedcs}}]

These are somewhat special. The \prm {csstring} primitive is like
\prm {string} but it omits the leading escape character. This can be
somewhat more efficient than stripping it afterwards.

The \lpr {begincsname} primitive is like \prm {csname} but doesn't create
a relaxed equivalent when there is no such name. It is equivalent to

\starttyping
\ifcsname foo\endcsname
  \csname foo\endcsname
\fi
\stoptyping

The advantage is that it saves a lookup (don't expect much speedup) but more
important is that it avoids using the \prm {if} test. The \lpr {lastnamedcs}
is one that should be used with care. The above example could be written as:

\starttyping
\ifcsname foo\endcsname
  \lastnamedcs
\fi
\stoptyping

This is slightly more efficient than constructing the string twice (deep down in
\LUATEX\ this also involves some \UTF8 juggling), but probably more relevant is
that it saves a few tokens and can make code a bit more readable.

\stopsubsection

\startsubsection[title={\lpr {clearmarks}}]

\topicindex {marks}

This primitive complements the \ETEX\ mark primitives and clears a mark class
completely, resetting all three connected mark texts to empty. It is an
immediate command (no synchronization node is used).

\startsyntax
\clearmarks <16-bit number>
\stopsyntax

\stopsubsection

\startsubsection[title={\lpr {alignmark} and \lpr {aligntab}}]

The primitive \lpr {alignmark} duplicates the functionality of \type {#} inside
alignment preambles, while \lpr {aligntab} duplicates the functionality of \type
{&}.

\stopsubsection

\startsubsection[title={\lpr {letcharcode}}]

This primitive can be used to assign a meaning to an active character, as in:

\starttyping
\def\foo{bar} \letcharcode123=\foo
\stoptyping

This can be a bit nicer than using the uppercase tricks (using the property of
\prm {uppercase} that it treats active characters special).

\stopsubsection

\startsubsection[title={\lpr {glet}}]

This primitive is similar to:

\starttyping
\protected\def\glet{\global\let}
\stoptyping

but faster (only measurable with millions of calls) and probably more convenient
(after all we also have \type {\gdef}).

\stopsubsection

\startsubsection[title={\lpr {defcsname}, \lpr {edefcsname}, \lpr {edefcsname} and \lpr {xdefcsname}}]

Although we can implement these primitives easily using macros it makes sense,
given the popularity of \prm {csname} to have these as primitives. It also saves
some \prm {expandafter} usage and it looks a bit better in the source.

\starttyping
\def\gdefcsname foo\endcsname{oof}
\stoptyping

\stopsubsection

\startsubsection[title={\lpr {expanded}}]

\topicindex {expansion}

The \lpr {expanded} primitive takes a token list and expands its content which
can come in handy: it avoids a tricky mix of \prm {expandafter} and \prm
{noexpand}. You can compare it with what happens inside the body of an \prm
{edef}. The \tex {immediateassignment} and \tex {immediateassigned} commands are
gone because we have the more powerful local control commands. They are a tad
slower but this mechanism isn't used that much anyway. Inside an \prm {edef} you
can use the \type {\immediate} prefix anyway, so if you really want these
primitives back you can say:

\starttyping
\let\immediateassignment\immediate
\let\immediateassigned  \localcontrolled
\stoptyping

\stopsubsection

% \startsubsection[title={\lpr {expanded}, \lpr {immediateassignment} and \lpr {immediateassigned}}]
%
% \topicindex {expansion}
%
% The \lpr {expanded} primitive takes a token list and expands its content which can
% come in handy: it avoids a tricky mix of \prm {expandafter} and \prm {noexpand}.
% You can compare it with what happens inside the body of an \prm {edef}. But this
% kind of expansion still doesn't expand some primitive operations.
%
% \startbuffer
% \newcount\NumberOfCalls
%
% \def\TestMe{\advance\NumberOfCalls1 }
%
% \edef\Tested{\TestMe foo:\the\NumberOfCalls}
% \edef\Tested{\TestMe foo:\the\NumberOfCalls}
% \edef\Tested{\TestMe foo:\the\NumberOfCalls}
%
% \meaning\Tested
% \stopbuffer
%
% \typebuffer
%
% The result is a macro that has the not expanded code in its body:
%
% \getbuffer
%
% Instead we can define \tex {TestMe} in a way that expands the assignment
% immediately. You need of course to be aware of preventing look ahead interference
% by using a space or \tex {relax} (often an expression works better as it doesn't
% leave an \tex {relax}).
%
% \startbuffer
% \def\TestMe{\immediateassignment\advance\NumberOfCalls1 }
%
% \edef\Tested{\TestMe foo:\the\NumberOfCalls}
% \edef\Tested{\TestMe foo:\the\NumberOfCalls}
% \edef\Tested{\TestMe foo:\the\NumberOfCalls}
%
% \meaning\Tested
% \stopbuffer
%
% \typebuffer
%
% This time the counter gets updates and we don't see interference in the
% resulting \tex {Tested} macro:
%
% \getbuffer
%
% Here is a somewhat silly example of expanded comparison:
%
% \startbuffer
% \def\expandeddoifelse#1#2#3#4%
%   {\immediateassignment\edef\tempa{#1}%
%    \immediateassignment\edef\tempb{#2}%
%    \ifx\tempa\tempb
%      \immediateassignment\def\next{#3}%
%    \else
%      \immediateassignment\def\next{#4}%
%    \fi
%    \next}
%
% \edef\Tested
%   {(\expandeddoifelse{abc}{def}{yes}{nop}/%
%     \expandeddoifelse{abc}{abc}{yes}{nop})}
%
% \meaning\Tested
% \stopbuffer
%
% \typebuffer
%
% It gives:
%
% \getbuffer
%
% A variant is:
%
% \starttyping
% \def\expandeddoifelse#1#2#3#4%
%   {\immediateassigned{
%      \edef\tempa{#1}%
%      \edef\tempb{#2}%
%    }%
%    \ifx\tempa\tempb
%      \immediateassignment\def\next{#3}%
%    \else
%      \immediateassignment\def\next{#4}%
%    \fi
%    \next}
% \stoptyping
%
% The possible error messages are the same as using assignments in preambles of
% alignments and after the \prm {accent} command. The supported assignments are the
% so called prefixed commands (except box assignments).
%
% \stopsubsection

\startsubsection[title={\lpr {ignorepars}}]

This primitive is like \prm {ignorespaces} but also skips paragraph ending
commands (normally \prm {par} and empty lines).

\stopsubsection

\startsubsection[title={\lpr {futureexpand}, \lpr {futureexpandis}, \lpr {futureexpandisap}}]

These commands are used as:

\starttyping
\futureexpand\sometoken\whenfound\whennotfound
\stoptyping

When there is no match and a space was gobbled a space will be put back. The
\type {is} variant doesn't do that while the \type {isap} even skips \type
{\pars}, These characters stand for \quote {ignorespaces} and \quote
{ignorespacesandpars}.

\stopsubsection

\startsubsection[title={\lpr {aftergrouped}}]

There is a new experimental feature that can inject multiple tokens to after the group
ends. An example demonstrate its use:

\startbuffer
{
    \aftergroup A \aftergroup B \aftergroup C
test 1 : }

{
    \aftergrouped{What comes next 1}
    \aftergrouped{What comes next 2}
    \aftergrouped{What comes next 3}
test 2 : }


{
    \aftergroup A \aftergrouped{What comes next 1}
    \aftergroup B \aftergrouped{What comes next 2}
    \aftergroup C \aftergrouped{What comes next 3}
test 3 : }

{
    \aftergrouped{What comes next 1} \aftergroup A
    \aftergrouped{What comes next 2} \aftergroup B
    \aftergrouped{What comes next 3} \aftergroup C
test 4 : }
\stopbuffer

\typebuffer

This gives:

\startpacked\getbuffer\stoppacked

\stopsubsection

\stopsection

\startsection[title=Conditions]

\startsubsection[title={\lpr{ifabsnum} and \lpr {ifabsdim}}]

There are two tests that we took from \PDFTEX:

\startbuffer
\ifabsnum -10 = 10
    the same number
\fi
\ifabsdim -10pt = 10pt
    the same dimension
\fi
\stopbuffer

\typebuffer

This gives

\blank {\tt \getbuffer} \blank

\stopsubsection

\startsubsection[title={\lpr{ifcmpnum}, \lpr {ifcmpdim}, \lpr {ifnumval}, \lpr
{ifdimval}, \lpr {ifchknum} and \lpr {ifchkdim}}]

\topicindex {conditions+numbers}
\topicindex {conditions+dimensions}
\topicindex {numbers}
\topicindex {dimensions}

New are the ones that compare two numbers or dimensions:

\startbuffer
\ifcmpnum 5 8 less \or equal \else more \fi
\ifcmpnum 5 5 less \or equal \else more \fi
\ifcmpnum 8 5 less \or equal \else more \fi
\stopbuffer

\typebuffer \blank {\tt \getbuffer} \blank

and

\startbuffer
\ifcmpdim 5pt 8pt less \or equal \else more \fi
\ifcmpdim 5pt 5pt less \or equal \else more \fi
\ifcmpdim 8pt 5pt less \or equal \else more \fi
\stopbuffer

\typebuffer \blank {\tt \getbuffer} \blank

There are also some number and dimension tests. All four expose the \type {\else}
branch when there is an error, but two also report if the number is less, equal
or more than zero.

\startbuffer
\ifnumval  -123  \or < \or = \or > \or ! \else ? \fi
\ifnumval     0  \or < \or = \or > \or ! \else ? \fi
\ifnumval   123  \or < \or = \or > \or ! \else ? \fi
\ifnumval   abc  \or < \or = \or > \or ! \else ? \fi

\ifdimval -123pt \or < \or = \or > \or ! \else ? \fi
\ifdimval    0pt \or < \or = \or > \or ! \else ? \fi
\ifdimval  123pt \or < \or = \or > \or ! \else ? \fi
\ifdimval  abcpt \or < \or = \or > \or ! \else ? \fi
\stopbuffer

\typebuffer \blank {\tt \getbuffer} \blank

\startbuffer
\ifchknum  -123  \or okay \else bad \fi
\ifchknum     0  \or okay \else bad \fi
\ifchknum   123  \or okay \else bad \fi
\ifchknum   abc  \or okay \else bad \fi

\ifchkdim -123pt \or okay \else bad \fi
\ifchkdim    0pt \or okay \else bad \fi
\ifchkdim  123pt \or okay \else bad \fi
\ifchkdim  abcpt \or okay \else bad \fi
\stopbuffer

\typebuffer \blank {\tt \getbuffer} \blank

The last checked values are available in \lpr {lastchknum} and \lpr {lastchkdim}.
These don't obey grouping.

\stopsubsection

\startsubsection[title={\lpr {ifmathstyle} and \lpr {ifmathparameter}}]

These two are variants on \prm {ifcase} where the first one operates with values
in ranging from zero (display style) to seven (cramped script script style) and
the second one can have three values: a parameter is zero, has a value or is
unset. The \type {\ifmathparameter} primitive takes a proper parameter name and a
valid style identifier (a primitive identifier or number). The \type
{\ifmathstyle} primitive is equivalent to \type {\ifcase \mathstyle}.

\stopsubsection

\startsubsection[title={\lpr {ifempty}}]

This primitive tests for the following token (control sequence) having no
content. Assuming that \type {\empty} is indeed empty, the following two are
equivalent:

\starttyping
\ifempty\whatever
\ifx\whatever\empty
\stoptyping

There is no real performance gain here, it's more one of these extensions that
lead to less clutter in tracing.

\stopsubsection

\startsubsection[title={\lpr {ifrelax}}]

This primitive complements \type {\ifdefined}, \type {\ifempty} and \type
{\ifcsname} so that we have all reasonable tests directly available.

\stopsubsection

\startsubsection[title={\lpr {ifboolean}}]

This primitive tests for non|-|zero, so the next variants are similar

\starttyping
       \ifcase   <integer>.F.\else .T.\fi
\unless\ifcase   <integer>.T.\else .F.\fi
       \ifboolean<integer>.T.\else .F.\fi
\stoptyping

\stopsubsection

\startsubsection[title={\lpr {iftok} and \lpr {ifcstok}}]

\topicindex {conditions+tokens}
\topicindex {tokens}

Comparing tokens and macros can be done with \type {\ifx}. Two extra test are
provided in \LUAMETATEX:

\startbuffer
\def\ABC{abc} \def\DEF{def} \def\PQR{abc} \newtoks\XYZ \XYZ {abc}

\iftok{abc}{def}\relax  (same) \else [different] \fi
\iftok{abc}{abc}\relax  [same] \else (different) \fi
\iftok\XYZ {abc}\relax  [same] \else (different) \fi

\ifcstok\ABC \DEF\relax (same) \else [different] \fi
\ifcstok\ABC \PQR\relax [same] \else (different) \fi
\ifcstok{abc}\ABC\relax [same] \else (different) \fi
\stopbuffer

\typebuffer \startpacked[blank] {\tt\nospacing\getbuffer} \stoppacked

You can check if a macro is defined as protected with \type {\ifprotected} while
frozen macros can be tested with \type {\iffrozen}. A provisional \type
{\ifusercmd} tests will check if a command is defined at the user level (and this
one might evolve).

\stopsubsection

\startsubsection[title={\lpr {ifarguments}, \lpr {ifparameters} and \lpr {ifparameter}}]

These are part of the extended macro argument parsing features. The \type
{\ifarguments} condition is like an \type {\ifcase} where the number is the
picked up number of arguments. The number reflects the {\em last} count, so
successive macro expansions will adapt the value. The \type {\ifparameters}
counts till the first empty parameter and the \type {\ifparameter} (singular)
takes a parameter reference (like \type {#2}) and again is an \type {\ifcase}
where zero means a bad reference, one a non|-|empty argument and two an empty
one. A typical usage is:

\starttyping
\def\foo#1#2%
  {\ifparameter#1\or one\fi
   \ifparameter#2\or two\fi}
\stoptyping

No expansion of arguments takes place here but you can use a test like this:

\starttyping
\def\foo#1#2%
  {\iftok{#1}{}\else one\fi
   \iftok{#2}{}\else two\fi}
\stoptyping

\stopsubsection

\startsubsection[title={\lpr {ifcondition}}]

\topicindex {conditions}

This is a somewhat special one. When you write macros conditions need to be
properly balanced in order to let \TEX's fast branch skipping work well. This new
primitive is basically a no||op flagged as a condition so that the scanner can
recognize it as an if|-|test. However, when a real test takes place the work is
done by what follows, in the next example \tex {something}.

\starttyping
\unexpanded\def\something#1#2%
  {\edef\tempa{#1}%
   \edef\tempb{#2}
   \ifx\tempa\tempb}

\ifcondition\something{a}{b}%
    \ifcondition\something{a}{a}%
        true 1
    \else
        false 1
    \fi
\else
    \ifcondition\something{a}{a}%
        true 2
    \else
        false 2
    \fi
\fi
\stoptyping

If you are familiar with \METAPOST, this is a bit like \type {vardef} where the macro
has a return value. Here the return value is a test.

Experiments with something \type {\ifdef} actually worked ok but were rejected
because in the end it gave no advantage so this generic one has to do. The \type
{\ifcondition} test is basically is a no|-|op except when branches are skipped.
However, when a test is expected, the scanner gobbles it and the next test result
is used. Here is an other example:

\startbuffer
\def\mytest#1%
  {\ifabsdim#1>0pt\else
     \expandafter \unless
   \fi
   \iftrue}

\ifcondition\mytest{10pt}\relax non-zero \else zero \fi
\ifcondition\mytest {0pt}\relax non-zero \else zero \fi
\stopbuffer

\typebuffer \blank {\tt \getbuffer} \blank

The last expansion in a macro like \type {\mytest} has to be a condition and here
we use \type {\unless} to negate the result.

\stopsubsection

\startsubsection[title={\lpr {orelse} and \lpr {orunless}}]

Sometimes you have successive tests that, when laid out in the source lead to
deep trees. The \type {\ifcase} test is an exception. Experiments with \type
{\ifcasex} worked out fine but eventually were rejected because we have many
tests so it would add a lot. As \LUAMETATEX\ permitted more experiments,
eventually an alternative was cooked up, one that has some restrictions but is
relative lightweight. It goes like this:

\starttyping
\ifnum\count0<10
    less
\orelse\ifnum\count0=10
    equal
\else
    more
\fi
\stoptyping

The \type {\orelse} has to be followed by one of the if test commands, except
\type {\ifcondition}, and there can be an \type {\unless} in front of such a
command. These restrictions make it possible to stay in the current condition
(read: at the same level). If you need something more complex, using \type
{\orelse} is probably unwise anyway. In case you wonder about performance, there
is a little more checking needed when skipping branches but that can be
neglected. There is some gain due to staying at the same level but that is only
measurable when you runs tens of millions of complex tests and in that case it is
very likely to drown in the real action. It's a convenience mechanism, in the
sense that it can make your code look a bit easier to follow.

There is a nice side effect of this mechanism. When you define:

\starttyping
\def\quitcondition{\orelse\iffalse}
\stoptyping

you can do this:

\starttyping
\ifnum\count0<10
    less
\orelse\ifnum\count0=10
    equal
    \quitcondition
    indeed
\else
    more
\fi
\stoptyping

Of course it is only useful at the right level, so you might end up with cases like

\starttyping
\ifnum\count0<10
    less
\orelse\ifnum\count0=10
    equal
    \ifnum\count2=30
        \expandafter\quitcondition
    \fi
    indeed
\else
    more
\fi
\stoptyping

The \lpr {orunless} variant negates the next test, just like \prm {unless}. In
some cases these commands look at the next token to see if it is an if|-|test so
a following negation will not work (read: making that work would complicate the
code and hurt efficiency too). Side note: interesting is that in \CONTEXT\ we
hardly use this kind of negation.

\stopsubsection

\startsubsection[title={\lpr {ifprotected}, \lpr {frozen}, \lpr {iffrozen} and \lpr {ifusercmd}}]

These checkers deal with control sequences. You can check if a command is a
protected one, that is, defined with the \type {\protected} prefix. A command is
frozen when it has been defined with the \type {\frozen} prefix. Beware: only
macros can be frozen. A user command is a command that is not part of the
predefined set of commands. This is an experimental command.

\stopsubsection

\stopsection

\startsection[title={Boxes, rules and leaders}]

\startsubsection[title={\lpr {outputbox}}]

\topicindex {output}

This integer parameter allows you to alter the number of the box that will be
used to store the page sent to the output routine. Its default value is 255, and
the acceptable range is from 0 to 65535.

\startsyntax
\outputbox = 12345
\stopsyntax

\stopsubsection

\startsubsection[title={\prm {hrule}, \prm {vrule}, \lpr {nohrule} and \lpr {novrule}}]

\topicindex {rules}

Both rule drawing commands take an optional \type {xoffset} and \type {yoffset}
parameter. The displacement is virtual and not taken into account when the
dimensions are calculated.

Two new primitives were introduced: \lpr {nohrule} and \lpr {novrule}. These can
be used to reserve space. This is often more efficient than creating an empty box
with fake dimensions. Of course this assumes that the backend implements them
being invisible but still taking space.

\stopsubsection

\startsubsection[title={\prm {vsplit}}]

\topicindex {splitting}

The \prm {vsplit} primitive has to be followed by a specification of the required
height. As alternative for the \type {to} keyword you can use \type {upto} to get
a split of the given size but result has the natural dimensions then.

\stopsubsection

\startsubsection[title={Images and reused box objects},reference=sec:imagesandforms]

In original \TEX\ image support is dealt with via specials. It's not a native
feature of the engine. All that \TEX\ cares about is dimensions, so in practice
that meant: using a box with known dimensions that wraps a special that instructs
the backend to include an image. The wrapping is needed because a special itself
is a whatsit and as such has no dimensions.

In \PDFTEX\ a special whatsit for images was introduced and that one {\em has}
dimensions. As a consequence, in several places where the engine deals with the
dimensions of nodes, it now has to check the details of whatsits. By inheriting
code from \PDFTEX, the \LUATEX\ engine also had that property. However, at some
point this approach was abandoned and a more natural trick was used: images (and
box resources) became a special kind of rules, and as rules already have
dimensions, the code could be simplified.

When direction nodes and (formerly local) par nodes also became first class
nodes, whatsits again became just that: nodes representing whatever you want, but
without dimensions, and therefore they could again be ignored when dimensions
mattered. And, because images were disguised as rules, as mentioned, their
dimensions automatically were taken into account. This separation between front
and backend cleaned up the code base already quite a bit.

In \LUAMETATEX\ we still have the image specific subtypes for rules, but the
engine never looks at subtypes of rules. That was up to the backend. This means
that image support is not present in \LUAMETATEX. When an image specification was
parsed the special properties, like the filename, or additional attributes, were
stored in the backend and all that \LUATEX\ does is registering a reference to an
image's specification in the rule node. But, having no backend means nothing is
stored, which in turn would make the image inclusion primitives kind of weird.

Therefore you need to realize that contrary to \LUATEX, {\em in \LUAMETATEX\
support for images and box reuse is not built in}! However, we can assume that
an implementation uses rules in a similar fashion as \LUATEX\ does. So, you can
still consider images and box reuse to be core concepts. Here we just mention the
primitives that \LUATEX\ provides. They are not available in the engine but can
of course be implemented in \LUA.

\starttabulate[|l|p|]
\DB command \BC explanation \NC \NR
\TB
\NC \lpr {saveboxresource}             \NC save the box as an object to be included later \NC \NR
\NC \lpr {saveimageresource}           \NC save the image as an object to be included later \NC \NR
\NC \lpr {useboxresource}              \NC include the saved box object here (by index) \NC \NR
\NC \lpr {useimageresource}            \NC include the saved image object here (by index) \NC \NR
\NC \lpr {lastsavedboxresourceindex}   \NC the index of the last saved box object \NC \NR
\NC \lpr {lastsavedimageresourceindex} \NC the index of the last saved image object \NC \NR
\NC \lpr {lastsavedimageresourcepages} \NC the number of pages in the last saved image object \NC \NR
\LL
\stoptabulate

An implementation probably should accept the usual optional dimension parameters
for \type {\use...resource} in the same format as for rules. With images, these
dimensions are then used instead of the ones given to \lpr {useimageresource} but
the original dimensions are not overwritten, so that a \lpr {useimageresource}
without dimensions still provides the image with dimensions defined by \lpr
{saveimageresource}. These optional parameters are not implemented for \lpr
{saveboxresource}.

\starttyping
\useimageresource width 20mm height 10mm depth 5mm \lastsavedimageresourceindex
\useboxresource   width 20mm height 10mm depth 5mm \lastsavedboxresourceindex
\stoptyping

Examples or optional entries are \type {attr} and \type {resources} that accept a
token list, and the \type {type} key. When set to non|-|zero the \type {/Type}
entry is omitted. A value of 1 or 3 still writes a \type {/BBox}, while 2 or 3
will write a \type {/Matrix}. But, as said: this is entirely up to the backend.
Generic macro packages (like \type {tikz}) can use these assumed primitives so
one can best provide them. It is probably, for historic reasons, the only more or
less standardized image inclusion interface one can expect to work in all macro
packages.

\stopsubsection

\startsubsection[title={\lpr {hpack}, \lpr {vpack} and \lpr {tpack}}]

These three primitives are the equivalents of \prm {hbox}, \prm {vbox} and
\prm {vtop} but they don't trigger the packaging related callbacks. Of course
one never know if content needs a treatment so using them should be done with
care. Apart from accepting more keywords (and therefore options) the normal
box behave the same as before. The \prm {vcenter} builder also works in text
mode.

\stopsubsection

\startsubsection[title={\lpr {gleaders}},reference=sec:gleaders]

\topicindex {leaders}

This type of leaders is anchored to the origin of the box to be shipped out. So
they are like normal \prm {leaders} in that they align nicely, except that the
alignment is based on the {\it largest\/} enclosing box instead of the {\it
smallest\/}. The \type {g} stresses this global nature.

\stopsubsection

\stopsection

\startsection[title={Languages}]

\startsubsection[title={\lpr {hyphenationmin}}]

\topicindex {languages}
\topicindex {hyphenation}

This primitive can be used to set the minimal word length, so setting it to a value
of~$5$ means that only words of 6 characters and more will be hyphenated, of course
within the constraints of the \prm {lefthyphenmin} and \prm {righthyphenmin}
values (as stored in the glyph node). This primitive accepts a number and stores
the value with the language.

\stopsubsection

\startsubsection[title={\prm {boundary}, \prm {noboundary}, \prm {protrusionboundary} and \prm {wordboundary}}]

The \prm {noboundary} command is used to inject a whatsit node but now injects a normal
node with type \nod {boundary} and subtype~0. In addition you can say:

\starttyping
x\boundary 123\relax y
\stoptyping

This has the same effect but the subtype is now~1 and the value~123 is stored.
The traditional ligature builder still sees this as a cancel boundary directive
but at the \LUA\ end you can implement different behaviour. The added benefit of
passing this value is a side effect of the generalization. The subtypes~2 and~3
are used to control protrusion and word boundaries in hyphenation and have
related primitives.

\stopsubsection

\stopsection

\startsection[title={Control and debugging}]

\startsubsection[title={Tracing}]

\topicindex {tracing}

If \prm {tracingonline} is larger than~2, the node list display will also print
the node number of the nodes.

\stopsubsection

% \startsubsection[title={\lpr {lastnodetype}, \lpr {lastnodesubtype}, \lpr
% {currentiftype} and \lpr {internalcodesmode}.}]
%
% The \ETEX\ command \type {\lastnodetype} is limited to some nodes. When the
% parameter \type {\internalcodesmode} is set to a non|-|zero value the normal
% (internally used) numbers are reported. The same is true for \type
% {\currentiftype}, as we have more conditionals and also use a different order.
% The \type {\lastnodesubtype} is a bonus.
%
% \stopsubsection

\startsubsection[title={\lpr {lastnodetype}, \lpr {lastnodesubtype}, \lpr
{currentiftype}}]

The \ETEX\ command \type {\lastnodetype} returns the node codes as used in the
engine. You can query the numbers at the \LUA\ end if you need the actual values.
The parameter \type {\internalcodesmode} is no longer provided as compatibility
switch because \LUATEX\ has more cq. some different nodes and it makes no sense
to be incompatible with the \LUA\ end of the engine. The same is true for \type
{\currentiftype}, as we have more conditionals and also use a different order.
The \type {\lastnodesubtype} is a bonus and again reports the codes used
internally. During development these might occasionally change, but eventually
they will be stable.

\stopsubsection

\stopsection

\startsection[title={Files}]

\startsubsection[title={File syntax}]

\topicindex {files+names}

\LUAMETATEX\ will accept a braced argument as a file name:

\starttyping
\input {plain}
\openin 0 {plain}
\stoptyping

This allows for embedded spaces, without the need for double quotes. Macro
expansion takes place inside the argument. Keep in mind that as side effect of
delegating \IO\ to \LUA\ the \prm {openin} primitive is nor provided by the
engine and has to be implemented by the macro package. This also means that the
limit on the number of open files is not enforced by the engine.

The \lpr {tracingfonts} primitive that has been inherited from \PDFTEX\ has
been adapted to support variants in reporting the font. The reason for this
extension is that a csname not always makes sense. The zero case is the default.

\starttabulate[|l|l|]
\DB value \BC reported \NC \NR
\TB
\NC \type{0} \NC \type{\foo xyz} \NC \NR
\NC \type{1} \NC \type{\foo (bar)} \NC \NR
\NC \type{2} \NC \type{<bar> xyz} \NC \NR
\NC \type{3} \NC \type{<bar @ ..pt> xyz} \NC \NR
\NC \type{4} \NC \type{<id>} \NC \NR
\NC \type{5} \NC \type{<id: bar>} \NC \NR
\NC \type{6} \NC \type{<id: bar @ ..pt> xyz} \NC \NR
\LL
\stoptabulate

\stopsubsection

\startsubsection[title={Writing to file}]

\topicindex {files+writing}

Writing to a file in \TEX\ has two forms: delayed and immediate. Delayed writing
means that the to be written text is anchored in the node list and flushed by the
backend. As all \IO\ is delegated to \LUA, this also means that it has to deal
with distinction. In \LUATEX\ the number of open files was already bumped to 127,
but in \LUAMETATEX\ it depends on the macro package. The special meaning of
channel 18 was already dropped in \LUATEX\ because we have \type {os.execute}.

\stopsubsection

\stopsection

\startsection[title={Math}]

\topicindex {math}

We will cover math extensions in its own chapter because not only the font
subsystem and spacing model have been enhanced (thereby introducing many new
primitives) but also because some more control has been added to existing
functionality. Much of this relates to the different approaches of traditional
\TEX\ fonts and \OPENTYPE\ math.

\stopsection

\startsection[title={Fonts}]

\topicindex {fonts}

Like math, we will cover fonts extensions in its own chapter. Here we stick to
mentioning that loading fonts is different in \LUAMETATEX. As in \LUATEX\ we have
the extra primitives \type {\fontid} and \type {\setfontid}, \type {\noligs} and
\type {\nokerns}, and \type {\nospaces}. The other new primitives in \LUATEX\
have been dropped.

\stopsection

\startsection[title=Directions]

\topicindex {\OMEGA}
\topicindex {\ALEPH}
\topicindex {directions}

\startsubsection[title={Two directions}]

The directional model in \LUAMETATEX\ is a simplified version the the model used
in \LUATEX. In fact, not much is happening at all: we only register a change in
direction.

\stopsubsection

\startsubsection[title={How it works}]

The approach is that we try to make node lists balanced but also try to avoid
some side effects. What happens is quite intuitive if we forget about spaces
(turned into glue) but even there what happens makes sense if you look at it in
detail. However that logic makes in|-|group switching kind of useless when no
properly nested grouping is used: switching from right to left several times
nested, results in spacing ending up after each other due to nested mirroring. Of
course a sane macro package will manage this for the user but here we are
discussing the low level injection of directional information.

This is what happens:

\starttyping
\textdirection 1 nur {\textdirection 0 run \textdirection 1 NUR} nur
\stoptyping

This becomes stepwise:

\startnarrower
\starttyping
injected: [push 1]nur {[push 0]run [push 1]NUR} nur
balanced: [push 1]nur {[push 0]run [pop 0][push 1]NUR[pop 1]} nur[pop 0]
result  : run {RUNrun } run
\stoptyping
\stopnarrower

And this:

\starttyping
\textdirection 1 nur {nur \textdirection 0 run \textdirection 1 NUR} nur
\stoptyping

becomes:

\startnarrower
\starttyping
injected: [+TRT]nur {nur [+TLT]run [+TRT]NUR} nur
balanced: [+TRT]nur {nur [+TLT]run [-TLT][+TRT]NUR[-TRT]} nur[-TRT]
result  : run {run RUNrun } run
\stoptyping
\stopnarrower

Now, in the following examples watch where we put the braces:

\startbuffer
\textdirection 1 nur {{\textdirection 0 run} {\textdirection 1 NUR}} nur
\stopbuffer

\typebuffer

This becomes:

\startnarrower
\getbuffer
\stopnarrower

Compare this to:

\startbuffer
\textdirection 1 nur {{\textdirection 0 run }{\textdirection 1 NUR}} nur
\stopbuffer

\typebuffer

Which renders as:

\startnarrower
\getbuffer
\stopnarrower

So how do we deal with the next?

\startbuffer
\def\ltr{\textdirection 0\relax}
\def\rtl{\textdirection 1\relax}

run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur}
run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run}
\stopbuffer

\typebuffer

It gets typeset as:

\startnarrower
\startlines
\getbuffer
\stoplines
\stopnarrower

We could define the two helpers to look back, pick up a skip, remove it and
inject it after the dir node. But that way we loose the subtype information that
for some applications can be handy to be kept as|-|is. This is why we now have a
variant of \lpr {textdirection} which injects the balanced node before the skip.
Instead of the previous definition we can use:

\startbuffer[def]
\def\ltr{\linedirection 0\relax}
\def\rtl{\linedirection 1\relax}
\stopbuffer

\typebuffer[def]

and this time:

\startbuffer[txt]
run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur}
run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run}
\stopbuffer

\typebuffer[txt]

comes out as a properly spaced:

\startnarrower
\startlines
\getbuffer[def,txt]
\stoplines
\stopnarrower

Anything more complex that this, like combination of skips and penalties, or
kerns, should be handled in the input or macro package because there is no way we
can predict the expected behaviour. In fact, the \lpr {linedir} is just a
convenience extra which could also have been implemented using node list parsing.

\stopsubsection

\startsubsection[title={Normalizing lines}]

The original \TEX\ machinery was never meant to be opened up. As a consequence a
constructed line can have different layouts. There can be left- and/or right
skips and hanging indentation or parshape can result in a shift and adapted
width. In \LUATEX\ glue got subtypes so we can recognize the left-, right and
parfill skips, but still there is no hundred percent certainty about the shape.

In \LUAMETATEX\ lines can be normalized. This is optional because we want to
preserve the original (for comparison) and is controlled by \lpr
{normalizelinemode}. That variable actually drives some more. An earlier version
provided a few more granular options (for instance: does a leftskip comes before
or after a left hanging indentation) but in the end that was dropped. Because
this normalization only is seen at the \LUA\ end there is no need to go into much
detail here.

At this moment a line has this pattern: left parfill, left hang, left skip,
indentation, content, right hang, right skip, right parfill. Of course the
indentation and fill skips are not present in every line.

Control over normalization happens via the mentioned mode variable and here is
what the engine provides right now. We use a bitmap:

\starttabulate[|l|l|]
\DB value \BC reported \NC \NR
\TB
\NC \type{0x0001} \NC normalize line as described above            \NC \NR
\NC \type{0x0002} \NC use a skip for parindent instead of a box    \NC \NR
\NC \type{0x0004} \NC swap hangindent in l2r mode                  \NC \NR
\NC \type{0x0008} \NC swap parshape in l2r mode                    \NC \NR
\NC \type{0x0010} \NC put breaks after dir in l2r mode             \NC \NR
\NC \type{0x0020} \NC remove margin kerns (\PDFTEX\ left-over)     \NC \NR
\NC \type{0x0040} \NC if needed clip width and use correction kern \NC \NR
\LL
\stoptabulate

Setting the bit enables the related normalization. More features might be added
in future releases.

% Swapping shapes
%
% Another adaptation to the \ALEPH\ directional model is control over shapes driven
% by \prm {hangindent} and \prm {parshape}. This is controlled by a new parameter
% \lpr {shapemode}:
%
% \starttabulate[|c|l|l|]
% \DB value    \BC \prm {hangindent} \BC \prm {parshape} \NC \NR
% \TB
% \BC \type{0} \NC  normal             \NC normal            \NC \NR
% \BC \type{1} \NC  mirrored           \NC normal            \NC \NR
% \BC \type{2} \NC  normal             \NC mirrored          \NC \NR
% \BC \type{3} \NC  mirrored           \NC mirrored          \NC \NR
% \LL
% \stoptabulate
%
% The value is reset to zero (like \prm {hangindent} and \prm {parshape})
% after the paragraph is done with. You can use negative values to prevent
% this. In \in {figure} [fig:shapemode] a few examples are given.
%
% \startplacefigure[reference=fig:shapemode,title={The effect of \type {shapemode}.}]
%     \startcombination[2*3]
%         {\ruledvbox \bgroup \setuptolerance[verytolerant]
%             \hsize .45\textwidth \switchtobodyfont[6pt]
%                 \pardirection 0 \textdirection 0
%                 \hangindent 40pt \hangafter -3
%                 \leftskip10pt \input tufte \par
%          \egroup} {TLT: hangindent}
%         {\ruledvbox \bgroup \setuptolerance[verytolerant]
%             \hsize .45\textwidth \switchtobodyfont[6pt]
%             \pardirection 0 \textdirection 0
%             \parshape 4 0pt .8\hsize 10pt .8\hsize 20pt .8\hsize 0pt \hsize
%             \input tufte \par
%          \egroup} {TLT: parshape}
%         {\ruledvbox \bgroup \setuptolerance[verytolerant]
%             \hsize .45\textwidth \switchtobodyfont[6pt]
%             \pardirection 1 \textdirection 1
%             \hangindent 40pt \hangafter -3
%             \leftskip10pt \input tufte \par
%          \egroup} {TRT: hangindent mode 0}
%         {\ruledvbox \bgroup \setuptolerance[verytolerant]
%             \hsize .45\textwidth \switchtobodyfont[6pt]
%             \pardirection 1 \textdirection 1
%             \parshape 4 0pt .8\hsize 10pt .8\hsize 20pt .8\hsize 0pt \hsize
%             \input tufte \par
%          \egroup} {TRT: parshape mode 0}
%         {\ruledvbox \bgroup \setuptolerance[verytolerant]
%             \hsize .45\textwidth \switchtobodyfont[6pt]
%             \shapemode=3
%             \pardirection 1 \textdirection 1
%             \hangindent 40pt \hangafter -3
%             \leftskip10pt \input tufte \par
%          \egroup} {TRT: hangindent mode 1 & 3}
%         {\ruledvbox \bgroup \setuptolerance[verytolerant]
%             \hsize .45\textwidth \switchtobodyfont[6pt]
%             \shapemode=3
%             \pardirection 1 \textdirection 1
%             \parshape 4 0pt .8\hsize 10pt .8\hsize 20pt .8\hsize 0pt \hsize
%             \input tufte \par
%          \egroup} {TRT: parshape mode 2 & 3}
%     \stopcombination
% \stopplacefigure
%
% We have \type {\pardirection}, \type {\textdirection}, \type {\mathdirection} and
% \type {\linedirection} that is like \type {\textdirection} but with some
% additional (inline) glue checking.

% Controlling glue with \lpr {breakafterdirmode}
%
% Glue after a dir node is ignored in the linebreak decision but you can bypass that
% by setting \lpr {breakafterdirmode} to~\type {1}. The following table shows the
% difference. Watch your spaces.
%
% \def\ShowSome#1{%
%     \BC \type{#1}
%     \NC \breakafterdirmode\zerocount\hsize\zeropoint#1
%     \NC
%     \NC \breakafterdirmode\plusone\hsize\zeropoint#1
%     \NC
%     \NC \NR
% }
%
% \starttabulate[|l|Tp(1pt)|w(5em)|Tp(1pt)|w(5em)|]
%     \DB
%     \BC \type{0}
%     \NC
%     \BC \type{1}
%     \NC
%     \NC \NR
%     \TB
%     \ShowSome{pre {\textdirection 0 xxx} post}
%     \ShowSome{pre {\textdirection 0 xxx }post}
%     \ShowSome{pre{ \textdirection 0 xxx} post}
%     \ShowSome{pre{ \textdirection 0 xxx }post}
%     \ShowSome{pre { \textdirection 0 xxx } post}
%     \ShowSome{pre {\textdirection 0\relax\space xxx} post}
%     \LL
% \stoptabulate

\stopsubsection

\startsubsection[title=Orientations]

As mentioned, the difference with \LUATEX\ is that we only have numeric
directions and that there are only two: left|-|to|-|right (\type {0}) and
right|-|to|-|left (\type {1}). The direction of a box is set with \type
{direction}.

In addition to that boxes can now have an \type {orientation} keyword followed by
optional \type {xoffset} and|/|or \type {yoffset} keywords. The offsets don't
have consequences for the dimensions. The alternatives \type {xmove} and \type
{ymove} on the contrary are reflected in the dimensions. Just play with them. The
offsets and moves only are accepted when there is also an orientation, so no time
is wasted on testing for these rarely used keywords. There are related primitives
\type {\box...} that set these properties.

As these are experimental it will not be explained here (yet). They are covered
in the descriptions of the development of \LUAMETATEX: articles and|/|or
documents in the \CONTEXT\ distribution. For now it is enough to know that the
orientation can be up, down, left or right (rotated) and that it has some
anchoring variants. Combined with the offsets this permits macro writers to
provide solutions for top|-|down and bottom|-|up writing directions, something
that is rather macro package specific and used for scripts that need
manipulations anyway. The \quote {old} vertical directions were never okay and
therefore not used.

There are a couple of properties in boxes that you can set and query but that
only really take effect when the backend supports them. When usage on \CONTEXT\
shows that is't okay, they will become official, so we just mention them: \type
{\boxdirection}, \type {\boxattr}, \type {\boxorientation}, \type {\boxxoffset},
\type {\boxyoffset}, \type {\boxxmove}, \type {\boxymove} and \type {\boxtotal}.

{\em This is still somewhat experimental and will be documented in more detail
when I've used it more in \CONTEXT\ and the specification is frozen. This might
take some time (and user input).}

\stopsubsection

\stopsection

\startsection[title=Keywords]

Some primitives accept one or more keywords and \LUAMETATEX\ adds some more. In
order to deal with this efficiently the keyword scanner has been optimized, where
even the context was taken into account. As a result the scanner was quite a bit
faster. This kind of optimization was a graduate process the eventually ended up
in what we have now. In traditional \TEX\ (and also \LUATEX) the order of
keywords is sometimes mixed and sometimes prescribed. In most cases only one
occurrence is permitted. So, for instance, this is valid in \LUATEX:

\starttyping
\hbox attr 123 456 attr 123 456 spread 10cm { }
\hrule width 10cm depth 3mm
\hskip 3pt plus 2pt minus 1pt
\stoptyping

The \type {attr} comes before the \type {spread}, rules can have multiple mixed
dimension specifiers, and in glue the optional \type {minus} part always comes
last. The last two commands are famous for look ahead side effects which is why
macro packages will end them with something not keyword, like \type {\relax},
when needed.

In \LUAMETATEX\ the following is okay. Watch the few more keywords in box and
rule specifications.

\starttyping
\hbox reverse to 10cm attr 123 456 orientation 4 xoffset 10pt spread 10cm { }
\hrule xoffset 10pt width 10cm depth 3mm
\hskip 3pt minus 1pt plus 2pt
\stoptyping

Here the order is not prescribed and, as demonstrated with the box specifier, for
instance dimensions (specified by \type {to} or \type {spread} can be overloaded
by later settings. In case you wonder if that breaks compatibility: in some way
it does but bad or sloppy keyword usage breaks a run anyway. For instance \type
{minuscule} results in \type {minus} with no dimension being seen. So, in the end
the user should not noticed it and when a user does, the macro package already
had an issue that had to be fixed.

\stopsection

\startsection[title=Expressions and \lpr {numericscale}]

The \type {*expr} parsers now accept \type {:} as operator for integer division
(the \type {/} operators does rounding. This can be used for division compatible
with \type {\divide}. I'm still wondering if adding a couple of bit operators
makes sense (for integers).

The \lpr{numericscale} parser is kind of special (and might evolve). For now it
converts a following number in a scale value as often used in \TEX, where 1000
means scaling by~1.0. The trick is in the presence of a digit (or comma): 1.234
becomes 1234 but 1234 stays 1234 and from this you can deduce that 12.34 becomes
123400. Internally \TEX\ calculates with integers, but this permits the macro
package to provide an efficient mix.

\stopsection

\startsection[title=Macro arguments]

Again this is experimental and (used and) discussed in document that come with the
\CONTEXT\ distribution. When defining a macro you can do this:

\starttyping
\def\foo(#1)#2{...}
\stoptyping

Here the first argument between parentheses is mandate. But the magic
prefix \lpr {tolerant} makes that limitation go away:

\starttyping
\tolerant\def\foo(#1)#2{...}
\stoptyping

A variant is this:

\starttyping
\tolerant\def\foo(#1)#*(#2){...}
\stoptyping

Here we have two optional arguments, possibly be separated by spaces. There are
more parsing options:

\starttabulate[|T|i2l|]
\FL
\NC +   \NC keep the braces \NC \NR
\NC -   \NC discard and don't count the argument \NC \NR
\NC /   \NC remove leading an trailing spaces and pars \NC \NR
\NC =   \NC braces are mandate \NC \NR
\NC _   \NC braces are mandate and kept \NC \NR
\NC ^   \NC keep leading spaces \NC \NR
\ML
\NC 1-9 \NC an argument \NC \NR
\NC 0   \NC discard but count the argument \NC \NR
\ML
\NC *   \NC ignore spaces \NC \NR
\NC :   \NC pick up scanning here  \NC \NR
\NC ;   \NC quit scanning \NC \NR
\LL
\stoptabulate

For the moment we leave it to your fantasy what these options do. Most probably
only make sense when you write a bit more complex macros. Just try to imagine
what this does:

\starttyping
\permanent\tolerant\global\protected\def\foo(#1)#*#;[#2]#:#3{...}
\stoptyping

Of course complex combinations can be confusing because after all \TEX\ is
parsing for (multi|-|token) delimiters and will happily gobble the whole file if
you are not careful. You can quit scanning if you want:

\starttyping
\mymacro 123\ignorearguments
\stoptyping

which of course only makes sense when used in a nested call where an already
picked up arguments is processed further. A not (yet) discussed feature of the
parser is that it will happily skip tokens that have the (probably seldom used)
ignored characters property.

When you use tracing or see error messages arguments defined using for instance
\type {#=} will have their usual number in the macro body, so you need to keep
track of the numbers.

All this is rather easy on the engine and although it might have a little impact
on performance this has been compensated by some more efficiency in the macro
parser and engine in general and of course you can gain back some by using these
features.

\stopsection

\startsection[title=Overload protection]

There is an experimental overload protection mechanism that we will test for a
while before declaring it stable. The reason for that is that we need to adapt
the \CONTEXT\ code base in order to test its usefulness. Protection is achieved
via prefixes. Depending on the value of the \lpr {overloadmode} variable
warnings or errors will be triggered. Examples of usage can be found in some
documents that come with \CONTEXT, so here we just stick to the basics.

\starttyping
\mutable  \def\foo{...}
\immutable\def\foo{...}
\permanent\def\foo{...}
\frozen   \def\foo{...}
\aliased  \def\foo{...}
\stoptyping

A \lpr {mutable} macro can always be changed contrary to an \lpr {immutable} one.
For instance a macro that acts as a variable is normally \lpr {mutable}, while a
constant can best be immutable. It makes sense to define a public core macro as
\lpr {permanent}. Primives start out a \lpr {permanent} ones but with a primitive
property instead.

\startbuffer
          \let\relaxone  \relax 1: \meaningfull\relaxone
\aliased  \let\relaxtwo  \relax 2: \meaningfull\relaxtwo
\permanent\let\relaxthree\relax 3: \meaningfull\relaxthree
\stopbuffer

\typebuffer

The \lpr {meaningfull} primitive is like \prm {meaning} but report the
properties too. The \lpr {meaningless} companion reports the body of a macro.
Anyway, this typesets:

\startlines \tttf \getbuffer \stoplines

So, the \lpr {aliased} prefix copies the properties. Keep in mind that a macro
package can redefine primitives, but \prm {relax} is an unlikely candidate.

There is an extra prefix \lpr {noaligned} that flags a macro as being valid
for \prm {noalign} compatible usage (which means that the body must contain that
one. The idea is that we then can do this:

\starttyping
\permanent\protected\noaligned\def\foo{\noalign{...}} % \foo is unexpandable
\stoptyping

that is: we can have protected macros that don't trigger an error in the parser
where there is a look ahead for \prm {noalign} which is why normally protection
doesn't work well. So: we have macro flagged as permanent (overload protection),
being protected (that is, not expandable by default) and a valid equivalent of
the noalign primitive. Of course we can also apply the \prm {global} and \lpr
{tolerant} prefixes here. The complete repertoire of extra prefixes is:

\starttabulate
\HL
\NC \type {frozen}     \NC a macro that has to be redefined in a managed way \NC \NR
\NC \type {permanent}  \NC a macro that had better not be redefined \NC \NR
\NC \type {primitive}  \NC a primitive that normally will not be adapted \NC \NR
\NC \type {immutable}  \NC a macro or quantity that cannot be changed, it is a constant \NC \NR
\NC \type {mutable}    \NC a macro that can be changed no matter how well protected it is \NC \NR
\HL
\NC \type {instance}   \NC a macro marked as (for instance) be generated by an interface \NC \NR
\HL
\NC \type {noaligned}  \NC the macro becomes acceptable as \type {\noalign} alias \NC \NR
\HL
\NC \type {overloaded} \NC when permitted the flags will be adapted \NC \NR
\NC \type {enforced}   \NC all is permitted (but only in zero mode or ini mode) \NC \NR
\NC \type {aliased}    \NC the macro gets the same flags as the original \NC \NR
\HL
\stoptabulate

The not yet discussed \lpr {instance} is just a flag with no special meaning
which can be used as classifier. The \lpr {frozen} also protects against overload
which brings amount of blockers to four.

To what extent the engine will complain when a property is changed in a way that
violates the flags depends on the parameter \lpr {overloadmode}. When this
parameter is set to zero no checking takes place. More interesting are values
larger than zero. If that is the case, when a control sequence is flagged as
mutable, it is always permitted to change. When it is set to immutable one can
never change it. The other flags determine the kind of checking done. Currently
the following overload values are used:

\starttabulate[|l|l|c|c|c|c|c|]
    \NC   \NC         \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR
    \NC 1 \NC warning \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
    \NC 2 \NC error   \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
    \NC 3 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
    \NC 4 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
    \NC 5 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
    \NC 6 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
\stoptabulate

The even values (except zero) will abort the run. A value of 255 will freeze this
parameter. At level five and above the \lpr {instance} flag is also checked but
no drastic action takes place. We use this to signal to the user that a specific
instance is redefined (of course the definition macros can check for that too).

The \lpr {overloaded} prefix can be used to overload a frozen macro. The \lpr
{enforced} is more powerful and forces an overload but that prefix is only
effective in ini mode or when it's embedded in the body of a macro or token list
at ini time unless of course at runtime the mode is zero.

So far for a short explanation. More details can be found in the \CONTEXT\
documentation where we can discuss it in a more relevant perspective. It must be
noted that this feature only makes sense a controlled situation, that is: user
modules or macros of unpredictable origin will probably suffer from warnings and
errors when de mode is set to non zero. In \CONTEXT\ we're okay unless of course
users redefine instances but there a warning or error is kind of welcome.

\stopsection

\startsection[title=Constants]

It is rather common to store constant values in a register or character
definition.

\starttyping
\newcount\MyConstantA \MyConstantA 123
\newdimen\MyConstantB \MyConstantB 123pt
\chardef \MyConstantC \MyConstantC 123
\stoptyping

But in \LUAMETATEX\ we also can do this:

\starttyping
\integerdef\MyConstantC 456
\dimendef  \MyConstantD 456pt
\stoptyping

These two are stored as efficient as a register but don't occupy a register slot.
They can be set as above, need \prm {the} for serializations and are seen as
valid number or dimension when needed.

Experiments with constant strings made the engine source more complex than I
wanted so that features was rejected. Of course we can use the prefixes mentioned
in a previous section.

\stopsection

\startsection[title=Nodes]

The \ETEX\ primitive \type {\lastnodetype} is not honest in reporting the
internal numbers as it uses its own values. But you can set \type
{\internalcodesmode} to a non|-|zero value to get the real id's instead. In
addition there is \type {\lastnodesubtype}.

Another last one is \type {\lastnamedcs} which holds the last match but this one
should be used with care because one never knows if in the meantime something
else \quote {last} has been seen.

\stopsection

\stopchapter

\stopcomponent