1 files changed, 254 insertions, 0 deletions
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex
new file mode 100644
index 00000000000..4f705e37caa
--- /dev/null
+++ b/Master/texmf-dist/doc/context/sources/general/manuals/lowlevel/lowlevel-security.tex
@@ -0,0 +1,254 @@
+% language=us
+
+% It took some time to get the right balance of using the overload related features
+% but at some point it started feeling right. Of course it will never be as
+% perfectly timed and integrated as Gavin Harrison performance on "Threatening War"
+% (youtube movie) but that doesn't mean I should not aim for perfection. But as
+% with drumming, it takes practising and that is what I did on a subset of sources
+% when writing the engine code.
+%
+% It tooks a few weeks but November 21 2020 the last of the core files was turned
+% \LMTX, not that the work was done (checking to be done (thanks Wolfgang!), imp
+% files to be checked, many \LUA\ files to be updated) but it's a start. This time
+% the musical timestamp is listening to Nikola Cvetkovic (piano) and history
+% podcasts.
+
+\usemodule[system-tokens]
+
+\environment lowlevel-style
+
+\startdocument
+  [title=security,
+   color=middleorange]
+
+\startsection[title=Preamble]
+
+Here I will discuss a moderate security subsystem of \LUAMETATEX\ and therefore
+\CONTEXT\ \LMTX. This is not about security in the sense of the typesetting
+machinery doing harm to your environment, but more about making sure that a user
+doesn't change the behavior of the macro package in ways that introduce
+interference and thereby unwanted side effect. It's all about protecting macros.
+
+This is all very experimental and we need to adapt the \CONTEXT\ source code to
+this. Actually that will happen a few times because experiments trigger that. It
+might take a few years before the security model is finalized and all files are
+updated accordingly. There are lots of files and macros involved. In the process
+the underlying features in the engine might evolve.
+
+\stopsection
+
+\startsection[title=Flags]
+
+Before we go into the security levels we see what flags can be set. The \TEX\
+language has a couple of so called prefixes that can be used when setting values
+and defining macros. Any engine that has traditional \TEX\ with \ETEX\ extensions
+can do this:
+
+\starttyping[option=TEX]
+                 \def\foo{foo}
+\global          \def\foo{foo}
+\global\protected\def\foo{foo}
+\stoptyping
+
+And \LUAMETATEX\ adds another one:
+
+\starttyping[option=TEX]
+       \tolerant          \def\foo{foo}
+\global\tolerant          \def\foo{foo}
+\global\tolerant\protected\def\foo{foo}
+\stoptyping
+
+What these prefixes do is discussed elsewhere. For now is is enough to know that
+the two optional prefixes \type {\protected} and \type {\tolerant} make for four
+distinctive cases of macro calls.
+
+But there are more prefixes:
+
+\starttabulate
+\HL
+\NC \type {frozen}     \NC a macro that has to be redefined in a managed way \NC \NR
+\NC \type {permanent}  \NC a macro that had better not be redefined \NC \NR
+\NC \type {primitive}  \NC a primitive that normally will not be adapted \NC \NR
+\NC \type {immutable}  \NC a macro or quantity that cannot be changed, it is a constant \NC \NR
+\NC \type {mutable}    \NC a macro that can be changed no matter how well protected it is \NC \NR
+\HL
+\NC \type {instance}   \NC a macro marked as (for instance) be generated by an interface \NC \NR
+\HL
+\NC \type {noaligned}  \NC the macro becomes acceptable as \type {\noalign} alias \NC \NR
+\HL
+\NC \type {overloaded} \NC when permitted the flags will be adapted \NC \NR
+\NC \type {enforced}   \NC all is permitted (but only in zero mode or ini mode) \NC \NR
+\NC \type {aliased}    \NC the macro gets the same flags as the original \NC \NR
+\HL
+\stoptabulate
+
+These prefixed set flags to the command at hand which can be a macro but
+basically any control sequence.
+
+To what extent the engine will complain when a property is changed in a way that
+violates the above depends on the parameter \type {\overloadmode}. When this
+parameter is set to zero no checking takes place. More interesting are values
+larger than zero. If that is the case, when a control sequence is flagged as
+mutable, it is always permitted to change. When it is set to immutable one can
+never change it. The other flags determine the kind of checking done. Currently
+the following overload values are used:
+
+\starttabulate[|l|l|c|c|c|c|c|]
+    \NC   \NC         \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR
+    \NC 1 \NC warning \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
+    \NC 2 \NC error   \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
+    \NC 3 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
+    \NC 4 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
+    \NC 5 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
+    \NC 6 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
+\stoptabulate
+
+The even values (except zero) will abort the run. In \CONTEXT\ we plug in a
+callback that deals with the messages. A value of 255 will freeze this parameter.
+At level five and above the \type {instance} flag is also checked but no drastic
+action takes place. We use this to signal to the user that a specific instance is
+redefined (of course the definition macros can check for that too).
+
+So, how does it work. The following is okay:
+
+\starttyping[option=TEX]
+\def\MacroA{A}
+\def\MacroB{B}
+\let\MyMacro\MacroA
+\let\MyMacro\MacroB
+\stoptyping
+
+The first two macros are ordinary ones, and the last two lines just create an alias. Such
+an alias shares the definition, but when for instance \type {\MacroA} is redefined, its
+new meaning will not be reflected in the alias.
+
+\starttyping[option=TEX]
+\permanent\protected\def\MacroA{A}
+\permanent\protected\def\MacroB{B}
+\let\MyMacro\MacroA
+\let\MyMacro\MacroB
+\stoptyping
+
+This also works, because the \type {\let} will create an alias with the protected
+property but it will not take the \type {permanent} propery along. For that we need
+to say:
+
+\starttyping[option=TEX]
+\permanent\protected\def\MacroA{A}
+\permanent\protected\def\MacroB{B}
+\permanent\let\MyMacro\MacroA
+\permanent\let\MyMacro\MacroB
+\stoptyping
+
+or, when we want to copy all properties:
+
+\starttyping[option=TEX]
+\permanent\protected\def\MacroA{A}
+\permanent\protected\def\MacroB{B}
+\aliased\let\MyMacro\MacroA
+\aliased\let\MyMacro\MacroB
+\stoptyping
+
+However, in \CONTEXT\ we have commands that we like to protect against
+overloading but at the same time have a different meaning depending on the use
+case. An example is the \type {\NC} (next column) command that has a different
+implementation in each of the table mechanisms.
+
+\starttyping[option=TEX]
+\permanent\protected\def\NC_in_table   {...}
+\permanent\protected\def\NC_in_tabulate{...}
+\aliased\let\NC\NC_in_table
+\aliased\let\NC\NC_in_tabulate
+\stoptyping
+
+Here the second aliasing of \type {\NC} fails (assuming of course that we enabled
+overload checking). One can argue that grouping can be used but often no grouping
+takes place when we redefine on the fly. Because \type {frozen} is less restrictive
+than \type {primitive} or \type {permanent}, and of course \type {immutable}, the
+next variant works:
+
+\starttyping[option=TEX]
+\frozen\protected\def\NC_in_table   {...}
+\frozen\protected\def\NC_in_tabulate{...}
+\overloaded\let\NC\NC_in_table
+\overloaded\let\NC\NC_in_tabulate
+\stoptyping
+
+However, in practice, as we want to keep the overload checking, we have to do:
+
+\starttyping[option=TEX]
+\frozen\protected\def\NC_in_table   {...}
+\frozen\protected\def\NC_in_tabulate{...}
+\overloaded\frozen\let\NC\NC_in_table
+\overloaded\frozen\let\NC\NC_in_tabulate
+\stoptyping
+
+or use \type {\aliased}, but there might be conflicting permissions. This is not
+that nice, so there is a kind of dirty trick possible. Consider this:
+
+\starttyping[option=TEX]
+\frozen\protected\def\NC_in_table   {...}
+\frozen\protected\def\NC_in_tabulate{...}
+\def\setNCintable   {\enforced\let\frozen\let\NC\NC_in_table}
+\def\setNCintabulate{\enforced\let\frozen\let\NC\NC_in_tabulate}
+\stoptyping
+
+When we're in so called \type {initex} mode or when the overload mode is zero,
+the \type {\enforced} prefix is internalized in a way that signals that the
+follow up is not limited by the overload mode and permissions. This definition
+time binding mechanism makes it possible to use \type {permanent} macros that
+users cannot redefine, but existing macros can, unless of course they tweak the
+mode parameter.
+
+Now keep in mind that users can always cheat but that is intentional. If you
+really want to avoid that you can set the overload mode to 255 after which it
+cannot be set any more. However, it can be useful to set the mode to zero (or
+some warning level) when foreign macro packages are used.
+
+\stopsection
+
+\startsection[title=Complications]
+
+One side effect of all this is that all those prefixes can lead to more code. On
+the other hand we save some due to the extended macro argument handling features.
+When you take the size of the format file as reference, in the end we get a
+somewhat smaller file. Every token that you add of remove gives a 8~bytes
+difference. The extra overhead that got added to the engine is compensated by the
+fact that some macro implementations can be more efficient. In the end, in spite
+of these new features and the more extensive testing of flags performance is
+about the same. \footnote {And if you wonder about memory, by compacting the used
+(often scattered) token memory before dumping I manages to save some 512K on the
+format file, so often the loss and gain are somewhere else.}
+
+\stopsection
+
+\startsection[title=Introspection]
+
+In case you want to get some details about the properties of a macro, you can
+check its meaning. The full variant shows all of them.
+
+\startbuffer
+% a macro with two optional arguments with optional spacing in between:
+
+\permanent\tolerant\protected\def\MyFoo[#1]#*[#2]{(#1)(#2)}
+
+\meaningless\MyFoo\par
+\meaning    \MyFoo\par
+\meaningfull\MyFoo\par
+\stopbuffer
+
+\typebuffer[option=TEX]
+
+\startpacked \getbuffer \stoppacked
+
+\stopsection
+
+% In \CONTEXT:
+% c! v! s! ??
+% newif newcount ... newconditional etc
+% userinterface (permanent)
+% primitives
+% noaligned
+% frozen is for users
+
+\stopdocument