diff options
Diffstat (limited to 'Master/texmf-dist/doc/context/sources/general/manuals/mk/mk-goingbeta.tex')
-rw-r--r-- | Master/texmf-dist/doc/context/sources/general/manuals/mk/mk-goingbeta.tex | 343 |
1 files changed, 343 insertions, 0 deletions
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/mk/mk-goingbeta.tex b/Master/texmf-dist/doc/context/sources/general/manuals/mk/mk-goingbeta.tex new file mode 100644 index 00000000000..9937a373dc2 --- /dev/null +++ b/Master/texmf-dist/doc/context/sources/general/manuals/mk/mk-goingbeta.tex @@ -0,0 +1,343 @@ +% language=uk + +\startcomponent mk-goingbeta + +\environment mk-environment + +\doifmodeelse {tug} { + + \title {Lua\TeX\ going beta} + + \subject{by Hans Hagen \& Taco Hoekwater} + + This is Chapter~XI from \notabene {\CONTEXT, from \MKII\ to \MKIV}, a document + that describes our explorations, experiments and decisions made while + we develop \LUATEX. + + \blank[3*big] + +} { + + \chapter {Going beta} + +} + +\subject{introduction} + +We're closing in on the day that we will go beta with \LUATEX\ (end of July +2007). By now we have a rather good picture of its potential and to what +extend \LUATEX\ will solve some of our persistent problems. Let's first +summarize our reasons for and objectives with \LUATEX. + +\startitemize + +\item The world has moved from 8~bits to 32~bits and more, and this is +quite noticeable in the arena of fonts. Although \TYPEONE\ fonts could host +more than 256 glyphs, the associated technology was limited to 256. The advent +of \OPENTYPE\ fonts will make it easier to support multiple languages at the +same time without the need to switch fonts at awkward times. + +\item At the same time \UNICODE\ is replacing 8~bit based encoding vectors and +code pages (input regimes). The most popular and rather efficient \UTF8 encoding +has become a de factor standard in document encoding and interchange. + +\item Although we can do real neat tricks with \TEX, given some nasty programming, +we are touching the limits of its possibilities. In order for it to survive we +need to extend the engine but not at the cost of base compatibility. + +\item Coding solutions in a macro language is fine, but sometimes you long to a more +procedural approach. Manipulating text, handling \IO, interfacing \unknown\ the +technology moves on and we need to move along too. + +\stopitemize + +Hence \LUATEX: a merge of the mainstream traditional \TEX\ engines, stripped from +broken or incomplete features and opened up to an embedded \LUA\ scripting engine. + +We will describe the impact of this new engine by starting from its core components +reflected in the specific \LUA\ interface libraries. Missing here is embedded support +for \METAPOST, because it's not yet there (apart from the fact that we use \LUA\ to +convert \METAPOST\ graphics into \TEX). Also missing is the interfacing to the \PDF\ +backend, which is also on the agenda for later. Special extensions, for instance those +dealing with runtime statistics are also not discussed. Since we use \CONTEXT\ as +testbed, we will refer to the \LUATEX\ aware version of this macro package, \MKIV, but +most conclusions are rather generic. + +\subject{tex internals} + +In order to manipulate \TEX's data structures, we need access to all those registers. +Already early in the development, dimension and counters were accessible and when +token and node interfaces were implemented, those registers also were interfaced. + +Those who read the previous chapters will have noticed that we hardly discussed this +option. The reason is that we didn't yet needed that access much in order to implement +font support and list processing. After all, most of the data that we need to access and +manipulate is not in the registers at all. Information meant for \LUA\ can be stored +in \LUA\ data structures. In fact, the basic call + +\starttyping +\directlua 0 {some lua code} +\stoptyping + +has shown to be a pretty good starting point and the fact that one can print back to +the \TEX\ engine overcomes the need to store results in shared variables. + +\starttyping +\def\valueofpi{\directlua0{tex.sprint(math.pi()}} +\stoptyping + +The number of such direct calls is not that large anyway. More often a call to \LUA\ +will be initiated by a callback, i.e.\ a hook into the \TEX\ machinery. + +What will be the impact of access on \CONTEXT\ \MKIV ? This is yet hard to tell. In a +later stage of the development, when parts of the \TEX\ machinery will be rewritten in +order to get rid of the current global nature of many variables, we will gain more +control and access to \TEX's internals. Core functionality will be isolated, can be +extended and|/|or overloaded and at that moment access to internals is much more +needed. But certainly that will be beyond the current registers and variables. + +\subject{callbacks} + +These are the spine of \LUATEX: here both worlds communicate with each other. A callback +is a place in the \TEX\ kernel where some information is passed to \LUA\ and some result +is returned that is then used along the road. The reference manual mentions them all and +we will not repeat them here. Interesting is that in \MKIV\ most of them are used and for +tasks that are rather natural to their place and function. + +\starttyping +callback.register("tex_wants_to_do_this", + function but_use_lua_to_do_it_instead(a,b,c) + -- do whatever you like with a, b and c + return a, b, c + end +) +\stoptyping + +The impact of callbacks on \MKIV\ is big. It provides us a way to solve persistent +problems or reimplement existing solutions in more convenient ways. Because we tested +realistic functionality on real (moderately complex) documents using a pretty large +macro package, we can safely conclude that callbacks are quite efficient. Stepwise +\LUA\ kicks in in order to: + +\startitemize[packed] +\item influence the input medium so that it provides a sequence of \UTF\ characters +\item manipulate the stream of characters that will be turned into a list of tokens +\item convert the list of tokens into another list of tokens +\item enhance the list of nodes that will be turned into a typeset paragraph +\item tweak the mechanisms that come into play when lines are constructed +\item finalize the result that will end up in the output medium +\stopitemize + +Interesting is that manipulating tokens is less useful than it may look at first +sight. This has to do with the fact that it's (mostly) an expanded stream and at that +time we've lost some information or need to do quite some coding in order to analyze +the information and act upon it. + +Will \CONTEXT\ users see any of this? Chances are small that they will, although we +will provide hooks so that they can add special code themselves. Users activating +a callback has some danger, since it may overload already existing functionality. +Chaining functionality in a callback also has drawbacks, if only that one may be +confronted with already processed results and|/|or may destroy this result in +unpredictable ways. So, as with most low level \TEX\ features, \CONTEXT\ users will +work with more abstract interfaces. + +\subject{in- and output} + +In \MKIV\ we will no longer use the \KPSE\ library directly. Instead we use a +reimplementation in \LUA\ that not only is more efficient, but also more powerful: +it can read from \ZIP\ files, use protocols, be more clever in searching, reencodes +the input streams when needed, etc. The impact on \MKIV\ is large. Most \TEX\ code +that deals with input reencoding has gone away and is replaced by \LUA\ code. + +Although it is not directly related with reading from the input medium, in that stage +we also replaced verbatim handling code. Such (often messy) catcode related situations +are now handled more flexible, thanks to fast catcode table switching (a new +\LUATEX\ feature) and features like syntax highlighting can be made more neat. + +Buffers, a quite old but frequently used feature of \CONTEXT, are now kept in +memory instead of files. This speeds up runs. Auxiliary data, aka multi||pass +information, will no longer be stored in \TEX\ files but in \LUA\ files. In +\CONTEXT\ we have one such auxiliary file and in \MKII\ this file is selectively +filtered, but in \MKIV\ we will be less careful with memory and load all that +data once. Such speed improvements compensate the fact that \LUATEX\ is somewhat +slower than it's ancestor \PDFTEX. (Actually, the fact that \LUATEX\ is a bit +slower that \PDFTEX\ is mostly due to the fact that it has \ALEPH\ code on +board.) + +Users often wonder why there are so many temporary files, but these mostly relate +to \METAPOST\ support. These will go away once we have \METAPOST\ as a library. + +In a similar way support for \XML\ will be enriched. We already have experimental +loaders, filters and other code, and integration is on the agenda. Since \CONTEXT\ uses +\XML\ for some sub systems, this may have some impact. + +Other \IO\ related improvements involve debugging, error handling and logging. We can pop +up helpers and debug screens (\MKIV\ can produce \XHTML\ output and then launch a +browser). Users can choose more verbose logging of \IO\ and ask for log data to be +formatted in \XML. These parts need some additional work, because in the end we will +also reimplement and extend \TEX's error handling. + +Another consequence of this will be that we will be able to package \TEX\ more +conveniently. We can put all the files that are needed into a \ZIP\ file so that we only +need to ship that \ZIP\ file and a binary. + + +\subject{font readers} + +Handling \OPENTYPE\ involves more that just loading yet another font format. Of course +loading an \OPENTYPE\ file is a necessity but we need to do more. Such fonts come with +features. Features can involve replacing one representation of a character by another +one of combining sequences into other sequences and finaly resolving them to one or more +glyphs. + +Given the numerous options we will have to spend quite some time on extending \CONTEXT\ +with new features. Instead of defining more and more font instances (the traditional \TEX\ way +of doing things) we will will provides feature switching. In the end this will make +the often confusing font mechanisms less complex for the user to understand. Instead of +for instance loading an extra font (set) that provides old style numerals, we will +decouple this completely from fonts and provide it as yet another property of a piece +of text. The good news is that much of the most important machinery is alresady in +place (ligature building and such). Here we also have to decide what we let \TEX\ do +and what we do by processing node lists. For instance kerning and ligature building +can either be done by \TEX\ or by \LUA. Given the fact that \TEX\ does some juggling +with character kerning while determining hyphenation points, we can as well disable +\TEX's kerning and let \LUA\ handle it. Thereby \TEX\ only has to deal with paragraph +building. (After all, we need to leave \TEX\ some core functionality to deal with.) + +Another everlasting burden on macro writers and users is dealing with character +representations missing from a font. Of course, since we use named glyphs in +\CONTEXT\ \MKII\ already much of this can be hidden, but in \MKIV\ we can +create virtual fonts on the fly and keep thinking in terms of characters and +glyphs instead of dealing with boxes and other structures that don't go well with +for instance hyphenating words. + +This brings us to hyphenation, historically bound to fonts in traditional \TEX. This +dependency will go away. In \MKII\ we already ship \UTF8\ based patterns fore some time +and these can be conveniently used in \MKIV\ too. We experimented with using hyphenated +word lists and this looks promising. You may expect more advanced ways of dealing with +words, hyphenation and paragraph building in the near future. When we presented the +first version of \LUATEX\ a few years ago, we only had the basic \type {\directlua} call +available and could do a bit of string manipulation on the input. A fancy demo was to +color wrongly spelled words. Now we can do that more robustly on the node lists. + +Loading and preparing fonts for usage in \LUATEX\ or actually \MKIV\ because this depends +on the macro package takes some runtime. For this reason we introduces caching +into \MKIV: data that is used frequently is written to a cache and converted to \LUA\ +bytecode. Loading the converted files is incredibly fast. Of course there is aprice to +pay: disk space, but that comes cheap these days. Also, it may as well be compensated +by the fact that we can kick out many redundant files from the core \TEX\ distributions +(metric files for instance). + +\subject{tokens handlers} + +Do we need to handle tokens? So far in experimental \MKIV\ code we only used these hooks +to demonstrate what \TEX\ does with your characters. For a while we also constructed +token lists when we wanted to inject \type {\pdfliteral} code in node lists, but that +became obsolete when automatic string to token conversion was introduced in the node +conversion code. Now we inject literal whatsit nodes. It may be worth noticing that +playing with token lists gave us some good insight in bottlenecks because quite some +small table allocation and garbage collections goes on. + +\subject{nodes and attributes} + +These are the most promissing new features. In itself, nodes are not new, nor are +attributes. In some sense when we use primitives like \type {\hbox}, \type {\vskip}, +\type {\lastpenalty} the result is a node, but we can only control and inspect their +properties within hard coded bounds. We cannot really look into boxes, and the last +penalty may be obscured by a whatsit (a mark, a special, a write, etc.). Attributes +could be fakes with marks and macro bases stacks of states. Native attributes +are more powerful and each node can cary a truckload of them. + +With \LUATEX, out of a sudden we can look into \TEX's internals and manipulate +them. Although I don't claim to be a real expert on these internals, even after +over a decade of \TEX\ programming, I'm sometimes surprised what I found there. +When we are playing with these interfaces, we often run into situations +where we need to add much print statements to the \LUA\ code in order to find +out what \TEX\ is returning. It all has to do with the way \TEX\ collects +information and when it decides to act. In regular \TEX\ much goes unnoticed, but +when one has for instance a callback that deals with page building there are many +places where this gets called and some of these places need special treatment. + +Undoubtely this will have a huge impact on \CONTEXT\ \MKIV. Instead of parsing +an input stream, we can now manipulate node lists in order to achieve (slight) +inter||character spacing which is often needed in sectioning titles. The nice +thing about this new approach is that we no longer have interference from +characters that need multiple tokens (input characters) in order to be +constructed, which complicates parsing (needed to split glyphs in \MKII). + +Signaling where to letterspace is done with the mentioned attributes. There can be +many of them and they behave like fonts: they obey grouping, travel with the nodes +and are therefore insensitive for box and page splitting. They can be set at the +\TEX\ end but needs to be handled at the \LUA\ side. One may wonder what kind +of macro packages would be around when \TEX\ has attributes right from its start. + +In \MKII\ letterspacing is handled by parsing the input and injecting skips. +Another approach would be to use a font where each character has more kerns or space +around it (a virtual font can do that). But that would not only demand knowledge of +what fonts need that that treatment, but also many more fonts and generating them is +no fun for users. In \PDFTEX\ there is a letterspace feature, where virtual fonts +are generated on the fly, and with such an approach one has to compensate for the +first and last character in a line, in order to get rid of the left- and +rightmost added space (being part of the glyph). The solution where nodes are +manipulated does put that burden upon the user. + +Another example of node processing is adding specific kerns around some punctuation +symbols, as is custom in French. You don't want to know what it takes to do that +in traditional \TEX, but if I mention the fact that colons become active characters +you can imagine the nightmare. Hours of hacking and maybe even days of dealing with +mechanisms that make these active colons workable in places where colons are used +for non text are now even more wasted time if you consider that it takes a few lines +of code in \MKIV. Currently we let \CONTEXT\ support both good old \TEX\ +(represented by \PDFTEX), \XETEX\ (a \UNICODE\ and \OPENTYPE\ aware variant) and +\LUATEX\ by shared and dedicated \MKII\ and \MKIV\ code. + +Vertical spacing can be a pain. Okay, currently \MKII\ has a rather sophisticated way to +deal with vertical spacing in ways that give documents a consistent look and feel, but +every now and then we run into border cases that cannot be dealt with simply because +we cannot look back in time. This is needed because \TEX\ adds content to the main +vertical list and then it's gone from our view. Take for instance section titles. We don't +want them dangling at the bottom of a page. But at the same time we want itemized lists +to look well, i.e.\ keep items together in some situations. Graphics that follow a section +title pose similar problems. Adding penalties helps but these may come too late, or +even worse, they may obscure previous skips which then cannot be dealt with by successive +skips. To simplify the problem: take a skip of 12pt, followed by a penalty, followed by +another skip of 24pt. In \CONTEXT\ this has to become a penalty followed by one skip +of 24pt. + +Dealing with this in the page builder is rather easy. Ok, due to the way \TEX\ adds +content to the page stream, we need to collect, treat and flush, but currently this +works all right. In \CONTEXT\ \MKIV\ we will have skips with three additional properties: +priority over other skips, penalties, and a category (think of: ignore, force, +replace, add). + +When we experimented with this kind of things we quickly decided that additional +experiments with grid snapping also made sense. These mechanisms are among the more +complex ones on \CONTEXT. A simple snap feature took a few lines of \LUA\ code and +hooking it into \MKIV\ was not that complex either. Eventually we will reimplement +all vertical spacing and grid snapping code of \MKII\ in \LUA. Because one of +\CONTEXT\ column mechanism is grid aware, we may as well adath that and|/|or implement +an additional mechanism. + +A side effect of being able to do this in \LUATEX\ is that the code taken from \PDFTEX\ +is cleaned up: all (recently added) static kerning code is removed (inter||character +spacing, pre- and post character kerning, experimental code that can fix the heights +and depths of lines, etc.). The core engine will only deal with dynamic features, +like \HZ\ and protruding. + +So, the impact on \MKIV\ of nodes and attributes is pretty big! Horizontal spacing isues, +vertical spacing, grid snapping are just a few of the things we will reimplement. Other +things are line numbering, multiple content streams with synchronization, both are +already present in \MKII\ but we can do a better job in \MKIV. + +\subject{generic code} + +In the previous text \MKIV\ was mentioned often, but some of the features are rather +generic in nature. So, how generic can interfaces be implemented? When the \MKIV\ code +has matured, much of the \LUA\ and glue||to||\TEX\ code will be generic in nature. +Eventually \CONTEXT\ will become a top layer on what we internally call \METATEX, a +collection of kernel modules that one can use to build specialized macro packages. +To some extent \METATEX\ can be for \LUATEX\ what plain is for \TEX. But if and how +fast this will be reality depends on the amount of time that we (and other members of +the \CONTEXT\ development team) can allocate to this. + +\stopcomponent |