diff options
Diffstat (limited to 'Master/texmf-dist/doc/context/sources/general/manuals/mk/mk-halfway.tex')
-rw-r--r-- | Master/texmf-dist/doc/context/sources/general/manuals/mk/mk-halfway.tex | 377 |
1 files changed, 377 insertions, 0 deletions
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/mk/mk-halfway.tex b/Master/texmf-dist/doc/context/sources/general/manuals/mk/mk-halfway.tex new file mode 100644 index 00000000000..badb5d42836 --- /dev/null +++ b/Master/texmf-dist/doc/context/sources/general/manuals/mk/mk-halfway.tex @@ -0,0 +1,377 @@ +% language=uk + +\environment mk-environment + +\startcomponent mk-halfway + +\chapter{Halfway} + +\subject{introduction} + +We are about halfway into the \LUATEX\ project now. At the time of +writing this document we are only a few days away from version +0.40 (the Bacho\TEX\ cq.\ \TEX Live version) and around euro\TEX\ +2009 we will release version 0.50. Starting with version 0.30 +(which we released around the conference of the Korean \TEX\ User +group meeting) all one-decimal releases are supported and usable +for (controlled) production work. We have always stated that all +interfaces may change until they are documented to be stable, and +we expect to document the first stable parts in version 0.50. +Currently we plan to release version 1.00 sometime in 2012, 30 +years after \TEX82, with 0.60 and 0.70 in 2010, 0.80 and 0.90 in +2011. But of course it might turn out different. + +In this update we assume that the reader knows what \LUATEX\ is and +what it does. + +\subject{design principles} + +We started this project because we wanted an extensible engine. +We chose \LUA\ as the glue language. We do not regret this choice as it +permitted us to open up \TEX's internals reasonably well. There have been +a few extensions to \TEX\ itself, and there will be a few more, but none +of them are fundamental in the sense that they influence + +typesetting. Extending \TEX\ in that area is up to the macro package +writer, who can use the \LUA\ language combined with \TEX\ macros. In a +similar fashion we made some decisions about \LUA\ libraries that are +included. What we have now is what you will get. Future versions of +\LUATEX\ will have the ability to load additional libraries but these +will not be part of the core distribution. There is simply too much +choice and we do not want to enter endless discussions about what is +best. More flexibility would also add a burden on maintenance that we +do not want. Portability has always been a virtue of \TEX\ and we want +to keep it that way. + +\subject{lua scripting} + +Before 0.40 there could be multiple instances of the \LUA\ interpreter +active at the same time, but we have now decided to limit the number of +instances to just one. The reason is simple: sharing all functionality +among multiple \LUA\ interpreter instances does more bad than good and +\LUA\ has enough possibilities to create namespaces anyway. The new +limit also simplifies the internal source code, which is a good +thing. While the \type {\directlua} command is now sort of frozen, we +might extend the functionality of \type {\latelua}, especially in +relation to what is possible in the backend. Both commands still +accept a number but this now refers to an index in a user||definable +name table that will be shown when an error occurs. + +\subject {input and output} + +The current \LUATEX\ release permits multiple instances of \KPSE\ +which can be handy if you mix, for instance, a macro package and +\MPLIB, as both have their own \quote{progname} (and engine) namespace. +However, right from the start it has been possible to bring most input +under \LUA\ control and one can overload the usual \KPSE\ +mechanisms. This is what we do in \CONTEXT\ (and probably only there). + +Logging, etc., is also under \LUA\ control. There is no support for +writing to \TEX's opened output channels except for the log and the +terminal. We are investigating limited write control to numbered +channels but this has a very low priority. + +Reading from zip files and sockets has been available +for a while now. + +Among the first things that have been implemented is a mechanism for +managing category codes (\type{\catcode}) although this is not really +needed for practical usage as we aim at full compatibility. It just +makes printing back to \TEX\ from \LUA\ a bit more comfortable. + +\subject {interface to tex} + +Registers can always be accessed from \LUA\ by number and (when +defined at the \TEX\ end) also by name. When writing to a register +grouping is honored. Most internal registers can be accessed +(mostly read-only). Box registers can be manipulated but users +need to be aware of potential memory management issues. + +There will be provisions to use the primitives related to setting +codes (lowercase codes and such). Some of this functionality will be +available in version 0.50. + +\subject {fonts} + +The internal font model has been extended to the full \UNICODE\ +range. There are readers for \OPENTYPE, \TYPEONE, and traditional +\TEX\ fonts. Users can create virtual fonts on the fly and have +complete control over what goes into \TEX. Font specific features +can either be mapped onto the traditional ligature and kerning +mechanisms or be implemented in \LUA. + +We use code from \FONTFORGE\ that has been stripped to get a +smaller code base. Using the \FONTFORGE\ code has the advantage +that we get a similar view on the fonts in \LUATEX\ as in this +editor which makes debugging easier and developing fonts more +convenient. + +The interface is already rather stable but some of the keys in loaded +tables might change. Almost all of the font interface will be stable +in version 0.50. + +\subject {tokens} + +It is possible to intercept tokenization. Once intercepted, a token +table can be manipulated before being piped back into \LUATEX. We +still support \OMEGA's translation processes but that might become +obsolete at some point. + +Future versions of \LUATEX\ might use \LUA's so-called \quote {user data} +concept but the interface will mostly be the same. Therefore this +subsystem will not be frozen yet in version 0.50. + +\subject {nodes} + +Users have access to the node lists in various stages. This interface +has already been quite stable for some time but some cleanup might +still take place. Currently the node memory maintenance is still +explicit, but eventually we will make releasing unused nodes automatic. + +We have plans for keeping more extensive information within +a paragraph (initial whatsit) so that one can build alternative +paragraph builders in \LUA. There will be a vertical packer (in +addition to the horizontal packer) and we will open up the page +builder (inserts etc.). The basic interface will be stable in version +0.50. + +\subject {attributes} + +This new kid on the block is now available for most subsystems but +we might change some of its default behaviour. As of 0.40 you can +also use negative values for attributes. The original idea of +using negative values for special purposes has been abandoned as +we consider a secondary (faster and more efficient) limited +variant. The basic principles will be stable around version 0.50, +but we reserve the freedom to change some aspects of attributes +until we reach version 1.00. + +\subject {hyphenation} + +In \LUATEX\ we have clearly separated hyphenation, ligature +building and kerning. Managing patterns as well as hyphenation is +reimplemented from scratch but uses the same principles as +traditional \TEX. Patterns can be loaded at run time and exceptions +are quite efficient now. There are a few extensions, like embedded +discretionaries in exceptions and pre- as well as posthyphens. + +On the agenda is fixing some \quote{hyphenchar} related issues and future +releases might deal with compound words as well. There are some +known limitations that we hope to have solved in version 0.50. + +\subject {images} + +Image handling is part of the backend. This part of the \PDFTEX\ +code has been rewritten and can now be controlled from \LUA. There +are already a few more options than in \PDFTEX\ (simple +transformations). The image code will also be integrated in the +virtual font handler. + +\subject {paragraph building} + +The paragraph builder has been rewritten in \CCODE\ (soon to be +converted back to \CWEB). There is a callback related to the builder +so it is possible to overload the default line breaker by one written +in \LUA. + +There are no further short|-|term revisions on the agenda, apart from +writing an advanced (third order) Arabic routine for the Oriental +\TEX\ project. + +Future releases may provide a bit more control over \type{\parshape}s +and multiple paragraph shapes. + +\subject {metapost} + +The closely related \MPLIB\ project has resulted in a \METAPOST\ +library that is included in \LUATEX. There can be multiple +instances active at the same time and \METAPOST\ processing is +very fast. Conversion to \PDF\ is to be done with \LUA. + +On the to-do list is a bit more interoperability (pre- and +postscript tables) and this will make it into release 0.50 +(maybe even in version 0.40 already). + +\subject {mathematics} + +Version 0.50 will have a stable version of \UNICODE\ +math support. Math is backward compatible but provides solutions +for dealing with \OPENTYPE\ math fonts. We provide math lists in +their intermediate form (noads) so that it is possible to +manipulate math in great detail. + +The relevant math parameters are reorganized according to what +\OPENTYPE\ math provides (we use the Cambria font as our reference). Parameters +are grouped by style. Future versions of \LUATEX\ will build upon +this base to provide a simple mechanism for switching style sets +and font families in-formula. + +There are new primitives for placing accents (top and bottom +variants and extensible characters), creating radicals, and making +delimiters. Math characters are permitted in text mode. + +There will be an additional alignment mechanism analogous to +what \MATHML\ provides. Expect more. + +\subject {page building} + +Not much work has been done on opening up the page builder +although we do have access to the intermediate lists. This +is unlikely to happen before 0.50. + +\subject {going cweb} + +After releasing version 0.50 around Euro\TEX\ 2009 there will be a +period of relative silence. Apart from bug fixes and (private) +experiments there will be no release for a while. At the time of the +0.50 release the \LUATEX\ source code will probably be in plain C +completely. After that is done, we will concentrate hard on +consolidating and upgrading the code base back into \CWEB. + +\subject {cleanup} + +Cleanup of code is a continuous process. Cleanup is needed because +we deal with a merge of traditional \TEX, \ETEX\ extensions, +\PDFTEX\ functionality and some \OMEGA\ (\ALEPH) code. + +Compatibility is a prerequisite, with the exception of logging and +rather special ligature reconstruction code. + +We also use the opportunity to slowly move away from all the global +variables that are used in the \PASCAL\ version. + +\subject {alignments} + +We do have some ideas about opening up alignments, but it has a +low priority and it will not happen before the 0.50 release. + +\subject {error handling} + +Once all code is converted to \CWEB, we will look into error +handling and recovery. It has no high priority as it is easier to +deal with after the conversion to \CWEB. + +\subject {backend} + +The backend code will be rewritten stepwise. The image related +code has already been redone, and currently everything related to +positioning and directions is redesigned and made more consistent. +Some bugs in the \ALEPH\ code (inherited from \OMEGA) have been +removed and we are trying to come up with a consistent way of dealing +with directions. Conceptually this is somewhat messy because much +directionality is delegated to the backend. + +We are experimenting with positioning (preroll) and better literal +injection. Currently we still use the somewhat fuzzy \PDFTEX\ methods +that evolved over time (direct, page and normal injection) but we +will come up with a clearer model. + +Accuracy of the output (\PDF) will be improved and character +extension (hz) will be done more efficiently. Experimental code +seems to work okay. This will become available from release 0.40 +and onwards and further cleanup will take place when the \CWEB\ +code is there, as much of the \PDF\ backend code is already \CCODE. + +\subject{context mkiv} + +When we started with \LUATEX\ we decided to use a branch of +\CONTEXT\ for testing as it involves quite drastic changes, many +rewrites, a tight connection with binary versions, etc. + +As a result for some time we now have two versions of \CONTEXT: \MKII\ +and \MKIV, where the former targets \PDFTEX\ and \XETEX, and +the latter exclusively uses \LUATEX. Although the user interface +is downward compatible the code base starts to diverge more and +more. Therefore at the last \CONTEXT\ meeting it was decided to +freeze the current version of \MKII\ and only apply bug fixes +and an occasional simple extension. + +This policy change opened the road to rather drastic splitting of the +code, also because full compatibility between \MKII\ and \MKIV\ is not +required. Around \LUATEX\ version 0.40 the new, currently still +experimental, document structure related code will be merged into the +regular \MKIV\ version. This might have some impact as it opens up new +possibilities. + +\subject {the future} + +In the future, \MKIV\ will try to create (more) clearly separated +layers of functionality so that it will become possible to make +subsets of \CONTEXT\ for special purposes. This is done under the name +\METATEX. Think of layering like: + +\startitemize[packed] +\item \IO, catcodes, callback management, helpers +\item input regimes, characters, filtering +\item nodes, attributes and noads +\item user interface +\item languages, scripts, fonts and math +\item spacing, par building and page construction +\item \XML, graphics, \METAPOST, job management, and structure (huge impact) +\item modules, styles, specific features +\item tools +\stopitemize + +\subject{fonts} + +At this moment \MKIV\ is already quite capable of dealing with +\OPENTYPE\ fonts. The driving force behind this is the Oriental +\TEX\ project which brings along some very complex and feature +rich Arabic font technology. Much time has gone into reverse +engineering the specification and behaviour of how these fonts +behave in Uniscribe (which we use as our reference for Arabic). + +Dealing with the huge \CJK\ fonts is less a font issue and more +a matter of node list processing. Around the annual meeting of +the Korean User Group we got much of the machinery working, thanks +to discussions on the spot and on the mailing list. + +\subject {math} + +Between \LUATEX\ versions 0.30 and 0.40 the math machinery was opened +up (stage one). In order to test this new functionality, \MKIV's math +subsystem (that was then already partially \UNICODE\ aware) had to be +adapted. + +First of all \UNICODE\ permits us to use only one math family and so +\MKIV\ now does that. The implementation uses Microsoft's Cambria +Math font as a benchmark. It creates virtual fonts from the other (old +and new) math fonts so they appear to match up to Cambria +Math. Because the \TEX\ Gyre math project is not yet up to speed \MKIV\ +currently uses virtual variants of these fonts that are created at +run time. The missing pieces in for instance Latin Modern and friends +are compensated for by means of virtual characters. + +Because it is now possible to parse the intermediate noad lists \MKIV\ can +do some manipulations before the formula is typeset. This is for +instance used for alphabet remapping, forcing sizes, and spacing +around punctuation. + +Although \MKIV\ already supports most of the math that users expect +there is still room for improvement once there is even more control +over the machinery. This is possible because \MKIV\ is not bound to +downward compatibility. + +As with all other \LUATEX\ related \MKIV\ code, it is expected that we +will have to rewrite most of the current code a few times as we +proceed, so \MKIV\ math support is not yet stable either. We can take +such drastic measures because \MKIV\ is still experimental and because +users are willing to do frequent synchronous updating of macros and +engine. In the process we hope to get away from all ad||hoc boxing and +kerning and whatever solutions for creating constructs, by using +the new accent, delimiter, and radical primitives. + +\subject {tracing and testing} + +Whenever possible we add tracing and visualization features to +\CONTEXT\ because the progress reports and articles need them. Recent +extensions concerned tracing math and tracing \OPENTYPE\ processing. + +The \OPENTYPE\ tracing options are a great help in stepwise +reaching the goals of the Oriental \TEX\ project. This project +gave the \LUATEX\ project its initial boost and aims at high +quality right|-|to|-|left typesetting. In the process complex (test) +fonts are made which, combined with the tracing mentioned, help us +to reveal the secrets of \OPENTYPE. + +\stopcomponent |