summaryrefslogtreecommitdiff
path: root/Master/texmf-dist/doc/context/sources/general/manuals/musings/musings-roadmap.tex
diff options
context:
space:
mode:
Diffstat (limited to 'Master/texmf-dist/doc/context/sources/general/manuals/musings/musings-roadmap.tex')
-rw-r--r--Master/texmf-dist/doc/context/sources/general/manuals/musings/musings-roadmap.tex372
1 files changed, 0 insertions, 372 deletions
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/musings/musings-roadmap.tex b/Master/texmf-dist/doc/context/sources/general/manuals/musings/musings-roadmap.tex
deleted file mode 100644
index f8771ba426b..00000000000
--- a/Master/texmf-dist/doc/context/sources/general/manuals/musings/musings-roadmap.tex
+++ /dev/null
@@ -1,372 +0,0 @@
-% language=uk
-
-% \showfontkerns
-
-\startcomponent musings-roadmap
-
-\environment musings-style
-
-\startchapter[title={\METATEX, a roadmap}]
-
-% \startlines \setupalign[flushright]
-% Hans Hagen
-% Hasselt NL
-% September 2018
-% \stoplines
-
-\startsection[title={Introduction}]
-
-Here I will shortly wrap up the state of \LUATEX\ and \CONTEXT\ in fall 2018. I
-made the first draft of this article as preparation for the \CONTEXT\ meeting
-where we also discussed the future. I updated the text afterwards to match the
-decisions made there. It's also a personal summary of thoughts and discussions
-with team members about where to move next.
-
-\stopsection
-
-\startsection[title={The state of affairs}]
-
-After a dozen years the development of \LUATEX\ has reached a state where adding
-more functionality and|/|or opening up more of the internals makes not much
-sense. Apart from fixes and maybe some minor extensions, version 1.10 is what you
-get. Users can do enough in \LUA\ and there is not much to gain in convenience
-and performance. Of course some of the code can and will be cleaned up, as we
-still see the effects of going from \PASCAL\ to \CWEB\ to \CCODE. In the process
-consistency is on the radar so we might occasionally add a helper. But we also
-don't want to move too far away from the original code, which is for instance why
-we keep names, keys and other properties found in original \TEX, which in turn
-leads to some inconsistencies with extensions added over time. We have to accept
-that.
-
-Because \LUATEX\ development is closely related to \CONTEXT\ development,
-especially \MKIV, we've also reached the moment that we can get rid of some older
-code and assume the latest \LUATEX\ to be used. Because we do so much in \LUA\
-the question is always to what extent the benefits outweigh the drawbacks. Just
-in case you wonder why we use \LUA\ extensively, the main reason is that it is
-easier and more efficient to manage data in this language and modern typesetting
-needs much data. It also permits us to extend regular \TEX\ functionality. But,
-one should not overrate the impact: we still let \TEX\ do what \TEX\ is best at!
-
-Performance is quite important. It doesn't make sense to create a powerful
-typesetting system where processing a page takes a second. We have discussed
-performance before since one of the complaints about \LUATEX\ is that it is slow.
-A simple, basic test is this:
-
-\starttyping
-\starttext
- \dorecurse{1000}{\input tufte \par}
-\stoptext
-\stoptyping
-
-This involves 1000 times loading a file (and reporting that on the console, which
-can influence runtime), typesetting paragraphs, splitting of a page and of course
-loading fonts and saving to the \PDF\ file. When I run this on a modest machine,
-I get these (relative) timings for the (about) 225 pages:
-
-\starttabulate[|l|c|c|c|c|]
-\BC \TEX\ engine used \BC \PDFTEX \BC \LUATEX \BC \LUAJITTEX \BC \XETEX \NC \NR
-\BC runtime in seconds \NC 2.0 \NC 3.9 \NC 3.0 \NC 8.4 \NC \NR
-\stoptabulate
-
-Now, as expected the 8 bit \PDFTEX\ is the winner here but \LUATEX\ is not doing
-that bad. I don't know why \XETEX\ is so much slower, maybe because its 64 bit
-binary is less optimal. I once noticed that a 64 bit \PDFTEX\ performed worse on
-such a test than \LUATEX, for which I always use 64 bit binaries.
-
-If you consider that often much more is done than in this example, you can take
-my word that \LUATEX\ quickly outpaces \PDFTEX\ on more complex tasks. In that
-sense it is now our benchmark. It must be said that the \MKIV\ code is probably a
-bit more efficient than the \MKII\ code but that doesn't matter much in this
-simple test because hardly any macro magic happens here; it mostly tests basic
-font processing, paragraph building and page construction. I don't think that I
-can squeeze out more pages per second, at least not without users telling me
-where they encounter bottlenecks that don't result from their style coding. It's
-no problem to write inefficient macros (or styles) so normally a user should
-first carefully check her|/|his own work. Using a more modern \CPU\ with proper
-caching and an \SSD\ helps too.
-
-So, to summarize, we can say that with version 1.10 \LUATEX\ is sort of finished.
-Our mission is now to make \LUATEX\ robust and stable. Things can be added and
-improved, but these are small and mostly consistency related.
-
-\stopsection
-
-\startsection[title={More in \LUA}]
-
-Till now I always managed to add functionality to \CONTEXT\ without hampering
-performance too much. Of course the biggest challenge is always in handling fonts
-and common features like color because that all happens in \LUA. So, the question
-is, what if we delegate more of the core functionality to \LUA ? I will discuss a
-few options because the \CONTEXT\ developers and users need to agree on the path
-to follow. One question there is, are the possible performance hits (which can be
-an inconvenience) compensated by better and easier typesetting.
-
-Fonts, colors, special typesetting features like spaced kerning, protrusion,
-expansion, but also dropped caps, line numbering, marginal notes, tables,
-structure related things, floats and spacing are not open for much discussion.
-All the things that happen in \LUA\ combined with macros is there and will stay.
-But how about hyphenation, paragraph building and page building? And how about a
-leaner and meaner, future safe engine?
-
-Hyphenation is handled in the \TEX\ core. But in \CONTEXT\ already for years one
-can also use a \LUA\ based variant. There is room for extensions and improvements
-there. Interesting is that performance is more or less the same, so this is an
-area where we might switch to the \LUA\ method eventually. It compares to fonts,
-where node mode is more or less the standard and base mode the old way.
-
-Building the paragraphs in \LUA\ is also available in \MKIV, although it needs an
-update. Again performance is not that bad, so when we add features not possible
-(or hard to do) in regular \TEX, it might actually pay of to default to the
-par builder written in \LUA.
-
-The page builder is also doable in \LUA\ but so far I only played a bit with a
-\LUA\ based variant. I might pick up that thread. However, when we would switch to
-\LUA\ there, it might have a bit of a penalty, unless we combine it with some
-other mechanisms which is not entirely trivial, as it would mean a diversion from
-the way \TEX\ does it normally.
-
-How about math? We could at some point do math rendering in \LUA\ but because the
-core mechanism is the standard, it doesn't really makes much sense. It would also
-touch the soul of \TEX. But, I might give it a try, just for fun, so that I can
-play with it a bit. It's typically something for cold and rainy days with some
-music in the background.
-
-We already use \LUA\ in the frontend: locating and reading files in \TEX,
-\XML, \LUA\ and whatever input format. Normalization and manipulation is all
-active and available. The backend is also depending on \LUA, like support for
-special \PDF\ features and exporting to \XML . The engine still handles the page
-stream conversion, font inclusion and object management.
-
-The inclusion of images is also handled by the engine, although in \CONTEXT\ we
-can delegate \PDF\ inclusion to \LUA. Interesting is that this has no performance
-hit.
-
-With some juggling the page stream conversion can also be done in \LUA, and I
-might move that code into the \CONTEXT\ distribution. Here we do have a
-performance hit: about one second more runtime on the 14 seconds needed for the
-300 page \LUATEX\ manual and just over more than half a second on a 11 second
-\LUAJITTEX\ run. The manual has lots of tables, verbatim, indices and uses color
-as well as a more than average number of fonts and much time is spent in \LUA. So
-there is a price to pay there. I tried to speed that up but there is not much to
-gain there.
-
-So, say that we default to \LUA\ based hyphenation, which enables some new
-functionality, \LUA\ based par building, which permits some heuristics for corner
-cases, and \LUA\ based page building, which might result in more control over
-tricky cases. A total performance hit of some 5\% is probably acceptable,
-especially because by that time I might have replaced my laptop and won't notice
-the degrade. This still fits in the normal progress and doesn't really demand a
-roadmap or wider acceptance. And of course we would still use the same strategies
-as implemented in traditional \TEX\ as default anyway.
-
-\stopsection
-
-\startsection[title={A more drastic move}]
-
-More fundamental is the question whether we delegate more backend activity to
-\LUA\ code. If we decide to handle the page stream in \LUA, then the next
-question is, why not also delegate object management and font inclusion to
-\LUA. Now, keep in mind that this is all very \CONTEXT\ specific! Already for
-more than a decade we delegate a lot to \LUA, and also we have a rather tight
-control over this core functionality. This would mean that \CONTEXT\ doesn't
-really need the backend code in the engine. \footnote {For generic packages like
-TikZ we (can) provide some primitive emulators, which is rather trivial to
-implement.}
-
-That situation is actually not unique. For instance, already for a while we don't
-need the \LUATEX\ font loader either, as loading the \OPENTYPE\ files is done in
-\LUA. So, we could also get rid of the font loader code. Currently some code is
-shared with the font inclusion in the backend but that can be isolated.
-
-You can see a \TEX\ engine as being made from several parts, but the core really
-concerns only two processes: reading, storing and expanding macros on the one
-hand, and converting a stream of characters into lines, paragraphs, pages etc.
-Fonts are mostly an abstraction: they are visible in so called glyph nodes as
-font identifier (a number) and character code (also a number) properties. The
-result, nowadays being \PDF, is also an abstraction: at some point the engine
-converts the to be shipped out box in \PDF\ instructions, and in our case,
-relatively simple ones. The backend registers which characters and fonts are used
-and also includes the right resources. But, the backend is not part of the core
-as such! It has been introduced in \PDFTEX\ and is a so called extension.
-
-So, what does that all mean for a future version of \CONTEXT\ and \LUATEX ? It
-means that we can decide to follow up with a \CONTEXT\ that does more in \LUA,
-which means not hard coded in a binary, on the one hand, but that we can also
-decide to strip the engine from non|-|core code. But, given that \LUATEX\ is also
-used in other macro packages, this would mean a different engine. We cannot say
-that \LUATEX\ is stable when we also experiment with core components.
-
-We've seen folks picking up experimental versions assuming that it is a precursor
-to official code. So, in order to move on we need to avoid confusion: we need to
-use another name. Choosing a name is always tricky but as Taco already registered
-the \METATEX\ domain, and because in the \CONTEXT\ distribution you will find
-references to \METATEX, we will use that name for the future engine. Adding \LUA\
-to that name makes sense but then the name would become too long.
-
-The main difference between \METATEX\ and \LUATEX\ would be that the former has
-no file lookup library, no hardcoded font loader, and no backend generator (but
-possibly some helpers, and these need time to evolve). We're basically back where
-\TEX\ started but instead of coding these extensions in \PASCAL\ or \CCODE\ we
-use \LUA. We're also kind of back to when we first started experimenting with
-\LUATEX\ in \CONTEXT\ where test, write and rewrite were going in parallel. But,
-as said, we cannot impose that on a wide audience.
-
-If we go for such a lean and mean follow up, then we can also do a more drastic
-cleanup of obsolete code in \CONTEXT\ (dating from \ETEX, \PDFTEX, \ALEPH, etc.).
-We then are sort of back to where it all started: we go back to the basics. This
-might mean dropping some primitives (one can define them as dummy). Of course we
-could generalize some of the \CONTEXT\ code to provide the kicked out
-functionality but would that pay of? Probably not.
-
-Just for the record: replacing the handling of macros, registers, grouping, etc.\
-to \LUA\ is not really an option as the performance hit would make a large system
-like \CONTEXT\ sort of unusable: it's no option and not even considered (although
-I must admit that I have some experimental \LUA\ based \TEX\ parser code around).
-
-It is quite likely that building \METATEX\ from source for the moment will be an
-option to the build script. But we can also decide to simplify that process,
-which is possible because we only need one binary. But in general we can assume
-that one can generate \METATEX\ and \LUATEX\ from the same source. A first step
-probably is a further isolation of the backend code. The fontloader and file
-handling code already can be made optional.
-
-Given that we only need one binary (it being \LUATEX\ or \METATEX) and nowadays
-only use \OPENTYPE\ fonts, one can even start thinking of a mini distribution,
-possibly with a zipped resource tree, something we experimented with in the early
-days of \LUATEX.
-
-Another though I have been playing with is a better separation between low level
-and high level \CONTEXT\ commands, and whether the low level layer should be more
-generic in nature (so that one can run specific packages on top of it instead of
-the whole of \CONTEXT) but that might not be worth the trouble.
-
-\stopsection
-
-\startsection[title={Interlude}]
-
-If we look at the future, it's good to also look at the past. Opening up \TEX\
-the way we did has many advantages but also potential drawbacks. It works quite
-well in \CONTEXT\ because we ship an integrated package. I don't think that there
-are many users who kick in their own callbacks. It is possible but completely up
-to the user to make sure things work out well. Performance hits, interference,
-crashes: those who interfere with the internals can sort that out themselves. I'm
-not sure how well that works out in other macro packages but it is a time bomb if
-users start doing that. Of course the documented interfaces to use \LUA\ in
-\CONTEXT\ are supported. So far I think we're not yet bitten in the tail. We keep
-this aspect out of the discussion.
-
-Another important aspect is stability of the engine. Sometimes we get suggestions
-for changes or patches that works for a specific case but for sure will have side
-effects on \CONTEXT. Just as we don't test \LATEX\ side effects, \LATEX\ users
-don't check \CONTEXT. And we're not even talking of users who expect their code
-to keep working. A tight control over the source is important but cannot be we
-will not be around for ever. This means that at some point \LUATEX\ should not be
-changed any more, even when we observe side effects we want to get rid of,
-because these side effects can be in use. This is another argument for a stripped
-down engine. The less there is to mess with, the less the mess.
-
-\stopsection
-
-\startsection[title={Audience}]
-
-So how about \CONTEXT\ itself? Of course we can make it better. We can add more
-examples and more documentation. We can try to improve support. The main question
-for us (as developers) is who actually is our audience. From the mails coming to
-the \CONTEXT\ support list it looks like a rather diverse group of users.
-
-At \TEX\ meetings there are often discussions about promoting \TEX. I can agree
-on the fact that even for simple documents it makes a lot of sense to use \TEX,
-but who will take the first hurdles? How many people really produce a lot of
-documents? And how many need \TEX\ after maybe a short period of (enforced) usage
-at the university?
-
-It's not trivial to recognize the possibilities and power of the
-\LUATEX|-|\CONTEXT\ combination. We never got any serious requests for support
-from large organizations. In fact, we do use this combination in a few projects
-for educational publishers, but there it's actually the authors and editors doing
-the work. It's seldom company policy to use tools that efficiently automate
-typesetting. I dare to say that publishers are not really an audience at all:
-they normally delegate the task. They might accept \TEX\ documents but let them
-rekey or adapt far|-|far|-|away and as cheap as possible. Thinking of it, the
-main reason for Don Knuth for writing \TEX\ in the first place was the ability to
-control the look and feel and quality. It were developments at typesetters and
-publishers that triggered development of \TEX . It was user demand. And the
-success of \TEX\ was largely due to the unique personality and competence of the
-author.
-
-System integrators qualify as audience but I fear that \TEX\ is not considered
-hip and modern. It doesn't seem to matter if you can demonstrate that it can do a
-wonderful job efficiently and relatively cheap. Also the fact that an
-installation can be very stable on the long run is of no importance. Maybe that
-audience (market place) is all about \quotation {The more we have to program and
-update regularly, the merrier.}. Marketing \TEX\ is difficult.
-
-Those who render multiple products, maintain manuals, have to render many
-documents automatically qualify as audience. But often company policies,
-preferred suppliers, so called standard tools etc.\ are used as argument against
-\TEX. It's a missed opportunity.
-
-One needs a certain mindset to recognize the potential and the question is, how
-do we reach that audience. Drawing a roadmap for that is not easy but worth
-discussing. We're open for suggestions.
-
-% \footnote {It's kind of interesting that recently the \TEX\ User Group announced
-% its presence on Facebook and Twitter. Apart from wondering how that gets updated,
-% one can also wonder how many potential (or even current) users go there, given
-% that these platforms are subjected to rise and fall. I'm on neither of them and
-% don't plan to. Kids (our future users) that I know already said goodbye to them.
-% We'll see how that works out.}
-
-\stopsection
-
-\startsection[title={Conclusion}]
-
-At the \CONTEXT\ user meeting those present agreed that moving forward this way
-makes sense. This means that we will explore a lean and mean \METATEX\ alongside
-\LUATEX. There is no rush and it's all volunteer work so we will take our time
-for this. It boils down to some reshuffling of code so that we can remove the
-built|-|in font loader, file handling, and probably also \SYNCTEX\ because we can
-emulate that. Then the backend with its font inclusion code will be cleaned up a
-bit (we even discussed only supporting modern wide fonts). It's no big deal to
-adapt \CONTEXT\ to this (so it can and will support both \LUATEX\ and \METATEX).
-Eventually the backend might go away but now we're talking years ahead. By then
-we can also explore the option to make \METATEX\ start out as a \LUA\ function
-call (the main control loop) and become reentrant. There will probably not be
-many changes to the opened up \TEX\ kernel, but we might extend the \METAPOST\
-part a bit (some of that was discussed at the meeting) especially because it is a
-nice tool to visualize big data.
-
-As with \LUATEX\ development we will go in small steps so that we keep a working
-system. Of course \LUATEX\ is always there as stable fallback. The experiments
-will mostly happen in the experimental branch and binaries will be generated
-using the compile farm on the \CONTEXT\ garden, just as happens now. This also
-limits testing and exploring to the \CONTEXT\ community so that there are no side
-effects for mainstream \LUATEX\ usage.
-
-Nowadays, instead if roadmaps, we tend to use navigational gadgets that adapt
-themselves to the situation. On the road by car this can mean a detour and when
-walking around it can be going to suggested points of interest. During the
-excursion at the meeting, we noticed that after the drivers (navigators)
-synchronized their gadget with Jano, the routes that were followed differed a
-bit. We saw cars in front of going a different direction and cars behind us
-arriving from a different direction. So, even when we talk about roadmaps, our
-route can be adapted to the situation.
-
-Now here is something to think about. If you look at the \TEX\ community you will
-notice that it's an aging community. User groups seem to loose members, although
-the \CONTEXT\ group is currently still growing. Fortunately we see a new
-generation taking interest and the \CONTEXT\ users are a pleasant mix and it
-makes me stay around. I see it as an \quote {old timers} responsibility to have
-\TEX\ and its environment in a healthy state by the time I retire from it
-(although I have no plans in that direction). In parallel to the upcoming
-development I think we will also see a change in \TEX\ use and usage. This aspect
-was also discussed at the meeting and for sure will get a follow up on the
-mailing lists and future meetings. It might as well influence the decisions we
-make the upcoming years. So far \TEX\ has never failed us in it's flexibility and
-capacity to adapt, so let's end on that positive note.
-
-\stopsection
-
-\stopchapter
-
-\stopcomponent