diff options
Diffstat (limited to 'Master/texmf-dist/doc/context/sources/general/manuals/musings/musings-roadmap.tex')
-rw-r--r-- | Master/texmf-dist/doc/context/sources/general/manuals/musings/musings-roadmap.tex | 372 |
1 files changed, 0 insertions, 372 deletions
diff --git a/Master/texmf-dist/doc/context/sources/general/manuals/musings/musings-roadmap.tex b/Master/texmf-dist/doc/context/sources/general/manuals/musings/musings-roadmap.tex deleted file mode 100644 index f8771ba426b..00000000000 --- a/Master/texmf-dist/doc/context/sources/general/manuals/musings/musings-roadmap.tex +++ /dev/null @@ -1,372 +0,0 @@ -% language=uk - -% \showfontkerns - -\startcomponent musings-roadmap - -\environment musings-style - -\startchapter[title={\METATEX, a roadmap}] - -% \startlines \setupalign[flushright] -% Hans Hagen -% Hasselt NL -% September 2018 -% \stoplines - -\startsection[title={Introduction}] - -Here I will shortly wrap up the state of \LUATEX\ and \CONTEXT\ in fall 2018. I -made the first draft of this article as preparation for the \CONTEXT\ meeting -where we also discussed the future. I updated the text afterwards to match the -decisions made there. It's also a personal summary of thoughts and discussions -with team members about where to move next. - -\stopsection - -\startsection[title={The state of affairs}] - -After a dozen years the development of \LUATEX\ has reached a state where adding -more functionality and|/|or opening up more of the internals makes not much -sense. Apart from fixes and maybe some minor extensions, version 1.10 is what you -get. Users can do enough in \LUA\ and there is not much to gain in convenience -and performance. Of course some of the code can and will be cleaned up, as we -still see the effects of going from \PASCAL\ to \CWEB\ to \CCODE. In the process -consistency is on the radar so we might occasionally add a helper. But we also -don't want to move too far away from the original code, which is for instance why -we keep names, keys and other properties found in original \TEX, which in turn -leads to some inconsistencies with extensions added over time. We have to accept -that. - -Because \LUATEX\ development is closely related to \CONTEXT\ development, -especially \MKIV, we've also reached the moment that we can get rid of some older -code and assume the latest \LUATEX\ to be used. Because we do so much in \LUA\ -the question is always to what extent the benefits outweigh the drawbacks. Just -in case you wonder why we use \LUA\ extensively, the main reason is that it is -easier and more efficient to manage data in this language and modern typesetting -needs much data. It also permits us to extend regular \TEX\ functionality. But, -one should not overrate the impact: we still let \TEX\ do what \TEX\ is best at! - -Performance is quite important. It doesn't make sense to create a powerful -typesetting system where processing a page takes a second. We have discussed -performance before since one of the complaints about \LUATEX\ is that it is slow. -A simple, basic test is this: - -\starttyping -\starttext - \dorecurse{1000}{\input tufte \par} -\stoptext -\stoptyping - -This involves 1000 times loading a file (and reporting that on the console, which -can influence runtime), typesetting paragraphs, splitting of a page and of course -loading fonts and saving to the \PDF\ file. When I run this on a modest machine, -I get these (relative) timings for the (about) 225 pages: - -\starttabulate[|l|c|c|c|c|] -\BC \TEX\ engine used \BC \PDFTEX \BC \LUATEX \BC \LUAJITTEX \BC \XETEX \NC \NR -\BC runtime in seconds \NC 2.0 \NC 3.9 \NC 3.0 \NC 8.4 \NC \NR -\stoptabulate - -Now, as expected the 8 bit \PDFTEX\ is the winner here but \LUATEX\ is not doing -that bad. I don't know why \XETEX\ is so much slower, maybe because its 64 bit -binary is less optimal. I once noticed that a 64 bit \PDFTEX\ performed worse on -such a test than \LUATEX, for which I always use 64 bit binaries. - -If you consider that often much more is done than in this example, you can take -my word that \LUATEX\ quickly outpaces \PDFTEX\ on more complex tasks. In that -sense it is now our benchmark. It must be said that the \MKIV\ code is probably a -bit more efficient than the \MKII\ code but that doesn't matter much in this -simple test because hardly any macro magic happens here; it mostly tests basic -font processing, paragraph building and page construction. I don't think that I -can squeeze out more pages per second, at least not without users telling me -where they encounter bottlenecks that don't result from their style coding. It's -no problem to write inefficient macros (or styles) so normally a user should -first carefully check her|/|his own work. Using a more modern \CPU\ with proper -caching and an \SSD\ helps too. - -So, to summarize, we can say that with version 1.10 \LUATEX\ is sort of finished. -Our mission is now to make \LUATEX\ robust and stable. Things can be added and -improved, but these are small and mostly consistency related. - -\stopsection - -\startsection[title={More in \LUA}] - -Till now I always managed to add functionality to \CONTEXT\ without hampering -performance too much. Of course the biggest challenge is always in handling fonts -and common features like color because that all happens in \LUA. So, the question -is, what if we delegate more of the core functionality to \LUA ? I will discuss a -few options because the \CONTEXT\ developers and users need to agree on the path -to follow. One question there is, are the possible performance hits (which can be -an inconvenience) compensated by better and easier typesetting. - -Fonts, colors, special typesetting features like spaced kerning, protrusion, -expansion, but also dropped caps, line numbering, marginal notes, tables, -structure related things, floats and spacing are not open for much discussion. -All the things that happen in \LUA\ combined with macros is there and will stay. -But how about hyphenation, paragraph building and page building? And how about a -leaner and meaner, future safe engine? - -Hyphenation is handled in the \TEX\ core. But in \CONTEXT\ already for years one -can also use a \LUA\ based variant. There is room for extensions and improvements -there. Interesting is that performance is more or less the same, so this is an -area where we might switch to the \LUA\ method eventually. It compares to fonts, -where node mode is more or less the standard and base mode the old way. - -Building the paragraphs in \LUA\ is also available in \MKIV, although it needs an -update. Again performance is not that bad, so when we add features not possible -(or hard to do) in regular \TEX, it might actually pay of to default to the -par builder written in \LUA. - -The page builder is also doable in \LUA\ but so far I only played a bit with a -\LUA\ based variant. I might pick up that thread. However, when we would switch to -\LUA\ there, it might have a bit of a penalty, unless we combine it with some -other mechanisms which is not entirely trivial, as it would mean a diversion from -the way \TEX\ does it normally. - -How about math? We could at some point do math rendering in \LUA\ but because the -core mechanism is the standard, it doesn't really makes much sense. It would also -touch the soul of \TEX. But, I might give it a try, just for fun, so that I can -play with it a bit. It's typically something for cold and rainy days with some -music in the background. - -We already use \LUA\ in the frontend: locating and reading files in \TEX, -\XML, \LUA\ and whatever input format. Normalization and manipulation is all -active and available. The backend is also depending on \LUA, like support for -special \PDF\ features and exporting to \XML . The engine still handles the page -stream conversion, font inclusion and object management. - -The inclusion of images is also handled by the engine, although in \CONTEXT\ we -can delegate \PDF\ inclusion to \LUA. Interesting is that this has no performance -hit. - -With some juggling the page stream conversion can also be done in \LUA, and I -might move that code into the \CONTEXT\ distribution. Here we do have a -performance hit: about one second more runtime on the 14 seconds needed for the -300 page \LUATEX\ manual and just over more than half a second on a 11 second -\LUAJITTEX\ run. The manual has lots of tables, verbatim, indices and uses color -as well as a more than average number of fonts and much time is spent in \LUA. So -there is a price to pay there. I tried to speed that up but there is not much to -gain there. - -So, say that we default to \LUA\ based hyphenation, which enables some new -functionality, \LUA\ based par building, which permits some heuristics for corner -cases, and \LUA\ based page building, which might result in more control over -tricky cases. A total performance hit of some 5\% is probably acceptable, -especially because by that time I might have replaced my laptop and won't notice -the degrade. This still fits in the normal progress and doesn't really demand a -roadmap or wider acceptance. And of course we would still use the same strategies -as implemented in traditional \TEX\ as default anyway. - -\stopsection - -\startsection[title={A more drastic move}] - -More fundamental is the question whether we delegate more backend activity to -\LUA\ code. If we decide to handle the page stream in \LUA, then the next -question is, why not also delegate object management and font inclusion to -\LUA. Now, keep in mind that this is all very \CONTEXT\ specific! Already for -more than a decade we delegate a lot to \LUA, and also we have a rather tight -control over this core functionality. This would mean that \CONTEXT\ doesn't -really need the backend code in the engine. \footnote {For generic packages like -TikZ we (can) provide some primitive emulators, which is rather trivial to -implement.} - -That situation is actually not unique. For instance, already for a while we don't -need the \LUATEX\ font loader either, as loading the \OPENTYPE\ files is done in -\LUA. So, we could also get rid of the font loader code. Currently some code is -shared with the font inclusion in the backend but that can be isolated. - -You can see a \TEX\ engine as being made from several parts, but the core really -concerns only two processes: reading, storing and expanding macros on the one -hand, and converting a stream of characters into lines, paragraphs, pages etc. -Fonts are mostly an abstraction: they are visible in so called glyph nodes as -font identifier (a number) and character code (also a number) properties. The -result, nowadays being \PDF, is also an abstraction: at some point the engine -converts the to be shipped out box in \PDF\ instructions, and in our case, -relatively simple ones. The backend registers which characters and fonts are used -and also includes the right resources. But, the backend is not part of the core -as such! It has been introduced in \PDFTEX\ and is a so called extension. - -So, what does that all mean for a future version of \CONTEXT\ and \LUATEX ? It -means that we can decide to follow up with a \CONTEXT\ that does more in \LUA, -which means not hard coded in a binary, on the one hand, but that we can also -decide to strip the engine from non|-|core code. But, given that \LUATEX\ is also -used in other macro packages, this would mean a different engine. We cannot say -that \LUATEX\ is stable when we also experiment with core components. - -We've seen folks picking up experimental versions assuming that it is a precursor -to official code. So, in order to move on we need to avoid confusion: we need to -use another name. Choosing a name is always tricky but as Taco already registered -the \METATEX\ domain, and because in the \CONTEXT\ distribution you will find -references to \METATEX, we will use that name for the future engine. Adding \LUA\ -to that name makes sense but then the name would become too long. - -The main difference between \METATEX\ and \LUATEX\ would be that the former has -no file lookup library, no hardcoded font loader, and no backend generator (but -possibly some helpers, and these need time to evolve). We're basically back where -\TEX\ started but instead of coding these extensions in \PASCAL\ or \CCODE\ we -use \LUA. We're also kind of back to when we first started experimenting with -\LUATEX\ in \CONTEXT\ where test, write and rewrite were going in parallel. But, -as said, we cannot impose that on a wide audience. - -If we go for such a lean and mean follow up, then we can also do a more drastic -cleanup of obsolete code in \CONTEXT\ (dating from \ETEX, \PDFTEX, \ALEPH, etc.). -We then are sort of back to where it all started: we go back to the basics. This -might mean dropping some primitives (one can define them as dummy). Of course we -could generalize some of the \CONTEXT\ code to provide the kicked out -functionality but would that pay of? Probably not. - -Just for the record: replacing the handling of macros, registers, grouping, etc.\ -to \LUA\ is not really an option as the performance hit would make a large system -like \CONTEXT\ sort of unusable: it's no option and not even considered (although -I must admit that I have some experimental \LUA\ based \TEX\ parser code around). - -It is quite likely that building \METATEX\ from source for the moment will be an -option to the build script. But we can also decide to simplify that process, -which is possible because we only need one binary. But in general we can assume -that one can generate \METATEX\ and \LUATEX\ from the same source. A first step -probably is a further isolation of the backend code. The fontloader and file -handling code already can be made optional. - -Given that we only need one binary (it being \LUATEX\ or \METATEX) and nowadays -only use \OPENTYPE\ fonts, one can even start thinking of a mini distribution, -possibly with a zipped resource tree, something we experimented with in the early -days of \LUATEX. - -Another though I have been playing with is a better separation between low level -and high level \CONTEXT\ commands, and whether the low level layer should be more -generic in nature (so that one can run specific packages on top of it instead of -the whole of \CONTEXT) but that might not be worth the trouble. - -\stopsection - -\startsection[title={Interlude}] - -If we look at the future, it's good to also look at the past. Opening up \TEX\ -the way we did has many advantages but also potential drawbacks. It works quite -well in \CONTEXT\ because we ship an integrated package. I don't think that there -are many users who kick in their own callbacks. It is possible but completely up -to the user to make sure things work out well. Performance hits, interference, -crashes: those who interfere with the internals can sort that out themselves. I'm -not sure how well that works out in other macro packages but it is a time bomb if -users start doing that. Of course the documented interfaces to use \LUA\ in -\CONTEXT\ are supported. So far I think we're not yet bitten in the tail. We keep -this aspect out of the discussion. - -Another important aspect is stability of the engine. Sometimes we get suggestions -for changes or patches that works for a specific case but for sure will have side -effects on \CONTEXT. Just as we don't test \LATEX\ side effects, \LATEX\ users -don't check \CONTEXT. And we're not even talking of users who expect their code -to keep working. A tight control over the source is important but cannot be we -will not be around for ever. This means that at some point \LUATEX\ should not be -changed any more, even when we observe side effects we want to get rid of, -because these side effects can be in use. This is another argument for a stripped -down engine. The less there is to mess with, the less the mess. - -\stopsection - -\startsection[title={Audience}] - -So how about \CONTEXT\ itself? Of course we can make it better. We can add more -examples and more documentation. We can try to improve support. The main question -for us (as developers) is who actually is our audience. From the mails coming to -the \CONTEXT\ support list it looks like a rather diverse group of users. - -At \TEX\ meetings there are often discussions about promoting \TEX. I can agree -on the fact that even for simple documents it makes a lot of sense to use \TEX, -but who will take the first hurdles? How many people really produce a lot of -documents? And how many need \TEX\ after maybe a short period of (enforced) usage -at the university? - -It's not trivial to recognize the possibilities and power of the -\LUATEX|-|\CONTEXT\ combination. We never got any serious requests for support -from large organizations. In fact, we do use this combination in a few projects -for educational publishers, but there it's actually the authors and editors doing -the work. It's seldom company policy to use tools that efficiently automate -typesetting. I dare to say that publishers are not really an audience at all: -they normally delegate the task. They might accept \TEX\ documents but let them -rekey or adapt far|-|far|-|away and as cheap as possible. Thinking of it, the -main reason for Don Knuth for writing \TEX\ in the first place was the ability to -control the look and feel and quality. It were developments at typesetters and -publishers that triggered development of \TEX . It was user demand. And the -success of \TEX\ was largely due to the unique personality and competence of the -author. - -System integrators qualify as audience but I fear that \TEX\ is not considered -hip and modern. It doesn't seem to matter if you can demonstrate that it can do a -wonderful job efficiently and relatively cheap. Also the fact that an -installation can be very stable on the long run is of no importance. Maybe that -audience (market place) is all about \quotation {The more we have to program and -update regularly, the merrier.}. Marketing \TEX\ is difficult. - -Those who render multiple products, maintain manuals, have to render many -documents automatically qualify as audience. But often company policies, -preferred suppliers, so called standard tools etc.\ are used as argument against -\TEX. It's a missed opportunity. - -One needs a certain mindset to recognize the potential and the question is, how -do we reach that audience. Drawing a roadmap for that is not easy but worth -discussing. We're open for suggestions. - -% \footnote {It's kind of interesting that recently the \TEX\ User Group announced -% its presence on Facebook and Twitter. Apart from wondering how that gets updated, -% one can also wonder how many potential (or even current) users go there, given -% that these platforms are subjected to rise and fall. I'm on neither of them and -% don't plan to. Kids (our future users) that I know already said goodbye to them. -% We'll see how that works out.} - -\stopsection - -\startsection[title={Conclusion}] - -At the \CONTEXT\ user meeting those present agreed that moving forward this way -makes sense. This means that we will explore a lean and mean \METATEX\ alongside -\LUATEX. There is no rush and it's all volunteer work so we will take our time -for this. It boils down to some reshuffling of code so that we can remove the -built|-|in font loader, file handling, and probably also \SYNCTEX\ because we can -emulate that. Then the backend with its font inclusion code will be cleaned up a -bit (we even discussed only supporting modern wide fonts). It's no big deal to -adapt \CONTEXT\ to this (so it can and will support both \LUATEX\ and \METATEX). -Eventually the backend might go away but now we're talking years ahead. By then -we can also explore the option to make \METATEX\ start out as a \LUA\ function -call (the main control loop) and become reentrant. There will probably not be -many changes to the opened up \TEX\ kernel, but we might extend the \METAPOST\ -part a bit (some of that was discussed at the meeting) especially because it is a -nice tool to visualize big data. - -As with \LUATEX\ development we will go in small steps so that we keep a working -system. Of course \LUATEX\ is always there as stable fallback. The experiments -will mostly happen in the experimental branch and binaries will be generated -using the compile farm on the \CONTEXT\ garden, just as happens now. This also -limits testing and exploring to the \CONTEXT\ community so that there are no side -effects for mainstream \LUATEX\ usage. - -Nowadays, instead if roadmaps, we tend to use navigational gadgets that adapt -themselves to the situation. On the road by car this can mean a detour and when -walking around it can be going to suggested points of interest. During the -excursion at the meeting, we noticed that after the drivers (navigators) -synchronized their gadget with Jano, the routes that were followed differed a -bit. We saw cars in front of going a different direction and cars behind us -arriving from a different direction. So, even when we talk about roadmaps, our -route can be adapted to the situation. - -Now here is something to think about. If you look at the \TEX\ community you will -notice that it's an aging community. User groups seem to loose members, although -the \CONTEXT\ group is currently still growing. Fortunately we see a new -generation taking interest and the \CONTEXT\ users are a pleasant mix and it -makes me stay around. I see it as an \quote {old timers} responsibility to have -\TEX\ and its environment in a healthy state by the time I retire from it -(although I have no plans in that direction). In parallel to the upcoming -development I think we will also see a change in \TEX\ use and usage. This aspect -was also discussed at the meeting and for sure will get a follow up on the -mailing lists and future meetings. It might as well influence the decisions we -make the upcoming years. So far \TEX\ has never failed us in it's flexibility and -capacity to adapt, so let's end on that positive note. - -\stopsection - -\stopchapter - -\stopcomponent |