summaryrefslogtreecommitdiff
path: root/Master/texmf-dist/doc/context/presentations/context/2020/context-2020-luametatex.tex
blob: 10b57660ea9ec7b3a5ba86e4dc805984a69a6071 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
% language=us

\usemodule[present-boring,abbreviations-logos]

\startdocument
  [title={LUAMETATEX},
   banner={where do we stand},
   location={context\enspace {\bf 2020}\enspace meeting}]

\starttitle[title=When it started]

\startitemize

\startitem
    About three years ago the idea came up to go this route.
\stopitem
\startitem
    At the 2018 meeting it was first mentioned and those present were okay with
    it.
\stopitem
\startitem
    Early 2019 the first beta release took place.
\stopitem
\startitem
    At the 2019 meeting the first more official version was presented.
\stopitem
\startitem
    Around the 2020 meeting we have more or less arrived at what I had in mind.
\stopitem
\startitem
    At the 2021 meeting I expect the code to be stable and repositories to be set
    up.
\stopitem
\startitem
    At the 2022 meeting we can make the official transition from \MKIV\ to \LMTX.
\stopitem
\startitem
    Some new options are only enabled in my local \type {cont-exp.tex} file.
\stopitem
\startitem
    Knowing that Wolfgang keeps an eye on all those changes makes me more daring.
\stopitem
\startitem
    We aim to get less (but more efficient) macro code that on the average looks
    better.
\stopitem

\stopitemize

\stoptitle

\starttitle[title=Why it started]

\startitemize

\startitem
    There was an increasing pressure for a stable \LUATEX.
\stopitem
\startitem
    There should be no more changes to the interfaces, no more extensions.
\stopitem
\startitem
    One can run into interesting comments on the web (as usual), like
    \startitemize[packed]
        \startitem The \LUATEX\ program has \quote {many bugs}. \stopitem
        \startitem The \LUATEX\ manual is bad. \stopitem
        \startitem The \LUATEX\ program is too slow to be useful. \stopitem
        \startitem The \LUATEX\ program will never end up in distributions. \stopitem
        \startitem The \LUATEX\ project is funded and developed in a commercial setting. \stopitem
    \stopitemize
\stopitem
\startitem
    I won't comment on how I read these (demotivating) comments because \unknown
\stopitem
\startitem
    \unknown\ it anyway often says more about the writer (attitudes) than about
    \LUATEX.
\stopitem
\startitem
    I also looks like (non \CONTEXT) users are charmed by \LUATEX, and the more
    they code, the more we need to freeze.
\stopitem
\startitem
    So, hopefully, the \LUAMETATEX\ development does not interfere badly with
    developments outside the \CONTEXT\ community.
\stopitem

\stopitemize

\stoptitle

\starttitle[title=The development]

The summary on the next pages is partial. More can be found in articles and
documents that come with the distribution.

\startitemize

\startitem
    \LUATEX\ started out as \CWEB\ code \unknown\ that eventually became just
    \CCODE\ \unknown\ which in \LUAMETATEX\ has been detached from the (complex)
    infrastructure.
\stopitem
\startitem
    The basic idea is to only keep the core of \TEX, but for instance font
    loading, file handling and the backend are gone.
\stopitem
\startitem
    As a consequence the code has been reorganized (shuffled around).
\stopitem
\startitem
    I experimented a lot without bothering about usage elsewhere and I like the
    result so far.
\stopitem
\startitem
    The \CONTEXT\ distribution will at some point ship with the source.
\stopitem

\stopitemize

\starttitle[title=File handling]

\startitemize

\startitem
    All file handling goes via \LUA, also read and write related primitives.
\stopitem
\startitem
    The same is true for terminal (console) handling.
\stopitem
\startitem
    Part of that (the writing) was actually kind of extension code in \TEX\ and
    partly a system dependency.
\stopitem
\startitem
    The \ETEX\ pseudo file \type {\scantokens} primitive uses the same mechanism
    as \LUA\ does.
\stopitem

\stopitemize

\stoptitle

\starttitle[title=The macro machinery]

\startitemize

\startitem
    There are extensions to the way macro arguments are handled (less clumsy
    macros).
\stopitem
\startitem
    There are extra if tests (makes for nicer macros).
\stopitem
\startitem
    Else branches in conditions can be collapsed using \type {\orelse} and \type
    {\orunless} which gives cleaner low level code.
\stopitem
\startitem
    Tracing gives more detail about node properties and also shows attributes.
\stopitem
\startitem
    Some new data carriers have been added that can be played with from \LUA .
\stopitem
\startitem
    Macros can efficiently be frozen (new) and protected (redone) and the
    concepts \quote {long} and \type {outer} are gone. \footnote {In \CONTEXT\
    macros were always \type {\long} and never \type {\outer}. Most commands were
    unexpandable (also in \MKII, pre \ETEX). So, users won't notice this.}
\stopitem
\startitem
    Saving and restoring is somewhat more efficient (partly a side effect of
    wider memory).
\stopitem

\stopitemize

\stoptitle

\starttitle[title=Language]

\startitemize

\startitem
    Language control settings now use less parameters but bit sets instead.
\stopitem
\startitem
    Only basic parameters are stored in the format file now.
\stopitem
\startitem
    There are all kind of small improvements.
\stopitem

\stopitemize

\stoptitle

\starttitle[title=Typesetting]

\startitemize

\startitem
    Attributes (the lists and states) are implemented more efficiently.
\stopitem
\startitem
    The paragraph state is stored with the paragraph.
\stopitem
\startitem
    Paragraphs can be normalized and options are now set with bit sets.
\stopitem
\startitem
    Boxes carry orientation related information (offsets, rotation, etc).
\stopitem
\startitem
    Some nodes carry more information.
\stopitem
\startitem
    Directions are mostly gone (it's up to the backend).
\stopitem
\startitem
    Migrated content is optionally kept with boxes.
\stopitem

\stopitemize

\stoptitle

\starttitle[title=Math]

\startitemize

\startitem
    Some math concepts have been extended (like prescripts and some more
    control over styles).
\stopitem
\startitem
    There are plenty of new control details.
\stopitem
\startitem
    The math parameter settings obey grouping in a math list.
\stopitem
\startitem
    We can have math in discretionaries in text and more advanced discretionaries
    in math as well.
\stopitem

\stopitemize

\stoptitle

\starttitle[title=Fonts]

\startitemize

\startitem
    Font specification information no longer uses the string pool (which saves a
    lot).
\stopitem
\startitem
    Of course we still have the basic font handler.
\stopitem
\startitem
    We only store what is needed for traditional \TEX\ font handling.
\stopitem
\startitem
    Virtual fonts are even more virtual (also a backend thing) so we can have
    more features.
\stopitem

\stopitemize

\stoptitle

\starttitle[title=The code]

\startitemize

\startitem
    Artifacts from \PASCAL\ and \CWEB\ have been removed.
\stopitem
\startitem
    Languages, fonts, marks etc are no longer \quote {register} based.
\stopitem
\startitem
    The token interface is more abstract and no longer presents strange numbers.
\stopitem
\startitem
    Some internals have been reconstructed because of cleaner \LUA\ interfacing.
\stopitem
\startitem
    A side effect of this is better abstraction of the equivalent ranges.
\stopitem
\startitem
    The code has been made more abstract (and looks easier in e.g. Visual Studio).
\stopitem
\startitem
    The compile farm is used to check if compilation works out of the box.
\stopitem
\startitem
    Compilation is fast and easy, otherwise this project was not possible.
\stopitem
\startitem
    Readability of the code is constantly improved (the usual: has to look okay
    in my editor).
\stopitem
\startitem
    The code has been made mostly independent of specific operating system needs.
\stopitem
\startitem
    Wide characters are dealt with in Windows interfaces.
\stopitem

\stopitemize

\stoptitle

\starttitle[title=Libraries]

\startitemize

\startitem
    We really want to stay lean and mean: the engine is also a \LUA\ engine.
\stopitem
\startitem
    All code is included, a few libraries are used, but these are small, old and
    stable.
\stopitem
\startitem
    In addition some helper libraries are made (including pplib by Pawel).
\stopitem
\startitem
    What we ship is what you get: \CONTEXT\ will not depend on more than that.
\stopitem
\startitem
    If something is updated (at all) the differences are checked first.
\stopitem

\stopitemize

\stoptitle

\starttitle[title=The \LUA\ engine]

\startitemize

\startitem
    We use the latest (even alpha) \LUA\ (5.4) because \LUAMETATEX\ is a good
    test.
\stopitem
\startitem
    There is no support for \LUAJIT\ and the \FFI\ interface is gone.
\stopitem
\startitem
    There is a limited set of libraries that we support but no code is (and will
    be) included.
\stopitem
\startitem
    There are less callbacks (because we only have a frontend).
\stopitem
\startitem
    There are more token scanners and some options have been added.
\stopitem

\stopitemize

\stoptitle

\starttitle[title=Efficiency]

\startitemize

\startitem
    We benefit some more from the wider memory words (some constructs could go).
\stopitem
\startitem
    The format file is smaller and not longer compressed.
\stopitem
\startitem
    Memory management is now mostly dynamic and usage is much lower.
\stopitem
\startitem
    There are more statistics (also as side effect of memory management).
\stopitem
\startitem
    Dumping the format has been made a bit more robust and is faster.
\stopitem
\startitem
    The core engine performs a bit better (machines don't get that much faster).
\stopitem
\startitem
    We want to be prepared for future architectures.
\stopitem
\startitem
    We manage to keep the binary way below 3 MB.
\stopitem
\startitem
    The lot runs quite well on e.g.\ a Raspberry Pi 4.
\stopitem

\stopitemize

\starttitle[title=Upgraded \METAPOST]

\startitemize

\startitem
    All (eight bit) font stuff has been stripped from the \METAPOST\ library.
\stopitem
\startitem
    The library no longer has a \POSTSCRIPT\ backend.
\stopitem
\startitem
    The library provides scanners that make extensions possible.
\stopitem
\startitem
    All file \IO\ goes via \LUA.
\stopitem
\startitem
    There are a few additions like pre|/|postscripts for clip and bounding boxes.
\stopitem
\stopitemize

\stoptitle

\starttitle[title=Praise for the users]

\startitemize

\startitem
    Much has been done and I probably forget to mention a lot.
\stopitem
\startitem
    The number of bugs is relative small compared to what gets changed and added.
\stopitem
\startitem
    The test suite gets ran very often, also to check if performance is okay.
\stopitem
\startitem
    I could only do this because the \CONTEXT\ users are so tolerant.
\stopitem
\startitem
    Some seem to constantly check for updates so they help with fast testing.
\stopitem
\startitem
    The \CONTEXT\ code base gets stepwise adapted (split files) which again
    forces users to test.
\stopitem
\startitem
    It takes a lot of time because we take small steps in order not to mess up.
\stopitem
\startitem
    I would not do it without the positive attribute of the \CONTEXT\ users.
\stopitem
\startitem
    It's all about motivation and I thank the \CONTEXT\ users for providing this
    friendly and non|-|competitive bubble!
\stopitem

\stopitemize

\starttitle[title=Todo]

\startitemize

\startitem
    Maybe add some more sanity checks in order to catch errors intruded by
    callbacks. Maybe add some more tracing too.
\stopitem
\startitem
    Explore variants, like having registers in dedicated eqtb tables so that we
    can allocate them dynamically (mostly for the fun of doing it).
\stopitem
\startitem
    Add some more documentation (read: addition cq.\ remarks about where the
    original documentation no longer applies, but we have years for doing that).
\stopitem
\startitem
    Update the manual (which is done occasionally in batch based on print|-|outs;
    there is no real need to hurry because we still experiment).
\stopitem
\startitem
    Apply some of the new stuff in \LMTX. Take up some challenges.
\stopitem
\startitem
    Wrap up new functionality (once it's stable) in articles and other documents.
\stopitem

\stopitemize

\stoptitle

\starttitle[title=And \LUATEX ?]

\startitemize

\startitem
    Of course \LUATEX\ will be maintained! After all, \MKIV\ needs it and it
    serves as reference for the front|-|end rendering and back|-|end generation
    when we're messing with \LUAMETATEX.
\stopitem
\startitem
    It is used by \LATEX\ and there are now also plain inspired packages. Because
    there are spin|-|offs (\LATEX\ has settled on a version with built|-|in font
    processing) we cannot change much.
\stopitem
\startitem
    And \LUATEX\ being nicely integrated into \TEXLIVE\ is another argument for
    not touching it too much.
\stopitem
\startitem
    I have no clue of \LUATEX\ usage but that fact alone already makes an
    argument for being even more careful. It's bad advertisement for \TEX\ when
    users who use the low level interfaces get confronted with conceptual
    changes.
\stopitem
\startitem
    So in the end not much will be back ported to \LUATEX: at some point the code
    base became too different and it's the price paid for the stability demand.
    That way we cannot introduce new bugs either. It also doesn't pay of.
\stopitem
\startitem
    But, a few non|-|intrusive things might actually trickle into it in due time,
    also out of self interest: it might help to share code between \MKIV\ and
    \LMTX.
\stopitem

\stopitemize

\stopdocument