short MAPS article by Taco on the now-integrated pool files

git-svn-id: svn://tug.org/texlive/trunk@8715 c570f23f-e606-0410-a88d-b1316a301751

1 files changed, 249 insertions, 0 deletions

diff --git a/Master/tlpkg/doc/integrated-pool.tex b/Master/tlpkg/doc/integrated-pool.tex
new file mode 100644
index 00000000000..bcd081e49ac
--- /dev/null
+++ b/Master/tlpkg/doc/integrated-pool.tex
@@ -0,0 +1,249 @@
+% Copyright 2008 Taco Hoekwater.
+% You may freely use, modify and/or distribute this file.
+
+\enablemode[realfonts]
+\enablemode[newstyle]
+
+\usemodule[map-se]
+
+\def\TEXMF{{\sc texmf}}
+
+%\setupwhitespace[big]
+%\setupindenting[no]
+
+\setuplayout[grid=yes]
+
+\starttext
+
+\startArticle[Title={Integrating the pool file},
+              SubTitle={},
+              Year=2007,
+              Number=35,
+              Page=9,
+              Period=VOORJAAR,
+              Author=Taco Hoekwater,
+              Email=taco(a)elvenkind.com]
+
+\startAbstract
+This short article discusses the method that is used in \METAPOST\ and
+\LUATEX\ to integrate the string pool file into the program.
+
+This method allows the redistribution of a single updated executable
+in place of both a program and a data file, and this makes
+updating those programs easier on both the user and the developer (me).
+\stopAbstract
+
+\subject{How a pool file is created}
+
+The readers who regularly update their (pdf)\TeX\ or \METAPOST\
+executables will probably be familiar with the concept of pool files
+already, but I will explain the mechanics in some detail.
+
+Programs written in the WEB language normally do not contain the
+strings inside the executable proper, but in a separate file, called
+the `pool file'.
+
+The most important reason for the existence of this file is that back
+when Don Knuth was creating \TeX\ and \METAFONT, there was not yet a
+standardized way to handle strings inside the Pascal language, so he
+had to invent his own solution for printing messages and warnings.  
+
+In order to illustrate what is in a pool file, I~will show you the
+required steps. First, here is a bit of WEB source from \METAPOST:
+
+\starttyping
+...
+if minx_val(h)>maxx_val(h) then 
+  print("0 0 0 0")
+else begin
+  ps_pair_out(minx_val(h),miny_val(h));
+  ps_pair_out(maxx_val(h),maxy_val(h));
+  end;
+print_nl("%%Creator: MetaPost ");
+print(metapost_version);
+print_nl("%%CreationDate: ");
+\stoptyping
+
+this excerpt is from one of the PostScript output routines. Here,
+there are still recognizable strings that are used as function
+arguments (as well as the symbolic value \type{metapost_version},
+that is actually a macro resolving to a string).
+
+The processor \type{tangle} converts this input into a proper Pascal
+source file. While doing so, it resolves all of the many WEB macros
+that are present in the code. \type{metapost_version} is one of those,
+but also the constructs like \type{minx_val(h)} and
+\type{maxx_val(h)}.  It also removes the underscores from function
+names, because traditional Pascal compilers did not allow \type{_} to
+appear in identifiers.
+
+\blank
+
+What we are focusing on now, is that it also collects all of the
+double||quoted strings in the input. It puts all of the unique
+multi||character strings into an internal array, and replaces the
+actual string in its output with the index number it has given the string
+inside that array. Of course, functions like \type{print()} are
+written in such a way that they expect numbers as arguments instead 
+of string values.
+
+The Pascal output file looks like this:
+
+\kern3pt
+
+\starttyping
+...
+if mem[h+2].int>mem[h+4].int then print(1279)
+else begin pspairout(mem[h+2].int,mem[h+3].int);
+pspairout(mem[h+4].int,mem[h+5].int);end;
+printnl(1281);print(256);printnl(1282);
+\stoptyping
+
+\kern3pt
+
+As you can see, this file is clearly intended for a compiler only.
+The complete lack of indentation makes it near impossible for a human
+to read the generated code, but of course a Pascal compiler has no
+problem with it.  
+
+\blank
+
+Nowadays, creating an executable program from the
+WEB source file happens in a few extra steps, and one of these steps
+is a conversion from Pascal to C source code, by means of the
+\type{web2c} system. You may find the output of \type{web2c} easier to
+read, because it re-indents the code for human reading:
+
+\starttyping
+...
+if ( mem [h + 2 ].cint > mem [h + 4 ].cint ) 
+  print ( 1279 ) ;
+else {
+  pspairout(mem [h + 2].cint,mem [h + 3].cint);
+  pspairout(mem [h + 4].cint,mem [h + 5].cint);
+} 
+printnl ( 1281 ) ;
+print ( 256 ) ;
+printnl ( 1282 ) ;
+\stoptyping
+
+\page
+
+So, where did the strings go? \type{tangle} put the multi||character
+strings into a separate file, in this case named \type{mp.pool}. Each
+line of that file contains two digits indicating the length of the
+string, followed by the string itself. Around line 1000, you will find
+this:
+
+\snaptogrid\vbox{
+\starttyping
+...
+070 0 0 0
+20%%HiResBoundingBox: 
+20%%Creator: MetaPost 
+16%%CreationDate: 
+...
+\stoptyping
+}
+
+\type{07} is the length in bytes of `\type{0 0 0 0}', \type{20} is the
+length of `\type{%%HiResBoundingBox: }', including the trailing space
+character, etcetera. Single character strings are not written to the
+pool file, because there is no need: all single||character strings
+simply have an assumed index value matching their contents, and the
+first string in the \type{pool} file receives index number 256.
+
+\blank
+
+The Pascal source code (or C source code) is now converted into an
+executable, and you end up with \type{mpost.exe} as well as
+\type{mp.pool}. The pool file is stored somewhere in the \TEXMF\ tree,
+and one of the very first things that the \type{--ini} version of
+\METAPOST\ does, is that it reads \type{mp.pool} to initialize its
+internal arrays. When the user executes the \type{dump} command,
+\METAPOST\ writes all of the string items to the \type{.mem} file,
+from where it will be retrieved by production runs of \METAPOST.
+
+There is nothing wrong with this system as such. In fact, it has
+worked well for nearly 30~years. But it does make updating executables
+a bit harder than desired: users not only have to copy the actual
+program to a folder in the path, but they also have to figure out
+where to place the new and improved \type{mp.pool} file.
+
+As the maintainer of \METAPOST\ and \LUATEX, both programs that are
+updated frequently, I was getting annoyed with having to explain to
+almost each updating user what a pool file was, why it was important,
+and where it should go in their \TEXMF\ tree.
+
+\snaptogrid\vbox{
+\subject{How a pool file disappears again}
+}
+
+So I decided to do something about it, and that was how the
+\type{makecpool} program was born. The concept is very simple: it
+converts the \type{mp.pool} into a C source file named
+\type{loadpool.c}.  In fact, it is so obvious that the idea has been
+proposed a number of times already, for instance by Fabrice Popineau.
+But somehow it has never made it to the core \TeX\ distribution yet.
+
+The structure of the created file is straightforward: there is one big
+static array, and a fairly simple C function that replaces the Pascal
+procedure for pool file reading. In abbreviated form, \type{loadpool.c}
+looks like this:
+
+
+\snaptogrid\vbox{\starttyping
+/* This file is auto-generated by makecpool */
+
+#include <stdio.h>
+#include "mpdir/mplib.h"
+
+static char *poolfilearr[] = {
+  "1.000",
+
+  ...
+
+  "0 0 0 0",
+  "%%HiResBoundingBox: ",
+  "%%Creator: MetaPost ",
+  "%%CreationDate: ",
+
+  ...
+
+  NULL };
+
+int loadpoolstrings (integer spare_size) {
+  char *s;
+  strnumber g=0;
+  int i=0,j=0;
+  while ((s = poolfilearr[j++])) {
+    int l = strlen (s);
+    i += l;
+    if (i>=spare_size) return 0;
+    while (l-- > 0) strpool[poolptr++] = *s++;
+    g = makestring();
+    strref[g]= 127;
+  }
+  return g;
+}
+\stoptyping
+}
+
+In the stage where the various C files are compiled into
+\type{mpost.exe}, this file is included in the list, and in that way
+the strings will be embedded in the program. At run-time, the C
+function is called to put the strings for the C array into the
+internal storage area instead of the original file reader.
+
+The result: there is only one single executable file that can be freely
+distributed to the users. The source code for \type{makecpool} is part
+of the \METAPOST\ and \LUATEX\ distribution package.
+
+Post-publication update: thanks to Akira Kakuto and others, this same
+method has now been adapted to all the \TeX\ family programs which used
+pool files, and \TeX\ Live 2008 contains no external pool files.  Knuth
+has approved the change for the baseline \type{tex} program.
+
+\stopArticle
+
+\stoptext