From e0c6872cf40896c7be36b11dcc744620f10adf1d Mon Sep 17 00:00:00 2001 From: Norbert Preining Date: Mon, 2 Sep 2019 13:46:59 +0900 Subject: Initial commit --- macros/generic/hyacc-cm/hyacc-cm.doc | 411 +++++++++++++++++++++++++++++++++++ 1 file changed, 411 insertions(+) create mode 100644 macros/generic/hyacc-cm/hyacc-cm.doc (limited to 'macros/generic/hyacc-cm/hyacc-cm.doc') diff --git a/macros/generic/hyacc-cm/hyacc-cm.doc b/macros/generic/hyacc-cm/hyacc-cm.doc new file mode 100644 index 0000000000..1d3a057a11 --- /dev/null +++ b/macros/generic/hyacc-cm/hyacc-cm.doc @@ -0,0 +1,411 @@ + %%% File HyAcc-CM.doc + %% Documentation for typographers and TeX programmers + %% For basic user instructions see the header + %% of HyAcc-CM.tex, the one example below + %% and the format compilation mechanism + %% fe-format-dumper-cm + + + OUR PERSPECTIVE ON THE HYPHENATION PROBLEM + + Jacques Desarmenien's devices for obtaining respectable +hyphenation in the presence of accents were never optimized nor +internationalized --- conjecturally because it was expected +that a version 3 of TeX would be based on MLTeX of M. Ferguson +and rather simply offer optimal hyphenation with no change in +the TeX environment. + + This did not come to pass. When TeX version 3 came along, +it instead offered something better inasmuch as the accented +characters can be tailor-made under the new 8-bit regime +(rather than composed of accent plus character). But, at the +same time, the cost was unexpectedly high: new fonts, new +drivers, new norms, and new formats. They are burdensome for +many European TeX users, and perhaps out of reach of many +non-European users. + + It was then announced that TeX and Metafont will not be +further changed by its author Don Knuth, nor by by anyone else +so long as they bear their present copyrighted names. + + Thus, rather unexpectedly, it has became desirable to have a +good international version of modest but decent hyphenation in +the presence of accents that works with absolutely all versions +of TeX since 1982, in particularly with TeX and Computer Modern +fonts as documented by Knuth. + + If you use TeX in a version <3, prior to fall 1989, you +should avoid recent hyphenation pattern files. Also, for better +portability, always prefer pattern files in ASCII format. + + Many 7-bit european hyphenation pattern tables are sufficiently +compact and portable to circulate along with .tex manuscripts by +electronic mail. For example, f7hyphen.tex the 7-bit French +hyphenation file of Desarmenien is just 4Ko, while HyAcc-CM.tex is +5Ko (without comments). + + + +A TYPICAL (FRENCH) USE OF HYACC-CM + + Let us consider the typical case of a French visitor to the +Anglo-Saxon world who, to process his French typescript, has to +use a "mainframe" computer exploiting exclusively CM fonts. If he +is fortunate enough to use Plain.tex, he can quite simply replace +TeX by INITEX at the cost of adding as a header to his typescript +the incantation: + + \input plain.tex %% standard, using US hyphenation for language 0 + \bgroup\catcode`@=11 + \global\def\US{\lccode`\'=0\nofrenchspacing\language=0} + \newlanguage\l@French + \global\def\FR{\lccode`\'=`\'\frenchspacing\language=\l@French} + \egroup + \FR + \input f7hyphen.tex + \input hyacc-cm.tex + +His TeX typescript will be set by INITEX starting with French +hyphenation; \US restores US English. This assumes the +typescript inputs adequate macros for French punctuation etc. + + Alternatively, he can make of this header a separate file +concluded by \dump newformat.fmt. Then INITEX dumps a format, +say newformat.fmt, which can in turn be used under INITEX +by reducing the header to + + &newformat.fmt + +This second procedure is the one that best generalizes to massive +formats such as LaTeX. Some ready-made recipes for them can be +found in fe-format-dumper-cm posted on ftp matups.matups.fr. +(These include fe-punct.tex and gllmts.tex giving rudimentary +macros for French versus English punctuation.) However, in +hostile environments, the use of simple formulas as above is often +the most cost-effective strategy. + + In a worst possible scenario, INITEX (initex) is unavailable. +Then one has to use the English hyphenation in conjunction with +HyAcc-CM. For this, the following simpler header can be used on +top of any standard English format: + + \def\US{\lccode`\'=0 \nofrenchspacing} + \def\FR{\lccode`\'=`\' \frenchspacing} + \FR + \input hyacc-cm.tex + \pretolerance=800 %% discourages hyphenation + +Hyphenation points that occur have to be checked for correctness +and replaced by explicit hypenations (via \-) when necessary. + + +A HISTORY AND EXPLANATION OF THE MACROS + + The background reference for all matters to be discussed is +Appendix H of the TeXbook, and particularly pages 454-455. + + The story begins with Jacques Desarmenien's thesis, Univ. +Louis Pasteur, Strasbourg, written in the early 1980's partly at +Stanford under Knuth's influence. The accessible reference is +Jacques D'esarm'enien: How to run TeX in a French environment: +hyphenation, fonts, typography, Tugboat, 5 (1984) 91-102. This +reference discusses hyphenation through the use of fonts +containing ready-made accented characters of the sort slowly +coming into use now some 10 years later. This package puts to +work devices for hyphenation which do NOT require new fonts, and +which were called upon a few years later when distribution of +new fonts was provisionally abandoned. + + It is often repeated that hyphenation of a word under 1982 +TeX is blocked at points at and after the first accented +character because of the explicit kerns that the \accent +primitive of TeX introduces there. + + The following formula appeared in an unsigned file +feHyphen.tex that was available from Strasbourg from the mid +1980's. According to Raymond Seroul, the author is J. +Desarmanien. The resulting bilingual fePlain format had wide +currency in France through Seroul's influence. + + \newbox\accentb@x \newdimen\wdb@x + \newbox\accentb@@x + \let\aig@=\'% + \def\'#1{\if e#1\setbox\accentb@x=\hbox{\char"13}% + \setbox\accentb@@x=\hbox{e}% + \wdb@x=.5\wd\accentb@@x \advance\wdb@x by -.5\wd\accentb@x + \kern\wdb@x \char"13 \kern-\wd\accentb@x + \penalty\@M \hskip-\wdb@x e% + \else \aig@#1\fi} + +A verbal interpretation of this is as follows. Let the one +argument macro \' for accute accents be redefined as follows in +case the letter being accented is e. Build the accented +character from the accute accent character "13 and the +character e using boxes and kerns as common sense recommends, +except for one detail. Let the last kern (which occurs just +before the letter e) become an hskip. This causes TeX to +consider that a new word has begun with a letter e and allows +hyphenation in that (sub)word. Normally an hskip is an +acceptable place for a linebreak, but this is prevented in this +case by insertion of an explicit penalty of 10000. + + Why was only the character \'e dealt with? Conjecturally +because effort was made via Desarmenien's hyphenation patterns +to assure that no incorrect hyphenation point resulted from this +trick. This is a standard we cannot maintain since it is +implausible that anything but a standard 8 bit hyphenation file +will be available to users. + + This means that the Desarmenien mechanism will occasionally +allow an unacceptable hyphenation point. (We do not yet have +statistics on this.) Some specialists, perhaps including Knuth, +would frown on this. Other specialists believe that hyphenation +should in any case be based more on phoenetics meaning and +etymology than on spelling, and that sort is sometimes +impossible to perfect with the Knuth-Liang approach. If +perfection is impossible why insist on complete rigour? + + In the late 1980's, in the Latex style file called +German.sty H. Partl (and collaborators) used a version of this +"penalty and hskip" trick which is so clear and simple that it +allows one to deal with all accents and all letters on an equal +footing. For example Partl gives + + \gdef\allowhyphens{\penalty\@M \hskip\z@skip} + % {\nobreak \hskip 0pt plus 0pt\relax} + + \gdef\highumlaut#1{\leavevmode\allowhyphens + \accent127 #1\allowhyphens} + +Here the one-argument macro \highumlaut#1 is a new version of +\"#1 designed to allow hyphenation after the accented character. +This is certainly shorter and faster than Desarmenien's +formulation. But there are some peculiarities to discuss. + + The subword following the accent does not include the +character accented. This unfortunately supresses any kern +between the character accented and a following character. Such +kerns are respected by both Knuth's regime (\DKaccents) and +(generalized) Desarmenien's (\JDaccents). (Note that kerning +with *preceeding* material is not within the scope of either, +but see below.) + + Another effect is to keep hyphens further from the +accented character, which is not in itself a good thing, but +it may also tend to eliminate bad hyphenations caused by the +fake word break. (No detailed evaluation has been made.) + + The positive role of \leavevmode and the first +\allowhyphens is unclear to us and we have let them drop. + + One of us (D.F.) noticed that in case the accented letter +occurs at the end of a word, linebreak becomes impossible there. +This may not occur in German but if one were to adopt the German +approach for accute accent, defining + + \def\'#1{{\accent"13 #1\allowhyphens}} + +then linebreak would be prevented after all the innumerable +French words terminating in e-acute! Here, surely, is the +reason why \allowhyphens never gained currency in France. + + This disaster does not occur with the Desarmenien approach +because the construction of the accented character concludes +with a character and not an hskip. + + Examine now our own definition of \'; under our \JDaccents +regime it is essentially + + \newbox\hy@accbox + \def\'#1{{\accent"13 #1}\setbox\hy@accbox=\hbox{#1} + \penalty10000\hskip-\wd\hy@accbox #1} + +We chose to use the standard \accent macro and nevertheless use +Desarmenien's "penalty-hskip-letter" approach, at the +(negligible!) cost of setting two copies of the letter to be +accented, one exactly on top of the other. Our approach is thus +not quite J. Desarmenien's but rather a trick simplification. It +would have been possible to pursue the Desarmenien approach in +all cases, but the coding would have been messy for capital +letters and miscellaneous accents. Incidentally, Desarmenien +makes it quite clear that he was privy to the folklore of Knuth +and his students and has made no explicit claim to this trick. +He did first bring it to fruition, and so we use his initials in +\JDaccents. + + Granting that one chooses an approach making the letter to +be accented part of the post-accent (sub)word, it might be +desirable to make the pre-accent (sub)word terminate with the +same letter. The two subwords then have that letter in common. +This might give slightly more acceptable hyphenation points. And +it would enable kerning with a preceeding letter, not just with +a following letter. In principle the method extends to do this: + + \def\'#1{{#1\setbox\hy@accbox=\hbox{#1}% + \penalty10000\hskip-\wd\hy@accbox + \accent"13 #1% + \penalty10000\hskip-\wd\hy@accbox #1}} + +THREE copies of the letter to be accented are then printed +directly on top of one another. + + In principle, with digital raster imaging, the result +should be identical to printing once. However many driver/font +combinations will at low resolution slightly misplace the first +of the three copies --- in case the accent occurs at the end of +a long string of ordinary letters. This is because character +width errors accumulate over a long string, and thereafter the +\accent macro makes a fresh calculation of absolute position. +The Textures screen driver dramatically exhibits this behavior. +Knowing no cure for this malady, we have abandoned this "three +copy" approach. Other versions are afflicted to a lesser degree +by this phenomenon, because the shift involved, although +present, is between characters several pixels apart. + + We believe that our "two-copy" or "double strike" approach +will very seldom cause a problem. + + Alas, with some combinations of low resolution drivers and +fonts some sort of drift has indeed been observed on screen +and/or paper. For these unfortunate cases we give another +approach \GFaccents (GF for German-French) which directly +salvages the German approach by use of an \if ...\fi clause +involving \futurelet. If the token following the character +accented is a letter (like a) then one inserts before it a +penalty of 1000 followed by an hskip of 0 points in order to +start a new word; but in other cases (notably when a space comes +next) one inserts nothing. This allows linebreaks after words +ending in an accented character and followed by a space. +Expressed in formulae + + \def\preLetterSkip@{\ifcat a\next\penalty\@M\hskip\z@skip\fi} + \def\@@@@Allowhyphens{\futurelet\next\preLetterSkip@} + \def\'#1{{\accent18#1}\@@@@Allowhyphens} + +is essentially our definition of the accute accent \' for the +\GFaccents regime. + + Unfortunately, this solution will suppress all kerns +between the character accented and a following character. Also, +it can be defeated by nonstandard typing; for example TeX braces +around an accented character will block hyphenation beyond. + + Further, 'futurelet' has occasional nasty side-effects in +LaTeX. For example, disaster occurs when an accented character +ends a 'tabular' environment as in: + + -- \begin{tabular} ... Painlev\'e\end{tabular} + -- \author{J. Dieudonn\'e} + +It's easy to find work-arounds: just add a space after the +terminal accented character (or put the whole word between curly +braces) and your problems are over. Thus the modified versions + + -- \begin{tabular} ... Painlev\'e \end{tabular} + -- \author{J. Dieudonn\'e } or \author{J. {\sc Dieudonn\'e}} + +will cause no problem. But all this shows that this solution is +fragile. + + + +LEFTOVERS AND IMPERFECTIONS + + 0) \ogonek #1 provides a provisional ogonek accent under character #1 +It is a translated copy of the left single quote. With this exception +we treat only the accents in the TeXbook, page 52. + + 1) This version makes no effort to eliminate the following +deficiency of Plain TeX: \uppercase and \lowercase fail when +applied to the miscellaneous characters \oe, \OE, \ae, \AE, \o, +\O, \ss and also to \i. Our \MLcodes macro merely serves to +allow some hyphenations in words involving these characters. +This is an imperfection. + + 2) If your computer and TeX have eight bit input capabilities, +you may wish to type some accented characters in ready-made +eight bit form using macros of the following sort. (Your TeX +installation probably has a ready-made collection.) + + \catcode =13 + \def{\'e} + +The formulas are here in an informal but portable ASCII format, +and you have to put your local e-accute in place of +before they will work. + + Next some changes that are common safety measures in various +national macro packages and which we have included here only in +the form of hints for the TeXpert, since we feel they are +primarily the responsibility of other macro packages. + + 3) Problems with \write. Only in the last few years have +simple and fast "sanitized" macros come into use for the purpose +of writing pieces of the .tex file to an external file. The +naked \write command makes a mess of accents and of many other +things, and even may choke. The basic reference seems to be R. +Whitney, TUGboat, November 1990. Here is a still simpler method: + + \newwrite\index \openout\index{myindex} + \def\iWrite#1#2{\toks0{#2}% + \immediate\write#1{\the\toks0} + +And a typical use is: + + \iWrite\index{...} + +The material ... then goes out to the file myindex.tex more or +less untouched. + + This does not work well for non-immediate writes --- +typically used to record page number of an index entry, but one +reasonable remedy is to write the dangerous material with +\iWrite to one file and the page numbers with \write to another +file and then collate the two using some device like serial +numbers or simply line numbers. + + 4) Cleaner messages. + + \message is of the same ilk as \write (and \edef). The above +"\the\toks..." non-expansion trick can be applied to it to +prevent mutilation or mishap. + + However most messages come from the bowels of +macro packages and it is not easy to be sure of their contents. +Active punctuation as well as accent macros often creep in. +Recall that character category (eg. 13=active) is fixed forever at TeX's +lips when the character is input. + + One can input the following redefinition while +;:?! are active. It will help combat these problems. +But first check your current meaning of \message. + + \let\@oldmessage@=\message + \def\message#1{\begingroup + \def\'{'}\def\`{`}\def\"{"}% + \def\i{i}\def\oe{oe}\def\ae{ae}\def\c{}% + \def\.{}\def\u{}\def\v{}% + \def\H{}\def\={}\def\relax{}% + \edef~{\string~}% + \edef\^{\string^^5e}% + \edef:{\string:}% + \edef;{\string;}% + \edef?{\string?}% + \edef!{\string!}% + \@oldmessage@{#1}\endgroup} + + + ------------------------- + + +If you notice errors or oversights please contact us. + + + + --- Daniel Flipo, Lille + + --- Laurent Siebenmannn, Orsay + + April, August, December 1992 + + -- cgit v1.2.3