%%% File HyAcc-CM.doc %% Documentation for typographers and TeX programmers %% For basic user instructions see the header %% of HyAcc-CM.tex, the one example below %% and the format compilation mechanism %% fe-format-dumper-cm OUR PERSPECTIVE ON THE HYPHENATION PROBLEM Jacques Desarmenien's devices for obtaining respectable hyphenation in the presence of accents were never optimized nor internationalized --- conjecturally because it was expected that a version 3 of TeX would be based on MLTeX of M. Ferguson and rather simply offer optimal hyphenation with no change in the TeX environment. This did not come to pass. When TeX version 3 came along, it instead offered something better inasmuch as the accented characters can be tailor-made under the new 8-bit regime (rather than composed of accent plus character). But, at the same time, the cost was unexpectedly high: new fonts, new drivers, new norms, and new formats. They are burdensome for many European TeX users, and perhaps out of reach of many non-European users. It was then announced that TeX and Metafont will not be further changed by its author Don Knuth, nor by by anyone else so long as they bear their present copyrighted names. Thus, rather unexpectedly, it has became desirable to have a good international version of modest but decent hyphenation in the presence of accents that works with absolutely all versions of TeX since 1982, in particularly with TeX and Computer Modern fonts as documented by Knuth. If you use TeX in a version <3, prior to fall 1989, you should avoid recent hyphenation pattern files. Also, for better portability, always prefer pattern files in ASCII format. Many 7-bit european hyphenation pattern tables are sufficiently compact and portable to circulate along with .tex manuscripts by electronic mail. For example, f7hyphen.tex the 7-bit French hyphenation file of Desarmenien is just 4Ko, while HyAcc-CM.tex is 5Ko (without comments). A TYPICAL (FRENCH) USE OF HYACC-CM Let us consider the typical case of a French visitor to the Anglo-Saxon world who, to process his French typescript, has to use a "mainframe" computer exploiting exclusively CM fonts. If he is fortunate enough to use Plain.tex, he can quite simply replace TeX by INITEX at the cost of adding as a header to his typescript the incantation: \input plain.tex %% standard, using US hyphenation for language 0 \bgroup\catcode`@=11 \global\def\US{\lccode`\'=0\nofrenchspacing\language=0} \newlanguage\l@French \global\def\FR{\lccode`\'=`\'\frenchspacing\language=\l@French} \egroup \FR \input f7hyphen.tex \input hyacc-cm.tex His TeX typescript will be set by INITEX starting with French hyphenation; \US restores US English. This assumes the typescript inputs adequate macros for French punctuation etc. Alternatively, he can make of this header a separate file concluded by \dump newformat.fmt. Then INITEX dumps a format, say newformat.fmt, which can in turn be used under INITEX by reducing the header to &newformat.fmt This second procedure is the one that best generalizes to massive formats such as LaTeX. Some ready-made recipes for them can be found in fe-format-dumper-cm posted on ftp matups.matups.fr. (These include fe-punct.tex and gllmts.tex giving rudimentary macros for French versus English punctuation.) However, in hostile environments, the use of simple formulas as above is often the most cost-effective strategy. In a worst possible scenario, INITEX (initex) is unavailable. Then one has to use the English hyphenation in conjunction with HyAcc-CM. For this, the following simpler header can be used on top of any standard English format: \def\US{\lccode`\'=0 \nofrenchspacing} \def\FR{\lccode`\'=`\' \frenchspacing} \FR \input hyacc-cm.tex \pretolerance=800 %% discourages hyphenation Hyphenation points that occur have to be checked for correctness and replaced by explicit hypenations (via \-) when necessary. A HISTORY AND EXPLANATION OF THE MACROS The background reference for all matters to be discussed is Appendix H of the TeXbook, and particularly pages 454-455. The story begins with Jacques Desarmenien's thesis, Univ. Louis Pasteur, Strasbourg, written in the early 1980's partly at Stanford under Knuth's influence. The accessible reference is Jacques D'esarm'enien: How to run TeX in a French environment: hyphenation, fonts, typography, Tugboat, 5 (1984) 91-102. This reference discusses hyphenation through the use of fonts containing ready-made accented characters of the sort slowly coming into use now some 10 years later. This package puts to work devices for hyphenation which do NOT require new fonts, and which were called upon a few years later when distribution of new fonts was provisionally abandoned. It is often repeated that hyphenation of a word under 1982 TeX is blocked at points at and after the first accented character because of the explicit kerns that the \accent primitive of TeX introduces there. The following formula appeared in an unsigned file feHyphen.tex that was available from Strasbourg from the mid 1980's. According to Raymond Seroul, the author is J. Desarmanien. The resulting bilingual fePlain format had wide currency in France through Seroul's influence. \newbox\accentb@x \newdimen\wdb@x \newbox\accentb@@x \let\aig@=\'% \def\'#1{\if e#1\setbox\accentb@x=\hbox{\char"13}% \setbox\accentb@@x=\hbox{e}% \wdb@x=.5\wd\accentb@@x \advance\wdb@x by -.5\wd\accentb@x \kern\wdb@x \char"13 \kern-\wd\accentb@x \penalty\@M \hskip-\wdb@x e% \else \aig@#1\fi} A verbal interpretation of this is as follows. Let the one argument macro \' for accute accents be redefined as follows in case the letter being accented is e. Build the accented character from the accute accent character "13 and the character e using boxes and kerns as common sense recommends, except for one detail. Let the last kern (which occurs just before the letter e) become an hskip. This causes TeX to consider that a new word has begun with a letter e and allows hyphenation in that (sub)word. Normally an hskip is an acceptable place for a linebreak, but this is prevented in this case by insertion of an explicit penalty of 10000. Why was only the character \'e dealt with? Conjecturally because effort was made via Desarmenien's hyphenation patterns to assure that no incorrect hyphenation point resulted from this trick. This is a standard we cannot maintain since it is implausible that anything but a standard 8 bit hyphenation file will be available to users. This means that the Desarmenien mechanism will occasionally allow an unacceptable hyphenation point. (We do not yet have statistics on this.) Some specialists, perhaps including Knuth, would frown on this. Other specialists believe that hyphenation should in any case be based more on phoenetics meaning and etymology than on spelling, and that sort is sometimes impossible to perfect with the Knuth-Liang approach. If perfection is impossible why insist on complete rigour? In the late 1980's, in the Latex style file called German.sty H. Partl (and collaborators) used a version of this "penalty and hskip" trick which is so clear and simple that it allows one to deal with all accents and all letters on an equal footing. For example Partl gives \gdef\allowhyphens{\penalty\@M \hskip\z@skip} % {\nobreak \hskip 0pt plus 0pt\relax} \gdef\highumlaut#1{\leavevmode\allowhyphens \accent127 #1\allowhyphens} Here the one-argument macro \highumlaut#1 is a new version of \"#1 designed to allow hyphenation after the accented character. This is certainly shorter and faster than Desarmenien's formulation. But there are some peculiarities to discuss. The subword following the accent does not include the character accented. This unfortunately supresses any kern between the character accented and a following character. Such kerns are respected by both Knuth's regime (\DKaccents) and (generalized) Desarmenien's (\JDaccents). (Note that kerning with *preceeding* material is not within the scope of either, but see below.) Another effect is to keep hyphens further from the accented character, which is not in itself a good thing, but it may also tend to eliminate bad hyphenations caused by the fake word break. (No detailed evaluation has been made.) The positive role of \leavevmode and the first \allowhyphens is unclear to us and we have let them drop. One of us (D.F.) noticed that in case the accented letter occurs at the end of a word, linebreak becomes impossible there. This may not occur in German but if one were to adopt the German approach for accute accent, defining \def\'#1{{\accent"13 #1\allowhyphens}} then linebreak would be prevented after all the innumerable French words terminating in e-acute! Here, surely, is the reason why \allowhyphens never gained currency in France. This disaster does not occur with the Desarmenien approach because the construction of the accented character concludes with a character and not an hskip. Examine now our own definition of \'; under our \JDaccents regime it is essentially \newbox\hy@accbox \def\'#1{{\accent"13 #1}\setbox\hy@accbox=\hbox{#1} \penalty10000\hskip-\wd\hy@accbox #1} We chose to use the standard \accent macro and nevertheless use Desarmenien's "penalty-hskip-letter" approach, at the (negligible!) cost of setting two copies of the letter to be accented, one exactly on top of the other. Our approach is thus not quite J. Desarmenien's but rather a trick simplification. It would have been possible to pursue the Desarmenien approach in all cases, but the coding would have been messy for capital letters and miscellaneous accents. Incidentally, Desarmenien makes it quite clear that he was privy to the folklore of Knuth and his students and has made no explicit claim to this trick. He did first bring it to fruition, and so we use his initials in \JDaccents. Granting that one chooses an approach making the letter to be accented part of the post-accent (sub)word, it might be desirable to make the pre-accent (sub)word terminate with the same letter. The two subwords then have that letter in common. This might give slightly more acceptable hyphenation points. And it would enable kerning with a preceeding letter, not just with a following letter. In principle the method extends to do this: \def\'#1{{#1\setbox\hy@accbox=\hbox{#1}% \penalty10000\hskip-\wd\hy@accbox \accent"13 #1% \penalty10000\hskip-\wd\hy@accbox #1}} THREE copies of the letter to be accented are then printed directly on top of one another. In principle, with digital raster imaging, the result should be identical to printing once. However many driver/font combinations will at low resolution slightly misplace the first of the three copies --- in case the accent occurs at the end of a long string of ordinary letters. This is because character width errors accumulate over a long string, and thereafter the \accent macro makes a fresh calculation of absolute position. The Textures screen driver dramatically exhibits this behavior. Knowing no cure for this malady, we have abandoned this "three copy" approach. Other versions are afflicted to a lesser degree by this phenomenon, because the shift involved, although present, is between characters several pixels apart. We believe that our "two-copy" or "double strike" approach will very seldom cause a problem. Alas, with some combinations of low resolution drivers and fonts some sort of drift has indeed been observed on screen and/or paper. For these unfortunate cases we give another approach \GFaccents (GF for German-French) which directly salvages the German approach by use of an \if ...\fi clause involving \futurelet. If the token following the character accented is a letter (like a) then one inserts before it a penalty of 1000 followed by an hskip of 0 points in order to start a new word; but in other cases (notably when a space comes next) one inserts nothing. This allows linebreaks after words ending in an accented character and followed by a space. Expressed in formulae \def\preLetterSkip@{\ifcat a\next\penalty\@M\hskip\z@skip\fi} \def\@@@@Allowhyphens{\futurelet\next\preLetterSkip@} \def\'#1{{\accent18#1}\@@@@Allowhyphens} is essentially our definition of the accute accent \' for the \GFaccents regime. Unfortunately, this solution will suppress all kerns between the character accented and a following character. Also, it can be defeated by nonstandard typing; for example TeX braces around an accented character will block hyphenation beyond. Further, 'futurelet' has occasional nasty side-effects in LaTeX. For example, disaster occurs when an accented character ends a 'tabular' environment as in: -- \begin{tabular} ... Painlev\'e\end{tabular} -- \author{J. Dieudonn\'e} It's easy to find work-arounds: just add a space after the terminal accented character (or put the whole word between curly braces) and your problems are over. Thus the modified versions -- \begin{tabular} ... Painlev\'e \end{tabular} -- \author{J. Dieudonn\'e } or \author{J. {\sc Dieudonn\'e}} will cause no problem. But all this shows that this solution is fragile. LEFTOVERS AND IMPERFECTIONS 0) \ogonek #1 provides a provisional ogonek accent under character #1 It is a translated copy of the left single quote. With this exception we treat only the accents in the TeXbook, page 52. 1) This version makes no effort to eliminate the following deficiency of Plain TeX: \uppercase and \lowercase fail when applied to the miscellaneous characters \oe, \OE, \ae, \AE, \o, \O, \ss and also to \i. Our \MLcodes macro merely serves to allow some hyphenations in words involving these characters. This is an imperfection. 2) If your computer and TeX have eight bit input capabilities, you may wish to type some accented characters in ready-made eight bit form using macros of the following sort. (Your TeX installation probably has a ready-made collection.) \catcode =13 \def{\'e} The formulas are here in an informal but portable ASCII format, and you have to put your local e-accute in place of before they will work. Next some changes that are common safety measures in various national macro packages and which we have included here only in the form of hints for the TeXpert, since we feel they are primarily the responsibility of other macro packages. 3) Problems with \write. Only in the last few years have simple and fast "sanitized" macros come into use for the purpose of writing pieces of the .tex file to an external file. The naked \write command makes a mess of accents and of many other things, and even may choke. The basic reference seems to be R. Whitney, TUGboat, November 1990. Here is a still simpler method: \newwrite\index \openout\index{myindex} \def\iWrite#1#2{\toks0{#2}% \immediate\write#1{\the\toks0} And a typical use is: \iWrite\index{...} The material ... then goes out to the file myindex.tex more or less untouched. This does not work well for non-immediate writes --- typically used to record page number of an index entry, but one reasonable remedy is to write the dangerous material with \iWrite to one file and the page numbers with \write to another file and then collate the two using some device like serial numbers or simply line numbers. 4) Cleaner messages. \message is of the same ilk as \write (and \edef). The above "\the\toks..." non-expansion trick can be applied to it to prevent mutilation or mishap. However most messages come from the bowels of macro packages and it is not easy to be sure of their contents. Active punctuation as well as accent macros often creep in. Recall that character category (eg. 13=active) is fixed forever at TeX's lips when the character is input. One can input the following redefinition while ;:?! are active. It will help combat these problems. But first check your current meaning of \message. \let\@oldmessage@=\message \def\message#1{\begingroup \def\'{'}\def\`{`}\def\"{"}% \def\i{i}\def\oe{oe}\def\ae{ae}\def\c{}% \def\.{}\def\u{}\def\v{}% \def\H{}\def\={}\def\relax{}% \edef~{\string~}% \edef\^{\string^^5e}% \edef:{\string:}% \edef;{\string;}% \edef?{\string?}% \edef!{\string!}% \@oldmessage@{#1}\endgroup} ------------------------- If you notice errors or oversights please contact us. --- Daniel Flipo, Lille --- Laurent Siebenmannn, Orsay April, August, December 1992