summaryrefslogtreecommitdiff
path: root/macros/generic/hyacc-cm/hyacc-cm.doc
blob: 1d3a057a11a124a8ad15757efb41fe66cfea764d (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
 %%% File HyAcc-CM.doc
 %%  Documentation for typographers and TeX programmers
 %%  For basic user instructions see the header
 %%  of HyAcc-CM.tex, the one example below
 %%  and the format compilation mechanism
 %%  fe-format-dumper-cm


       OUR PERSPECTIVE ON THE HYPHENATION PROBLEM

    Jacques Desarmenien's devices for obtaining respectable
hyphenation in the presence of accents were never optimized nor
internationalized --- conjecturally because it was expected
that a version 3 of TeX would be based on MLTeX of M. Ferguson
and rather simply offer optimal hyphenation with no change in
the TeX environment.

    This did not come to pass.  When TeX version 3 came along,
it instead offered something better inasmuch as the accented
characters can be tailor-made under the new 8-bit regime
(rather than composed of accent plus character).  But, at the
same time, the cost was unexpectedly high:  new fonts, new
drivers, new norms, and new formats.  They are burdensome for
many European TeX users, and perhaps out of reach of many
non-European users.

    It was then announced that TeX and Metafont will not be
further changed by its author Don Knuth, nor by by anyone else
so long as they bear their present copyrighted names.

    Thus, rather unexpectedly, it has became desirable to have a
good international version of modest but decent hyphenation in
the presence of accents that works with absolutely all versions
of TeX since 1982, in particularly with TeX and Computer Modern
fonts as documented by Knuth.

    If you use TeX in a version <3, prior to fall 1989, you
should avoid recent hyphenation pattern files.  Also, for better
portability, always prefer pattern files in ASCII format.

    Many 7-bit european hyphenation pattern tables are sufficiently
compact and portable to circulate along with .tex manuscripts by
electronic mail. For example, f7hyphen.tex the 7-bit French
hyphenation file of Desarmenien is just 4Ko, while HyAcc-CM.tex is
5Ko (without comments).



A TYPICAL (FRENCH) USE OF HYACC-CM

    Let us consider the typical case of a  French visitor to the
Anglo-Saxon world who, to process his French typescript, has to
use a "mainframe" computer exploiting exclusively CM fonts.  If he
is fortunate enough to use Plain.tex, he can quite simply replace
TeX by INITEX at the cost of adding as a header to his typescript
the incantation:

 \input plain.tex %% standard, using US hyphenation for language 0
 \bgroup\catcode`@=11
  \global\def\US{\lccode`\'=0\nofrenchspacing\language=0}
  \newlanguage\l@French
  \global\def\FR{\lccode`\'=`\'\frenchspacing\language=\l@French}
 \egroup
 \FR
 \input f7hyphen.tex
 \input hyacc-cm.tex
 
His TeX typescript will be set by INITEX starting with French
hyphenation; \US restores US English.  This assumes the
typescript inputs adequate macros for French punctuation etc.

     Alternatively, he can make of this header a separate file
concluded by \dump newformat.fmt.  Then INITEX dumps a format,
say newformat.fmt, which can in turn be used under INITEX
by reducing the header to 

   &newformat.fmt

This second procedure is the one that best generalizes to massive
formats such as LaTeX.  Some ready-made recipes for them can be
found in fe-format-dumper-cm posted on ftp matups.matups.fr.
(These include fe-punct.tex and gllmts.tex giving rudimentary
macros for French versus English punctuation.)  However, in
hostile environments, the use of simple formulas as above is often
the most cost-effective strategy.

     In a worst possible scenario, INITEX (initex) is unavailable.
Then one has to use the English hyphenation in conjunction with
HyAcc-CM.  For this, the following simpler header can be used on
top of any standard English format:

 \def\US{\lccode`\'=0 \nofrenchspacing}
 \def\FR{\lccode`\'=`\' \frenchspacing}
 \FR
 \input hyacc-cm.tex
 \pretolerance=800 %% discourages hyphenation

Hyphenation points that occur have to be checked for correctness
and replaced by explicit hypenations (via \-) when necessary.


A HISTORY AND EXPLANATION OF THE MACROS

     The background reference for all matters to be discussed is
Appendix H of the TeXbook, and particularly pages 454-455.

    The story begins with Jacques Desarmenien's thesis, Univ.
Louis Pasteur, Strasbourg, written in the early 1980's partly at
Stanford under Knuth's influence.  The accessible reference is
Jacques D'esarm'enien: How to run TeX in a French environment:
hyphenation, fonts, typography, Tugboat, 5 (1984) 91-102.  This
reference discusses hyphenation through the use of fonts
containing ready-made accented characters of the sort slowly
coming into use now some 10 years later.  This package puts to
work devices for hyphenation which do NOT require new fonts, and
which were called upon a few years later when distribution of
new fonts was provisionally abandoned.

    It is often repeated that hyphenation of a word under 1982
TeX is blocked at points at and after the first accented
character because of the explicit kerns that the \accent
primitive of TeX introduces there.

    The following formula appeared in an unsigned file
feHyphen.tex that was available from Strasbourg from the mid
1980's.  According to Raymond Seroul, the author is J.
Desarmanien.  The resulting  bilingual fePlain format had wide
currency in France through Seroul's influence.  

 \newbox\accentb@x \newdimen\wdb@x
 \newbox\accentb@@x
 \let\aig@=\'%
 \def\'#1{\if e#1\setbox\accentb@x=\hbox{\char"13}%
    \setbox\accentb@@x=\hbox{e}%
    \wdb@x=.5\wd\accentb@@x \advance\wdb@x by -.5\wd\accentb@x
    \kern\wdb@x \char"13 \kern-\wd\accentb@x
    \penalty\@M \hskip-\wdb@x e%
    \else \aig@#1\fi}

A verbal interpretation of this is as follows.  Let the one
argument macro \' for accute accents be redefined as follows in
case the letter being accented is  e.  Build the accented
character <e-acute> from the accute accent character "13 and the
character  e  using boxes and kerns as common sense recommends,
except for one detail.  Let the last kern (which occurs just
before the letter  e)  become an hskip.  This causes TeX to
consider that a new word has begun with a letter  e  and allows
hyphenation in that (sub)word.  Normally an hskip is an
acceptable place for a linebreak, but this is prevented in this
case by insertion of an explicit penalty of 10000.

     Why was only the character \'e dealt with? Conjecturally
because effort was made via Desarmenien's hyphenation patterns
to assure that no incorrect hyphenation point resulted from this
trick.  This is a standard we cannot maintain since it is
implausible that anything but a standard 8 bit hyphenation file
will be available to users.

     This means that the Desarmenien mechanism will occasionally
allow an unacceptable hyphenation point.  (We do not yet have
statistics on this.)  Some specialists, perhaps including Knuth,
would frown on this.  Other specialists believe that hyphenation
should in any case be based more on phoenetics meaning and
etymology than on spelling, and that sort is sometimes
impossible to perfect with the Knuth-Liang approach.  If
perfection is impossible why insist on complete rigour?

    In the late 1980's, in the Latex style file called
German.sty H. Partl (and collaborators) used a version of this
"penalty and hskip" trick which is so clear and simple that it
allows one to deal with all accents and all letters on an equal
footing.  For example Partl gives

     \gdef\allowhyphens{\penalty\@M \hskip\z@skip}
  %          {\nobreak    \hskip 0pt plus 0pt\relax}

     \gdef\highumlaut#1{\leavevmode\allowhyphens
             \accent127 #1\allowhyphens}

Here the one-argument macro \highumlaut#1 is a new version of
\"#1 designed to allow hyphenation after the accented character.
This is certainly shorter and faster than Desarmenien's
formulation.  But there are some peculiarities to discuss.

      The subword following the accent does not include the
character accented.  This unfortunately supresses any kern
between the character accented and a following character.  Such
kerns are respected by both Knuth's regime (\DKaccents) and
(generalized) Desarmenien's (\JDaccents).  (Note that kerning
with *preceeding* material is not within the scope of either,
but see below.)

      Another effect is to keep hyphens further from the
accented character, which is not in itself a good thing, but
it may also tend to eliminate bad hyphenations caused by the
fake word break.  (No detailed evaluation has been made.)

      The positive role of \leavevmode and the first
\allowhyphens is unclear to us and we have let them drop.

      One of us (D.F.) noticed that in case the accented letter
occurs at the end of a word, linebreak becomes impossible there.
This may not occur in German but if one were to adopt the German
approach for accute accent, defining

   \def\'#1{{\accent"13 #1\allowhyphens}}

then linebreak would be prevented after all the innumerable
French words terminating in e-acute!  Here, surely, is the
reason why \allowhyphens never gained currency in France.

      This disaster does not occur with the Desarmenien approach
because the construction of the accented character concludes
with a character and not an hskip.

      Examine now our own definition of \'; under our \JDaccents
regime it is essentially

  \newbox\hy@accbox
  \def\'#1{{\accent"13 #1}\setbox\hy@accbox=\hbox{#1}
                \penalty10000\hskip-\wd\hy@accbox #1}

We chose to use the standard \accent macro and nevertheless use
Desarmenien's  "penalty-hskip-letter" approach, at the
(negligible!) cost of setting two copies of the letter to be
accented, one exactly on top of the other.  Our approach is thus
not quite J. Desarmenien's but rather a trick simplification.  It
would have been possible to pursue the Desarmenien approach in
all cases, but the coding would have been messy for capital
letters and miscellaneous accents.  Incidentally, Desarmenien
makes it quite clear that he was privy to the folklore of Knuth
and his students and has made no explicit claim to this trick.
He did first bring it to fruition, and so we use his initials in
\JDaccents.

      Granting that one chooses an approach making the letter to
be accented part of the post-accent (sub)word, it might be
desirable to make the pre-accent (sub)word terminate with the
same letter.  The two subwords then have that letter in common.
This might give slightly more acceptable hyphenation points.  And
it would enable kerning with a preceeding letter, not just with
a following letter.  In principle the method extends to do this:

  \def\'#1{{#1\setbox\hy@accbox=\hbox{#1}%
               \penalty10000\hskip-\wd\hy@accbox
                 \accent"13 #1%
                   \penalty10000\hskip-\wd\hy@accbox #1}}

THREE copies of the letter to be accented are then printed
directly on top of one another.

     In principle,  with digital raster imaging, the result
should be identical to printing once.  However many driver/font
combinations will at low resolution slightly misplace the first
of the three copies --- in case the accent occurs at the end of
a long string of ordinary letters.   This is because character
width errors accumulate over a long string, and thereafter the
\accent macro makes a fresh calculation of absolute position.
The Textures screen driver dramatically exhibits this behavior.
Knowing no cure for this malady, we have abandoned this "three
copy" approach.  Other versions are afflicted to a lesser degree
by this phenomenon, because the shift involved, although
present, is between characters several pixels apart.

    We believe that our "two-copy" or "double strike" approach
will very seldom cause a problem.

    Alas, with some combinations of low resolution drivers and
fonts some sort of drift has indeed been observed on screen
and/or paper.  For these unfortunate cases we give another
approach \GFaccents (GF for German-French) which directly
salvages the German approach by use of an \if ...\fi clause
involving \futurelet.  If the token following the character
accented is a letter (like a) then one inserts before it a
penalty of 1000 followed by an hskip of 0 points in order to
start a new word; but in other cases (notably when a space comes
next) one inserts nothing.  This allows linebreaks after words
ending in an accented character and followed by a space.
Expressed in formulae

 \def\preLetterSkip@{\ifcat a\next\penalty\@M\hskip\z@skip\fi}
 \def\@@@@Allowhyphens{\futurelet\next\preLetterSkip@}
 \def\'#1{{\accent18#1}\@@@@Allowhyphens}

is essentially our definition of the accute accent \' for the
\GFaccents regime.

     Unfortunately, this solution will suppress all kerns
between the character accented and a following character.  Also,
it can be defeated by nonstandard typing; for example TeX braces
around an accented character will block hyphenation beyond.

     Further, 'futurelet' has occasional nasty side-effects in
LaTeX. For example, disaster occurs when an accented character
ends a 'tabular' environment as in:

 -- \begin{tabular} ...  Painlev\'e\end{tabular}
 -- \author{J. Dieudonn\'e}

It's easy to find work-arounds: just add a space after the
terminal accented character (or put the whole word between curly
braces) and your problems are over.  Thus the modified versions

 -- \begin{tabular} ...  Painlev\'e \end{tabular}
 -- \author{J. Dieudonn\'e } or \author{J. {\sc Dieudonn\'e}}

will cause no problem.  But all this shows that this solution is
fragile.



LEFTOVERS AND IMPERFECTIONS

  0)  \ogonek #1 provides a provisional ogonek accent under character #1
It is a translated copy of the left single quote.  With this exception
we treat only the accents in the TeXbook, page 52.

  1)  This version makes no effort to eliminate the following
deficiency of Plain TeX:  \uppercase and \lowercase fail when
applied to the miscellaneous characters \oe, \OE, \ae, \AE,  \o,
\O, \ss and also to \i.  Our \MLcodes macro merely serves to
allow some hyphenations in words involving these characters.
This is an imperfection.

 2)  If your computer and TeX have eight bit input capabilities,
you may wish to type some accented characters in ready-made
eight bit form using macros of the following sort.  (Your TeX
installation probably has a ready-made collection.)

   \catcode <e-accute>=13 
   \def<e-accute>{\'e}

The formulas are here in an informal but portable ASCII format,
and you have to put your local e-accute in place of <e-accute>
before they will work.

    Next some changes that are common safety measures in various
national macro packages and which we have included here only in
the form of hints for the TeXpert, since we feel they are
primarily the responsibility of other macro packages.

  3)  Problems with \write.  Only in the last few years have
simple and fast "sanitized" macros come into use for the purpose
of writing pieces of the .tex file to an external file.  The
naked \write command makes a mess of accents and of many other
things, and even may choke.  The basic reference seems to be R.
Whitney, TUGboat, November 1990.  Here is a still simpler method:

    \newwrite\index \openout\index{myindex}
    \def\iWrite#1#2{\toks0{#2}%
      \immediate\write#1{\the\toks0}

And a typical use is:
     
    \iWrite\index{...}

The material ...  then goes out to the file myindex.tex more or
less untouched.

     This does not work well for non-immediate writes ---
typically used to record page number of an index entry, but one
reasonable remedy is to write the dangerous material with
\iWrite to one file and the page numbers with \write to another
file and then collate the two using some device like serial
numbers or simply line numbers.  

 4)  Cleaner messages.

     \message is of the same ilk as \write (and \edef). The above
"\the\toks..." non-expansion trick can be applied to it to
prevent mutilation or mishap.  

     However most messages come from the bowels of
macro packages and it is not easy to be sure of their contents.
Active punctuation as well as accent macros often creep in.
Recall that character category (eg. 13=active) is fixed forever at TeX's
lips when the character is input.

    One can input the following redefinition while
;:?! are active. It will help combat these problems.
But first check your current meaning of \message.

 \let\@oldmessage@=\message
 \def\message#1{\begingroup
    \def\'{'}\def\`{`}\def\"{"}%
    \def\i{i}\def\oe{oe}\def\ae{ae}\def\c{}%
    \def\.{}\def\u{}\def\v{}%
    \def\H{}\def\={}\def\relax{}%
    \edef~{\string~}%
    \edef\^{\string^^5e}%
    \edef:{\string:}%
    \edef;{\string;}%
    \edef?{\string?}%
    \edef!{\string!}%
    \@oldmessage@{#1}\endgroup}


 -------------------------


If you notice errors or oversights please contact us.



 --- Daniel Flipo, Lille <flipo@frcitl81.bitnet>

 --- Laurent Siebenmannn, Orsay <siebenmann@LALCLS.IN2P3.FR>

  April, August, December 1992