diff options
author | Norbert Preining <norbert@preining.info> | 2019-09-02 13:46:59 +0900 |
---|---|---|
committer | Norbert Preining <norbert@preining.info> | 2019-09-02 13:46:59 +0900 |
commit | e0c6872cf40896c7be36b11dcc744620f10adf1d (patch) | |
tree | 60335e10d2f4354b0674ec22d7b53f0f8abee672 /usergrps/uktug/baskervi/4_4/taylor.tex |
Initial commit
Diffstat (limited to 'usergrps/uktug/baskervi/4_4/taylor.tex')
-rw-r--r-- | usergrps/uktug/baskervi/4_4/taylor.tex | 138 |
1 files changed, 138 insertions, 0 deletions
diff --git a/usergrps/uktug/baskervi/4_4/taylor.tex b/usergrps/uktug/baskervi/4_4/taylor.tex new file mode 100644 index 0000000000..9614d41a7c --- /dev/null +++ b/usergrps/uktug/baskervi/4_4/taylor.tex @@ -0,0 +1,138 @@ +\title{Hyphenating British English} +\author[Philip Taylor]{Philip Taylor\\RHBNC\\\texttt{P.Taylor@Vax.Rhbnc.Ac.Uk}} +\begin{Article} + +\noindent +Many members of \ukt\ will already be aware that an enormous debt of +gratitude is owed to Dominik Wujastyk, who undertook the initial +generation of a set of hyphenation patterns for \TeX\ which were based +on a British (as opposed to American) dictionary. That debt of +gratitude is also owed to Oxford University Press, who donated their +internal word-list of some 160$\,$000 entries with primary, secondary and +tertiary breakpoints shewn as well as a `frequency-of-use' index for +each word. + +Dominik struggled against seemingly insuperable odds to process this +vast word-list; the standard \emph{Patgen} simply wasn't up to the task, and +despite the best efforts of Peter Breitenlohner, Wayne Sullivan and +many others, an attempt to build a suitably large DOS/Pascal version +was doomed to failure. In the end, Dominik discovered the \emph{web2c} +implementation of Karl Berry, and this, together with D J Delorie's +DJGPP C compiler, eventually enabled him to build a version of \emph{Patgen} +which could cope with a 160$\,$000-entry word-list. + +But although he did not know it, his troubles were but starting: once +he could read the word-list, he had to supply values for three of the +most cryptic and arcane variables in the known \TeX\ world: +good_wt+, ++bad_wt+ and +threshold+. These three variables control the entire pattern +generation process, yet even their inventor, Frank Liang, was forced to +confess in his Ph.D thesis (``Word Hy-phen-a-tion by Com-put-er'') that +he was unable to justify the values which he had used to generate the +American patterns other than by purely empirical means. And so Dominik, +too, used Frank's values, and produced patterns which, statistically +at least, were as valid as Frank Liang's. Dominik recorded +his experiences in a talk which he gave to the UK \TeX\ Users' Group +Easter meeting which was held at RHBNC last year. + +However, the generation of patterns is not a once-and-forever task: +those patterns which Dominik had produced were larger than the +American equivalent, requiring for some systems at least either a +specially `large' \TeX\ or at least a \TeX\ tuned to accommodate a +larger pattern set. Furthermore it correctly hyphenated only 90\% of +the words in the 160$\,$000-entry word-list, missing about 10\% +completely. There were also a few words which it was known would be +hyphenated incorrectly using Dominik's patterns, and which were +subsequently documented in the distributed \texttt{ukhyphen.tex}. + +With a sabbatical year in India on the horizon, Dominik felt that it +was time to hand over the baton; he had created a viable set of +patterns, and if someone else wanted to improve on them, that was up +to them. As Dominik knew that I had a considerable interest in +pattern generation, and that I had, in fact, offered to run +\emph{Patgen} on my VAX/VMS system if he had been unable to get a copy +working on any of the systems to which he had access, he asked if I +would like to become `custodian of the patterns', and I willingly +agreed. After all, Dominik had done all the hard work --- acquired a +suitable machine-readable dictionary, created the initial pattern set, +ascertained suitable values for +good_wt+, +bad_wt+, ++threshold+{\ldots} So my task should be infinitely more +straightforward: just build on what Dominik had already done. + +But of course, life is rarely that straightforward: as soon as I came +to build a large \emph{Patgen} for VAX/VMS, I discovered that the +Kellerman \& Smith changefile which I had no longer worked. +Furthermore, K\&S were unwilling to allow it to pass into the public +domain, so any development work on it would have been futile. My +saviour turned out to be Christian Spieler, who had already ported the +remainder of the standard \TeX\ distribution to Alpha/VMS; only +\emph{Patgen} remained, and once I had explained to him the importance +of that little-known utility, he willingly and promptly undertook an +Alpha/VMS port, including as standard the additional workspace which +it was known would be required. Within 24 hours a test version was +ready, and it worked beyond my wildest dreams: no second version was +needed, the very first version went straight into production, and that +same day I was able to produce a set of patterns which, statistically +at least, were as good as those produced by Dominik. + +But just as Dominik had had to battle with +good_wt+, +bad_wt+ and ++threshold+, I too had my own windmills at which to tilt: in my case +the problem came about because Christian had, very reasonably, based +\emph{his} implementation on \emph{Patgen2} (Peter Breitenlohner's +8-bit modifications to DEK's standard 7-bit Patgen). And Patgen2 has +four new variables with which to cope: +hyph_start+, +hyph_finish+, ++pat_start+ and +pat_finish+! + +Fortunately for me, these are nowhere near as arcane as +good_wt+ and +its ilk: the two +hyph_+ parameters allow multiple passes through the +dictionary to be subsumed into a single run, whilst the two +pat_+ +parameters allow the minimum and maximum length of pattern for each +pass to be separately specified. I do not pretend for one instant +that I \emph{fully} understand these, and I certainly don't pretend to +have more than the vaguest comprehension of the full implications of ++good_wt+, etc., but at least I can now generate patterns to my +heart's content, and the Alpha is busy doing that at the very time +that I am writing this report\ldots + +Between now and the time of publication of the next \BV, I hope to +have a much clearer understanding of the possible interactions between +the various parameters to \emph{Patgen}. And I hope, too, to have +prepared a new set of patterns which the UK community will be able to +adopt as a standard, together with a minimal set of exceptions which I +am sure will still be necessary. But work will not then stop: I have +already enlisted the help of a friend and sometime colleague, Chris +McManus, who I hope will be able to define some \emph{rules} for the +choice of values for the various parameters (Chris is a medic, +statistician and polymath \emph{extraordinaire}, and if anyone can +formulate rules for this problem, I am convinced that it is he); and +between us I hope that we will be able to publish some guidelines for +the use of \emph{Patgen2} --- guidelines which are sadly lacking at +the moment. + +And finally I hope that you, too --- the UK \TeX\ Community --- will +contribute to this project: for someone has to identify the mistakes +which the patterns allow, and such a task is far beyond the ability of +any one individual to undertake. Once a new definitive set of +patterns is announced, I will ask you all to look carefully at every +document that you typeset thereafter; and note whenever a hyphenation +looks strange; and to check it with a definitive list of valid +hyphenation points (I am using ``The Oxford Minidictionary of Spelling +and Word-Division'', but pointers to other definitive sources will be +most welcome); and if you find a genuine instance of a +wrongly-hyphenated word, then \emph{please} report it to me. I will +probably set up an e-mail list solely for this purpose, since I lose +paper mail almost by definition whilst e-mail remains accessible in +perpetuity. + +So, to summarise: building on previous work by Don Knuth, Frank Liang, +Peter Breitenlohner, The Oxford University Press, Dominik Wujastyk and +Christian Spieler (doubtless among many others), I am now in a +position to generate British English hyphenation patterns. In +conjunction with Chris McManus, I hope that we will be able to +formalise much that has been heuristic, or at best stochastic, in the +past. And with your help, I hope to be able to produce not only a +definitive set of British English patterns, but an equally definitive +(but, one hopes, very small!) set of exceptions. I look forward to +this challenge very much indeed. + +\end{Article} + |