summaryrefslogtreecommitdiff
path: root/Master/texmf-dist/doc/fonts/itrans/tech.tex
blob: 3633921314bc4423ebda43a8983c1700706b5257 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
% $Header: /home/cvsroot/itrans/nextrel/doc/tech.tex,v 1.1.1.1 1996/11/29 01:45:31 avinash Exp $
\renewcommand{\arraystretch}{1.25}
\newcommand{\rarrow}{\mbox{---\hspace{-8pt}$>$}}
\newcommand{\SM}{\verb+\char92marathi+}
\newcommand{\EM}{\verb+\char92endmarathi+}
\documentstyle[11pt]{article}
\setlength{\textwidth}{5.75in}
\newfont{\devnf}{dnh at 15pt}

\begin{document}

\centerline{\LARGE\bf The {\em itrans} Preprocessor}
\medskip
\centerline{\bf TECHNICAL INFORMATION}
\medskip
\centerline{\bf \copyright 1991 Avinash Chopde}
\medskip
\hrule
\bigskip

\section{Reference}

{\tt
WARNING: The information in this document relates to ITRANS 2.11 and
earlier only.
THIS INFORMATION MAY BE INCORRECT/INCOMPLETE AS OF ITRANS 3.00 versions
onwards.
}

This is a project that aims to develop a single tool for handling the
printing of various indian language documents, assuming that the input is
in some transliterated form.
The transliteration mechanism for all indian languages will be through
the english language.

See the document {\em Printing Transliterated Indian Language Documents},
in the file {\em idoc.itx} for more details---that document is the
user manual for the {\em itrans} package.
This document discusses the technical aspects of the program.

\section{Input Parsing}

I have made the following assumptions regarding how the words in a
indian language (well, not all, but most of them) are assembled.
(In the following discussion, {\em alphabet} refers to the list of basic
characters in the language script, a
{\em letter} refers to an element from the complete list of characters---basic
plus all the composite forms.)

The alphabet is divided into two groups: vowels and consonants.

Each indian language letter may be a vowel, or a consonant-vowel pair, or a
ligature-vowel pair.

Ligatures are sequences of consonants.

If a consonant {\em xxx} is followed by another consonant {\em yyy}, then it is
assumed to imply a half-consonant, i.e., the half-form of {\em xxx} must be
displayed at that point.
Of course, if the ligature for the pair xxx-yyy exists, the ligature
is used instead.
Note that not all indian languages make use of ligatures.

Apart from the vowels and consonants, some special forms are also
provided, such as the chandra-bindu, the anuswara, virama, etc.
These special forms always form the suffix on the letter they affect,
i.e., you specify the letter first, then the special form.
A bunch of letters separated by white space or punctuation forms a word.

Based on these assumptions, a simple parser has been built, to recognize
the basic unit---the letter.

\section{Letter Construction}

The fonts that are used here provide only the basic characters---vowels,
consonants (both full and half forms), special forms, and, some ligatures. 
Every letter is built by printing its constituent characters.
This mapping of the consonant-vowel pairs, of the ligatures, etc, to
actual characters to be used from the specific font is specified through
the IFM file (indian language font metric file).
The IFM file is a text file, and follows somewhat the syntax of Adobe AFM
files.
In it is described how to construct every consonant-vowel form using the
given font, how to construct a ligature of consonant-consonant pairs,
the character codes of the special forms, etc.
The description is provided as a list of character codes, and their
offsets.
For example a description for some letter looks like:

PCC 97 0 0 ; PCC 129 -70 0 ;

This is a two character chain, first character has code 97, second
character has a code of 129.
The first character is printed with an offset (x,y) of (0,0).
Then the second character is printed with an offset of (-70,0), i.e., a
kern of -70 units.
One unit is equivalent of $1/1000$ of the current font size.
The entire IFM file structure is described in a later section.

As shown in the example above, the chain of characters is printed out by
performing two operations on every character code in the chain:

\begin{verbatim}
for every component of the form:    PCC <code> <dx> <dy> ;

move <dx>, <dy>;

display character <code>;

/*POSTSCRIPT FONT ONLY*/
    if (<dy> is non-zero) move 0, -<dy>;
    if (width of char <code> is defined as zero) move -<dx>, 0;
/*POSTSCRIPT FONT ONLY*/

\end{verbatim}

This relies on the fact that both \TeX\ and PostScript have a notion of a
current point.
Two values allow kerning to be specified in  both X and Y  directions.
If a movement in Y direction is des
Note that if the font being used is a PostScript font, then two other
actions may be performed.
If \verb+ <dy>+ is non-zero, itrans reapplies a negative offset to
restore the baseline y coordinate (otherwise the line will keep getting
skewed! Note that this step is not necessary for \TeX---it takes care 
of keeping the baseline horizontal).
Secondly, if the width of the character has been defined to be zero, a
negative x kern is applied.
This is done just for convenience---zero width characters are always
used as accent marks, and they should not contribute to the actual
character width, so the current position in X is restored to the original
value after a accent mark (zero width character) is printed.

Only PostScript fonts may have zero-width characters, the above does not
apply to \TeX\ Matafont fonts.
I.e, for \TeX, the last {\em move -<dx>, 0;} is never executed, even if
the charwidth for the given character is zero.
(This is partly due to my laziness, I did not want to write code that
reads in a TFM file to get the char widths!)

The above given description is used to specify both consonant-vowel
forms, and consonant-consonant (ligature) forms.
Currently, only ligatures composed of two consonants can be defined.

If a consonant-vowel form appears in the input text, but has not been
defined in the IFM file, an error message is printed on the screen, and
the character is omitted.

If a consonant-consonant-vowel (ligature) form appears in the input text,
but has not been
defined in the IFM file, it is not considered an error, instead, the
first consonant is printed in the half-form, and the second consonant
is printed out as its appropriate vowel form.
Long consonant chains, such as consonant-consonant-$\cdots$-consonant-vowel 
are also handled in a similar manner---first a check if made if a ligature
exists for any two consecutive consonants, if yes, it is used, else, the
half forms of the consonants are used.
Beginning with the first consonant in the list,
itrans checks if a double-consonant ligature has
been defined for  that consonant and  the next one in the list.
If such a character exists, then it is used and both consonants are consumed,
and itrans repeats the procedure for the next consonant.

There is an exception to the above rule: if at all possible, the last two
consonants are handled together, that is if a ligature of the last two
consonants
exists, that is used over the pairing that would result from the above method.
(See the user manual idoc.itx for example in the section Usage Hints.)

\section{IFM File Code Names}

The IFM file uses code names to refer to vowels and consonants.
For example, the ``I'' char in the input text for devanagari refers
to the ``ii'' code name in the IFM file (in this case, so does the input
char pair ``ii'').
Figure 1 shows the relationship between  the input vowel text forms
accepted by the
lexer, and the code name for the vowels in the IFM file.
(Note that no indian language script makes use of the complete set
of code names available in the IFM file.)

(Figures 1 \& 2 may be out-of-date, see the files imap.h and ilex.l
for correct info.)

Similarly, the consonant names are as given in figure 2.
Again, only a subset of these may be used by a particular indian
language script.

\bigskip
\begin{figure}
\begin{center}
\begin{tabular}{|c|c|c|}
\hline
{\em Input Text}	& {\em IFM Code Name}  & {\em Used For} \\
\hline\hline
a	& a	& devanagari, tamil \\ \hline
aa	& aa	& devanagari, tamil \\ \hline
i	& i	& devanagari, tamil \\ \hline
ii or I	& ii	& devanagari, tamil \\ \hline
u	& u	& devanagari, tamil \\ \hline
uu or U	& uu	& devanagari, tamil \\ \hline
Ri	& ri	& devanagari \\ \hline
RI	& rii	& devanagari \\ \hline
Li	& li	& devanagari \\ \hline
LI	& lii	& devanagari \\ \hline
e	& ay	& devanagari, tamil \\ \hline
E	& aay	& tamil \\ \hline
ai	& ai	& devanagari, tamil \\ \hline
o	& o	& devanagari, tamil \\ \hline
O	& oo	& tamil \\ \hline
au	& au	& devanagari, tamil \\ \hline
aM	& am	& devanagari\\ \hline
H	& aha	& devanagari\\ \hline
\end{tabular}
\end{center}
\caption {Vowel Input Forms and Code Names}
\end{figure}

\begin{figure}
\begin{center}
\begin{tabular}{|c|c|}
\hline
{\em Input Text}	& {\em IFM Code Name}\\ \hline \hline
k	& ka	\\ \hline
q	& kadot	\\ \hline
kh	& kha	\\ \hline
K	& khadot	\\ \hline
g	& ga	\\ \hline
G	& gadot	\\ \hline
gh	& gha	\\ \hline
ng	& nga	\\ \hline
c	& cha	\\ \hline
ch	& chha	\\ \hline
j	& ja	\\ \hline
z	& jadot	\\ \hline
jh	& jha	\\ \hline
jn	& jnh	\\ \hline
T	& tta	\\ \hline
Th	& ttha	\\ \hline
D	& dda	\\ \hline
.D	& ddadot	\\ \hline
Dh	& ddha	\\ \hline
.Dh	& ddhadot	\\ \hline
N	& nna	\\ \hline
t	& ta	\\ \hline
th	& tha	\\ \hline
d	& da	\\ \hline
dh	& dha	\\ \hline
n	& na	\\ \hline
p	& pa	\\ \hline
\end{tabular}
\hfill
\begin{tabular}{|c|c|}
\hline
{\em Input Text}	& {\em IFM Code Name}\\ \hline \hline
ph	& pha	\\ \hline
f	& phadot	\\ \hline
b	& ba	\\ \hline
bh	& bha	\\ \hline
m	& ma	\\ \hline
y	& ya	\\ \hline
r	& ra	\\ \hline
l	& la	\\ \hline
v	& va	\\ \hline
sh	& sha	\\ \hline
shh	& shha	\\ \hline
s	& sa	\\ \hline
h	& ha	\\ \hline
ld	& lda	\\ \hline
x	& ksha	\\ \hline
gy	& gya	\\ \hline
ny	& nya	\\ \hline
n\^	& nnx	\\ \hline
R	& rra	\\ \hline
.r	& rahalf	\\ \hline
.n	& anusvara	\\ \hline
.c	& chandra	\\ \hline
.C	& chandrabindu	\\ \hline
.h	& viraam	\\ \hline
SRI	& sri	\\ \hline
AUM	& aum	\\ \hline
\end{tabular}
\\
\end{center}
\caption {Consonant Input Forms and Code Names}
\end{figure}

\section{IFM File Data}

Currently, the IFM data resides in a file of its own, file name ends in 
{\em .ifm}, example {\em dvnc.ifm}.
I still hope
that I may some day be able to include the IFM data into the Adobe AFM file.
Hence, all the lines in the IFM file start off with the word {\em
Comment\/}, so that regular Adobe programs that scan the AFM file may
skip over this data.
The marker {\em -I-} follows the comment word, and this marks the line as
containing legal metric data for the indian language.

Every line in the AFM file consists of semicolon separated fields.
Each line describes a composite alphabet form, such as a consonant-vowel
description or a consonant-consonant form, or a vowel form, or a half
form of a consonant, etc.
The english words used to specify the characters of the indian language
alphabet are meant to sound right, in some vague manner, and are usually
self-descriptive.
These english words (such as ii, aa, aha, etc) have no relation to what
the user types to get the required character, that mapping is defined by
the lexer input file, {\em ilex.l}.

Every field in a line in the IFM file starts off with an opcode,
describing what to expect next.

The following lines are representative of the data in the AFM file:

\begin{verbatim}
Comment -I- StartINDIAN
Comment -I- FONT marathi dmta.afm
Comment -I- CC a 2 ; PCC 97 0 0 ; PCC 129 -70 0 ;
Comment -I- CC a 2 ; PCC 97 0 0 ; PCC 129 -70 0 ;
Comment -I- CC gha-ii 3 ; PCC implicit 0 0 ; PCC 129 -70 0 ; PCC 132 0 0 ;
Comment -I- CCS gha ga ;
Comment -I- CCADD tmplA ;
Comment -I- EndINDIAN
\end{verbatim}


The opcodes StartINDIAN and EndIndian are used to bracket the indian
language character description data.

The FONT opcode gives the name of the language, and the name of the file
which contains the AFM data for this font.
If the font is a Metafont description, then the TFM file name is
specified here.
The TFM file name is just used as an indication that the font used is a
\TeX\ font, the file is not actually opened or read.
Thus, if the font used is a Metafont description, you could as well say
``junk.tfm'' in the FONT opcode---only the extension ``.tfm'' is
important.
The first word in the FONT line is the language, and it must be one of the
languages ITRANS recognizes (see the function get\_lang\_tok() in marker.c).
This word determines how the input is scanned for tokens.
For example, the input letters ``Ri''refer to a vowel in
sanskrit/marathi/hindi, but in tamil, that vowel is not present.
But tamil uses the ``R'' symbol for a consonant, so the input ``Ri''
is scanned as the consonant ``R'' followed by the vowel ``i'', while in
sanksrit it is scanned as the vowel ``Ri''.
Thus, it is important to define the correct language in the FONT statement
in the IFM file.

The CC words stands for composite character, and it defines how to
construct the given character using the font.
In the example above, it shows that the {\em a} character is made up of
two units: char code 97, and char code 129.
See the previous section regarding the PCC opcode.
The second CC line above describes how to create the {\em ii} vowel form
of the consonant {\em gha}.
The first PCC character code in the line is not a number, but states {\em
implicit}.
This requires that a special character, called {\em gha-implicit} be
defined earlier.
Semantically, gha-implicit means the implicit form of the consonant gha.
Thus, whenever the code implicit appears in the description of some
consonant xxx, at that point, the program inserts the definition of the
xxx-implicit letter.
(Naturally, the description of xxx-implicit cannot have the code implicit 
in its description.)

Most consonants are similar in the manner in which the vowel forms are
constructed, using the implicit form of the consonant, and a few other
character forms.
For example, kha-aa, gha-aa, da-aa, dda-aa, are all constructed by using
this description:
PCC implicit 0 0 ; PCC 130 -70 0 ;
So, instead of restating this description for all these consonants, the
CCS keyword can be used instead.
The CCS keyword assigns equivalences, 

CCS xxx yyy ;

states that a given consonant
xxx is similar to a already defined consonant yyy, and  if some vowel
form (x) is missing, i.e. xxx-x description is missing, then it looks up
the description for yyy-x, and uses that.
This chaining can be made as deep as necessary:

CCS bbb aaa ;

CCS ccc bbb ;

CCC ddd ccc ;

etc

CCS also works similarly for ligature forms:

CCS ga-ra tmplC ;

which states that the ga-ra ligature should use the form of the tmplC
dummy consonant (dummy consonants are explained further in this section).
Note that it is usually dangerous to specify ligature equivalence to one
of the constituent consonants, since most of the consonants do not use the
codename "implicit" in all their form definitions.
Thus, be careful of such definitions:

CCS ga-ra ga ;

or

CCS ga-ra ra ;

This will create problems if the half form of the ga-ra ligature is
required since both ga and ra (and every other consonant too)
have a half-form definition that does not include the code "implicit",
hence instead of the ga-ra-half form, what will print out will
be just the ga-half form or the ra-half form, depending on which
CCS line (from the above two lines) is present in the IFM file.
One way around this problem is to specifically define the half-form of
the ga-ra ligature, so as to stop the program from looking for it 
through the CCS chain.
It is usually only the half form that causes problems (as of october 1991),
since all the other forms do contain the "implicit" code in their
definitions.

A special consonant form, ``*'' is available for use with the CCS
keyword for ligatures:
It acts as a meta-character implying all the consonants.

CCS *-ra tmplC ;

CCS ga-* tmplC ;

Instead of providing consonant equivalences between consonants,
additional dummy consonants may be created, these exist only in the IFM
file, and used only for equivalencing a real consonant.
The CCADD line defines the creation of a placeholder consonant.

CCADD tmplA ;

This states that the IFM will make use of a dummy consonant called tmplA,
and then all its vowel forms (including the half form) can be defined.
See the file dvnc.ifm for complete example.
All that tmplA is used for is to define equivalences, again from the
dvnc.ifm file, you will see lines like:

CCS chha tmplA ;

CCS tta tmplA ;

which state that if some required vowel form of chha or tta is missing,
then try to use the definition of the same vowel form in tmplA.

Each IFM file can make up to 10 such dummy consonants.

\section{Example Usage or When to Make New IFM Files}

So, to what end  can this knowledge  of the IFM file format be applied ?

\begin{enumerate}
\item Assume that you need to disable all ligature printing, you
are printing at a very small font size, and ligatures do not print out
legibly.
To do this, you should copy the dvnc.ifm file to sdvnc.ifm, and remove all
ligature definitions from sdvnc.ifm.
Ligature definitions are those which contain two consonants separated by
a dash: examples:
{\obeylines
CCS tta-tta  tmplA ;
CC tta-tta-implicit 1 ; PCC 147 0 0 ;
}
All such ligature definitions can be deleted from sdvnc.ifm.
And then, in your input file that contains the transliterated text, add
the command:

$\backslash$indianifm=sdvnc.ifm

(See the user manual on itrans for more information on the indianifm
keyword.)

Now, no ligatures are available, and nothing needs to be done to the
input text at all to suppress the ligatures.
Instead of the ligatures, the half-forms of the consonants will be used
wherever required.

\item Another use would be if you do not like the spacing of certain
characters when they are composed to form consonant forms.
Well, as seen in the previous example, it is quite simple to copy the IFM
file into a new one, change the deltas (the numbers) as required, and use
the new IFM in the input text.

\item By changing the character definitions in the IFM file, one can
produce different printed forms for the character.
For example, to print the normal form of {\em ii} in devanagari prints out
as: {\devnf \char105\kern-0.04em\char128}.
% itrans: CC ii 2 ; PCC 105 0 0 ; PCC 128 -40 0 ;
Now, what if you need to make it print out as: {\devnf \char97\char132} ?
This is easily done, just comment out the old definition of the {\em ii}
character (it is called {\em ii} in the ifm file), and make it read:

{\obeylines
Comment -I- CC ii 2 ; PCC 97 0 0 ; PCC 132 0 0 ;
}

Now, all instances of {\em ii} in the input text will print out as:
{\devnf \char97\char132 }.

\renewcommand{\arraystretch}{1.25}
\renewcommand{\arraystretch}{1.25}
\renewcommand{\arraystretch}{1.25}
\renewcommand{\arraystretch}{1.25}
This method can be used to change any character form you need, as long as
the constituent parts of the new character are available in the font.

\item You have received a new font, either you developed it yourself, or
\renewcommand{\arraystretch}{1.25}
got it elsewhere, and you would like to use it through the itrans
mechanism.
This involves two steps.
First, you need to map the vowels and consonants available through itrans
(see figures 1 and 2) to the font characters.
Second, you create a IFM file for the font.
The first step is the biggest hurdle, especially if you feel that you
need to add new consonant names to ilex.l.
(Of course, that implies lots of source code changes: now iyacc.y will
have to be modified to accept the new token, imap.h will have to edited
to add the codename for the font, and font.c will have to be edited to
add the new codename into a static data structure!)
If that happens, send me e-mail, maybe I will permanently add the name to
ilex.l, but I  hope that current set of names accepted by ilex.l will
suffice.
If they do, then no source code changes are necessary, just need to
create the IFM file and you are in business.

If you wish, you could send me the IFM file you create, and the font
(if it is in public domain, otherwise the IFM file only will suffice),

\end{enumerate}

\end{document}