1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<HTML>
<HEAD>
<META NAME="GENERATOR" CONTENT="SGML-Tools 1.0.9">
<TITLE>xindy Manual: How the Index Processor Works</TITLE>
<LINK HREF="manual-3.html" REL=next>
<LINK HREF="manual-1.html" REL=previous>
<LINK HREF="manual.html#toc2" REL=contents>
</HEAD>
<BODY>
<A HREF="manual-3.html">Next</A>
<A HREF="manual-1.html">Previous</A>
<A HREF="manual.html#toc2">Contents</A>
<HR>
<H2><A NAME="how-works"></A> <A NAME="s2">2. How the Index Processor Works</A></H2>
<P>
<P>
<H2><A NAME="ss2.1">2.1 Data Flow</A>
</H2>
<P>Usually document preparation systems produce some form data that
describe the <EM>index entries</EM> and the <EM>locations</EM> they point to.
This data is called the <EM>raw index</EM> since it consists of raw data
which contains structured information about the entries of the index.
The raw index is fed into the index processor and is processed
according to a specification called the <EM>index style</EM>. This is a
user-definable description how the index is to be processed, what
sort rules for the keywords should be used, which kind of locations
may appear, and finally, what tags should be emitted when writing the
index into the output file. The result is sometimes fed back into the
document preparation system, as it is in the case of TeX, or used
otherwise. The following figure illustrates this embedding into the
document preparation process.
<P>
<FIGURE>
<EPS FILE="dataflow.eps">
<IMG SRC="dataflow.gif">
</FIGURE>
<P>The <SF>xindy</SF> system is oriented on book-like indexes in a form that
is described later in detail. The format of the input data is fixed
and a small tool that converts some system dependent raw index in a
form suitable for <SF>xindy</SF> should not be a big hurdle. The user is
normally not involved in the correct format suitable for <SF>xindy</SF>
and some tools will make this work invisible to him.
<P>In contrast, the output of the <SF>xindy</SF> system is highly
configurable and can produce a variety of different markups.
Additionally, the way <SF>xindy</SF> processes the index data can be
parametrized and controlled in many ways and gives the user a powerful
framework for describing complex indexes. The user can tell <SF>xindy</SF>
how it should produce an output according to his wishes or simply use
some of the predefined styles that come with the distribution.
<P>
<P>
<P>
<P>
<H2><A NAME="general-concepts"></A> <A NAME="ss2.2">2.2 General Concepts and Terms</A>
</H2>
<P>In the following sections we introduce some terms that are necessary
to understand the way <SF>xindy</SF> works.
<P>The development of <SF>xindy</SF> started with the devlopment of a model
of a <EM>book-like</EM> index. This rather vague term will now be fleshed
out by giving a semi-formal description of the model. If the structure
of an index conforms to this model it can be processed by <SF>xindy</SF>.
We first present a diagram that describes the structure of a processed
index. It is a hierarchical recursive data structure which we will
describe now.
<P>
<FIGURE>
<EPS FILE="index.eps">
<IMG SRC="index.gif">
</FIGURE>
<P>Boxed objects contain elements from the raw index such as the keywords
or the location reference layers. We start from the root of the
diagram and explain the different elements.
<P>
<P>
<P>
<DL>
<P>
<DT><B>index</B><DD><P>An index consists of a list of <SF>letter-groups</SF>.
<P>
<P>
<A NAME="def letter-group"></A> <DT><B>letter-group</B><DD><P>The letter groups serve as containers for
<SF>indexentries</SF>. The letter groups are intended to group all
index entries sharing a common <SF>keyword</SF> prefix, allowing the
assignment of a specific markup for each group. For example, indexes
are often organized in a way that new index entries beginning with a
new letter are separated by a specific markup. The following example
illustrates this style.
<P>
<PRE>
B
bread 25, 27
butter 26
M
marmalade 19
milk 21
</PRE>
<P>Here we have two letter groups for the letters `B' and `M' because the
keywords `bread' and `butter' share the same prefix `b' and
`marmalade' and `milk' share the prefix `m'. <SF>xindy</SF> can be advised
to define arbitrary letter groups with a particular markup for each of
them. See command <CODE>define-letter-group</CODE> for further details.
<P>
<P>
<DT><B>indexentry</B><DD><P>An index entry consists of three components: the
<SF>keyword</SF> and optionally the list of <SF>location-class-groups</SF> and
the sub-<SF>indexentries</SF>. An example:
<P>
<PRE>
bread 25-30
brown 26
white 27
</PRE>
<P>This index entry consists of the keyword `bread', the
location class list `25-30' and the sub-entries `brown 26' and `white
27'. The sub-entries themselves are complete index entries as well,
forming a recursive data structure.
<P>
<P>
<DT><B>keyword</B><DD><P>The keyword is the identifier of an index entry. It
consists of a list of strings being the <EM>key</EM> if the index entry. In
our model the keyword constists of a list of strings, since indexes
are often organized hierarchically with different layers. In the
example
<P>
<PRE>
bread 25-30
bread, brown 26
bread, white 27
</PRE>
<P>we have keywords `bread', `bread, brown' and `bread, white'.
<P>A keyword is actually separated into four different keys. The
<SF>main-key</SF> is the key that is specified from the author who indexes
a document. The <SF>merge-key</SF> is derived from the main key with the
so-called <EM>merge mapping</EM>. This mapping is accomplished with a set
of rewrite rules. The <SF>sort-key</SF> is derived from the merge key with
another set of rewrite rules called the <EM>sort mapping</EM>. The
<SF>print-key</SF> can be specified by the user explicitely or, if it is
ommited, equals the main key. The rewrite rules consist of string
substitutions that also include regular expression features.
<P>The following figure describes this:
<P>
<FIGURE>
<EPS FILE="mappings.eps">
<IMG SRC="mappings.gif">
</FIGURE>
<P>
<P>The <SF>merge-key</SF> is necessary to define the equality of
index entry keys. See command <CODE>merge-rule</CODE> for further details about
defining the merge mapping.
<P>The <SF>sort-key</SF> is necessary to define a total order on the
keywords. Since we need a scheme that is independent from any specific
character set such as ASCII or ISO-Latin, we must be able to define
completely new alphabets. The index entries are then sorted according
to the ISO-Latin alphabet. See the command <CODE>sort-rule</CODE> for an
explanation how these mapping can be specified.
<P>The <SF>print-key</SF> is used for the final markup and contains the
string that appears in the generated markup.
<P>
<P>
<DT><B>location-class-group</B><DD><P>There exist two kinds of
location class groups namely <SF>location-reference-groups</SF> and
<SF>cross-reference-groups</SF>.
<P>
<DT><B>location-reference-group</B><DD><P>consists of a list of
<SF>attribute-groups</SF>.
<P>
<P>
<DT><B>attribute-groups</B><DD><P>Each <SF>location-reference-group</SF> consists of
a list of <SF>attribute-groups</SF>. Attribute groups are defined by a set
of <SF>attributes</SF>. In the markup phase, all location references are
sorted according to the group number their attributes belong to.
Additionally all location references of a particular attribute group
appear together in the index.
<P>See command <CODE>define-attribute-groups</CODE> for further information about
the handling of attribute-groups.
<P>An attribute group consists of a list of <SF>location-references</SF>
and/or <SF>ranges</SF>.
<P>
<P>
<A NAME="def locref-class"></A> <DT><B>location-reference-class, location-reference, location</B><DD><P>Each
index entry of an index references at least one <SF>location</SF> in the
document. A <SF>location</SF> is a logical or physical part of a document,
such as a single page, a section or subsection, etc. The symbolic name
of a location is called a <SF>location-reference</SF>. The
location reference is always represented as a string (a sequence of
characters). Each location reference is associated a special tag
called <SF>attribute</SF>. The following examples may give you an idea how
location references could look like.
<P>
<BLOCKQUOTE>
<CODE>"25" </CODE> represents a page number.
</BLOCKQUOTE>
<P>
<BLOCKQUOTE>
<CODE>"Chapter-3"</CODE> represents the third chapter of a document.
</BLOCKQUOTE>
<P>
<BLOCKQUOTE>
<CODE>"A-I" </CODE> represents the first page, written in uppercase roman
numerals of the appendix `A', separated by a hyphen.
</BLOCKQUOTE>
<P>
<BLOCKQUOTE>
<CODE>"2.3.4" </CODE> represents the sub-sub-section 2.3.4 of a document.
</BLOCKQUOTE>
<P>As we can see, the location references are often composed
hierarchically of smaller entities, such as numbers or names. All
location references that belong to the same class form a
<SF>location-class</SF>. A location class is composed of <SF>alphabets</SF>
and <SF>separators</SF>. See command <CODE>define-location-class</CODE> for a
description how classes can be defined.
<P>
<P>
<DT><B>alphabet, separator</B><DD><P>As we have seen in the previous section,
location classes consist of alphabets and separators. xindy has the
following built-in set of basic alphabets:
<P>
<BLOCKQUOTE><CODE>
<PRE>
<tag/<tt/arabic-numbers// contains all non-negative numbers beginning
with zero: 0, 1, 2, ....
<tag/<tt/roman-numerals-uppercase// the roman numerals I, II, III,
(IIII/IV), V, ... It recognizes the old and new roman numeral system.
<tag/<tt/roman-numerals-lowercase// the same for the lowercase roman
numerals.
<tag/<tt/ALPHA// the US-ASCII alphabet (uppercase letters).
<tag/<tt/alpha// the US-ASCII alphabet (lowercase letters).
<tag/<tt/digits// the digits 0, 1, 2, 3, 4, 5, 6, 7, 8 and 9 in this
order.
</PRE>
</CODE></BLOCKQUOTE>
<P>The user is free to define new alphabets as he wishes. See the
commands <CODE>define-alphabet</CODE> and <CODE>define-enumeration</CODE> for further
details.
<P>A complete location class can be defined by composing alphabets and
separators. The location class of the example <CODE>"A-I"</CODE> can be
defined as follows
<P>
<PRE>
(define-location-class "appendix"
(ALPHA "-" roman-numerals-uppercase))
</PRE>
<P>This is a list that completely describes all possible instances of
this location class. Other valid members are <CODE>A-II</CODE>, <CODE>B-VI</CODE>,
etc. Location classes are defined with the command
<CODE>define-location-class</CODE>.
<P><SF>xindy</SF> reads the incoming strings representing location references
and tries to match them against a previously defined location class.
There exist no predefined location classes.
<P>
<DT><B>range</B><DD><P>A <SF>range</SF> is a pair of <SF>location-references</SF>. A
sequence of location references following one another is transformed
into an range if the length of the sequence exceeds a certain
user-definable limit. For example, the location references 11, 12, 13
and 14 may be transformed into the range 11--14. See command
<CODE>define-location-class</CODE> for further details.
<P>
<P>
<DT><B>cross-reference-group</B><DD><P>Container for all
<SF>cross-references</SF> of a particular <SF>cross-reference-class</SF>.
<P>
<P>
<A NAME="def crossref-class"></A> <DT><B>cross-reference-class, cross-reference</B><DD><P>A <SF>cross-reference</SF> is
an unstructured reference. <SF>Cross-reference-classes</SF> are used to
define different classes of cross references. All elements of a
cross reference class appear together in the index. A cross reference
consists of a list of strings.
<P>
<P>
<DT><B>attributes</B><DD><P>Each location reference contains a so-called
<SF>attribute</SF>. Attributes may be used to give a location reference a
different meaning that may influence the processing and markup of this
reference. Sometimes location references pointing to semantically
different locations such as a definition or a usage of a term are
tagged differently. They often appear in <EM>italics</EM> or <B>boldface</B>
indicating this difference. An attribute may now be used to express
this difference. Since <SF>xindy</SF> needs to know the possible
attributes before the index entries are read in, the command
<CODE>define-attributes</CODE> must be used to define a set of attributes and
how they interact.
<P>
</DL>
<P>
<P>
<P>
<P>
<HR>
<A HREF="manual-3.html">Next</A>
<A HREF="manual-1.html">Previous</A>
<A HREF="manual.html#toc2">Contents</A>
</BODY>
</HTML>
|
