path: root/info/laan/pwt/pwt.ind

\input blue.tex \loadindexmacros \report 

 \bluepictures\indmodelpic

\bluechapter Creating an Index

\beginsummary
The creation of a modest index within 
a one-pass \TeX{} job has been treated. 
In general a proof run and a final run are needed.
\endsummary

Making an index is an art. The fundamental problem is 
\bluedisplaycenterline What to include in an index?

Computer-assisted indexing is not simple either.
Issues are 
\bitem the markup of keywords or phrases
\bitem to associate page numbers 
\bitem to sort and compress raw Index Reminders (^{IR}s), and
\bitem to typeset the result.
\smallbreak
My approach is to create proof indexes\Dash also called mini-indexes\Dash
for each chapter and 
learn from those what should be included in the total index.
I perceived this as very pleasant in practice. 
Even if you prefer \cs{makeindex} for the real index, this
processing on the fly of a chapter index can be of great help.\ftn{It is said
   that the automatic generation of an index is a feature of the Literate Programming tools.
   For LP with \TeX{} as such, as for example Gurari's Pro\TeX, this on-the-fly indexing
   within \TeX{} can be used.}

\bluehead Use

I'll show how to mark up Knuth's four types of IRs, 
how to mark up accents,  
how to mark up font switching, and 
how to mark up spaces as part of the IR. 

\blueexample Markup, commands and resulting index

The right column has been obtained via 
\bitem ^|\loadindexmacros|, at the beginning of the script 
\bitem ^|\sortindex|, at the place of indexing, and
\bitem ^|\pasteupindex|, for the pasteup of
       the index.
\smallbreak
 \thisverbatim{\catcode`\|=12
   \catcode`!=12 \unmc
   \catcode`\*=0 }
\begindemo
Types of IR 
 0  ^{return}
 1  ^|verbatim|
 2  ^|\controlsequence|
 3  ^\<syntactic quantity>
Accents ^{\'el\`eve!},
font changing ^{\bf bold}
and spaces ^{control\ symbol}
Control sequences 
  ^{\TeX, and \AmSTeX}
  ^{Lamport and \LaTeX}
brackets ^{\tt< \rm and \tt>}
\newpage ^{return}
\newpage ^{return}%on purpose
\sortindex\pasteupindex\bye
*yields\obeylines
\quad {\tt {}< \rm {}and \tt {}>}{} {\oldstyle1}
\quad {\bf {}bold}{} {\oldstyle1}
\quad {control\ symbol}{} {\oldstyle1}
\quad {\tt \char 92\hbox {controlsequence}}{} {\oldstyle1}
\quad {\'el\`eve!}{} {\oldstyle1}
\quad {Lamport and \LaTeX{}} {\oldstyle1}
\quad {\TeX, and \AmSTeX{}} {\oldstyle1}
\quad {return}{} {\oldstyle1}--{\oldstyle3}
\quad $\langle \hbox {syntactic\ 
             quantity}\rangle ${} {\oldstyle1}
\quad {\tt verbatim}{} {\oldstyle1}
\enddemo
The  representation of page numbers as a range comes out automatically.

\exercise What makes a good index?  Of course this is a million-dollar
          question. Let us concentrate on the number of entries and on the
          number of page numbers per entry. Which of the two extremes
          sketched below is the better one in your opinion?
          One with many entries pointing to issues spread throughout the book\Dash
          like \TB{} ;-))), and pushing the limits just for the imagination,
          an index with pointers to related work on the internet, accessible
          by just clicking the mouse\Dash
          or one with few page numbers per entry\ftn{Courtesy Erik Frambach.}?
\answer As usual it all depends on your application. End of answer.
        But\Dash there is always a but\Dash the complaint I heard
        most about \TB{} was that the information is spread all over,
        and that it is hard to find what you are looking for.
        Therefore I consider a few page numbers per entry beneficial.
        (Let us forget about the intrinsic complexity of the 
         subject, certainly at the time.)
        BLUe's format supports scrutinizing parts of an index, 
        because it is so easy to generate an index per chapter on the fly.
        It is hardly not more difficult than generating a table of contents. 
        An index per chapter can be scrutinized more easily, and 
        redundancies removed. 
        That the index provides a mechanism
        to link things over chapters is a good thing, however.
        Don't misunderstand me. 
        But don't overuse it, IMHO, with all respect. Remember DeVinne's
        adage `The last thing to learn is simplicity.'

%end answer

\bluehead Markup of Index Reminders

IR-s are at the heart of the process. ^^{IR,\ markup} 
Knuth distinguished {\oldstyle4} types to facilitate the outside processing. 
I'll adopt his IRs syntax and types.

\bluesubhead Syntax

Knuth's IRs obey the following syntax. ^^{IR,\ syntax} 
\begincenterverbatim
<word(s)>!]!!<digit>!]<page number>.
!endcenterverbatim
The digits  {\oldstyle0}, {\oldstyle1}, {\oldstyle2}, or {\oldstyle3}
denote the types: 
words,
verbatim words, 
control sequences, and 
syntactic quantities.
A user does not have to bother about the digits nor about the
page numbers.
Knuth has adopted the accompanying conventions for 
the word(s) of IRs.\ftn{See \TB{} {\oldstyle424}, for the IR types, 
   and what is typeset in the result. 
   In \cs{vref} the markup is inserted as replacement
   text of \cs{next}. What is set in the index is governed by the macros
   which are included after \cs{begindoublecolumns} in the \TeX book script.}
$$\vbox{\offinterlineskip\def\tstrut{\vrule height2.5ex depth.5ex width0pt}
\halign{\tstrut#\hfill\quad\vrule\quad&#\hfill\quad\vrule\quad&#\hfill\cr
Mark up&Typeset in copy$^*$&IR \cr
\noalign{\hrule}
|^{...}|  &\dots        &|... !!0 |$\langle page\, no\rangle$.\cr
|^!vrt...!vrt|  &|!vrt...!vrt|&
                              |... !!1 |$\langle page\, no\rangle$.\cr
|^!vrt\...!vrt| &|!vrt\...!vrt| &
                               |... !!2 |$\langle page\, no\rangle$.\cr
|^\<...>|&$\langle\dots\rangle^{**}$&
                               |... !!3 |$\langle page\, no\rangle$.\cr
\noalign{\vskip.5ex\hrule width1cm\relax\vskip1ex}
\multispan3{\quad$^*\,$\vrt\dots\vrt\ denotes manmac's, TUGboat's,\dots
   verbatim  \hfil}\cr
\multispan3{\quad$^{**}\,$in \cs{rm} \hfil}\cr
}}$$
For the user the word(s) is (are) important. 
The markup allowed  for the IRs and the result in the copy are
given in the accompanying table. 

\bluesubhead Markup
 
The markup for IRs is near to natural.
Precede the entry by a circumflex, or a double one in case of
a silent\ftn{Silent IRs mean that these  will appear
   only in the index, not on the page.}
index entry.

\blueexample IR markup

\thisverbatim{\catcode`\|=12 \catcode`\^=12 
   \catcode`!=12 \catcode`*=0 \unmc}
\beginverbatim
^{\'el\`eve!}^|verbatim text|^|\controlsequence|^\<a metalinguistic variable>
^^\<a metalinguistic variable>  %for silent ones, double the ^ 
{\sl^{ligatures}} |'$|^|\,||$''|%from the TeX book script
^^{markup commands, see control sequences}
^{Lamport and \LaTeX}           %text and control sequences with sort keys
*endverbatim

\thissubsubhead{\runintrue}
\bluesubsubhead Spaces \par are  difficult as always. ^^{IR\ and\ spaces} 
In the IR they separate parts of the IR and are used
in the word part. 

\bitem Just typing a space has as an effect that
       it will be neglected during  sorting   
\bitem The markup `\cs{\char32}', a control space,
       will yield a space subject to sorting, according to
       the ordering table
\bitem \cs{space} as markup will be neglected during sorting. 
       This token is default member of
       the set of  control sequences to be ignored.
       It will be set in the index as \cs{\char32}.
\smallbreak

\exercise What to do when part of a title should reappear in the index?
\answer The naive approach is to enclose that part by braces and precede
        it by a circumflex. However, that goes wrong because a title
        is stored and reused in many places. So copy the words and
        mark them as a silent IR.

\blueexample Spaces
 
\thisverbatim{\catcode`\|=12 }
\begindemo
^{\space}%an ignored cs
^{a\ a}  %control space
^{aa} 
^{a\ b}
^{a \TeX}
^{a\ \bf a}
^{\TeX book}
^{xyz beta}%space neglected in 
           %sorting
^{xyza}
^|\space|
\sortindex\pasteupindex\bye
!yields
\noindent  Sorted result in file index.srt
\thisverbatim{\catcode`\!=12
              \catcode`\;=0 }
\beginverbatim
\space {} !0 1.
a\ \bf a{} !0 1.
a\ a{} !0 1.
a\ b{} !0 1.
aa{} !0 1.
a \TeX {} !0 1.
space{} !2 1.
\TeX book{} !0 1.
xyza{} !0 1.
xyz beta{} !0 1.
;endverbatim
\enddemo
Explanation. \cs{space} belongs to the set of control sequences
to be ignored, ^^{ignored\ control\ sequences}
ICSs for short. 
This means that it is skipped with respect to sorting,
except when it occurs as the last token of the word part. 
In that case they are ordered as a space, 
i.e., according to the lowest value. 
This explains the position of `\cs{space}.'  

`\cs{TeX},' and `\cs{TeX} book,' are subject to the default sorting keys.
 
`xyza' precedes `xyz beta,' because the space is silent. 
When word ordering is preferred a \cs{\char32}, a control space,  must
be included.

\bluehead Special tokens

Tokens are either neglected 
or replaced by another sequence while sorting.
\bluetex{} provides two sets of tokens to be ignored while sorting: 
^|\conseqs| and ^|\consyms|.\ftn{There are two sets because of
   the handling of the space after the token in the result.}
Replacing a control sequence  by another sequence
is called associating a sorting key to the
control sequence.

Active symbols can't be part of the IR, for the moment.

\bluesubhead Tokens to be ignored 

In practice I needed things like \cs{tt}
as part of the IR, which must be neglected while sorting.\ftn{The reason is
  that {\tt <, and >} are used, and printed wrongly.}
I decided to ignore those tokens while sorting and to include
the tokens in the final index.elm as such.
Default \bluetex{} knows about the following sets of tokens to be ignored.
\begincenterverbatim
\conseqs{\c\space\bf\it\rm\tt\sub\relax}
\consyms{\`\'\"\^\~}
!endcenterverbatim

\bluesubhead Sorting keys

In order to  extend a set, use the macro \cs{add}.

\blueexample Use of sorting keys

Default \bluetex{} provides the  following
sorting keys. 
\begincenterverbatim
\srtkeypairs{\AmSTeX{amstex}
             \LAMSTeX{lamstex}
             \LaTeX{latex}
             \TeX{tex}
             \PS{PostScript}}
!endcenterverbatim
Suppose that we have \cs{fourtex} and that we like this to be sorted
as `4tex.' 
This can be done by extending the set of ^|\srtkeypairs|, ^^|\add|
as follows.
\thisverbatim{\unmc}
\begindemo
\add\fourtex{4tex}to\srtkeypairs
Copy with ^{IR \fourtex}
^{IR 1}
^{IR 5}
^{IR a}
%
\sortindex   %with 4tex for \fourtex 
\pasteupindex%Set `IR \fourtex{} 
             %<pagenumbers>'
\bye
!yields
then the file index.srt will contain the IRs
\thisverbatim{\catcode`\!=12 
              \catcode`\;=0 }
\beginverbatim
IR 1 !0 <pageno>.
IR \fourtex{} !0 <pageno>.
IR 5 !0 <pageno>.
IR a !0 <pageno>.
;endverbatim
\enddemo
with \cs{fourtex}  sorted on 4tex.

\exercise What to do when `to' is part of the sorting key?
\answer Add an extra level of braces.


\bluehead Ordering

A fundamental issue with indexes is the ordering. ^^{ordering}
The ^{ASCII} table is not suited because lowercase and uppercase
letters differ by 32. I decided to rank these as equal, more
precisely to assign the lowercase ASCII values to both. I prefer
from the accompanying table the {\oldstyle1}$^{st}$ column to the 
{\oldstyle2}$^{nd}$ one.

Moreover, accented letters are not part of ASCII. How should we
order for example e, \'e, \`e, \^e, \"e?
I decided to rank accented letters equal to those without an accent,
because I prefer from the accompanying table
the {\oldstyle3}$^{rd}$ column to the {\oldstyle4}$^{th}$ one.

I know that non-letters precede letters, but what about their relative
ordering? I decided to stay as close as possible to the ASCII ordering.

Then there is the problem of digits. In IRs they come as part of the 
word(s) and as page numbers. For the latter I used the numerical ordering.
For the former I used the alphabetical ordering.\ftn{I could have applied
  a look ahead mechanism and use numerical ordering throughout.
  Maybe another time.}

Furthermore, a user can select
the so-called ^{word\ ordering},\ftn{This means that a space precedes
    all letters. A space as such is neglected in the ordering.}
by \cs{\char32}, \TeX nically a control space, as markup for a space.
Personally, I like from the accompanying table 
the {\oldstyle5}$^{th}$ column  better than the {\oldstyle6}$^{th}$.
\def\btablecaption{}
\def\footer{}
\nonframed
\def\rowstblst{}
$$\def\header{lower vs.\ upper case\cs 
              accents vs.\ unaccented\cs 
              word ordering}
\vruled\btable{\vtop{\halign{&\tstrut\quad#\hfil\cr
el      &  el               \cr       
El\`eve &  em               \cr 
em      &  El\`eve \cr}}\cs
\vtop{\halign{&\tstrut\quad#\hfil\cr
el                 & el                \cr        
\hbox{\'el\`eve}   & em                \cr        
em                 & \hbox{\'el\`eve}  \cr}}
\cs\vtop{\halign{&\tstrut\quad#\hfil\cr
sea lion & seal    \cr
seal     & sea lion\cr}}
}%end \btable
$$

\bluehead Typesetting the index

The specifications for typesetting a \bluetex{} index are
\bitem represent the four IR types the same as in the \TeX book
\bitem set in two-columns, balanced, possibly preceded
       by one-column copy
\bitem set subsidiary entries analogous to the \TeX book
\bitem indent continuation lines by {\oldstyle2}em
\bitem indent subsidiary entries by {\oldstyle1}em
%\bitem underline page numbers which represent the definition or
%       the main source of information
%\bitem represent a page number in italics when that page contains
%       an instructive example of the concept in question.
\smallbreak
Users can  edit index.elm\Dash read: add markup\Dash and 
provide the necessary macros in for example \cs{preindex}. 
In short follow  Knuth. To please Frans Goddijn I introduced the tag
^|\numberstyle|, by default  equal to \cs{oldstyle}.

\exercise And what about subentries?
\answer  My approach is to consider subentries as a typesetting problem in the sense
   that the full entries are specified and only when sorted and typeset redundant first
   parts can be suppressed, if one considers this better. This is similar to how
   BLUe's format system typesets references, inspired by the \AMS.
   For the moment I did not implement subentries handling, because as of {\oldstyle1995}
   I consider it of low priority.

\bluehead Customization

A user might wish to interfere in places
\bitem to include other tokens to be ignored while sorting
\bitem to supply an ordering of his/her own
\bitem to enrich the sorted and compressed file index.elm.
\smallbreak

\bluesubhead Adding tokens

What are reasonable requirements to impose upon the handling of
markup control sequences (cs for short)? In my opinion 
\bitem the cs must be defined
\bitem ^|\makexref| writes the cs unexpanded
\bitem ordering? unknown, and therefore must be supplied
\bitem ^|\setupnxtokens| guards that the cs-s are written %, unexpanded,
       to ^{index.srt} and ^{index.elm}.
\smallbreak
As a consequence I decided to neglect the `in between'
control sequences while sorting.
For those who favour a one-pass job, I have provided the following,
though.\ftn{It is simpler to add those control sequences to 
       index.elm.} 
 
The extension of a set of tokens can be done via ^^|\add|
\beginverbatim
\add\hfil to\conseqs  or  \add\`to\consyms  or  \add\hfil{hfil}to\srtkeypairs
%with auxiliary \def\add#1to#2{...}
!endverbatim
Each element  from \cs{conseqs} is redefined in such a way 
that the control sequence token is written to the file  
with a space appended.\ftn{\cs{noexpand} is used instead of \cs{string}.} 

\bluesubhead Modifying ordering

A general way is to `copy' the ordering table 
and to modify it.\ftn{My \cs{fifo} is just a shortcut, which also
   prevents typos in assigning the ASCII values. For \cs{fifo}, see
   my `FIFO and LIFO sing the BLUes.'}

And what about a macro to add to the table?
This can be done easily, and superficially looks convenient 
for an innocent user.
At the moment I don't trust the macros to be worthwhile
for an innocent user, unless a very modest index has to be made.
And this completes the circle: different ordering is not wanted, I guess.

\bluesubhead The process and files involved

Like in manmac, \bluetex{} stores the raw IRs in the file index. 
^^{IR,\ processes\ and\ files}
The file index^^{index,\ file}\ftn{Default index is the value of
  the toks variable ^|\irfile|, which is used in \cs{sortindex}.}
is read and stored in an array for internal sorting.
After sorting, the number of  entries is 
reduced,\ftn{Those which differ by page
   number are collected in one entry.} 
and the result is written to the file ^{index.srt}.
Then, index.srt is transformed into the file
index.elm.\ftn{Default ^{index.elm} is the value of the toks variable
      ^|\indexfile|, which is used in |\pasteupindex|.
      The transformation abandons the IR syntax.
      The part which specifies the kind of IR
      is deleted and the word part marked up
      accordingly.}
The result is typeset via ^|\pasteupindex|.
Schematically it comes down to the following.
\bigskip
$$\vbox{\hsize.5\hsize%
\indmodelpic
}$$
^|\loadindexmacros| loads the index and sorting macros, and performs
initializations. It is safeguarded against double loading.\ftn{I introduced 
    this because I  start each chapter with \cs{loadindexmacros},
    independent from whether it is run on its own or as part of the total.}


\bluesubhead Enriching the index

This use is necessary when for example ^^{index,\ enrich}
\bitem control sequences have to be typeset
\bitem special symbols are needed, or
\bitem cross-references within the index are required.
\smallbreak
The best way  is to start from the ^{index.elm} file.

\bluesubhead Typesetting the enriched file

When the default name is used\Dash index.elm\Dash
just say \cs{pasteupindex}.
For another file name  assign this name to the toks 
variable \cs{indexfile},
prior to the invocation of \cs{pasteupindex}. 

\bluehead Extras

Ubdoubtedly people favour their own subset of \TeX, or more likely \LaTeX.
There is good news. You don't have to use BLUe's format system. I gathered the sorting
and indexing stuff as an independent self-contained set in the file plainindex.tpl.

The bad news is that up till now I did not do much about preventing name clashes.

\bluesubhead \TeX nical details 

The details with respect to indexing have been treated in `BLUe's Indexes,' and 
the sorting aspects have been treated in `Sorting in BLUe,' both  available from the CTAN. 
\endinput
\bye