summaryrefslogtreecommitdiff
path: root/Master/texmf-dist/doc/latex/nicetext/docsrc/fifinddo.tex
blob: 59484968382fd3309e50e9b65ee14400bcb55f0b (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
\ProvidesFile{fifinddo.tex}[2010/03/23 documentation for `fifinddo']

\title{\textsf{fifinddo}\\---\\Filtering \TeX(t) Files by \TeX}

\author{Uwe L\"uck\,---\,{\tt http://contact-ednotes.sty.de.vu}}

% \listfiles

\documentclass{article}

\usepackage{makedoc}

% \usepackage{substr} %% for failure examples TODO 2010/03/17

\input{makedoc.cfg}

\begin{document}

\maketitle



\begin{flushright}\small\it FIDO, FIND!\\or:\\FIND FIDO!\\oder:\\FIFI, 

                            SUCH!\end{flushright} 

\begin{abstract}\noindent\hspace{1sp}%% for niceverb-apostrophe

'fifinddo' starts implementing parsing of plain text or \TeX\ files 

using \TeX, generalizing the philosophy behind 'docstrip', 

based on how \TeX\ reads macro arguments. 

Rather than typsetting the edited input stream immediately, 

results are written to another file, 

in the first instance as input for \TeX.

Rather than presenting a ``complete study" of a computer-scientific 

idea, it aims at practical applications. 

The main one at present is '\mbox{makedoc}' which removes certain comment 

marks from package files and inserts listing commands. 

Parsing macros are not defined anew at every input chunk, 

but once before a file is processed. 

This also allows for \emph{expandable} sequences of replacements, 

e.g., with `txt'$\to$\TeX\ functionality. 

The method of testing for substrings is carefully discussed,

revealing an earlier mistake (then) shared with 'substr.sty' 

and \LaTeX's internal `\in@'. 

\end{abstract}

\tableofcontents



\section{Introduction: The Gnome of the Aim} 

\subsection{Parsing by \TeX---are you mad?}

The package name 'fifinddo' is a &\listfiles-compatible abbreviation 

of \lq file-find-do\rq\footnote{\lq file\rq\ possibly for ``searching 

  \TeX(t) files" (I don't remember my thoughts!), 

  while there were requests for doing replacements on 

  \LaTeX\ \emph{environments} on 'texhax'. However, the package might 

  be enhanced in this direction $\dots$ so the name may be wrong 

  $\dots$ but now I like it so much .\,.\,. Or the reason was that 

  results are written to a \emph{separate file}, not typeset 

  immediately.---Let me also mention that \emph{\lq Fifi\rq} (as the 

  package name starts) is a kind of German equivalent to the 

  ``English" \emph{\lq Fido\rq}, or may have been.}

(or think of \lq if found do\rq).

'fifinddo' implements (or aims at) general parsing 

(extracting, replacing [converting], expanding, \dots) 

using \TeX\ where 'texhax' posters strongly urge to use 'sed', 

'awk', or 'Perl'.

'fifinddo''s opposed rationales are: 

\begin{itemize}

\item It works instantly on any \TeX\ installation. 

      (\emph{Restrictions:} Some \TeX\ versions &\write certain hex 

      codes for certain characters, cf.\ \TeX book p.~45, 

      I have seen this with PC\TeX. However, some applications of 

      'fifinddo' are nothing but technical steps where you will read 

      the result files rarely anyway. 

      %% And I have not yet explored extended encodings.)

      %% <- done meanwhile 2010/03/22

\item You can apply and customize it like any \TeX\ macros, knowing 

      just \TeX\ (or even only the documentation of some user-friendly 

      extension of 'fifinddo'), without the need of learning any 

      additional script language. 

\item The syntax of usual utilities (e.g., ``wildcards") 

      is sometimes difficult with \emph{\TeX} files 

      with all their backslashes, square brackets, stars, 

      question marks \dots

\end{itemize}

%

At least the first item is just the philosophy of the 'docstrip' program, 

standard for installing \TeX\ packages; 

and while I am typing this, I find at least 14 other similar packages 

in J\"urgen Fenn's \emph{Topic Index} of the \emph{\TeX\ Catalogue:}

% \begin{quote}\small

\[\texttt{%

%   \hbox\bgroup\url{%                %% \url impossible!? 2010/03/19

    http://mirror.ctan.org/help/Catalogue/bytopic.html\string#parsingfiles}% 

      \footnotemark

%        \egroup\] 

\]

\footnotetext{%

  \href{http://mirror.ctan.org/help/Catalogue/bytopic.html\string#parsingfiles}%

       {Click here!}}

% \end{quote}

(Some of them may have been \emph{reactance} to 'texhax' and other 

postings urging not to try something like this; some seem just to be 

celebrations of the power of \TeX---yes, celebrate!)



Actually, \TeX's mechanism of collecting macro arguments is hard-wired 

parsing at quite a high level. \LaTeX\ hides this from 

``simple-minded" users by a convention \emph{not} to use that full 

power of \TeX\ for \emph{end-user macros}. 

\emph{Internally}, \LaTeX\ \emph{does} use it in reading 

lists of options and file dates as well as to implement certain 

`FOR'- and `WHILE'-like loop programming structures. 

\LaTeX's `\in@'/`\ifin@' construction is an implementation of a 

``<string1> occurs in <string2>" test. More packages seem to use 

this idea for extracting file informations, like 

'texshade'.\footnote{\url{http://ctan.org/pkg/texshade}} 





However, such packages don't make much ado about parsing, 

there seems to be no general setup mechanism as are presented by 

'fifinddo'. Indeed, tayloring parsing macros to specific applications 

may often be more efficient than a general approach. 



\subsection{Useful for \dots}

My main application of 'fifinddo' at present is typesetting 

documentations of packages using 'makedoc' which removes certain 

percent marks and inserts listing commands, so you edit a package file 

with as little documentation markup as possible. 

This may be extended to other kinds of documents as an alternative 

to 'easylatex' or 'wiki' (the approach of which is dangerous and 

incompatible with certain other things).



I have used a similar own package 'txtproc' successfully, 

where more features were implemented for practical purposes 

than are here so far, yet I don't like its implementation, want to 

improve it here. This package also \emph{created batch files}, e.g., 

to remove temporary files.

This could be used for package handling: 

typset the documentation at the desired place in the tree, 

write the packages to another, write a batch file to remove 

files that are not needed any more after installation

(cf.\ 'make'). %%% !? cf. Wiki `Make (software) 2009/04/11



I used 'txtproc' also for \emph{large-scale substitutions} 

(it had been decided to change the orthography in a part of a book).

Other large-scale substitutions may be: 

\begin{itemize}

\item inserting &\index commands;

\item inserting (soft) hyphenation commands near accents; 

\item manual umlaut-conversion.\footnote{If you know the ``names" 

        of the encodings, Heiko Oberdiek's 

        \ctanpkgref{stringenc}

        may be preferable.}

\item typographical (or even orthographical) corrections (same mistake 

      many times on each of hundreds of pages). 

      You may turn `...' into `$\dots$' and `etc.' into `etc.\' 

      etc.\footnote{But what when a new sentence is starting indeed? 

        Well, `cf.' is an easier example.---`etc.' even showed 

        a problem in 'niceverb'. 'mdoccorr.cfg' replaces `etc. ' 

        only, so you can keep the extra space by a code line break.}

      This could replace packages like 

      'easylatex',\footnote{\url{http://ctan.org/pkg/easylatex}} 

      'txt2latex',\footnote{\url{http://ctan.org/pkg/txt2latex}} 

      'txt2tex'\footnote{\url{http://ctan.org/pkg/txt2tex}} 

      in a customizable way, using, e.g., the ``correct"

      hook from 'makedoc.sty' as exemplified in 'mdoccorr.cfg' 

      (see examples section of 'makedoc.pdf'). 

      You should find 'fdtxttex.tpl', a 'fifinddo' script 

      to try or apply `\MakeDocCorrectHook' from 'mdoccorr.cfg', 

      as well as 'fdtxttex.tex' that runs a dialogue for the 

      same purpose if you can manage to run it ('WinShell'?).

      You can then try to create your own `\MakeDocCorrectHook'. 

      Section~\ref{sec:replchain} provides setup for macros 

      of this kind. 

\item as to 'easylatex' again, \emph{lists} could be detected and 

      transformed into \LaTeX\ list commands. This could re-implement 

      the lists functionality of 'wiki.sty' that is somewhat 

      dangerous. %% TODO 2009/04/15

\item introduce your own \emph{shorthands} to be expanded not as \TeX\ macros, 

      but by text substitution;

\end{itemize}

%% % <- TODO present macros/"jobs" doing this. 

%%           %% <- TOOD single quote tests could be used for 

%%                      double quote character as well 

%%                      to turn it into left quotes if ... 2010/03/19

%

In certain cases, insertions deteriorate readability, hyphenation 

corrections even make text search difficult. 

It is therefore suggested to 

\begin{enumerate}

\item keep editing the file without the insertions, 

\item run the script (commands based on 'fifinddo') for insertions 

      in the preamble of the main file 

      (```\jobname.tex'", maybe &\input the script file) and 

\item &\input the result file within the `document' environment. 

\end{enumerate}



In general, differences to ``manual" replacing by the substitution 

function of your \emph{text editor} is that 

\begin{itemize}

\item you first keep the original version, 

\item you can check the resulting file before you replace the original 

      file by it, 

\item you can store the replacement script in order to check for 

      mistakes at a later stage of your work, 

\item you can do \emph{all} the replacements in \emph{one run} 

      (by \emph{one} script to check for mistakes), 

\item you can store replacement scripts for future applications, 

      so you needn't type the patterns and replacement strings anew. 

\end{itemize}





\subsubsection{Comparisons} %% was `Missing' 2010/03/18

% \subsection{To implement or not to implement \dots} %% TODO!? 2009/04/13

% Actually, 'fifinddo' doesn't implement very much, and this possibly 

% won't change, even if I am lucky. ``Practical applications" tend to 

% show (in my view) that there is a very complex space of possibilities 

% what parsing macros could do. If I try to ``attack" 'Perl' or so, 

% cf.\ 'CPAN'\@. If I am lucky and become famous, 'fifinddo' will get many 

% extensions in separate packages, cf.\ 'pstricks'. I also like it when 

% you can choose among packages for similar purposes, and this may apply 

% to extensions of 'fifinddo' as well. 

% 

% What \emph{really} is missing very badly at present is matching 

% \emph{sequences} of strings. However, 'makedoc' contains a practical 

% application of this for the case that both strings appear \emph{in the same 

% line}. A better algorithm would have to read more lines from the input 

% file etc. 

% 

% As it just comes to my frightened mind, there is even a problem with 

% ``strings"/``patterns" that are \emph{phrases} of words, containing 

% spaces \dots

% 

% Clearly much comfort concerning \emph{routine} jobs is missing now, 

% 'txtproc' has more. Please sponsor (or otherwise support) the work for 

% improving this. 

% 

It should be noted (perhaps here) that the present approach to parsing 

is a quite \emph{simple} one and in this respect much different to the 

string handling mechanisms of 

'stringstrings',\footnote{\url{http://ctan.org/pkg/stringstrings}} 

'ted',\footnote{\url{http://ctan.org/pkg/ted}} 

'xstrings'\footnote{\url{http://ctan.org/pkg/xstrings}} 

(as I understand them, perhaps also 

'coolstr'\footnote{\url{http://ctan.org/pkg/coolstr}}) 

which are \emph{much more powerful} than what is offered here---but 

perhaps slow and for practical applications possibly replaceable by 

the present approach.

\emph{Expandable replacement} seems not to exist outside 'fifinddo' 

(2009/04/13).



Much is missing, I know.\footnote{There 

  is more in my badly implemented 'txtproc.sty'.}

I am just implementing what I actually need 

and what could show that this approach is worth being pursued. 

  %% removed 2010/03/18:

% It may need being sponsored or otherwise supported. 



\subsection{For insiders}



\emph{Warning:} You may (at least at the present state of the work) 

have little success with this package, if you don't know about 

\TeX's category codes and how \TeX\ macros are defined. 

The package rather provides tools for package writers. 

You may, however, be able to run other packages which just load 

'fifinddo' as required background.



That 'fifinddo' acts on ``\TeX(t)" files or so means that 

(at present) I think of applications on ``plain text" files which 

will usually be \TeX\ input files. ``At present" they are read 

without ``special characters," so essentially category codes of input 

characters are either 11 (``letter") or 12 (``other"). 

This way some things are easier than with usual \TeX\ applications: 

\begin{enumerate}

\item You can ``look into" curly braces and ``behind" comment characters. 

\item There are exact or safe tests especially of \emph{empty macro arguments} 

      that are ``expandable," i.e., they are ``robust," 

      don't need assignments, can be executed in `\write'ing and in 

      `\edef' definitions. ``Usually," the safe way to test emptiness 

      is storing a macro argument as a macro, say `\tempo', in order 

      to test `\ifx\tempo\empty' where `\empty' has been defined 

      by `\def\empty{}' in the format. But this requires some 

      `\def\tempo{#<n>}' which breaks in ``mere expanding" 

      (\TeX\ \emph{evaluates} `\tempo' instead of defining it).

      An \emph{expandable} test on emptiness is, e.g. `\ifx$#<n>$', 

      where we hope that it becomes `\iftrue' just if macro argument 

      `#<n>' is empty indeed. However, ``usually" it may \emph{also} 

      become `\iftrue' when `#<n>' starts with `$'---if the latter 

      has category code 3 (``math shift"). But 'fifinddo' does 

      not assign category code 3 to any character from the input file! 

      Therefore `\ifx$#<n>$' is `\iftrue' \emph{exactly} if `#<n>' is 

      empty. 

\item You can avoid interference with packages that are needed for 

      typesetting. You can do the ``preprocessing" in one run with 

      typesetting, but you should do the preprocessing before you 

      load packages needed for typesetting. One may even try to keep 

      the macros and settings for preprocessing local to a group. 

\end{enumerate}

%

Once there may be an option to read input with some usual \TeX\ 

category codes as well, it may be useful to (some of)

\begin{itemize}

\item avoid matching substrings of control words, 

\item skip blank spaces as \TeX\ does it usually,

\item catch \emph{balanced} input pieces,

\item ignore comments, 

\item ignore certain characters. 

\end{itemize}



The essential approach of 'fifinddo' to looking for single strings is 

described in some detail in section~\ref{sec:theory}.



The implementation of 'fifinddo' is as follows.

User commands are specially highlighted (boxed\slash coloured), 

together with their syntax description. 



%   \end{document}



\section{Preliminaries}

\subsection{Head of file (Legalese)}

\sloppy 

\ProcessLineMessage{}

\MakeInputJobDoc{23}{\SectionLevelTwoParseInput}

\end{document}



TODO cf. Kastrup 'makematch' and others 2009/04/09



2009/04/10: title break, by -> with(!?); `boxed/coloured'

2009/04/12: useful: own shorthands

2009/04/13: substr.sty, \emph{expandable}, w/o `implement'!?

2009/04/15: same with \in@

2010/03/08: EPS application removed

2010/03/17: use \MakeInputJobDoc

2010/03/18: `Missing' -> `Comparisons' etc.; \Require... 

2010/03/19: Catalogue \url; '' -> "

2010/03/20:  \ctanpkgref

2010/03/22: debugging; done ...

2010/03/23: more URLs; abstract "then shared"; 

            removed dates from 2009, fdtxttex.tex/tpl