attempt to document the lack of disk searching when ls-R is present more clearly

git-svn-id: svn://tug.org/texlive/trunk@50595 c570f23f-e606-0410-a88d-b1316a301751

5 files changed, 162 insertions, 114 deletions

diff --git a/Build/source/texk/kpathsea/ChangeLog b/Build/source/texk/kpathsea/ChangeLog
index 5162fda2e3a..2c04367bddd 100644
--- a/Build/source/texk/kpathsea/ChangeLog
+++ b/Build/source/texk/kpathsea/ChangeLog
@@ -1,3 +1,11 @@
+2019-03-25  Karl Berry  <karl@freefriends.org>
+
+	* texmf.cnf (TEXMFDBS),
+	* doc/kpathsea.texi (ls-R): attempt to describe the lack of
+	disk searching when ls-R is present, regardless of !!, more clearly.
+	Report from Yann Salmon, tex-k thread of 27 Feb 2019 21:00:46.
+	* kpsewhich.c: doc fix.
+
 2019-02-11  Karl Berry  <karl@tug.org>
 
 	* version.ac,
diff --git a/Build/source/texk/kpathsea/doc/kpathsea.info b/Build/source/texk/kpathsea/doc/kpathsea.info
index 401a14dd46c..d5d49858ae5 100644
--- a/Build/source/texk/kpathsea/doc/kpathsea.info
+++ b/Build/source/texk/kpathsea/doc/kpathsea.info
@@ -1,4 +1,4 @@
-This is kpathsea.info, produced by makeinfo version 6.5 from
+This is kpathsea.info, produced by makeinfo version 6.6 from
 kpathsea.texi.
 
 This file documents the Kpathsea library for path searching.
@@ -37,7 +37,7 @@ Kpathsea library
 ****************
 
 This manual documents the Kpathsea library for path searching.  It
-corresponds to version 6.3.1, released in February 2019.
+corresponds to version 6.3.1, released in March 2019.
 
 * Menu:
 
@@ -62,7 +62,7 @@ File: kpathsea.info,  Node: Introduction,  Next: unixtex.ftp,  Prev: Top,  Up: T
 **************
 
 This manual corresponds to version 6.3.1 of the Kpathsea library,
-released in February 2019.
+released in March 2019.
 
    The library's fundamental purpose is to return a filename from a list
 of directories specified by the user, similar to what shells do when
@@ -989,9 +989,13 @@ File: kpathsea.info,  Node: ls-R,  Next: Filename aliases,  Up: Filename databas
 
 As mentioned above, you must name the main filename database 'ls-R'.
 You can put one at the root of each TeX installation hierarchy you wish
-to search ('$TEXMF' by default); most sites have only one hierarchy.
-Kpathsea looks for 'ls-R' files along the 'TEXMFDBS' path, so that
-should presumably match the list of hierarchies.
+to search ('$TEXMF' by default, which expands to a braced list of
+several hierarchies in TeX Live).
+
+   Kpathsea looks for 'ls-R' files along the 'TEXMFDBS' path.  It is
+best for this to contain all and only those hierarchies from '$TEXMF'
+which are specified with '!!'--and also to specify them with '!!' in
+'TEXMFDBS'.  (See the end of this section for more on '!!'.)
 
    The recommended way to create and maintain 'ls-R' is to run the
 'mktexlsr' script, which is installed in '$(bindir)' ('/usr/local/bin'
@@ -1012,7 +1016,9 @@ definition of the file format, see *note Database format::.
    Regardless of whether you use the supplied script or your own, you
 will almost certainly want to invoke it via 'cron', so when you make
 changes in the installed files (say if you install a new LaTeX package),
-'ls-R' will be automatically updated.
+'ls-R' will be automatically updated.  However, for those using TeX Live
+or system distributions, the package managers should run 'mktexlsr' as
+needed.
 
    The '-A' option to 'ls' includes files beginning with '.' (except for
 '.' and '..'), such as the file '.tex' included with the LaTeX tools
@@ -1032,14 +1038,24 @@ instead of 'ls -LR ./' or 'ls -R /YOUR/TEXMF/ROOT'.  Another possibility
 is some system directory name starting with a '.' (perhaps if you are
 using AFS); Kpathsea ignores everything under such directories.
 
-   Because the database may be out-of-date for a particular run, if a
-file is not found in the database, by default Kpathsea goes ahead and
-searches the disk.  If a particular path element begins with '!!',
-however, _only_ the database will be searched for that element, never
-the disk.  If the database does not exist, nothing will be searched.
-Because this can surprise users ("I see the font 'foo.tfm' when I do an
-'ls'; why can't Dvips find it?"), it is not in any of the default search
-paths.
+   If a particular path element begins with '!!', _only_ the database
+will be searched for that element, never the disk; and if the database
+does not exist, nothing at all will be searched.  In TeX Live, most of
+the trees are specified with '!!'.
+
+   For path elements that do not begin with '!!', if the database
+exists, it will be used, and the disk will not be searched.  However, in
+this case, if the database does not exist, the disk will be searched.
+In TeX Live, the most notable case of this is the 'TEXMFHOME' tree, to
+allow users to add and remove files from their own tree without having
+to worry about 'ls-R'.
+
+   (Aside: there are uncommon cases where a '!!' tree will be searched
+on disk even if the 'ls-R' file exists; they are too obscure to try to
+explain here.  See 'pathsearch.c' in the source if you need to know.)
+
+   To sum up: do not create an 'ls-R' file unless you also take care to
+keep it up to date.  Otherwise newly-installed files will not be found.
 
 
 File: kpathsea.info,  Node: Filename aliases,  Next: Database format,  Prev: ls-R,  Up: Filename database
@@ -2963,13 +2979,16 @@ Index
 
 * Menu:
 
-* !! in path specifications:             ls-R.                (line  51)
+* !! and casefolding:                    Casefolding examples.
+                                                              (line  57)
+* !! in path specifications:             ls-R.                (line  57)
+* !! in TEXMFDBS:                        ls-R.                (line  11)
 * $ expansion:                           Variable expansion.  (line   6)
 * --all:                                 Path searching options.
                                                               (line  12)
 * --casefold-search:                     Path searching options.
                                                               (line  19)
-* --color=tty:                           ls-R.                (line  21)
+* --color=tty:                           ls-R.                (line  25)
 * --debug=NUM:                           Auxiliary tasks.     (line   9)
 * --dpi=NUM:                             Path searching options.
                                                               (line  28)
@@ -3014,14 +3033,14 @@ Index
 * --without-mktexpk-default:             mktex configuration. (line  12)
 * --without-mktextfm-default:            mktex configuration. (line  12)
 * -1 debugging value:                    Debugging.           (line  23)
-* -A option to ls:                       ls-R.                (line  33)
+* -A option to ls:                       ls-R.                (line  39)
 * -D NUM:                                Path searching options.
                                                               (line  28)
 * -iname, find predicate:                Casefolding examples.
                                                               (line  78)
-* -L option to ls:                       ls-R.                (line  38)
-* . directories, ignored:                ls-R.                (line  33)
-* . files:                               ls-R.                (line  33)
+* -L option to ls:                       ls-R.                (line  44)
+* . directories, ignored:                ls-R.                (line  39)
+* . files:                               ls-R.                (line  39)
 * .2602gf:                               Unable to generate fonts.
                                                               (line  36)
 * .afm:                                  Supported file formats.
@@ -3105,7 +3124,7 @@ Index
                                                               (line  42)
 * .tex:                                  Supported file formats.
                                                               (line 180)
-* .tex file, included in ls-R:           ls-R.                (line  33)
+* .tex file, included in ls-R:           ls-R.                (line  39)
 * .tfm:                                  Supported file formats.
                                                               (line 199)
 * .ttc:                                  Supported file formats.
@@ -3126,7 +3145,7 @@ Index
                                                               (line   6)
 * /etc/profile:                          Unable to find files.
                                                               (line  14)
-* /etc/profile and aliases:              ls-R.                (line  21)
+* /etc/profile and aliases:              ls-R.                (line  25)
 * /var/tmp/texfonts:                     mktex configuration. (line 113)
 * 2602gf:                                Unable to generate fonts.
                                                               (line  36)
@@ -3164,7 +3183,7 @@ Index
                                                               (line   6)
 * argv[0]:                               Calling sequence.    (line  14)
 * autoconf, recommended:                 Calling sequence.    (line 117)
-* automounter, and ls-R:                 ls-R.                (line  40)
+* automounter, and ls-R:                 ls-R.                (line  46)
 * auxiliary tasks:                       Auxiliary tasks.     (line   6)
 * Bach, Johann Sebastian:                Default expansion.   (line  41)
 * backslash-newline:                     Config files.        (line  37)
@@ -3266,13 +3285,13 @@ Index
                                                               (line   6)
 * disabling mktex scripts:               mktex configuration. (line   6)
 * disk search:                           Searching overview.  (line  22)
-* disk searching, avoiding:              ls-R.                (line  51)
+* disk searching, avoiding:              ls-R.                (line  57)
 * disk usage, reducing:                  Logging.             (line   6)
 * doc files:                             Supported file formats.
                                                               (line 185)
 * DOS compatible names:                  mktex configuration. (line  68)
 * dosnames:                              mktex configuration. (line  67)
-* dot files:                             ls-R.                (line  33)
+* dot files:                             ls-R.                (line  39)
 * doubled colons:                        Default expansion.   (line   6)
 * dpiNNN directories:                    mktex configuration. (line  68)
 * DVILJMAKEPK:                           mktex script names.  (line  32)
@@ -3471,7 +3490,7 @@ Index
 * ls-R:                                  Supported file formats.
                                                               (line  91)
 * ls-R database file:                    ls-R.                (line   6)
-* ls-R, simplest build:                  ls-R.                (line  18)
+* ls-R, simplest build:                  ls-R.                (line  22)
 * Mac filesystem, case-insensitive:      Casefolding rationale.
                                                               (line   6)
 * MacKenzie, David:                      History.             (line  44)
@@ -3544,7 +3563,7 @@ Index
 * must exist:                            Searching overview.  (line  31)
 * names for mktex scripts:               mktex script names.  (line   6)
 * Neumann, Gustaf:                       History.             (line  56)
-* NFS and ls-R:                          ls-R.                (line  40)
+* NFS and ls-R:                          ls-R.                (line  46)
 * nomfdrivers:                           mktex configuration. (line  92)
 * nomode:                                mktex configuration. (line  97)
 * none:                                  Suppressing warnings.
@@ -3692,7 +3711,7 @@ Index
                                                               (line   6)
 * symbolic links not found:              Unable to find files.
                                                               (line  21)
-* symbolic links, and ls-R:              ls-R.                (line  38)
+* symbolic links, and ls-R:              ls-R.                (line  44)
 * symlinks, resolving:                   Calling sequence.    (line  31)
 * system C compiler bugs:                TeX or Metafont failing.
                                                               (line  16)
@@ -3765,7 +3784,7 @@ Index
 * TEXMFCNF:                              Config files.        (line   6)
 * TEXMFCNF <1>:                          Supported file formats.
                                                               (line  49)
-* TEXMFDBS:                              ls-R.                (line   6)
+* TEXMFDBS:                              ls-R.                (line  11)
 * TEXMFDBS <1>:                          Supported file formats.
                                                               (line  91)
 * TEXMFINI:                              Supported file formats.
@@ -3827,7 +3846,7 @@ Index
 * unreadable file warnings:              Suppressing warnings.
                                                               (line  27)
 * unreadable files:                      Searching overview.  (line  63)
-* unusable ls-R warning:                 ls-R.                (line  45)
+* unusable ls-R warning:                 ls-R.                (line  51)
 * usage patterns, finding:               Logging.             (line   6)
 * USERPROFILE, as ~ expansion:           Tilde expansion.     (line   6)
 * USE_TEXMFVAR:                          mktex configuration. (line 128)
@@ -3845,7 +3864,7 @@ Index
                                                               (line 219)
 * Vojta, Paul:                           History.             (line  30)
 * Walsh, Norman:                         History.             (line  56)
-* warning about unusable ls-R:           ls-R.                (line  45)
+* warning about unusable ls-R:           ls-R.                (line  51)
 * warning, about missing texmf.cnf:      Config files.        (line  18)
 * warnings, file access:                 Searching overview.  (line  63)
 * warnings, suppressing:                 Suppressing warnings.
@@ -3873,61 +3892,61 @@ Index
 
 Tag Table:
 Node: Top1480
-Node: Introduction2263
-Node: History4336
-Node: unixtex.ftp8932
-Node: Security10357
-Node: TeX directory structure12861
-Node: Path searching16909
-Node: Searching overview17636
-Node: Path sources21455
-Node: Config files22546
-Node: Path expansion26473
-Node: Default expansion27426
-Node: Variable expansion29496
-Node: Tilde expansion30897
-Node: Brace expansion31877
-Node: KPSE_DOT expansion32816
-Node: Subdirectory expansion33329
-Node: Casefolding search35683
-Node: Casefolding rationale36452
-Node: Casefolding examples37791
-Node: Filename database42841
-Node: ls-R43899
-Node: Filename aliases46794
-Node: Database format47972
-Node: Invoking kpsewhich48985
-Node: Path searching options49940
-Node: Specially-recognized files58459
-Node: Auxiliary tasks59814
-Node: Standard options63520
-Node: TeX support63876
-Node: Supported file formats65230
-Node: File lookup72895
-Node: Glyph lookup74644
-Node: Basic glyph lookup75768
-Node: Fontmap76648
-Node: Fallback font79177
-Node: Suppressing warnings80089
-Node: mktex scripts81216
-Node: mktex configuration82431
-Node: mktex script names88234
-Node: mktex script arguments89620
-Node: Programming90499
-Node: Programming overview91072
-Node: Calling sequence93933
-Node: Program-specific files100465
-Node: Programming with config files101488
-Node: Reporting bugs103075
-Node: Bug checklist103753
-Node: Mailing lists107225
-Node: Debugging107900
-Node: Logging112977
-Node: Common problems114844
-Node: Unable to find files115321
-Node: Slow path searching117731
-Node: Unable to generate fonts119106
-Node: TeX or Metafont failing121577
-Node: Index122779
+Node: Introduction2260
+Node: History4330
+Node: unixtex.ftp8926
+Node: Security10351
+Node: TeX directory structure12855
+Node: Path searching16903
+Node: Searching overview17630
+Node: Path sources21449
+Node: Config files22540
+Node: Path expansion26467
+Node: Default expansion27420
+Node: Variable expansion29490
+Node: Tilde expansion30891
+Node: Brace expansion31871
+Node: KPSE_DOT expansion32810
+Node: Subdirectory expansion33323
+Node: Casefolding search35677
+Node: Casefolding rationale36446
+Node: Casefolding examples37785
+Node: Filename database42835
+Node: ls-R43893
+Node: Filename aliases47569
+Node: Database format48747
+Node: Invoking kpsewhich49760
+Node: Path searching options50715
+Node: Specially-recognized files59234
+Node: Auxiliary tasks60589
+Node: Standard options64295
+Node: TeX support64651
+Node: Supported file formats66005
+Node: File lookup73670
+Node: Glyph lookup75419
+Node: Basic glyph lookup76543
+Node: Fontmap77423
+Node: Fallback font79952
+Node: Suppressing warnings80864
+Node: mktex scripts81991
+Node: mktex configuration83206
+Node: mktex script names89009
+Node: mktex script arguments90395
+Node: Programming91274
+Node: Programming overview91847
+Node: Calling sequence94708
+Node: Program-specific files101240
+Node: Programming with config files102263
+Node: Reporting bugs103850
+Node: Bug checklist104528
+Node: Mailing lists108000
+Node: Debugging108675
+Node: Logging113752
+Node: Common problems115619
+Node: Unable to find files116096
+Node: Slow path searching118506
+Node: Unable to generate fonts119881
+Node: TeX or Metafont failing122352
+Node: Index123554
 
 End Tag Table
diff --git a/Build/source/texk/kpathsea/doc/kpathsea.texi b/Build/source/texk/kpathsea/doc/kpathsea.texi
index 3fef753b886..cbaa101867e 100644
--- a/Build/source/texk/kpathsea/doc/kpathsea.texi
+++ b/Build/source/texk/kpathsea/doc/kpathsea.texi
@@ -3,7 +3,7 @@
 @settitle Kpathsea: A library for path searching
 
 @set version 6.3.1
-@set month-year February 2019
+@set month-year March 2019
 
 @copying
 This file documents the Kpathsea library for path searching.
@@ -1138,6 +1138,7 @@ searches the disk in the big system directory, the casefolding is not
 done, and a search for @samp{stix-regular.otf} will fail (on
 case-sensitive systems), as it always has.
 
+@kindex !! @r{and casefolding}
 The caveat about not searching the disk amounts to saying that
 casefolding does not happen in the trees specified with @samp{!!}
 (@pxref{ls-R}), that is, where only database (@file{ls-R}) searching
@@ -1224,14 +1225,19 @@ source file @file{kpathsea/db.c}.
 @subsection @file{ls-R}
 
 @flindex ls-R @r{database file}
-@vindex TEXMFDBS
 
 As mentioned above, you must name the main filename database
 @file{ls-R}.  You can put one at the root of each @TeX{} installation
-hierarchy you wish to search (@code{$TEXMF} by default); most sites have
-only one hierarchy.  Kpathsea looks for @file{ls-R} files along the
-@code{TEXMFDBS} path, so that should presumably match the list of
-hierarchies.
+hierarchy you wish to search (@code{$TEXMF} by default, which expands
+to a braced list of several hierarchies in @TeX{} Live). 
+
+@vindex TEXMFDBS
+@kindex !! @r{in @code{TEXMFDBS}}
+Kpathsea looks for @file{ls-R} files along the @code{TEXMFDBS}
+path. It is best for this to contain all and only those hierarchies
+from @code{$TEXMF} which are specified with @code{!!}---and also to
+specify them with @code{!!} in @code{TEXMFDBS}. (See the end of this
+section for more on @code{!!}.)
 
 The recommended way to create and maintain @samp{ls-R} is to run the
 @code{mktexlsr} script, which is installed in @samp{$(bindir)}
@@ -1256,10 +1262,12 @@ Also presuming your @code{ls} hasn't been aliased in a system file
 generating @file{ls-R}.  For the precise definition of the file format,
 see @ref{Database format}.
 
-Regardless of whether you use the supplied script or your own, you will
-almost certainly want to invoke it via @code{cron}, so when you make
-changes in the installed files (say if you install a new La@TeX{}
-package), @file{ls-R} will be automatically updated.
+Regardless of whether you use the supplied script or your own, you
+will almost certainly want to invoke it via @code{cron}, so when you
+make changes in the installed files (say if you install a new @LaTeX{}
+package), @file{ls-R} will be automatically updated.  However, for
+those using @TeX{} Live or system distributions, the package managers
+should run @code{mktexlsr} as needed.
 
 @opindex -A @r{option to @code{ls}}
 @cindex dot files
@@ -1268,7 +1276,7 @@ package), @file{ls-R} will be automatically updated.
 @flindex .tex @r{file, included in @file{ls-R}}
 The @samp{-A} option to @code{ls} includes files beginning with @samp{.}
 (except for @file{.} and @file{..}), such as the file @file{.tex}
-included with the La@TeX{} tools package.  (On the other hand,
+included with the @LaTeX{} tools package.  (On the other hand,
 @emph{directories} whose names begin with @samp{.} are always ignored.)
 
 @cindex symbolic links, and @file{ls-R}
@@ -1293,14 +1301,27 @@ ignores everything under such directories.
 
 @kindex !! @r{in path specifications}
 @cindex disk searching, avoiding
-Because the database may be out-of-date for a particular run, if a file
-is not found in the database, by default Kpathsea goes ahead and
-searches the disk. If a particular path element begins with @samp{!!},
-however, @emph{only} the database will be searched for that element,
-never the disk.  If the database does not exist, nothing will be
-searched.  Because this can surprise users (``I see the font
-@file{foo.tfm} when I do an @code{ls}; why can't Dvips find it?''), it
-is not in any of the default search paths.
+If a particular path element begins with @samp{!!}, @emph{only} the
+database will be searched for that element, never the disk; and if the
+database does not exist, nothing at all will be searched.  In @TeX{}
+Live, most of the trees are specified with @samp{!!}.
+
+For path elements that do not begin with @samp{!!}, if the database
+exists, it will be used, and the disk will not be searched.  However,
+in this case, if the database does not exist, the disk will be
+searched.  In @TeX{} Live, the most notable case of this is the
+@code{TEXMFHOME} tree, to allow users to add and remove files from
+their own tree without having to worry about @code{ls-R}.
+
+(Aside: there are uncommon cases where a @samp{!!} tree will be
+searched on disk even if the @code{ls-R} file exists; they are too
+obscure to try to explain here.  See @file{pathsearch.c} in the source
+if you need to know.)
+
+To sum up: do not create an @code{ls-R} file unless you also take care
+to keep it up to date. Otherwise newly-installed files will not be
+found.
+
 
 
 @node Filename aliases
diff --git a/Build/source/texk/kpathsea/kpsewhich.c b/Build/source/texk/kpathsea/kpsewhich.c
index a3d095ae19a..1be21f39889 100644
--- a/Build/source/texk/kpathsea/kpsewhich.c
+++ b/Build/source/texk/kpathsea/kpsewhich.c
@@ -632,7 +632,7 @@ read_command_line (kpathsea kpse, int argc, string *argv)
     assert (g == 0); /* We have no short option names.  */
 
     if (ARGUMENT_IS ("casefold-search")) {
-      /* We can't just a boolean for casefold-search because we want to
+      /* We can't use a boolean for casefold-search because we want to
          distinguish it being set with an option vs. leaving the default
          (by default).  */
       xputenv ("texmf_casefold_search", "1");      
diff --git a/Build/source/texk/kpathsea/texmf.cnf b/Build/source/texk/kpathsea/texmf.cnf
index c951999fa74..97ed2a78714 100644
--- a/Build/source/texk/kpathsea/texmf.cnf
+++ b/Build/source/texk/kpathsea/texmf.cnf
@@ -109,12 +109,12 @@ TEXMFAUXTREES = {}
 % the search paths. tlmgr takes care to end any value with a trailing comma.
 TEXMF = {$TEXMFAUXTREES$TEXMFCONFIG,$TEXMFVAR,$TEXMFHOME,!!$TEXMFLOCAL,!!$TEXMFSYSCONFIG,!!$TEXMFSYSVAR,!!$TEXMFDIST}
 
-% Where to look for ls-R files.  There need not be an ls-R in the
-% directories in this path, but if there is one, Kpathsea will use it.
-% By default, this is only the !! elements of TEXMF, so that mktexlsr
-% does not create ls-R files in the non-!! elements -- because if an
-% ls-R is present, it will be used, and the disk will not be searched.
-% This is arguably a bug in kpathsea.
+% Where to look for, and where mktexlsr creates, ls-R files. By default,
+% this is all and only the !! elements of TEXMF, so that mktexlsr does not
+% create ls-R files in the non-!! elements -- because if an ls-R is
+% present, it will be used, and the disk will not (usually) be searched,
+% regardless of !!.  Although in principle a directory listed here need
+% not contain an ls-R file, in practice they all should.
 TEXMFDBS = {!!$TEXMFLOCAL,!!$TEXMFSYSCONFIG,!!$TEXMFSYSVAR,!!$TEXMFDIST}
 
 % The system trees.  These are the trees that are shared by all users.