summaryrefslogtreecommitdiff
path: root/Master/tlpkg/doc/repository-setup.txt
blob: f72af9c8f44aa79844a95ea0ceae58babf5b75f2 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
$Id$
(This document is public domain.)

How to set up your own TeX Live Packages repository
===================================================
TeX Live 2008 and later contain the TeX Live Manager, which allows 
updates over the network. By specifying different installation sources
the user can in theory install packages using tlmgr from a different
source as the default TeX Live repository.

This document describes how to set up a repository in a way that tlmgr
can use it. Possible applications include distribution of company-internal
packages or non-free stuff not included in TeX Live.

The current document is based on the status of the TeX Live repository
scripts as of 2018-02-12.

Please write tex-live@tug.org if questions or suggestions.

Contents
--------
0. Preliminaries
1. Prerequisites
2. Preparing the "repository"
3. tlpsrc files
4. Creating the TeX Live Database
5. Adjusting the revision numbers
6. Making the web distribution
7. Testing your distribution
8. Signing repositories
9. Example update script


0. Preliminaries
----------------
First of all, before you start with providing your own packages, please
be reminded that you should take good care not to
- overwrite files which are in a different package
  The current system does NOT check for files that got overwritten
- take care not to hose the rest of the TeX Live system,
  imagine you install a package that removes/deletes tlmgr --
  the user will hardly be able to revert to any working state!
- test, test, and again TEST before shipping your stuff
- might be good to alert us at tex-live@tug.org so that we can take a look
  and comment, or check for other problems. Also, if we change something
  in the infrastructure you have a better chance in getting notifications
  and better treatment ;-)

In principle what you have to do is to set up some directory in a layout
paralleling the one in our repository under Master. See
http://www.tug.org/texlive/svn/ for details on accessing our repo.

That means, with ROOT as the root of your local "repository":
	ROOT/tlpkg/		infrastructure stuff, modules, scripts
	ROOT/texmf-dist/	stuff where you should put your 
				TeX code, fonts etc

After that you call the very same scripts as we do when building the
distribution database in
	ROOT/tlpkg/texlive.tlpdb

Finally you create/update your web distribution from that ROOT.


1. Prerequisites
----------------
You need from the TeX Live Repository (see http://www.tug.org/texlive/svn/)
the contents of at least
	Master/tlpkg/TeXLive		the perl modules
	Master/tlpkg/bin		the scripts for managing
These files can be checked out via svn, or downloaded from the net.
We recommend svn, so that you can easily update them regularly.

Note that it is not necessary to have the above files and the files
you are distributing in the same file hierarchy. It is completely valid
to have a full checkout of TeX Live (svn or git) and use the scripts
from there. See Section 9 for an example management script.



2. Preparing your repository
----------------------------
Create a root folder which will mirror our directory layout. We will call
that root folder ROOT from now on.

Put all of YOUR files in a TDS compliant way into the texmf-dist folder:
	ROOT/texmf-dist
Usage of 'texmf' as tree is strongly discouraged but still possible.

IMPORTANT: Create the directory
  ROOT/texmf-dist/web2c
(in case of a git repository put in a dummy .dont-delete-me to make sure
the directory is present). tl-update-tlpdb and tl-update-containers rely
on this directory existence to determine the type of repository
(files based or container based).

If you really want to replace packages with files in texmf, please think
twice, and maybe contact us before.


3. tlpsrc files
---------------
Now you have to create several TeX Live Package Source files, .tlpsrc files
and place them into
	ROOT/tlpkg/tlpsrc

You have to have two .tlpsrc file for each package you want to ship, and
furthermore three configuration files for the overall installation:
  00texlive.config.tlpsrc
  00texlive.installation.tlpsrc
These files can be copied from the main TeX Live repository, and only
00texlive.config.tlpsrc needs adjustment. This package defines some 
basic configurations for the tlpdb, like how it announces itself,
whether separate src and doc containers are build, and the release 
years it supports.

In the end the generate "package" 00texlive.config should contain
(at least) the following lines:
	category TLCore
	depend container_format/xz
	depend release/NNNN-YOUR_ID

and might optionally contain:
	depend container_split_src_files
	depend container_split_doc_files
	depend minrelease/MMMM-YOUR_ID

The 
	depend container_format/xz
is *obligatory* because currently we do NOT support any other format.

The "container_split_src_files" and "container_split_doc_files" are
options, they specify whether at a later stage the container script will
generate separate .tar.xz archives for source and doc files, or put
all of them into one .tar.xz. It is up to you.

Finally, the 
	depend release/NNNN-YOUR_ID
and (optionally)
	depend minrelease/MMMM-YOUR_ID
determine(s) which release(s) of TL your repository works with.
If minrelease is present, the year specified in the local (user's) 
installation's $TLConfig::ReleaseYear must be in the range 
[minrelease,release].  
If minrelease is not present, the release/NNNN value must exactly 
match the the TL release number NNNN.
For example, you might have:
	depend release/2010-myorg
and if your packages work with both TL'09 and TL'10, you could add:
	depend minrelease/2009-myorg


For the rest of your package you should create .tlpsrc files describing
their content. Either copy one of our .tlpsrc files, or (much better) 
PLEASE read the documentation of TLPSRC.pm (in POD format).

If you want to use the auto-generated patterns we are using in TeX Live 
you have to copy our 00texlive.autopatterns.tlpsrc into your tlpkg/tlpsrc
directory.

Some guidelines: For simple packages, say a LaTeX package foobar, put all
the files in 
	ROOT/texmf-dist/tex/latex/foobar/
	ROOT/texmf-dist/source/latex/foobar/
	ROOT/texmf-dist/doc/latex/foobar/
then create a two-line .tlpsrc file foobar.tlpsrc containing:
	category Package
(that is all). The rest will be automatically done by our scripts. If you
want to understand that, read the section on auto-generated patterns in 
the TLPSRC.pm documentation (and don't forget to copy 
00texlive.autopatterns.tlpsrc).


4. Creating the TeX Live Database
---------------------------------
If you have set up all your local files, and the tlpsrc files, you can 
create the TeX Live Database in ROOT/tlpkg/texlive.tlpdb by calling
	perl PATH-TO/tl-update-tlpdb OPTIONS
Here PATH-TO points to the directory Master/tlpkg/bin in your TeX Live
checkout, or where you have saved the files from TeXLive and tlpkg 
directories.

How revision numbers are computed and which files are used can be controlled
by one of the several -from-XXXX options. By default Subversion is used.
Other options, if your repository is not subversion based, are
  -from-files          use find to get list of files, revisions are all 1
  -from-git            use git and compute the history (number of commits)
  -from-gitsvn         use git over svn repo

By default tl-update-tlpdb uses all tlpsrc files present in 
ROOT/tlpkg/tlpsrc/. Presumably this won't be very many files.
But if you like, you can specify individual package names together 
with -tlpsrc-from-cmdline.

You can add the option
	-with-w32-pattern-warning
which will warn on missing files for Windows; some of those warnings are
by default suppressed.

Another option that might be of interest is
	-no-reverse-revision-check
Normally if there is an old tlpdb the packages in the new one are compared
with the packages in the old one and checked that the revisions are
increasing.

As usual, please read the documentation of tl-update-tlpdb (again in POD)
for further information.

That process will probably show you some warnings; read them carefully.

After it runs successfully, you should have a file
	ROOT/tlpkg/texlive.tlpdb
	
Your next job is to check that your new texlive.tlpdb has the files you
expected in the corresponding packages that you expected.  Go through
the entire file (each package starts with a line "name foobar") and
the file list..

If you are not content, work on your patterns to fix that.

There are some scripts in tlpkg/bin/ starting with "check-" which you might
use for doing some checks on your repository.


5. Adjusting the revision numbers
---------------------------------
If you used -from-files above all the revision numbers in the generated
texlive.tlpdb will be set to 0:
	revision 0
The revision is used for checking which package is newer. So if you 
have different "releases" of a package you should increase the revision
number from one step to the next.

If you want to change that, there are several options:
- do it by hand in the texlive.tlpdb
  ... if you can remember what numbers you already shipped

- do it automatically with some script which takes into account the already
  shipped version of the packages, or the version of the package in
  the main TeX Live repository (if you replace a TeX Live package)
  An example for doing that take a look at our repository
  tlpkg/etc/tlrepo-from-minimals.pl

- use either subversion or git for your repository and get automatically
  increasing numbers.


6. Making the web distribution
------------------------------
Now that the texlive.tlpdb has been set up, revision numbers adjusted, you
can create or update your web distribution with
	perl PATH-TO/tl-update-containers OPTIONS

OPTIONS:
-location DEST
The -location gives the place where your former distribution was or where
it should be created. In the former case only those containers with larger
revision numbers are recreated. In the latter case all containers are
created.

-master `pwd`
The option -master defines the location of *your* files, that is ROOT
in this document.

-all
Update all containers

-no-sign
-gpgcmd
See below for information about signing repositories.

For other options, please see the documentation (in POD) of
tl-update-containers.


7. Testing your distribution
----------------------------
Having completed all these steps, you can test your distribution by calling
the TeX Live Manager with the location argument. Continuing the above
exmaple we use DEST:
	tlmgr -location DEST update --list
or 
	tlmgr -location DEST update --all
or 
	tlmgr -location DEST list
etc.  Please see the documentation of tlmgr (POD again) for details.


8. Signing repositories
-----------------------
Since 2016 tlmgr is able to check cryptographic signatures and thus
verify that the downloaded packages are what is distributed by the
author/maintainer (you reading this document).

If you don't to sign, please pass
  -no-sign
to tl-update-containers above.

If you want to sign you should provide a signing script that accepts
one file name and creates a detached signature for that file.
Note that in this case you should publish your public key, and ask users
to import your public key with 
  tlmgr key add ...
(see documentation of tlmgr for details).

The signing script can be as simple as
  gpg --armor --detach-sign --local-user 0xYOUR_ID "$1"
See below for a bit more complicated script.



9. Example update script
------------------------
For the TeX Live Contrib repository which is managed in git, I have the 
following files:
ROOT/bin/...
    /texmf-dist
               /web2c/.dont-delete-this-dir
               /....   all the other files are also under texmf-dist
    /tlpkg
          /tlpsrc
                 /00texlive.autopatterns.tlpsrc
                 /00texlive.config.tlpsrc
                 /00texlive.installation.tlpsrc
                 /...tlpsrc   all the other .tlpsrc files, mostly empty
          /texlive.tlpdb 
    /tl-sign-file
    /update.sh
    
I am using the following script update.sh (more or less, for details
see the git repo of tlcontrib):

TLCHECKOUT=${TLCHECKOUT:-$HOME/Development/TeX/texlive.git}
TLNETDEST=${TLNETDEST:-$HOME/Domains/server/texlive.info/contrib/2017}
TLCATALOGUE=${TLCATALOGUE:-$HOME/Development/TeX/texcatalogue-svn}

$TLCHECKOUT/Master/tlpkg/bin/tl-update-tlpdb \
    -with-w32-pattern-warning \
    -from-git \
    -catalogue=$TLCATALOGUE \
    -master=`pwd`

$TLCHECKOUT/Master/tlpkg/bin/tl-update-containers \
    -master `pwd` \
    -location $TLNETDEST    \
    -gpgcmd `pwd`/tl-sign-file \
    -all # sometimes we need -recreate

and the tl-sign-file is very similar to the TeX Live version, only
that the --local-user is changed and I need to enter the passphrase
manually.

tl-sign-file:

if test $# -ne 1; then
  echo "$0: Exactly one argument must be given, the file to sign." >&2
  exit 1
fi

# remove previous signature
rm -f "$1.asc"

prg=gpg
if ! [ "x$TL_GNUPG" = "x" ] ; then
  # use the environment variable TL_GNUPG
  prg=$TL_GNUPG
fi

# sign
$prg --armor \
    --detach-sign \
    --local-user 0xEC00B8DAD32266AA \
    "$1"


===================================================================

Closing remark:
We cannot repeat it often enough, please don't play around with files not
belonging to your packages. That is dangerous!


If you have further comments, questions, additions to this file please
let us know at tex-live@tug.org.


Last changed 2018-02-12 by np