
COMPLETE BUILD INSTRUCTIONS

COMPILING

This is the central document describing building the binaries and submitting them to TeX Live. But two others documents are important for the first steps. The document

	BuildStatus-2022
	
explains how to get the sources from TUG, and how to build these sources on Arm and on Intel. We support the last three versions of macOS (and this year one earlier system for good measure), and compile on the oldest systems supported each processor. In 2022, the Intel binaries were created on an Intel machine running Mojave, and the Arm binaries were created on an Arm machine running Big Sur. These sets of binaries come in folders named x86_64-apple-darwin18.7.0 and aarech64-apple-darwin20.6.0.

The program xindy requires CLisp to build, but not to run. Before 2022, we used a prebuilt clisp-2.49 on Intel because we had trouble building later versions of CLisp. But in the summer of 2021, instructions for building the most recent version were sent by Roberto Avanzi, roberto.av anzi@gmail.com. His instructions work on both Intel and Arm, so in 2022 our Xindy uses the latest CLisp and is universal. Details are in the BuildStatus document.

The program asymptote is built as a separate step, but the source code is part of the TeX Live source code. Building it is explained in a separate document in AsymptoteBuild named 

	AsymptoteBuild-2022 
	
This involves building several libraries just once each year. Then we rebuild asy each time the code changes, and name the resulting binaries asy-Intel and asy-Arm.

The documents BuildStatus-2022 and AsymptoteBuild-2022 explain all details of building on Arm and on Intel. From now on, assume these steps have been done.

LIPO AND SIGNING

This is the primary document for all further steps. Usually we create MacTeX on the latest Apple Hardware. In 2022,
remaining steps were done on the latest 16" Arm Macbook Pro running Monterey. 

There are two separate ways to combine, sign, process and submit the binaries. The easier is for an initial build, where all binaries are submitted to Karl Berry for inclusion in TeX Live. The more complicated way is for later builds where only some of the files have changed. In that case, Karl's software uses file creation dates to detect the changed binaries. Unfortunately, the lipo operation that makes universal binaries from arm and intel binaries destroys creation dates, so Karl's script marked all binaries as changed, and Mac users were forced to update their full set of binaries. We now use a more complicated procedure so Karl's script correctly detects changed files.

We first describe the initial build.

The Binary folder has a subfolder named RawCode. After the Arm and Intel binaries are created, place them in RawCode. This folder will then contain two files and two folders:

	asy-Arm
	asy-Intel
	aarch64-apple-darwin20.6.0
	x86_64-apple-darwin18.7.0
	
A note of caution: In 2021, Apple issued a minor update of the system during the pretest period and the number at the end of the aarch64 name changed. Be on the lookout for such changes and revise the scripts described in this document appropriately.

SIGNING CERTIFICATES

Another note of caution: In 2022, just after the initial binaries were created, Apple sent me an email explaining that my signing signature would expire in thirty days and a new signature should be created immediately. Two significant signatures are involved: "Developer ID Application" and "Developer ID Installer". The first is used to sign binaries and applications, and the second is used to sign install packages. Signatures are valid for five years after first issued. When an application signature expires, the applications signed by it continue to run. When an installer signature expires, packages signed by it immediately refuse to install. When an install package is notarized, Apple requires that all binaries and applications inside be signed with a currently valid application signature.

My signatures are called "Developer ID Application: Richard Koch" and "Developer ID Installer: Richard Koch". When scripts in the build system are told to use these signatures, they reach inside the keychain on my Monterey machine to find the desired information. When another developer takes over my job, they will obtain their own signatures, install them in their keychain, and modify the scripts described below to use their name rather than mine for signing operations.

Obtaining new signatures is very easy, but installing them --- not so much. Apple's instructions tell a developer to download the new certificate and double click it. This opens KeyChain, but that program asks which keychain to install into:

	login
	system
	iCloud
	
It turns out that "login" is the correct answer (after brutal experiments with other choices). The result is two signatures with exactly the same name, one the older certificate and one the new certificate. Google sources explain that the older certificates must immediately be removed from keychain. I didn't know that, and some scripts then gave error messages because they didn't know which certificate to use. 

Removing a certificate from the keychain leads to a scary message, and sure enough trouble then arose. In some cases I could not re-sign binaries with my new signature because the system claimed that it could not find the desired information to remove the old signature. 

The new certificates work with recent versions of XCode, but not older versions. Recall that binaries are actually created on older systems running on older machines. None of the build instructions on these older machines refer to signing. I did not update my signatures on older machines.

All recent versions of macOS contain a script named "codesign". This script does the actual signing, or removing signatures, or inspection of signatures. For example

	codesign --display pdftex

will display signature information. When this command is run on an intel binary like pdftex created on Mojave, it returns

	pdftex: code object is not signed at all
	
When it is run on an arm binary like xz-arm created on Big Sur, it returns something like

	Executable=/Users/koch/MacTeX-2022/xz-and-wget/RawCodeOriginal/xz-arm
	
Therefore, apparently the Intel binaries are not signed, but the Arm binaries mysteriously are. This may not be so surprising, since XCode knows about developer certificates.

Experiments show that if the Arm binaries are owned by a regular user rather than root, then the command

	codesign --remove-signature pdftex
	
removes the signature, even if run on Monterey where only the new signatures exist.

We want all signing to be done on the latest system, using the latest signatures, and we want signatures on older systems to be irrelevant. So in 2022, after importing binaries from Arm to the final Monterey machine, we run a script which removes all the signatures. This script is called "unsignarm.sh". Several copies many exist, in various kinds of RawCode folders, but it is always the same script and it looks at all files, picks out those containing arm binary code, and removes the signature of such files.

MAKING THE INITIAL BUILD

In Binary there is now a folder named RawCode containing binaries from other machines and one script "unsignarm.sh"

	asy-Arm
	asy-Intel
	aarch64-apple-darwin20.6.0
	x86_64-apple-darwin18.7.0
	unsignarm.sh
	
Change directory to RawCode and issue the command

	sh unsignarm.sh
	
This removes signatures from the arm binaries (curiously, removing the asy-Arm signature is not necessary).

Now change directory to Binary. This folder has two scripts called 
	
	lipocreateuniversaldarwin.sh
	liposignuniversaldarwin.sh
	
Run the first by typing

	sh lipocreateuniversaldarwin.sh
	
It is very important to run this as a regular user, so do not begin with sudo. The script removes the folder universal-darwin
and the file universal-darwin.tar.xz from Binary, if they exist. Then it create the folder universal-darwin. It then makes a list of all files in the Intel binary directory. If a file is a binary, it uses lipo to combine this Intel code with the corresponding Arm code and writes the resulting universal file to the folder universal-darwin. For all other scripts, links, and the like it just copies the Intel version to universal-darwin.  Next the script combines asy.Intel and asy.Arm and writes the resulting asy to the folder universal-darwin. It creates the xasy symbolic link in universal-darwin. At this point, universal-darwin has an intel file named xindy.mem. The script renames this xindy-x86_64.mem and then copies the corresponding arm file, naming it
xindy-arm64.mem. This is done because xindy.mem, created by the compile script, is not binary code and yet is processor dependent. The Xindy program running on a Macintosh knows that it must use the appropriate version of this file.

Finally, run the second script by typing

	sh liposignuniversaldarwin.sh
	
Again, it is important to run this as a regular user, without sudo. This signs all binary files using the latest Developer ID Application signature, adopts a hardened runtime for each, and applies appropriate entitlement files for a few binaries. For example, applications running with a hardened runtime are not allowed to link to third party libraries, but a small number of programs like xdvi-raw need to link to the X11 library, which is open source and provided by a third party. An appropriate entitlement file allows this.

After signing, the script compresses universal-darwin with tar and again with xz, producing

	universal-darwin.tar.xz
	
A copy of this file is added to RawCode for reasons made clear in the next section.
	
Then universal-darwin.tar.xz is uploaded to TUG, and Karl Berry is notified to find it and install the binaries in TeX Live.

(Note that one binary, biber, is supplied by the author. Karl will obtain and install it separately.)


UPDATING PREVIOUS BUILDS

We now come to the harder problem of updating previous builds. First we describe the idea, and then how it is carried out.

We will start with two folders named "RawCodePrevious" and "RawCodeRebuild". The first folder contains all the information used to create the previous build. This means that it contains the raw unsigned Intel  code, the raw unsigned Arm  code, and the universal-darwin folder this code created. The second folder contains information for the new build. So it contains the raw unsigned Intel code and the raw unsigned Arm code.

From this, we create a new universal-darwin folder and a new univeral-darwin.tar.xz file at the top level of the Binary folder.
How do we do this? We examine each new Intel file piece by piece and determine whether it is a binary file or something else, like a script or symbolic link. If it is not a binary, then we copy it to universal-darwin.  After all, dates are fine when we compile for only one processor; they are broken only when we lipo two files together.

If the file is an intel binary, we compare it to the corresponding intel binary in RawCodePrevious. If these binaries are the same, then nothing changed. So we copy the corresponding universal-darwin binary in RawCodePrevious to the new universal-darwin. But if the two Intel files are different, then we lipo the new intel and new arm files, and sign and harden the new file with appropriate entitlements, and add that to the new universal-darwin.

So in detail, here is how the process proceeds. The first step is to create the folder "RawCodePrevious". If this is the first time we have updated the original binaries, we just make a copy of RawCode and rename the copy RawCodePrevious. Recall that this folder contains universal-darwin.tar.xz. We reach into the new RawCodePrevious and expand this file, creating a folder universal-darwin in RawCodePrevious.

On the other hand, if this is a subsequent update, we trash the previous "RawCodePrevious". Then we rename "RawCodeRebuild" to "RawCodePrevious".  As we will see shortly, this folder contains universal-darwin.tar.xz from the previous build. We expand this file, creating "universal-darwin" in RawCodePrevious.

We then create a new file "RawCodeRebuild". Next we rebuild binaries for Intel, for instance on Mojave in 2022. This should be a full rebuild including Xindy, but without asy. Copy the resulting binaries to RawCodeRebuild on the main Monterey machine. Similarly rebuild binaries for Arm, for instance on Big Sur in 2022. Again this should be a full rebuild including Xindy, but without asy.  Copy this to RawCodeRebuild on the main Monterey machine.

If Karl announces that asy has changed, also create asy.Intel and asy.Arm on the build machines, and add these files to RawCodeRebuild on the Monterey machine. But if not, copy asy.Intel and asy.Arm from RawCodePrevious to RawCodeRebuild.

Put a copy of the script unsignarm.sh in RawCodeRebuild, and run it by changing directory to RawCodeRebuild and

	sh unsignarm.sh
	
In the main Binary directory there is a script named "rebuild.sh".  Run it as a regular user, not with sudo. So

	sh rebuild.sh
	
This script will remove universal-darwin and universal-darwin.tar.xz from the Binary folder. Then it will create a new universal-darwin folder in Binary and populate it by following the logic described at the start of this section. Sometimes it will copy scripts and links from the new Intel, sometimes it will copy binaries from the old universal-darwin, and sometimes it will lipo new intel and arm versions of a binary, sign and harden the result, and apply entitlements as needed. In the end it will produce a new universal-darwin.tar.xz file and add a copy of this file to RawCodeRebuild.

Transfer universal-darwin.tar.xz to TUG and notify Karl.

Incidentally, when rebuild.sh is executed, it will print a list of the changed binaries. These can be sent to Karl as a check that we are doing the correct thing.

WHEW.	

	
