path: root/macros/latex/contrib/l3kernel/l3news12.tex

% Copyright 2019 The LaTeX3 Project
\documentclass{ltnews}

\usepackage{csquotes}
\usepackage{hologo}
\usepackage{ragged2e}

\PassOptionsToPackage{colorlinks}{hyperref}

%%% wrong in ltnews.cls ... so some tmp fix here at the moment:
\makeatletter
\renewcommand{\subsubsection}{%
   \@startsection
      {subsubsection}{2}{\leftmargini}{-1.5ex \@plus -1ex \@minus -.2ex}%
      {1sp}{\@subheadingfont}%
}
\makeatother

\AtBeginDocument{%
  \renewcommand*{\LaTeXNews}{\LaTeX3~News}%
  \RaggedRight
  \setlength\parindent{1em}%
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\publicationmonth{January}
\publicationyear{2020}
\publicationissue{12}

% Avoid hyphenation of csnames
\makeatletter
\protected\edef\cs#1{%
  \noexpand\path{\@backslashchar#1}%
}
\makeatother

\begin{document}

\maketitle

\tableofcontents

\section{Introduction}

There has been quite a gap since the last \emph{\LaTeX3 News} (Issue~11,
February 2018), and so there is quite a bit to cover here. Luckily, one of the
things there \emph{is} to cover is that we are using a more formalised approach
for logging changes, so writing up what has happened is a bit easier.
(By mistake \LaTeX3 News~11 itself did not get \emph{published} when written, but is
now available: we have kept the information it contains separate as it
is a good summary of the work that had happened in 2017.)

Work has continued apace across the \LaTeX3 codebase in the last (nearly) two
years. A lot of this is ultimately focussed on making the core of \pkg{expl3}
even more stable: \emph{squeezing} out more experimental ideas, refining
ones we have and making it a serious option for core \LaTeX{} programming.

As a result of these activities, the \LaTeX3 programming layer will be
available as part of the kernel of \LaTeXe{} from 2020-02-02 onwards, i.e., can
be used without explicitly loading \pkg{expl3}. See \emph{\LaTeX{}
  News~31}~\cite{12:site-news} for more details on this.

\section{New features in \pkg{expl3}}

\subsection{A new argument specifier: \texttt{e}-type}

During 2018, the team worked with the \TeX{} Live, \hologo{XeTeX} and
(u)p\TeX{} developers to add the \cs{expanded} primitive to \hologo{pdfTeX}{},
\hologo{XeTeX} and (u)p\TeX{}. This primitive was originally suggested for
\hologo{pdfTeX}{} v1.50 (never released), and was present in \hologo{LuaTeX}{}
from the start of that project.

Adding \cs{expanded} lets us create a new argument specifier: \texttt{e}-type
expansion. This is \emph{almost} the same as \texttt{x}-type, but is itself
expandable. (It also doesn't need doubled \verb|#| tokens.) That's incredibly
useful for creating function-like macros: you can ensure that \emph{everything}
is expanded in an argument before you go near it, with not an \cs{expandafter}
in sight.

\subsection{New functions}

New programming tools have appeared in various places across \pkg{expl3}.
The highlights are
\begin{itemize}
  \item Shuffling of sequences to allow randomization
  \item Arrays of integers and floating point values; these have constant-time
    access
  \item Functions to return values after system shell usage
  \item Expandable access to file information, including file size, MD5
    hash and modification date
\end{itemize}

For the latter, we have revised handling of file names considerably. There is
now support for finding files in expansion contexts (by using the
\cs{(pdf)filesize} primitive). Spaces and quotes in file names are now fully
normalised, in a similar manner to the approach used by the latest \LaTeXe{}
kernel.

\subsection{String conversion moves to \pkg{expl3}}

In addition to entirely new functions, the team have moved the
\pkg{l3str-convert} module from the \pkg{l3experimental} bundle into the
\pkg{expl3} core. This module is essential for dealing with the need to produce
\textsc{utf}-16 and \textsc{utf}-32 strings in some contexts, and also offers
built-in escape for \textsc{url}s and \textsc{pdf} strings.

\subsection{Case changing of text}

Within \pkg{expl3}, the team have re-named and reworked the ideas from
\cs{tl_upper_case:n} and so on, creating a new module \pkg{l3text}. This is a
\enquote{final} home for functions to manipulate \emph{text}; token lists that
can reasonably expected to expand to plain text plus limited mark-up, for
example emphasis and labels/references. Moving these functions, we have also
made a small number of changes in other modules to give consistent names to
functions: see the change log for full details.

Over time, we anticipate that functions for other textual manipulation will be
added to this module.

\section{Notable fixes and changes}

\subsection{File name parsing}

The functions for parsing file names have been entirely re-written, partly as
this is required for the expandable access to file information mentioned above.
The new code correctly deals with spaces and quote marks in file names and
splits the path/name/extension.

\subsection{Message formatting}

The format of messages in \pkg{expl3} was originally quite text-heavy, the idea
being that they would stand out in the \texttt{.log} file. However, this made
them hard to find by a regular expression search, and was very different from
the \LaTeXe{} message approach. The formatting of \pkg{expl3} messages has been
aligned with that from the \LaTeXe{} kernel, such that \textsc{ide} scripts and
similar will be able to find and extract them directly.

\subsection{Key inheritance}

A number of changes have been made to the inheritance code for keys,
to allow inheritance to work \enquote{as expected} in (almost) all
cases.

\subsection{Floating point juxtaposition}

Implicit multiplication by juxtaposition, such as |2pi|, is now handled
separately from parenthetic values. Thus for example |1in/1cm| is treated
as equal to |(1in)/(1cm)| and thus yields |2.54|, and | 1/2(pi+pi)| is
equal to |pi|.

\subsection{Changing box dimensions}

\TeX{}'s handling of boxes is subtly-different from other registers, and this
shows up in particular when you want to resize a box. To bring treatment of
boxes, or rather the grouping behavior of boxes, into line with other
registers, we have made some internal changes to how functions such as
\cs{box_set_wd:N} are implemented. This will be transparent for
\enquote{well-behaved} use cases of these functions.

\subsection{More functions moved to stable}

A large number of functions which were introduced as candidates have been
evaluated and moved to stable status. The team hopes to move all functions
in \pkg{expl3} to stable status, or move them out of the core, over the coming
months.

\subsection{Deprecations}

There have been two notable sets of deprecations over the past 18 months.
First, we have rationalised all of the \enquote{raw} primitive names to the
form \cs{tex_<name>:D}. This means that the older names, starting
\cs{pdftex_...}, \cs{xetex_...}, etc., have been removed.

Secondly, the use of integer constants, which dates back to the earliest days
of \pkg{expl3}, is today more likely to make the code harder to read than
anything else. Speed improvements in engines mean that the tiny enhancements in
reading such constants are no longer required. Thus for example \cs{c_two} is
deprecated in favour of simply using \texttt{2}.

In parallel with this, a number of older \texttt{.sty} files have been removed.
These older files provided legacy stubs for files which have now been
integrated in the \pkg{expl3} core. They have now had sufficient time to allow
users to update their code.

\section{Internal improvements}

\subsection{Cross-module functions}

The team introduced the idea of internal module functions some time ago. Within
the kernel, there are places where functions need to be used in multiple
modules. To make the nature of the kernel interactions clearer, we have worked
on several aspects
\begin{itemize}
  \item Reducing as far as possible cross-module functions
  \item Making more generally-useful functions public, for example
    scan marks
  \item Creating an explicit cross-kernel naming convention for
    functions which are internal but are essential to use in multiple
    kernel modules
\end{itemize}

\subsection{The backend}

Creating graphics, working with color, setting up hyperlinks and so on require
backend-specific code. Here, backends are for example \texttt{dvips},
\texttt{xdvipdfmx} and the direct \textsc{pdf} mode in \hologo{pdfTeX} and
\hologo{LuaTeX}. These functions are needed across the \LaTeX3 codebase and
have to be updated separate from the \pkg{expl3} core. To facilitate that, we
have split those sources into a separate bundle, which can then be updated as
required.

At the same time, the code these files contain is very low-level and is best
described as internal. We have re-structured how the entire set of functions
are referred to such that they are now internal for the area they implement,
for example image inclusion, box affine transformations, etc.

\section{Better support for (u)p\TeX{}}

The developers behind (u)p\TeX{} (Japanese \TeX{}) have recently enhanced their
English documentation (see \url{https://github.com/texjporg/ptex-manual}).
Using this new information, we have been able to make internal adjustments to
\pkg{expl3} to better support these engines.

\section{Options}

A new option \texttt{undo-recent-deprecations} is now available for cases where
a document (or package) requires some \pkg{expl3} functions that have been
formally removed after deprecation. This is to allow \emph{temporary}
work-arounds for documents to be compiled whilst code is begin updated.

The \enquote{classical} options for selecting backends (\texttt{dvips},
\texttt{pdftex}, etc.) are now recognised in addition to the native key--value
versions. This should make it much easier to use \pkg{expl3} image and color
support as it is brought up to fully-workable standards.

\section{Engine requirements}

The minimum engine versions needed to use \pkg{expl3} have been
incremented a little:
\begin{itemize}
  \item \hologo{pdfTeX} v1.40
  \item \hologo{XeTeX} v0.99992
  \item \hologo{LuaTeX} v0.95
  \item $\epsilon$-(u)p\TeX{} mid-2012
\end{itemize}

The team have also worked with the \hologo{XeTeX} and (u)p\TeX{} developers to
standardise the set of post-\eTeX{} utility primitives that are available: the
so-called \enquote{\hologo{pdfTeX} utilities}. These are now available in all
supported engine, and in time will all be \emph{required}. This primarily
impacts \hologo{XeTeX}, which gained most of these primitives in the 2019
\TeX{} Live cycle. (Examples are the random number primitives and expandable
file data provision.) See \emph{\LaTeX{} News~31}~\cite{12:site-news} for more on this.

\section{Documentation}

\subsection{News}

The \emph{\LaTeX3 News} files were until recently only used to create
\textsc{pdf} files on the team website~\cite{12:site}. We have now
integrated those into the \pkg{l3kernel} (\pkg{expl3} core)
bundle. The news files cover all of \LaTeX3 files, as the core files
are always available.

\subsection{ChangeLog}

Since the start of 2018, the team have commenced a comprehensive
change log for each of the bundles which make up the \LaTeX3 code.
These are simple Markdown text files, which means that they can be displayed
formatted in web views.

\section{Changes in \pkg{xparse}}

A number of new features have been added to \pkg{xparse}. To allow
handling of the fact that skipping spaces may be required only in
some cases when searching for optional arguments, a new modifier
\texttt{!} is available in argument specifiers. This causes
\pkg{xparse} to \emph{require} that an optional argument follows
immediately with no intervening spaces.

There is a new argument type purely for environments: \texttt{b}-type
for collecting a \cs{begin}/\cs{end} pair, i.e., collecting the body of
an environment. This is similar in concept to the \pkg{environ} package,
but is integrated directly into \pkg{xparse}.

Finally, it is now possible to refer to one argument as the
default for another optional one, for example
\begin{verbatim}
\NewDocumentCommand{\caption}{O{#2} m}
  ...
\end{verbatim}

\section{New experimental modules}

A number of new experimental modules have been added within the
\pkg{l3experimental} bundle
\begin{itemize}
  \item \pkg{l3benchmark} Performance-testing system using the timing
    function in modern \TeX{} engines
  \item \pkg{l3cctab} Category code tables for all engines, not
    just \hologo{LuaTeX}
  \item \pkg{l3color} Color support, similar in interface to \pkg{xcolor}
  \item \pkg{l3draw} Creation of drawings, inspired by \pkg{pgf}, but
    using the \LaTeX3 \textsc{fpu} for calculations
  \item \pkg{l3pdf} Support for \textsc{pdf} features such as compression,
    hyper-links, etc.
  \item \pkg{l3sys-shell} Shell escape functions for file manipulation
\end{itemize}

\section{\pkg{l3build} changes}

The \pkg{l3build} tool for testing and releasing \TeX{} packages
has seen a number of incremental improvements. It is now available directly
as a script in \TeX{} Live and MiK\TeX{}, meaning you can call it simply
as
\begin{verbatim}
l3build <target>
\end{verbatim}
Accompanying this, we have added support for installing scripts and script
\texttt{man} files.

There is a new \texttt{upload} target that can take a zip file and send it to
\textsc{ctan}: you just have to fill in release information for \emph{this}
upload at the prompts.

Testing using \textsc{pdf} files rather than logs has been heavily revised:
this is vital for work on \textsc{pdf} tagging.

There is also better support for complex directory structures, including the
ability to manually specify \textsc{tds} location for all installed files. This
is particularly targeted at package with both generic and format-specific file
to install.


\begin{thebibliography}{9}

\fontsize{9.3}{11.3}\selectfont


\bibitem{12:site}
  \emph{\LaTeX{} Project Website}.
  \url{https://latex-project.org/}

\bibitem{12:site-news}
  \emph{\LaTeXe{} release news letters on the \LaTeX{} Project Website}.
  \url{https://latex-project.org/news/latex2e-news/}

  
\end{thebibliography}

\end{document}