path: root/macros/latex-dev/base/ltnews28.tex

% \iffalse meta-comment
%
% Copyright (C) 2017-2020
% The LaTeX3 Project and any individual authors listed elsewhere
% in this file.
%
% This file is part of the LaTeX base system.
% -------------------------------------------
%
% It may be distributed and/or modified under the
% conditions of the LaTeX Project Public License, either version 1.3c
% of this license or (at your option) any later version.
% The latest version of this license is in
%    http://www.latex-project.org/lppl.txt
% and version 1.3c or later is part of all distributions of LaTeX
% version 2008 or later.
%
% This file has the LPPL maintenance status "maintained".
%
% The list of all files belonging to the LaTeX base distribution is
% given in the file `manifest.txt'. See also `legal.txt' for additional
% information.
%
% The list of derived (unpacked) files belonging to the distribution
% and covered by LPPL is defined by the unpacking scripts (with
% extension .ins) which are part of the distribution.
%
% \fi
% Filename: ltnews28.tex
%
% This is issue 28 of LaTeX News.

\documentclass{ltnews}
\usepackage[T1]{fontenc}

\usepackage{lmodern,url,hologo}

\providecommand\acro[1]{\textsc{#1}}
\providecommand\meta[1]{$\langle$\textit{#1}$\rangle$}


\publicationmonth{April}
\publicationyear{2018}

\publicationissue{28}

\begin{document}

\maketitle
\tableofcontents

\setlength\rightskip{0pt plus 3em}

\section{A new home for \LaTeXe{} sources}

In the past the development version of the \LaTeXe{} source files has
been managed in a Subversion source control system with read access
for the public. This way it was possible to download in an emergency
the latest version even before it was released to CTAN and made its
way into the various distributions.

We have recently changed this setup and now manage the sources using
Git and placed the master sources on GitHub at
\begin{quote}
\url{https://github.com/latex3/latex2e}
\end{quote}
where we already store the sources
for \pkg{expl3} and other work. As before, direct write access is restricted
to \LaTeX{} Project Team members, but everything is publicly accessible
including the ability to download, clone (using Git) or checkout
(using SVN). More details are given in~\cite{Mittelbach:TB39-1}.

\section{Bug reports for core \LaTeXe{}}

For more than two decades we used GNATS, an open source bug tracking
system developed by the FSF. While that has served us well in the past
it started to show its age more and more. So as part of this move we
also decided to finally retire the old \LaTeX{} bug database and replace
it with the standard ``Issue Tracker'' available at Github.

The requirements and the workflow for reporting a bug in the core
\LaTeX{} software is documented at
\begin{quote}
\url{https://www.latex-project.org/bugs/}
\end{quote}
and with further details also discussed in~\cite{Mittelbach:TB39-1}.


\section{UTF-8: the new default input encoding}

The first \TeX{} implementations only supported reading 7-bit
\acro{ascii} files---any accented or otherwise ``special'' character
had to be entered using commands, if it could be represented at
all. For example to obtain an ``\"a'' one would enter \verb=\"a=, and to
typeset a ``\ss'' the command \verb=\ss=. Furthermore fonts at that
time had 128 glyphs inside, holding the \acro{ascii} characters, some
accents to build composite glyphs from a letter and an accent, and a
few special symbols such as parentheses, etc.

With 8-bit \TeX{} engines such as \hologo{pdfTeX} this situation changed
somewhat: it was now possible to process 8-bit files, i.e., files that
could encode 256 different characters. However, 256 is still a fairly
small number and with this limitation it is only possible to encode a
few languages and for other languages one would need to change the
encoding (i.e., interpret the character positions 0--255 in a
different way). The first code points 0--127 were essentially normed
(corresponding to \acro{ascii}) while the second half 128--255 would
vary by holding different accented characters to support a certain set
of languages.

Each computer used one of these encodings when storing or interpreting
files and as long as two computers used the same encoding it was
(easily) possible to exchange files between them and have them
interpreted and processed correctly.

But different computers may have used different encodings and given
that a computer file is simply a sequence of bytes with no indication for
which encoding is intended, chaos could easily happen and has
happened. For example, the German word ``Gr\"o\ss e'' (height) entered on a
German keyboard could show up as ``Gr\v T\`ae'' on a different
computer using a different encoding by default.

So in summmary the situation wasn't at all good and it was clear in
the early nineties that \LaTeXe{} (that was being developed to provide
a \LaTeX{} version usable across the world) had to provide a solution
to this issue.

The \LaTeXe{} answer was the introduction of the \package{inputenc}
package~\cite{Mittelbach:Brno95} through which it is possible to
provide support for multiple encodings. It also allows to correctly
process a file written in one encoding on a computer using a different
encoding and even supports documents where the encoding changes
midway.

Since the first release of \LaTeXe{} in 1994, \LaTeX{} documents that
used any characters outside \acro{ascii} in the source (i.e. any
characters in the range of 128--255) were supposed to load
\package{inputenc} and specify in which file encoding they were
written and stored.
%
If the \package{inputenc} package was not loaded then \LaTeX{} used a
``raw'' encoding which essentially took each byte from the input file
and typeset the glyph that happened to be in that position in the
current font---something that sometimes produces the right result but
often enough will not.

In 1992 Ken Thompson and Rob Pike developed the UTF-8 encoding scheme
which enables the encoding of all Unicode characters within 8-bit sequences.
Over time this encoding has gradually taken over the world,
replacing the legacy 8-bit encodings used before. These days all major
computer operating systems use UTF-8 to store their files and it
requires some effort to explicitly store files in one of the legacy
encodings.

As a result, whenever \LaTeX{} users want to use any accented
characters from their keyboard (instead of resorting to \verb=\"a= and
the like) they always have to use
\begin{verbatim}
  \usepackage[utf8]{inputenc}
\end{verbatim}
in the preamble of their documents as otherwise \LaTeX{} will produce
gibberish.

\subsection{The new default}

With this release, the default encoding for \LaTeX\ files has been
changed from the ``fall through raw'' encoding to UTF-8 if used with
classic \TeX\ or \hologo{pdfTeX}. The implementation is essentially
the same as the existing UTF-8 support from
\verb|\usepackage[utf8]{inputenc}|.

The \hologo{LuaTeX} and \hologo{XeTeX} engines always supported the
UTF-8 encoding as their native (and only) input encoding, so with
these engines \package{inputenc} was always a no-op.

This means that with new documents one can assume UTF-8 input and it
is no longer required to always specify
\verb|\usepackage[utf8]{inputenc}|. But if this line is present it
will not hurt either.


\subsection{Compatibility}

For most existing documents this change will be transparent:
\begin{itemize}
\item documents using only \acro{ascii} in the input file and
  accessing accented characters via commands;
\item documents that specified the encoding of their file via an
  option to the \package{inputenc} package and then used 8-bit
  characters in that encoding;
\item documents that already had been stored in UTF-8 (whether or not
  specifying this via \package{inputenc}).
\end{itemize}
Only documents that have been stored in a legacy encoding and used
accented letters from the keyboard \emph{without} loading
\package{inputenc} (relying on the similarities between the input used
and the T1 font encoding) are affected.

These documents will now generate an error that they contain invalid
UTF-8 sequences.  However, such documents may be easily processed by
adding the new command \verb|\UseRawInputEncoding| as the first line
of the file. This will re-instate the previous ``raw'' encoding
default.

\verb|\UseRawInputEncoding| may also be used on the command line to
process existing files without requiring the file to be edited
\begin{verbatim}
  pdflatex '\UseRawInputEncoding \input'  file
\end{verbatim}
will process the file using the previous default encoding.

Possible alternatives are reencoding the file to UTF-8 using a tool
(such as recode or iconv or an editor) or adding the line
\begin{flushleft}
\verb=  \usepackage[=\meta{encoding}\verb=]{inputenc}=
\end{flushleft}
to the preamble specifying the \meta{encoding} that fits the file
encoding.  In many cases this will be \texttt{latin1} or
\texttt{cp1252}. For other encoding names and their meaning see the
\package{inputenc} documentation.

As usual, this change may also be reverted via the more general
\package{latexrelease} package mechanism, by speciying a release date
earlier than this release.


\subsection{BOM: byte order mark handling}

When using Unicode the first bytes of a file may be a, so called, BOM
character (byte order mark) to indicate the byte oder used in the
file. While this is not required with UTF-8 encoded files (where the
byte order is known) it is nevertheless allowed by the standard and
some editors add that byte sequence to the beginning of a file. In the
past such files would have generated a ``Missing begin document''
error or displayed strange characters when loaded at a later stage.

With the addition of UTF-8 support to the kernel it is now possible to
identify and ignore such BOMs characters even before
\verb=\documentclass= so that these issues will no longer be showing
up.



\section[A general rollback concept]
        {A general rollback concept for packages and classes}

  In 2015 a rollback concept for the \LaTeX{} kernel was introduced.
  Providing this feature allowed us to make corrections to the
  software (which more or less didn't happen for nearly two decades)
  while continuing to maintain backward compatibility to the highest
  degree.

  In this release we have now extended this concept to
  the world of packages and classes which was not covered
  initially. As the classes and the extension packages have different
  requirements compared to the kernel, the approach is different (and
  simplified). This should make it easy for package developers to
  apply it to their packages and authors to use when necessary.

  The documentation of this new feature is given in an article
  submitted to TUGboat and also available from our
  website~\cite{Mittelbach:TB39-2}.


\section[Integration of \pkg{remreset} and \pkg{chngcntr} packages]
         {Integration of \pkg{remreset} and \pkg{chngcntr} packages
         into the kernel}

With the optional argument to \cs{newcounter} \LaTeX{} offers to
automatically reset counters when some counter is stepped, e.g.,
stepping a \texttt{chapter} counter resets the \texttt{section}
counter (and recursively all other heading counters). However, what
was until now missing was a way to undo such a link between counters
or to link two counters after they have been defined.

This can be now be done with \cs{counterwithin} and \cs{counterwithout},
respectively. In the past one had to load the \pkg{chngcntr} package
for this. For the programming level we also added
\cs{@removefromreset} as the counterpart of the already existing
\cs{@addtoreset} command. Up to now this was offered by the
\pkg{remreset} package.

\section{Testing for undefined commands}
\LaTeX\ packages often use a test \verb|\@ifundefined| to test if a command
is defined. Unfortunately this had the side effect of \emph{defining}
the command to \verb|\relax| in the case that it had no definition.
 The new release uses a modified definition
(using extra testing possibilities available in \hologo{eTeX}. The new definition
is more natural, however code that was relying on the side effect of the
command being tested being defined if it was previously undefined may have to add
\verb|\let\|\meta{command}\verb|\relax|.


\section{Changes to packages in the tools category}

\subsection{\LaTeX{} table columns with fixed widths}

Frank published a short paper in
TUGboat~\cite{Mittelbach:TB38-2-213} on producing tables that have
columns with fixed widths. The outlined approach using column
specifiers ``\texttt{w}'' and ``\texttt{W}'' has now been integrated
into the \pkg{array} package.

\subsection{Obscure overprinting with \pkg{multicol} fixed}

A rather peculiar bug was reported on StackExchange for
\pkg{multicol}. If the column/page breaking was fully controlled by
the user (through \cs{columnbreak}) instead of letting the environment
do its job and if then more \cs{columnbreak} commands showed up on the
last page then the balancing algorithm was thrown off track.
As a result some parts of the columns did overprint each other.

The fix required a redesign of the output routines used by
\pkg{multicol} and while it ``should'' be transparent in other cases
(and all tests in the regession test suite came out fine) there is the
off-chance that code that hooked into internals of \pkg{multicol}
needs adjustment.


\section{Changes to packages in the amsmath category}

With this release of \LaTeX{} a few minor issues with \pkg{amsmath}
have been corrected.

\subsection{Updated user's guide}

Furthermore, \texttt{amsldoc.pdf}, the AMS user's
guide for the \pkg{amsmath} package~\cite{amsldoc}, has been updated
from version~2.0 to~2.1 to incorporate changes and corrections made
between 2016 and 2018.


\begin{thebibliography}{9}

\bibitem{Mittelbach:TB39-1} Frank Mittelbach:
  \emph{New rules for reporting bugs in the \LaTeX{} core software}.
  In: TUGboat, 39\#1, 2018.
  \url{https://www.latex-project.org/publications/}

\bibitem{Mittelbach:Brno95} Frank Mittelbach:
  \emph{\LaTeXe{} Encoding Interface --- Purpose, concepts, and
   Open Problems}.
  Talk given in Brno June 1995.
  \url{https://www.latex-project.org/publications/}

\bibitem{Mittelbach:TB39-2} Frank Mittelbach:
  \emph{A rollback concept for packages and classes}.
  Submitted to TUGboat.
  \url{https://www.latex-project.org/publications/}

\bibitem{Mittelbach:TB38-2-213} Frank Mittelbach:
  \emph{\LaTeX{} table columns with fixed widths}.
  In: TUGboat, 38\#2, 2017.
  \url{https://www.latex-project.org/publications/}

\bibitem{amsldoc} American Mathematical Society and The \LaTeX3 Project:
  \emph{User's Guide for the \texttt{amsmath} package} (Version 2.1).
  April 2018.
  Available from
  \url{https://www.ctan.org}
  and distributed as part of every \LaTeX{} distribution.

\end{thebibliography}

\end{document}