%\pdfoutput=1% Do not remove, needed for arxiv \documentclass[a4paper,doc2]{ltxdoc} % doc2 is needed to force the old version, or links get colored in a weird red way even with hidelinks. https://github.com/latex3/latex2e/issues/822 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%% Packages %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Warning: if you compile and get: %% ERROR: Argument of \tikz@lib@matrix@with@options has an extra }. %% make sure to fix catcodes around it as | is given a different meaning in ltxdoc. \usepackage{amsmath} \usepackage[margin=3cm]{geometry} \usepackage{calc} \usepackage{siunitx} \usepackage{tikz} \usepackage{tikzit} \input{new_zxstyle.tikzstyles} \usetikzlibrary{shadows,fit} % \usetikzlibrary fails because file is not in current directory, lazy to setup TEXINPUTS \makeatletter \usepackage{robust-externalize} \makeatother \robExtConfigure{ enable optimizations, copy file to cache={zx-calculus.sty}, copy file to cache={tikzlibraryzx-calculus.code.tex}, % You need to uncomment that line to rename the .tex files into .tex-backup % backup source for arxiv, rename backup files for arxiv, print source when saving, % debug } \usepackage{zx-calculus} \usepackage{forest} \setlength{\marginparwidth}{2cm} \usepackage{todonotes} % Loads the great package that produces tikz-like manual (see also tikzcd for examples) \input{pgfmanual-en-macros.tex} % Is supposed to be included in recent TeX distributions, but I get errors... \usepackage{gnuplot-lua-tikz} %% Generate with gnuplot -e "set terminal tikz createstyle" %% For verbatim environments: \NewDocumentEnvironment{codeAndResult}{}{\XSIMfilewritestart{\jobname-codeAndResult-tmp-file-you-can-remove.tmp}}% {% \XSIMfilewritestop% \par \medskip \noindent\colorbox{graphicbackground}{\noindent\begin{minipage}{1.0\linewidth} {\input{\jobname-codeAndResult-tmp-file-you-can-remove.tmp}} \end{minipage}} % I don't know why, but sometimes this prints nothing, while input works: %{\codeexample[width=0pt, leave comments, vbox, from file={\jobname-codeAndResult-tmp-file-you-can-remove.tmp}]}% {\codeexample[code only, from file={\jobname-codeAndResult-tmp-file-you-can-remove.tmp}]}% } \usepackage{makeidx} % Produces an index of commands. \makeindex % Useful or not index will be created \usepackage{alertmessage} % For warning, info... \newcommand{\mylink}[2]{\href{#1}{#2}\footnote{\url{#1}}} \usepackage{verbatim} \usepackage{mathtools} \usepackage{float} %figure inside minipage \usepackage{pythonhighlight} \usepackage{tcolorbox} \usepackage{listings} \definecolor{codegreen}{rgb}{0,0.6,0} \definecolor{codegray}{rgb}{0.5,0.5,0.5} \definecolor{codepurple}{rgb}{0.58,0,0.82} \definecolor{backcolour}{rgb}{0.95,0.95,0.92} \lstdefinestyle{mystyle}{ backgroundcolor=\color{backcolour}, commentstyle=\color{codegreen}, keywordstyle=\color{magenta}, numberstyle=\tiny\color{codegray}, stringstyle=\color{codepurple}, basicstyle=\ttfamily\footnotesize, breakatwhitespace=false, breaklines=true, captionpos=b, keepspaces=true, numbers=left, numbersep=5pt, showspaces=false, showstringspaces=false, showtabs=false, tabsize=2 } \lstset{style=mystyle} \usepackage[hidelinks]{hyperref} \usepackage{cleveref} \begin{document} %%% Title: thanks tikzcd for the styling \begin{center} \vspace*{1em} % Thanks tikzcd \tikz\node[scale=1.2]{% \color{gray}\Huge\ttfamily \char`\{\raisebox{.09em}{\textcolor{red!75!black}{robust\raisebox{-0.1em}{-}externalize}}\char`\}}; \vspace{0.5em} {\Large\bfseries Cache anything (\tikzname, tikz-cd, python…),\\in a robust, efficient and pure way.} \vspace{1em} {Léo Colisson \quad Version 2.9}\\[3mm] {\href{https://github.com/leo-colisson/robust-externalize}{\texttt{github.com/leo-colisson/robust-externalize}}} \end{center} \tableofcontents \bigskip \textbf{WARNING: Even if we try to stay backward compatible, the only guaranteed way to be 100\% immune to changes is to copy/paste the\ .sty file of the library in your main project folder. Please report any bug to \url{https://github.com/leo-colisson/robust-externalize}!} \textbf{WARNING 2: the recent versions 2.1 and 2.2 improves significantly the compilation time compared with 2.0 that was surprisingly slow. Now, it is fairly well optimized, and version 2.3 is even more optimized.} \textbf{WARNING 3: If you read this documentation from the arXiv or CTAN, you might prefer to read it from \mylink{https://github.com/leo-colisson/robust-externalize}{Github} to get the latest version.} \section{A taste of this library} This library allows you to cache any language: not only \LaTeX{} documents and \tikzname{} images, taking into account depth and overlays, but also arbitrary code. To use it, make sure to have v2.0 installed (see instructions below), and load: \begin{verbatim} \usepackage{robust-externalize} \robExtConfigure{ % To display instructions in the PDF to manually compile pictures if % you forgot/do not want to compile with -shell-escape enable fallback to manual mode, } \end{verbatim} Then type something like this (note the |C| for |cached| at the end of |tikzpictureC|, see below to override the original tikz commands), and compile with |pdflatex -shell-escape yourfile.tex| (or \textcolor{red}{\textbf{read below if you do not want to use }}|-shell-escape|\textcolor{red}{\textbf{ (note that overleaf enables shell-escape by default, and it is not needed once all pictures are cached)}}): \begin{codeexample}[width=0pt,vbox] \robExtConfigure{ add to preset={tikz}{ % we load some packages that will be loaded by figures based on the tikz preset add to preamble={\usepackage{pifont}} } } The next picture is cached % \begin{tikzpictureC}[baseline=(A.base)] \node[fill=red, rounded corners](A){My node respecting baseline \ding{164}.}; \node[fill=red, rounded corners, opacity=.3,overlay] at (A.north east){I am an overlay text}; \end{tikzpictureC} and you can see that overlay and depth works. \end{codeexample} Since v2.3 we also provide a way to include online pictures with |\includegraphicsWeb|: \begin{codeexample}[width=0pt,vbox] Here is a picture of cat downloaded online: \includegraphicsWeb[width=3cm]{http://placekitten.com/400/300}. \end{codeexample} You can also cache arbitrary code (e.g.\ python). You can also define arbitrary compilation commands, inclusion commands, and presets to fit you need. For instance, you can create a preset to obtain: \begin{codeAndResult} \begin{CacheMeCode}{python print code and result, set title={The for loop}} for name in ["Alice", "Bob"]: print(f"Hello {name}") \end{CacheMeCode} \end{codeAndResult} Actually, we also provide this style by default (and explain how to write it yourself), just use the above code (WARNING: in v2.7, a bug was introduced, fixed in v2.8, that was adding a new line in front of the code), just make sure to load: \begin{verbatim} \usepackage{pythonhighlight} \usepackage[most]{tcolorbox} \end{verbatim} You can also cache any environment and command using something like: \begin{itemize} \item For environments: \begin{codeAndResult} % We cache the environment forest \cacheEnvironment{forest}{latex, add to preamble={\usepackage{forest}}} A forest tree automatically cached % \begin{forest} [VP [DP[John]] [V’ [V[sent]] [DP[Mary]] [DP[D[a]][NP[letter]]] ] ] \end{forest}\\ And one not cached: % \begin{forest} [VP [DP[John]] [V’ [V[sent]] [DP[Mary]] [DP[D[a]][NP[letter]]] ] ] \end{forest} \end{codeAndResult} People interested by |tikz-cd| might like this example: \begin{codeAndResult} \cacheEnvironment{tikzcd}{tikz, add to preamble={\usepackage{tikz-cd}}} \begin{tikzcd} A \rar & B \end{tikzcd} \end{codeAndResult} \item For commands: \begin{codeAndResult} % We cache the command \zx % O{} = optional argument, m = mandatory argument. % Autodetected if command defined with xparse (NewDocumentCommand) % (in this example it is redundant since zx IS defined with xparse syntax) \cacheCommand{zx}[O{}O{}O{}m]{latex, add to preamble={\usepackage{zx-calculus}}} A ZX figure % \zx{ \zxX{\alpha} \rar \ar[d,C] & \zxZ*{a\pi} \\ \zxZ{\beta} \rar & \zxX*{b\pi} }, we can disable externalization: % \zx{ \zxX{\alpha} \rar \ar[d,C] & \zxZ*{a\pi} \\ \zxZ{\beta} \rar & \zxX*{b\pi} } or add more options: % \zx{ \zxX{\alpha} \rar \ar[d,C] & \zxZ{\text{\hello{Alice}}} \\ \zxZ{\beta} \rar & \zxX*{b\pi} } \end{codeAndResult} \end{itemize} We also provide methods to cache automatically \mylink{https://tikzit.github.io/}{TikZit} pictures (|\cacheCommand| alone would not be sufficient as the source relies on an external file): \begin{codeexample}[vbox] %%%%%%% In your preamble: \cacheTikzitWithStyle{new_zxstyle.tikzstyles} %%%%%%% In your document: \tikzfig{demo} %figures/ can be omitted \end{codeexample} You can also cache all tikz pictures by default, except those that are run from some commands (as remember picture does not work yet, this is important). For instance, this code will allow you to freely use the |todonotes| package (that uses internally tikz but is not cachable with this library since it uses remember picture) while caching all other elements: \begin{codeexample}[code only] %% this will cache \tikz and tikzpictures by default (using the tikz preset) \cacheTikz %% possibly add stuff to the tikz preset \robExtConfigure{ add to preset={tikz}{ add to preamble={ \def\whereIAm{(I am cached)} } }, }% %% We say not to cache the todo commands: \cacheCommand{todo}[O{}m]{disable externalization}% \todo{Check how to use cacheCommand and cacheEnvironment} \tikz[baseline=(A.base)] \node(A)[rounded corners,fill=green]{my cached node \whereIAm}; \end{codeexample} will give you: {% %% this will cache \tikz and tikzpictures by default (using the tikz preset) \cacheTikz %% possibly add stuff to the tikz preset \robExtConfigure{ add to preset={tikz}{ add to preamble={ \def\whereIAm{(I am cached)} } }, }% %% We say not to cache the todo commands: \cacheCommand{todo}[O{}m]{disable externalization}% \todo{Check how to use cacheCommand and cacheEnvironment} \tikz[baseline=(A.base)] \node(A)[rounded corners,fill=green]{my cached node \whereIAm}; } Note that most options can either be specified anywhere in the document like \begin{verbatim} \robExtConfigure{ disable externalization } \end{verbatim} , inside a preset (that is nothing more than a pgf style), or in the options of the cached environment (either inside |CacheMe| or inside the first parameter |<...>| for automatically cached environments): \begin{codeexample}[vbox] \robExtConfigure{ new preset={my preset with xcolor}{ latex, % load the latex preset, you can also use the tikz preset and more add to preamble={ % load xcolor in the preamble of the cached document \usepackage{xcolor} }, }, } \begin{CacheMe}{my preset with xcolor, add to preamble={\def\hello#1{Hello #1!}} } \begin{minipage}{5cm} \textcolor{red}{\hello{Alice} See, you can cache any data, including minipages, define macros, combine presets, override a single picture or a whole part of a document etc.} \end{minipage} \end{CacheMe} \end{codeexample} (see that |CacheMe| can be used to cache arbitrary pictures) Or include images generated in python: \begin{codeexample}[code only] \begin{CacheMeCode}{python, set includegraphics options={width=.8\linewidth}} import matplotlib.pyplot as plt year = [2014, 2015, 2016, 2017, 2018, 2019] tutorial_count = [39, 117, 111, 110, 67, 29] plt.plot(year, tutorial_count, color="#6c3376", linewidth=3) plt.xlabel('Year') plt.ylabel('Number of futurestud.io Tutorials') plt.savefig("__ROBEXT_OUTPUT_PDF__") \end{CacheMeCode} \end{codeexample} You can also include files from the root folder, if you want, recompile all pictures if this files changes (nice to ensure maximum purity): \begin{codeexample}[width=0pt,vbox] \begin{CacheMe}{latex, add dependencies={common_inputs.tex}, add to preamble={\input{__ROBEXT_WAY_BACK__common_inputs.tex}}} The answer is \myValueDefinedInCommonInputs. \end{CacheMe} \end{codeexample} Since v2.1, we also provide easy ways to export counters: \begin{codeexample}[width=0pt] \begin{tikzpictureC} \node[rounded corners, fill=red]{The current page is \thepage.}; \end{tikzpictureC} \end{codeexample} or even macros (we can also export evaluated macros): \begin{codeexample}[width=0pt] \cacheTikz \newcommand{\MyNodeWithNewCommand}[2][draw,thick]{\node[rounded corners,fill=red,#1]{#2};} \begin{tikzpicture} \MyNodeWithNewCommand{Alice} \MyNodeWithNewCommand[xshift=2cm]{Bob} \end{tikzpicture} \end{codeexample} Since manually forwarding elements can be quite verbose, it is also possible to automatically apply some styles depending on the content of the element to cache. For instance, we provide a way to automatically forward some macros: \begin{codeexample}[vbox] \cacheTikz \robExtConfigure{add to preset={tikz}{auto forward}} \newcommandAutoForward{\MyNode}[2][draw,thick]{\node[rounded corners,fill=red,#1]{#2};} \newcommandAutoForward{\MyGreenNode}[2][draw,thick]{\node[rounded corners,fill=green,#1]{#2};} \begin{tikzpicture} \MyNode{Recompiled only if MyNode is changed} \MyNode[xshift=8cm]{but not if the (unused) MyGreenNode is changed.} \end{tikzpicture}\\ \begin{tikzpicture} \MyGreenNode{Recompiled only if MyGreenNode is changed} \MyGreenNode[xshift=8cm]{but not if the (unused) MyNode is changed.} \end{tikzpicture} \end{codeexample} We also provide a number of other functionalities, among others: \begin{itemize} \item To achieve a pleasant and configurable interface (e.g. pass options to a preset with default values), we introduced placeholders, that can be of independent interest. \item To get maximum purity while minimizing the number of rebuild, we provide functions to forward macros defined in the parent document, and the set of macros that must be forwarded can be automatically detected for each element to cache. It is also possible to conditionally import a library if the element contains a given macro. \item We provide a way to compile a template via:\\ |new compiled preset={compiled ZX}{templateZX, compile latex template}{}|\\ in order to make the compilation even faster (but you cannot use placeholders except |add to preamble|) \item We give a highly configurable template, compilation and inclusion system, to cache basically anything \item We show how to compile all images in parallel using |\robExtConfigure{compile in parallel}| or |\robExtConfigure{compile in parallel after=5}| (make sure to have xargs installed, which is available by default on linux but must be installed on Windows by installing the lightweight GNU On Windows (Gow) \url{https://github.com/bmatzelle/gow} if you use the default command) even during the first run, so that the first run becomes faster to compile than a normal run(!) \item We furnish commands to automatically clean the cache while keeping cached elements that are still needed. \item And more\dots \end{itemize} \section{Acknowledgments} I am deeply indebted to many users on \url{tex.stackexchange.com} that made the writing of this library possible. I can't list you all, but thank you so much! A big thanks also to the project \url{https://github.com/sasozivanovic/memoize/} from which I borrowed most of the code to automatically wrap commands. Thanks to kiryph and dflvunoooooo for providing very useful feedbacks. \section{Support} If you find this library useful, please consider citing the arXiv version of this documentation, possibly using |\nocite|. If you want to support me financially and offer me a virtual coffee, you can use my \mylink{https://github.com/sponsors/tobiasBora}{GitHub sponsor page}. \section{Introduction} \subsection{Why do I need to cache (a.k.a. externalize) parts of my document?} One often wants to cache (i.e.\ store pre-compiled parts of the document, like figures) operations that are long to do: For instance, TikZ is great, but TikZ figures often take time to compile (it can easily take a few seconds per picture). This can become really annoying with documents containing many pictures, as the compilation can take multiple minutes: for instance my thesis needed roughly 30mn to compile as it contains many tiny figures, and LaTeX needs to compile the document multiple times before converging to the final result. But even on much smaller documents you can easily reach a few minutes of compilation, which is not only high to get a useful feedback in real time, but worse, when using online \LaTeX{} providers (e.g. overleaf), this can be a real pain as you are unable to process your document due to timeouts. Similarly, you might want to cache the result of some codes, for instance a text or an image generated via python and matplotlib, without manually compiling them externally. \subsection{Why not using \tikzname{}'s externalize library?} \tikzname{} has an externalize library to pre-compile these images on the first run. Even if this library is quite simple to use, it has multiple issues: \begin{itemize} \item If you add a picture before existing pre-compiled pictures, the pictures that are placed after will be recompiled from scratch. This can be mitigated by manually adding a different prefix to each picture, but it is highly not practical to use. \item To compile each picture, TikZ's externalize library reads the document's preamble and needs to process (quickly) the whole document. In large documents (or in documents relying on many packages), this can result in a significant loading time, sometimes much bigger than the time to compile the document without the externalize library: for instance, if the document takes 10 seconds to be processed, and if you have 200 pictures that take 1s each to be compiled, the first compilation with the TikZ's externalize library will take roughly half an hour (200 $\times$ 10s = 33mn) instead of 3mn without the library. And if you add a single picture at the beginning of the document… you need to restart everything from scratch. For these reasons, I was not even able to compile my thesis with TikZ's external library in a reasonable time. \item If two pictures share the same code, it will be compiled twice \item Little purity is enforced: if a macro changes before a pre-compiled picture that uses this macro, the figure will not be updated. This can result in different documents depending on whether the cache is cleared or not. \item As far as I know, it is made for TikZ picture mostly, and is not really made for inserting other stuff, like matplotlib images generated from python etc... \item According to some maintainers of TikZ, ``\mylink{https://github.com/pgf-tikz/pgf/issues/758}{the code of the externalization library is mostly unreadable gibberish}'', and therefore most of the above issues are unlikely to be solved in a foreseable future. \end{itemize} \subsection{FAQ} \paragraph{What is supported?} You can cache most things, including tikz pictures, (including ones with overlays (but not with remember picture), with depth etc.), python code etc. We tried to make the library as customizable as possible to be useful in most scenarios. You can also send some data (e.g.\ : the current page) to the compiled pictures, and feed some data back to the main document (say that you want to compute a value that takes time to compute, or compute the number of pages of the produced document in order to increase the number of pages accordingly\dots{}). Since v2.0, you can also cache automatically any environment. \paragraph{What is not supported?} We do not yet support remember picture, and you can't use (yet) cross-references inside your images (at least not without further hacks) or links as links are stripped when the pdf is included. I might have some ideas to solve this\footnote{\url{https://tex.stackexchange.com/questions/695277/clickable-includegraphics-for-cross-reference-data}}, meanwhile you can just disable locally the library on problematic figures. Note that this library is quite young, so expect untested things. \paragraph{What OS are supported?} I tested the library on Linux and Windows, but the library should work on all OS, including MacOs. Please let me know if it fails for you. \paragraph{Do I need to compile using -shell-escape?} Since we need to compile the images via an external command the first time, the simpler option is to add the argument |-shell-escape| to let the library run the compilation command automatically (this is also the case of \tikzname's externalize library). Note that overleaf already defines this option by default, so you don't need to do anything special in overleaf. Moreover, this is only needed \textbf{during the first compilation} since once the image is cached, |-shell-escape| is not needed anymore (therefore, on the arXiv website, you can upload the |robustExternalize| folder containing the cached images without any issue, see \cref{sec:arxiv} for details). However, people worried by security issues of |-shell-escape| (that allows arbitrary code execution if you don't trust the \LaTeX{} code) might be interested by these facts: \begin{itemize} \item You can choose to display a dummy content explaining how to manually compile the pictures instead of giving an error until they are compiled, see |enable fallback to manual mode|. \item You can compile manually the images: all the commands that are left to be executed are also listed in \texttt{JOBNAME-robExt-compile-missing-figures.sh} and you can just inspect and run them, either with \texttt{bash robExt-compile-missing-figures.sh} or by typing them manually (most of the time it's only a matter of running \texttt{pdflatex somefile.tex} from the \texttt{robustExternalize} folder). \item If you allowed: \begin{itemize} \item |pdflatex| (needed to compile latex documents) \item |cd| (not needed when using |no cache folder|) \item and |mkdir| (not needed when using |no cache folder| or if the cache folder that defaults to |robustExternalize| is already created) \end{itemize} to run in restricted mode (so without enabling |-shell-escape|), then there is no need for |-shell-escape|. In that case, set |force compilation| and this library will compile even if |-shell-escape| is disabled. \item If images are all already cached, you don't need to enable \texttt{-shell-escape} (this might be interesting e.g. to send a pre-cached document to the arxiv or to a publisher: just make sure to include the cache folder). \item We use very few commands to compile latex files, basically only pdflatex, mkdir (to create the cache folder if needed) and cd (if the cache folder is not present). You might want to allow them in restricted mode. \end{itemize} \paragraph{Is it working on overleaf?} Yes: overleaf automatically compiles documents with |-shell-escape|, so nothing special needs to be done there (of course, if you use this library to run some code, the programming language might not be available, but I heard that python is installed on overleaf servers for instance, even if this needs to be doubled checked). If the first compilation of the document to cache images times out, you can just repeat this operation multiple times until all images are cached. In case of doubt, you can look at the log file to see which file is getting cached. If overleaf still times out, read the next paragraph. \paragraph{Overleaf still times-out, why?} During the first compilation, the pictures are not compiled, so you may obtain a time out. However, at each compilation, new pictures should be added to the cache, so you should eventually be able to compile (check the log file, you should read ``No need to recompile XX.pdf'' if the picture is already in the cache). If you want this to be faster, you can try to enable |\robExtConfigure{compile in parallel}| in order to compile the pictures in parallel (need 2 compilations). If you do not often change all pictures, you can also locally download the project (it might be more practical to use the git bridge if you have it enabled), compile them locally, and copy back the |robustExternalize| folder containing all cached pictures. If all pictures are still compiled, but you still experience timeouts, you can try to use compiled presets, with something like: \begin{verbatim} \robExtConfigure{ new compiled preset={compiled ZX}{templateZX, compile latex template}{} } \end{verbatim} to create a new, more efficient (we call it compiled) preset called |compiled ZX| from a preset |templateZX|. You can also try to reduce the compilation time by reducing the number of compilation passes. Indeed, overleaf will run |pdflatex| multiple times to make it converge to the final version, increasing mechanically the compilation time (for instance a file that takes 6s to compile locally was not able to compile online). You can force overleaf to compile a single time the document by creating a |latexmkrc| file at the root of the project, containing: \begin{verbatim} $max_repeat=1 \end{verbatim} Of course, you might need to recompile more than once the document to reach the final version. \paragraph{Do you have some benchmarks?} We completely rewrote the \href{https://github.com/leo-colisson/robust-externalize/commit/09a05a86ec9c04eb6687b75831ebb84185786ac5}{original library} to introduce the notion of placeholders (before, the template was fixed forever), allowing for greater flexibility. The original rewrite (1.0) had really poor performance compared to the original library (only x3 improvement compared to no externalization, while the old library could be 20x faster), but we made some changes (v2.0) that correct this issue to recover the original speed (at least when using compiled templates). Here is a benchmark we made on an article with 159 small pictures (ZX diagrams from the zx-calculus library): \begin{itemize} \item without externalization: 1mn25 \item with externalization, v2.0 (first run, not in parallel): 3mn05, so takes twice the time; but (first run in parallel) takes 60s (so faster by a factor of 1.5). \item with externalization, v2.0 (next runs): 5.77s (it is 15x times faster) \item with externalization, v2.0, with a ``compiled'' template (less flexible) like in the old version: 4.0s (22x faster) \item with externalization, old library (next runs): 4.16s (so 21x faster) \end{itemize} \paragraph{Can I compile pictures in parallel to speed up notably the first run?} The first run might be particularly long to compile since many elements are still not cached and re-starting \LaTeX{} on each picture can take quite some time. If you have v2.1 or above, and if you have xargs installed (installed by default on most linux, on Windows you need to install the lightweight GNU On Windows (Gow) \url{https://github.com/bmatzelle/gow}), you can simply compile all images in parallel by typing at the beginning of the file: \begin{codeexample}[code only] \robExtConfigure{ compile in parallel } \end{codeexample} In my benchmark, I observed a compilation 1.5x faster than a normal compilation without any caching involved. We provide more options in \cref{sec:compileParallel}. \paragraph{Can I use version-control to keep the cached files in my repository?} Sure, each cached figure is stored in a few files (typically one pdf and one \LaTeX{} file, plus the source) having the same prefix (the hash), avoiding collision between runs. Just commit these files and you are good to go. \paragraph{Can I deal with baseline position and overlays ?} Yes, the depth of the box is automatically computed and used to include the figure by default, and additional margin is added to the image (30cm by default) to allow overlay text. \paragraph{How is purity enforced?} Purity is the property that if you remove the cached files and recompile your document, you should end-up with the same output. To enforce purity, we compute the hash of the final program, including the compilation command, the dependency files used for instance in |\input{include.tex}| (unless you prefer not to, for instance to keep parts of the process impure for efficiency reasons), the forwarded macros, and put the code in a file named based on this hash. Then we compile it if it has not been used before, and include the output. Changing a single character in the file, the tracked dependencies, or the compilation command will lead to a new hash, and therefore to a new generated picture. \paragraph{What if I don't want purity for all files?} If you do not want your files to be recompiled if you modify a given file, then just do not add this file to the list of dependencies. \paragraph{Can I extend it easily?} We tried to take a quite modular approach in order to allow easy extensions. Internally, to support a new cache scheme, we only expect a string containing the program (possibly produced using a template), a list of dependencies, a command to compile this program (e.g. producing a pdf and possibly a tex file with the properties (depth…) of the pdf), and a command to load the result of the compilation into the final document (called after loading the previously mentioned optional tex file). Thanks to pgfkeys, it is then possible to create simple pre-made settings to automatically apply when needed. \paragraph{How does it compare with \url{https://github.com/sasozivanovic/memoize}?} I recently became aware of the great \url{https://github.com/sasozivanovic/memoize}. While we aim to solve a similar goal, our approaches are quite different. While I focus on purity, and therefore create a different file for each picture, the above project puts all pictures in a single file and compile them all at once to avoid losing time to run the latex command for each picture (this mostly makes a difference for the first compilation). Our understanding of the main differences is the following: Pros of \url{https://github.com/sasozivanovic/memoize}: \begin{itemize} \item The above library might be easier to setup as they do not need to specify the preamble of the file to compile (the preamble of the main file is used), and tikz pictures are automatically memoized (we provide |\cacheTikz| for that, but you still need to specify the preamble once). \item This library allows to compile multiple pictures while loading \LaTeX{} only once, saving the time to start \LaTeX{} for each picture. However, it is not clear that it brings a faster compilation time compared to our library for multiple reasons: \begin{itemize} \item While during the first run all pictures are compiled with a single \LaTeX{} instance\footnote{We could also do something similar using some scripting to merge multiple cached files together, but this is not yet implemented.} (and inserted at the beginning of the document), they still need to extract each picture in a separate pdf file to include it at the right place in the document. This is done by default using |pdflatex|, which means that you need to call |pdflatex| once per picture, mitigating the advantage of compiling all pictures at once. Note that they also provide a python script to separate the pictures more efficiently, but this needs further manual interventions. \item We provide an option |compile in parallel| to compile all images in parallel, and the time is in my benchmark 1.5x faster than a normal compilation without any caching. Using this option, we should therefore compile faster than |memoized| that is not able to run jobs in parallel. \end{itemize} \item Even if it does not officially support contexts, context might be automatically transmitted (while we need to do a bit of work to manually or semi-automatically define the context needed to compile a picture). However, this can break purity (see below). \end{itemize} Cons of \url{https://github.com/sasozivanovic/memoize}: \begin{itemize} \item The purity is not enforced so strongly since all images are in the same file. Notably, the hash only depends on the picture, but not on its context. So for instance if you define |\def\mycolor{blue}| before the picture, and use |\mycolor| inside the picture, if you change later the color to, say, |\def\mycolor{red}|, the picture will not be recompiled (so cleaning the cache and recompiling would produce a different result). In our case, the purity is always strictly enforced (unless you choose not to) since the context must be passed (explicitly or semi-automatically) to the compiled file. \item As a result, the above library has poor support for contexts (in our case, you can easily, for instance, make a picture depend on the current page, and recompile the picture only if the current page changes: you can also do it the other way, and change some counters, say, depending on the cached file). \item The above library only focuses on \LaTeX{} while our library works for any language. \item The above library can only cache in pdf format, while we can generate any format (text, tex, jpg\dots{} and even videos that I include in my beamer presentations\dots{} but it is another topic!). \item We have an arguably more complete documentation. \end{itemize} Note that |remember picture| is not supported in both libraries, but we can provide multiple methods to disable externalizations when |remember picture| is used. \subsection{How to use with arXiv}\label{sec:arxiv} You can completely use this library with the arXiv's website (this documentation is also compiled in arXiv), but you need to follow a few steps due to arXiv's particularities: \begin{itemize} \item First, arXiv does NOT allow |-shell-escape|. So you either need to disable externalization (not recommended since this will of course increase the compilation time and will not work when working with non-latex code like python or external images), or you need to compile locally the document to cache all images locally\footnote{Make sure to recompile from scratch, by removing the .aux file, if your cached pictures can depend on counters (like the current page) that can change across compilations.}, and upload the |robustExternalize/| folder containing all cached images online in a zip file. \item Moreover, if you choose to upload the |robustExternalize/| folder, the arXiv website will automatically remove all |.pdf| files if a |.tex| file is present (which will break this library, since it puts all sources in the |.tex| file and the output in a |.pdf| file). The trick is to rename all the source |.tex| files into |.tex-backup| (this library will automatically recreate the |.tex| from the |.tex-backup| starting from v2.4). We provide two approaches for that: \begin{itemize} \item \textbf{Solution 1}: if you have python installed the most robust solution is to run \begin{verbatim} $ python robExt-prepare-for-arxiv.py \end{verbatim} before sending the document to the arXiv. This will rename all the |.tex| files in the |robust-externalize| folder that have a matching |*.pdf| file into |.tex-backup|, and list these files in |robExt-arxiv-files-to-rename.txt|. Then, add in the preamble of your document: \begin{codeexample}[code only] \robExtConfigure{ rename backup files for arxiv, } \end{codeexample} This will automatically rename back the |.tex-backup| files into |.tex| (without needing shell-escape), based on the |robExt-arxiv-files-to-rename.txt| file (if this file exists and if the files have not already been renamed: it is therefore safe to keep this option on even without running the python code), allowing the rest of the document to compile normally. \item \textbf{Solution 2}: if you do \textbf{not} have python installed, you can simply run: \begin{codeexample}[code only] \robExtConfigure{ backup source for arxiv, } \end{codeexample} This will automatically rename the source files |.tex| into |.tex-backup| (make sure to comment this line on the final pdf sent to the arXiv). Note however that this is less robust than solution 1 since the |.tex-backup| will be renamed into |.tex| only if it is in the source, but sometimes other |.tex| files might need to be renamed as well: for instance, |gnuplot| with |cairolatex| terminal creates two files |-gnuplot-.tex| and |-gnuplot-.pdf|, and this file will not be renamed with this method, so the |-gnuplot-.pdf| file will be lost. \end{itemize} \item This should be enough to compile most documents. But note that since the arXiv uses a different Texlive version and might try to compile other document formats, it might be the case that the number of page varies from your original document: if you uses things like |forward counter=page|, it might cause you some troubles since your local cache might contain the cache for |page=42| while the arXiv would ask for the cached value for |page=45|, producing an error like |enable -shell-escape|. There are a few steps you can follow to fix these issues: \begin{itemize} \item First, make sure to have the same margins/paper on your local file and online (by default arXiv will create a document in |letterpaper|). To that end, you can force the paper format with |\documentclass[a4paper]{article}|. \item ArXiv might try to compile with various output formats like eps, so it might help to force a pdf output by starting your document with |\pdfoutput=1| (not sure how helpful this is, it might already be forced when we insert a pdf?). \item ArXiv will automatically remove all empty files. While for some files it is not an issue (aux files and compilation logs), if this file is the |.pdf| it will be an issue. So make sure to always generate non-empty output files. \item If despite these options, arXiv still tries to fetch a cached value that is not present in your local cache, you can enable \begin{verbatim} \robExtConfigure{print source when saving} \end{verbatim} This will print in the log file the content of the |.deps| file, of the |.tex| file, and the compilation command that would have been run without the error. You can then recreate these files in the cache and compile it locally. If you do not want to copy/paste the content of the file, it might be simpler to simply see the value that differs from your own local cache and just temporarily change this value in your local document to automatically generate the correct cache entry. If the value is obtained via |forward counter=page|, you can use |forward counter force value={page}{45}| to force the value of the counter. \end{itemize} \end{itemize} \section{Tutorial} \subsection{Installation} To install the library, just copy the |robust-externalize.sty| file into the root of the project. Then, load the library using:\\ \begin{verbatim} \usepackage{robust-externalize} \end{verbatim} If you want to display a message in the pdf on how to manually compile the file if |-shell-escape| is disabled, you can also load this configuration (only available on v2.0): \begin{verbatim} \robExtConfigure{ enable fallback to manual mode, } \end{verbatim} \noindent (otherwise, it will give you an error if |-shell-escape| is disabled and if some pictures are not yet cached) If you forget/do not want to enable |-shell-escape|, this will give you this kind of message with instructions to follow to compile without shell escape: \noindent { \robExtConfigure{ enable manual mode } \begin{tikzpictureC}[baseline=(A.base)] \node[fill=red, rounded corners](A){I must be manually compiled.}; \node[fill=red, rounded corners, opacity=.3,overlay] at (A.north east){I am an overlay text}; \end{tikzpictureC} } If you use an editor, you can usually configure |-shell-escape| directly inside the editor, even on a per-document basis. Shell-escape is enabled by default on overleaf. In emacs/auctex, just add at the end of your document: \begin{verbatim} % Local Variables: % TeX-command-extra-options: "--shell-escape" % End: \end{verbatim} save, and run |M-x revert-buffer| (Note that it seems that |--shell-escape| can be used with one or two dashes on Linux, I am not sure about the number of dashes needed in Windows.) For TeXstudio, you can enable shell-escape on the current document by adding in your document: \begin{verbatim} % !TeX TXS-program:compile = txs:///pdflatex/[--shell-escape] \end{verbatim} and you might also want to enable: \begin{verbatim} \robExtConfigure{textstudio} \end{verbatim} to get clearer error messages as TeXstudio is \mylink{https://github.com/texstudio-org/texstudio/issues/1410}{fairly bad at parsing error messages} (see also |print whole file in error message|). If you use |latexmk| (with any editor), you can create a file called |.latexmkrc| containing: \begin{verbatim} $pdf_mode = 1; $pdflatex = 'pdflatex --shell-escape'; \end{verbatim} For other editors, \mylink{https://tex.stackexchange.com/questions/598818/how-can-i-enable-shell-escape}{this answer} describe pretty accurately the solutions you have depending on your software. \subsection{Caching a tikz picture} If you only care about \tikzname's picture, you have 3 options: \begin{enumerate} \item Call once |\cacheTikz| that will redefine |tikzpicture| and |\tikz| to use our library (if you use this solution, make sure to read how to disable externalization (\cref{sec:disableExternalization}) as we do not support for instance |remember picture|). Then, configure the default preamble for cached files as explained below by extending the |tikz| or |tikzpicture| presets (that first loads |tikz|). \item Use the generic commands that allows you to wrap an arbitrary environment: \begin{verbatim} % name environment ---v v----- preset options \cacheEnvironment{tikzpicture}{tikzpicture} \end{verbatim} Note that you can also wrap commands (that you call with |\foo| instead of |\begin{foo}...\end{foo}|) using this:\\ |\cacheCommand{yourcommand}[O{default val}m]{your preset options}|\\ (|O{default val}m| means that the command accepts one optional argument with default value |default val| and one mandatory argument) but \textbf{we do recommend} to use |\cacheCommand| to cache |\tikz| since |\tikz| has a quite complicated parsing strategy (e.g.\ you can write |\tikz[options] \node{foo};| which has no mandatory argument enclosed in |{}|). |\cacheTikz| takes care of this already and caches both the environment |\begin{tikzpicture}| and the macro |\tikz|. \item Use |tikzpictureC| instead of |tikzpicture| (this is mostly done to easily convert existing code to this library, but works only for |tikz| pictures). \item Use the more general |CacheMe| environment, that can cache \tikzname, \LaTeX{}, python, and much more. \end{enumerate} These 4 options are illustrated below (note that all commands accept a first optional argument enclosed in |<...>| that contains the options to pass to |CacheMe| after loading the |tikzpicture| preset, that loads itself the |tikz| preset first): Option 1: \begin{codeexample}[width=0pt] %% We override the default tikzpicture environment %% to externalize all pictures %% Warning: it will cause troubles with pictures relying on |remember pictures| \cacheTikz I am a cached picture: \begin{tikzpicture}[baseline=(A.base)] \node[draw,rounded corners,fill=pink!60](A){Hello World!}; \end{tikzpicture}. % I am a non-cached picture: \begin{tikzpicture}[baseline=(A.base)] \node[draw,rounded corners,fill=pink!60](A){Hello World!}; \end{tikzpicture}. \end{codeexample} Option 2: \begin{codeexample}[width=0pt] \cacheEnvironment{tikzpicture}{tikzpicture} I am a cached picture: \begin{tikzpicture}[baseline=(A.base)] \node[draw,rounded corners,fill=pink!60](A){Hello World!}; \end{tikzpicture}. \end{codeexample} Option 3: \begin{codeexample}[width=0pt] I am a cached picture: \begin{tikzpictureC}[baseline=(A.base)] \node[draw,rounded corners,fill=pink!60](A){Hello World!}; \end{tikzpictureC}. \end{codeexample} Option 4: \begin{codeexample}[width=0pt] I am a cached picture: \begin{CacheMe}{tikzpicture}[baseline=(A.base)] \node[draw,rounded corners,fill=pink!60](A){Hello World!}; \end{CacheMe}. \end{codeexample} Since |CacheMe| is more general as it applies also to non-tikz pictures (just replace |tikzpicture| with the style of your choice), we will mostly use this syntax from now. \subsection{Custom preamble} Note that the pictures are compiled in a separate document, with a different preamble and class (we use the standalone class). This is interesting to reduce the compilation time of each picture (loading a large preamble is really time consuming) and to avoid unnecessary recompilation (do you want to recompile all your pictures when you add a single new macro?) without sacrificing the purity. But of course, you need to provide the preamble of the pictures. The easiest way is probably to modify the |tikz| preset (since |tikzpicture| loads this style first, it will also apply to tikzpictures. Note that you can also modify the |latex| preset if you want the change to apply to all \LaTeX{} documents): \begin{codeexample}[width=0pt,vbox] \robExtConfigure{ add to preset={tikz}{ add to preamble={\usetikzlibrary{shadows}}, }, } See, tikz's style now packs the |shadows| library by default: % \begin{CacheMe}{tikzpicture}[even odd rule] \filldraw [drop shadow,fill=white] (0,0) circle (.5) (0.5,0) circle (.5); \end{CacheMe} \end{codeexample} and you can also create a new preset, and why not mix multiple presets together: \begin{codeexample}[width=0pt,vbox] \robExtConfigure{ new preset={load forest}{ latex, add to preamble={\usepackage{forest}}, }, new preset={load hello}{ add to preamble={\def\hello#1{Hello #1.}}, }, } \begin{CacheMe}{load forest, load hello} \begin{forest} [VP [DP[\hello{John}]] [V’ [V[sent]] [DP[Mary]] [DP[D[a]][NP[letter]]] ] ] \end{forest} \end{CacheMe} \end{codeexample} \textbf{Note:} the |add to preset| and |new preset| directives have been added in v2.0, together with the |tikzpicture| preset. In v1.0, you would do |tikz/.append style={...}| (and you can still do this if you prefer), the difference is that |.append style| and |.style| require the user to double all hashes like |\def\mymacro##1{Hello ##1.}| which can lead to confusing errors. You can also choose to overwrite the preset options for a single picture (or even a block of picture if you run the |\robExtConfigure| and |CacheMe| inside a group |{ ... }|): \begin{codeexample}[width=0pt,vbox] See, you can add something to the preamble of a single picture: % \begin{CacheMe}{tikzpicture, add to preamble={\usetikzlibrary{shadows}}}[even odd rule] \filldraw [drop shadow,fill=white] (0,0) circle (.5) (0.5,0) circle (.5); \end{CacheMe} \end{codeexample} Note that if you use the |tikzpictureC| or |tikzpicture| syntax, you want to add the options right after the name of the command or environment, enclosed in |<>| (by default): \begin{codeexample}[width=0pt,vbox] See, you can add something to the preamble of a single picture: % \begin{tikzpictureC}[even odd rule] \filldraw [drop shadow,fill=white] (0,0) circle (.5) (0.5,0) circle (.5); \end{tikzpictureC} \end{codeexample} \textbf{Important}: Note that the preset options can be specified in a number of places: in |\robExtConfigure| (possibly in a group, or inside a preset), in the options of the picture, in the default options of |cacheEnvironment| and |cacheCommand| etc. Most options can be used in all these places, the only difference being the scope of the options. If you find yourself duplicating code between the main document and the preamble of the cached file, we provide multiple solutions with different properties: \begin{itemize} \item Recommended: |\newcommandAutoForward| etc, to automatically forward specific macros/colors etc: see \cref{sec:forwardIntro}. Very useful if you want maximum purity without annoying recompilation. \item |\runHereAndInPreambleOfCachedFiles|: mostly useful if you want to always add a text both in the current file and in the cached files, like a font etc. \item Share a common input file with or without dependencies (see \cref{sec:dependencies}), useful if you need to work with external files or if you do NOT want purity (i.e.\ you can choose not to recompile the file even if its content changes). \end{itemize} |\runHereAndInPreambleOfCachedFiles{XXX}| is very simple (need at least version 2.3), as it is simply a shortcut for typing |XXX| in the current file and in the preamble of the cached files (default to latex preset). For instance if you want the current file and all cached files to use the Times New Roman font, you can do: \begin{codeexample}[code only] % By default adds \runHereAndInPreambleOfCachedFiles{ \usepackage{fontspec} % need lualatex or xelatex to compile \setmainfont{Times New Roman} } \robExtConfigure{ add to preset={latex}{ use lualatex, % Make sure to compile with lualatex the cached images. }, } \end{codeexample} You can also define your common macros (even if we recommend |\newcommandAutoForward| to avoid frequent recompilations when the definitions changes): \begin{codeexample}[width=0pt,vbox] \runHereAndInPreambleOfCachedFiles{ \def\name#1{The great lady #1 and her friends.} } \cacheTikz This is the picture of \name{Alice}: \begin{tikzpicture} \node[fill=orange]{\name{Alice}}; \end{tikzpicture} \end{codeexample} \subsection{Dependencies}\label{sec:dependencies} \textbf{Note: dependencies can sometimes be advantageously replaced with auto-forwarding of arguments, cf.\ \cref{sec:forwardIntro}, or |\runHereAndInPreambleOfCachedFiles|.} It might be handy to have a file that is loaded in both the main document and in the cached pictures. For instance, if you have a logo that might be included in the cached files, or if you have a file |common_inputs.tex| that you want to input in both the main file and in the cached files, that contains, say:\\ |\def\myValueDefinedInCommonInputs{42}|\\ then you can add it as a dependency this way (here we use the |latex| preset that does not wrap the code inside a |tikzpicture| only to illustrate that we can also cache things that are not generated by tikz): \begin{codeexample}[width=0pt,vbox] \begin{CacheMe}{latex, add dependencies={common_inputs.tex}, add to preamble={\input{__ROBEXT_WAY_BACK__common_inputs.tex}}} The answer is \myValueDefinedInCommonInputs. \end{CacheMe} \end{codeexample} Note that the placeholder |__ROBEXT_WAY_BACK__| contains the path from the cache folder (containing the |.tex| that will be cached) to the root folder, and will be replaced when creating the file. This way, you can easily input files contained in the root folder. You can also create your own placeholders, read more below. You can note that we used |add dependencies={common_inputs.tex}|: this allows us to recompile the files if |common_inputs.tex| changes. If you do not want this behavior (e.g. |common_inputs.tex| changes too often and you do not want to recompile everything at every change), you can remove this line, but beware: if you do a breaking changes in |common_inputs.tex| (e.g.\ redefine |42| to |43|), then the previously cached picture will not be recompiled! (So you will still read 42 instead of 43.) Sometimes, it can also be useful to copy a file in the cache folder. This can be done via: \begin{codeexample}[code only] \robExtConfigure{ copy file to cache={zx-calculus.sty}, } \end{codeexample} \subsection{Wrap arbitrary environments} You can wrap arbitrary environments using the already presented |cacheEnvironment| and |cacheCommand|, where the first mandatory argument is the name of the environment/macro, and the second mandatory argument contains the default preset. |cacheEnvironment| works for any environment while |cacheCommand| might need a bit of help to determine the signature of the macro if the function is defined via xparse. Long story short, |O{foo}| is an optional argument with default value foo, |m| is a mandatory argument. By default, you can pass an optional arguments via the first argument enclosed in |<>|: \begin{codeexample}[width=0pt,vbox] \cacheCommand{zx}[O{}O{}O{}m]{latex, add to preamble={\usepackage{zx-calculus}\def\hello#1{Hello #1.}}} \zx[mbr=1]{ % amsmath provides \text \zxX{\alpha} \rar \ar[d,C] & \zxZ{\text{\hello{Alice}}} \\ \zxZ{\beta} \rar & \zxZ{\text{\bye{Bob}}} } \end{codeexample} you can also disable externalization for some commands using this same command, here is for instance the code to use |todonotes|: \begin{codeexample}[code only] \cacheCommand{todo}[O{}m]{disable externalization} \todo{Check how to use cacheCommand and cacheEnvironment} \end{codeexample} Which gives you { \cacheCommand{todo}[O{}m]{disable externalization} \todo{Check how to use cacheCommand and cacheEnvironment} } See \cref{sec:wrapAutomatically} for more details. \subsection{Disabling externalization} You can use |disable externalization| to disable externalization (which is particularly practical if you set |\cacheTikz|). You can configure the exact command run in that case using |command if no externalization/.code={...}|, but most of the time it should work out of the box (see \cref{sec:disableExternalization} for details). \begin{codeexample}[width=0pt,vbox] % In theory all pictures should be externalized (so remember picture should fail) \tikz[remember picture,baseline=(pointtome1.base)] \node[rounded corners, fill=orange](pointtome1){Point to me if you can};\\ \cacheTikz % But we can disable it temporarily \begin{tikzpicture}[remember picture] \node[rounded corners, fill=red](A){This figure is not externalized. This way, it can use remember picture.}; \draw[->,overlay] (A) to[bend right] (pointtome1); \end{tikzpicture}\\ % You can also disable it globally/in a group: { \robExtConfigure{disable externalization} \begin{tikzpicture}[remember picture] \node[rounded corners, fill=red](A){This figure is not externalized. This way, it can use remember picture.}; \draw[->,overlay] (A.west) to[bend left] (pointtome1); \end{tikzpicture}\\ \begin{tikzpicture}[remember picture] \node[rounded corners, fill=red](A){This figure is not externalized. This way, it can use remember picture.}; \draw[->,overlay] (A.east) to[bend right] (pointtome1); \end{tikzpicture}\\ } \begin{tikzpicture} \node[rounded corners, fill=green](A){This figure is externalized, but cannot use remember picture.}; \end{tikzpicture} \end{codeexample} You can also disable externalization for some kinds of commands. For instance, the package |todonotes| requires |remember picture| and is therefore not compatible with externalization provided by this package. To disable externalization on all |\todo|, you can do: \begin{codeexample}[code only] \cacheCommand{todo}[O{}m]{disable externalization} \todo{Check how to use cacheCommand and cacheEnvironment} \end{codeexample} Which gives you { \cacheCommand{todo}[O{}m]{disable externalization} \todo{Check how to use cacheCommand and cacheEnvironment} } You can also disable externalization on elements that contain a specific string/regex. For instance, you can disable externalization on all elements containing |remember picture|: \begin{codeexample}[vbox] \cacheTikz \robExtConfigure{ add to preset={tikz}{ if matches={remember picture}{disable externalization}, }, } \begin{tikzpicture}[remember picture] \node[fill=green](my node){Point to me}; \end{tikzpicture} % Some text % \begin{tikzpicture}[overlay, remember picture] \draw[->] (0,0) to[bend left] (my node); \end{tikzpicture} \end{codeexample} \subsection{Feeding data from the main document to the cached documents}\label{sec:forwardIntro} You can feed data from the main document to the cached file using placeholders, since |set placeholder eval={__foo__}{\bar}| will evaluate |\bar| and put the result in |__foo__|. For instance, if the picture depends on the current page, you can do: \message{We will export the page: \thepage} \begin{codeexample}[width=0pt,vbox] \begin{tikzpictureC} \node[rounded corners, fill=red]{The current page is __thepage__.}; \end{tikzpictureC} \end{codeexample} However, since v2.1, we also provide a method to specifically export a counter using |forward counter=my counter| and a macro using |forward=\macroToExport|, possibly by evaluating first using |forward eval=\macroToEval|: \begin{codeexample}[width=0pt] \cacheTikz \NewDocumentCommand{\MyNode}{O{}m}{\node[rounded corners,fill=red,#1]{#2};} \begin{tikzpicture} \MyNode{Alice} \MyNode[xshift=2cm]{Bob} \end{tikzpicture} \end{codeexample} To avoid manually writing |forward=...| for each picture, we can instead load |auto forward| and define the macro for instance using |\newcommandAutoForward| (we also provide alternatives for xparse and def-based definitions): \begin{codeexample}[vbox] \cacheTikz \robExtConfigure{add to preset={tikz}{auto forward}} \newcommandAutoForward{\MyNode}[2][draw,thick]{\node[rounded corners,fill=red,#1]{#2};} \newcommandAutoForward{\MyGreenNode}[2][draw,thick]{\node[rounded corners,fill=green,#1]{#2};} \begin{tikzpicture} \MyNode{Recompiled only if MyNode is changed} \MyNode[xshift=8cm]{but not if the (unused) MyGreenNode is changed.} \end{tikzpicture}\\ \begin{tikzpicture} \MyGreenNode{Recompiled only if MyGreenNode is changed} \MyGreenNode[xshift=8cm]{but not if the (unused) MyNode is changed.} \end{tikzpicture} \end{codeexample} If a macro depends on another macro/package, it is possible to load any additional style like using the last optional argument of |\newcommandAutoForward|: \begin{codeexample}[vbox] \cacheTikz \robExtConfigure{add to preset={tikz}{auto forward}} \newcommandAutoForward{\MyName}{Alice} \newcommandAutoForward{\MyNode}[2][draw,thick]{ \node[rounded corners,fill=red,#1]{\ding{164} \MyName: #2}; }[forward=\MyName, add to preamble={\usepackage{pifont}}] \begin{tikzpicture} \MyNode{Recompiled if MyNode or MyName are changed} \end{tikzpicture} \end{codeexample} If instead of forwarding the macro you want to, say, load a package or any other style, you should use instead |\robExtConfigIfMacroPresent|: \begin{codeexample}[vbox] \cacheTikz \robExtConfigure{add to preset={tikz}{auto forward}} \robExtConfigIfMacroPresent{\ding}{add to preamble={\usepackage{pifont}}} \begin{tikzpicture} \node[fill=green, circle]{\ding{164}}; \end{tikzpicture} \end{codeexample} For more details and functions, see \cref{sec:forward}. \subsection{Defining macros} You can define macros in the preamble: \begin{codeexample}[width=0pt,vbox] \cacheMe[latex, add to preamble={\def\sayhello#1{Hello #1.}}]{ \sayhello{my friend} } \end{codeexample} \subsection{Feeding data back into the main document} For more advanced usage, you might want to compute a data and cache the result in a macro that you could use later. This is possible if you write into the file |\jobname-out.tex| during the compilation of the cached file (by default, we already open |\writeRobExt| to write to this file). This file will be automatically loaded before loading the pdf (but you can customize all these operations, for instance if you do not want to load the pdf at all; the only requirement is that you should generate a |.pdf| file to specify that the compilation is finished). For instance: \begin{codeexample}[width=0pt,vbox] \begin{CacheMe}{latex, add to preamble={\usepackage{tikz}}, do not include pdf} We compute this data that is long to compute: \pgfmathparse{(1 + sqrt(5))/2}% result is stored in \pgfmathresult % We write the result to the -out file (\string\foo writes \foo to the file without evaluating it, % so this will write "\gdef\myLongResult{1.61803}"): % Note that CacheMe is evaluated in a group, so you want to use \gdef to define it % outside of the group \immediate\write\writeRobExt{% \string\gdef\string\myLongResult{\pgfmathresult}% } \end{CacheMe} We computed the cached value \myLongResult. \end{codeexample} \subsection{A note on font handling} Cached pictures are basically simple latex files that are compiled in a separate |standalone| document. Therefore, if the main document loads a different font, there is no reason for the cached document to also use that font. Fully-automatically forwarding the current font by default is not so easy because there are so many ways to load a font in LaTeX (by loading a package like |\usepackage{times}|, by using xelatex or lualatex, by calling locally |\rmfamily|, by loading a different class… and here I'm not even mentioning math font where each symbol \mylink{https://tex.stackexchange.com/questions/603475/is-there-any-command-to-find-out-the-current-math-font-used)}{can use a different font}, and this is even more complicated if the main file is compiled with a different engine (e.g.\ what should I do if the main document is compiled with lualatex but the inner cached images are compiled with pdflatex, that supports very few fonts by default?). For these reasons, it is simpler to simply let the user load fonts as they want, for instance using something like: \begin{codeexample}[width=0pt] \robExtConfigure{ add to preset={latex}{ add to preamble={\usepackage{times}}, }, } \begin{tikzpictureC} \node[fill=green]{See I am using times font}; \end{tikzpictureC} \end{codeexample} or: \begin{codeexample}[width=0pt] \robExtConfigure{ add to preset={latex}{ add to latex options={12pt}, }, } \begin{tikzpictureC} \node[fill=green]{See I am using a 12pt font}; \end{tikzpictureC} \end{codeexample} But if you want to use the same font as the current document, you might want to avoid code duplication (this might also be useful for any arbitrary code). Since v2.3, you can do: \begin{codeexample}[code only] \runHereAndInPreambleOfCachedFiles{ \usepackage{fontspec} % need lualatex or xelatex to compile \setmainfont{Times New Roman} } \robExtConfigure{ add to preset={latex}{ use lualatex, % Make sure to compile with lualatex the cached images. }, } \end{codeexample} and it will automatically compile the cached images with lualatex, and configure the font of the current document and of the cached documents to follow |Times New Roman|. If you like to often change the font in the document, you can also do something like that to always compile the document with the font currently in use if you use lualatex (this code fails with \mylink{https://tex.stackexchange.com/questions/705208/get-current-font}{xelatex}, currently trying to understand why): \begin{codeexample}[code only] \usepackage{fontspec} \setmainfont{Times New Roman} %% needed since \f@family contains a @ in its name: \makeatletter \def\myCurrentFont{\f@family} \makeatother \robExtConfigure{ add to preset={latex}{ use lualatex, % create a new placeholder containing the name of the current font: set placeholder eval={__MY_FONT__}{\myCurrentFont}, add to preamble={ \usepackage{fontspec} \setmainfont{__MY_FONT__} }, }, } \end{codeexample} If you want this to work on xelatex, you can either manually attribute a |NFSSFamily| name like: \begin{codeexample}[code only] \runHereAndInPreambleOfCachedFiles{ \usepackage{fontspec} % need lualatex or xelatex to compile, make sure to use a different NFSSFamily for each new font you add here \setmainfont[NFSSFamily=tmsnr]{Times New Roman} } \end{codeexample} and use the above solution for lualatex (you can change the font with |\fontfamily{tmsnr}\selectfont|), or do instead something like this: \begin{codeexample}[code only] % Create a new command \mysetmainfont which forwards the font to the picture. \NewDocumentCommand{\mysetmainfont}{m}{% \def\myCurrentFont{#1}% Stores the name of the current font in \myCurrentFont \setmainfont{#1}% } % Use mysetmainfont from now to change font: \mysetmainfont{Times New Roman} \usepackage{robust-externalize} \robExtConfigure{ add to preset={latex}{ use xelatex, set placeholder eval={__MY_FONT__}{\myCurrentFont}, add to preamble={ \usepackage{fontspec} \setmainfont{__MY_FONT__} }, }, } \end{codeexample} \subsection{Compile in parallel} Make sure to have xargs installed (installed by default on most linux, on Windows you need to install the lightweight GNU On Windows (Gow) \url{https://github.com/bmatzelle/gow}), and type in your preamble: \begin{codeexample}[code only] \robExtConfigure{ compile in parallel } \end{codeexample} to compile all figures in parallel (you need to compile your document twice). Your document should build significantly faster, possibly faster than a normal run. If you want to compile in parallel only if you have more than, say, 5 new elements to cache, do: \begin{codeexample}[code only] \robExtConfigure{ compile in parallel after=5 } \end{codeexample} \subsection{Compile a preset}\label{sec:compilePreset} Long story short: you can compile even faster (around 1.5x in our tests) by compiling presets, but beware that you will not be able to modify the placeholders except |add to preamble| with the default compiler we provide: \begin{codeexample}[vbox] % We create a latex-based preset and compile it \robExtConfigure{ new preset={templateZX}{ latex, add to preamble={ \usepackage{tikz} \usepackage{tikz-cd} \usepackage{zx-calculus} }, %% possibly add some dependencies }, % We compile it into a new preset new compiled preset={compiled ZX}{templateZX, compile latex template}{}, } % we use that preset automatically for ZX environments \cacheEnvironment{ZX}{compiled ZX} \cacheCommand{zx}{compiled ZX} % Usage: (you can't use placeholders except for the preamble, trade-off of the compiled template) \begin{ZX} \zxX{\sayHey{Bob}} \end{ZX} \end{codeexample} For details, see \cref{sec:compileFaster}. \subsection{For non-\LaTeX{} code} Due to the way \LaTeX{} works, non-\LaTeX{} code can't be reliably read inside macros and some environments that parse their body (e.g. align) as some characters are removed (e.g. percent symbols are comments and are removed). For this reason, we sometimes need to separate the time where we define the code and where we insert it (this is done using placeholders, see |PlaceholderFromCode|), and we need to introduce new environments to populate the template (see \cref{sec:placeholders} for more details, to generate them from filename, to get the path of a file etc). The environment |CacheMeCode| can be used for this purpose. \subsubsection{Python code} \paragraph{Generate an image} For instance, you can use the default |python| template to generate an image with python. The following code: \begin{codeexample}[code only] \begin{CacheMeCode}{python, set includegraphics options={width=.8\linewidth}} import matplotlib.pyplot as plt year = [2014, 2015, 2016, 2017, 2018, 2019] tutorial_count = [39, 117, 111, 110, 67, 29] plt.plot(year, tutorial_count, color="#6c3376", linewidth=3) plt.xlabel('Year') plt.ylabel('Number of futurestud.io Tutorials') plt.savefig("__ROBEXT_OUTPUT_PDF__") \end{CacheMeCode} \end{codeexample} will produce the image visible in \cref{fig:pythonGeneratedImage}. \textbf{Importantly: you do not want to indent the content of CacheMeCode, or the space will also appear in the final code.} \begin{figure} \centering \begin{CacheMeCode}{python, set includegraphics options={width=.8\linewidth}} import matplotlib.pyplot as plt year = [2014, 2015, 2016, 2017, 2018, 2019] tutorial_count = [39, 117, 111, 110, 67, 29] plt.plot(year, tutorial_count, color="#6c3376", linewidth=3) plt.xlabel('Year') plt.ylabel('Number of futurestud.io Tutorials') plt.savefig("__ROBEXT_OUTPUT_PDF__") \end{CacheMeCode} \caption{Image generated with python.} \label{fig:pythonGeneratedImage} \end{figure} \paragraph{Compute a value} We also provide by default a number of helper functions. For instance, |write_to_out(text)| will write |text| to the |*-out.tex| file that is loaded automatically by \LaTeX{}. This is useful to compute data that is not an image (note that |r"some string"| does not consider backslash as an escape string, which is handy to write \LaTeX{} code in python): For instance: \begin{codeAndResult} \begin{CacheMeCode}{python, do not include pdf} import math write_to_out(r"\gdef\cosComputedInPython{" + str(math.cos(1)) + r"}") \end{CacheMeCode} $\rightarrow$ The cosinus of 1 is \cosComputedInPython. \end{codeAndResult} \paragraph{Improve an existing preset} If you often use the same code (e.g.\ load matplotlib, save the file etc), you can directly modify the |__ROBEXT_MAIN_CONTENT__| placeholder to add the redundant information (or create a new template from scratch, see below). By default, it points to |__ROBEXT_MAIN_CONTENT_ORIG__| that contains directly the code typed by the user (this is true for all presets, as |CacheMe*| is in charge of setting this placeholder). When dealing with \LaTeX{} code, |__ROBEXT_MAIN_CONTENT__| should ideally contain a code that can be inserted as-it in the document in order to be compatible by default with |disable externalization|. So if you want to wrap the content of the user in an environment like |\begin{tikzpicture}...\end{tikzpicture}|, this is the placeholder to modify. \begin{codeAndResult} %% Create your style: \begin{PlaceholderFromCode}{__MY_MATPLOTLIB_TEMPLATE_BEFORE__} import matplotlib.pyplot as plt import sys \end{PlaceholderFromCode} \begin{PlaceholderFromCode}{__MY_MATPLOTLIB_TEMPLATE_AFTER__} plt.savefig("__ROBEXT_OUTPUT_PDF__") \end{PlaceholderFromCode} \robExtConfigure{ new preset={my matplotlib}{ python, add before placeholder no space={__ROBEXT_MAIN_CONTENT__}{__MY_MATPLOTLIB_TEMPLATE_BEFORE__}, add to placeholder no space={__ROBEXT_MAIN_CONTENT__}{__MY_MATPLOTLIB_TEMPLATE_AFTER__}, }, } %% Use your style: %% See, you don't need to load matplotlib or save the file: \begin{CacheMeCode}{my matplotlib, set includegraphics options={width=.5\linewidth}} year = [2014, 2015, 2016, 2017, 2018, 2019] tutorial_count = [39, 117, 111, 110, 67, 29] plt.plot(year, tutorial_count, color="#6c3376", linewidth=3) plt.xlabel('Year') plt.ylabel('Number of futurestud.io Tutorials') \end{CacheMeCode} \end{codeAndResult} \paragraph{Custom parameters and placeholders} Let us say that you would like to define a default font size for your figure, but that you would like to allow the user to change this font size. Then, you should create a new placeholder with your default value, and use |set placeholder| to change this value later (see also the documentation of |CacheMeCode| to see how to create a new command to avoid typing |set placeholder|): \begin{codeAndResult} %% Create your style: \begin{PlaceholderFromCode}{__MY_MATPLOTLIB_TEMPLATE_BEFORE__} import matplotlib as mpl import matplotlib.pyplot as plt import sys mpl.rcParams['font.size'] = __MY_MATPLOTLIB_FONT_SIZE__ \end{PlaceholderFromCode} \begin{PlaceholderFromCode}{__MY_MATPLOTLIB_TEMPLATE_AFTER__} plt.savefig("__ROBEXT_OUTPUT_PDF__") \end{PlaceholderFromCode} \robExtConfigure{ new preset={my matplotlib}{ python, % We create a new placeholder (it is simple enough that you don't need to use PlaceholderFromCode) set placeholder={__MY_MATPLOTLIB_FONT_SIZE__}{12}, add before placeholder no space={__ROBEXT_MAIN_CONTENT__}{__MY_MATPLOTLIB_TEMPLATE_BEFORE__}, add to placeholder no space={__ROBEXT_MAIN_CONTENT__}{__MY_MATPLOTLIB_TEMPLATE_AFTER__}, }, } %% Use your style: %% See, you don't need to load matplotlib or save the file: Default font size: \begin{CacheMeCode}{my matplotlib, set includegraphics options={width=.5\linewidth}} year = [2014, 2015, 2016, 2017, 2018, 2019] tutorial_count = [39, 117, 111, 110, 67, 29] plt.plot(year, tutorial_count, color="#6c3376", linewidth=3) plt.xlabel('Year') plt.ylabel('Number of futurestud.io Tutorials') \end{CacheMeCode} With font size 16: \begin{CacheMeCode}{my matplotlib, set includegraphics options={width=.5\linewidth}, set placeholder={__MY_MATPLOTLIB_FONT_SIZE__}{16}} year = [2014, 2015, 2016, 2017, 2018, 2019] tutorial_count = [39, 117, 111, 110, 67, 29] plt.plot(year, tutorial_count, color="#6c3376", linewidth=3) plt.xlabel('Year') plt.ylabel('Number of futurestud.io Tutorials') \end{CacheMeCode} \end{codeAndResult} Note that if you manage to move all the code in the template and that the user can configure everything using the options and an empty content, you can use |CacheMeNoContent| that takes no argument and that consider its body as the options. \paragraph{Custom include command} There may be some cases where you do not want to include a picture. We already saw the option |do not include pdf| if you do not want to include anything. But you can customize the include function, using notably:\\ |custom include command={your include command}| For instance, let us say that you would like to display both the source code used to obtain a given code, together with the output of this code. Then, you can write this style: \begin{codeexample}[code only] { %% Create your style: \begin{PlaceholderFromCode}{__MY_PRINT_BOTH_TEMPLATE_BEFORE__} # File where print("bla") should be redirected # get_filename_from_extension("-foo.txt") will give you the path of the file # in the cache that looks like robExt-somehash-foo.txt print_file = open(get_filename_from_extension("-print.txt"), "w") sys.stdout = print_file # This code will read the current code, and extract the lines between # that starts with "### CODESTARTSHERE" and "### CODESTOPSHERE", and will write # it into the *-code.text (we do not want to print all these functions in # the final code) with open(get_filename_from_extension("-code.txt"), "w") as f: # The current script has extension .tex with open(get_current_script(), "r") as script: should_write = False for line in script: if line.startswith("### CODESTARTSHERE"): should_write = True elif line.startswith("### CODESTOPSHERE"): should_write = False elif "HIDEME" in line: pass else: if should_write: f.write(line) ### CODESTARTSHERE \end{PlaceholderFromCode} \begin{PlaceholderFromCode}{__MY_PRINT_BOTH_TEMPLATE_AFTER__} ### CODESTOPSHERE print_file.close() \end{PlaceholderFromCode} \robExtConfigure{ new preset={my python print both}{ python, add before placeholder no space={__ROBEXT_MAIN_CONTENT__}{__MY_PRINT_BOTH_TEMPLATE_BEFORE__}, add to placeholder no space={__ROBEXT_MAIN_CONTENT__}{__MY_PRINT_BOTH_TEMPLATE_AFTER__}, set title/.style={ set placeholder={__MY_TITLE__}{#1}, }, set title={Example}, custom include command={ % Useful to replace __MY_TITLE__: \evalPlaceholder{ \begin{tcolorbox}[colback=red!5!white,colframe=red!75!black,title=__MY_TITLE__] \lstinputlisting[frame=single, breakindent=.5\textwidth, frame=single, breaklines=true, style=mypython]{\robExtAddCachePathAndName{\robExtFinalHash-code.txt}} Output: \verbatiminput{\robExtAddCachePathAndName{\robExtFinalHash-print.txt}} \end{tcolorbox} } }, }, } \end{codeexample} Once the style is defined (actually we already defined in the library under the name |python print code and result|), you can just write: \begin{codeexample}[code only] \begin{CacheMeCode}{my python print both, set title={The for loop}} for name in ["Alice", "Bob"]: print(f"Hello {name}") \end{CacheMeCode} \end{codeexample} to get: { %% Create your style: \begin{PlaceholderFromCode}{__MY_PRINT_BOTH_TEMPLATE_BEFORE__} # File where print("bla") should be redirected # get_filename_from_extension("-foo.txt") will give you the path of the file # in the cache that looks like robExt-somehash-foo.txt print_file = open(get_filename_from_extension("-print.txt"), "w") sys.stdout = print_file # This code will read the current code, and extract the lines between # that starts with "### CODESTARTSHERE" and "### CODESTOPSHERE", and will write # it into the *-code.text (we do not want to print all these functions in # the final code) with open(get_filename_from_extension("-code.txt"), "w") as f: # The current script has extension .tex with open(get_current_script(), "r") as script: should_write = False for line in script: if line.startswith("### CODESTARTSHERE"): should_write = True elif line.startswith("### CODESTOPSHERE"): should_write = False elif "HIDEME" in line: pass else: if should_write: f.write(line) ### CODESTARTSHERE \end{PlaceholderFromCode} \begin{PlaceholderFromCode}{__MY_PRINT_BOTH_TEMPLATE_AFTER__} ### CODESTOPSHERE print_file.close() \end{PlaceholderFromCode} \robExtConfigure{ new preset={my python print both}{ python, add before placeholder no space={__ROBEXT_MAIN_CONTENT__}{__MY_PRINT_BOTH_TEMPLATE_BEFORE__}, add to placeholder no space={__ROBEXT_MAIN_CONTENT__}{__MY_PRINT_BOTH_TEMPLATE_AFTER__}, set title/.style={ set placeholder={__MY_TITLE__}{#1}, }, set title={Example}, custom include command={ % Useful to replace __MY_TITLE__: \evalPlaceholder{ \begin{tcolorbox}[colback=red!5!white,colframe=red!75!black,title=__MY_TITLE__] % \noindent Code: \lstinputlisting[frame=single,breakindent=.5\textwidth,frame=single,breaklines=true,style=mypython]{\robExtAddCachePathAndName{\robExtFinalHash-code.txt}} Output: \verbatiminput{\robExtAddCachePathAndName{\robExtFinalHash-print.txt}} \end{tcolorbox} } }, }, } %% Use your style: \begin{CacheMeCode}{my python print both, set title={The for loop}} for name in ["Alice", "Bob"]: print(f"Hello {name}") \end{CacheMeCode} } \subsubsection{Other languages}\label{sec:otherLanguages} We also provide support for other languages, notably |bash|, but it is relatively easy to add basic support for any new language. You only need to configure |set compilation command| to your command, |set template| to the file to compile (|__ROBEXT_MAIN_CONTENT__| contains the code typed by the user), and possibly a custom include command with |custom include command| if you do not want to do |\includegraphics| on the final pdf. For instance, to define a basic template for bash, you just need to use: \begin{codeAndResult} % Create your style \begin{PlaceholderFromCode}{__MY_BASH_TEMPLATE__} # Quit if there is an error set -e __ROBEXT_MAIN_CONTENT__ # Create the pdf file to certify that no compilation error occured # (non empty file to avoid arXiv error) echo "ok" > "__ROBEXT_OUTPUT_PDF__" \end{PlaceholderFromCode} \robExtConfigure{ new preset={my bash}{ set compilation command={bash "__ROBEXT_SOURCE_FILE__"}, set template={__MY_BASH_TEMPLATE__}, %%% Version 1: % verbatim output, %%% Version 2: custom include command={% \evalPlaceholder{% \verbatiminput{__ROBEXT_CACHE_FOLDER____ROBEXT_OUTPUT_PREFIX__-out.txt}% }% }, % Ensure that the code does not break when externalization is disabled print verbatim if no externalization, } } % Use your style \begin{CacheMeCode}{my bash} # Write the system conf to a file *-out.txt uname -srv > "__ROBEXT_OUTPUT_PREFIX__-out.txt" \end{CacheMeCode} \end{codeAndResult} \paragraph{Code inside a macro} Due to fundamental \LaTeX{} restrictions, it is impossible to use |CacheMeCode| inside a macro or some environments as \LaTeX{} will strip all lines containing a percent character for instance. The solution here is to define our main content before, and then set it using |set main content| (that simply sets |__ROBEXT_MAIN_CONTENT_ORIG__|). In this example, we also show how |CacheMeNoContent| can be used when their is no content (the arguments to |CacheMe| are directly given in the body of |CacheMeNoContent|): \begin{codeAndResult} \begin{PlaceholderFromCode}{__TMP_MAIN_CONTENT__} import matplotlib.pyplot as plt year = [2014, 2015, 2016, 2017, 2018, 2019] tutorial_count = [39, 117, 111, 110, 67, 29] plt.plot(year, tutorial_count, color="#6c3376", linewidth=3) plt.xlabel('Year') plt.ylabel('Number of futurestud.io Tutorials') plt.savefig("__ROBEXT_OUTPUT_PDF__") \end{PlaceholderFromCode} \fbox{\begin{CacheMeNoContent} python, set includegraphics options={width=.8\linewidth}, set main content={__TMP_MAIN_CONTENT__}, \end{CacheMeNoContent}} \end{codeAndResult} \subsection{Force to recompile or remove a cached item} If you want to recompile a file (e.g. an untracked dependency was changed\dots{} and you do not want to track it to avoid recompilation when you change a single line in this file), you can use the |recompile| style since v2.1: \begin{codeexample}[vbox, %TURN_INTO_CODE_ONLY_ARXIV ] \cacheEnvironment{tikzcd}{tikz, add to preamble={\usepackage{tikz-cd}}} \begin{tikzcd} A \rar & B \end{tikzcd} \end{codeexample} Note that this assumes that your compilation command is idempotent (so running it twice is like running it once) since the aux files are not cached. If you want to clean aux files or if you run an older version, see the documentation of |recompile| in \cref{sec:configureCompilationCommand}. \section{Documentation} Before starting this documentation, note that all commands are prefixed with |robExt| and all environments are prefixed with |RobExt|, but we also often define aliases without this prefix. The user is free to use any version, but we recommend to use the non-prefixed version unless a clash with a package forbids you from using it. In the following, we will only print the non-prefixed name when it exists. Note also that we follow the convention that environment names start with an upper case letter while commands start with a lower case letter. \subsection{How it works} This library must be able to generate 3 elements for any cached content: \begin{itemize} \item a source file, that will be compiled, and is obtained by expanding the placeholder |__ROBEXT_TEMPLATE__| (see \cref{sec:placeholders}), \item a compilation command obtained by expanding the placeholder |__ROBEXT_COMPILATION_COMMAND__|, \item a dependency file, that contains the hash of all the dependencies (see \cref{sec:dependencies} for details) and the compilation command, \item an inclusion command (this one is not used during the caching process, it is only used when including the compiled document in the main document), that you can set using |custom include command={your command}|. \end{itemize} The hash of all these elements is computed in order to obtain a reference hash, denoted |somehash| that looks like a unique random value (note that |__ROBEXT_OUTPUT_PDF__| and alike are expanded after knowing the hash since they depend on the final hash value). This hash |somehash| will change whenever a dependency changes, or if the compilation command changes, ensuring purity. Then, the dependency file and the source file are written in the cache, by default in |robustExternalize/robExt-somehash.tex| and |robustExternalize/robExt-somehash.deps|. Then, the compilation command will be run from the cache folder. At the end, by default, we check if a file |robustExternalize/robExt-somehash.pdf| exists: if not we abort, otherwise we |\input| the file |robustExternalize/robExt-somehash-out.tex| and we run the include command (that includes the pdf by default). As we saw earlier, this command can be customized to use other files. \textbf{Importantly, all the files created during the compilation must be prefixed by} |robExt-somehash|, which can be obtained at runtime using |__ROBEXT_OUTPUT_PREFIX__|. This way, we can easily clean the cache while ensuring maximum purity. In the following, we will denote by |*-foo.bar| the file in:\\ |robustExternalize/robExt-somehash-foo.bar|. Note also that we usually define two names for each function, one normal and one prefixed with |robExt| (or |RobExt|) for environments. In this documentation, we only write the first form, but the second form is kept in case a conflicting package redefines some functions. \subsection{Placeholders}\label{sec:placeholders} Placeholders are the main concept allowing this library to generate the content of a source file based on a template (a template will itself be a placeholder containing other placeholders). A placeholder is a special strings like |__COLOR_IMAGE__| that should start and end with two underscores, ideally containing no space or double underscores inside the name directly (we might use this property even more in a future release to improve further the efficiency). Placeholders are inserted for instance in a string and will be given a value later. This value will be used to replace (recursively) the placeholder in the template. For instance, if a placeholder |__LIKES__| contains |I like __FRUIT__ and __VEGETABLE__|, if the placeholder |__FRUIT__| contains |oranges| and if the placeholder |__VEGETABLE__| contains |salad|, then evaluating |__LIKES__| will output |I like oranges and salad.| Note that you are not strictly forced to follow the above convention (it allows us to optimize the code to find and replace placeholders), but in that case you should enable |not all placeholders have underscores|. Placeholders are local variables (internally just some \LaTeX{} 3 strings). You can therefore define a placeholder in a local group surrounded by brackets |{ ... }| if you want it to have a reduced scope. \subsubsection{Reading a placeholder} \begin{pgfmanualentry} \extractcommand\getPlaceholder\opt{\oarg{new placeholder name}}\marg{name placeholder or string}\@@ \extractcommand\getPlaceholderInResult\opt{\oarg{new placeholder name}}\marg{name placeholder or string}\@@ \extractcommand\getPlaceholderInResultFromList\marg{list,of,placeholders,to,replace}\opt{\oarg{new placeholder name}}\marg{name placeholder or string}\@@ \pgfmanualbody Get the value of a placeholder after replacing (recursively) all the inner placeholders. |\getPlaceholderInResult| puts the resulting string in a \LaTeX{} 3 string |\l_robExt_result_str| and in |\robExtResult|, while |\getPlaceholder| directly outputs this string. You can also put inside the argument any arbitrary string, allowing you, for instance, to concatenate multiple placeholders, copy a placeholder etc. Note that you will get a string, but this string will not be evaluated by \LaTeX{} (see |\evalPlaceholder| for that), for instance math will not be interpreted: \begin{codeexample}[width=0pt,vbox] \setPlaceholder{__MY_PLACEHOLDER__}{Hello __NAME__, I am a template $\delta_n$.} \setPlaceholder{__NAME__}{Alice __NICKNAME__} \setPlaceholder{__NICKNAME__}{the great} The placeholder evaluates to:\\ \texttt{\getPlaceholder{__MY_PLACEHOLDER__}}\\ Combining placeholders produces:\\ \texttt{\getPlaceholder{In ``__MY_PLACEHOLDER__'', the name is __NAME__.}} \end{codeexample} You can also specify the optional argument in order to additionally define a new placeholder containing the resulting string (but you might prefer to use its alias |\setPlaceholderRec| described below): \begin{codeexample}[width=0pt,vbox] \setPlaceholder{__MY_PLACEHOLDER__}{Hello __NAME__, I am a template $\delta_n$.} \setPlaceholder{__NAME__}{Alice __NICKNAME__} \setPlaceholder{__NICKNAME__}{the great} \getPlaceholderInResult[__NEW_PLACEHOLDER__]{In ``__MY_PLACEHOLDER__'', the name is __NAME__.} \printImportedPlaceholdersExceptDefaults \end{codeexample} The variation |\getPlaceholderInResultFromList| allows you to specify a set of placeholder to replace from: \begin{codeexample}[width=0pt,vbox] \setPlaceholder{__MY_PLACEHOLDER__}{Hello __NAME1__, __NAME2__ and __NAME3__, I am a template $\delta_n$.} \setPlaceholder{__NAME1__}{Alice} \setPlaceholder{__NAME2__}{Bob} \setPlaceholder{__NAME3__}{Charlie} \getPlaceholderInResultReplaceFromList{__MY_PLACEHOLDER__,__NAME1__,__NAME2__}[__NEW_PLACEHOLDER__]{ Here we go: __MY_PLACEHOLDER__ } \printImportedPlaceholdersExceptDefaults \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\evalPlaceholder\marg{name placeholder or string}\@@ \pgfmanualbody Evaluate the value of a placeholder after replacing (recursively) all the inner placeholders. You can also put inside any arbitrary string. \begin{codeexample}[width=0pt,vbox] \setPlaceholder{__MY_PLACEHOLDER__}{Hello __NAME__, I am a template $\delta_n$.} \setPlaceholder{__NAME__}{Alice __NICKNAME__} \setPlaceholder{__NICKNAME__}{the great} % The placeholder evaluates to \texttt{\getPlaceholder{__MY_PLACEHOLDER__}}. The placeholder evaluates to:\\ \evalPlaceholder{__MY_PLACEHOLDER__}\\ Combining placeholders produces:\\ \evalPlaceholder{In ``__MY_PLACEHOLDER__'', the name is __NAME__.} \end{codeexample} \end{pgfmanualentry} \subsubsection{List and debug placeholders}\label{sec:listAndDebugPlaceholders} It can sometimes be handy to list all placeholders, print their contents etc (see also \cref{sec:debug}). We list here commands that are mostly useful for debugging purposes. \begin{pgfmanualentry} \extractcommand\printImportedPlaceholdersExceptDefaults\opt{*}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/print imported placeholders except default\@nil \makeatother% \pgfmanualbody Prints the verbatim content of all defined and imported placeholders (without performing any replacement of inner placeholders), except for the placeholders that are defined by default in this library (that we identify as they start with |__ROBEXT_|). The stared version does print the name of the placeholder defined in this library, but not their definition. This is mostly for debugging purposes. \begin{codeexample}[width=0pt,vbox] \placeholderFromContent{__LIKES__}{Hello __NAME__ I am a really basic template $\delta_n$.} \placeholderFromContent{__NAME__}{Alice} \printImportedPlaceholdersExceptDefaults \end{codeexample} Compare with: \begin{codeexample}[width=0pt,vbox] \placeholderFromContent{__LIKES__}{Hello __NAME__ I am a really basic template $\delta_n$.} \placeholderFromContent{__NAME__}{Alice} \printImportedPlaceholdersExceptDefaults* \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\printImportedPlaceholders\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/print imported placeholders\@nil \makeatother% \pgfmanualbody Prints the verbatim content of all defined and imported placeholders (without performing any replacement of inner placeholders), including the placeholders that are defined by default in this library (those starting with |__ROBEXT_|). This is mostly for debugging purposes. Here is the result of |\printImportedPlaceholders|: \printImportedPlaceholders \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\printPlaceholderNoReplacement\marg{name placeholder}\@@ \pgfmanualbody Prints the verbatim content of a given placeholder, without evaluating it and \textbf{without replacing inner placeholders: it is used mostly for debugging purposes} and will be used in this documentation to display the content of the placeholder for educational purposes. The stared version prints it inline. \begin{codeexample}[width=0pt,vbox] \placeholderFromContent{__LIKES__}{Hello __NAME__ I am a really basic template $\delta_n$.} \placeholderFromContent{__NAME__}{Alice} The (unexpanded) template contains \printPlaceholderNoReplacement{__LIKES__}.\\ The (unexpanded) template contains \printPlaceholderNoReplacement*{__LIKES__} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\printPlaceholder\marg{name placeholder}\@@ \pgfmanualbody Like |\printPlaceholderNoReplacement| except that it first replaces the inner placeholders. The stared version prints it inline. \begin{codeexample}[width=0pt,vbox] \placeholderFromContent{__LIKES__}{Hello __NAME__ I am a really basic template $\delta_n$.} \placeholderFromContent{__NAME__}{Alice} The (unexpanded) template contains \printPlaceholder{__LIKES__}.\\ The (unexpanded) template contains \printPlaceholder*{__LIKES__} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\evalPlaceholderNoReplacement\marg{name placeholder}\@@ \pgfmanualbody Evaluates the content of a given placeholder as a \LaTeX{} code, \textbf{without replacing the placeholders contained inside (mostly used for debugging purposes).} \begin{codeexample}[width=0pt,vbox] \placeholderFromContent{__LIKES__}{Hello I am a really basic template $\delta_n$.} The (unexpanded) template evaluates to ``\evalPlaceholderNoReplacement{__LIKES__}''. \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\rescanPlaceholderInVariableNoReplacement\marg{name macro}\marg{name placeholder}\@@ \pgfmanualbody (new v2.0) Create a new macro that executes the \LaTeX{} code in the placeholder. \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\getPlaceholderNoReplacement\marg{name placeholder}\@@ \pgfmanualbody Like |\evalPlaceholderNoReplacement| except that it only outputs the string without evaluating the macros inside. \begin{codeexample}[width=0pt,vbox] \placeholderFromContent{__LIKES__}{Hello __NAME__ I am a really basic template $\delta_n$.} \placeholderFromContent{__NAME__}{Alice} The (unexpanded) template contains \texttt{\getPlaceholderNoReplacement{__LIKES__}} \end{codeexample} Note that this macro is expandable, which can be helpful for instance if you use |siunitx| that expects an expandable content: \begin{codeexample}[width=0pt,vbox] \setPlaceholder{__TEST__}{4.42345} \num[round-mode=figures,round-precision=3]{\getPlaceholderNoReplacement{__TEST__}} \end{codeexample} This is contrary to |\getPlaceholder| that is not expandable, so if you do want expandable |\getPlaceholder| with replacements, you should rather first evaluate the placeholder with |\getPlaceholderInResult{__TEST_VARIABLE__}|: \begin{codeexample}[width=0pt,vbox] \setPlaceholder{__TEST_A__}{4.42345} \setPlaceholder{__TEST__}{__TEST_A__} \getPlaceholderInResult{__TEST_A__} \num[round-mode=figures,round-precision=3]{\robExtResult} \end{codeexample} \end{pgfmanualentry} \subsubsection{Setting a value to a placeholder} \begin{pgfmanualentry} \extractcommand\placeholderFromContent\opt{*}\marg{name placeholder}\marg{content placeholder}\@@ \extractcommand\setPlaceholder\opt{*}\marg{name placeholder}\marg{content placeholder}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/set placeholder=\marg{name placeholder}\marg{content placeholder}\@nil \extractkey/robExt/set placeholder no import=\marg{name placeholder}\marg{content placeholder}\@nil \extractkey/robExt/set placeholder from content=\marg{name placeholder}\marg{content placeholder}\@nil \makeatother% \pgfmanualbody |\placeholderFromContent| (and its alias |\setPlaceholder| and its equivalent pgf styles |/robExt/set placeholder| and |/robExt/set placeholder from content|) is useful to set a value to a given placeholder. \begin{codeexample}[width=0pt,vbox] \placeholderFromContent{__LIKES__}{Hello I am a basic template with math $\delta_n$ and macros \hello} The (unexpanded) template contains \printPlaceholderNoReplacement{__LIKES__} and % after evaluation and setting the value of hello,% \def\hello{Hello my friend!}% you get ``\evalPlaceholder{__LIKES__}''. \end{codeexample} As you can see, \textbf{the precise content is not exactly identical to the original string}: \LaTeX{} comments are removed, spaces are added after macros, some newlines are removed etc. While this is usually not an issue when dealing with \LaTeX{} code, it causes some troubles when dealing with non-\LaTeX{} code. For this reason, we define \textbf{other commands} (see for instance |PlaceholderFromCode| below) that can accept verbatim content; the downside being that \LaTeX{} forbids usage of these verbatim commands inside other macros, so you should always define them at the top level (this seems to be fundamental to how \LaTeX{} works, as any input to a macro gets interpreted first as a \LaTeX{} string, losing all comments for instance). Note that this is not as restrictive as it may sound, as it is always possible to define the needed placeholders before any macro, while using them inside the macro, possibly combining them with other placeholders (defined either before or inside the macro). \end{pgfmanualentry} \begin{pgfmanualentry} \extractenvironement{PlaceholderFromCode}\opt{*}\opt{\oarg{style}}\marg{name placeholder}\@@ \extractenvironement{SetPlaceholderCode}\opt{*}\opt{\oarg{style}}\marg{name placeholder}\@@ \pgfmanualbody These two (aliased) environments are useful to set a verbatim value to a given placeholder: the advantage is that you can put inside any code, including \LaTeX{} comments, the downside is that you cannot use it inside macros and some environments (so you typically define it before the macros and call it inside). % \begin{codeexample}[width=0pt,vbox] % \placeholderFromContent{__PYTHON_CODE__}{} % The (unexpanded) template contains \printPlaceholderNoReplacement{__LIKES__} and % % after evaluation (no replacement), you get ``\evalPlaceholderNoReplacement{__LIKES__}''. % \end{codeexample} \begin{codeexample}[width=0pt,vbox] \begin{PlaceholderFromCode}{__PYTHON_CODE__} def my_function(b): # this is a python code c = {} d[42] = 0 return b \end{PlaceholderFromCode} \printImportedPlaceholdersExceptDefaults \end{codeexample} Note that |PlaceholderFromCode| should not be used inside other macros or inside some environments (notably the ones that need to evaluate the body of the environment, e.g. using |+b| argument or |environ|) as verbatim content is parsed first by the macro, meaning that some characters might be changed or removed. For instance, any percent character would be considered as a comment, removing the rest of the line. However, this should not be be problem if you use it outside of any macro or environment, or if you load it from a file. For instance this code: \begin{verbatim} \begin{PlaceholderFromCode}{__PYTHON_CODE__} def my_function(b): # this is a python code c = {} d[42] = 0 return b % 2 \end{PlaceholderFromCode} \printImportedPlaceholdersExceptDefaults \end{verbatim} would produce: { \begin{PlaceholderFromCode}{__PYTHON_CODE__} def my_function(b): # this is a python code c = {} d[42] = 0 return b % 2 \end{PlaceholderFromCode} \begin{codeexample}[width=0pt,vbox] \printImportedPlaceholdersExceptDefaults \end{codeexample} } Note that of course, you can define a placeholder before a macro and call it inside (explaining how we can generate this documentation). You can use the optional style (only after 2.7; note that it defaults to |defaultPlaceholderFromCodeStyle|, which contains by default |remove leading spaces|) to apply some operations on the style. This way, by default, it will remove any leading indentation, allowing you, for instance, to indent your python code based on the surrounding LaTeX environment): \begin{codeAndResult} \begin{PlaceholderFromCode}{__TMP_MAIN_CONTENT__} # See, you can indent this code with two spaces without disturbing python (that expects no indentation) import matplotlib.pyplot as plt year = [2014, 2015, 2016, 2017, 2018, 2019] tutorial_count = [39, 117, 111, 110, 67, 29] plt.plot(year, tutorial_count, color="#6c3376", linewidth=3) plt.xlabel('Year') plt.ylabel('Number of futurestud.io Tutorials') plt.savefig("__ROBEXT_OUTPUT_PDF__") \end{PlaceholderFromCode} \fbox{\begin{CacheMeNoContent} python, set includegraphics options={width=.8\linewidth}, set main content={__TMP_MAIN_CONTENT__}, \end{CacheMeNoContent}} \end{codeAndResult} Note that the star and no import version does NOT import the placeholder it the main group (unless you try to optimize the compilation time you should not need it, but see \cref{sec:importSystem} for details). \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\placeholderPathFromFilename\opt{*}\marg{name placeholder}\marg{filename}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/set placeholder path from filename=\marg{name placeholder}\marg{filename}\@nil \makeatother% \pgfmanualbody |\placeholderPathFromFilename{__MYLIB__}{mylib.py}| will copy |mylib.py| in the cache (setting its hash depending on its content), and set the content of the placeholder |__MYLIB__| to the \textbf{path} of the library in the cache. Note that the path is relative to the cache folder (it is easier to use for instance if you want to call this library from a code already in the cache). \begin{codeexample}[width=0pt,vbox] \placeholderPathFromFilename{__MYLIB__}{mylib.py} \printImportedPlaceholdersExceptDefaults You can also get the path relative to the root folder:\\ \robExtAddCachePath{\getPlaceholderNoReplacement{__MYLIB__}} \end{codeexample} Note that the star and no import version does NOT import the placeholder it the main group (unless you try to optimize the compilation time you should not need it, but see \cref{sec:importSystem} for details). \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\placeholderFromFileContent\opt{*}\marg{name placeholder}\marg{filename}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/set placeholder from file content=\marg{name placeholder}\marg{filename}\@nil \extractkey/robExt/set placeholder from file content no import=\marg{name placeholder}\marg{filename}\@nil \makeatother% \pgfmanualbody |\placeholderFromFileContent{__MYLIB__}{mylib.py}| will set the content of the placeholder |__MYLIB__| to the content of |mylib.py|. \begin{codeexample}[width=0pt,vbox] \placeholderFromFileContent{__MYLIB__}{mylib.py} \printImportedPlaceholdersExceptDefaults \end{codeexample} Note that the star and no import version does NOT import the placeholder it the main group (unless you try to optimize the compilation time you should not need it, but see \cref{sec:importSystem} for details). \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\placeholderPathFromContent\opt{*}\marg{name placeholder}\opt{\oarg{suffix}}\marg{content}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/set placeholder path from content=\marg{name placeholder}\marg{suffix}\marg{content}\@nil \extractkey/robExt/set placeholder path from content no import=\marg{name placeholder}\marg{suffix}\marg{content}\@nil \makeatother% \pgfmanualbody |\placeholderPathFromContent{__MYLIB__}{some content}| will copy |some content| in a file in the cache (setting its hash depending on its content, the filename will end with |suffix| that defaults to |.tex|), and set the content of the placeholder |__MYLIB__| to the \textbf{path} of the file in the cache. Note that the path is relative to the cache folder (it is easier to use for instance if you want to call this library from a code already in the cache). \begin{codeexample}[width=0pt,vbox] \placeholderPathFromContent{__MYLIB__}[.py]{some contents b} \printImportedPlaceholdersExceptDefaults You can also get the path relative to the root folder:\\ \robExtAddCachePath{\getPlaceholderNoReplacement{__MYLIB__}}\\ As a sanity check, this file contains \verbatiminput{\robExtAddCachePath{\getPlaceholderNoReplacement{__MYLIB__}}} \end{codeexample} Note that the star and no import version does NOT import the placeholder it the main group (unless you try to optimize the compilation time you should not need it, but see \cref{sec:importSystem} for details). \end{pgfmanualentry} \begin{pgfmanualentry} \extractenvironement{PlaceholderPathFromCode}\opt{*}\opt{\oarg{suffix}}\marg{name placeholder}\@@ \pgfmanualbody This environment is similar to |\placeholderPathFromContent| except that it accepts verbatim code (therefore \LaTeX{} comments, newlines etc. will not be removed). However, due to \LaTeX{} limitations, this environment cannot be used inside macros or some environments, or this property will not be preserved. For instance, if you create your placeholder using: \begin{verbatim} \begin{PlaceholderPathFromCode}[.py]{__MYLIB__} def my_function(b): # this is a python code c = {} d[42] = 0 return b % 2 \end{PlaceholderPathFromCode} \end{verbatim} %% The code cannot be placed inside codeexample as it needs to parse the body: \begin{PlaceholderPathFromCode}[.py]{__MYLIB__} def my_function(b): # this is a python code c = {} d[42] = 0 return b % 2 \end{PlaceholderPathFromCode} You can then use it like: \begin{codeexample}[width=0pt,vbox] \printImportedPlaceholdersExceptDefaults You can also get the path relative to the root folder:\\ \robExtAddCachePath{\getPlaceholderNoReplacement{__MYLIB__}}\\ As a sanity check, this file contains \verbatiminput{\robExtAddCachePath{\getPlaceholderNoReplacement{__MYLIB__}}} \end{codeexample} Note that the star version does NOT import the placeholder it the main group (unless you try to optimize the compilation time you should not need it, but see \cref{sec:importSystem} for details). \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\copyPlaceholder\opt{*}\marg{new placeholder}\marg{old placeholder}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/copy placeholder=\marg{new placeholder}\marg{old placeholder}\@nil \extractkey/robExt/copy placeholder no import=\marg{new placeholder}\marg{old placeholder}\@nil \makeatother% \pgfmanualbody This creates a new placeholder with the content of |old placeholder|. Note that this is different from:\\ |\setPlaceholder{new placeholder}{old placeholder}|\\ because if we modify |old placeholder|, this will not affect |new placeholder|. \begin{codeexample}[width=0pt,vbox] \setPlaceholder{__MY_CONTENT__}{Some content} \copyPlaceholder{__MY_OLD_CONTENT__}{__MY_CONTENT__} \setPlaceholder{__MY_CONTENT__}{The content used to be __MY_OLD_CONTENT__} \printImportedPlaceholdersExceptDefaults \end{codeexample} Note that the star and no import version does NOT import the placeholder it the main group (unless you try to optimize the compilation time you should not need it, but see \cref{sec:importSystem} for details). \end{pgfmanualentry} Now, we see how we can define a placeholder recursively, by giving it a value based on its previous value (useful for instance in order to add stuff to it). \begin{pgfmanualentry} \extractcommand\setPlaceholderRec\marg{new placeholder}\marg{content with placeholder}\@@ \extractcommand\setPlaceholderRecReplaceFromList\marg{list,of,placeholder,to,replace}\marg{new placeholder}\marg{content with placeholder}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/set placeholder rec=\marg{name placeholder}\marg{content placeholder}\@nil \extractkey/robExt/set placeholder rec replace from list=\marg{list,of,placeholder,to,replace}\marg{name placeholder}\marg{content placeholder}\@nil \makeatother% \pgfmanualbody |\setPlaceholderRec{foo}{bar}| is actually an alias for |\getPlaceholderInResult[foo]{bar}|. Note that contrary to |\setPlaceholder|, it recursively replaces all inner placeholders. This is particularly useful to add stuff to an existing (or not) placeholder: \begin{codeexample}[width=0pt,vbox] \setPlaceholderRec{__MY_COMMAND__}{pdflatex} \setPlaceholderRec{__MY_COMMAND__}{__MY_COMMAND__ myfile} \printImportedPlaceholdersExceptDefaults \end{codeexample} Note that the if the placeholder content contains at the end the placeholder name, we will automatically remove it to avoid infinite recursion at evaluation time. This has the benefit that you can add something to a placeholder even if this placeholder does not exists yet (in which case it will be understood as the empty string): \begin{codeexample}[width=0pt,vbox] \setPlaceholderRec{__COMMAND_ARGS__}{__COMMAND_ARGS__ -l} \setPlaceholderRec{__COMMAND_ARGS__}{__COMMAND_ARGS__ -s} \printImportedPlaceholdersExceptDefaults \end{codeexample} The variation |\setPlaceholderRecReplaceFromList| allows us to specify a subset of placeholder that will be allowed to be expanded, and is an alias for |\getPlaceholderInResultReplaceFromList| (except that the optional argument is mandatory): \begin{codeexample}[width=0pt,vbox] \placeholderFromContent{__MY_PLACEHOLDER__}{Hello __NAME1__, __NAME2__ and __NAME3__, I am a template $\delta_n$. } \placeholderFromContent{__NAME1__}{Alice} \placeholderFromContent{__NAME2__}{Bob} \placeholderFromContent{__NAME3__}{Charlie} \setPlaceholderRecReplaceFromList{__MY_PLACEHOLDER__,__NAME2__,__NAME3__}{__OTHER_PLACEHOLDER__}{ Here we go: __MY_PLACEHOLDER__ } \printImportedPlaceholdersExceptDefaults \end{codeexample} Note that the star and no import version does NOT import the placeholder it the main group (unless you try to optimize the compilation time you should not need it, but see \cref{sec:importSystem} for details). \end{pgfmanualentry} Note that sometimes, you might not want to use |\setPlaceholderRec| to simply append some data to the placeholder as it will also evaluate the inner placeholders (meaning that you will not be able to redefine them later). For this reason, we also provide functions to add something to the placeholder without evaluating it first: \begin{pgfmanualentry} \extractcommand\addToPlaceholder\opt{*}\marg{placeholder}\marg{content to add}\@@ \extractcommand\addToPlaceholderNoImport\opt{*}\marg{placeholder}\marg{content to add}\@@ \extractcommand\addBeforePlaceholder\opt{*}\marg{placeholder}\marg{content to add}\@@ \extractcommand\addBeforePlaceholderNoImport\opt{*}\marg{placeholder}\marg{content to add}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/add to placeholder=\marg{name placeholder}\marg{content to add}\@nil \extractkey/robExt/add to placeholder no space=\marg{name placeholder}\marg{content to add}\@nil \extractkey/robExt/add before placeholder=\marg{name placeholder}\marg{content to add}\@nil \extractkey/robExt/add before placeholder no space=\marg{name placeholder}\marg{content to add}\@nil \extractkey/robExt/add before main content=\marg{name placeholder}\marg{content to add}\@nil \extractkey/robExt/add to placeholder no import=\marg{name placeholder}\marg{content to add}\@nil \extractkey/robExt/add to placeholder no space no import=\marg{name placeholder}\marg{content to add}\@nil \extractkey/robExt/add before placeholder no import=\marg{name placeholder}\marg{content to add}\@nil \extractkey/robExt/add before placeholder no space no import=\marg{name placeholder}\marg{content to add}\@nil \extractkey/robExt/add before main content no import=\marg{name placeholder}\marg{content to add}\@nil \makeatother% \pgfmanualbody |\addToPlaceholder{foo}{bar}| adds |bar| at the end of the placeholder |foo| (by default it also adds a space, unless you use the star version), creating it if it does not exist (the |before| variants add the content\dots{} before). \begin{codeexample}[width=0pt,vbox] \setPlaceholder{__ENGINE__}{pdflatex} \setPlaceholder{__COMMAND__}{__ENGINE__ --option} \addToPlaceholder{__COMMAND__}{--other} \addToPlaceholder*{__COMMAND__}{-option} \addBeforePlaceholder{__COMMAND__}{time} \printImportedPlaceholdersExceptDefaults \end{codeexample} |add before main content| is a particular case where the placeholder is |__ROBEXT_MAIN_CONTENT__|. It is practical if you want to define for instance a macro, but in a way that even if you disable externalization, the command should still compile (if you define the macro in the preamble, it will not be added when disabling externalization). For instance: \begin{codeexample}[width=0pt,vbox] \robExtConfigure{ new preset={my preset}{latex, add before main content={\def\hello#1{Hello #1.}}}, } \cacheMe[my preset]{Here I am cached (\hello{Alice})} and \cacheMe[my preset, disable externalization]{Here I am not (\hello{Bob}).} \end{codeexample} The |no import| versions do not import the placeholder in the current group (only needed if you want to optimize the compilation time, see \cref{sec:importSystem} for details). \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\placeholderFromString\opt{*}\marg{latex3 string}\@@ \extractcommand\setPlaceholderFromString\opt{*}\marg{latex3 string}\@@ \pgfmanualbody (new in v2.0) This allows you to assign an existing \LaTeX{}3 string to a placeholder (the star version does not import the placeholder, see \cref{sec:importSystem}). \begin{codeexample}[width=0pt,vbox] \ExplSyntaxOn \setPlaceholderFromString{__my_percent_string__}{\c_percent_str} \printImportedPlaceholdersExceptDefaults \ExplSyntaxOff \end{codeexample} We provide a list of placeholders that are useful to escape parts of the strings (but you should not really need them, if you need weird characters like percent, most of the time you want to use placeholderFromCode): \begin{codeexample}[width=0pt,vbox] \printPlaceholder{ String containing __ROBEXT_LEFT_BRACE__, __ROBEXT_RIGHT_BRACE__, __ROBEXT_BACKSLASH__, __ROBEXT_HASH__, __ROBEXT_UNDERSCORE__, __ROBEXT_PERCENT__. } \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\placeholderReplaceInplace\marg{placeholder}\marg{from}\marg{to}\@@ \extractcommand\placeholderReplaceInplaceEval\marg{placeholder}\marg{from}\marg{to}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/placeholder replace in place=\marg{placeholder}\marg{from}\marg{to}\@nil \extractkey/robExt/placeholder replace in place eval=\marg{placeholder}\marg{from}\marg{to}\@nil \makeatother% \pgfmanualbody (new in v2.0) This allows you to replace a value in a placeholder. The |eval| variation first evaluates the string. \begin{codeexample}[width=0pt,vbox] \def\nameFrom{Bob} \def\nameTo{Dylan} \robExtConfigure{ set placeholder={__NAMES__}{Hello Alice and Bob.}, placeholder replace in place={__NAMES__}{Alice}{Charlie}, placeholder replace in place eval={__NAMES__}{\nameFrom}{\nameTo}, } \printImportedPlaceholdersExceptDefaults \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\placeholderRemoveSpacesUntil\marg{placeholder}\opt{\oarg{nb of additional letters to remove}}\marg{separator}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/placeholder remove spaces until=\marg{placeholder}\marg{separator}\@nil \extractkey/robExt/remove spaces until=\marg{separator}\@nil \extractkey/robExt/placeholder strictly remove spaces until=\marg{placeholder}\marg{separator}\@nil \extractkey/robExt/strictly remove spaces until=\marg{separator}\@nil \makeatother% \pgfmanualbody (new in v2.6) Removes all the spaces before |separator|, together with the separator itself and the next character (unless you use the |strictly| version, in that case the next character is not dropped, or if you change the default value of |additional letters to remove| that defaults to 1). Note that |prepend all lines| automatically apply it to the |__ROBEXT_MAIN_CONTENT_ORIG__| placeholder, i.e. the text typed by the user. This is useful, for instance, if your editor tends to automatically remove the indentation of your python code while indenting automatically the document. \begin{codeAndResult} %% See that the python code can be freely indented now: \begin{CacheMeCode}{python print code and result, remove spaces until={>>>}} >>> for i in range(2): >>> print(f"Hello {i}") \end{CacheMeCode} \end{codeAndResult} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\placeholderPrependAllLines\marg{placeholder}\marg{text}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/placeholder prepend all lines=\marg{placeholder}\marg{text}\@nil \extractkey/robExt/prepend all lines=\marg{text}\@nil \makeatother% \pgfmanualbody (new in v2.6) Prepend all lines with |text|. This can be useful, for instance, if you want to indent a whole block of python code automatically. Note that the text is evaluated, so in case you want to insert multiple spaces, use |\space| like in:\\ |prepend all lines={\space\space\space\space}|\\ \begin{codeAndResult} \begin{PlaceholderFromCode}{__TMP__} >>> def f(x): % hye # 42 >>> return 42 >>> >>> def g(z): >>> return 48 \end{PlaceholderFromCode} \placeholderRemoveSpacesUntil{__TMP__}{>>>} \placeholderPrependAllLines{__TMP__}{My text\space\space\space\space} \printPlaceholderNoReplacement{__TMP__} \end{codeAndResult} Note that |prepend all lines| automatically adds the text to the |__ROBEXT_MAIN_CONTENT_ORIG__| placeholder, i.e. the text typed by the user. \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\placeholderRemoveLeadingSpaces\marg{placeholder}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/placeholder remove leading spaces=\marg{placeholder}\@nil \extractkey/robExt/remove leading spaces\@nil \extractkey/robExt/remove leading spaces if not disabled\@nil \extractkey/robExt/do not remove leading spaces\@nil \makeatother% \pgfmanualbody (new in v2.6) Remove the indentation of a placeholder (the |__ROBEXT_MAIN_CONTENT_ORIG__| placeholder, containing the content typed by the user, is used when not specified), by removing on each line the highest possible number of leading spaces (the same number of spaces is removed on all lines). \begin{codeAndResult} \begin{PlaceholderFromCode}{__TMP__} def f(x): % hye # 42 return 42 def g(z): return 48 \end{PlaceholderFromCode} \placeholderRemoveLeadingSpaces{__TMP__} \printPlaceholderNoReplacement{__TMP__} \end{codeAndResult} Note that |remove leading spaces if not disabled| is like |remove leading spaces|, except that it is not run if the |do not remove leading spaces| style is applied before. This is used inside some styles like |python| that cares about indentation to remove the leading spaces (before v2.6, it was forbidden to add any indentation to the python code), while allowing the user to disable it temporarily (who knows why?) by using the style: \begin{verbatim} do not remove leading spaces, python \end{verbatim} Since |remove leading spaces if not disabled| is enabled by default on python, it is therefore fine to indent your python code like in: \begin{codeAndResult} \begin{CacheMeCode}{python print code and result} for i in range(3): print(f"Hello {i}") \end{CacheMeCode} \end{codeAndResult} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\placeholderHalveNumberHashesInplace\marg{placeholder}\@@ \extractcommand\placeholderDoubleNumberHashesInplace\marg{placeholder}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/placeholder halve number hashes in place=\marg{placeholder}\@nil \extractkey/robExt/placeholder double number hashes in place=\marg{placeholder}\@nil \makeatother% \pgfmanualbody (new in v2.0) This allows you to either turn any |##| into |#| or the other way around (may be practical when dealing with arguments to functions). \begin{codeexample}[width=0pt,vbox] \robExtConfigure{ set placeholder={__DEMO__}{\def\hey##1{Hey ##1.}}, placeholder halve number hashes in place={__DEMO__}, } \printImportedPlaceholdersExceptDefaults \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\removePlaceholder\marg{placeholder}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/remove placeholder=\marg{placeholder}\@nil \extractkey/robExt/remove placeholders=\marg{list,of,placeholder}\@nil \makeatother% \pgfmanualbody (new in v2.0) Remove placeholders. \begin{codeexample}[width=0pt,vbox] \robExtConfigure{ set placeholder={__DEMOA__}{Alice}, set placeholder={__DEMOB__}{Bob}, set placeholder={__DEMOC__}{Charlie}, remove placeholders={__DEMOA__,__DEMOC__}, } \printImportedPlaceholdersExceptDefaults \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\evalPlaceholderInplace\marg{name placeholder}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/eval placeholder inplace=\marg{name placeholder}\@nil \makeatother% \pgfmanualbody This command will update (inplace) the content of a macro by first replacing recursively the placeholders, and finally by expanding the \LaTeX{} macros. \begin{codeexample}[width=0pt,vbox] \def\mymacro{Initial value} \placeholderFromContent{__MACRO_NOT_EVALUATED__}{\mymacro} \placeholderFromContent{__MACRO_EVALUATED__}{\mymacro} \evalPlaceholderInplace{__MACRO_EVALUATED__} \printImportedPlaceholdersExceptDefaults \def\mymacro{Final value} Compare \evalPlaceholder{__MACRO_EVALUATED__} and \evalPlaceholder{__MACRO_NOT_EVALUATED__}. \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/set placeholder eval=\marg{name placeholder}\marg{content placeholder}\@nil \makeatother% \pgfmanualbody Alias for |\setPlaceholderRec{#1}{#2}\evalPlaceholderInplace{#1}|: set and evaluate recursively the placeholders and macros. This can be practical to pass the value of a counter/macro to the template (of course, if this value is fixed, you can also directly load it from the preambule): \begin{codeexample}[width=0pt,vbox] \begin{CacheMe}{tikzpicture, set placeholder eval={__thepage__}{\thepage}} \node[rounded corners, fill=red]{The current page is __thepage__.}; \end{CacheMe} \end{codeexample} Note that this works well for commands that expand completely, but some more complex commands might not expand properly (like |cref|). I need to investigate how to solve this issue, meanwhile you can still disable externalization for these pictures. \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\lenToCm\opt{\oarg{unit}}\marg{expression}\@@ \extractcommand\lenToCmNoUnit\opt{\oarg{unit}}\marg{expression}\@@ \pgfmanualbody |\lenToCm{.75\linewidth}| will output the length of |.75\linewidth| in cm by default, like |10cm|. If you want to change the unit, you can use the optional argument to specify a different unit (like |in| for inches, see |\dim_to_decimal_in_unit| for details). You can remove the unit at the end using |\lenToCmNoUnit{.75\linewidth}| (needs v2.6) to obtain something like |10| instead of |10cm|. These commands are helpful with |set placeholder eval| to send length from \LaTeX{} to your environment: \begin{codeAndResult} \begin{CacheMeCode}{python, set placeholder eval={__LINEWIDTH__}{\lenToCmNoUnit[in]{.75\linewidth}}} import matplotlib.pyplot as plt import matplotlib from matplotlib.pyplot import figure figure(figsize=(__LINEWIDTH__, __LINEWIDTH__)) year = [2014, 2015, 2016, 2017, 2018, 2019] tutorial_count = [39, 117, 111, 110, 67, 29] plt.plot(year, tutorial_count, color="#6c3376", linewidth=2) plt.title("Simple plot") plt.xlabel('Year') plt.ylabel('Number of futurestud.io Tutorials') print(get_filename_from_extension(".pgf")) # https://stackoverflow.com/a/52587591/4987648 plt.savefig("__ROBEXT_OUTPUT_PDF__", bbox_inches="tight") \end{CacheMeCode} \end{codeAndResult} \end{pgfmanualentry} \subsubsection{Groups: the import system}\label{sec:importSystem} (This whole system was added v2.0.) In order to replace placeholders, we need to know their list, but to get the best possible performance we do not maintain a single list of all placeholders since \LaTeX{} is quite slow when doing list manipulations. Therefore, we group them inside smaller groups (e.g.\ one group for |latex|, one group for |python| etc), and the user is free to import groups. By default, new placeholders are imported inside the |main| group, and this group will be used when doing placeholder replacements. Note that for most of the case, we define a star variant of functions like |\setPlaceholder*{__FOO__}{bar}| (or |set placeholder no import| for styles) in order to create a placeholder without importing it (but this is certainly not used a lot by end users unless they want to make them even faster). Note that basically all commands to create a placeholder have a command/environment where you add a star like |\setPlaceholder*{}| or a style where you add |no import| like |set placeholder no import| in order to create a placeholder that is not imported at all. \begin{pgfmanualentry} \extractcommand\importPlaceholder\marg{name placeholder}\@@ \extractcommand\importPlaceholdersFromGroup\marg{name placeholder}\@@ \extractcommand\importAllPlaceholders\@@ \extractcommand\importPlaceholderFirst\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/import placeholder=\marg{name placeholder}\@nil \extractkey/robExt/import placeholders from group=\marg{name group}\@nil \extractkey/robExt/import all placeholders\@nil \extractkey/robExt/import placeholder first=\marg{name placeholders}\@nil \makeatother% \pgfmanualbody Import placeholders, either individually, from a group of placeholders (exists either in style form or command), or from all registered groups (warning: you should avoid importing all placeholders if care too much about efficiency, as this can significantly slow down the replacement procedure). The |first| variant inserts the placeholder at the beginning of the import list, which can speed up compilation when this placeholder must be replaced first (but not so useful for basic usage). \begin{codeexample}[width=0pt,vbox] \robExtConfigure{ set placeholder no import={__name__}{Alice}, set placeholder no import={__name2__}{Bob}, print imported placeholders except default, import placeholder={__name__}, print imported placeholders except default, print imported placeholders, import placeholders from group={latex}, print imported placeholders, } \end{codeexample} \begin{codeexample}[width=0pt,vbox] \printPlaceholder{First try: __ROBEXT_LATEX_TRIM_LENGTH__} \importAllPlaceholders \printPlaceholder{Second try: __ROBEXT_LATEX_TRIM_LENGTH__} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\clearImportedPlaceholders\marg{name placeholder}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/clear imported placeholders\@nil \makeatother% \pgfmanualbody Clear all imported placeholders (comma separated list of placeholders) \begin{codeexample}[width=0pt,vbox] \robExtConfigure{ set placeholder={__name__}{Alice}, print imported placeholders except default, clear imported placeholders, print imported placeholders except default, } \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\removeImportedPlaceholder\marg{name placeholder}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/remove imported placeholders\@nil \makeatother% \pgfmanualbody Comma separated list of placeholders to remove from the list of imported placeholders. \begin{codeexample}[width=0pt,vbox] \robExtConfigure{ set placeholder={__name__}{Alice}, set placeholder={__name2__}{Alice}, print imported placeholders except default, remove imported placeholders={__name__,__name2__}, print imported placeholders except default, } \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\printAllRegisteredGroups\@@ \extractcommand\printAllRegisteredGroupsAndPlaceholders\opt{*}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/print all registered groups\@nil \extractkey/robExt/print all registered groups and placeholders\@nil \extractkey/robExt/show all registered groups\@nil \extractkey/robExt/show all registered groups and placeholders\@nil \makeatother% \pgfmanualbody Print (or shows in the console for the alternative versions) all registered groups of placeholders (possibly with the placeholders they contain), mostly for debugging purpose, with the star prints also the content of the inner placeholders. \begin{codeexample}[width=0pt,vbox] \printAllRegisteredGroups \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\printGroupPlaceholders\opt{*}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/print group placeholders=\marg{name group}\@nil \makeatother% \pgfmanualbody Print all placeholders (star = with content) of a given group. \begin{codeexample}[width=0pt,vbox] \printGroupPlaceholders*{bash} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\printAllRegisteredGroupsAndPlaceholders\opt{*}\@@ \pgfmanualbody Print all registered groups of placeholders with their inner content, mostly for debugging purpose, with the star prints only the content of the inner placeholders. \begin{codeexample}[width=0pt,vbox] \printAllRegisteredGroupsAndPlaceholders \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\newGroupPlaceholders\marg{name group}\@@ \extractcommand\addPlaceholderToGroup\marg{name group}\marg{list,of,placeholders}\@@ \extractcommand\addPlaceholdersToGroup\marg{name group}\marg{list,of,placeholders}\@@ \extractcommand\removePlaceholderFromGroup\marg{name group}\marg{list,of,placeholders}\@@ \extractcommand\removePlaceholdersFromGroup\marg{name group}\marg{list,of,placeholders}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/new group placeholders=\marg{name group}\@nil \extractkey/robExt/add placeholder to group=\marg{name group}\marg{name placeholder}\@nil \extractkey/robExt/add placeholders to group=\marg{name group}\marg{list,of,placeholders}\@nil \extractkey/robExt/add placeholders to group=\marg{name group}\marg{list,of,placeholders}\@nil \extractkey/robExt/remove placeholders from group=\marg{name group}\marg{list,of,placeholders}\@nil \extractkey/robExt/remove placeholder from group=\marg{name group}\marg{list,of,placeholders}\@nil \makeatother% \pgfmanualbody Create a new group and add placeholders to it (you can put multiple placeholders separated by commas). \begin{codeAndResult} \robExtConfigure{ new group placeholders={my dummy group}, set placeholder no import={__name__}{Alice}, add placeholders to group={my dummy group}{__name__}, } \printGroupPlaceholders{my dummy group} \removePlaceholdersFromGroup{my dummy group}{__name__} \printGroupPlaceholders{my dummy group} \end{codeAndResult} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\clearGroupPlaceholders\marg{name group}\@@ \makeatletter% \def\extrakeytext{style, } % \extractkey/robExt/new group placeholders=\marg{name group}\@nil \makeatother% \pgfmanualbody Remove all elements in a group. \begin{codeexample}[vbox] \robExtConfigure{ new group placeholders={my dummy group}, set placeholder no import={__name__}{Alice}, add placeholders to group={my dummy group}{__name__}, } \clearGroupPlaceholders{my dummy group} \printGroupPlaceholders{my dummy group} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\copyGroupPlaceholders\marg{name group}\marg{name group to copy}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/copy group placeholders=\marg{name group}\marg{name group to copy}\@nil \makeatother% \pgfmanualbody Remove all elements in a group. \begin{codeexample}[vbox] \robExtConfigure{ new group placeholders={my dummy group}, set placeholder no import={__name__}{Alice}, add placeholders to group={my dummy group}{__name__}, } \copyGroupPlaceholders{my copy}{my dummy group} \printGroupPlaceholders{my copy} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\appendGroupPlaceholders\marg{name group}\marg{name group to append}\@@ \extractcommand\appendBeforeGroupPlaceholders\marg{name group}\marg{name group to append}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/append group placeholders=\marg{name group}\marg{name group to copy}\@nil \extractkey/robExt/append before group placeholders=\marg{name group}\marg{name group to copy}\@nil \makeatother% \pgfmanualbody Append a group to another group (after or before). \begin{codeexample}[vbox] \robExtConfigure{ new group placeholders={my dummy group}, set placeholder no import={__name__}{Alice}, add placeholders to group={my dummy group}{__name__}, new group placeholders={my dummy group2}, set placeholder no import={__name2__}{Bob}, add placeholders to group={my dummy group2}{__name2__}, append group placeholders={my dummy group}{my dummy group2}, } \printGroupPlaceholders{my dummy group} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\importPlaceholdersFromGroup\marg{name group}\@@ \extractcommand\importAllPlaceholders\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/import placeholders from group=\marg{name group}\@nil \extractkey/robExt/import all placeholders\@nil \makeatother% \pgfmanualbody Import a group/all registered groups (note that this is equivalent to appending to the group called |main|). Note that for the later, you should be sure that the group is ``registered'', which is the case if you created it via |new group placeholders|. \begin{codeexample}[vbox] \robExtConfigure{ new group placeholders={my dummy group}, set placeholder no import={__name__}{Alice}, add placeholder to group={my dummy group}{__name__}, import all placeholders, } \printImportedPlaceholdersExceptDefaults \end{codeexample} \end{pgfmanualentry} \subsection{Caching a content} \subsubsection{Basics} \begin{pgfmanualentry} \extractcommand\cacheMe\opt{\oarg{preset style}}\marg{content to cache}\@@ \extractenvironement{CacheMe}\marg{preset style}\@@ \pgfmanualbody This command (and its environment alias) is the main entry point if you want to cache the result of a file. The preset style is a pgfkeys-based style that is used to configure the template that is used, the compilation command, and more. You can either inline the style, or use some presets that configure the style automatically. After evaluating the style, the placeholders |__ROBEXT_TEMPLATE__| (containing the content of the file) and |__ROBEXT_COMPILATION_COMMAND__| (containing the compilation command run in the cache folder, that can use other placeholders internally like |__ROBEXT_SOURCE_FILE__| to get the path to the source file) should be set. Note that we provide some basic styles that allow settings these placeholders easily. See \cref{sec:placeholders} for a list of existing placeholders and presets. The placeholder |__ROBEXT_MAIN_CONTENT_ORIG__| will automatically be set by this command (or environment) so that it equals the content of the second argument (or the body of the environment). By default, |__ROBEXT_MAIN_CONTENT__| will point to |__ROBEXT_MAIN_CONTENT_ORIG__|, possibly wrapping it inside an environment like |\begin{tikzpicture}| (most of the time, you want to modify and display |__ROBEXT_MAIN_CONTENT__| rather than the |_ORIG_| to easily recover the input of the user). This style can also configure the command to use to include the file and more. By default it will insert the compiled PDF, making sure that the depth is respected (internally, we read the depth from an aux file created by our \LaTeX{} preset), but you can easily change it to anything you like. For an educational purpose, we write here an example that does not exploit any preset. In practice, we recommend however to use our presets, or to define new presets based on our presets (see below for examples). \begin{codeexample}[width=0pt,vbox] \begin{CacheMe}{set template={ \documentclass{standalone} \begin{document} __ROBEXT_MAIN_CONTENT__ \end{document} }, set compilation command={pdflatex -shell-escape -halt-on-error "__ROBEXT_SOURCE_FILE__"}, custom include command={% \includegraphics[width=4cm,angle=45]{\robExtAddCachePathAndName{\robExtFinalHash.pdf}}% }, } This content is cached $\delta$. \end{CacheMe} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\robExtConfigure\marg{preset style}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/new preset=\marg{name preset}\marg{preset options}\@nil \extractkey/robExt/add to preset=\marg{name preset}\marg{preset options}\@nil \extractkey/robExt/add before preset=\marg{name preset}\marg{preset options}\@nil \makeatother% \pgfmanualbody You can then create your own style (or preset) in |\robExtConfigure| (that is basically an alias for |\pgfkeys{/robExt/.cd,#1}|) containing your template, add your own placeholders and commands to configure them etc. We provide two helper functions since v2.0:\\ |new preset={your preset}{your configuration}|\\ and\\ |add to preset={your preset}{your configuration}|\\ in order to create/modify the presets. You can also use |my preset/.style| or\\ |my preset/.append style|\\ to configure them instead, but in that case make sure to double the number of hashes like in |\def\mymacro##1{hello ##1}|, as the |#1| in |\def\mymacro#1{hello #1}.| would be understood as the (non-existent) argument of |my preset|. \begin{codeexample}[width=0pt,vbox] %% Define your presets once: \robExtConfigure{% new preset={my latex preset}{ %% Create a default value for my new placeholders: set placeholder={__MY_COLOR__}{red}, set placeholder={__MY_ANGLE__}{45}, % We can also create custom commands to "hide" the notion of placeholder set my angle/.style={ set placeholder={__MY_ANGLE__}{#1} }, set template={ \documentclass{standalone} \usepackage{xcolor} \begin{document} \color{__MY_COLOR__}__ROBEXT_MAIN_CONTENT__ \end{document} }, set compilation command={pdflatex -shell-escape -halt-on-error "__ROBEXT_SOURCE_FILE__"}, custom include command={% % The include command is a regular LaTeX command, but using % \evalPlaceholder avoids the need to play with expandafter, getPlaceholder etc... \evalPlaceholder{% \includegraphics[width=4cm,angle=__MY_ANGLE__,origin=c]{% \robExtAddCachePathAndName{\robExtFinalHash.pdf}% }% }% }, }, } % Reuse them later... \begin{CacheMe}{my latex preset} This content is cached $\delta$. \end{CacheMe} % And configure them at will \begin{CacheMe}{my latex preset, set placeholder={__MY_COLOR__}{green}, set my angle=-45} This content is cached $\delta$. \end{CacheMe} \end{codeexample} The |before| variation (from 2.6) simply adds the style before the existing style. May be useful to disable default options of a style. \end{pgfmanualentry} \begin{pgfmanualentry} \extractenvironement{CacheMeCode}\marg{preset style}\@@ \pgfmanualbody Like |CacheMe|, except that the code is read verbatim by \LaTeX{}. This way, you can put non-\LaTeX{} code inside safely, but you will not be able to use it inside a macro or some environments that read their body. Here is an example where we define an environment that automatically import matplotlib, save the figure, and insert it into a figure. Note that we define in this example new commands to type |set caption=foo| instead of |set placeholder={__FIG_CAPTION__}{foo}|. %% codeexample cannot deal with verbatim content \begin{codeexample}[code only] %% Define the python code to use as a template %% (impossible to define it in \robExtConfigure directly since %% it is a verbatim environment) \begin{PlaceholderFromCode}{__MY_MATPLOTLIB_TEMPLATE__} import matplotlib.pyplot as plt import sys __ROBEXT_MAIN_CONTENT__ plt.savefig("__ROBEXT_OUTPUT_PDF__") \end{PlaceholderFromCode} % Create a new preset called matplotlib \robExtConfigure{ new preset={matplotlib figure}{ set template={__MY_MATPLOTLIB_TEMPLATE__}, set compilation command={python "__ROBEXT_SOURCE_FILE__"}, set caption/.style={ set placeholder={__FIG_CAPTION__}{#1} }, set label/.style={ set placeholder={__FIG_LABEL__}{#1} }, set includegraphics options/.style={ set placeholder={__INCLUDEGRAPHICS_OPTIONS__}{#1} }, set caption={}, set label={}, set includegraphics options={width=1cm}, custom include command={% \evalPlaceholder{% \begin{figure} \centering \includegraphics[__INCLUDEGRAPHICS_OPTIONS__]{\robExtAddCachePathAndName{\robExtFinalHash.pdf}}% \caption{__FIG_CAPTION__a} \label{__FIG_LABEL__} \end{figure}% }% }, }, } %% Use it \begin{CacheMeCode}{matplotlib figure, set includegraphics options={width=.8\linewidth}, set caption={Hello}} year = [2014, 2015, 2016, 2017, 2018, 2019] tutorial_count = [39, 117, 111, 110, 67, 29] plt.plot(year, tutorial_count, color="#6c3376", linewidth=3) plt.xlabel('Year') plt.ylabel('Number of futurestud.io Tutorials') \end{CacheMeCode} \end{codeexample} %% Define the python code to use as a template %% (impossible to define it in \robExtConfigure directly since %% it is a verbatim environment) \begin{PlaceholderFromCode}{__MY_MATPLOTLIB_TEMPLATE__} import matplotlib.pyplot as plt import sys __ROBEXT_MAIN_CONTENT__ plt.savefig("__ROBEXT_OUTPUT_PDF__") \end{PlaceholderFromCode} % Create a new preset called matplotlib \robExtConfigure{ new preset={matplotlib figure}{ set template={__MY_MATPLOTLIB_TEMPLATE__}, set compilation command={python "__ROBEXT_SOURCE_FILE__"}, set caption/.style={ set placeholder={__FIG_CAPTION__}{#1} }, set label/.style={ set placeholder={__FIG_LABEL__}{#1} }, set includegraphics options/.style={ set placeholder={__INCLUDEGRAPHICS_OPTIONS__}{#1} }, set caption={}, set label={}, set includegraphics options={width=1cm}, custom include command={% \evalPlaceholder{% \begin{figure} \centering \includegraphics[__INCLUDEGRAPHICS_OPTIONS__]{\robExtAddCachePathAndName{\robExtFinalHash.pdf}}% \caption{__FIG_CAPTION__} \label{__FIG_LABEL__} \end{figure}% }% }, }, } %% Use it \begin{CacheMeCode}{matplotlib figure, set includegraphics options={width=.8\linewidth}, set caption={An example to show how matplotlib pictures can be inserted}} year = [2014, 2015, 2016, 2017, 2018, 2019] tutorial_count = [39, 117, 111, 110, 67, 29] plt.plot(year, tutorial_count, color="#6c3376", linewidth=3) plt.xlabel('Year') plt.ylabel('Number of futurestud.io Tutorials') \end{CacheMeCode} Note that as we explained it before, due to \LaTeX{} limitations, it is impossible to call |CacheMeCode| inside macros and inside some environments that evaluate their body. To avoid that issue, it is always possible to define the macro before and call it inside. We will exemplify this on the previous example, but note that \textbf{this example is only for educational purposes} since the environment |figure| does not evaluate its body, and |CacheMeCode| can therefore safely be used inside without using this trickery: \begin{codeexample}[code only] %% Define the python code to use as a template %% (impossible to define it in \robExtConfigure directly since %% it is a verbatim environment) \begin{PlaceholderFromCode}{__MY_MATPLOTLIB_TEMPLATE__} import matplotlib.pyplot as plt import sys __ROBEXT_MAIN_CONTENT__ plt.savefig("__ROBEXT_OUTPUT_PDF__") \end{PlaceholderFromCode} % Create a new preset called matplotlib \robExtConfigure{ new preset={matplotlib}{ set template={__MY_MATPLOTLIB_TEMPLATE__}, set compilation command={python "__ROBEXT_SOURCE_FILE__"}, set includegraphics options/.style={ set placeholder={__INCLUDEGRAPHICS_OPTIONS__}{#1} }, set includegraphics options={width=1cm}, custom include command={% \evalPlaceholder{% \includegraphics[__INCLUDEGRAPHICS_OPTIONS__]{\robExtAddCachePathAndName{\robExtFinalHash.pdf}}% }% }, }, } %% You cannot use CacheMeCode inside some macros or environments due to fundamental LaTeX limitations. %% But you can always define them before, and call them inside: \begin{SetPlaceholderCode}{__TMP__} year = [2014, 2015, 2016, 2017, 2018, 2019] tutorial_count = [39, 117, 111, 110, 67, 29] plt.plot(year, tutorial_count, color="#6c3376", linewidth=3) plt.xlabel('Year') plt.ylabel('Number of futurestud.io Tutorials') \end{SetPlaceholderCode} \begin{figure} \centering \cacheMe[matplotlib, set includegraphics options={width=.8\linewidth}, set caption={Hello}]{__TMP__} \caption{An example to show how code can be inserted into macros or environments that evaluate their contents (this trick is actually not needed for figures)} \end{figure} \end{codeexample} \end{pgfmanualentry} \subsubsection{Options to configure the template} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/set template=\marg{content template}\@nil \makeatother% \pgfmanualbody Style that alias to |set placeholder={__ROBEXT_TEMPLATE__}{#1}|, in order to define the placeholder that will hold the template of the final file. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/set template=\marg{content template}\@nil \makeatother% \pgfmanualbody Style that alias to |set placeholder={__ROBEXT_TEMPLATE__}{#1}|, in order to define the placeholder that will hold the template of the final file. \end{pgfmanualentry} \subsubsection{Options to configure the compilation command}\label{sec:configureCompilationCommand} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/set compilation command=\marg{compilation command}\@nil \extractkey/robExt/set compilation command create if no error=\marg{compilation command}\@nil \extractkey/robExt/set compilation command move if no error=\marg{compilation command}\@nil \makeatother% \pgfmanualbody |set compilation command| is equivalent to |set placeholder={__ROBEXT_COMPILATION_COMMAND__}{#1}|, in order to define the placeholder that will hold the compilation command. Then, |set compilation command move if no error| (new in v2.4) is like |set compilation command| except that it will copy the file located in |__ROBEXT_OUTPUT_PDF__-tmp| into |__ROBEXT_OUTPUT_PDF__| if the command succeeds (returns a status code of 0), and otherwise it will not create the |__ROBEXT_OUTPUT_PDF__| at all, creating an error message. This is needed by some programs that will create an output file even if an error occurs, for instance wget will still create an empty file if there is no internet connection, producing no error during the compilation, which can be confusing. |set compilation command create if no error| (new in v2.4) will, on the other hand, create an empty |__ROBEXT_OUTPUT_PDF__| file if the command succeeds (only useful if you use a custom include command). \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/add argument to compilation command=\marg{argument}\@nil \extractkey/robExt/add arguments to compilation command=\marg{argument}\@nil \makeatother% \pgfmanualbody |add argument to compilation command| is a style that alias to:\\ |set placeholder={__ROBEXT_COMPILATION_COMMAND__}{__ROBEXT_COMPILATION_COMMAND__ "#1"}| in order to add an argument to the compilation command. |add arguments to compilation command| (note the |s|) accepts multiple arguments separated by a comma. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/add key value argument to compilation command=\marg{key=value}\@nil \makeatother% \pgfmanualbody Adds to the command line two arguments |key| and |value|. This is a way to quickly pass arguments to a script: the script just needs to loop over the arguments and consider the odd elements as keys and the next elements as the value. Another option is to insert some placeholders directly in the script. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/add key and file argument to compilation command=\marg{key=filename}\@nil \makeatother% \pgfmanualbody |filename| is the path to a file in the root folder. This adds, as:\\ |add key value argument to compilation command|\\ two arguments, where the first argument is the key, but this time the second argument is the path of |filename| relative to the cache folder (useful since scripts run from this folder). Moreover, it automatically ensures that when |filename| changes, the file gets recompiled. Note that contrary to some other commands, this does not copy the file in the cache, which is practical notably for large files like videos. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/force compilation\@nil \extractkey/robExt/do not force compilation\@nil \makeatother% \pgfmanualbody By default, we compile cached documents only if |-shell-escape| is enabled. However, if the user allowed |pdflatex| (needed to compile latex documents), |cd| (not needed when using |no cache folder|), and |mkdir| (not needed when using |no cache folder| or if the cache folder that defaults to |robustExternalize| is already created) to run in restricted mode (so without enabling |-shell-escape|), then there is no need for |-shell-escape|. In that case, set |force compilation| and this library will compile even if |-shell-escape| is disabled. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/recompile\@nil \extractkey/robExt/do not recompile\@nil \makeatother% \pgfmanualbody (new to v2.1) |recompile| will force a compilation even if the image is already cached (mostly useful if you made a change to a non-tracked dependency). Note that since this library does not want to remove any file on the hard drive (we do not want to risk to remove important files in case of a bug), this command will NOT clear the auxiliary files already present, so for maximum purity, ensure that your compiling command is idempotent (so that compiling twice gives the same outcome as compiling once). \begin{codeexample}[vbox, %TURN_INTO_CODE_ONLY_ARXIV ] \cacheEnvironment{tikzcd}{tikz, add to preamble={\usepackage{tikz-cd}}} \begin{tikzcd} A \rar & B \end{tikzcd} \end{codeexample} If you want to actually remove the aux files, you can either change the compilation command, or add something like this to remove the file (note that it will prompt a message before running the actual command), using the hook |robust-externalize/just-before-writing-files| that we introduced in v2.1: \begin{codeexample}[code only] \cacheTikz \robExtConfigure{ clean and recompile/.code={% \AddToHook{robust-externalize/just-before-writing-files}{% \edef\command{rm -f "\robExtAddCachePathAndName{\robExtFinalHash.pdf}" && % rm -f "\robExtAddCachePathAndName{\robExtFinalHash.aux}"}% \message{You will run the command ``\command''}% \message{Press X if you do NOT want to run it, otherwise press ENTER.}% \show\def% this is just used to wait a user input \immediate\write18{\command}% }% },% } \begin{tikzpicture} \node[fill=red] {B}; \end{tikzpicture} \end{codeexample} Note that if you do not want to run a command from \LaTeX{}, or if you are on a version smaller than 2.1, you can also simply print the name of the file in the pdf and remove it manually: \begin{codeexample}[width=0cm,vbox] \begin{CacheMe}{tikzpicture, name output=filetodelete}[baseline=(A.base)] \node[draw,rounded corners,fill=pink!60](A){Hello World!}; \end{CacheMe}\\ You want to delete the file \texttt{\filetodeleteInCache.pdf}. % \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\robExtMv\@@ \extractcommand\robExtCp\@@ \pgfmanualbody (new in v2.4) Respectively expands to |mv| and |cp| in linux/unix and to |move| and |copy| in Windows (needed to create a compilation command that works in both linux and windows). Note that |set compilation command| does NOT expand its argument by default and spaces after macros are removed by latex, so you need to use |/.expanded| and |\space| like in: \begin{codeexample}[code only] set compilation command/.expanded={\robExtCp\space"__ROBEXT_SOURCE_FILE__" "__ROBEXT_OUTPUT_PDF__"} \end{codeexample} \end{pgfmanualentry} \subsubsection{Options to configure the inclusion command} The inclusion command is the command that is run to include the cached file back in the pdf (e.g. based on |\includegraphics|). We describe now how to configure this command. \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/custom include command advanced=\marg{include command}\@nil \makeatother% \pgfmanualbody Sets the command to run to include the compiled file. You can use:\\ |\robExtAddCachePathAndName{\robExtFinalHash.pdf}|\\ in order to get the path of the compiled pdf file. Note that we recommend rather to use |custom include command| that automatically checks if the file compiled correctly and that load the |*-out.tex| file if it exists (useful to pass information back to the pdf). \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/custom include command=\marg{include command}\@nil \extractkey/robExt/include command is input=\opt{\oarg{additional code to run}}\@nil \makeatother% \pgfmanualbody Sets the command to run to include the compiled file, after checking if the file has been correctly compiled and loading |*-out.tex| (useful to pass information back to the pdf). |include command is input={SOME CODE}| is a shortcut for: \begin{verbatim} custom include command={SOME CODE\input{\robExtAddCachePathAndName{\robExtFinalHash.pdf}}}, \end{verbatim} \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/do not include pdf\@nil \extractkey/robExt/include command is input=\opt{additional command}\@nil \makeatother% \pgfmanualbody |do not include pdf| do not include the pdf. Useful if you only want to compile the file but use it later (note that you should still generate a |.pdf| file, possibly empty, to indicate that the compilation runs smootly). Equivalent to:\\ |custom include command={}| Use |include command is input| (v2.3 and after) in order to specify that you should |\input| the |.pdf| file (note that despite the extension, it should still contain a LaTeX code). You can also specify an optional command to run before. Equivalent to: |custom include command={#1\input{\robExtAddCachePathAndName{\robExtFinalHash.pdf}}}| \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/enable manual mode\@nil \extractkey/robExt/disable manual mode\@nil \extractkey/robExt/enable fallback to manual mode\@nil \extractkey/robExt/disable fallback to manual mode\@nil \makeatother% \pgfmanualbody If you do or do not want to ask latex to run the compilation commands itself (for instance for security reasons), you can use these commands and run the command manually later: \begin{codeexample}[width=0pt,vbox] \robExtConfigure{ enable manual mode } The next picture must be manually compiled % (see JOBNAME-robExt-compile-missing-figures.sh):\\ % \begin{tikzpictureC}[baseline=(A.base)] \node[fill=red, rounded corners](A){I must be manually compiled.}; \node[fill=red, rounded corners, opacity=.3,overlay] at (A.north east){I am an overlay text}; \end{tikzpictureC} \end{codeexample} The main difference between |manual mode| and |fallback to manual mode| is that |fallback to manual mode| will try to compile the file if |-shell-escape| is enabled, while |manual mode| will never run any command, even if |-shell-escape| is enabled. Note that |enable fallback to manual mode| is available starting from v2.0. See \cref{sec:operationsCache} for more details. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/include graphics args\@nil \makeatother% \pgfmanualbody By default, the include commands runs |\includegraphics| on the pdf, and possibly raises it if needed. You can customize the arguments passed to |\includegraphics| here. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/add cache to graphicspath\@nil \makeatother% \pgfmanualbody This will add the folder containing the cache in the |\graphicspath{}|. Mostly useful if you need to input a file from the cache that does |\includegraphics{foo}| where |foo| is an image from the cache itself (looking at you gnuplot). \end{pgfmanualentry} \subsubsection{Configuration of the cache} If needed, you can configure the cache: \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/set filename prefix=\marg{prefix}\@nil \makeatother% \pgfmanualbody By default, the files in the cache starts with |robExt-|. If needed you can change this here, or by manually defining |\def\robExtPrefixFilename{yourPrefix-}|. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/set subfolder and way back=\marg{cache folder}\marg{path to project from cache}\@nil \extractkey/robExt/set cache folder and way back=\marg{cache folder}\marg{path to project from cache}\@nil \makeatother% \pgfmanualbody By default, the cache is located in |robustExternalize/|, using:\\ |set cache folder and way back={robustExternalize/}{../},|\\ You can customize it the way you want, just be make sure that going to the second arguments after going to the first argument leads you back to the original position, and make sure to terminate the path with a |/| so that |__ROBEXT_WAY_BACK__common_inputs.tex| gives the path of the file |common_inputs.tex| in the root folder (do not write |__ROBEXT_WAY_BACK__/common_inputs.tex| as this would expand to the absolute path |/common_inputs.tex| if you disable the cache folder). Note that both:\\ |set cache folder and way back|\\ and\\ |set subfolder and way back|\\ are alias, but the first one was introduced in v2.0 as it is clearer, the second one being kept for backward compatibility. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/set output directory=\marg{output directory}\@nil \makeatother% \pgfmanualbody If you compile with |--output-directory build|, you also need to tell this library that you are compiling with |--output-directory build|. As I can't find a reliable way to get this value, even using |TEXMF_OUTPUT_DIRECTORY|, you need to specify manually the compilation directory. To do that, you can either compile with: \begin{codeexample}[code only] pdflatex -shell-escape -output-directory=build "\def\robExtOutputDirectory{build}\input{test.tex}" \end{codeexample} or add in your file: \begin{codeexample}[code only] \robExtConfigure{ set output directory=build, } \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/no cache folder\@nil \makeatother% \pgfmanualbody Do not put the cache in a separate subfolder. \end{pgfmanualentry} \newpage \subsubsection{Customize or disable externalization}\label{sec:disableExternalization} You might want (sometimes or always) to disable externalization, for instance to use |remember picture| \tikz[remember picture,baseline=(pointtome.base)] \node[rounded corners, fill=orange](pointtome){Point to me if you can};, even if you used |\cacheTikz|: \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/disable externalization\@nil \extractkey/robExt/de\@nil \extractkey/robExt/disable externalization now\@nil \extractkey/robExt/enable externalization\@nil \makeatother% \pgfmanualbody Enable or disable externalization. \begin{codeexample}[width=0pt,vbox] % In theory all pictures should be externalized (so remember picture should fail) \cacheTikz % But we can disable it temporarily \begin{tikzpicture}[remember picture] \node[rounded corners, fill=red](A){This figure is not externalized. This way, it can use remember picture.}; \draw[->,overlay] (A) to[bend right] (pointtome); \end{tikzpicture}\\ % You can also disable it globally/in a group: { \robExtConfigure{disable externalization} \begin{tikzpicture}[remember picture] \node[rounded corners, fill=red](A){This figure is not externalized. This way, it can use remember picture.}; \draw[->,overlay] (A.west) to[bend left] (pointtome); \end{tikzpicture}\\ \begin{tikzpicture}[remember picture] \node[rounded corners, fill=red](A){This figure is not externalized. This way, it can use remember picture.}; \draw[->,overlay] (A.east) to[bend right] (pointtome); \end{tikzpicture} } \end{codeexample} |disable externalization now| additionally redefines all automatically cached commands and environments to their default value right now (|disable externalization| would only do so when running the |command if no externalization| is run, which should be preferred if possible). This is mostly useful when an automatically cached command cannot be parsed (e.g.\ you specified the signature |O{}m| but in fact the command expects a more complicated parsing algorithm. For instance, in |tikz| we can omit the brackets, and it might confuse the system if we defined it as: \begin{codeexample}[width=0pt,vbox] %% Not the recommended way to proceed, but this is for the example \cacheCommand{tikz}[O{}m]{tikz} { % Needed, or the parser would be confused by the next command \robExtConfigure{disable externalization now} \tikz\node[fill=green]{The short version of tikz can be confusing.}; } \end{codeexample} Since v2.1 you can also automatically disable externalization if the input contains a given string, for instance |remember picture| (that is not cachable with this library): \begin{codeexample}[vbox] \cacheTikz \robExtConfigure{ add to preset={tikzpicture}{ % \b means "word boundary", and spaces must be escaped if matches={remember picture}{disable externalization}, }, } \begin{tikzpicture}[remember picture] \node[fill=green](my original node){Point to me}; \end{tikzpicture} % Some text % \begin{tikzpicture}[overlay, remember picture] \draw[->] (0,0) to[bend left] (my original node); \end{tikzpicture} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/command if no externalization\@nil \extractkey/robExt/command if externalization\@nil \extractkey/robExt/run command if externalization=\marg{code}\@nil \makeatother% \pgfmanualbody You can easily change the command to run if externalization is disabled using by setting the \textbf{.code} of this key. By default, it is configured as:\\ \begin{verbatim} command if no externalization/.code={% \robExtDisableTikzpictureOverwrite\evalPlaceholder{__ROBEXT_MAIN_CONTENT__}% } \end{verbatim} Unless you know what you are doing, you should include |\robExtDisableTikzpictureOverwrite| as it is often necessary to avoid infinite recursion when externalization is disabled and the original command has been replaced with a cached version (for instance done by |\cacheTikz|). Note that if you write your own style, try to modify |__ROBEXT_MAIN_CONTENT__| so that it can be included as-it in a document: this way you do not need to change this command. You can also use |command if externalization/.append code={...}| if you want to run a code only if externalization is enabled (this is used for instance by the |forward| functionality), but we advise you rather to use |run command if externalization={...}| as this will also work if the style is executed while |command if externalization| is already running (this might occur when calling the style inside |if matches word|, for instance when forwarding colors). \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/print verbatim if no externalization\@nil \makeatother% \pgfmanualbody Sets |command if no externalization| to print the verbatim content of |__ROBEXT_MAIN_CONTENT__| if externalization is disabled. Internally, it just sets it to:\\ |\printPlaceholder{__ROBEXT_MAIN_CONTENT__}|\\ This is mostly useful when typesetting |__ROBEXT_MAIN_CONTENT__| directly does not make sense (e.g. in python code). This style is used for instance in the |python| preset, allowing us to get: \begin{codeAndResult} \begin{CacheMeCode}{python, verbatim output, set placeholder eval={__thepage__}{\thepage}, %% We disable externalization disable externalization} with open("__ROBEXT_OUTPUT_PREFIX__-out.txt", "w") as f: for i in range(5): f.write(f"Hello {i}, we are on page __thepage__\n") \end{CacheMeCode} \end{codeAndResult} You can also disable the externalization on all elements that use a common preset, for instance you can disable externalization on all |bash| instances (useful if you are on Windows for instance): \begin{codeAndResult} \robExtConfigure{ % bash code will not be compiled (useful on windows for instance) add to preset={bash}{ disable externalization }, } \begin{CacheMeCode}{bash, verbatim output} # $outputTxt contains the path of the file that will be printed via \verbatiminput uname -srv > "${outputTxt}" \end{CacheMeCode} \end{codeAndResult} \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/execute after each externalization\@nil \extractkey/robExt/execute before each externalization\@nil \makeatother% \pgfmanualbody By doing |execute after each externalization={some code}|, you will run some code after the externalization. This might be practical for instance to update a counter (e.g. the number of pages\dots) based on the result of the compiled file. \end{pgfmanualentry} \subsubsection{Dependencies}\label{sec:dependenciesDoc} In order to enforce reproducibility, you should tell what are the files that your code depends on, by adding this file as a dependency. This has the advantage that if this file is changed, your code is automatically recompiled. On the other hand, you might not want this behavior (e.g. if this file often changes in a non-important way): in that case, just don't add the file as a dependency (but keep that in mind as you might not be able to recompile your file if you clear the cache if you introduced breaking changes). \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/dependencies=\marg{list,of,dependencies}\@nil \extractkey/robExt/add dependencies=\marg{list,of,dependencies}\@nil \extractkey/robExt/reset dependencies\@nil \makeatother% \pgfmanualbody Set/add/reset the dependencies (you can put multiple files separated by commas). These files should be relative to the main compiled file. For instance, if you have a file |common_inputs.tex| that you want to input in both the main file and in the cached files, that contains, say:\\ |\def\myValueDefinedInCommonInputs{42}|\\ then you can add it as a dependency using: \begin{codeexample}[width=0pt,vbox] \begin{CacheMe}{latex, add dependencies={common_inputs.tex}, add to preamble={\input{__ROBEXT_WAY_BACK__common_inputs.tex}}} The answer is \myValueDefinedInCommonInputs. \end{CacheMe} \end{codeexample} Note that the placeholder |__ROBEXT_WAY_BACK__| contains the path from the cache folder (containing the |.tex| that will be cached) to the root folder.\\ This way, you can easily input files contained in the root folder. \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\copyFileToCache\@@ \extractcommand\forceCopyFileToCache\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/copy file to cache=\marg{file}\@nil \makeatother% \pgfmanualbody It can sometimes be handy to copy a file to the cache folder to avoid using |__ROBEXT_WAY_BACK__| for instance to refer to another file. This can be done with |copy file to cache|, either in a preset or in |\robExtConfigure| like: \begin{codeexample}[code only] \robExtConfigure{ copy file to cache={zx-calculus.sty}, } \end{codeexample} Note that |copy file to cache| will always compare the md5 sum to check if a copy must be done or not. \end{pgfmanualentry} \subsubsection{Forward macros}\label{sec:forward} (since v2.1) You can also see the tutorial (\cref{sec:forwardIntro}) for some examples. So far, we have seen multiple approaches to use a macro in both the main document and in cached pictures, having different advantages: \begin{itemize} \item if we define the macro in a file say |common_inputs.tex|, if we include that file in both files \textbf{and} if we add it to the list of dependencies (\cref{sec:dependenciesDoc}), then you will have great purity (if |common_inputs.tex| changes, all files are recompiled), but this might be a problem if you often change the file |common_inputs.tex| since you will need to recompile all pictures every time you add a new macro \item If you follow the same procedure, but \textbf{without} adding the file in the list of dependencies, then you will save compilation time (no need to recompile everything if you add a macro in |common_inputs.tex|), but you will lose purity (if you change the definition of a macro in that file, you will need to manually specify the list of figures to recompile, e.g. using |recompile|). \item You can also create multiple files like \begin{verbatim} common_inputs_functionality_A.tex \end{verbatim} and \begin{verbatim} common_inputs_functionality_B.tex \end{verbatim} and use the first solution (maybe by creating a preset that automatically calls |add to preamble| and |add dependencies|), and only input the appropriate file depending on the set of macro that you need. This gives a tradeoff of above approaches (good purity, with less recompilation), but you will still compile often if you put many macros in a single file (and nobody wants to create one macro per file!). \end{itemize} In this section, we provide a third solution that tries to solve the above issue (purity without frequent recompilations), by allowing the user to forward only a specific macro, with something like |forward=\myMacro|. In order to avoid to manually forward all macros used for each picture, we also provide some functions to automatically detect the macros to forward. \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/forward=macro to forward\@nil \extractkey/robExt/fw=macro to forward\@nil \extractkey/robExt/forward at letter=macro to forward\@nil \makeatother% \pgfmanualbody |forward| (alias |fw|) will forward the definition of a macro to the picture if externalization is enabled: \begin{codeexample}[width=0pt] \cacheTikz \def\myName{Alice} \begin{tikzpicture} \node[fill=red]{\myName}; \end{tikzpicture} \end{codeexample} Note that it works for macro defined with |\def|, |\newcommand| (or alike) and |\NewDocumentCommand| (or other xparse commands). For instance with xparse: \begin{codeexample}[width=0pt] \cacheTikz \NewDocumentCommand{\MyNode}{O{}m}{\node[rounded corners,fill=red,#1]{#2};} \begin{tikzpicture} \MyNode{Alice} \MyNode[xshift=2cm]{Bob} \end{tikzpicture} \end{codeexample} and with |\newcommand|: \begin{codeexample}[width=0pt] \cacheTikz \newcommand{\MyNodeWithNewCommand}[2][draw,thick]{\node[rounded corners,fill=red,#1]{#2};} \begin{tikzpicture} \MyNodeWithNewCommand{Alice} \MyNodeWithNewCommand[xshift=2cm]{Bob} \end{tikzpicture} \end{codeexample} Note that this might not work with quite involved macros. Notably, if the macro is defined by calling other macros, you need to also forward these macros. \begin{codeexample}[width=0pt] \cacheTikz \def\myName{Alice} \NewDocumentCommand{\MyNodeWithInnerMacros}{O{}m}{\node[rounded corners,fill=red,#1]{\myName: #2};} \begin{tikzpicture} \MyNodeWithInnerMacros{she is tall} \end{tikzpicture} \end{codeexample} Really involved macros (e.g.\ defined inside libraries) might call many other macros, possibly using the @ letter (used only in module code). If you REALLY want, you can forward such macros if you forward \textbf{all} macros that are used internally, using |forward at letter| that will automatically wrap the macro inside |\makeatletter...\makeatother|, but you should certainly use |add to preamble={\usepackage{yourlib}}| if you need to do something like that: \begin{codeexample}[width=0pt] \cacheTikz \makeatletter \def\module@name{Alice} \newcommand{\MyNodeWithAtCommands}[2][draw,thick]{\node[rounded corners,fill=red,#1]{\module@name};} \begin{tikzpicture} \MyNodeWithAtCommands{Alice} \end{tikzpicture} \makeatother \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/forward eval=macro to eval and forward\@nil \makeatother% \pgfmanualbody Evaluate a macro, and defines it using this new value when compiling the function. This might be useful for instance if your macro depends on other macros that you do not want to export: \begin{codeexample}[width=0pt] \def\name{Alice} \def\fullName{\name} \begin{tikzpictureC} \node[rounded corners, fill=red]{\fullName.}; \end{tikzpictureC} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/forward counter=counter to forward\@nil \extractkey/robExt/forward counter force value=\marg{counter to forward}\marg{value of counter}\@nil \makeatother% \pgfmanualbody Forward a counter: \begin{codeexample}[width=0pt] \begin{tikzpictureC} \node[rounded corners, fill=red]{The current page is \thepage.}; \end{tikzpictureC} \end{codeexample} The force value also assigns a value to the counter irrespective of the current value (might be useful for instance to generate a cached value for the arXiv). \begin{codeexample}[width=0pt] \begin{tikzpictureC} \node[rounded corners, fill=red]{The current page is \thepage.}; \end{tikzpictureC} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/forward color=color to forward\@nil \extractkey/robExt/auto forward color=\marg{preset}\marg{color to forward}\@nil \makeatother% \pgfmanualbody |forward color| forwards a color (defined with xcolor). It works with colors defined with |\definecolor|: \begin{codeexample}[width=0pt] \definecolor{myred}{HTML}{F01122} \begin{tikzpictureC} \node[fill=myred]{A}; \end{tikzpictureC} \end{codeexample} but also with colors defined with |\colorlet|: \begin{codeexample}[width=0pt] \definecolor{myred}{HTML}{F01122} \colorlet{myviolet}{blue!50!myred} \begin{tikzpictureC} \node[fill=myviolet,yshift=-1cm]{B}; \end{tikzpictureC} \end{codeexample} Note that using |if matches word|, you can also automatically forward colors. This command also scales better than |if matches|: if |if matches| is used to register $N$ words, the compilation time will increase for each new |if matches| (since we need to check each time if the string contains the expression). |if matches word|, on the other hand, is more clever, and extracts in a single loop all the words of the string: the running time is therefore independent of the number of times |if matches word| was called. You can call |if matches word| outside of any preset like in: \begin{codeexample}[width=0pt] \definecolor{myred}{HTML}{F01122} \colorlet{myviolet}{blue!50!myred} \colorlet{mypink}{pink!90!orange} \robExtConfigure{ if matches word={mypink}{forward color=mypink}, } \begin{tikzpictureC} \node[fill=mypink,yshift=-1cm]{B}; \end{tikzpictureC} \end{codeexample} And this will automatically allow all presets based on |latex| to forward this color (see also |\definecolorAutoForward| to do that even quicker). You can also simply call it on any preset of your choice like: \begin{codeexample}[width=0pt] \definecolor{myred}{HTML}{F01122} \colorlet{myviolet}{blue!50!myred} \colorlet{mypink}{pink!90!orange} \robExtConfigure{ add to preset={latex}{ if matches word={mypink}{forward color=mypink}, }, } \begin{tikzpictureC} \node[fill=mypink,yshift=-1cm]{B}; \end{tikzpictureC} \end{codeexample} or, equivalently: \begin{codeexample}[width=0pt] \definecolor{myred}{HTML}{F01122} \colorlet{myviolet}{blue!50!myred} \colorlet{mypink}{pink!90!orange} \robExtConfigure{ auto forward color={tikz}{mypink}, } \begin{tikzpictureC} \node[fill=mypink,yshift=-1cm]{B}; \end{tikzpictureC} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/auto forward\@nil \extractkey/robExt/auto forward only macros\@nil \extractkey/robExt/auto forward words\@nil \makeatother% \pgfmanualbody |auto forward| enables automatic forwarding of macros marked with |*AutoForward| commands or |\configIfMacroPresent| or |if matches word|. This command naively extracts all used macros via a simple regex match and forwards them/execute the style configured in |\configIfMacroPresent| if it exists. It does a similar thing for words registered via |if matches word| or |register word with namespace|. If you want to only forward macros or words, you can use |auto forward only macros| or |auto forward words|. \begin{codeexample}[vbox] \cacheTikz \robExtConfigure{add to preset={tikz}{auto forward}} \newcommandAutoForward{\MyNode}[2][draw,thick]{\node[rounded corners,fill=red,#1]{#2};} \newcommandAutoForward{\MyGreenNode}[2][draw,thick]{\node[rounded corners,fill=green,#1]{#2};} \begin{tikzpicture} \MyNode{Recompiled only if MyNode is changed} \MyNode[xshift=8cm]{but not if the (unused) MyGreenNode is changed.} \end{tikzpicture}\\ \begin{tikzpicture} \MyGreenNode{Recompiled only if MyGreenNode is changed} \MyGreenNode[xshift=8cm]{but not if the (unused) MyNode is changed.} \end{tikzpicture} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\configIfMacroPresent\marg{macro}\marg{style to run if macro is present}\@@ \pgfmanualbody Runs the corresponding style if the macro |macro| is present (make sure to enable |auto forward|). \begin{codeexample}[vbox] \cacheTikz \robExtConfigure{add to preset={tikz}{auto forward}} \configIfMacroPresent{\ding}{add to preamble={\usepackage{pifont}}} \begin{tikzpicture} \node[fill=green, circle]{\ding{164}}; \end{tikzpicture} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\newcommandAutoForward\marg{macro}\opt{\oarg{nb args}}\opt{\oarg{optional default value}}\marg{definition}\opt{\oarg{additional style}}\@@ \extractcommand\renewcommandAutoForward\marg{macro}\opt{\oarg{nb args}}\opt{\oarg{optional default value}}\marg{definition}\opt{\oarg{additional style}}\@@ \extractcommand\providecommandAutoForward\marg{macro}\opt{\oarg{nb args}}\opt{\oarg{optional default value}}\marg{definition}\opt{\oarg{additional style}}\@@ \extractcommand\NewDocumentCommandAutoForward\marg{macro}\marg{arg spec}\opt{\oarg{additional style}}\marg{definition}\@@ \extractcommand\RenewDocumentCommandAutoForward\marg{macro}\marg{arg spec}\opt{\oarg{additional style}}\marg{definition}\@@ \extractcommand\ProvideDocumentCommandAutoForward\marg{macro}\marg{arg spec}\opt{\oarg{additional style}}\marg{definition}\@@ \extractcommand\DeclareDocumentCommandAutoForward\marg{macro}\marg{arg spec}\opt{\oarg{additional style}}\marg{definition}\@@ \extractcommand\NewExpandableDocumentCommandAutoForward\marg{macro}\marg{arg spec}\opt{\oarg{additional style}}\marg{definition}\@@ \extractcommand\RenewExpandableDocumentCommandAutoForward\marg{macro}\marg{arg spec}\opt{\oarg{additional style}}\marg{definition}\@@ \extractcommand\ProvideExpandableDocumentCommandAutoForward\marg{macro}\marg{arg spec}\opt{\oarg{additional style}}\marg{definition}\@@ \extractcommand\DeclareExpandableDocumentCommandAutoForward\marg{macro}\marg{arg spec}\opt{\oarg{additional style}}\marg{definition}\@@ \extractcommand\defAutoForward\marg{macro}\opt{\oarg{def-style arg spec}}\marg{macro}\opt{\oarg{additional style}}\@@ \extractcommand\genericAutoForward\opt{*}\opt{\oarg{preset}}\opt{\oarg{namespace}}\marg{word}\opt{\oarg{additional style}}\marg{code to run}\@@ \extractcommand\genericAutoForwardStringMatch\opt{*}\opt{\oarg{preset}}\marg{string}\opt{\oarg{additional style}}\marg{code to run}\@@ \pgfmanualbody \begin{verbatim} \newcommandAutoForward{\macro}[nb args][default value]{def}[STYLE] \end{verbatim} is like \begin{verbatim} \newcommand{\macro}[nb args][default value]{def} \end{verbatim} but it additionally runs \begin{verbatim} \configIfMacroPresent{\macro}{forward=\macro,STYLE} \end{verbatim} to automatically forward the macro if it is used. It is really practical to automatically define a macro and forward it: \begin{codeexample}[vbox] \cacheTikz \robExtConfigure{add to preset={tikz}{auto forward}} \newcommandAutoForward{\MyName}{\ding{164} Alice}[add to preamble={\usepackage{pifont}}] \newcommandAutoForward{\MyNode}[2][draw,thick]{\node[rounded corners,fill=green,#1]{\MyName: #2};}[ load auto forward macro config=\MyName ] \begin{tikzpicture} \MyNode{Recompiled only if MyNode is changed} \MyNode[yshift=-1cm]{or if MyName is changed} \end{tikzpicture} \end{codeexample} This works similarly for other commands (note that for |\defAutoForward|, you need to put the arguments in a bracket, like |\defAutoForward\myMacro[#1 and #2]{Hey #1, #2}| for |\def\myMacro#1 and #2{Hey #1, #2}|). |\genericAutoForward| and |\genericAutoForwardStringMatch| are a bit special, as it can be used to run any code assuming that |word|/|string to match| is present (|\genericAutoForward| is more efficient as it does not scale with the number of words, but do not allow spaces etc in the name). For instance, you can use it to automatically forward the code that defines a new tikz style assuming |mystyle| is preset: \begin{codeexample}[width=0pt] \cacheTikz \genericAutoForward{mystyle}{ \tikzset{mystyle/.style={fill=#1!10!white,text=#1!50!black}} } \begin{tikzpicture} \node[mystyle=green]{A}; \end{tikzpicture} \end{codeexample} This will also run the code to define it in the current document (practical for instance if some pictures are not externalized), use the stared version of |\genericAutoForward| if you do not want to run the code in the current document. By default, the code is added to the |latex| preset, but you can choose another preset if you want using the optional option. The additional style might be specified to run other style if the matches is true. Note that you should be able to use definitions inside the code (a bug in v2.1 was making this code fail): \begin{codeexample}[width=0pt] \genericAutoForward{mystyle}{ \tikzset{mystyle/.style={fill=#1!10!white,text=#1!50!black}} } \newcommandAutoForward{\myName}{Alice} \robExtConfigure{ add to preset={tikz}{ auto forward, % not needed for genericAutoForward, but for newcommandAutoForward if matches word={pmatrix}{add to preamble={\usepackage{amsmath}}} } } \begin{tikzpicture} \node[mystyle=green]{% \def\hey#1{Hey #1 :)}% You can define macros inside the cached elements \hey{Coucou \myName} % $\begin{pmatrix}% Fixing a bug introduced in v2.1, we can also use macros A & B\\ C & D\\ \end{pmatrix}$ }; \end{tikzpicture} \end{codeexample} \begin{codeexample}[vbox] % space is not allowed in the name with this more efficient version: \robExtGenericAutoForward{mystyle}{% \tikzset{% mystyle/.style={fill=#1!30!white,text=#1!40!black}, }% } \begin{tikzpictureC} \node[mystyle=green]{My style automatically forwarded}; \end{tikzpictureC} \end{codeexample} \begin{codeexample}[vbox] % space is allowed in the name with this slightly less efficient version: \robExtGenericAutoForwardStringMatch{my style}{% \tikzset{% my style/.style={fill=#1!30!white,text=#1!40!black}, }% } \begin{tikzpictureC} \node[my style=green]{My style with a space in the name automatically forwarded}; \end{tikzpictureC} \end{codeexample} Note that by default, this forwards elements in the |latex| preset, but you can change it via |preset|. You can also configure additional style to run, and choose another namespace for the word version (practical only if you want to forward different codes to different presets: note that a single namespace is allowed per preset). \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\runHereAndInPreambleOfCachedFiles\opt{\oarg{preset}}\marg{code}\@@ \pgfmanualbody (from v2.3) This will run |code| right now and it will add it in the preamble of the files that use the preset |preset| (defaults to |latex|). This is practical to avoid duplicating the code into |add to preamble|. For instance, if want the configure the font of the current document and of all latex documents at the same time, use: \begin{codeexample}[code only] \runHereAndInPreambleOfCachedFiles{ \usepackage{fontspec} \setmainfont{Times New Roman} } \robExtConfigure{ add to preset={latex}{ use lualatex, % Make sure to compile with lualatex the cached images. }, } \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\definecolorAutoForward\marg{color}\marg{model}\marg{value}\opt{\oarg{additional style}}\@@ \extractcommand\colorletAutoForward\marg{color}\marg{value}\opt{\oarg{additional style}}\@@ \pgfmanualbody Define colors like |\definecolor| and |\colorlet|, but also runs \begin{verbatim} if matches word={COLOR}{forward color=COLOR} \end{verbatim} to automatically forward the color (this will forward it only in latex-based presets). You can provide additional style to run as well in option. \begin{codeexample}[vbox] \definecolorAutoForward{myred}{HTML}{F01122} \colorletAutoForward{myviolet}{blue!50!myred} \begin{tikzpictureC} \node[fill=myred,yshift=-1cm]{Uses myred}; \end{tikzpictureC} \begin{tikzpictureC} \node[fill=myviolet,yshift=-1cm]{Myviolet}; \end{tikzpictureC} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/load auto forward macro config=\marg{macro}\@nil \makeatother% \pgfmanualbody Like |forward| except that it loads the configuration that would have been loaded if the macro was present in the file. This is mostly useful to say that a macro depends on the style of another macro without copy/pasting the style of the second macro inside the first one. \begin{codeexample}[vbox] \cacheTikz \robExtConfigure{add to preset={tikz}{auto forward}} \newcommandAutoForward{\MyName}{\ding{164} Alice}[add to preamble={\usepackage{pifont}}] \newcommandAutoForward{\MyNode}[2][draw,thick]{\node[rounded corners,fill=green,#1]{\MyName: #2};}[ load auto forward macro config=\MyName ] \begin{tikzpicture} \MyNode{Recompiled only if MyNode is changed} \MyNode[yshift=-1cm]{or if MyName is changed} \end{tikzpicture} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/if matches=\marg{string}\marg{style to apply}\@nil \extractkey/robExt/if matches word=\marg{string}\marg{style to apply}\@nil \extractkey/robExt/if matches regex=\marg{latex3 regex}\marg{style to apply}\@nil \extractkey/robExt/register word with namespace=\marg{namespace}\marg{word}\marg{style}\@nil \extractkey/robExt/auto forward words\@nil \extractkey/robExt/auto forward words namespace=\marg{namespace}\@nil \makeatother% \pgfmanualbody |if matches *| applies the corresponding style if the string (resp.\ word or regex in \LaTeX{}3 format) matches is matched in the content. |if matches word| is more efficient that |if matches string| since the running time is independent of the number of times |if matches word| is called\footnote{Scaling with $O(s)$ instead of $O(sn)$, where $s$ is the size of the string typed in |cacheMe| and $n$ is the number of times |if matches| is used, for instance when replacing multiple strings.}. |if matches word| make it only match words that are mostly made of consecutive letters and numbers |[A-Za-z0-9]| (more precisely separated by elements in |\robExtWordSeparators|). The regex version can be more expressive, but is significantly slower (it can easily multiply by 2 the compilation time). \begin{codeexample}[vbox] \robExtConfigure{ add to preset={my python}{ python print code and result, % \b is for word border, and ( needs to be escaped in regex if matches={cos(}{add import={from math import cos}}, if matches={sin(}{add import={from math import sin}}, }, } \begin{CacheMeCode}{my python} print(cos(1)+sin(2)) \end{CacheMeCode} \end{codeexample} With regex: \begin{codeexample}[vbox] \robExtConfigure{ add to preset={my python}{ python print code and result, % \b is for word border, and ( needs to be escaped in regex if matches regex={\b cos\(}{add import={from math import cos}}, if matches regex={\b sin\(}{add import={from math import sin}}, }, } \begin{CacheMeCode}{my python} print(cos(1)+sin(2)) \end{CacheMeCode} \end{codeexample} This can also be practical to disable caching if some pictures uses |remember picture| (which is not supported by this library): \begin{codeexample}[vbox] \cacheTikz \robExtConfigure{ add to preset={tikzpicture}{ % \b means "word boundary", and spaces must be escaped if matches={remember picture}{disable externalization}, }, } \begin{tikzpicture}[remember picture] \node[fill=green](my original node){Point to me}; \end{tikzpicture} % Some text % \begin{tikzpicture}[overlay, remember picture] \draw[->] (0,0) to[bend left] (my original node); \end{tikzpicture} \end{codeexample} Note that to make |if matches word| even more efficient, you can call it before the preset. It will "register" the corresponding word with |register word| (itself a shortcut for |register word with namespace| with an empty namespace), and call |auto forward words| (shortcut for |auto forward words namespace={}|) in the |latex| preset (or in the current preset if we are already in a preset) to forward all words registered in the corresponding namespace (empty by default). Unless you want to forward different words to different presets, these more advanced functions involving namespaces should not be really useful for the end user. \end{pgfmanualentry} \subsubsection{Pass compiled file to another template}\label{sec:passOutputToOtherTemplate} It can sometimes be handy to use the result of a previous cached file to cache another file, or to do anything else (e.g.\ it can also be practical to debug an issue). |name output| can be used to do that \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/name output=\marg{macro name}\@nil \makeatother% \pgfmanualbody |name output=foo| will create two global macros |\foo| and |\fooInCache|: |\foo| expands to the prefix of the files created in the template like |robExt-somehash|, and |\fooInCache| also adds the cache folder like |robustExternalize/robExt-somehash|. You can then use |set placeholder eval| to send it to another cached file. It is then your role to add the extension, usually |.tex| to get the source (even if the source is a python file), |.pdf| to get the pdf, |-out.tex| to get the file that is loaded before the import, |-out.txt| if you wanted to make it compatible with |verbatim output| (this list is not exhaustive as each script might decide to create a different output file). Here is a demo: \begin{codeexample}[width=0cm,vbox] \begin{CacheMe}{tikzpicture, do not add margins, name output=mycode}[baseline=(A.base)] \node[draw,rounded corners,fill=pink!60](A){Hello World!}; \end{CacheMe}\\[3mm] The prefix is \texttt{\mycode} and with the cache folder it is in:\\ \texttt{\mycodeInCache}.\\ It this can be helpful for instance to debug, as you can inspect the source: \verbatiminput{\mycodeInCache.tex} but it is also practical to define a template based on the previously cached files:\\ \begin{CacheMe}{tikzpicture, set placeholder eval={__previous__}{\mycode.pdf}} \node[rounded corners, fill=green!50]{A cached file can use result from another cached file: \includegraphics[width=2cm]{__previous__}\includegraphics[width=2cm]{__previous__}}; \end{CacheMe} \end{codeexample} Note that if you do not want to display the first cached file, you can use |do not include pdf| to hide it. \end{pgfmanualentry} \subsubsection{Compile in parallel}\label{sec:compileParallel} (introduced in v2.1, read below on how to do it manually on v2.0) \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/compile in parallel=\opt{nb of pictures to compile normally} (default 0)\@nil \extractkey/robExt/compile in parallel after=\opt{nb pictures to compile normally} (default 0)\@nil \makeatother% \pgfmanualbody These two commands are alias. Typing in the preamble: \begin{codeexample}[code only] \robExtConfigure{ compile in parallel } \end{codeexample} will cause the cached elements to be compiled in parallel (this requires two compilations of the main project). For this to work out of the box, you need to have |xargs| installed (on windows, install the lightweight GNU On Windows (Gow) \url{https://github.com/bmatzelle/gow} to get |xargs|). Typing |compile in parallel after=3| will start the compilation in parallel only if you have more than 3 new elements to compile, and compile the first 3 elements normally (useful when you work on one picture at a time since you do not need to compile twice the document). \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/compile in parallel command=\marg{command to run}\@nil \extractkey/robExt/compile in parallel with xargs=\opt{number threads} (default 16\%)\@nil \extractkey/robExt/compile in parallel with gnu parallel=\opt{--jobs option} (default 200\%)\@nil \makeatother% \pgfmanualbody You can customize the compilation command to run via something like: \begin{codeexample}[code only] \robExtConfigure{ compile in parallel command={ parallel --jobs 200\% :::: '\jobname-\robExtAddPrefixName{compile-missing-figures.sh} }, } \end{codeexample} The other styles are predefined commands to compile with xargs (by default, already installed in linux, on Windows you can get it by installing the lightweight GNU On Windows (Gow) \url{https://github.com/bmatzelle/gow}) or gnu parallel. You can use these commands directly like: \begin{codeexample}[code only] \robExtConfigure{ compile in parallel with gnu parallel } \end{codeexample} or use the optional parameter to configure the number of threads (xargs starts with 16 threads by default, and gnu parallel takes twice the number of CPU threads available), like in: \begin{codeexample}[code only] \robExtConfigure{ compile in parallel with xargs=64,% Compiles with 64 processes at a time } \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/if unix=\marg{style to run if running linux or unix}\@nil \extractkey/robExt/if windows=\marg{style to run if windows}\@nil \makeatother% \pgfmanualbody Since windows does not pack |xparse| by default, you might want to enable parallel compilation only in Linux/MacOs. This can be done using the above flags, like: \begin{codeexample}[code only] \robExtConfigure{ if unix={ compile in parallel }, } \end{codeexample} (note that |unix| is understood as ``non windows'', i.e.\ would run in Unix/Linux/MacOs) \end{pgfmanualentry} NB: in version smaller than v2.1 (or if you prefer the manual method), you can instead run it in manual mode and use for instance GNU |parallel| (or any tool of you choice) to run all commands in the file |\jobname-robExt-compile-missing-figures.sh| in parallel. To do that: \begin{itemize} \item First add |\robExtConfigure{enable manual mode}| in your file \item Create the cache folder, e.g.\ using |mkdir robustExternalize| \item Compile your file, with, e.g.\ |pdflatex yourfile.tex| (in my benchmark: 4.2s) \item This should create a file |yourfile-robExt-compile-missing-figures.sh|, with each line containing a compilation command like this (the command should be compatible with Windows, Mac and Linux, if not, let me know on github): \begin{verbatim} cd robustExternalize/ && pdflatex -halt-on-error "robExt-4A806941E86C6B657A0CB3D160CFFF3E.tex \end{verbatim} \textbf{IMPORTANT}: \LaTeX{} might render \textbf{multiple times} the same picture if it is inside some particular environments (align, tables\dots), so this file can contain duplicates if you inserted cached data inside. In version 2.1 this is solved, but on older versions, to avoid running the same command twice, \textbf{be sure to remove duplicates!} On Linux, this can be done by piping the file into |sort| and |uniq| (if your files can depend on previously compiled elements as explained in \cref{sec:passOutputToOtherTemplate}, you should additionally preserve \mylink{https://unix.stackexchange.com/questions/194780/remove-duplicate-lines-while-keeping-the-order-of-the-lines}{the order of the lines}). Since this is anyway a rather advanced use case, we will not consider that case for simplicity. \item Run all these command in parallel, for instance on Linux you can install |parallel|, and run the following command (|--bar| displays a progress bar): \begin{verbatim} cat yourfile-robExt-compile-missing-figures.sh | sort | uniq | parallel --bar --jobs 200% \end{verbatim} (note that since v2.1, it is not necessary to remove duplicates using |sort| and |uniq| since the file contains no more duplicates) In my benchmark, it ran during 52s, so 1.6x faster than the original compilation without caching) \item Recompile the original document with |pdflatex yourfile.tex| \end{itemize} In my benchmark, the total time is 2x4.2s + 52s = 60s, so 1.5 times faster than a normal command (of course, this depends a lot on the preamble of the file that you compile, since the loading time of the file is the main bottleneck for the first compilation; the advantage here is that it is easy to include only the necessary things in the preamble of cached pictures, possibly creating different presets if it is easier to manage it this way). \subsubsection{Compile a template to compile even faster}\label{sec:compileFaster} Long story short: you can compile even faster (at least 1.5x in our tests, but we expect this to be more visible on larger tests due to the loading time of the preamble of the main document that takes most of the time of the compilation) by compiling presets, but beware that you will not be able to modify the placeholders except |add to preamble| with the default compiler we provide. The gain comes from the fact that instead of trying to find all placeholders to replace in the string, we can directly replace a few hard-coded placeholders to save time (yes, \LaTeX{} is not really efficient so it can make a difference\dots). \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/new compiled preset=\marg{preset options to compile}\marg{runtime options}\@nil \extractkey/robExt/compile latex template\@nil \makeatother% \pgfmanualbody Compile a preset by creating a new placeholders that removes all placeholders except |__ROBEXT_MAIN_CONTENT__|, |__ROBEXT_MAIN_CONTENT_ORIG__| and |__ROBEXT_LATEX_PREAMBLE__|. |preset options to compile| should be sure that the template, compilation command and include command placeholders contain the final version: this can be easily done for latex presets by adding |compile latex template| at the end of |preset options to compile|:. \begin{codeexample}[vbox] % We create a latex-based preset and compile it \robExtConfigure{ new preset={templateZX}{ latex, add to preamble={ \usepackage{tikz} \usepackage{tikz-cd} \usepackage{zx-calculus} }, %% possibly add some dependencies }, % We compile it into a new preset new compiled preset={compiled ZX}{templateZX, compile latex template}{}, } % we use that preset automatically for ZX environments \cacheEnvironment{ZX}{compiled ZX} \cacheCommand{zx}{compiled ZX} % Usage: (you can't use placeholders except for the preamble, trade-off of the compiled template) \begin{ZX} \zxX{\sayHey{Bob}} \end{ZX} \end{codeexample} \end{pgfmanualentry} Explanations: in v2.0, we made substantial improvements in order to improve significantly the compilation time (our benchmark went from 20s (v1.0) to only 5.77s (v2.0)), but you might want to make this even faster: with a compiled preset, you can improve it further, experimentation showed an additional improvement factor of 1.4x (final compilation time was 4s in our tests, reminder: without the library it would take 1mn25s). The main bottleneck in term of time is the expansion of the placeholders (that allows great flexibility, but can add a significant time). At a high level, we keep track in a list of all declared placeholders, and loop over them to replace each of them until the string is left unmodified by a full turn. This is simple to implement, does not need to assume any shape for the placeholder\dots{} but not extremely efficient in \LaTeX{} where string and list manipulations are costly. We can play on multiple parameters to speed up the process: \begin{itemize} \item Disable completely the placeholders and only replace a small amount of fixed placeholders in a fixed order and stop expanding the placeholder. This is what we do by default when we compile a preset: it is the quickest solution, but you cannot use arbitrary placeholders. \item Start the expansion using a fixed (preset-dependent) list of placeholders, check if some templates are still present (assuming that templates must contain at least two consecutive underscores |__|), if not stop, otherwise go to the normal (less efficient) placeholder replacement. It is what we do for instance in the latex preset, and gives a great balance between efficiency and flexibility. Details can be found in \cref{sec:advancedOperations}. \item Stop iterating over all placeholders, only choose the one meaningful (why would I care about python placeholders in latex?). For this, we introduce an import system, where you can create placeholders that are not added to the ``global'' list of placeholders, and import them where you want. In practice, you should not really need to use it unless you do more advanced stuff as we already take care of setting it properly, but details are in \cref{sec:importSystem}. \end{itemize} Another option could be to rewrite the code to search directly the list of placeholders in the string, but it is certainly not trivial to do in latex, would require even more constraints on the shape of placeholders, and would certainly still no more efficient than the compiled stuff. But anyway the already implemented solutions already give fairly good performances. \subsection{Default presets}\label{sec:presets} We provide by default some presets for famous languages (for now \LaTeX{} and python). \subsubsection{All languages} First, here are a few options that are available irrespective of the used language. \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/set includegraphics options=\marg{options}\@nil \extractkey/robExt/add to includegraphics options=\marg{options}\@nil \makeatother% \pgfmanualbody Set/add options to the |\includegraphics| run when inserting the pdf (by the default include command). By default it is empty, but the |latex| preset sets it to:\\ |trim=__ROBEXT_LATEX_TRIM_LENGTH__ __ROBEXT_LATEX_TRIM_LENGTH__|\\ |__ROBEXT_LATEX_TRIM_LENGTH__ __ROBEXT_LATEX_TRIM_LENGTH__|\\ in order to remove the margin added in the standalone package options, which is needed to display overlay texts. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/verbatim output\@nil \makeatother% \pgfmanualbody Shortcut for: \begin{verbatim} custom include command={% \evalPlaceholder{% __ROBEXT_VERBATIM_COMMAND__{% __ROBEXT_CACHE_FOLDER____ROBEXT_OUTPUT_PREFIX__-out.txt}% }% }, \end{verbatim} i.e.\ instead of printing the pdf we print the content of the file |__ROBEXT_OUTPUT_PREFIX__-out.txt| using the command in |__ROBEXT_VERBATIM_COMMAND__|, that defaults to |\verbatiminput|: \begin{codeAndResult} \begin{CacheMeCode}{python, verbatim output} with open("__ROBEXT_OUTPUT_PREFIX__-out.txt", "w") as f: for i in range(5): f.write(f"Hello {i}\n") \end{CacheMeCode} \end{codeAndResult} \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/debug\@nil \extractkey/robExt/print command and source\@nil \makeatother% \pgfmanualbody (new in v2.0) These two (alias) commands allow you to print the compilation command, the dependency file (useful to be sure you tracked all dependencies), and the source file: \begin{codeexample}[width=0pt,vbox] \begin{CacheMe}{latex, add dependencies={common_inputs.tex}, add to preamble={\input{__ROBEXT_WAY_BACK__common_inputs.tex}}, debug } The answer is \myValueDefinedInCommonInputs. \end{CacheMe} \end{codeexample} \end{pgfmanualentry} \subsubsection{\LaTeX{} and \tikzname{}} The |latex| preset is used to cache any \LaTeX{} content, like tikz pictures. Note that as of today, it supports overlay content out of the box (if the overlay is more than 30cm long, you might want to customize a placeholder), but not images that need to use |remember picture|. \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/latex\@nil \makeatother% \pgfmanualbody This style sets the template |__ROBEXT_LATEX__| and the compilation command:\\ |__ROBEXT_LATEX_COMPILATION_COMMAND__|\\ (cf \cref{sec:placeholders} for details), and adds a number of styles described below, to easily configure the most common options. You can use it as follows: \begin{codeexample}[width=0pt,vbox] The next picture is cached % \begin{CacheMe}{latex, add to preamble={\usepackage{tikz}}} \begin{tikzpicture}[baseline=(A.base)] \node[fill=red, rounded corners](A){My node that respects baseline.}; \node[fill=red, rounded corners, opacity=.3,overlay] at (A.north east){I am an overlay text}; \end{tikzpicture} \end{CacheMe} and you can see that overlay and depth works. \end{codeexample} \end{pgfmanualentry} To see how to create your own preset or automatically load a library, see \cref{sec:customize}. The next options can be used after calling the |latex| style: \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/latex/use latexmk\@nil \extractkey/robExt/latex/use lualatex\@nil \extractkey/robExt/latex/use xelatex\@nil \makeatother% \pgfmanualbody Use latexmk/lualatex/xelatex to compile. It is a shortcut for:\\ |set placeholder={__ROBEXT_LATEX_ENGINE__}{yourfavoriteengine}| \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/latex/set latex options=\marg{latex options}\@nil \extractkey/robExt/latex/add to latex options=\marg{latex options}\@nil \makeatother% \pgfmanualbody Set/add elements to the set of latex options of the |\documentclass| (it will automatically add a comma before if you add an element). Internally it sets |__ROBEXT_LATEX_OPTIONS__|. By default, it sets:\\ |margin=__ROBEXT_LATEX_TRIM_LENGTH__| (where |__ROBEXT_LATEX_TRIM_LENGTH__| is defined as 30cm by default) in order to add a margin that will be trimmed later in the |\includegraphics|. This is useful not to cut stuff displayed outside of the bounding box (overlays). \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/latex/set documentclass=\marg{documentclass}\@nil \makeatother% \pgfmanualbody Set the documentclass of the document (defaults to |standalone|). Internally, it sets the placeholder |__ROBEXT_LATEX_DOCUMENT_CLASS__|. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/latex/set preamble=\marg{code of preamble}\@nil \extractkey/robExt/latex/add to preamble=\marg{code of preamble}\@nil \extractkey/robExt/latex/set preamble hyperref=\marg{code of preamble}\@nil \extractkey/robExt/latex/add to preamble hyperref=\marg{code of preamble}\@nil \extractkey/robExt/latex/set preamble after hyperref=\marg{code of preamble}\@nil \extractkey/robExt/latex/add to preamble after hyperref=\marg{code of preamble}\@nil \makeatother% \pgfmanualbody Set/add element to the preamble (defaults to |standalone|). Internally, it sets the placeholder |__ROBEXT_LATEX_PREAMBLE__|. The variations |hyperref| and |after hyperref| are used to put stuff after the preambule, as |hyperref| typically needs to be loaded last (or nearly): packages that must be loaded after |hyperref| like |cref| can be added via |add to preamble after hyperref|. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/latex/do not wrap code\@nil \makeatother% \pgfmanualbody By default, the main content is wrapped into a box in order to measure its depth to properly set the baseline. If you do not want to do this wrapping, you can set this option. Internally, it is a shortcut for:\\ |set placeholder={__ROBEXT_LATEX_MAIN_CONTENT_WRAPPED__}{__ROBEXT_MAIN_CONTENT__}| \textbf{IMPORTANT}: note that it means that you might need to adapt your code to take into account the fact that they are inside a box (I don't know of any other solution to compute the depth, but it does not mean that there is none). \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/latex/in command\@nil \makeatother% \pgfmanualbody Sets |__ROBEXT_MAIN_CONTENT| to point to |__ROBEXT_MAIN_CONTENT_ORIG__| to remove any wrapping of the user input, for instance made by tikzpicture. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/tikz\@nil \extractkey/robExt/tikzpicture\@nil \makeatother% \pgfmanualbody |tikz| loads |latex| and then adds |tikz| to the preamble. |tikzpicture| first load |tikz| and wraps the main content within |\begin{tikzpicture}| and |\end{tikzpicture}| using:\\ \begin{verbatim} set placeholder={__ROBEXT_MAIN_CONTENT__}{% \begin{tikzpicture}__ROBEXT_MAIN_CONTENT_ORIG__\end{tikzpicture}% } \end{verbatim} so that the user does not need to type it. See for instance the introduction for examples of use. \end{pgfmanualentry} \subsubsection{Python} We provide support for python: \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/python\@nil \makeatother% \pgfmanualbody Load the |python| preset (inspect |__ROBEXT_PYTHON__|) for details on the exact template, but note that this template might be subject to changes. We also provide a few helper functions: \begin{itemize} \item |write_to_out(text)| writes |text| to the |*-out.tex| file that will be loaded automatically before running the include function \item |parse_args()| is a function that returns a dictionary mapping some keys to values depending on the called arguments: for instance, if you call the python file with |python script key1 value1 key2 value2|, then the dictionary will map |key1| to |value1| and |key2| to |value2|. You might like this in conjunction with commands presented in \cref{sec:configureCompilationCommand}. Note that if you place placeholders in your code, you might not need this, but this is used if you plan to use your script outside of this library. \item |get_cache_folder()| outputs the cache folder. \item |get_file_base()| outputs the prefix of all files that should be created by this script, that looks like |robExt-somehash|. \item |get_current_script()| returns the current script. \item |get_filename_from_extension(extension)| outputs the prefix |robExt-somehash| concatenated with the extension. You often need this function to get the path of a file that your script is creating, for instance, |get_filename_from_extension("-out.txt")| is the path |*-out.txt| of the file that is read by |verbatim output|. \item |get_verbatim_output()| returns |get_filename_from_extension("-out.txt")| \item |finished_with_no_error()| creates the pdf file if it does not exists (to certify that the compilation ran without issues). The template automatically runs this function at the end. \end{itemize} We demonstrate its usage on a few examples: \begin{codeAndResult} \begin{CacheMeCode}{python, verbatim output} with open(get_verbatim_output(), "w") as f: for i in range(5): f.write(f"Hello {i}\n") \end{CacheMeCode} \end{codeAndResult} Note that if you use a version before 2.6, you cannot indent further the python code (i.e.\ the first line must not start with any space). This is fixed in 2.6 with the addition of |remove leading spaces if not disabled| in the python preset by default, that will remove automatically any unnecessary indentation. If you use automatic indentation in your text editor, you might also like to use |remove spaces until| so that automatically indenting your whole document does not ruin the indentation of the python code. You can also generate some images. This code will produce the image in \cref{fig:pythonGeneratedImage2}: \begin{codeexample}[code only] \begin{CacheMeCode}{python, set includegraphics options={width=.8\linewidth}} import matplotlib.pyplot as plt year = [2014, 2015, 2016, 2017, 2018, 2019] tutorial_count = [39, 117, 111, 110, 67, 29] plt.plot(year, tutorial_count, color="#6c3376", linewidth=3) plt.xlabel('Year') plt.ylabel('Number of futurestud.io Tutorials') plt.savefig("__ROBEXT_OUTPUT_PDF__") \end{CacheMeCode} \end{codeexample} {\begin{figure} \centering \begin{CacheMeCode}{python, set includegraphics options={width=.8\linewidth}} import matplotlib.pyplot as plt year = [2014, 2015, 2016, 2017, 2018, 2019] tutorial_count = [39, 117, 111, 110, 67, 29] plt.plot(year, tutorial_count, color="#6c3376", linewidth=3) plt.xlabel('Year') plt.ylabel('Number of futurestud.io Tutorials') plt.savefig("__ROBEXT_OUTPUT_PDF__") \end{CacheMeCode} \caption{Image generated with python.} \label{fig:pythonGeneratedImage2} \end{figure} } Note that by default, the executable called |python| is run. It seems like on windows |python3| is not created and only |python| exists, while on linux the user can choose whether |python| should point to |python3| or |python2| (on NixOs, I directly have |python| pointing to |python3|, and in ubuntu, you might need to install |python-is-python3| or create a symlink, as explained \href{https://askubuntu.com/questions/1296790/python-is-python3-package-in-ubuntu-20-04-what-is-it-and-what-does-it-actually}{here}). In any case, you can customize the name of the executable by setting something like: \begin{verbatim} \setPlaceholder{__ROBEXT_PYTHON_EXEC__}{python3} \end{verbatim} or using the style |force python3| that forces |python3|. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/python print code and result\@nil \makeatother% \pgfmanualbody (Warning: in 2.7 a bug was introduced (fixed in v2.8) where an empty line was automatically added in front of the code) This is a demo style that can print a python code and its result. \begin{codeAndResult} \begin{CacheMeCode}{python print code and result, set title={The for loop}} for name in ["Alice", "Bob"]: print(f"Hello {name}") \end{CacheMeCode} \end{codeAndResult} You can set |__ROBEXT_PYTHON_TCOLORBOX_PROPS__| the options of the tcolorbox,\\ |__ROBEXT_PYTHON_CODE_MESSAGE__| and |__ROBEXT_PYTHON_RESULT_MESSAGE__| which are displayed before the corresponding block, |__ROBEXT_PYTHON_LSTINPUT_STYLE__| which contains the default lstinput style and |__MY_TITLE__| (cf |set title|) that contains the title of the box. Make sure to have the following packages to use the default styling: \begin{verbatim} \usepackage{pythonhighlight} \usepackage{tcolorbox} \end{verbatim} \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/python/add import\@nil \makeatother% \pgfmanualbody (since v2.1) Add an import statement in the first part of the file. \begin{codeexample}[vbox] \robExtConfigure{ add to preset={my python}{ python print code and result, add import={from math import cos}, add import={from math import sin}, }, } \begin{CacheMeCode}{my python} print(cos(1)+sin(2)) \end{CacheMeCode} \end{codeexample} \end{pgfmanualentry} \subsubsection{Bash} We provide a basic bash template, that sets: \begin{verbatim} set -e outputTxt="__ROBEXT_OUTPUT_PREFIX__-out.txt" outputTex="__ROBEXT_OUTPUT_PREFIX__-out.tex" outputPdf="__ROBEXT_OUTPUT_PDF__" \end{verbatim} in order to quit when an error occurs, and to define two variables containing the path to the |pdf| file and to the file that is read by the |verbatim output| setting (that just apply a |\verbatiminput| on that file). Finally, it also creates the file |outputPdf| with |echo "ok"| in order to notify that the compilation succeeded. In practice: \begin{codeAndResult} \begin{CacheMeCode}{bash, verbatim output} # $outputTxt contains the path of the file that will be printed via \verbatiminput uname -srv > "${outputTxt}" \end{CacheMeCode} \end{codeAndResult} \subsubsection{Verbatim text} Sometimes, it might be handy to write the text to a file and use it somehow. This is possible using |verbatim text|, that defaults to calling |\verbatiminput| on that file: \begin{codeAndResult} \begin{CacheMeCode}{verbatim text} def some_verbatim_fct(a): # See this is a verbatim code where I can use the % symbol return a % b \end{CacheMeCode} \end{codeAndResult} You can also call |verbatim text no include|: it will not include the text, but it sets a macro |\robExtPathToInput| containing the path to the input file. Use it the way you like! For instance, we define here a macro |codeAndResult| that prints the code and runs it (we use a pretty printer from pgf, so you need to load |\usepackage{tikz}\input{pgfmanual-en-macros.tex}| to use it). It is what we use right now in this documentation for verbatim blocks like here. You can obtain a simpler version using: \begin{codeAndResult} \begin{CacheMeCode}{verbatim text no include} \NewDocumentCommand{\testVerbatim}{+v}{ \begin{flushleft}\ttfamily% #1 \end{flushleft}} \testVerbatim{Demo % with percent} \end{CacheMeCode} We will input the file \robExtPathToInput{}: \input{\robExtPathToInput} This file contains: \verbatiminput{\robExtPathToInput} \end{codeAndResult} You might also like to use |name output=yourfile| that will create two macros |\yourfile| and |\yourfileInCache|, equal respectively to the prefix |robExt-somehash| and |pathOfCache/robExt-somehash|. \subsubsection{Online images} We also provide support for online images since v2.3. \begin{pgfmanualentry} \extractcommand\includegraphicsWeb\opt{}\opt{\oarg{includegraphics options}}\marg{url image}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/web image\@nil \extractkey/robExt/web image/wget\@nil \extractkey/robExt/web image/curl\@nil \makeatother% \pgfmanualbody |\includegraphicsWeb| is used in place of |\includegraphics| to download online images: \begin{codeexample}[width=0pt,vbox] Here is a cat downloaded online: \includegraphicsWeb[width=3cm]{http://placekitten.com/400/300}. \end{codeexample} By default, the engine uses |wget| in linux (since it is installed by default) and |curl| on windows (should also be installed by default on recent windows). But you can change it by modifying the |web image| preset (or using the optional robExt options that are loaded after |web image|): \begin{codeexample}[width=0pt,vbox] \robExtConfigure{ add to preset={web image}{ curl }, } Here is a cat downloaded online: \includegraphicsWeb[width=3cm]{http://placekitten.com/400/303}. \end{codeexample} By default, the |wget| and |curl| presets are defined as: \begin{verbatim} wget/.style={ set compilation command={wget "__ROBEXT_MAIN_CONTENT__" -O "__ROBEXT_OUTPUT_PDF__"}, }, curl/.style={ set compilation command={curl "__ROBEXT_MAIN_CONTENT__" -L -o "__ROBEXT_OUTPUT_PDF__"}, }, \end{verbatim} \end{pgfmanualentry} \subsubsection{Gnuplot} We also provide support for gnuplot since v2.3. \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/gnuplot\@nil \extractkey/robExt/gnuplot/set terminal=\marg{terminal and options}\@nil \extractkey/robExt/gnuplot/pdf terminal=\opt{pdf terminal options}\@nil \extractkey/robExt/gnuplot/tikz terminal=\opt{tikz terminal options}\@nil \makeatother% \pgfmanualbody Loads the |gnuplot| preset. By default, it uses the |pdf| terminal: \begin{codeAndResult} \begin{CacheMeCode}{gnuplot} plot sin(x) \end{CacheMeCode} \end{codeAndResult} You can also specify the gnuplot options to the pdf terminal, like the size, using: \begin{codeAndResult} \begin{CacheMeCode}{gnuplot,pdf terminal={size 5cm,4cm}} plot sin(x) \end{CacheMeCode} \end{codeAndResult} You can also change the options of the |includegraphics| command used to include the image, using as usual: \begin{codeAndResult} \begin{CacheMeCode}{gnuplot,set includegraphics options={width=5cm,height=4cm}} plot sin(x) \end{CacheMeCode} \end{codeAndResult} You can also use other terminals using the |set terminal| option. For instance, you can use the png terminal and configure its size using: \begin{codeAndResult} \begin{CacheMeCode}{gnuplot,set terminal={png size 400,300}} plot sin(x) \end{CacheMeCode} \end{codeAndResult} The tikz terminal is a bit special since we do not want to use |\includegraphics| but |\input| to include the file. You can use it like this, just make sure: \begin{itemize} \item to have gnuplot installed with lua support. With nix you can get it with:\\ |nix-shell -p '(gnuplot.override { withLua = true; })'| \item Add |\usepackage{gnuplot-lua-tikz}| after making sure to run\\ |gnuplot -e "set terminal tikz createstyle"| once to create the |.sty| file. \end{itemize} \begin{codeAndResult} % \usepackage{gnuplot-lua-tikz} %% Generate with gnuplot -e "set terminal tikz createstyle" \begin{CacheMeCode}{gnuplot,tikz terminal={size 5cm,4cm}} plot sin(x) \end{CacheMeCode} \end{codeAndResult} (Note that if you do not provide any option to |tikz terminal|, it will just use the default value in gnuplot, you can also add any other option to the terminal here.) By default, we only cache the gnuplot compilation, but the |\input| is not cached (so gnuplot will not be run again the next time, but tikz will run every time). But it can also be cached, just add |cache tikz| for that: \begin{codeAndResult} \robExtConfigure{ add to preset={tikz}{ % Run gnuplot -e "set terminal tikz createstyle" in the robustExternalize folder first % to create this file, or, better, run this command in your local folder and use % "copy file to cache" to copy the 3 generated files.: add to preamble={\usepackage{gnuplot-lua-tikz}}, }, add to preset={gnuplot}{ tikz terminal/.append style={ cache tikz, }, }, } \begin{CacheMeCode}{gnuplot, tikz terminal={size 8cm,5cm}} plot cos(x) \end{CacheMeCode} \end{codeAndResult} We also automatically add |__ROBEXT_WAY_BACK__| to the |loadpath|, this way you do NOT need to type |plot "__ROBEXT_WAY_BACK__yourfile.dat"`| if the data is in the main folder. Note however that you need to add this file to the list of dependencies, otherwise it will not be recompiled if the file changes: \begin{codeAndResult} \begin{CacheMeCode}{gnuplot, dependencies={gnuplot_data.dat}} plot "gnuplot_data.dat" \end{CacheMeCode} \end{codeAndResult} You can also use the current text width when using the tikz terminal using something like:\\ |tikz terminal/.expanded={size \lenToCm{\textwidth},\lenToCm{.3\textwidth}}|\\ Full example: \begin{codeAndResult} % Just to check size: \noindent\rule{\textwidth}{1mm}\\% \begin{CacheMeCode}{gnuplot, tikz terminal/.expanded={size \lenToCm{\textwidth},\lenToCm{.3\textwidth}}} plot cos(x) \end{CacheMeCode} \end{codeAndResult} Similar to the tikz terminal, you can also use the |cairolatex| terminal like this: \begin{codeAndResult} \begin{CacheMeCode}{gnuplot,cairolatex terminal} plot sin(x) \end{CacheMeCode} \end{codeAndResult} As before, you can also specify the size, for instance it term of the current textwidth: \begin{codeAndResult} \begin{CacheMeCode}{gnuplot, cairolatex terminal/.expanded={size \lenToCm{\textwidth},\lenToCm{.3\textwidth}}} plot cos(x) \end{CacheMeCode} \end{codeAndResult} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\robExtGpgetvar\marg{variable name}\@@ \extractcommand\robExtGpgetvarNb\opt{*}\opt{\oarg{text, default 404}}\marg{variable name}\@@ \pgfmanualbody Like |\gpgetvar{}| from |gnuplot-lua-tikz| but does not produce an error if the variable does not exist yet (mostly useful if you compile in parallel as shown \mylink{https://github.com/leo-colisson/robust-externalize/issues/17}{here}). The first time, you will get a message like: \robExtGpgetvar{test_variable} and later this will compile fine: \begin{codeAndResult} \begin{CacheMeCode}{gnuplot, tikz terminal={providevars test_variable}} test_variable = 5 plot cos(test_variable * x) title "Test2" \end{CacheMeCode} Variable is \robExtGpgetvar{test_variable}. \end{codeAndResult} |\robExtGpgetvarNb| is similar, except that it prints a number 404 by default (useful if you use it in siunit) and is expandable (also \mylink{https://github.com/leo-colisson/robust-externalize/issues/17}{needed for siunit}). You can choose another text using the optional argument. The star version prints a warning in the log if the variable does not exist, but can only be used in LuaLaTeX: \begin{codeAndResult} Variable is \num[round-mode=figures,round-precision=3]{\robExtGpgetvarNb{non_existing_variable}}. \end{codeAndResult} Note that you will get a warning if the variable does not exist (yet?) only in LuaLaTeX as in other engines there is no \mylink{https://tex.stackexchange.com/questions/705299/print-message-in-expandable-macro/705304}{way to write in the log with an expandable function}. \end{pgfmanualentry} \subsubsection{TikZit} \mylink{https://tikzit.github.io/}{TikZit} is a software to produce \tikzname{} images via a graphical interface. \begin{pgfmanualentry} \extractcommand\cacheTikzit\opt{\oarg{preset}}\@@ \extractcommand\cacheTikzitWithStyle\opt{\oarg{preset}}\marg{name of style file}\@@ \pgfmanualbody (since v2.5) These command will automatically cache the |\tikzfig| command (and therefore also |\ctikzfig|). While |\cacheTikzit| will let you create the preset (named |tikzit| preset by default), |\cacheTikzitWithStyle| will automatically create the preset style for you, loading your style given as parameter: \begin{codeexample}[vbox] %%%%%%% In your preamble: \cacheTikzitWithStyle{new_zxstyle.tikzstyles} %%%%%%% In your document: \tikzfig{demo} % figures/ is optional \end{codeexample} If you prefer to do it manually, you can do instead: \begin{codeexample}[vbox] %%%%%%% In your preamble: \robExtConfigure{ % First, we copy the tikzit-related files to the cache: copy file to cache={tikzit.sty}, copy file to cache={new_zxstyle.tikzstyles}, % We create the tikzit preset loaded by default: new preset={tikzit}{ latex, add to preamble={ \usepackage{tikzit} \input{new_zxstyle.tikzstyles} }, }, } % Cache all tikzit figures: \cacheTikzit %%%%%%% In your document: \tikzfig{figures/demo} \end{codeexample} \textbf{WARNING:} since the tikz picture uses |baseline| that relies on the current font size to vertically align the figure, and since we do not yet propagate the current font size to the cached picture (there are so many ways to configure the font in \LaTeX\dots{}), you should make sure that the |tikzit| preset sets the font size like in your main document, for instance via: \begin{codeexample}[vbox] %%%%%%% In your preamble: \cacheTikzitWithStyle{new_zxstyle.tikzstyles} \robExtConfigure{ add to preset={tikzit}{ add to latex options={12pt}, }, } %%%%%%% In your document: \tikzfig{demo} % figures/ is optional \end{codeexample} Note that |\tikzfig| accepts an additional optional argument for further options, for instance to forward locally a macro or to disable externalization: \begin{codeexample}[vbox] %%%%%%% In your preamble, make sure to load: %% \usepackage{tikzit} %% \input{new_zxstyle.tikzstyles} %% if you want "disable externalization" to work. \cacheTikzitWithStyle{new_zxstyle.tikzstyles} %%%%%%% In your document: \tikzfig{demo} % figures/ is optional \end{codeexample} \end{pgfmanualentry} \subsubsection{Custom languages} To add support to new languages, see \cref{sec:otherLanguages} for an example. \subsection{List of special placeholders and presets}\label{sec:placeholders} This library defines a number of pre-existing placeholders, or placeholders playing a special role. We list some of them in this section. All placeholders created by this library start with |__ROBEXT_|. Note that you can list all predefined placeholders (at least those globally defined) using |\printImportedPlaceholdersExceptDefaults| (note that some other placeholders might be created directly in the style set right before the command, and may not appear in this list if you call it before setting the style). \subsubsection{Generic placeholders} We define two special placeholders that should be defined by the user (possibly indirectly, using presets offered by this library): \begin{itemize} \item |__ROBEXT_TEMPLATE__| is a placeholder that should contain the code of the file to compile. \item |__ROBEXT_MAIN_CONTENT_ORIG__| is a placeholder containing the text typed by the user, automatically set by |CacheMe|, |CacheMeCode| etc. You rarely need to deal with this placeholder directly since |__ROBEXT_MAIN_CONTENT__| will typically point to it and add some necessary wrapping. \item |__ROBEXT_MAIN_CONTENT__| is a placeholder that might be used inside |__ROBEXT_TEMPLATE__|, that points by default to |__ROBEXT_MAIN_CONTENT_ORIG__| and that contains the content that should be typed inside the document. For instance, this might be a tikz picture, a python function without the import etc. Note that it is often used to wrap the text of the user |__ROBEXT_MAIN_CONTENT_ORIG__|: for instance, the |tikzpicture| preset adds the |\begin{tikzpicture}| around the user code automatically: this way we do not need to edit the command to disable externalization. \item |__ROBEXT_COMPILATION_COMMAND__| contains the compilation command to run to compile the file (assuming we are in the cache folder). \end{itemize} We also provide a number of predefined placeholders in order to get the name of the source file etc... Note that most of these placeholders are defined (and/or expanded inplace) late during the compilation stage as one needs first to obtain the hash of the file, and therefore all dependencies, the content of the template etc. \begin{itemize} \item |__ROBEXT_SOURCE_FILE__| contains the path of the file to compile (containing the content of |__ROBEXT_TEMPLATE__|) like |robExt-somehash.tex|, relative to the cache folder (since we always go to this folder before doing any action, you most likely want to use this directly in the compilation command). \item |__ROBEXT_OUTPUT_PDF__| contains the path of the pdf file produced after the compilation command relative to the cache folder (like |robExt-somehash.pdf|). Even if you do not plan to output a pdf file, you should still create that file at the end of the compilation so that this library can know whether the compilation succeeded. \item |__ROBEXT_OUTPUT_PREFIX__| contains the prefix that all newly created file should follow, like |robExt-somehash|. If you want to create additional files (e.g. a picture, a video, a console output etc...) make sure to make it start with this string. It will not only help to ensure purity, but it also allows us to garbage collect useless files easily. \item |__ROBEXT_WAY_BACK__| contains the path to go back to the main project from the cache folder, like |../| (internally it is equals to the expanded value of |\robExtPrefixPathWayBack|). \item |__ROBEXT_CACHE_FOLDER__| contains the path to the cache folder. Since most commands are run from the cache folder, this should not be really useful to the user. \end{itemize} You can also use these placeholders to customize the default include function: \begin{itemize} \item |__ROBEXT_INCLUDEGRAPHICS_OPTIONS__| contains the options given to |\includegraphics| when loading the pdf \item |__ROBEXT_INCLUDEGRAPHICS_FILE__| contains the file loaded by |\includegraphics|, defaults to |\robExtAddCachePathAndName{\robExtFinalHash.pdf}|, that is itself equivalent to |__ROBEXT_CACHE_FOLDER____ROBEXT_OUTPUT_PDF__| or\\|__ROBEXT_CACHE_FOLDER____ROBEXT_OUTPUT_PREFIX__.pdf|. \end{itemize} \subsubsection{Placeholders related to \LaTeX{}} Some placeholders are reserved only when dealing with \LaTeX{} code: \begin{itemize} \item |__ROBEXT_LATEX__| is the main entrypoint, containing all the latex template. It internally calls other placeholders listed below. \item |__ROBEXT_LATEX_OPTIONS__|: contains the options to compile the document, like |a4paper|. Empty by default. \item |__ROBEXT_LATEX_DOCUMENT_CLASS__|: contains the class of the document. Defaults to |standalone|. \item |__ROBEXT_LATEX_PREAMBLE__|: contains the preamble. Is empty by default. \item |__ROBEXT_LATEX_MAIN_CONTENT_WRAPPED__|: content inside the |document| environment. It will wrap the actual content typed by the user |__ROBEXT_MAIN_CONTENT__| around a box to compute its depth. If you do not want this behavior, you can set |__ROBEXT_LATEX_MAIN_CONTENT_WRAPPED__| to be equal to |__ROBEXT_MAIN_CONTENT__|. It calls internally |__ROBEXT_LATEX_CREATE_OUT_FILE__| and |__ROBEXT_LATEX_WRITE_DEPTH_TO_OUT_FILE__| to do this computation. \item |__ROBEXT_LATEX_CREATE_OUT_FILE__| creates a new file called |\jobname-out.tex| and open it in the handle called |\writeRobExt| \item |__ROBEXT_LATEX_WRITE_DEPTH_TO_OUT_FILE__| writes the height, depth and width of the box |\boxRobExt| into the filed opened in |\writeRobExt|. \item |__ROBEXT_LATEX_COMPILATION_COMMAND__| is the command used to compile a \LaTeX{} document. It uses internally other placeholders: \item |__ROBEXT_LATEX_ENGINE__| is the engine used to compile the document (defaults to |pdflatex|) \item |__ROBEXT_LATEX_COMPILATION_COMMAND_OPTIONS__| contains the options used to compile the document (defaults to |-shell-escape -halt-on-error|) \end{itemize} \subsubsection{Placeholders related to python} \begin{itemize} \item |__ROBEXT_PYTHON_EXEC__| contains the python executable (defaults to |python|) used to compile \item |__ROBEXT_PYTHON__| contains the python template \item |__ROBEXT_PYTHON_IMPORT__| can contain import statements \item |__ROBEXT_PYTHON_MAIN_CONTENT_WRAPPED__| is used to add all the above functions. You can set it to |__ROBEXT_MAIN_CONTENT__| if you do not want them \item |__ROBEXT_PYTHON_FINISHED_WITH_NO_ERROR__| is called at the end to create the pdf file even if it is not created, you can set it to the empty string if you do not want to do that. \end{itemize} \subsubsection{Placeholders related to bash} \begin{itemize} \item |__ROBEXT_BASH_TEMPLATE__| contains the bash template. By default, it sets |set -e|, creates |outputTxt|, |outputTex| and |outputPdf| pointing to the corresponding files, and it created the pdf file at the end. \item |__ROBEXT_SHELL__| contains the shell (defaults to |bash|). \end{itemize} \subsection{Customize presets and create your own style}\label{sec:customize} Note that you can define your own presets simply by creating a new pgf style (please refer to tikz-pgf's documentation for more details). For instance, we defined the |tikz| and |tikzpicture| presets using: \begin{codeexample}[code only] \robExtConfigure{ new preset={tikz}{ latex, add to preamble={\usepackage{tikz}}, }, new preset={tikzpicture}{ tikz, set placeholder={__ROBEXT_MAIN_CONTENT__}{\begin{tikzpicture}__ROBEXT_MAIN_CONTENT_ORIG__\end{tikzpicture}}, }, } \end{codeexample} in order to automatically load |tikz| and add the surrounding |tikzpicture| when needed (note that the style is always loaded \textbf{after} the definition of |__ROBEXT_MAIN_CONTENT_ORIG__|, so in theory you could also modify it directly even if it is not recommended). You can also customize an existing style by adding stuff to it using |add to preset| (or |.append style| but make sure to double the hashes). For instance, here, we add the |shadows| library to the |tikz| preset by default: \begin{codeexample}[width=0pt,vbox] \robExtConfigure{ add to preset={tikz}{ add to preamble={\usetikzlibrary{shadows}}, }, } See, tikz's style now packs the |shadows| library by default: % \begin{CacheMe}{tikzpicture}[even odd rule] \filldraw [drop shadow,fill=white] (0,0) circle (.5) (0.5,0) circle (.5); \end{CacheMe} \end{codeexample} \subsection{Cache automatically a given environment}\label{sec:wrapAutomatically} It might be handy to cache automatically a given environment: we already provide:\\ |\cacheTikz|\\ to cache all tikz pictures (unless externalization is disabled), but we also provide tools to handle arbitrary environments. \begin{pgfmanualentry} \extractcommand\robExtExternalizeAllTikzpictures\opt{\oarg{preset for tikz}}\opt{\oarg{preset for tikzpicture}}\opt{\oarg{delimiters}}\@@ \extractcommand\cacheTikz\opt{\oarg{preset for tikz}}\opt{\oarg{preset for tikzpicture}}\opt{\oarg{delimiters}}\@@ \extractcommand\robExtDoNotExternalizeAllTikzpictures\@@ \extractcommand\doNotCacheTikz\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/cache tikz\@nil \extractkey/robExt/cache tikz\@nil \extractkey/robExt/cache tikz 2 args=\marg{tikz preset}\marg{tikzpicture preset}\@nil \extractkey/robExt/cache tikz 3 args=\marg{tikz preset}\marg{tikzpicture preset}\marg{delimiters cacheMe options}\@nil \makeatother% \pgfmanualbody (|cache tikz*| from 2.3, simply style alias) This will automatically cache all tikz pictures (the |*ExternalizeAllTikzpicturers| are alias of the |*cacheTikz| which is available from v2.0). Since v2.0, we added the delimiters options and we allow to specify custom presets for |tikz| and |tikzpicture|, and we parse |\tikz| as well (the |tikz| preset is now used by |\tikz| by default while the |tikzpicture| preset is used by |tikzpicture|). Note that we add an additional optional argument to the tikz picture via its first argument delimited by |delimiters| (defaults to |<>|) to specify preset options, which can for instance be practical to disable externalization on individual pictures. Cf.\ \cref{sec:disableExternalization} to see an example of use. \begin{codeexample}[width=0pt,vbox] \cacheTikz \def\whereIAm{(not cached)} \robExtConfigure{ add to preset={tikz}{ add to preamble={ \def\whereIAm{(cached)} }, }, } \tikz \node[fill=pink, rounded corners]{See that this syntax can be safely used. \whereIAm}; \tikz \node[fill=green, rounded corners]{\sayHello{Alice} \whereIAm}; \tikz \node[fill=green, rounded corners]{Foo \whereIAm}; \end{codeexample} Here is an example to specify arguments via parenthesis: \begin{codeexample}[width=0pt,vbox] % the first two arguments are the default presets used by \tikz and \tikzpicture \cacheTikz[tikz][tikzpicture][()] \def\whereIAm{(not cached)} \robExtConfigure{ add to preset={tikz}{ add to preamble={ \def\whereIAm{(cached)} }, }, } \tikz \node[fill=pink, rounded corners]{See that this syntax can be safely used. \whereIAm}; \tikz(add to preamble={\def\sayHello#1{Hello #1!}}) \node[fill=green, rounded corners]{\sayHello{Alice} \whereIAm}; \tikz(disable externalization) \node[fill=green, rounded corners]{Foo \whereIAm}; \end{codeexample} You can also change the default style loaded for tikz and tikzpicture (note that you might prefer to modify |tikz| directly using |add to preset|), but you need v2.1 (buggy in v2.0): \begin{codeexample}[vbox] % You might prefer to modify tikz directly using "add to preset" \cacheTikz[tikz, add to preamble={\def\hello{H}},][tikzpicture, add to preamble={\def\hello{HH}}, ] \tikz[baseline=(A.base)] \node(A){In tikz \hello}; % \begin{tikzpicture}[baseline=(A.base)] \node(A){In tikzpicture \hello}; \end{tikzpicture} \end{codeexample} Note that the |cache tikz| styles (the variation just provide default values) is useful for instance if you only want to locally enable |cache tikz| for the |\input| of a style (this way you can cache the output of a cached thing\dots{}). For instance, |gnuplot| with a tikz terminal will compile into a tex file that will be inputted. But unless |\cacheTikz| is enabled, the inputted picture will not be cached (i.e. tikz will need to run). If you still want to cache it without enabling |\cacheTikz| globally, you can load this style in the |gnuplot| style, or do something like: |\doNotCacheTikz| just undo |\cacheTikz|: \begin{codeexample}[width=0pt,vbox] \def\whereIAm{(not cached)} \robExtConfigure{ add to preset={tikz}{ add to preamble={ \def\whereIAm{(cached)} }, }, } \cacheTikz \doNotCacheTikz \tikz \node[fill=pink, rounded corners]{\whereIAm}; \begin{tikzpicture} \node[fill=pink, rounded corners]{\whereIAm}; \end{tikzpicture} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\robExtDisableTikzpictureOverwrite\@@ \pgfmanualbody This is useful to temporarily reset the current environment to their original value (since v2.0 it also resets other environments, not just |tikzpicture|). This must typically be called at the beginning of |command if no externalization| to avoid infinite recursion if you redefine it, but expect for this case the user is not expected to use this option. \end{pgfmanualentry} Let us say that you want to cache all elements of a given environment, like |minipage| or |zx-calculus| pictures (another package of mine): \begin{pgfmanualentry} \extractcommand\cacheEnvironment\opt{\marg{delimiters}}\marg{name environment}\marg{default preset options}\@@ \pgfmanualbody This will automatically cache the corresponding environment (but note that you still need to define it in the preamble of the cached files, for instance by loading the appropriate package): \begin{codeexample}[width=0pt,vbox] \cacheEnvironment{ZX}{latex, add to preamble={\usepackage{zx-calculus}}} A ZX figure % \begin{ZX} \zxX{\alpha} \rar \ar[d,C] & \zxZ*{a\pi} \\ \zxZ{\beta} \rar & \zxX*{b\pi} \end{ZX} and the same but aligned on the second line % \begin{ZX}[mbr=2] \zxX{\alpha} \rar \ar[d,C] & \zxZ*{a\pi} \\ \zxZ{\beta} \rar & \zxX*{b\pi} \end{ZX}. \end{codeexample} You can of course configure them using globally defined configuration, but you can also provide arguments to a single picture using the delimiters |delimiters| that default to |<...>| as the optional argument, like: % \begin{codeexample}[width=0pt,vbox] \cacheEnvironment{ZX}{latex, add to preamble={\usepackage{zx-calculus}}} \begin{ZX} % amsmath provides \text \zxX{\alpha} \rar \ar[d,C] & \zxZ{\text{yes}} \\ \zxZ{\beta} \rar & \zxX*{b\pi} \end{ZX} \end{codeexample} If you do not like |<>| or if your command already have this parameter, you can change it, for instance to get parens as delimiters, use: \begin{codeexample}[width=0pt,vbox] \cacheEnvironment[()]{ZX}{latex, add to preamble={\usepackage{zx-calculus}}} \begin{ZX}(add to preamble={\usepackage{amsmath}}) % amsmath provides \text \zxX{\alpha} \rar \ar[d,C] & \zxZ{\text{yes}} \\ \zxZ{\beta} \rar & \zxX*{b\pi} \end{ZX} \end{codeexample} otherwise you must provide some presets arguments, possibly empty:\\ |\begin{someweirdenv}<>| You can also use this argument to disable externalisation on a per-picture basis: \begin{codeexample}[width=0pt,vbox] \cacheEnvironment{ZX}{latex, add to preamble={\usepackage{zx-calculus}}} \begin{ZX} % amsmath provides \text \zxX{\alpha} \rar \ar[d,C] & \zxZ*{a\pi} \\ \zxZ{\beta} \rar & \zxX*{b\pi} \end{ZX} \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\cacheCommand\opt{\oarg{delimiter}}\marg{name command}\opt{\oarg{xparse signature}}\marg{default preset options}\@@ \pgfmanualbody This will automatically cache the corresponding command, in a similar way to environment. It work exactly the same, except that we (usually) need to specify the signature of the command or it is impossible to know where the command stops. If the command is defined using xparse-compatible commands like |\NewDocumentCommand|, this detection is automatically done. Otherwise, you need to specify it in the xparse format: long story short, |O{foo}| is an optional argument with default value foo, |m| is a mandatory argument. First, let us see an example with a command defined using xparse: \begin{codeexample}[width=0pt,vbox] % We cache the command \zx that is defined with NewDocumentCommand: no need to specify the signature \cacheCommand{zx}{latex, add to preamble={\usepackage{zx-calculus}}} A ZX figure % \zx{ \zxX{\alpha} \rar \ar[d,C] & \zxZ*{a\pi} \\ \zxZ{\beta} \rar & \zxX*{b\pi} } and the same but aligned on the second line % \zx[mbr=2]{ \zxX{\alpha} \rar \ar[d,C] & \zxZ*{a\pi} \\ \zxZ{\beta} \rar & \zxX*{b\pi} }. \end{codeexample} You can of course configure them using globally defined configuration, but you can also provide arguments to a single picture using the delimiters that default to |<>| as the optional argument, like: % \begin{codeexample}[width=0pt,vbox] \cacheCommand{zx}{latex, add to preamble={\usepackage{zx-calculus}\def\hello#1{Hello #1.}}} \zx[mbr=1]{ % amsmath provides \text \zxX{\alpha} \rar \ar[d,C] & \zxZ{\text{\hello{Alice}}} \\ \zxZ{\beta} \rar & \zxZ{\text{\bye{Bob}}} } \end{codeexample} or with parens as delimiters: \begin{codeexample}[width=0pt,vbox] \cacheCommand[()]{zx}{latex, add to preamble={\usepackage{zx-calculus}}} \zx(add to preamble={\usepackage{amsmath}})[mbr=1]{ % amsmath provides \text \zxX{\alpha} \rar \ar[d,C] & \zxZ{\text{yes}} \\ \zxZ{\beta} \rar & \zxX*{b\pi} } \end{codeexample} Here is an example with a command not defined with |\NewDocumentCommand| (we also show how you can use the optional argument to disable externalisation on a per-picture basis): \begin{codeexample}[width=0pt,vbox] \def\fromWhere{I am not cached} \newcommand{\myMacroNotDefinedWithNewDocumentCommand}[3][default value]{ Optional argument 1: #1, Argument 2: #2, Argument 3: #3 (\fromWhere). } \cacheCommand{myMacroNotDefinedWithNewDocumentCommand}[O{default value}mm]{ latex, add to preamble={% \def\fromWhere{I am cached} \newcommand{\myMacroNotDefinedWithNewDocumentCommand}[3][default value]{ Optional argument 1: #1, Argument 2: #2, Argument 3: #3 (\fromWhere). } } } \myMacroNotDefinedWithNewDocumentCommand{second arg}{last arg}\\ \myMacroNotDefinedWithNewDocumentCommand{second arg}{last arg} \end{codeexample} \end{pgfmanualentry} \paragraph{Defining its own macro without repeating its definition.} Note that when you define yourself a macro function, the above structure might not be optimal as you need to define the macro in the main document (in case you disable the externalization) and in the template. One option to avoid repeated code is to write the definition in a common input file (see \cref{sec:dependencies}), use |\runHereAndInPreambleOfCachedFiles| or |forward=\macroToExport|. But sometimes, you might need to do more involved operations, for instance to add a file as a dependency etc. So another solution it to wrap yourself your function into |cacheMe| yourself. Say you want your function to draw a circle like, |\tikz \draw[fill=#2] (0,0) circle [radius=#1];|, then wrap in inside |CacheMe|: \begin{codeexample}[width=0mm,vbox] % D<>{} is an optional argument enclosed in <>, we use this to pass arguments to CacheMe \NewDocumentCommand{\drawMyCircle}{D<>{}O{2mm}m}{ \begin{CacheMe}{tikz, #1} \tikz \draw[fill=#3] (0,0) circle [radius=#2]; \end{CacheMe} } You can externalize it \drawMyCircle{pink} % or disable externalization \drawMyCircle[4mm]{red}. \end{codeexample} \paragraph{Manually wrap a command.} In some more advanced cases, if you do not know the definition of the command and if |\cacheCommand| does not work (e.g. the number of mandatory argument depends on the first argument), you can wrap it manually. To do so, first copy first your command using |\DeclareCommandCopy{\robExtCommandOrigNAMEOFCOMMAND}{\NAMEOFCOMMAND}| and run |\robExtAddToCommandResetList{NAMEOFCOMMAND}| (this is needed to disable externalization), and override the command by wrapping it into a cache. Make sure to add |\def\robExtCommandOrigName{NAMEOFCOMMAND}| at the beginning of the function or |disable externalization| will not work: \begin{codeAndResult} Original command: \zx{\zxZ{\alpha} \rar & \zxX{\beta}}. % \DeclareCommandCopy{\robExtCommandOrigzx}{\zx}% used to recover the original function \robExtAddToCommandResetList{zx}% let us know that this function should be reset when externalization is disabled \DeclareDocumentCommand{\zx}{D<>{}O{}m}{% \def\robExtCommandOrigName{zx}% \begin{CacheMe}{latex, add to preamble={\usepackage{zx-calculus}}, #1}% \zx[#2]{#3}% \end{CacheMe}% } This is cached \zx<>{\zxZ{\alpha} \rar & \zxX{\beta}}, you can add options like % \zx{\zxZ{\alpha} \rar & \zxX{\text{Hey}}} % and we can disable the externalization: % \zx{\zxZ{\alpha} \rar & \zxX{\text{Hey}}}. \end{codeAndResult} You can do a similar trick for environment, by using instead \begin{verbatim} \NewEnvironmentCopy{robExtEnvironmentOrigNAMEENV}{NAMEENV} \robExtAddToEnvironmentResetList{NAMEENV} \def\robExtEnvironmentOrigName{NAMEENV} \end{verbatim} \subsection{Operations on the cache}\label{sec:operationsCache} Every time we compile a document, we create automatically a bunch of files: \begin{itemize} \item the cache is located by default in the |robustExternalize| folder. Feel free to remove this folder if you want to completely clear the cache (but then you need to recompile everything). See below if you want to clean it in a better way. \item |\jobname-robExt-all-figures.txt| contains the list of all figures contained in the document. Mostly useful to help the script that remove other figures. \item |robExt-remove-old-figures.py| is a python script that will remove all cached files that are not used anymore. Just run |python robExt-remove-old-figures.py| to clean it. You will then see the list of files that the script wants to remove: make sure it does not remove any important data, and press ``y''. Note that it will search for all files that look like |*robExt-all-figures.txt| to see the list of pictures that are still in use, and by default it will only remove the images in the |robustExternalize| folder that start with |robExt-|. If you change the path of the cache or the prefix, edit the script (should not be hard to do). \item |\jobname-robExt-compile-missing-figures.sh| contains a list of commands that you need to run to compile the images not yet compiled in the cache (this list will only be created if you enable the manual compilation mode). \item |\jobname-robExt-tmp-file-you-can-remove.tmp| is a temporary file. Feel free to remove it. \end{itemize} We go over some of these scripts. \subsubsection{Cleaning the cache} You might want to clean the cache. Of course you can remove all generated files, but if you want to keep the picture in use in the latest version of the document, we provide a python script (automatically generated in the root folder) to do this. Just install python3 and run:\\ |python3 robExt-remove-old-figures.py|\\ (on windows, the executable might be called |python|) You will then be prompted for a confirmation after providing the list of files that will be removed. \subsubsection{Listing all figures in use} After the compilation of the document, a file |robExt-all-figures.txt| is created with the list of the |.tex| file of all figures used in the current document. \subsubsection{Manually compiling the figures} When enabling the manual mode (useful if we don't want to enable |-shell-escape|): \begin{verbatim} \robExtConfigure{ enable manual mode } \end{verbatim} or the fallback to manual mode (it will only enable the manual mode if |-shell-escape| is disabled): \begin{verbatim} \robExtConfigure{ enable fallback to manual mode, } \end{verbatim} the library creates a file |JOBNAME-robExt-compile-missing-figures.sh| that contains the instructions to build the figures that are not yet in the cache (each line contains the compilation command to run). On Linux (or on Windows with bash/cygwin/… installed, it possibly even work out of the box without) you can easily execute them using: \begin{verbatim} bash JOBNAME-robExt-compile-missing-figures.sh \end{verbatim} \subsection{How to debug}\label{sec:debug} If for some reasons you are unable to understand why a build fails, first check if you compiled your document with |-shell-escape| (not that this must appear \textbf{before} the filename). Since v2.3, we provide a more advanced error handling system in order to provide more meaningful errors to the user, and to write the full compilation output in the log in case of an error. We do so by wrapping commands around something like |(my command) > somelog.tex 2>&1|; therefore nothing is printed on screen during the compilation of the cached picture, you only get a message if there is an error. \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/do not redirect compilation output\@nil \makeatother If you want to come back to the old behavior, that just runs the command is it either set this option, or set |\def\robExtDoNotRedirectOutput{}| and |\def\robExtDoNotRedirectOutputInManuallyCompileMissingFigures{}| (the last one will only concern commands written to the |robExt-compile-missing-figures.sh|, which is used in manual mode and in parallel mode). \end{pgfmanualentry} Starting from 2.3, we also automatically print an actual error message like: \begin{verbatim} ! Package robExt Error: On line 20: the pdf file (robExt) robustExternalize/robExt-F27CD43D192AA2668A16DEAA616591FF.pdf (robExt) is not present. The compilation command "cd (robExt) robustExternalize/ && xelatex -halt-on-error (robExt) "robExt-F27CD43D192AA2668A16DEAA616591FF.tex"" used (robExt) to compile the environment on line 20 certainly (robExt) failed with errors: (robExt) vvvvvv (robExt) ! Package tikz Error: Giving up on this path. Did (robExt) you forget a semicolon?. (robExt) (robExt) ^^^^^^ (robExt) See full logs below (you might need to press ENTER (robExt) to go to the next error) or in (robExt) robustExternalize/robExt-F27CD43D192AA2668A16DEAA616591FF-compilation.log. \end{verbatim} You can see inside the location in the main tex file of the block containing the error, we provide also the compilation command that lead to this error, and the lines of the log containing the word ``error'' (case insensitive), as well as the path to the full log of the compilation of the inner command, that you can also see if you continue the compilation by pressing Enter or by looking at the full log file, for instance in TeXstudio. This smaller message is often good enough in LaTeX, but usually not very good for languages that do not print the word ``error''. In that case, add to your preset: \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/print whole file in error message\@nil \extractkey/robExt/do not print whole file in error message\@nil \makeatother (Add this only in preset, do NOT set it globally) If you want to print the whole file in the small error message because you do not want to inspect the full log file, you can use the style |print whole file in error message|. This is enabled by default on the gnuplot, python and bash presets for instance, but disabled in \LaTeX{} has it makes little sense to print thousands of log lines in the error. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/texstudio\@nil \extractkey/robExt/logs should show as errors\@nil \extractkey/robExt/prefix log message with=\marg{string to prepend to log message}\@nil \extractkey/robExt/spacing between lines in log=\marg{content to separate lines in log}\@nil \makeatother Some IDE like TeXstudio will only print lines starting with an exclamation mark, assuming that they are spaced enough, so to make the compilation errors more visible in the log, we can start every line in the log with a |!| (you can set it globally via |logs should show as errors| that internally calls |prefix log message with|) and separate all lines with enough empty lines using |spacing between lines in log|. For simplicity, we create |texstudio| to automatically configure these values so that errors print nicely in texstudio via: \begin{verbatim} \robExtConfigure{texstudio} \end{verbatim} This way you will get errors like: \begin{verbatim} -------- We print now the full log -------- ! sh: ligne 1: gnuplot : commande introuvable -------- End of the full log -------- \end{verbatim} and the line |sh: ligne 1: gnuplot : commande introuvable| will appear in your IDE without going in the log file. Note that this will only be printed for items that enable |print whole file in error message| not to pollute the errors space when the log contains many lines (e.g. \LaTeX{}), but this is automatically enabled on most default presets. \end{pgfmanualentry} Then, you can look at the log files\footnote{Either the latex one or the |-compilation.log| file, and sometimes try in a terminal if you really can't see any useful error as it might be the case that the error is not printed in the log file, which occurs for instance if xparse is not installed on Windows and you use the compilation command in parallel.} to get more advices, and run yourself the compilation command printed in the above error message. This way, you can easily check the content of the file and see why it fails to compile. In version before 2.3, or if you use |do not redirect compilation output|, the compilation errors are also displayed directly in the output, and you can read the log file for details. You might often get an error: \begin{verbatim} ! Missing $ inserted. \end{verbatim} This is typically when a placeholder was left unreplaced (e.g.\ you forgot to define it, import it, or you forgot to wrap a command in |\evalPlaceholder{}|): since \LaTeX{} is asked to typeset |__something__|, it thinks that you are trying to write a subscript, and asks you to start first the math mode. To get more advanced info during the compilation command, set the style |more logs| (that does |\def\robExtDebugMessage#1{\message{^^J[robExt] #1}}|) in order to print some messages on the replacement process (there is also |\robExtDebugWarning| for more important messages, but it prints the logs by default. Then, it is often handy to check the content of the generated template. You can either get the name in the log, via |name output|, or print it directly in the file: \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/debug\@nil \extractkey/robExt/print command and source\@nil \makeatother% \pgfmanualbody You can also use |debug| (or its alias |print command and source|) in order to disable the compilation and print instead the compilation command and the source file: \begin{codeexample}[width=0pt,vbox] \begin{tikzpictureC}[baseline=(A.base)] \node[fill=red, rounded corners](A){My node that respects baseline \ding{164}.}; \node[fill=red, rounded corners, opacity=.3,overlay] at (A.north east){I am an overlay text}; \end{tikzpictureC} \end{codeexample} \begin{codeexample}[width=0pt,vbox] \end{codeexample} \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\robExtShowPlaceholder\opt{*}\marg{placeholder name}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/show placeholder=\marg{placeholder name}\@nil \makeatother% \pgfmanualbody |\robExtShowPlaceholder{__ROBEXT_MAIN_CONTENT__}| will show in the terminal the content of the placeholder. Use the start not to pause. \end{pgfmanualentry} \begin{pgfmanualentry} \extractcommand\robExtShowPlaceholders\opt{*}\@@ \extractcommand\robExtShowPlaceholdersContents\opt{*}\@@ \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/show placeholders\@nil \extractkey/robExt/show placeholders contents\@nil \makeatother% \pgfmanualbody Show all imported placeholders and their content. Use the start not to pause. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/nb lines after error to show=\marg{nb lines to keep}\@nil \extractkey/robExt/remove line number\@nil \extractkey/robExt/do not remove line number\@nil \makeatother In order to make the small error message more meaningful (to avoid getting to the full log), we try to extract some errors automatically, reading lines staring with |!|, or containing |error| or |invalid|. But this line alone is often not enough, so you can use |nb lines after error to show| to determine the number of lines to show after (defaults to 3). |remove line number| will remove in the short error message any |l.| at the beginning of the line, otherwise some editors (at least emacs), will interpret it as a line number in the current file and will jump to a bad position (this is enabled by default). These options would work best if set globally when you use |compile in parallel|. \end{pgfmanualentry} \subsection{Advanced optimizations}\label{sec:advancedOperations} We made a number of optimizations to get near instant feedback on documents having hundreds of pictures, basically by saying which placeholder should be replaced, in which order, when to stop replacing placeholders, and possibly by compiling templates (see \cref{sec:compilePreset}). Most of them are invisible to the end user and pre-configured for presets like |latex| or when compiling a preset, but you might want to also use them on your own. First note that the order used to import placeholders is important, as it will first replace the first imported placeholders: so if you imported the placeholders in the right order, a single loop should be enough to replace them all, otherwise you might need $n$ loops if you need to replace $n$ placeholders (the basic evaluation procedure just loops over placeholders and replace all occurrences in the string until there is either no remaining placeholders (see below), or if you obtain the same result after two loops). \begin{pgfmanualentry} \extractcommand\onlyPlaceholdersInCompilationCommand\marg{list,of,placeholders}\@@ \extractcommand\firstPlaceholdersInCompilationCommand\marg{list,of,placeholders}\@@ \extractcommand\onlyPlaceholdersInTemplate\marg{list,of,placeholders}\@@ \extractcommand\firstPlaceholdersInTemplate\marg{list,of,placeholders}\@@ \pgfmanualbody Using these commands (possibly inside |/utils/exec={\yourcommand{foo}}| in a style), you will tell to the system to evaluate the compilation/template placeholders (depending on the name of the command) only using the placeholders in the argument, or (for the commands starting with |first|), by starting with the placeholders in argument before continuing with others placeholders. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/all placeholders have underscores\@nil \extractkey/robExt/not all placeholders have underscores\@nil \makeatother% \pgfmanualbody In order to know when to stop replacing placeholders earlier (i.e.\ without running until it converges), it is practical to know how placeholders look like in order to stop before. This (enabled by default) says to the library that all placeholders should start with two underscores |__|, allowing us to easily check if a string contains a placeholder. If you want to use placeholders with fancy names and no underscores, disable this feature (but it might be a bit slower). \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/do not load special placeholders\@nil \extractkey/robExt/load special placeholders\@nil \makeatother% \pgfmanualbody Similarly, to save some time, we do not import some special placeholders and only replace them manually at the end, the list being: |__ROBEXT_OUTPUT_PREFIX__|, |__ROBEXT_SOURCE_FILE__|, |__ROBEXT_OUTPUT_PDF__|, |__ROBEXT_WAY_BACK__|, |__ROBEXT_CACHE_FOLDER__|. If for some reasons you do want to import them earlier, you can do it this way. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/disable placeholders\@nil \extractkey/robExt/enable placeholders\@nil \makeatother% \pgfmanualbody To save even more time, you can disable the placeholder system completely, and only replace the following placeholders: |__ROBEXT_MAIN_CONTENT__|, |__ROBEXT_LATEX_PREAMBLE__| and |__ROBEXT_MAIN_CONTENT_ORIG__|. This is done notably inside |new compiled preset| in order to speed up the compilation by maybe a factor of 1.5 or 2, but then you cannot use any other placeholders. Note that the default presets like |latex| enable placeholders, so you should be able to call |latex| inside a compiled template to come back to the uncompiled version. \end{pgfmanualentry} \begin{pgfmanualentry} \makeatletter% \def\extrakeytext{style, } \extractkey/robExt/disable import mechanism\@nil \extractkey/robExt/enable optimizations\@nil \makeatother% \pgfmanualbody If you disable the import mechanism (enabled by default), it will do an |\importAllPlaceholders| while evaluating the template. This is not recommended as it is quite inefficient, it is better to selectively import the group you need. \end{pgfmanualentry} \section{Frequent errors} \subsection{Xelatex and tabs} It seems that xelatex (I never had issues with pdflatex and lualatex, but if you get this error on other engines you can try the same fix) \mylink{https://tex.stackexchange.com/a/14776/116348}{has a hard time dealing with tabs} if you do not compile with: \begin{verbatim} xelatex -8bit yourfile.tex \end{verbatim} If you see that tabs in your input file are turned into |^^I| in the source code, you should either add the |-8bit| option\dots{} or simply follow egreg's advice: \begin{center} \emph{The strategy is very simple: never use TAB when dealing with TeX. Reserve the TAB key to trigger autocompletion and your life will be happier.} \end{center} If you use TeXstudio that uses Tabs by default, you can disable this feature by setting these two options: \begin{verbatim} Options > Configure TeXstudio > Editor > Replace indentation tabs with spaces Options > Configure TeXstudio > Editor > Replace tabs in text with spaces \end{verbatim} \section{TODO and known bugs:} \begin{itemize} \item See how to deal with label, links and cite inside pictures (without disabling externalization). For label and links, I have a proof of concept, I ``just'' need to write it down. \url{https://tex.stackexchange.com/questions/695277/clickable-includegraphics-for-cross-reference-data} \item Deal with remember picture \item Since externalizing is all about speed, it would be nice to do more benchmarks. But overall, it seems fairly optimized, at least when using the compiled presets. One option to gain in term of efficiency and simplicity might be to directly use the fact that the placeholders start and end with two dashes: a more efficient implementation might be doable using something like |\def\findPlaceholders#1__#2__{}| instead of string replacement. I will try to investigate in future releases. \end{itemize} \section{Changelog} \begin{itemize} \item v2.9 (2024/03/15) \begin{itemize} \item Bug that may forbid the package to load has been fixed (you might encounter |ERROR: Missing = inserted for \ifnum|) \end{itemize} \item v2.8 (current master, not yet released): \begin{itemize} \item Fix a regression bug introduced in 2.7: |python print code and result| was adding a new line above the code printed. This is fixed now. \end{itemize} \item v2.7: \begin{itemize} \item apply |remove leading spaces| by default also in |placeholderFromCode| and |PlaceholderPathFromCode|, and allow the user to customize the style. This produces a very minor backward incompatibility as it might disturb code where all lines start with a space. \end{itemize} \item v2.6: \begin{itemize} \item Added |\lenToCmNoUnit| \item Added |\placeholderPrependAllLines| and alike \item Added |remove spaces until| and alike \item Added |placeholderRemoveLeadingSpaces| and alike \end{itemize} \item v2.5: \begin{itemize} \item Add tikzit support \item Add support for |--output-directory| \end{itemize} \item v2.4: \begin{itemize} \item Better error message in wget if file not found \item Add support for arXiv with |rename backup files for arxiv|, |print source when saving|, |backup source for arxiv|, |forward counter force value| \item Add |\robExtMv|, |\robExtCp|, |set compilation command create if no error| and |set compilation command move if no error|. \item Add |copy file to cache| \end{itemize} \item v2.3: \begin{itemize} \item Fix bug in |more logs| and |less logs| \item Make |if matches word| more efficient, independent on the number of runs, allowing registration, namespaces, etc. \item Added |\definecolorAutoForward| and |\colorletAutoForward| \item Add support for gnuplot \item Added |\runHereAndInPreambleOfCachedFiles| \item Clearer and more advanced error handling \item Add support for online images with |\includegraphicsWeb| \end{itemize} \item v2.2 \begin{itemize} \item v2.1 introduced two important bugs where hashes could not be used in a |\cacheMe| environment, and |auto forward| was not working if the string was containing an environment. This is fixed now. \item Fix issue with |compile in parallel| on MacOs (xargs there is not GNU xargs and is missing |-a| option) \item Added |*ExpandableDocumentCommandAutoForward|. \item Added |\genericAutoForward| to forward arbitrary code. \item Added |if windows| and |if unix| tests. \item Added details on the future difference between |if matches| and |if matches word|. \end{itemize} \item v2.1: \begin{itemize} \item Fix a bug when compiling the main document with lualatex or xelatex \item Fix a bug when compiling a template that was not exporting dependencies \item Add |recompile| to recompile a file even if no dependencies changed. \item Add a hook |robust-externalize/just-before-writing-files|. \item Fixed a bug in |\cacheTikz| that was not working with a custom argument for tikzpicture \item Added options |forward*| to forward macros and counters \item Added |auto forward| and co \item Added |compile in parallel| and co \end{itemize} \item v2.0: \begin{itemize} \item \textbf{not backward compatible}: the preset |tikzpicture| should be used instead of |tikz|: |tikz| does not wrap anymore its content into |\begin{tikzpicture}| in order to be usable inside the command version |\tikz| (but |tikzpicture| does load |tikz| first). Moreover, all arguments to |tikzpictureC| or |tikzpicture| (if you loaded |cacheTikz|) should be specified in the first argument, using ||. This is used to provide a more uniform interface with the new |\cacheEnvironment| command. Hopefully this should \textbf{not cause too much trouble} since the first version of the library has been published on CTAN only a few days ago. \item not really compatible: Renamed some latex placeholders to prefix all of them with |__ROBEXT_LATEX_|. Should not be a problem if you used the style made to change them. \item Added an alias to |\robExtExternalizeAllTikzpictures| called |\cacheTikz|. Now, this also configures |\tikz|. \item You can now use ampersands etc freely thanks to lrbox. \item You can now automatically cache environments using |\cacheEnvironment| and commands using |\cacheCommand|. \item We provide |new preset| and |add to preset| to avoid doubling the number of hashes. \item We provide new functions like |\setPlaceholderRecReplaceFromList| to only expand a subset of placeholders. \item An error message is given if you forget to set a template. \item Add |enable fallback to manual mode| to fallback to manual mode only if shell escape is missing. \item Allow restricted mode \item |debug| mode available \item Rename |set subfolder and way back| to |set cache folder and way back| (in a backward compatible way) \item Waaaaaaay more efficient. Also added a method to compile presets. \item I added the import system for efficiency reasons (not present in v1.0 at all). Note that I also added some other functions that I forgot to notify as new in the doc... \end{itemize} \end{itemize} \printindex \end{document} % Luatex is needed, otherwise quotes ' are turned into another symbol in documentation % https://github.com/leo-colisson/robust-externalize/issues/11 % Local Variables: % TeX-command-extra-options: "--shell-escape -halt-on-error" % TeX-engine: luatex % jinx-languages: "en" % End: