summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
-rw-r--r--Master/texmf-dist/doc/context/third/filter/filter.txt316
-rw-r--r--Master/texmf-dist/tex/context/third/filter/t-filter.tex167
-rwxr-xr-xMaster/tlpkg/bin/tlpkg-ctan-check2
-rwxr-xr-xMaster/tlpkg/libexec/ctan2tds1
-rw-r--r--Master/tlpkg/tlpsrc/collection-context.tlpsrc1
-rw-r--r--Master/tlpkg/tlpsrc/context-filter.tlpsrc1
6 files changed, 487 insertions, 1 deletions
diff --git a/Master/texmf-dist/doc/context/third/filter/filter.txt b/Master/texmf-dist/doc/context/third/filter/filter.txt
new file mode 100644
index 00000000000..4609d606eb3
--- /dev/null
+++ b/Master/texmf-dist/doc/context/third/filter/filter.txt
@@ -0,0 +1,316 @@
+The filter module
+=================
+
+History
+-------
+
+This module started with a simple idea. I wanted an environment
+
+ \startmarkdown
+ ...
+ \stopmarkdown
+
+to write content in Markdown. Such an environment requires a parser that
+converts markdown to TeX. A TeX wizard would write such a parser in TeX (which
+after all is Turing complete). A lua wizard would write it in LPEG (and wonder
+why some users still use MkII). I, being no wizard myself, found an existing
+tool that does the job---pandoc. But how could I use pandoc inside ConTeXt?
+
+As for almost everything else in ConTeXt, Hans had already done this; albeit for
+a different tool---the R programming language. The _R
+module_ (`m-r.tex`) allows the user to execute snippets of R code. I wanted the
+same for Markdown.
+
+I copied the R-module, made a couple of changes, and had a Markdown module
+ready. But what if tomorrow I wanted to use ReStructured Text instead of
+Markdown? Or Matlab code instead R? Surely, copying the R-module for each
+program would be a waste of effort. Each new program requires only a few changes
+in the R-module; what I needed was a R-module _template_ so that I could fill in
+the blanks with the appropriate values. And so, the filter module was born.
+
+Installation
+------------
+
+Writing installation instructions is always boring. Hopefully, by the time this
+article is published, the filter module will be available from ConTeXt garden.
+If so, and if you are using ConTeXt minimals, you already have the module. To
+verify, check if
+
+ kpsewhich t-filter.tex
+
+returns a meaningful path. If not, you have to manually install the module.
+
+Create a directory `tex/context/third/filter` in your `$TEXMFHOME` or
+`$TEXMFLOCAL` directory. Copy `t-filter.tex` and `t-filter.lua` from
+this git repository
+[http://github.com/adityam/filter](http://github.com/adityam/filter) to the
+directory that you just created. Run
+
+ mktexlsr
+
+and
+
+ mtxrun --generate
+
+to refresh the TeX file database (for MkII and MkIV, respectively). If
+everything went well
+
+ kpsewhich t-filter
+
+will return the path where you stored the file.
+
+Unfortunately, that is not enough. For the module to work, TeX must be able to
+call an external program. This feature is a potential security risk and is
+disabled by default on most TeX distributions. To enable this feature, you must
+set
+
+ shell_escape=t
+
+in your `texmf.cnf` file. See this page
+[http://wiki.contextgarden.net/write18](http://wiki.contextgarden.net/write18)
+on the ConTeXt wiki for detailed instructions.
+
+Basic Usage
+-----------
+
+The steps involved in calling a filter on the contents of an evironment are:
+
+1. Write the contents to an external file. This file is the input to the filter,
+ and is, therefore, called `\externalfilterinputfile`
+
+2. Run the filter on `\externalfilterinputfile` to generate an output, which is
+ called `\externalfilteroutputfile`.
+
+3. (Optional) Read `\externalfilteroutputfile` in ConTeXt.
+
+Lets start from the simplest case: a filter that inputs a text file and outputs
+a valid ConTeXt file, like `pandoc` to convert from Markdown to ConTeXt. The
+command line syntax of this filter is
+
+ pandoc -t context -o outputfile inputfile
+
+Using this filter from within ConTeXt is pretty simple:
+
+ \usemodule[filter]
+
+ \defineexternalfilter
+ [markdown]
+ [filtercommand={pandoc -t context -o \externalfilteroutputfile\space \externalfilterinputfile}]
+
+Yes, its that easy! The only thing to note is that TeX macros gobble spaces, so
+we have to manually insert a space after `\externalfilteroutputfile`.
+
+This defines an environment
+
+ \startmarkdown
+ ...
+ \stopmarkdown
+
+and a macro
+
+ \processmarkdownfile[...]
+
+The contents of the environment are processed by `pandoc` and the output is
+included back in ConTeXt.
+
+The argument to the macro is a filename, which is processed by `pandoc` and the
+output is included back in ConTeXt.
+
+Dealing with slow filters
+-------------------------
+
+The above definition of a markdown filter creates two additional files: an
+"input" file and an "output" file, *irrespective of the
+number of times the environment is called*. For each markdown environment,
+ConTeXt overwrites the input file and pandoc overwrites the output file. This
+behavior is the default as I do not want to clutter the current directory with
+temporary files. The trade off is that for each document run, the filter is
+invoked as many times as the number of markdown environments. Since getting
+cross-referencing right normally takes two or three runs, effectively the filter
+is run two or three times more than required. A filter like `pandoc` is fairly
+fast, so these extra runs are not noticeable. But some filters, like the
+R-programming language for which simply startup and exit takes about 0.3
+seconds, are slow. In such cases, the additional runs start adding up. A better
+trade off is to store the contents of each environment in a separate file and
+invoke the filter only if a file *changes in between successive runes*.
+
+The second behavior is achieved by adding `continue=yes` option to the
+definition:
+
+ \defineexternalfilter
+ [...]
+ [...
+ continue=yes,
+ ...]
+
+Reading the input
+----------------
+
+By default, after the filter is executed, `\externalfilteroutputfile` is read
+using `\ReadFile`. To change this behavior, use the `readcommand` option. For
+example:
+
+ \defineexternalfilter
+ [...]
+ [....
+ readcommand=\typefile,
+ ...]
+
+types the output file verbatim. The value of read command must be a macro that
+takes the name of the output file as a (brace-delimited) argument and does
+something sensible with it.
+
+Sometimes, it is desirable to ignore the output, which is done by
+
+ \defineexternalfilter
+ [...]
+ [....
+ read=no,
+ ...]
+
+
+Names of temporary files
+------------------------
+
+By default, `\externalfilterinputfile` is set to `\jobname-<filter>.tmp`, where
+`<filter>` is the first argument of `\defineexternalfilter`. When `continue=yes`
+is set, `\externalfilterinputfile` equals `\jobname-<filter>-<n>.tmp`, where
+`<n>` is the number of filter environments that have appeared so far. In this
+case, a `\jobname-<filter>-<n>.tmp.md5` file, which stores the `md5` sum of the
+input file` is also created.
+
+A macro `\externalfilterbasefile` stores the name of the input file without the
+extension. By default, the value of `\externalfilteroutputfile` is
+`\externalfilterbasefile.tex`. Having a `.tex` extension is not always
+desirable. For example, if the filter generates a PNG file, a `.png` extension
+is more descriptive. The name of the output file is changed using the `output`
+key. For example
+
+ \defineexternalfilter
+ [...]
+ [filtercommand={...},
+ output={\externalfilterbasefile.png}]
+
+changes the output extension to `.png`. We also need to either set
+
+ \defineexternalfilter
+ [...]
+ [....
+ read=no,
+ ...]
+
+or set
+
+ \defineexternalfilter
+ [...]
+ [....
+ readcommand=\readPNGfile,
+ ...]
+
+where `\readPNGfile` is defined as
+
+ \def\readPNGfile#1{\externalfigure[#1]}
+
+
+Standard options
+---------------
+
+Like most commands in ConTeXt, `\defineexternalfilter` also accepts the `before`
+and `after` options. These are executed before and after the output file is read
+using `readcommand`. Typically, these options are used to set the spacing around
+the environment or enclose the output in a frame, etc.
+
+`\defineexternalfilter` also accepts a `setup` option to specify a list of
+setups (defined using `\startsetup`). These setups may be used to define
+commands that are needed inside the environment.
+
+The order in which these options are executed is:
+
+ \def\dodoreadprocessedfile
+ {\bgroup
+ \externalfilterparameter\c!before
+ \processcommacommand[\externalfilterparameter\c!setups]\directsetup
+ \externalfilterparameter\c!readcommand\externalfilteroutputfile
+ \externalfilterparameter\c!after
+ \egroup}
+
+Options to a specific environment
+---------------------------------
+
+Each `\start<filter>` macro also accepts options. However, unlike other ConTeXt
+environment, these options cannot be on a separate line; they must be on the same
+line as `\start<filter>`. For example, suppose I define an environment to run
+R-code
+
+ \defineexternalfilter
+ [R]
+ [filtercommand={R CMD BATCH -q \externalfilterinputfile\space \externalfilteroutputfile},
+ output=\externalfilterbasefile.out,
+ continue=yes]
+
+I can hide the output of a particular R-environment by
+
+ \startR[read=no]
+ ...
+ \stopR
+
+
+A setup to control them all
+---------------------------
+
+The macro `\setupexternalfilters` sets the default options for all the filters
+created using `\defineexternalfilter`. This is responsible for the default values
+of all options. The current defaults are
+
+ \setupexternalfilters
+ [before=,
+ after=,
+ setups=,
+ continue=no,
+ read=yes,
+ readcommand=\ReadFile,
+ output=\externalfilterbasefile.tex,
+ ]
+
+
+Passing options to filters
+--------------------------
+
+Sometimes it is useful to pass options to a filter. For example, `pandoc`
+converts many different formats to ConTeXt (actually, to many different output
+formats, but that is irrelevant here). Instead of defining a separate
+environment for each input format, can I define a universal pandoc environment
+and specify the input format on a case by case basis. For example,
+
+ \startpandoc
+ ...
+ \stoppandoc
+
+for the default Markdown input,
+
+ \startpandoc[format=rst]
+ ...
+ \stoppandoc
+
+for reStructured Text input, and
+
+ \startpandoc[format=latex]
+ ...
+ \stoppandoc
+
+for LaTeX input. In `pandoc`, the input format is specified as
+
+ pandoc -f format -t context -o outputfile inputfile
+
+So, we need a mechanism to access the value of the `format` option to
+`\startpandoc`. This value is accessed using `\externalfilterparameter{format}`.
+Thus, the pandoc environment may be defined as
+
+ \defineexternalfilter
+ [pandoc]
+ [filtercommand={pandoc -f \externalfilterparameter{format} -t context
+ -o \externalfilteroutputfile\space \externalfilterinputfile},
+ format=markdown]
+
+
+
diff --git a/Master/texmf-dist/tex/context/third/filter/t-filter.tex b/Master/texmf-dist/tex/context/third/filter/t-filter.tex
new file mode 100644
index 00000000000..9ae777aa3b7
--- /dev/null
+++ b/Master/texmf-dist/tex/context/third/filter/t-filter.tex
@@ -0,0 +1,167 @@
+%D \module
+%D [ file=t-filter,
+%D version=2010.09.26,
+%D title=\CONTEXT\ User Module,
+%D subtitle=Filter,
+%D author=Aditya Mahajan,
+%D date=\currentdate,
+%D copyright=Aditya Mahajan,
+%D email=adityam <at> umich <dot> edu,
+%D license=Simplified BSD License]
+
+\writestatus{loading}{ConTeXt User Module / Filter}
+
+\startmodule [filter]
+
+\unprotect
+
+\startinterface all
+ \setinterfaceconstant {filter} {filter}
+ \setinterfaceconstant {filtercommand} {filtercommand}
+ \setinterfaceconstant {output} {output}
+ \setinterfaceconstant {read} {read}
+ \setinterfaceconstant {readcommand} {readcommand}
+\stopinterface
+
+\def\??externalfilter??{externalfilter}
+\def\currentexternalfilter{}
+\def\externalfiltercountername{\??externalfilter??-\currentexternalfilter-counter}
+
+\def\externalfilterparameter#1%
+ {\csname
+ \docheckparentparameter{\??externalfilter??\currentexternalfilter}{#1}%
+ \endcsname}
+
+\def\docheckparentparameter#1#2%
+ {\ifcsname#1#2\endcsname
+ #1#2%
+ \else
+ \expandafter\redocheckparentparameter\csname#1\s!parent\endcsname{#2}%
+ \fi}
+
+\def\redocheckparentparameter#1#2%
+ {\ifx#1\relax
+ \s!empty
+ \else
+ \docheckparentparameter{#1}{#2}%
+ \fi}
+
+\def\setupexternalfilters
+ {\dodoubleargument\dosetupexternalfilters}
+
+\def\dosetupexternalfilters[#1][#2]%
+ {\ifsecondargument
+ \getparameters[\??externalfilter??#1][#2]%
+ \else
+ \getparameters[\??externalfilter??][#1]%
+ \fi}
+
+
+\def\defineexternalfilter
+ {\dodoubleargument\dodefineexternalfilter}
+
+\def\dodefineexternalfilter[#1][#2]%
+ {\edef\currentexternalfilter{#1}%
+ \getparameters[\??externalfilter??#1][\s!parent=\??externalfilter??,#2]%
+ \doif{\externalfilterparameter\c!continue}\v!yes
+ {\expandafter\newcounter\csname\externalfiltercountername\endcsname}%
+ \setvalue{\e!start#1}{\bgroup\obeylines\dodoubleargument\dostartexternalfilter[#1]}%
+ \setvalue{\e!stop#1}{\doprocessexternalfilter}%
+ \setvalue{process#1file}{\dodoubleargument\doprocessexternalfile[#1]}%
+ %TODO: Inline.
+ %\setvalue{#1}##1{\save ##1 to file}
+ }
+
+
+\def\dostartexternalfilter[#1][#2]% filter options
+ {% Initializations
+ \egroup %\bgroup in \start#1
+ \begingroup % to keep assignments local
+ \edef\currentexternalfilter {#1}%
+ \getparameters[\??externalfilter??#1][\c!name=,#2]%
+ % Set the name of temp file for os filter
+ \doifelse{\externalfilterparameter\c!continue}\v!yes
+ {\edef\externalfiltertmpfile{\??externalfilter??-#1-\csname\externalfiltercountername\endcsname}}
+ {\edef\externalfiltertmpfile{\??externalfilter??-#1}}
+ \doifsomething{\externalfilterparameter\c!name}
+ {\edef\externalfiltertmpfile{\externalfilterparameter\c!name}}
+ % The following macros are useful for filter= and filtercommand= options
+ % The basename of the external file
+ \edef\externalfilterbasefile {\jobname-\externalfiltertmpfile}%
+ % The name of the buffer to which data is written
+ \edef\externalfilterinputfile {\externalfilterbasefile.tmp}%
+ % The name of the file to which the output is written
+ \edef\externalfilteroutputfile{\externalfilterparameter\c!output}%
+ % Capture the contents of the buffer
+ \dostartbuffer[\externalfiltertmpfile][\e!start#1][\e!stop#1]}
+
+\def\doprocessexternalfilter%
+ {% By defualt, buffers are in memory in MkIV
+ \doifmode\s!mkiv{\savebuffer[\externalfiltertmpfile]}%
+ % Run external command
+ \doexecuteexternalfilter
+ \doreadprocessedfile
+ \endgroup
+ % Finalization
+ \doif{\externalfilterparameter\c!continue}\v!yes
+ {\expandafter\increment\csname\externalfiltercountername\endcsname}%
+ }
+
+\def\doexecuteexternalfilter
+ {\doifelse{\externalfilterparameter\c!continue}\v!yes
+ {\doifmode{*first}
+ {\executesystemcommand
+ {mtxrun --ifchanged=\externalfilterinputfile\space
+ bin:\externalfilterparameter\c!filtercommand}}}
+ {\executesystemcommand
+ {\externalfilterparameter\c!filtercommand}}}
+
+\def\doreadprocessedfile
+ {\doif{\externalfilterparameter\c!read}\v!yes
+ {\doiffileelse{\externalfilteroutputfile}
+ {\dodoreadprocessedfile}
+ {{\tttf TODO: File \externalfilteroutputfile\ not found! Check your definition}}}}
+
+\def\dodoreadprocessedfile
+ {\begingroup
+ \externalfilterparameter\c!before
+ \processcommacommand[\externalfilterparameter\c!setups]\directsetup
+ \externalfilterparameter\c!readcommand\externalfilteroutputfile
+ \externalfilterparameter\c!after
+ \endgroup}
+
+\def\doprocessexternalfile[#1][#2]%
+ {\edef\currentexternalfilter {#1}%
+ \edef\externalfilterinputfile {#2}%
+ \splitfiletype {#2}%
+ %BEWARE. \edef doesn not work
+ \def\externalfilterbasefile {\splitoffname}
+ \def\externalfilteroutputfile{\externalfilterparameter\c!output}%
+ \doexecuteexternalfilter
+ \doreadprocessedfile}
+
+
+\setupexternalfilters
+ [
+ \c!before=,
+ \c!after=,
+ \c!setups=,
+ \c!continue=\v!no,
+ \c!read=\v!yes,
+ \c!readcommand=\ReadFile,
+ \c!output=\externalfilterbasefile.tex,
+ \c!filter=,
+ \c!filtercommand={\externalfilterparameter\c!filter\space \externalfilterinputfile},
+ ]
+
+% For tracing
+\def\showexternalfilterstate
+ {\starttable[|l|l|]
+ \NC currentexternalfilter \NC \currentexternalfilter \NC \AR
+ \NC externalfilterinputfile \NC \externalfilterinputfile \NC \AR
+ \NC externalfilteroutputfile \NC \externalfilteroutputfile \NC \AR
+ \NC externalfilterbasefile \NC \externalfilterbasefile \NC \AR
+ \stoptable}
+
+\protect
+\stopmodule
diff --git a/Master/tlpkg/bin/tlpkg-ctan-check b/Master/tlpkg/bin/tlpkg-ctan-check
index 1a715fbc6af..dc432f96c6a 100755
--- a/Master/tlpkg/bin/tlpkg-ctan-check
+++ b/Master/tlpkg/bin/tlpkg-ctan-check
@@ -83,7 +83,7 @@ my @TLP_working = qw(
complexity comprehensive computational-complexity
concmath concmath-fonts concprog confproc constants
context-account context-algorithmic context-bnf context-chromato
- context-construction-plan context-degrade context-fixme
+ context-construction-plan context-degrade context-filter context-fixme
context-french context-fullpage context-games context-gnuplot
context-letter context-lettrine context-lilypond context-mathsets
context-notes-zh-cn context-ruby
diff --git a/Master/tlpkg/libexec/ctan2tds b/Master/tlpkg/libexec/ctan2tds
index d446a351a5f..a80bd4c84ea 100755
--- a/Master/tlpkg/libexec/ctan2tds
+++ b/Master/tlpkg/libexec/ctan2tds
@@ -185,6 +185,7 @@ $Master = "$mydir/../..";
'context-chromato', "&MAKEcopy",
'context-construction-plan', "&MAKEcopy",
'context-degrade', "&MAKEcopy",
+ 'context-filter', "&MAKEcopy",
'context-fixme', "&MAKEcopy",
'context-french', "&MAKEcopy",
'context-fullpage', "&MAKEcopy",
diff --git a/Master/tlpkg/tlpsrc/collection-context.tlpsrc b/Master/tlpkg/tlpsrc/collection-context.tlpsrc
index 1f07a8bc0b6..aa8e2d22540 100644
--- a/Master/tlpkg/tlpsrc/collection-context.tlpsrc
+++ b/Master/tlpkg/tlpsrc/collection-context.tlpsrc
@@ -11,6 +11,7 @@ depend context-bnf
depend context-chromato
depend context-construction-plan
depend context-degrade
+depend context-filter
depend context-fixme
depend context-french
depend context-fullpage
diff --git a/Master/tlpkg/tlpsrc/context-filter.tlpsrc b/Master/tlpkg/tlpsrc/context-filter.tlpsrc
new file mode 100644
index 00000000000..8e194d47208
--- /dev/null
+++ b/Master/tlpkg/tlpsrc/context-filter.tlpsrc
@@ -0,0 +1 @@
+category ConTeXt