new context pkg context-filter (28sep10)

git-svn-id: svn://tug.org/texlive/trunk@19939 c570f23f-e606-0410-a88d-b1316a301751

2 files changed, 483 insertions, 0 deletions

diff --git a/Master/texmf-dist/doc/context/third/filter/filter.txt b/Master/texmf-dist/doc/context/third/filter/filter.txt
new file mode 100644
index 00000000000..4609d606eb3
--- /dev/null
+++ b/Master/texmf-dist/doc/context/third/filter/filter.txt
@@ -0,0 +1,316 @@
+The filter module
+=================
+
+History
+-------
+
+This module started with a simple idea. I wanted an environment
+    
+    \startmarkdown
+    ...
+    \stopmarkdown
+
+to write content in Markdown. Such an environment requires a parser that
+converts markdown to TeX. A TeX wizard would write such a parser in TeX (which
+after all is Turing complete). A lua wizard would write it in LPEG (and wonder
+why some users still use MkII). I, being no wizard myself, found an existing
+tool that does the job---pandoc. But how could I use pandoc inside ConTeXt?
+
+As for almost everything else in ConTeXt, Hans had already done this; albeit for
+a different tool---the R programming language.  The _R
+module_ (`m-r.tex`) allows the user to execute snippets of R code. I wanted the
+same for Markdown.
+
+I copied the R-module, made a couple of changes, and had a Markdown module
+ready. But what if tomorrow I wanted to use ReStructured Text instead of
+Markdown? Or Matlab code instead R? Surely, copying the R-module for each
+program would be a waste of effort. Each new program requires only a few changes
+in the R-module; what I needed was a R-module _template_ so that I could fill in
+the blanks with the appropriate values. And so, the filter module was born.
+
+Installation
+------------
+
+Writing installation instructions is always boring. Hopefully, by the time this
+article is published, the filter module will be available from ConTeXt garden.
+If so, and if you are using ConTeXt minimals, you already have the module. To
+verify, check if
+
+    kpsewhich t-filter.tex
+
+returns a meaningful path. If not, you have to manually install the module.
+
+Create a directory `tex/context/third/filter` in your `$TEXMFHOME` or
+`$TEXMFLOCAL` directory. Copy `t-filter.tex` and `t-filter.lua` from 
+this git repository
+[http://github.com/adityam/filter](http://github.com/adityam/filter) to the
+directory that you just created. Run
+
+    mktexlsr
+
+and
+
+    mtxrun --generate
+
+to refresh the TeX file database (for MkII and MkIV, respectively). If
+everything went well
+
+    kpsewhich t-filter
+
+will return the path where you stored the file.
+
+Unfortunately, that is not enough. For the module to work, TeX must be able to
+call an external program. This feature is a potential security risk and is
+disabled by default on most TeX distributions. To enable this feature, you must
+set
+
+    shell_escape=t
+
+in your `texmf.cnf` file. See this page
+[http://wiki.contextgarden.net/write18](http://wiki.contextgarden.net/write18)
+on the ConTeXt wiki for detailed instructions.
+
+Basic Usage
+-----------
+
+The steps involved in calling a filter on the contents of an evironment are:
+
+1. Write the contents to an external file. This file is the input to the filter,
+   and is, therefore, called `\externalfilterinputfile`
+
+2. Run the filter on `\externalfilterinputfile` to generate an output, which is
+   called `\externalfilteroutputfile`.
+
+3. (Optional) Read `\externalfilteroutputfile` in ConTeXt.
+
+Lets start from the simplest case: a filter that inputs a text file and outputs
+a valid ConTeXt file, like `pandoc` to convert from Markdown to ConTeXt. The
+command line syntax of this filter is
+
+    pandoc -t context -o outputfile inputfile
+
+Using this filter from within ConTeXt is pretty simple:
+
+    \usemodule[filter]
+
+    \defineexternalfilter
+        [markdown]
+        [filtercommand={pandoc -t context -o \externalfilteroutputfile\space \externalfilterinputfile}]
+
+Yes, its that easy! The only thing to note is that TeX macros gobble spaces, so
+we have to manually insert a space after `\externalfilteroutputfile`.
+
+This defines an environment
+
+    \startmarkdown
+      ...
+    \stopmarkdown
+
+and a macro
+
+    \processmarkdownfile[...]
+
+The contents of the environment are processed by `pandoc` and the output is
+included back in ConTeXt.
+
+The argument to the macro is a filename, which is processed by `pandoc` and the
+output is included back in ConTeXt.
+
+Dealing with slow filters
+-------------------------
+
+The above definition of a markdown filter creates two additional files: an
+"input" file and an "output" file, *irrespective of the
+number of times the environment is called*. For each markdown environment,
+ConTeXt overwrites the input file and pandoc overwrites the output file. This
+behavior is the default as I do not want to clutter the current directory with
+temporary files. The trade off is that for each document run, the filter is
+invoked as many times as the number of markdown environments. Since getting
+cross-referencing right normally takes two or three runs, effectively the filter
+is run two or three times more than required. A filter like `pandoc` is fairly
+fast, so these extra runs are not noticeable. But some filters, like the
+R-programming language for which simply startup and exit takes about 0.3
+seconds, are slow. In such cases, the additional runs start adding up. A better
+trade off is to store the contents of each environment in a separate file and
+invoke the filter only if a file *changes in between successive runes*.
+
+The second behavior is achieved by adding `continue=yes` option to the
+definition:
+
+    \defineexternalfilter
+        [...]
+        [...
+         continue=yes,
+         ...]
+
+Reading the input
+----------------
+
+By default, after the filter is executed, `\externalfilteroutputfile` is read
+using `\ReadFile`. To change this behavior, use the `readcommand` option. For
+example:
+
+    \defineexternalfilter
+      [...]
+      [....
+       readcommand=\typefile,
+       ...]
+
+types the output file verbatim. The value of read command must be a macro that
+takes the name of the output file as a (brace-delimited) argument and does
+something sensible with it. 
+
+Sometimes, it is desirable to ignore the output, which is done by
+
+    \defineexternalfilter
+      [...]
+      [....
+       read=no,
+       ...]
+
+
+Names of temporary files
+------------------------
+
+By default, `\externalfilterinputfile` is set to `\jobname-<filter>.tmp`, where
+`<filter>` is the first argument of `\defineexternalfilter`. When `continue=yes`
+is set, `\externalfilterinputfile` equals `\jobname-<filter>-<n>.tmp`, where
+`<n>` is the number of filter environments that have appeared so far. In this
+case,  a `\jobname-<filter>-<n>.tmp.md5` file, which stores the `md5` sum of the
+input file` is also created.
+
+A macro `\externalfilterbasefile` stores the name of the input file without the
+extension. By default, the value of `\externalfilteroutputfile` is
+`\externalfilterbasefile.tex`. Having a `.tex` extension is not always
+desirable. For example, if the filter generates a PNG file, a `.png` extension
+is more descriptive. The name of the output file is changed using the `output`
+key. For example
+
+    \defineexternalfilter
+        [...]
+        [filtercommand={...},
+         output={\externalfilterbasefile.png}]
+
+changes the output extension to `.png`. We also need to either set
+
+    \defineexternalfilter
+      [...]
+      [....
+       read=no,
+       ...]
+
+or set 
+
+    \defineexternalfilter
+      [...]
+      [....
+       readcommand=\readPNGfile,
+       ...]
+
+where `\readPNGfile` is defined as
+
+    \def\readPNGfile#1{\externalfigure[#1]}
+
+
+Standard options
+---------------
+
+Like most commands in ConTeXt, `\defineexternalfilter` also accepts the `before`
+and `after` options. These are executed before and after the output file is read
+using `readcommand`. Typically, these options are used to set the spacing around
+the environment or enclose the output in a frame, etc.
+
+`\defineexternalfilter` also accepts a `setup` option to specify a list of 
+setups (defined using `\startsetup`). These setups may be used to define
+commands that are needed inside the environment.
+
+The order in which these options are executed is:
+
+    \def\dodoreadprocessedfile
+      {\bgroup
+       \externalfilterparameter\c!before
+       \processcommacommand[\externalfilterparameter\c!setups]\directsetup
+       \externalfilterparameter\c!readcommand\externalfilteroutputfile
+       \externalfilterparameter\c!after
+       \egroup}
+
+Options to a specific environment
+---------------------------------
+
+Each `\start<filter>` macro also accepts options. However, unlike other ConTeXt
+environment, these options cannot be on a separate line; they must be on the same
+line as `\start<filter>`. For example, suppose I define an environment to run
+R-code
+
+    \defineexternalfilter
+      [R]
+      [filtercommand={R CMD BATCH -q  \externalfilterinputfile\space \externalfilteroutputfile},
+       output=\externalfilterbasefile.out,
+       continue=yes]
+
+I can hide the output of a particular R-environment by
+
+    \startR[read=no]
+    ...
+    \stopR
+
+
+A setup to control them all
+---------------------------
+
+The macro `\setupexternalfilters` sets the default options for all the filters
+created using `\defineexternalfilter`. This is responsible for the default values
+of all options. The current defaults are
+
+    \setupexternalfilters
+      [before=,
+       after=,
+       setups=,
+       continue=no,
+       read=yes,
+       readcommand=\ReadFile,
+       output=\externalfilterbasefile.tex,
+      ]
+
+
+Passing options to filters
+--------------------------
+
+Sometimes it is useful to pass options to a filter. For example, `pandoc`
+converts many different formats to ConTeXt (actually, to many different output
+formats, but that is irrelevant here). Instead of defining a separate
+environment for each input format, can I define a universal pandoc environment
+and specify the input format on a case by case basis. For example,
+
+    \startpandoc
+    ...
+    \stoppandoc
+
+for the default Markdown input,
+
+    \startpandoc[format=rst]
+    ...
+    \stoppandoc
+
+for reStructured Text input, and
+
+    \startpandoc[format=latex]
+    ...
+    \stoppandoc
+
+for LaTeX input. In `pandoc`, the input format is specified as
+
+    pandoc -f format -t context -o outputfile inputfile
+
+So, we need a mechanism to access the value of the `format` option to
+`\startpandoc`. This value is accessed using `\externalfilterparameter{format}`.
+Thus, the pandoc environment may be defined as
+
+    \defineexternalfilter
+      [pandoc]
+      [filtercommand={pandoc -f \externalfilterparameter{format} -t context 
+                       -o \externalfilteroutputfile\space \externalfilterinputfile},
+       format=markdown]
+
+
+
diff --git a/Master/texmf-dist/tex/context/third/filter/t-filter.tex b/Master/texmf-dist/tex/context/third/filter/t-filter.tex
new file mode 100644
index 00000000000..9ae777aa3b7
--- /dev/null
+++ b/Master/texmf-dist/tex/context/third/filter/t-filter.tex
@@ -0,0 +1,167 @@
+%D \module
+%D   [     file=t-filter,
+%D      version=2010.09.26,
+%D        title=\CONTEXT\ User Module,
+%D     subtitle=Filter,
+%D       author=Aditya Mahajan,
+%D         date=\currentdate,
+%D    copyright=Aditya Mahajan,
+%D        email=adityam <at> umich <dot> edu,
+%D      license=Simplified BSD License]
+
+\writestatus{loading}{ConTeXt User Module / Filter}
+
+\startmodule    [filter]
+
+\unprotect
+
+\startinterface all
+  \setinterfaceconstant {filter}           {filter}
+  \setinterfaceconstant {filtercommand}    {filtercommand}
+  \setinterfaceconstant {output}           {output} 
+  \setinterfaceconstant {read}             {read} 
+  \setinterfaceconstant {readcommand}      {readcommand} 
+\stopinterface
+
+\def\??externalfilter??{externalfilter}
+\def\currentexternalfilter{}
+\def\externalfiltercountername{\??externalfilter??-\currentexternalfilter-counter}
+
+\def\externalfilterparameter#1%
+  {\csname
+     \docheckparentparameter{\??externalfilter??\currentexternalfilter}{#1}%
+   \endcsname}
+
+\def\docheckparentparameter#1#2%
+  {\ifcsname#1#2\endcsname
+      #1#2%
+    \else
+      \expandafter\redocheckparentparameter\csname#1\s!parent\endcsname{#2}%
+    \fi}
+
+\def\redocheckparentparameter#1#2%
+  {\ifx#1\relax
+      \s!empty
+   \else
+      \docheckparentparameter{#1}{#2}%
+   \fi}
+
+\def\setupexternalfilters
+  {\dodoubleargument\dosetupexternalfilters}
+  
+\def\dosetupexternalfilters[#1][#2]%
+  {\ifsecondargument
+     \getparameters[\??externalfilter??#1][#2]%
+   \else
+     \getparameters[\??externalfilter??][#1]%
+   \fi}
+
+
+\def\defineexternalfilter
+  {\dodoubleargument\dodefineexternalfilter}
+
+\def\dodefineexternalfilter[#1][#2]%
+  {\edef\currentexternalfilter{#1}%
+  \getparameters[\??externalfilter??#1][\s!parent=\??externalfilter??,#2]%
+   \doif{\externalfilterparameter\c!continue}\v!yes
+      {\expandafter\newcounter\csname\externalfiltercountername\endcsname}%
+   \setvalue{\e!start#1}{\bgroup\obeylines\dodoubleargument\dostartexternalfilter[#1]}%
+   \setvalue{\e!stop#1}{\doprocessexternalfilter}%
+   \setvalue{process#1file}{\dodoubleargument\doprocessexternalfile[#1]}%
+   %TODO: Inline.
+   %\setvalue{#1}##1{\save ##1 to file}
+   }
+
+
+\def\dostartexternalfilter[#1][#2]% filter options
+  {% Initializations
+   \egroup %\bgroup in \start#1
+   \begingroup % to keep assignments local
+   \edef\currentexternalfilter   {#1}%
+   \getparameters[\??externalfilter??#1][\c!name=,#2]%
+   % Set the name of temp file for os filter
+   \doifelse{\externalfilterparameter\c!continue}\v!yes
+       {\edef\externalfiltertmpfile{\??externalfilter??-#1-\csname\externalfiltercountername\endcsname}}
+       {\edef\externalfiltertmpfile{\??externalfilter??-#1}}
+   \doifsomething{\externalfilterparameter\c!name}
+       {\edef\externalfiltertmpfile{\externalfilterparameter\c!name}}
+   % The following  macros are useful for filter= and filtercommand= options
+   % The basename of the external file 
+   \edef\externalfilterbasefile  {\jobname-\externalfiltertmpfile}%
+   % The name of the buffer to which data is written
+   \edef\externalfilterinputfile {\externalfilterbasefile.tmp}%
+   % The name of the file to which the output is written
+   \edef\externalfilteroutputfile{\externalfilterparameter\c!output}%
+   % Capture the contents of the buffer
+   \dostartbuffer[\externalfiltertmpfile][\e!start#1][\e!stop#1]}
+
+\def\doprocessexternalfilter%
+  {% By defualt, buffers are in memory in MkIV
+   \doifmode\s!mkiv{\savebuffer[\externalfiltertmpfile]}%
+   % Run external command
+   \doexecuteexternalfilter
+   \doreadprocessedfile
+   \endgroup 
+   % Finalization
+   \doif{\externalfilterparameter\c!continue}\v!yes
+        {\expandafter\increment\csname\externalfiltercountername\endcsname}%
+   }
+
+\def\doexecuteexternalfilter
+  {\doifelse{\externalfilterparameter\c!continue}\v!yes
+   {\doifmode{*first}
+     {\executesystemcommand
+      {mtxrun --ifchanged=\externalfilterinputfile\space 
+          bin:\externalfilterparameter\c!filtercommand}}}
+   {\executesystemcommand
+      {\externalfilterparameter\c!filtercommand}}}
+
+\def\doreadprocessedfile
+  {\doif{\externalfilterparameter\c!read}\v!yes
+      {\doiffileelse{\externalfilteroutputfile}
+          {\dodoreadprocessedfile}
+          {{\tttf TODO: File \externalfilteroutputfile\ not found! Check your definition}}}}
+
+\def\dodoreadprocessedfile
+  {\begingroup
+   \externalfilterparameter\c!before
+   \processcommacommand[\externalfilterparameter\c!setups]\directsetup
+   \externalfilterparameter\c!readcommand\externalfilteroutputfile
+   \externalfilterparameter\c!after
+   \endgroup}
+
+\def\doprocessexternalfile[#1][#2]%
+  {\edef\currentexternalfilter    {#1}%
+   \edef\externalfilterinputfile  {#2}%
+   \splitfiletype {#2}%
+   %BEWARE. \edef doesn not work
+   \def\externalfilterbasefile   {\splitoffname}
+   \def\externalfilteroutputfile{\externalfilterparameter\c!output}%
+   \doexecuteexternalfilter
+   \doreadprocessedfile}
+   
+
+\setupexternalfilters
+  [
+   \c!before=,
+   \c!after=,
+   \c!setups=,
+   \c!continue=\v!no,
+   \c!read=\v!yes,
+   \c!readcommand=\ReadFile,
+   \c!output=\externalfilterbasefile.tex,
+   \c!filter=,
+   \c!filtercommand={\externalfilterparameter\c!filter\space \externalfilterinputfile},
+ ]
+   
+% For tracing
+\def\showexternalfilterstate
+  {\starttable[|l|l|]
+  \NC currentexternalfilter     \NC \currentexternalfilter      \NC \AR
+  \NC externalfilterinputfile   \NC \externalfilterinputfile    \NC \AR
+  \NC externalfilteroutputfile  \NC \externalfilteroutputfile   \NC \AR
+  \NC externalfilterbasefile    \NC \externalfilterbasefile     \NC \AR
+  \stoptable}
+   
+\protect
+\stopmodule