diff options
author | Norbert Preining <norbert@preining.info> | 2019-09-02 13:46:59 +0900 |
---|---|---|
committer | Norbert Preining <norbert@preining.info> | 2019-09-02 13:46:59 +0900 |
commit | e0c6872cf40896c7be36b11dcc744620f10adf1d (patch) | |
tree | 60335e10d2f4354b0674ec22d7b53f0f8abee672 /macros/luatex/generic/lualibs |
Initial commit
Diffstat (limited to 'macros/luatex/generic/lualibs')
38 files changed, 25981 insertions, 0 deletions
diff --git a/macros/luatex/generic/lualibs/LICENSE b/macros/luatex/generic/lualibs/LICENSE new file mode 100644 index 0000000000..d159169d10 --- /dev/null +++ b/macros/luatex/generic/lualibs/LICENSE @@ -0,0 +1,339 @@ + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Lesser General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) <year> <name of author> + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + <signature of Ty Coon>, 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. diff --git a/macros/luatex/generic/lualibs/NEWS b/macros/luatex/generic/lualibs/NEWS new file mode 100644 index 0000000000..66ee05e8cb --- /dev/null +++ b/macros/luatex/generic/lualibs/NEWS @@ -0,0 +1,88 @@ + History of the lualibs package +2019/08/11 v2.67/ + * sync with Context current as of 2019/08/11 + +2019/07/04 v2.66/ + * sync with Context current as of 2019/07/04 + * changed kpse initialization to avoid conflicts with latex-dev format + +2019/05/18 v2.65/ + * sync with Context current as of 2019/05/18 + +2019/02/14 v2.64/ + * sync with Context beta as of 2019/02/14 + +2019/01/28 v2.63/ + * sync with Context beta as of 2019/01/28 + +2019/01/28 v2.62/ + * sync with Context beta as of 2018/12/19 + +2018/10/18 v2.61/ + * sync with Context beta as of 2018-10-18 + +2018/09/20 v2.6/ + * sync with Context beta as of 2018-09-20 + + +2017/02/01 v2.5/ + * sync with Context beta as of 2017-02-01 + +2016/04/06 v2.4/ + * sync with Context beta as of 2016-04-06 + * basic maintenance + * include util-fil.lua, a fontloader requirement, in the extended + collection + +2015/12/02 v2.3/ + * sync with Context beta as of 2015-11-19 + * basic maintenance + +2014/05/05 v2.2/ + * sync with Context beta as of 2014-07-26 + * remove util-env.lua + +2013/31/03 v2.1a/ + * sync with Context beta as of 2013-12-30 + +2013/11/03 v2.1/ + * sync with Context beta as of 2013-11-01 + * add l-gzip + +2013/07/23 v2.0c/ + * sync with Context beta as of 2013-07-14 + +2013/05/18 v2.0/ + * sync with Context beta as of 2013.04.29 20:30 + * merge with mtx-package + * split in basic and extended collection, allowing for partial + loading + * include additional files from Context + * drop module() in favor of locals + +2012/10/19 v0.9/ + * sync with ConTeXt beta 2012.10.17 + * move some files to util-* prefix + * add util-sto util-lua util-sto util-jsn + +2011/01/20 v0.96 + * Fix computability with lfs in luatex 0.65 + +2010/06/28 v0.95 + * don't use kpse.set_program_name(), people loading lualibs with texlua + should set it themselves, to avoid possible clashes + * drop deprecated functions and aliases + * update the documentation + +2010/05/28 v0.94 + * sync with ConTeXt beta 2010.05.20, adding new functions used + by luaotfload + +2010/05/10 v0.93 + * renamed the package from luaextra to lualibs + * now a separate package (previously part of the luatextra bundle) + +2010/01/10 v0.92 + * making the file loadable without luatextra (useful for lua scripts) + * fixing typo in localize_sep + diff --git a/macros/luatex/generic/lualibs/README.md b/macros/luatex/generic/lualibs/README.md new file mode 100644 index 0000000000..b89ca40fb7 --- /dev/null +++ b/macros/luatex/generic/lualibs/README.md @@ -0,0 +1,99 @@ +# The Lualibs Package + +VERSION: 2.67 + +DATE: 2019-08-11 + +Lualibs is a collection of Lua modules useful for general programming. + +This work is based on Lua modules shipped with ConTeXt, to make them available +for use independent of ConTeXt. + +This package has been developed by the LuaLaTeX development team on +<https://github.com/lualatex/lualibs>. + +The current version has been build by Ulrike Fischer on +<https://github.com/u-fischer/lualibs>. The date equals the date of the context files. + +See the 'NEWS' file for version history. + + + +## Installation + + +1. If you are using TeX Live 2010 or later use 'tlmgr install lualibs'. + With miktex, use the miktex console. + Alternatively, use your (TeX or Linux) distribution's package management + system. + +2. + 1. Grab lualibs.tds.zip on CTAN. + 2. Unzip it at the root of one or your TDS trees. + 3. You may need to update some filename database after this, see your TeX + distribution's manual for details. + + +## Manifest + +### Source files: + +| file | TDS location +|------------------------------------|----------------------------------- +| lualibs.dtx |source/luatex/lualibs/lualibs.dtx +| lualibs-boolean.lua |tex/luatex/lualibs/lualibs-boolean.lua +| lualibs-compat.lua |tex/luatex/lualibs/lualibs-compat.lua +| lualibs-dir.lua |tex/luatex/lualibs/lualibs-dir.lua +| lualibs-file.lua | tex/luatex/lualibs/lualibs-file.lua +| lualibs-gzip.lua | tex/luatex/lualibs/lualibs-gzip.lua +| lualibs-function.lua | tex/luatex/lualibs/lualibs-function.lua +| lualibs-io.lua | tex/luatex/lualibs/lualibs-io.lua +| lualibs-lpeg.lua | tex/luatex/lualibs/lualibs-lpeg.lua +| lualibs-lua.lua | tex/luatex/lualibs/lualibs-lua.lua +| lualibs-math.lua | tex/luatex/lualibs/lualibs-math.lua +| lualibs-md5.lua | tex/luatex/lualibs/lualibs-md5.lua +| lualibs-number.lua | tex/luatex/lualibs/lualibs-number.lua +| lualibs-os.lua | tex/luatex/lualibs/lualibs-os.lua +| lualibs-package.lua | tex/luatex/lualibs/lualibs-package.lua +| lualibs-set.lua | tex/luatex/lualibs/lualibs-set.lua +| lualibs-string.lua | tex/luatex/lualibs/lualibs-string.lua +| lualibs-table.lua | tex/luatex/lualibs/lualibs-table.lua +| lualibs-trac-inf.lua | tex/luatex/lualibs/lualibs-trac-inf.lua +| lualibs-unicode.lua\* | tex/luatex/lualibs/lualibs-unicode.lua +| lualibs-url.lua | tex/luatex/lualibs/lualibs-url.lua +| lualibs-util-deb.lua | tex/luatex/lualibs/lualibs-util-deb.lua +| lualibs-util-dim.lua | tex/luatex/lualibs/lualibs-util-dim.lua +| lualibs-util-jsn.lua | tex/luatex/lualibs/lualibs-util-jsn.lua +| lualibs-util-lua.lua | tex/luatex/lualibs/lualibs-util-lua.lua +| lualibs-util-prs.lua | tex/luatex/lualibs/lualibs-util-prs.lua +| lualibs-util-sta.lua | tex/luatex/lualibs/lualibs-util-sta.lua +| lualibs-util-sto.lua | tex/luatex/lualibs/lualibs-util-sto.lua +| lualibs-util-str.lua | tex/luatex/lualibs/lualibs-util-str.lua +| lualibs-util-tab.lua | tex/luatex/lualibs/lualibs-util-tab.lua +| lualibs-util-tpl.lua | tex/luatex/lualibs/lualibs-util-tpl.lua +| LICENSE | doc/luatex/lualibs/LICENSE +| NEWS | doc/luatex/lualibs/NEWS +| README | doc/luatex/lualibs/README + +\* the file has been modified to enable the unicode library again + +### Derived files: + +| file | TDS location +|------------------------------------|----------------------------------- +| lualibs.lua | tex/luatex/lualibs/lualibs.lua +| lualibs-basic.lua | tex/luatex/lualibs/lualibs-basic.lua +| lualibs-basic-merged.lua | tex/luatex/lualibs/lualibs-basic-merged.lua +| lualibs-extended.lua | tex/luatex/lualibs/lualibs-extended.lua +| lualibs-extended-merged.lua | tex/luatex/lualibs/lualibs-extended-merged.lua +| lualibs.pdf | doc/luatex/lualibs/lualibs.pdf + +## License and Copyright + +As a derived work of Context, this work and the derived files are under the +GNU GPL v.2.0 license. +The full text of the license is included in the file LICENSE. + +The files taken from Context (lualibs-*.lua) are copyright PRAGMA ADE / ConTeXt +Development Team. + diff --git a/macros/luatex/generic/lualibs/lualibs-basic-merged.lua b/macros/luatex/generic/lualibs/lualibs-basic-merged.lua new file mode 100644 index 0000000000..0cb07cea2a --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-basic-merged.lua @@ -0,0 +1,5640 @@ +-- merged file : lualibs-basic-merged.lua +-- parent file : lualibs-basic.lua +-- merge date : Tue Aug 13 20:13:08 2019 + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-lua']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local next,type,tonumber=next,type,tonumber +LUAMAJORVERSION,LUAMINORVERSION=string.match(_VERSION,"^[^%d]+(%d+)%.(%d+).*$") +LUAMAJORVERSION=tonumber(LUAMAJORVERSION) or 5 +LUAMINORVERSION=tonumber(LUAMINORVERSION) or 1 +LUAVERSION=LUAMAJORVERSION+LUAMINORVERSION/10 +if LUAVERSION<5.2 and jit then + MINORVERSION=2 + LUAVERSION=5.2 +end +if not lpeg then + lpeg=require("lpeg") +end +if loadstring then + local loadnormal=load + function load(first,...) + if type(first)=="string" then + return loadstring(first,...) + else + return loadnormal(first,...) + end + end +else + loadstring=load +end +if not ipairs then + local function iterate(a,i) + i=i+1 + local v=a[i] + if v~=nil then + return i,v + end + end + function ipairs(a) + return iterate,a,0 + end +end +if not pairs then + function pairs(t) + return next,t + end +end +if not table.unpack then + table.unpack=_G.unpack +elseif not unpack then + _G.unpack=table.unpack +end +if not package.loaders then + package.loaders=package.searchers +end +local print,select,tostring=print,select,tostring +local inspectors={} +function setinspector(kind,inspector) + inspectors[kind]=inspector +end +function inspect(...) + for s=1,select("#",...) do + local value=select(s,...) + if value==nil then + print("nil") + else + local done=false + local kind=type(value) + local inspector=inspectors[kind] + if inspector then + done=inspector(value) + if done then + break + end + end + for kind,inspector in next,inspectors do + done=inspector(value) + if done then + break + end + end + if not done then + print(tostring(value)) + end + end + end +end +local dummy=function() end +function optionalrequire(...) + local ok,result=xpcall(require,dummy,...) + if ok then + return result + end +end +if lua then + lua.mask=load([[τεχ = 1]]) and "utf" or "ascii" +end +local flush=io.flush +if flush then + local execute=os.execute if execute then function os.execute(...) flush() return execute(...) end end + local exec=os.exec if exec then function os.exec (...) flush() return exec (...) end end + local spawn=os.spawn if spawn then function os.spawn (...) flush() return spawn (...) end end + local popen=io.popen if popen then function io.popen (...) flush() return popen (...) end end +end +FFISUPPORTED=type(ffi)=="table" and ffi.os~="" and ffi.arch~="" and ffi.load +if not FFISUPPORTED then + local okay;okay,ffi=pcall(require,"ffi") + FFISUPPORTED=type(ffi)=="table" and ffi.os~="" and ffi.arch~="" and ffi.load +end +if not FFISUPPORTED then + ffi=nil +elseif not ffi.number then + ffi.number=tonumber +end +if LUAVERSION>5.3 then +end +if status and os.setenv then + os.setenv("engine",string.lower(status.luatex_engine or "unknown")) +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-package']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local type=type +local gsub,format,find=string.gsub,string.format,string.find +local insert,remove=table.insert,table.remove +local P,S,Cs,lpegmatch=lpeg.P,lpeg.S,lpeg.Cs,lpeg.match +local package=package +local searchers=package.searchers or package.loaders +local filejoin=file and file.join or function(path,name) return path.."/"..name end +local isreadable=file and file.is_readable or function(name) local f=io.open(name) if f then f:close() return true end end +local addsuffix=file and file.addsuffix or function(name,suffix) return name.."."..suffix end +local function cleanpath(path) + return path +end +local pattern=Cs((((1-S("\\/"))^0*(S("\\/")^1/"/"))^0*(P(".")^1/"/"+P(1))^1)*-1) +local function lualibfile(name) + return lpegmatch(pattern,name) or name +end +local offset=luarocks and 1 or 0 +local helpers=package.helpers or { + cleanpath=cleanpath, + lualibfile=lualibfile, + trace=false, + report=function(...) print(format(...)) end, + builtin={ + ["preload table"]=searchers[1+offset], + ["path specification"]=searchers[2+offset], + ["cpath specification"]=searchers[3+offset], + ["all in one fallback"]=searchers[4+offset], + }, + methods={}, + sequence={ + "already loaded", + "preload table", + "qualified path", + "lua extra list", + "lib extra list", + "path specification", + "cpath specification", + "all in one fallback", + "not loaded", + } +} +package.helpers=helpers +local methods=helpers.methods +local builtin=helpers.builtin +local extraluapaths={} +local extralibpaths={} +local luapaths=nil +local libpaths=nil +local oldluapath=nil +local oldlibpath=nil +local nofextralua=-1 +local nofextralib=-1 +local nofpathlua=-1 +local nofpathlib=-1 +local function listpaths(what,paths) + local nofpaths=#paths + if nofpaths>0 then + for i=1,nofpaths do + helpers.report("using %s path %i: %s",what,i,paths[i]) + end + else + helpers.report("no %s paths defined",what) + end + return nofpaths +end +local function getextraluapaths() + if helpers.trace and #extraluapaths~=nofextralua then + nofextralua=listpaths("extra lua",extraluapaths) + end + return extraluapaths +end +local function getextralibpaths() + if helpers.trace and #extralibpaths~=nofextralib then + nofextralib=listpaths("extra lib",extralibpaths) + end + return extralibpaths +end +local function getluapaths() + local luapath=package.path or "" + if oldluapath~=luapath then + luapaths=file.splitpath(luapath,";") + oldluapath=luapath + nofpathlua=-1 + end + if helpers.trace and #luapaths~=nofpathlua then + nofpathlua=listpaths("builtin lua",luapaths) + end + return luapaths +end +local function getlibpaths() + local libpath=package.cpath or "" + if oldlibpath~=libpath then + libpaths=file.splitpath(libpath,";") + oldlibpath=libpath + nofpathlib=-1 + end + if helpers.trace and #libpaths~=nofpathlib then + nofpathlib=listpaths("builtin lib",libpaths) + end + return libpaths +end +package.luapaths=getluapaths +package.libpaths=getlibpaths +package.extraluapaths=getextraluapaths +package.extralibpaths=getextralibpaths +local hashes={ + lua={}, + lib={}, +} +local function registerpath(tag,what,target,...) + local pathlist={... } + local cleanpath=helpers.cleanpath + local trace=helpers.trace + local report=helpers.report + local hash=hashes[what] + local function add(path) + local path=cleanpath(path) + if not hash[path] then + target[#target+1]=path + hash[path]=true + if trace then + report("registered %s path %s: %s",tag,#target,path) + end + else + if trace then + report("duplicate %s path: %s",tag,path) + end + end + end + for p=1,#pathlist do + local path=pathlist[p] + if type(path)=="table" then + for i=1,#path do + add(path[i]) + end + else + add(path) + end + end +end +local function pushpath(tag,what,target,path) + local path=helpers.cleanpath(path) + insert(target,1,path) + if helpers.trace then + helpers.report("pushing %s path in front: %s",tag,path) + end +end +local function poppath(tag,what,target) + local path=remove(target,1) + if helpers.trace then + if path then + helpers.report("popping %s path from front: %s",tag,path) + else + helpers.report("no %s path to pop",tag) + end + end +end +helpers.registerpath=registerpath +function package.extraluapath(...) + registerpath("extra lua","lua",extraluapaths,...) +end +function package.pushluapath(path) + pushpath("extra lua","lua",extraluapaths,path) +end +function package.popluapath() + poppath("extra lua","lua",extraluapaths) +end +function package.extralibpath(...) + registerpath("extra lib","lib",extralibpaths,...) +end +function package.pushlibpath(path) + pushpath("extra lib","lib",extralibpaths,path) +end +function package.poplibpath() + poppath("extra lib","lua",extralibpaths) +end +local function loadedaslib(resolved,rawname) + local base=gsub(rawname,"%.","_") + local init="luaopen_"..gsub(base,"%.","_") + if helpers.trace then + helpers.report("calling loadlib with '%s' with init '%s'",resolved,init) + end + return package.loadlib(resolved,init) +end +helpers.loadedaslib=loadedaslib +local function loadedbypath(name,rawname,paths,islib,what) + local trace=helpers.trace + for p=1,#paths do + local path=paths[p] + local resolved=filejoin(path,name) + if trace then + helpers.report("%s path, identifying '%s' on '%s'",what,name,path) + end + if isreadable(resolved) then + if trace then + helpers.report("%s path, '%s' found on '%s'",what,name,resolved) + end + if islib then + return loadedaslib(resolved,rawname) + else + return loadfile(resolved) + end + end + end +end +helpers.loadedbypath=loadedbypath +local function loadedbyname(name,rawname) + if find(name,"^/") or find(name,"^[a-zA-Z]:/") then + local trace=helpers.trace + if trace then + helpers.report("qualified name, identifying '%s'",what,name) + end + if isreadable(name) then + if trace then + helpers.report("qualified name, '%s' found",what,name) + end + return loadfile(name) + end + end +end +helpers.loadedbyname=loadedbyname +methods["already loaded"]=function(name) + return package.loaded[name] +end +methods["preload table"]=function(name) + return builtin["preload table"](name) +end +methods["qualified path"]=function(name) + return loadedbyname(addsuffix(lualibfile(name),"lua"),name) +end +methods["lua extra list"]=function(name) + return loadedbypath(addsuffix(lualibfile(name),"lua"),name,getextraluapaths(),false,"lua") +end +methods["lib extra list"]=function(name) + return loadedbypath(addsuffix(lualibfile(name),os.libsuffix),name,getextralibpaths(),true,"lib") +end +methods["path specification"]=function(name) + getluapaths() + return builtin["path specification"](name) +end +methods["cpath specification"]=function(name) + getlibpaths() + return builtin["cpath specification"](name) +end +methods["all in one fallback"]=function(name) + return builtin["all in one fallback"](name) +end +methods["not loaded"]=function(name) + if helpers.trace then + helpers.report("unable to locate '%s'",name or "?") + end + return nil +end +local level=0 +local used={} +helpers.traceused=false +function helpers.loaded(name) + local sequence=helpers.sequence + level=level+1 + for i=1,#sequence do + local method=sequence[i] + if helpers.trace then + helpers.report("%s, level '%s', method '%s', name '%s'","locating",level,method,name) + end + local result,rest=methods[method](name) + if type(result)=="function" then + if helpers.trace then + helpers.report("%s, level '%s', method '%s', name '%s'","found",level,method,name) + end + if helpers.traceused then + used[#used+1]={ level=level,name=name } + end + level=level-1 + return result,rest + end + end + level=level-1 + return nil +end +function helpers.showused() + local n=#used + if n>0 then + helpers.report("%s libraries loaded:",n) + helpers.report() + for i=1,n do + local u=used[i] + helpers.report("%i %a",u.level,u.name) + end + helpers.report() + end +end +function helpers.unload(name) + if helpers.trace then + if package.loaded[name] then + helpers.report("unloading, name '%s', %s",name,"done") + else + helpers.report("unloading, name '%s', %s",name,"not loaded") + end + end + package.loaded[name]=nil +end +table.insert(searchers,1,helpers.loaded) +if context then + package.path="" +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-lpeg']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +lpeg=require("lpeg") +local lpeg=lpeg +if not lpeg.print then function lpeg.print(...) print(lpeg.pcode(...)) end end +local type,next,tostring=type,next,tostring +local byte,char,gmatch,format=string.byte,string.char,string.gmatch,string.format +local floor=math.floor +local P,R,S,V,Ct,C,Cs,Cc,Cp,Cmt=lpeg.P,lpeg.R,lpeg.S,lpeg.V,lpeg.Ct,lpeg.C,lpeg.Cs,lpeg.Cc,lpeg.Cp,lpeg.Cmt +local lpegtype,lpegmatch,lpegprint=lpeg.type,lpeg.match,lpeg.print +if setinspector then + setinspector("lpeg",function(v) if lpegtype(v) then lpegprint(v) return true end end) +end +lpeg.patterns=lpeg.patterns or {} +local patterns=lpeg.patterns +local anything=P(1) +local endofstring=P(-1) +local alwaysmatched=P(true) +patterns.anything=anything +patterns.endofstring=endofstring +patterns.beginofstring=alwaysmatched +patterns.alwaysmatched=alwaysmatched +local sign=S('+-') +local zero=P('0') +local digit=R('09') +local digits=digit^1 +local octdigit=R("07") +local octdigits=octdigit^1 +local lowercase=R("az") +local uppercase=R("AZ") +local underscore=P("_") +local hexdigit=digit+lowercase+uppercase +local hexdigits=hexdigit^1 +local cr,lf,crlf=P("\r"),P("\n"),P("\r\n") +local newline=P("\r")*(P("\n")+P(true))+P("\n") +local escaped=P("\\")*anything +local squote=P("'") +local dquote=P('"') +local space=P(" ") +local period=P(".") +local comma=P(",") +local utfbom_32_be=P('\000\000\254\255') +local utfbom_32_le=P('\255\254\000\000') +local utfbom_16_be=P('\254\255') +local utfbom_16_le=P('\255\254') +local utfbom_8=P('\239\187\191') +local utfbom=utfbom_32_be+utfbom_32_le+utfbom_16_be+utfbom_16_le+utfbom_8 +local utftype=utfbom_32_be*Cc("utf-32-be")+utfbom_32_le*Cc("utf-32-le")+utfbom_16_be*Cc("utf-16-be")+utfbom_16_le*Cc("utf-16-le")+utfbom_8*Cc("utf-8")+alwaysmatched*Cc("utf-8") +local utfstricttype=utfbom_32_be*Cc("utf-32-be")+utfbom_32_le*Cc("utf-32-le")+utfbom_16_be*Cc("utf-16-be")+utfbom_16_le*Cc("utf-16-le")+utfbom_8*Cc("utf-8") +local utfoffset=utfbom_32_be*Cc(4)+utfbom_32_le*Cc(4)+utfbom_16_be*Cc(2)+utfbom_16_le*Cc(2)+utfbom_8*Cc(3)+Cc(0) +local utf8next=R("\128\191") +patterns.utfbom_32_be=utfbom_32_be +patterns.utfbom_32_le=utfbom_32_le +patterns.utfbom_16_be=utfbom_16_be +patterns.utfbom_16_le=utfbom_16_le +patterns.utfbom_8=utfbom_8 +patterns.utf_16_be_nl=P("\000\r\000\n")+P("\000\r")+P("\000\n") +patterns.utf_16_le_nl=P("\r\000\n\000")+P("\r\000")+P("\n\000") +patterns.utf_32_be_nl=P("\000\000\000\r\000\000\000\n")+P("\000\000\000\r")+P("\000\000\000\n") +patterns.utf_32_le_nl=P("\r\000\000\000\n\000\000\000")+P("\r\000\000\000")+P("\n\000\000\000") +patterns.utf8one=R("\000\127") +patterns.utf8two=R("\194\223")*utf8next +patterns.utf8three=R("\224\239")*utf8next*utf8next +patterns.utf8four=R("\240\244")*utf8next*utf8next*utf8next +patterns.utfbom=utfbom +patterns.utftype=utftype +patterns.utfstricttype=utfstricttype +patterns.utfoffset=utfoffset +local utf8char=patterns.utf8one+patterns.utf8two+patterns.utf8three+patterns.utf8four +local validutf8char=utf8char^0*endofstring*Cc(true)+Cc(false) +local utf8character=P(1)*R("\128\191")^0 +patterns.utf8=utf8char +patterns.utf8char=utf8char +patterns.utf8character=utf8character +patterns.validutf8=validutf8char +patterns.validutf8char=validutf8char +local eol=S("\n\r") +local spacer=S(" \t\f\v") +local whitespace=eol+spacer +local nonspacer=1-spacer +local nonwhitespace=1-whitespace +patterns.eol=eol +patterns.spacer=spacer +patterns.whitespace=whitespace +patterns.nonspacer=nonspacer +patterns.nonwhitespace=nonwhitespace +local stripper=spacer^0*C((spacer^0*nonspacer^1)^0) +local fullstripper=whitespace^0*C((whitespace^0*nonwhitespace^1)^0) +local collapser=Cs(spacer^0/""*nonspacer^0*((spacer^0/" "*nonspacer^1)^0)) +local nospacer=Cs((whitespace^1/""+nonwhitespace^1)^0) +local b_collapser=Cs(whitespace^0/""*(nonwhitespace^1+whitespace^1/" ")^0) +local e_collapser=Cs((whitespace^1*endofstring/""+nonwhitespace^1+whitespace^1/" ")^0) +local m_collapser=Cs((nonwhitespace^1+whitespace^1/" ")^0) +local b_stripper=Cs(spacer^0/""*(nonspacer^1+spacer^1/" ")^0) +local e_stripper=Cs((spacer^1*endofstring/""+nonspacer^1+spacer^1/" ")^0) +local m_stripper=Cs((nonspacer^1+spacer^1/" ")^0) +patterns.stripper=stripper +patterns.fullstripper=fullstripper +patterns.collapser=collapser +patterns.nospacer=nospacer +patterns.b_collapser=b_collapser +patterns.m_collapser=m_collapser +patterns.e_collapser=e_collapser +patterns.b_stripper=b_stripper +patterns.m_stripper=m_stripper +patterns.e_stripper=e_stripper +patterns.lowercase=lowercase +patterns.uppercase=uppercase +patterns.letter=patterns.lowercase+patterns.uppercase +patterns.space=space +patterns.tab=P("\t") +patterns.spaceortab=patterns.space+patterns.tab +patterns.newline=newline +patterns.emptyline=newline^1 +patterns.equal=P("=") +patterns.comma=comma +patterns.commaspacer=comma*spacer^0 +patterns.period=period +patterns.colon=P(":") +patterns.semicolon=P(";") +patterns.underscore=underscore +patterns.escaped=escaped +patterns.squote=squote +patterns.dquote=dquote +patterns.nosquote=(escaped+(1-squote))^0 +patterns.nodquote=(escaped+(1-dquote))^0 +patterns.unsingle=(squote/"")*patterns.nosquote*(squote/"") +patterns.undouble=(dquote/"")*patterns.nodquote*(dquote/"") +patterns.unquoted=patterns.undouble+patterns.unsingle +patterns.unspacer=((patterns.spacer^1)/"")^0 +patterns.singlequoted=squote*patterns.nosquote*squote +patterns.doublequoted=dquote*patterns.nodquote*dquote +patterns.quoted=patterns.doublequoted+patterns.singlequoted +patterns.digit=digit +patterns.digits=digits +patterns.octdigit=octdigit +patterns.octdigits=octdigits +patterns.hexdigit=hexdigit +patterns.hexdigits=hexdigits +patterns.sign=sign +patterns.cardinal=digits +patterns.integer=sign^-1*digits +patterns.unsigned=digit^0*period*digits +patterns.float=sign^-1*patterns.unsigned +patterns.cunsigned=digit^0*comma*digits +patterns.cpunsigned=digit^0*(period+comma)*digits +patterns.cfloat=sign^-1*patterns.cunsigned +patterns.cpfloat=sign^-1*patterns.cpunsigned +patterns.number=patterns.float+patterns.integer +patterns.cnumber=patterns.cfloat+patterns.integer +patterns.cpnumber=patterns.cpfloat+patterns.integer +patterns.oct=zero*octdigits +patterns.octal=patterns.oct +patterns.HEX=zero*P("X")*(digit+uppercase)^1 +patterns.hex=zero*P("x")*(digit+lowercase)^1 +patterns.hexadecimal=zero*S("xX")*hexdigits +patterns.hexafloat=sign^-1*zero*S("xX")*(hexdigit^0*period*hexdigits+hexdigits*period*hexdigit^0+hexdigits)*(S("pP")*sign^-1*hexdigits)^-1 +patterns.decafloat=sign^-1*(digit^0*period*digits+digits*period*digit^0+digits)*S("eE")*sign^-1*digits +patterns.propername=(uppercase+lowercase+underscore)*(uppercase+lowercase+underscore+digit)^0*endofstring +patterns.somecontent=(anything-newline-space)^1 +patterns.beginline=#(1-newline) +patterns.longtostring=Cs(whitespace^0/""*((patterns.quoted+nonwhitespace^1+whitespace^1/""*(endofstring+Cc(" ")))^0)) +function anywhere(pattern) + return (1-P(pattern))^0*P(pattern) +end +lpeg.anywhere=anywhere +function lpeg.instringchecker(p) + p=anywhere(p) + return function(str) + return lpegmatch(p,str) and true or false + end +end +function lpeg.splitter(pattern,action) + if action then + return (((1-P(pattern))^1)/action+1)^0 + else + return (Cs((1-P(pattern))^1)+1)^0 + end +end +function lpeg.tsplitter(pattern,action) + if action then + return Ct((((1-P(pattern))^1)/action+1)^0) + else + return Ct((Cs((1-P(pattern))^1)+1)^0) + end +end +local splitters_s,splitters_m,splitters_t={},{},{} +local function splitat(separator,single) + local splitter=(single and splitters_s[separator]) or splitters_m[separator] + if not splitter then + separator=P(separator) + local other=C((1-separator)^0) + if single then + local any=anything + splitter=other*(separator*C(any^0)+"") + splitters_s[separator]=splitter + else + splitter=other*(separator*other)^0 + splitters_m[separator]=splitter + end + end + return splitter +end +local function tsplitat(separator) + local splitter=splitters_t[separator] + if not splitter then + splitter=Ct(splitat(separator)) + splitters_t[separator]=splitter + end + return splitter +end +lpeg.splitat=splitat +lpeg.tsplitat=tsplitat +function string.splitup(str,separator) + if not separator then + separator="," + end + return lpegmatch(splitters_m[separator] or splitat(separator),str) +end +local cache={} +function lpeg.split(separator,str) + local c=cache[separator] + if not c then + c=tsplitat(separator) + cache[separator]=c + end + return lpegmatch(c,str) +end +function string.split(str,separator) + if separator then + local c=cache[separator] + if not c then + c=tsplitat(separator) + cache[separator]=c + end + return lpegmatch(c,str) + else + return { str } + end +end +local spacing=patterns.spacer^0*newline +local empty=spacing*Cc("") +local nonempty=Cs((1-spacing)^1)*spacing^-1 +local content=(empty+nonempty)^1 +patterns.textline=content +local linesplitter=tsplitat(newline) +patterns.linesplitter=linesplitter +function string.splitlines(str) + return lpegmatch(linesplitter,str) +end +local cache={} +function lpeg.checkedsplit(separator,str) + local c=cache[separator] + if not c then + separator=P(separator) + local other=C((1-separator)^1) + c=Ct(separator^0*other*(separator^1*other)^0) + cache[separator]=c + end + return lpegmatch(c,str) +end +function string.checkedsplit(str,separator) + local c=cache[separator] + if not c then + separator=P(separator) + local other=C((1-separator)^1) + c=Ct(separator^0*other*(separator^1*other)^0) + cache[separator]=c + end + return lpegmatch(c,str) +end +local function f2(s) local c1,c2=byte(s,1,2) return c1*64+c2-12416 end +local function f3(s) local c1,c2,c3=byte(s,1,3) return (c1*64+c2)*64+c3-925824 end +local function f4(s) local c1,c2,c3,c4=byte(s,1,4) return ((c1*64+c2)*64+c3)*64+c4-63447168 end +local utf8byte=patterns.utf8one/byte+patterns.utf8two/f2+patterns.utf8three/f3+patterns.utf8four/f4 +patterns.utf8byte=utf8byte +local cache={} +function lpeg.stripper(str) + if type(str)=="string" then + local s=cache[str] + if not s then + s=Cs(((S(str)^1)/""+1)^0) + cache[str]=s + end + return s + else + return Cs(((str^1)/""+1)^0) + end +end +local cache={} +function lpeg.keeper(str) + if type(str)=="string" then + local s=cache[str] + if not s then + s=Cs((((1-S(str))^1)/""+1)^0) + cache[str]=s + end + return s + else + return Cs((((1-str)^1)/""+1)^0) + end +end +function lpeg.frontstripper(str) + return (P(str)+P(true))*Cs(anything^0) +end +function lpeg.endstripper(str) + return Cs((1-P(str)*endofstring)^0) +end +function lpeg.replacer(one,two,makefunction,isutf) + local pattern + local u=isutf and utf8char or 1 + if type(one)=="table" then + local no=#one + local p=P(false) + if no==0 then + for k,v in next,one do + p=p+P(k)/v + end + pattern=Cs((p+u)^0) + elseif no==1 then + local o=one[1] + one,two=P(o[1]),o[2] + pattern=Cs((one/two+u)^0) + else + for i=1,no do + local o=one[i] + p=p+P(o[1])/o[2] + end + pattern=Cs((p+u)^0) + end + else + pattern=Cs((P(one)/(two or "")+u)^0) + end + if makefunction then + return function(str) + return lpegmatch(pattern,str) + end + else + return pattern + end +end +function lpeg.finder(lst,makefunction,isutf) + local pattern + if type(lst)=="table" then + pattern=P(false) + if #lst==0 then + for k,v in next,lst do + pattern=pattern+P(k) + end + else + for i=1,#lst do + pattern=pattern+P(lst[i]) + end + end + else + pattern=P(lst) + end + if isutf then + pattern=((utf8char or 1)-pattern)^0*pattern + else + pattern=(1-pattern)^0*pattern + end + if makefunction then + return function(str) + return lpegmatch(pattern,str) + end + else + return pattern + end +end +local splitters_f,splitters_s={},{} +function lpeg.firstofsplit(separator) + local splitter=splitters_f[separator] + if not splitter then + local pattern=P(separator) + splitter=C((1-pattern)^0) + splitters_f[separator]=splitter + end + return splitter +end +function lpeg.secondofsplit(separator) + local splitter=splitters_s[separator] + if not splitter then + local pattern=P(separator) + splitter=(1-pattern)^0*pattern*C(anything^0) + splitters_s[separator]=splitter + end + return splitter +end +local splitters_s,splitters_p={},{} +function lpeg.beforesuffix(separator) + local splitter=splitters_s[separator] + if not splitter then + local pattern=P(separator) + splitter=C((1-pattern)^0)*pattern*endofstring + splitters_s[separator]=splitter + end + return splitter +end +function lpeg.afterprefix(separator) + local splitter=splitters_p[separator] + if not splitter then + local pattern=P(separator) + splitter=pattern*C(anything^0) + splitters_p[separator]=splitter + end + return splitter +end +function lpeg.balancer(left,right) + left,right=P(left),P(right) + return P { left*((1-left-right)+V(1))^0*right } +end +function lpeg.counter(pattern,action) + local n=0 + local pattern=(P(pattern)/function() n=n+1 end+anything)^0 + if action then + return function(str) n=0;lpegmatch(pattern,str);action(n) end + else + return function(str) n=0;lpegmatch(pattern,str);return n end + end +end +function lpeg.is_lpeg(p) + return p and lpegtype(p)=="pattern" +end +function lpeg.oneof(list,...) + if type(list)~="table" then + list={ list,... } + end + local p=P(list[1]) + for l=2,#list do + p=p+P(list[l]) + end + return p +end +local sort=table.sort +local function copyindexed(old) + local new={} + for i=1,#old do + new[i]=old + end + return new +end +local function sortedkeys(tab) + local keys,s={},0 + for key,_ in next,tab do + s=s+1 + keys[s]=key + end + sort(keys) + return keys +end +function lpeg.append(list,pp,delayed,checked) + local p=pp + if #list>0 then + local keys=copyindexed(list) + sort(keys) + for i=#keys,1,-1 do + local k=keys[i] + if p then + p=P(k)+p + else + p=P(k) + end + end + elseif delayed then + local keys=sortedkeys(list) + if p then + for i=1,#keys,1 do + local k=keys[i] + local v=list[k] + p=P(k)/list+p + end + else + for i=1,#keys do + local k=keys[i] + local v=list[k] + if p then + p=P(k)+p + else + p=P(k) + end + end + if p then + p=p/list + end + end + elseif checked then + local keys=sortedkeys(list) + for i=1,#keys do + local k=keys[i] + local v=list[k] + if p then + if k==v then + p=P(k)+p + else + p=P(k)/v+p + end + else + if k==v then + p=P(k) + else + p=P(k)/v + end + end + end + else + local keys=sortedkeys(list) + for i=1,#keys do + local k=keys[i] + local v=list[k] + if p then + p=P(k)/v+p + else + p=P(k)/v + end + end + end + return p +end +local p_false=P(false) +local p_true=P(true) +local lower=utf and utf.lower or string.lower +local upper=utf and utf.upper or string.upper +function lpeg.setutfcasers(l,u) + lower=l or lower + upper=u or upper +end +local function make1(t,rest) + local p=p_false + local keys=sortedkeys(t) + for i=1,#keys do + local k=keys[i] + if k~="" then + local v=t[k] + if v==true then + p=p+P(k)*p_true + elseif v==false then + else + p=p+P(k)*make1(v,v[""]) + end + end + end + if rest then + p=p+p_true + end + return p +end +local function make2(t,rest) + local p=p_false + local keys=sortedkeys(t) + for i=1,#keys do + local k=keys[i] + if k~="" then + local v=t[k] + if v==true then + p=p+(P(lower(k))+P(upper(k)))*p_true + elseif v==false then + else + p=p+(P(lower(k))+P(upper(k)))*make2(v,v[""]) + end + end + end + if rest then + p=p+p_true + end + return p +end +local function utfchartabletopattern(list,insensitive) + local tree={} + local n=#list + if n==0 then + for s in next,list do + local t=tree + local p,pk + for c in gmatch(s,".") do + if t==true then + t={ [c]=true,[""]=true } + p[pk]=t + p=t + t=false + elseif t==false then + t={ [c]=false } + p[pk]=t + p=t + t=false + else + local tc=t[c] + if not tc then + tc=false + t[c]=false + end + p=t + t=tc + end + pk=c + end + if t==false then + p[pk]=true + elseif t==true then + else + t[""]=true + end + end + else + for i=1,n do + local s=list[i] + local t=tree + local p,pk + for c in gmatch(s,".") do + if t==true then + t={ [c]=true,[""]=true } + p[pk]=t + p=t + t=false + elseif t==false then + t={ [c]=false } + p[pk]=t + p=t + t=false + else + local tc=t[c] + if not tc then + tc=false + t[c]=false + end + p=t + t=tc + end + pk=c + end + if t==false then + p[pk]=true + elseif t==true then + else + t[""]=true + end + end + end + return (insensitive and make2 or make1)(tree) +end +lpeg.utfchartabletopattern=utfchartabletopattern +function lpeg.utfreplacer(list,insensitive) + local pattern=Cs((utfchartabletopattern(list,insensitive)/list+utf8character)^0) + return function(str) + return lpegmatch(pattern,str) or str + end +end +patterns.containseol=lpeg.finder(eol) +local function nextstep(n,step,result) + local m=n%step + local d=floor(n/step) + if d>0 then + local v=V(tostring(step)) + local s=result.start + for i=1,d do + if s then + s=v*s + else + s=v + end + end + result.start=s + end + if step>1 and result.start then + local v=V(tostring(step/2)) + result[tostring(step)]=v*v + end + if step>0 then + return nextstep(m,step/2,result) + else + return result + end +end +function lpeg.times(pattern,n) + return P(nextstep(n,2^16,{ "start",["1"]=pattern })) +end +do + local trailingzeros=zero^0*-digit + local stripper=Cs(( + digits*( + period*trailingzeros/""+period*(digit-trailingzeros)^1*(trailingzeros/"") + )+1 + )^0) + lpeg.patterns.stripzeros=stripper + local nonzero=digit-zero + local trailingzeros=zero^1*endofstring + local stripper=Cs((1-period)^0*( + period*trailingzeros/""+period*(nonzero^1+(trailingzeros/"")+zero^1)^0+endofstring + )) + lpeg.patterns.stripzero=stripper +end +local byte_to_HEX={} +local byte_to_hex={} +local byte_to_dec={} +local hex_to_byte={} +for i=0,255 do + local H=format("%02X",i) + local h=format("%02x",i) + local d=format("%03i",i) + local c=char(i) + byte_to_HEX[c]=H + byte_to_hex[c]=h + byte_to_dec[c]=d + hex_to_byte[h]=c + hex_to_byte[H]=c +end +local hextobyte=P(2)/hex_to_byte +local bytetoHEX=P(1)/byte_to_HEX +local bytetohex=P(1)/byte_to_hex +local bytetodec=P(1)/byte_to_dec +local hextobytes=Cs(hextobyte^0) +local bytestoHEX=Cs(bytetoHEX^0) +local bytestohex=Cs(bytetohex^0) +local bytestodec=Cs(bytetodec^0) +patterns.hextobyte=hextobyte +patterns.bytetoHEX=bytetoHEX +patterns.bytetohex=bytetohex +patterns.bytetodec=bytetodec +patterns.hextobytes=hextobytes +patterns.bytestoHEX=bytestoHEX +patterns.bytestohex=bytestohex +patterns.bytestodec=bytestodec +function string.toHEX(s) + if not s or s=="" then + return s + else + return lpegmatch(bytestoHEX,s) + end +end +function string.tohex(s) + if not s or s=="" then + return s + else + return lpegmatch(bytestohex,s) + end +end +function string.todec(s) + if not s or s=="" then + return s + else + return lpegmatch(bytestodec,s) + end +end +function string.tobytes(s) + if not s or s=="" then + return s + else + return lpegmatch(hextobytes,s) + end +end +local patterns={} +local function containsws(what) + local p=patterns[what] + if not p then + local p1=P(what)*(whitespace+endofstring)*Cc(true) + local p2=whitespace*P(p1) + p=P(p1)+P(1-p2)^0*p2+Cc(false) + patterns[what]=p + end + return p +end +lpeg.containsws=containsws +function string.containsws(str,what) + return lpegmatch(patterns[what] or containsws(what),str) +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-functions']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +functions=functions or {} +function functions.dummy() end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-string']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local string=string +local sub,gmatch,format,char,byte,rep,lower=string.sub,string.gmatch,string.format,string.char,string.byte,string.rep,string.lower +local lpegmatch,patterns=lpeg.match,lpeg.patterns +local P,S,C,Ct,Cc,Cs=lpeg.P,lpeg.S,lpeg.C,lpeg.Ct,lpeg.Cc,lpeg.Cs +local unquoted=patterns.squote*C(patterns.nosquote)*patterns.squote+patterns.dquote*C(patterns.nodquote)*patterns.dquote +function string.unquoted(str) + return lpegmatch(unquoted,str) or str +end +function string.quoted(str) + return format("%q",str) +end +function string.count(str,pattern) + local n=0 + for _ in gmatch(str,pattern) do + n=n+1 + end + return n +end +function string.limit(str,n,sentinel) + if #str>n then + sentinel=sentinel or "..." + return sub(str,1,(n-#sentinel))..sentinel + else + return str + end +end +local stripper=patterns.stripper +local fullstripper=patterns.fullstripper +local collapser=patterns.collapser +local nospacer=patterns.nospacer +local longtostring=patterns.longtostring +function string.strip(str) + return str and lpegmatch(stripper,str) or "" +end +function string.fullstrip(str) + return str and lpegmatch(fullstripper,str) or "" +end +function string.collapsespaces(str) + return str and lpegmatch(collapser,str) or "" +end +function string.nospaces(str) + return str and lpegmatch(nospacer,str) or "" +end +function string.longtostring(str) + return str and lpegmatch(longtostring,str) or "" +end +local pattern=P(" ")^0*P(-1) +function string.is_empty(str) + if not str or str=="" then + return true + else + return lpegmatch(pattern,str) and true or false + end +end +local anything=patterns.anything +local moreescapes=Cc("%")*S(".-+%?()[]*$^{}") +local allescapes=Cc("%")*S(".-+%?()[]*") +local someescapes=Cc("%")*S(".-+%()[]") +local matchescapes=Cc(".")*S("*?") +local pattern_m=Cs ((moreescapes+anything )^0 ) +local pattern_a=Cs ((allescapes+anything )^0 ) +local pattern_b=Cs ((someescapes+matchescapes+anything )^0 ) +local pattern_c=Cs (Cc("^")*(someescapes+matchescapes+anything )^0*Cc("$") ) +function string.escapedpattern(str,simple) + return lpegmatch(simple and pattern_b or pattern_a,str) +end +function string.topattern(str,lowercase,strict) + if str=="" or type(str)~="string" then + return ".*" + elseif strict=="all" then + str=lpegmatch(pattern_m,str) + elseif strict then + str=lpegmatch(pattern_c,str) + else + str=lpegmatch(pattern_b,str) + end + if lowercase then + return lower(str) + else + return str + end +end +function string.valid(str,default) + return (type(str)=="string" and str~="" and str) or default or nil +end +string.itself=function(s) return s end +local pattern_c=Ct(C(1)^0) +local pattern_b=Ct((C(1)/byte)^0) +function string.totable(str,bytes) + return lpegmatch(bytes and pattern_b or pattern_c,str) +end +local replacer=lpeg.replacer("@","%%") +function string.tformat(fmt,...) + return format(lpegmatch(replacer,fmt),...) +end +string.quote=string.quoted +string.unquote=string.unquoted +if not string.bytetable then + local limit=5000 + function string.bytetable(str) + local n=#str + if n>limit then + local t={ byte(str,1,limit) } + for i=limit+1,n do + t[i]=byte(str,i) + end + return t + else + return { byte(str,1,n) } + end + end +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-table']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local type,next,tostring,tonumber,select=type,next,tostring,tonumber,select +local table,string=table,string +local concat,sort=table.concat,table.sort +local format,lower,dump=string.format,string.lower,string.dump +local getmetatable,setmetatable=getmetatable,setmetatable +local lpegmatch,patterns=lpeg.match,lpeg.patterns +local floor=math.floor +local stripper=patterns.stripper +function table.getn(t) + return t and #t +end +function table.strip(tab) + local lst={} + local l=0 + for i=1,#tab do + local s=lpegmatch(stripper,tab[i]) or "" + if s=="" then + else + l=l+1 + lst[l]=s + end + end + return lst +end +function table.keys(t) + if t then + local keys={} + local k=0 + for key in next,t do + k=k+1 + keys[k]=key + end + return keys + else + return {} + end +end +local function compare(a,b) + local ta=type(a) + if ta=="number" then + local tb=type(b) + if ta==tb then + return a<b + elseif tb=="string" then + return tostring(a)<b + end + elseif ta=="string" then + local tb=type(b) + if ta==tb then + return a<b + else + return a<tostring(b) + end + end + return tostring(a)<tostring(b) +end +local function sortedkeys(tab) + if tab then + local srt={} + local category=0 + local s=0 + for key in next,tab do + s=s+1 + srt[s]=key + if category~=3 then + local tkey=type(key) + if category==1 then + if tkey~="string" then + category=3 + end + elseif category==2 then + if tkey~="number" then + category=3 + end + else + if tkey=="string" then + category=1 + elseif tkey=="number" then + category=2 + else + category=3 + end + end + end + end + if s<2 then + elseif category==3 then + sort(srt,compare) + else + sort(srt) + end + return srt + else + return {} + end +end +local function sortedhashonly(tab) + if tab then + local srt={} + local s=0 + for key in next,tab do + if type(key)=="string" then + s=s+1 + srt[s]=key + end + end + if s>1 then + sort(srt) + end + return srt + else + return {} + end +end +local function sortedindexonly(tab) + if tab then + local srt={} + local s=0 + for key in next,tab do + if type(key)=="number" then + s=s+1 + srt[s]=key + end + end + if s>1 then + sort(srt) + end + return srt + else + return {} + end +end +local function sortedhashkeys(tab,cmp) + if tab then + local srt={} + local s=0 + for key in next,tab do + if key then + s=s+1 + srt[s]=key + end + end + if s>1 then + sort(srt,cmp) + end + return srt + else + return {} + end +end +function table.allkeys(t) + local keys={} + for k,v in next,t do + for k in next,v do + keys[k]=true + end + end + return sortedkeys(keys) +end +table.sortedkeys=sortedkeys +table.sortedhashonly=sortedhashonly +table.sortedindexonly=sortedindexonly +table.sortedhashkeys=sortedhashkeys +local function nothing() end +local function sortedhash(t,cmp) + if t then + local s + if cmp then + s=sortedhashkeys(t,function(a,b) return cmp(t,a,b) end) + else + s=sortedkeys(t) + end + local m=#s + if m==1 then + return next,t + elseif m>0 then + local n=0 + return function() + if n<m then + n=n+1 + local k=s[n] + return k,t[k] + end + end + end + end + return nothing +end +table.sortedhash=sortedhash +table.sortedpairs=sortedhash +function table.append(t,list) + local n=#t + for i=1,#list do + n=n+1 + t[n]=list[i] + end + return t +end +function table.prepend(t,list) + local nl=#list + local nt=nl+#t + for i=#t,1,-1 do + t[nt]=t[i] + nt=nt-1 + end + for i=1,#list do + t[i]=list[i] + end + return t +end +function table.merge(t,...) + if not t then + t={} + end + for i=1,select("#",...) do + for k,v in next,(select(i,...)) do + t[k]=v + end + end + return t +end +function table.merged(...) + local t={} + for i=1,select("#",...) do + for k,v in next,(select(i,...)) do + t[k]=v + end + end + return t +end +function table.imerge(t,...) + local nt=#t + for i=1,select("#",...) do + local nst=select(i,...) + for j=1,#nst do + nt=nt+1 + t[nt]=nst[j] + end + end + return t +end +function table.imerged(...) + local tmp={} + local ntmp=0 + for i=1,select("#",...) do + local nst=select(i,...) + for j=1,#nst do + ntmp=ntmp+1 + tmp[ntmp]=nst[j] + end + end + return tmp +end +local function fastcopy(old,metatabletoo) + if old then + local new={} + for k,v in next,old do + if type(v)=="table" then + new[k]=fastcopy(v,metatabletoo) + else + new[k]=v + end + end + if metatabletoo then + local mt=getmetatable(old) + if mt then + setmetatable(new,mt) + end + end + return new + else + return {} + end +end +local function copy(t,tables) + if not tables then + tables={} + end + local tcopy={} + if not tables[t] then + tables[t]=tcopy + end + for i,v in next,t do + if type(i)=="table" then + if tables[i] then + i=tables[i] + else + i=copy(i,tables) + end + end + if type(v)~="table" then + tcopy[i]=v + elseif tables[v] then + tcopy[i]=tables[v] + else + tcopy[i]=copy(v,tables) + end + end + local mt=getmetatable(t) + if mt then + setmetatable(tcopy,mt) + end + return tcopy +end +table.fastcopy=fastcopy +table.copy=copy +function table.derive(parent) + local child={} + if parent then + setmetatable(child,{ __index=parent }) + end + return child +end +function table.tohash(t,value) + local h={} + if t then + if value==nil then value=true end + for _,v in next,t do + h[v]=value + end + end + return h +end +function table.fromhash(t) + local hsh={} + local h=0 + for k,v in next,t do + if v then + h=h+1 + hsh[h]=k + end + end + return hsh +end +local noquotes,hexify,handle,compact,inline,functions,metacheck +local reserved=table.tohash { + 'and','break','do','else','elseif','end','false','for','function','if', + 'in','local','nil','not','or','repeat','return','then','true','until','while', + 'NaN','goto', +} +local function is_simple_table(t,hexify) + local nt=#t + if nt>0 then + local n=0 + for _,v in next,t do + n=n+1 + if type(v)=="table" then + return nil + end + end + local haszero=rawget(t,0) + if n==nt then + local tt={} + for i=1,nt do + local v=t[i] + local tv=type(v) + if tv=="number" then + if hexify then + tt[i]=format("0x%X",v) + else + tt[i]=v + end + elseif tv=="string" then + tt[i]=format("%q",v) + elseif tv=="boolean" then + tt[i]=v and "true" or "false" + else + return nil + end + end + return tt + elseif haszero and (n==nt+1) then + local tt={} + for i=0,nt do + local v=t[i] + local tv=type(v) + if tv=="number" then + if hexify then + tt[i+1]=format("0x%X",v) + else + tt[i+1]=v + end + elseif tv=="string" then + tt[i+1]=format("%q",v) + elseif tv=="boolean" then + tt[i+1]=v and "true" or "false" + else + return nil + end + end + tt[1]="[0] = "..tt[1] + return tt + end + end + return nil +end +table.is_simple_table=is_simple_table +local propername=patterns.propername +local function dummy() end +local function do_serialize(root,name,depth,level,indexed) + if level>0 then + depth=depth.." " + if indexed then + handle(format("%s{",depth)) + else + local tn=type(name) + if tn=="number" then + if hexify then + handle(format("%s[0x%X]={",depth,name)) + else + handle(format("%s[%s]={",depth,name)) + end + elseif tn=="string" then + if noquotes and not reserved[name] and lpegmatch(propername,name) then + handle(format("%s%s={",depth,name)) + else + handle(format("%s[%q]={",depth,name)) + end + elseif tn=="boolean" then + handle(format("%s[%s]={",depth,name and "true" or "false")) + else + handle(format("%s{",depth)) + end + end + end + if root and next(root)~=nil then + local first=nil + local last=0 + if compact then + last=#root + for k=1,last do + if rawget(root,k)==nil then + last=k-1 + break + end + end + if last>0 then + first=1 + end + end + local sk=sortedkeys(root) + for i=1,#sk do + local k=sk[i] + local v=root[k] + local tv=type(v) + local tk=type(k) + if compact and first and tk=="number" and k>=first and k<=last then + if tv=="number" then + if hexify then + handle(format("%s 0x%X,",depth,v)) + else + handle(format("%s %s,",depth,v)) + end + elseif tv=="string" then + handle(format("%s %q,",depth,v)) + elseif tv=="table" then + if next(v)==nil then + handle(format("%s {},",depth)) + elseif inline then + local st=is_simple_table(v,hexify) + if st then + handle(format("%s { %s },",depth,concat(st,", "))) + else + do_serialize(v,k,depth,level+1,true) + end + else + do_serialize(v,k,depth,level+1,true) + end + elseif tv=="boolean" then + handle(format("%s %s,",depth,v and "true" or "false")) + elseif tv=="function" then + if functions then + handle(format('%s load(%q),',depth,dump(v))) + else + handle(format('%s "function",',depth)) + end + else + handle(format("%s %q,",depth,tostring(v))) + end + elseif k=="__p__" then + if false then + handle(format("%s __p__=nil,",depth)) + end + elseif tv=="number" then + if tk=="number" then + if hexify then + handle(format("%s [0x%X]=0x%X,",depth,k,v)) + else + handle(format("%s [%s]=%s,",depth,k,v)) + end + elseif tk=="boolean" then + if hexify then + handle(format("%s [%s]=0x%X,",depth,k and "true" or "false",v)) + else + handle(format("%s [%s]=%s,",depth,k and "true" or "false",v)) + end + elseif tk~="string" then + elseif noquotes and not reserved[k] and lpegmatch(propername,k) then + if hexify then + handle(format("%s %s=0x%X,",depth,k,v)) + else + handle(format("%s %s=%s,",depth,k,v)) + end + else + if hexify then + handle(format("%s [%q]=0x%X,",depth,k,v)) + else + handle(format("%s [%q]=%s,",depth,k,v)) + end + end + elseif tv=="string" then + if tk=="number" then + if hexify then + handle(format("%s [0x%X]=%q,",depth,k,v)) + else + handle(format("%s [%s]=%q,",depth,k,v)) + end + elseif tk=="boolean" then + handle(format("%s [%s]=%q,",depth,k and "true" or "false",v)) + elseif tk~="string" then + elseif noquotes and not reserved[k] and lpegmatch(propername,k) then + handle(format("%s %s=%q,",depth,k,v)) + else + handle(format("%s [%q]=%q,",depth,k,v)) + end + elseif tv=="table" then + if next(v)==nil then + if tk=="number" then + if hexify then + handle(format("%s [0x%X]={},",depth,k)) + else + handle(format("%s [%s]={},",depth,k)) + end + elseif tk=="boolean" then + handle(format("%s [%s]={},",depth,k and "true" or "false")) + elseif tk~="string" then + elseif noquotes and not reserved[k] and lpegmatch(propername,k) then + handle(format("%s %s={},",depth,k)) + else + handle(format("%s [%q]={},",depth,k)) + end + elseif inline then + local st=is_simple_table(v,hexify) + if st then + if tk=="number" then + if hexify then + handle(format("%s [0x%X]={ %s },",depth,k,concat(st,", "))) + else + handle(format("%s [%s]={ %s },",depth,k,concat(st,", "))) + end + elseif tk=="boolean" then + handle(format("%s [%s]={ %s },",depth,k and "true" or "false",concat(st,", "))) + elseif tk~="string" then + elseif noquotes and not reserved[k] and lpegmatch(propername,k) then + handle(format("%s %s={ %s },",depth,k,concat(st,", "))) + else + handle(format("%s [%q]={ %s },",depth,k,concat(st,", "))) + end + else + do_serialize(v,k,depth,level+1) + end + else + do_serialize(v,k,depth,level+1) + end + elseif tv=="boolean" then + if tk=="number" then + if hexify then + handle(format("%s [0x%X]=%s,",depth,k,v and "true" or "false")) + else + handle(format("%s [%s]=%s,",depth,k,v and "true" or "false")) + end + elseif tk=="boolean" then + handle(format("%s [%s]=%s,",depth,tostring(k),v and "true" or "false")) + elseif tk~="string" then + elseif noquotes and not reserved[k] and lpegmatch(propername,k) then + handle(format("%s %s=%s,",depth,k,v and "true" or "false")) + else + handle(format("%s [%q]=%s,",depth,k,v and "true" or "false")) + end + elseif tv=="function" then + if functions then + local getinfo=debug and debug.getinfo + if getinfo then + local f=getinfo(v).what=="C" and dump(dummy) or dump(v) + if tk=="number" then + if hexify then + handle(format("%s [0x%X]=load(%q),",depth,k,f)) + else + handle(format("%s [%s]=load(%q),",depth,k,f)) + end + elseif tk=="boolean" then + handle(format("%s [%s]=load(%q),",depth,k and "true" or "false",f)) + elseif tk~="string" then + elseif noquotes and not reserved[k] and lpegmatch(propername,k) then + handle(format("%s %s=load(%q),",depth,k,f)) + else + handle(format("%s [%q]=load(%q),",depth,k,f)) + end + end + end + else + if tk=="number" then + if hexify then + handle(format("%s [0x%X]=%q,",depth,k,tostring(v))) + else + handle(format("%s [%s]=%q,",depth,k,tostring(v))) + end + elseif tk=="boolean" then + handle(format("%s [%s]=%q,",depth,k and "true" or "false",tostring(v))) + elseif tk~="string" then + elseif noquotes and not reserved[k] and lpegmatch(propername,k) then + handle(format("%s %s=%q,",depth,k,tostring(v))) + else + handle(format("%s [%q]=%q,",depth,k,tostring(v))) + end + end + end + end + if level>0 then + handle(format("%s},",depth)) + end +end +local function serialize(_handle,root,name,specification) + local tname=type(name) + if type(specification)=="table" then + noquotes=specification.noquotes + hexify=specification.hexify + handle=_handle or specification.handle or print + functions=specification.functions + compact=specification.compact + inline=specification.inline and compact + metacheck=specification.metacheck + if functions==nil then + functions=true + end + if compact==nil then + compact=true + end + if inline==nil then + inline=compact + end + if metacheck==nil then + metacheck=true + end + else + noquotes=false + hexify=false + handle=_handle or print + compact=true + inline=true + functions=true + metacheck=true + end + if tname=="string" then + if name=="return" then + handle("return {") + else + handle(name.."={") + end + elseif tname=="number" then + if hexify then + handle(format("[0x%X]={",name)) + else + handle("["..name.."]={") + end + elseif tname=="boolean" then + if name then + handle("return {") + else + handle("{") + end + else + handle("t={") + end + if root then + if metacheck and getmetatable(root) then + local dummy=root._w_h_a_t_e_v_e_r_ + root._w_h_a_t_e_v_e_r_=nil + end + if next(root)~=nil then + do_serialize(root,name,"",0) + end + end + handle("}") +end +function table.serialize(root,name,specification) + local t={} + local n=0 + local function flush(s) + n=n+1 + t[n]=s + end + serialize(flush,root,name,specification) + return concat(t,"\n") +end +table.tohandle=serialize +local maxtab=2*1024 +function table.tofile(filename,root,name,specification) + local f=io.open(filename,'w') + if f then + if maxtab>1 then + local t={} + local n=0 + local function flush(s) + n=n+1 + t[n]=s + if n>maxtab then + f:write(concat(t,"\n"),"\n") + t={} + n=0 + end + end + serialize(flush,root,name,specification) + f:write(concat(t,"\n"),"\n") + else + local function flush(s) + f:write(s,"\n") + end + serialize(flush,root,name,specification) + end + f:close() + io.flush() + end +end +local function flattened(t,f,depth) + if f==nil then + f={} + depth=0xFFFF + elseif tonumber(f) then + depth=f + f={} + elseif not depth then + depth=0xFFFF + end + for k,v in next,t do + if type(k)~="number" then + if depth>0 and type(v)=="table" then + flattened(v,f,depth-1) + else + f[#f+1]=v + end + end + end + for k=1,#t do + local v=t[k] + if depth>0 and type(v)=="table" then + flattened(v,f,depth-1) + else + f[#f+1]=v + end + end + return f +end +table.flattened=flattened +local function collapsed(t,f,h) + if f==nil then + f={} + h={} + end + for k=1,#t do + local v=t[k] + if type(v)=="table" then + collapsed(v,f,h) + elseif not h[v] then + f[#f+1]=v + h[v]=true + end + end + return f +end +local function collapsedhash(t,h) + if h==nil then + h={} + end + for k=1,#t do + local v=t[k] + if type(v)=="table" then + collapsedhash(v,h) + else + h[v]=true + end + end + return h +end +table.collapsed=collapsed +table.collapsedhash=collapsedhash +local function unnest(t,f) + if not f then + f={} + end + for i=1,#t do + local v=t[i] + if type(v)=="table" then + if type(v[1])=="table" then + unnest(v,f) + else + f[#f+1]=v + end + else + f[#f+1]=v + end + end + return f +end +function table.unnest(t) + return unnest(t) +end +local function are_equal(a,b,n,m) + if a==b then + return true + elseif a and b and #a==#b then + if not n then + n=1 + end + if not m then + m=#a + end + for i=n,m do + local ai,bi=a[i],b[i] + if ai==bi then + elseif type(ai)=="table" and type(bi)=="table" then + if not are_equal(ai,bi) then + return false + end + else + return false + end + end + return true + else + return false + end +end +local function identical(a,b) + if a~=b then + for ka,va in next,a do + local vb=b[ka] + if va==vb then + elseif type(va)=="table" and type(vb)=="table" then + if not identical(va,vb) then + return false + end + else + return false + end + end + end + return true +end +table.identical=identical +table.are_equal=are_equal +local function sparse(old,nest,keeptables) + local new={} + for k,v in next,old do + if not (v=="" or v==false) then + if nest and type(v)=="table" then + v=sparse(v,nest) + if keeptables or next(v)~=nil then + new[k]=v + end + else + new[k]=v + end + end + end + return new +end +table.sparse=sparse +function table.compact(t) + return sparse(t,true,true) +end +function table.contains(t,v) + if t then + for i=1,#t do + if t[i]==v then + return i + end + end + end + return false +end +function table.count(t) + local n=0 + for k,v in next,t do + n=n+1 + end + return n +end +function table.swapped(t,s) + local n={} + if s then + for k,v in next,s do + n[k]=v + end + end + for k,v in next,t do + n[v]=k + end + return n +end +function table.hashed(t) + for i=1,#t do + t[t[i]]=i + end + return t +end +function table.mirrored(t) + local n={} + for k,v in next,t do + n[v]=k + n[k]=v + end + return n +end +function table.reversed(t) + if t then + local tt={} + local tn=#t + if tn>0 then + local ttn=0 + for i=tn,1,-1 do + ttn=ttn+1 + tt[ttn]=t[i] + end + end + return tt + end +end +function table.reverse(t) + if t then + local n=#t + local m=n+1 + for i=1,floor(n/2) do + local j=m-i + t[i],t[j]=t[j],t[i] + end + return t + end +end +local function sequenced(t,sep,simple) + if not t then + return "" + elseif type(t)~="table" then + return t + end + local n=#t + local s={} + if n>0 then + for i=1,n do + local v=t[i] + if type(v)=="table" then + s[i]="{"..sequenced(v,sep,simple).."}" + else + s[i]=tostring(t[i]) + end + end + else + n=0 + for k,v in sortedhash(t) do + if simple then + if v==true then + n=n+1 + s[n]=k + elseif v and v~="" then + n=n+1 + if type(v)=="table" then + s[n]=k.."={"..sequenced(v,sep,simple).."}" + else + s[n]=k.."="..tostring(v) + end + end + else + n=n+1 + if type(v)=="table" then + s[n]=k.."={"..sequenced(v,sep,simple).."}" + else + s[n]=k.."="..tostring(v) + end + end + end + end + if sep==true then + return "{ "..concat(s,", ").." }" + else + return concat(s,sep or " | ") + end +end +table.sequenced=sequenced +function table.print(t,...) + if type(t)~="table" then + print(tostring(t)) + else + serialize(print,t,...) + end +end +if setinspector then + setinspector("table",function(v) if type(v)=="table" then serialize(print,v,"table") return true end end) +end +function table.sub(t,i,j) + return { unpack(t,i,j) } +end +function table.is_empty(t) + return not t or next(t)==nil +end +function table.has_one_entry(t) + return t and next(t,next(t))==nil +end +function table.loweredkeys(t) + local l={} + for k,v in next,t do + l[lower(k)]=v + end + return l +end +function table.unique(old) + local hash={} + local new={} + local n=0 + for i=1,#old do + local oi=old[i] + if not hash[oi] then + n=n+1 + new[n]=oi + hash[oi]=true + end + end + return new +end +function table.sorted(t,...) + sort(t,...) + return t +end +function table.values(t,s) + if t then + local values={} + local keys={} + local v=0 + for key,value in next,t do + if not keys[value] then + v=v+1 + values[v]=value + keys[k]=key + end + end + if s then + sort(values) + end + return values + else + return {} + end +end +function table.filtered(t,pattern,sort,cmp) + if t and type(pattern)=="string" then + if sort then + local s + if cmp then + s=sortedhashkeys(t,function(a,b) return cmp(t,a,b) end) + else + s=sortedkeys(t) + end + local n=0 + local m=#s + local function kv(s) + while n<m do + n=n+1 + local k=s[n] + if find(k,pattern) then + return k,t[k] + end + end + end + return kv,s + else + local n=next(t) + local function iterator() + while n~=nil do + local k=n + n=next(t,k) + if find(k,pattern) then + return k,t[k] + end + end + end + return iterator,t + end + else + return nothing + end +end +if not table.move then + function table.move(a1,f,e,t,a2) + if a2 and a1~=a2 then + for i=f,e do + a2[t]=a1[i] + t=t+1 + end + return a2 + else + t=t+e-f + for i=e,f,-1 do + a1[t]=a1[i] + t=t-1 + end + return a1 + end + end +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-boolean']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local type,tonumber=type,tonumber +boolean=boolean or {} +local boolean=boolean +function boolean.tonumber(b) + if b then return 1 else return 0 end +end +function toboolean(str,tolerant) + if str==nil then + return false + elseif str==false then + return false + elseif str==true then + return true + elseif str=="true" then + return true + elseif str=="false" then + return false + elseif not tolerant then + return false + elseif str==0 then + return false + elseif (tonumber(str) or 0)>0 then + return true + else + return str=="yes" or str=="on" or str=="t" + end +end +string.toboolean=toboolean +function string.booleanstring(str) + if str=="0" then + return false + elseif str=="1" then + return true + elseif str=="" then + return false + elseif str=="false" then + return false + elseif str=="true" then + return true + elseif (tonumber(str) or 0)>0 then + return true + else + return str=="yes" or str=="on" or str=="t" + end +end +function string.is_boolean(str,default,strict) + if type(str)=="string" then + if str=="true" or str=="yes" or str=="on" or str=="t" or (not strict and str=="1") then + return true + elseif str=="false" or str=="no" or str=="off" or str=="f" or (not strict and str=="0") then + return false + end + end + return default +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-number']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local tostring,tonumber=tostring,tonumber +local format,floor,match,rep=string.format,math.floor,string.match,string.rep +local concat,insert=table.concat,table.insert +local lpegmatch=lpeg.match +local floor=math.floor +number=number or {} +local number=number +if bit32 then + local bextract=bit32.extract + local t={ + "0","0","0","0","0","0","0","0", + "0","0","0","0","0","0","0","0", + "0","0","0","0","0","0","0","0", + "0","0","0","0","0","0","0","0", + } + function number.tobitstring(b,m,w) + if not w then + w=32 + end + local n=w + for i=0,w-1 do + local v=bextract(b,i) + local k=w-i + if v==1 then + n=k + t[k]="1" + else + t[k]="0" + end + end + if w then + return concat(t,"",1,w) + elseif m then + m=33-m*8 + if m<1 then + m=1 + end + return concat(t,"",1,m) + elseif n<8 then + return concat(t) + elseif n<16 then + return concat(t,"",9) + elseif n<24 then + return concat(t,"",17) + else + return concat(t,"",25) + end + end +else + function number.tobitstring(n,m) + if n>0 then + local t={} + while n>0 do + insert(t,1,n%2>0 and 1 or 0) + n=floor(n/2) + end + local nn=8-#t%8 + if nn>0 and nn<8 then + for i=1,nn do + insert(t,1,0) + end + end + if m then + m=m*8-#t + if m>0 then + insert(t,1,rep("0",m)) + end + end + return concat(t) + elseif m then + rep("00000000",m) + else + return "00000000" + end + end +end +function number.valid(str,default) + return tonumber(str) or default or nil +end +function number.toevenhex(n) + local s=format("%X",n) + if #s%2==0 then + return s + else + return "0"..s + end +end +function number.bytetodecimal(b) + local d=floor(b*100/255+0.5) + if d>100 then + return 100 + elseif d<-100 then + return -100 + else + return d + end +end +function number.decimaltobyte(d) + local b=floor(d*255/100+0.5) + if b>255 then + return 255 + elseif b<-255 then + return -255 + else + return b + end +end +function number.idiv(i,d) + return floor(i/d) +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-math']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +if not math.ceiling then + math.ceiling=math.ceil +end +if not math.round then + local floor=math.floor + function math.round(x) return floor(x+0.5) end +end +if not math.div then + local floor=math.floor + function math.div(n,m) return floor(n/m) end +end +if not math.mod then + function math.mod(n,m) return n%m end +end +if not math.sind then + local sin,cos,tan=math.sin,math.cos,math.tan + local pipi=2*math.pi/360 + function math.sind(d) return sin(d*pipi) end + function math.cosd(d) return cos(d*pipi) end + function math.tand(d) return tan(d*pipi) end +end +if not math.odd then + function math.odd (n) return n%2~=0 end + function math.even(n) return n%2==0 end +end +if not math.cosh then + local exp=math.exp + function math.cosh(x) + local xx=exp(x) + return (xx+1/xx)/2 + end + function math.sinh(x) + local xx=exp(x) + return (xx-1/xx)/2 + end + function math.tanh(x) + local xx=exp(x) + return (xx-1/xx)/(xx+1/xx) + end +end +if not math.pow then + function math.pow(x,y) + return x^y + end +end +if not math.atan2 then + math.atan2=math.atan +end +if not math.ldexp then + function math.ldexp(x,e) + return x*2.0^e + end +end +if not math.log10 then + local log=math.log + function math.log10(x) + return log(x,10) + end +end +if not math.type then + function math.type() + return "float" + end +end +if not math.tointeger then + math.mininteger=-0x4FFFFFFFFFFF + math.maxinteger=0x4FFFFFFFFFFF + local floor=math.floor + function math.tointeger(n) + local f=floor(n) + return f==n and f or nil + end +end +if not math.ult then + local floor=math.floor + function math.tointeger(m,n) + return floor(m)<floor(n) + end +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-io']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local io=io +local open,flush,write,read=io.open,io.flush,io.write,io.read +local byte,find,gsub,format=string.byte,string.find,string.gsub,string.format +local concat=table.concat +local type=type +if string.find(os.getenv("PATH"),";",1,true) then + io.fileseparator,io.pathseparator="\\",";" +else + io.fileseparator,io.pathseparator="/",":" +end +local large=0x01000000 +local medium=0x00100000 +local small=0x00020000 +local function readall(f) + local size=f:seek("end") + if size>0 then + f:seek("set",0) + return f:read(size) + else + return "" + end +end +io.readall=readall +function io.loaddata(filename,textmode) + local f=open(filename,(textmode and 'r') or 'rb') + if f then + local size=f:seek("end") + local data=nil + if size>0 then + f:seek("set",0) + data=f:read(size) + end + f:close() + return data + end +end +function io.copydata(source,target,action) + local f=open(source,"rb") + if f then + local g=open(target,"wb") + if g then + local size=f:seek("end") + if size>0 then + f:seek("set",0) + local data=f:read(size) + if action then + data=action(data) + end + if data then + g:write(data) + end + end + g:close() + end + f:close() + flush() + end +end +function io.savedata(filename,data,joiner) + local f=open(filename,"wb") + if f then + if type(data)=="table" then + f:write(concat(data,joiner or "")) + elseif type(data)=="function" then + data(f) + else + f:write(data or "") + end + f:close() + flush() + return true + else + return false + end +end +if fio and fio.readline then + local readline=fio.readline + function io.loadlines(filename,n) + local f=open(filename,'r') + if not f then + elseif n then + local lines={} + for i=1,n do + local line=readline(f) + if line then + lines[i]=line + else + break + end + end + f:close() + lines=concat(lines,"\n") + if #lines>0 then + return lines + end + else + local line=readline(f) + f:close() + if line and #line>0 then + return line + end + end + end +else + function io.loadlines(filename,n) + local f=open(filename,'r') + if not f then + elseif n then + local lines={} + for i=1,n do + local line=f:read("*lines") + if line then + lines[i]=line + else + break + end + end + f:close() + lines=concat(lines,"\n") + if #lines>0 then + return lines + end + else + local line=f:read("*line") or "" + f:close() + if #line>0 then + return line + end + end + end +end +function io.loadchunk(filename,n) + local f=open(filename,'rb') + if f then + local data=f:read(n or 1024) + f:close() + if #data>0 then + return data + end + end +end +function io.exists(filename) + local f=open(filename) + if f==nil then + return false + else + f:close() + return true + end +end +function io.size(filename) + local f=open(filename) + if f==nil then + return 0 + else + local s=f:seek("end") + f:close() + return s + end +end +local function noflines(f) + if type(f)=="string" then + local f=open(filename) + if f then + local n=f and noflines(f) or 0 + f:close() + return n + else + return 0 + end + else + local n=0 + for _ in f:lines() do + n=n+1 + end + f:seek('set',0) + return n + end +end +io.noflines=noflines +local nextchar={ + [ 4]=function(f) + return f:read(1,1,1,1) + end, + [ 2]=function(f) + return f:read(1,1) + end, + [ 1]=function(f) + return f:read(1) + end, + [-2]=function(f) + local a,b=f:read(1,1) + return b,a + end, + [-4]=function(f) + local a,b,c,d=f:read(1,1,1,1) + return d,c,b,a + end +} +function io.characters(f,n) + if f then + return nextchar[n or 1],f + end +end +local nextbyte={ + [4]=function(f) + local a,b,c,d=f:read(1,1,1,1) + if d then + return byte(a),byte(b),byte(c),byte(d) + end + end, + [3]=function(f) + local a,b,c=f:read(1,1,1) + if b then + return byte(a),byte(b),byte(c) + end + end, + [2]=function(f) + local a,b=f:read(1,1) + if b then + return byte(a),byte(b) + end + end, + [1]=function (f) + local a=f:read(1) + if a then + return byte(a) + end + end, + [-2]=function (f) + local a,b=f:read(1,1) + if b then + return byte(b),byte(a) + end + end, + [-3]=function(f) + local a,b,c=f:read(1,1,1) + if b then + return byte(c),byte(b),byte(a) + end + end, + [-4]=function(f) + local a,b,c,d=f:read(1,1,1,1) + if d then + return byte(d),byte(c),byte(b),byte(a) + end + end +} +function io.bytes(f,n) + if f then + return nextbyte[n or 1],f + else + return nil,nil + end +end +function io.ask(question,default,options) + while true do + write(question) + if options then + write(format(" [%s]",concat(options,"|"))) + end + if default then + write(format(" [%s]",default)) + end + write(format(" ")) + flush() + local answer=read() + answer=gsub(answer,"^%s*(.*)%s*$","%1") + if answer=="" and default then + return default + elseif not options then + return answer + else + for k=1,#options do + if options[k]==answer then + return answer + end + end + local pattern="^"..answer + for k=1,#options do + local v=options[k] + if find(v,pattern) then + return v + end + end + end + end +end +local function readnumber(f,n,m) + if m then + f:seek("set",n) + n=m + end + if n==1 then + return byte(f:read(1)) + elseif n==2 then + local a,b=byte(f:read(2),1,2) + return 0x100*a+b + elseif n==3 then + local a,b,c=byte(f:read(3),1,3) + return 0x10000*a+0x100*b+c + elseif n==4 then + local a,b,c,d=byte(f:read(4),1,4) + return 0x1000000*a+0x10000*b+0x100*c+d + elseif n==8 then + local a,b=readnumber(f,4),readnumber(f,4) + return 0x100*a+b + elseif n==12 then + local a,b,c=readnumber(f,4),readnumber(f,4),readnumber(f,4) + return 0x10000*a+0x100*b+c + elseif n==-2 then + local b,a=byte(f:read(2),1,2) + return 0x100*a+b + elseif n==-3 then + local c,b,a=byte(f:read(3),1,3) + return 0x10000*a+0x100*b+c + elseif n==-4 then + local d,c,b,a=byte(f:read(4),1,4) + return 0x1000000*a+0x10000*b+0x100*c+d + elseif n==-8 then + local h,g,f,e,d,c,b,a=byte(f:read(8),1,8) + return 0x100000000000000*a+0x1000000000000*b+0x10000000000*c+0x100000000*d+0x1000000*e+0x10000*f+0x100*g+h + else + return 0 + end +end +io.readnumber=readnumber +function io.readstring(f,n,m) + if m then + f:seek("set",n) + n=m + end + local str=gsub(f:read(n),"\000","") + return str +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-os']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local os=os +local date,time=os.date,os.time +local find,format,gsub,upper,gmatch=string.find,string.format,string.gsub,string.upper,string.gmatch +local concat=table.concat +local random,ceil,randomseed=math.random,math.ceil,math.randomseed +local rawget,rawset,type,getmetatable,setmetatable,tonumber,tostring=rawget,rawset,type,getmetatable,setmetatable,tonumber,tostring +do + local selfdir=os.selfdir + if selfdir=="" then + selfdir=nil + end + if not selfdir then + if arg then + for i=1,#arg do + local a=arg[i] + if find(a,"^%-%-[c:]*texmfbinpath=") then + selfdir=gsub(a,"^.-=","") + break + end + end + end + if not selfdir then + selfdir=os.selfbin or "luatex" + if find(selfdir,"[/\\]") then + selfdir=gsub(selfdir,"[/\\][^/\\]*$","") + elseif os.getenv then + local path=os.getenv("PATH") + local name=gsub(selfdir,"^.*[/\\][^/\\]","") + local patt="[^:]+" + if os.type=="windows" then + patt="[^;]+" + name=name..".exe" + end + local isfile + if lfs then + local attributes=lfs.attributes + isfile=function(name) + local a=attributes(name,"mode") + return a=="file" or a=="link" or nil + end + else + local open=io.open + isfile=function(name) + local f=open(name) + if f then + f:close() + return true + end + end + end + for p in gmatch(path,patt) do + if isfile(p.."/"..name) then + selfdir=p + break + end + end + end + end + os.selfdir=selfdir or "." + end +end +math.initialseed=tonumber(string.sub(string.reverse(tostring(ceil(socket and socket.gettime()*10000 or time()))),1,6)) +randomseed(math.initialseed) +if not os.__getenv__ then + os.__getenv__=os.getenv + os.__setenv__=os.setenv + if os.env then + local osgetenv=os.getenv + local ossetenv=os.setenv + local osenv=os.env local _=osenv.PATH + function os.setenv(k,v) + if v==nil then + v="" + end + local K=upper(k) + osenv[K]=v + if type(v)=="table" then + v=concat(v,";") + end + ossetenv(K,v) + end + function os.getenv(k) + local K=upper(k) + local v=osenv[K] or osenv[k] or osgetenv(K) or osgetenv(k) + if v=="" then + return nil + else + return v + end + end + else + local ossetenv=os.setenv + local osgetenv=os.getenv + local osenv={} + function os.setenv(k,v) + if v==nil then + v="" + end + local K=upper(k) + osenv[K]=v + end + function os.getenv(k) + local K=upper(k) + local v=osenv[K] or osgetenv(K) or osgetenv(k) + if v=="" then + return nil + else + return v + end + end + local function __index(t,k) + return os.getenv(k) + end + local function __newindex(t,k,v) + os.setenv(k,v) + end + os.env={} + setmetatable(os.env,{ __index=__index,__newindex=__newindex } ) + end +end +local execute=os.execute +local iopopen=io.popen +local function resultof(command) + local handle=iopopen(command,"r") + if handle then + local result=handle:read("*all") or "" + handle:close() + return result + else + return "" + end +end +os.resultof=resultof +function os.pipeto(command) + return iopopen(command,"w") +end +if not io.fileseparator then + if find(os.getenv("PATH"),";",1,true) then + io.fileseparator,io.pathseparator,os.type="\\",";",os.type or "windows" + else + io.fileseparator,io.pathseparator,os.type="/",":",os.type or "unix" + end +end +os.type=os.type or (io.pathseparator==";" and "windows") or "unix" +os.name=os.name or (os.type=="windows" and "mswin" ) or "linux" +if os.type=="windows" then + os.libsuffix,os.binsuffix,os.binsuffixes='dll','exe',{ 'exe','cmd','bat' } +else + os.libsuffix,os.binsuffix,os.binsuffixes='so','',{ '' } +end +local launchers={ + windows="start %s", + macosx="open %s", + unix="xdg-open %s &> /dev/null &", +} +function os.launch(str) + local command=format(launchers[os.name] or launchers.unix,str) + execute(command) +end +local gettimeofday=os.gettimeofday or os.clock +os.gettimeofday=gettimeofday +local startuptime=gettimeofday() +function os.runtime() + return gettimeofday()-startuptime +end +local resolvers=os.resolvers or {} +os.resolvers=resolvers +setmetatable(os,{ __index=function(t,k) + local r=resolvers[k] + return r and r(t,k) or nil +end }) +local name,platform=os.name or "linux",os.getenv("MTX_PLATFORM") or "" +if platform~="" then + os.platform=platform +elseif os.type=="windows" then + function resolvers.platform(t,k) + local architecture=os.getenv("PROCESSOR_ARCHITECTURE") or "" + local platform="" + if find(architecture,"AMD64",1,true) then + platform="win64" + else + platform="mswin" + end + os.setenv("MTX_PLATFORM",platform) + os.platform=platform + return platform + end +elseif name=="linux" then + function resolvers.platform(t,k) + local architecture=os.getenv("HOSTTYPE") or resultof("uname -m") or "" + local platform=os.getenv("MTX_PLATFORM") or "" + local musl=find(os.selfdir or "","linuxmusl") + if platform~="" then + elseif find(architecture,"x86_64",1,true) then + platform=musl and "linuxmusl" or "linux-64" + elseif find(architecture,"ppc",1,true) then + platform="linux-ppc" + else + platform=musl and "linuxmusl" or "linux" + end + os.setenv("MTX_PLATFORM",platform) + os.platform=platform + return platform + end +elseif name=="macosx" then + function resolvers.platform(t,k) + local architecture=resultof("echo $HOSTTYPE") or "" + local platform="" + if architecture=="" then + platform="osx-intel" + elseif find(architecture,"i386",1,true) then + platform="osx-intel" + elseif find(architecture,"x86_64",1,true) then + platform="osx-64" + else + platform="osx-ppc" + end + os.setenv("MTX_PLATFORM",platform) + os.platform=platform + return platform + end +elseif name=="sunos" then + function resolvers.platform(t,k) + local architecture=resultof("uname -m") or "" + local platform="" + if find(architecture,"sparc",1,true) then + platform="solaris-sparc" + else + platform="solaris-intel" + end + os.setenv("MTX_PLATFORM",platform) + os.platform=platform + return platform + end +elseif name=="freebsd" then + function resolvers.platform(t,k) + local architecture=resultof("uname -m") or "" + local platform="" + if find(architecture,"amd64",1,true) then + platform="freebsd-amd64" + else + platform="freebsd" + end + os.setenv("MTX_PLATFORM",platform) + os.platform=platform + return platform + end +elseif name=="kfreebsd" then + function resolvers.platform(t,k) + local architecture=os.getenv("HOSTTYPE") or resultof("uname -m") or "" + local platform="" + if find(architecture,"x86_64",1,true) then + platform="kfreebsd-amd64" + else + platform="kfreebsd-i386" + end + os.setenv("MTX_PLATFORM",platform) + os.platform=platform + return platform + end +else + function resolvers.platform(t,k) + local platform="linux" + os.setenv("MTX_PLATFORM",platform) + os.platform=platform + return platform + end +end +os.newline=name=="windows" and "\013\010" or "\010" +function resolvers.bits(t,k) + local bits=find(os.platform,"64",1,true) and 64 or 32 + os.bits=bits + return bits +end +local t={ 8,9,"a","b" } +function os.uuid() + return format("%04x%04x-4%03x-%s%03x-%04x-%04x%04x%04x", + random(0xFFFF),random(0xFFFF), + random(0x0FFF), + t[ceil(random(4))] or 8,random(0x0FFF), + random(0xFFFF), + random(0xFFFF),random(0xFFFF),random(0xFFFF) + ) +end +local d +function os.timezone(delta) + d=d or tonumber(tonumber(date("%H")-date("!%H"))) + if delta then + if d>0 then + return format("+%02i:00",d) + else + return format("-%02i:00",-d) + end + else + return 1 + end +end +local timeformat=format("%%s%s",os.timezone(true)) +local dateformat="!%Y-%m-%d %H:%M:%S" +local lasttime=nil +local lastdate=nil +function os.fulltime(t,default) + t=t and tonumber(t) or 0 + if t>0 then + elseif default then + return default + else + t=time() + end + if t~=lasttime then + lasttime=t + lastdate=format(timeformat,date(dateformat)) + end + return lastdate +end +local dateformat="%Y-%m-%d %H:%M:%S" +local lasttime=nil +local lastdate=nil +function os.localtime(t,default) + t=t and tonumber(t) or 0 + if t>0 then + elseif default then + return default + else + t=time() + end + if t~=lasttime then + lasttime=t + lastdate=date(dateformat,t) + end + return lastdate +end +function os.converttime(t,default) + local t=tonumber(t) + if t and t>0 then + return date(dateformat,t) + else + return default or "-" + end +end +local memory={} +local function which(filename) + local fullname=memory[filename] + if fullname==nil then + local suffix=file.suffix(filename) + local suffixes=suffix=="" and os.binsuffixes or { suffix } + for directory in gmatch(os.getenv("PATH"),"[^"..io.pathseparator.."]+") do + local df=file.join(directory,filename) + for i=1,#suffixes do + local dfs=file.addsuffix(df,suffixes[i]) + if io.exists(dfs) then + fullname=dfs + break + end + end + end + if not fullname then + fullname=false + end + memory[filename]=fullname + end + return fullname +end +os.which=which +os.where=which +function os.today() + return date("!*t") +end +function os.now() + return date("!%Y-%m-%d %H:%M:%S") +end +if not os.sleep then + local socket=socket + function os.sleep(n) + if not socket then + socket=require("socket") + end + socket.sleep(n) + end +end +local function isleapyear(year) + return (year%4==0) and (year%100~=0 or year%400==0) +end +os.isleapyear=isleapyear +local days={ 31,28,31,30,31,30,31,31,30,31,30,31 } +local function nofdays(year,month) + if not month then + return isleapyear(year) and 365 or 364 + else + return month==2 and isleapyear(year) and 29 or days[month] + end +end +os.nofdays=nofdays +function os.weekday(day,month,year) + return date("%w",time { year=year,month=month,day=day })+1 +end +function os.validdate(year,month,day) + if month<1 then + month=1 + elseif month>12 then + month=12 + end + if day<1 then + day=1 + else + local max=nofdays(year,month) + if day>max then + day=max + end + end + return year,month,day +end +local osexit=os.exit +local exitcode=nil +function os.setexitcode(code) + exitcode=code +end +function os.exit(c) + if exitcode~=nil then + return osexit(exitcode) + end + if c~=nil then + return osexit(c) + end + return osexit() +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-file']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +file=file or {} +local file=file +if not lfs then + lfs=optionalrequire("lfs") +end +local insert,concat=table.insert,table.concat +local match,find,gmatch=string.match,string.find,string.gmatch +local lpegmatch=lpeg.match +local getcurrentdir,attributes=lfs.currentdir,lfs.attributes +local checkedsplit=string.checkedsplit +local P,R,S,C,Cs,Cp,Cc,Ct=lpeg.P,lpeg.R,lpeg.S,lpeg.C,lpeg.Cs,lpeg.Cp,lpeg.Cc,lpeg.Ct +local attributes=lfs.attributes +function lfs.isdir(name) + return attributes(name,"mode")=="directory" +end +function lfs.isfile(name) + local a=attributes(name,"mode") + return a=="file" or a=="link" or nil +end +function lfs.isfound(name) + local a=attributes(name,"mode") + return (a=="file" or a=="link") and name or nil +end +if sandbox then + sandbox.redefine(lfs.isfile,"lfs.isfile") + sandbox.redefine(lfs.isdir,"lfs.isdir") + sandbox.redefine(lfs.isfound,"lfs.isfound") +end +local colon=P(":") +local period=P(".") +local periods=P("..") +local fwslash=P("/") +local bwslash=P("\\") +local slashes=S("\\/") +local noperiod=1-period +local noslashes=1-slashes +local name=noperiod^1 +local suffix=period/""*(1-period-slashes)^1*-1 +local pattern=C((1-(slashes^1*noslashes^1*-1))^1)*P(1) +local function pathpart(name,default) + return name and lpegmatch(pattern,name) or default or "" +end +local pattern=(noslashes^0*slashes)^1*C(noslashes^1)*-1 +local function basename(name) + return name and lpegmatch(pattern,name) or name +end +local pattern=(noslashes^0*slashes^1)^0*Cs((1-suffix)^1)*suffix^0 +local function nameonly(name) + return name and lpegmatch(pattern,name) or name +end +local pattern=(noslashes^0*slashes)^0*(noperiod^1*period)^1*C(noperiod^1)*-1 +local function suffixonly(name) + return name and lpegmatch(pattern,name) or "" +end +local pattern=(noslashes^0*slashes)^0*noperiod^1*((period*C(noperiod^1))^1)*-1+Cc("") +local function suffixesonly(name) + if name then + return lpegmatch(pattern,name) + else + return "" + end +end +file.pathpart=pathpart +file.basename=basename +file.nameonly=nameonly +file.suffixonly=suffixonly +file.suffix=suffixonly +file.suffixesonly=suffixesonly +file.suffixes=suffixesonly +file.dirname=pathpart +file.extname=suffixonly +local drive=C(R("az","AZ"))*colon +local path=C((noslashes^0*slashes)^0) +local suffix=period*C(P(1-period)^0*P(-1)) +local base=C((1-suffix)^0) +local rest=C(P(1)^0) +drive=drive+Cc("") +path=path+Cc("") +base=base+Cc("") +suffix=suffix+Cc("") +local pattern_a=drive*path*base*suffix +local pattern_b=path*base*suffix +local pattern_c=C(drive*path)*C(base*suffix) +local pattern_d=path*rest +function file.splitname(str,splitdrive) + if not str then + elseif splitdrive then + return lpegmatch(pattern_a,str) + else + return lpegmatch(pattern_b,str) + end +end +function file.splitbase(str) + if str then + return lpegmatch(pattern_d,str) + else + return "",str + end +end +function file.nametotable(str,splitdrive) + if str then + local path,drive,subpath,name,base,suffix=lpegmatch(pattern_c,str) + if splitdrive then + return { + path=path, + drive=drive, + subpath=subpath, + name=name, + base=base, + suffix=suffix, + } + else + return { + path=path, + name=name, + base=base, + suffix=suffix, + } + end + end +end +local pattern=Cs(((period*(1-period-slashes)^1*-1)/""+1)^1) +function file.removesuffix(name) + return name and lpegmatch(pattern,name) +end +local suffix=period/""*(1-period-slashes)^1*-1 +local pattern=Cs((noslashes^0*slashes^1)^0*((1-suffix)^1))*Cs(suffix) +function file.addsuffix(filename,suffix,criterium) + if not filename or not suffix or suffix=="" then + return filename + elseif criterium==true then + return filename.."."..suffix + elseif not criterium then + local n,s=lpegmatch(pattern,filename) + if not s or s=="" then + return filename.."."..suffix + else + return filename + end + else + local n,s=lpegmatch(pattern,filename) + if s and s~="" then + local t=type(criterium) + if t=="table" then + for i=1,#criterium do + if s==criterium[i] then + return filename + end + end + elseif t=="string" then + if s==criterium then + return filename + end + end + end + return (n or filename).."."..suffix + end +end +local suffix=period*(1-period-slashes)^1*-1 +local pattern=Cs((1-suffix)^0) +function file.replacesuffix(name,suffix) + if name and suffix and suffix~="" then + return lpegmatch(pattern,name).."."..suffix + else + return name + end +end +local reslasher=lpeg.replacer(P("\\"),"/") +function file.reslash(str) + return str and lpegmatch(reslasher,str) +end +if lfs.isreadablefile and lfs.iswritablefile then + file.is_readable=lfs.isreadablefile + file.is_writable=lfs.iswritablefile +else + function file.is_writable(name) + if not name then + elseif lfs.isdir(name) then + name=name.."/m_t_x_t_e_s_t.tmp" + local f=io.open(name,"wb") + if f then + f:close() + os.remove(name) + return true + end + elseif lfs.isfile(name) then + local f=io.open(name,"ab") + if f then + f:close() + return true + end + else + local f=io.open(name,"ab") + if f then + f:close() + os.remove(name) + return true + end + end + return false + end + local readable=P("r")*Cc(true) + function file.is_readable(name) + if name then + local a=attributes(name) + return a and lpegmatch(readable,a.permissions) or false + else + return false + end + end +end +file.isreadable=file.is_readable +file.iswritable=file.is_writable +function file.size(name) + if name then + local a=attributes(name) + return a and a.size or 0 + else + return 0 + end +end +function file.splitpath(str,separator) + return str and checkedsplit(lpegmatch(reslasher,str),separator or io.pathseparator) +end +function file.joinpath(tab,separator) + return tab and concat(tab,separator or io.pathseparator) +end +local someslash=S("\\/") +local stripper=Cs(P(fwslash)^0/""*reslasher) +local isnetwork=someslash*someslash*(1-someslash)+(1-fwslash-colon)^1*colon +local isroot=fwslash^1*-1 +local hasroot=fwslash^1 +local reslasher=lpeg.replacer(S("\\/"),"/") +local deslasher=lpeg.replacer(S("\\/")^1,"/") +function file.join(one,two,three,...) + if not two then + return one=="" and one or lpegmatch(reslasher,one) + end + if one=="" then + return lpegmatch(stripper,three and concat({ two,three,... },"/") or two) + end + if lpegmatch(isnetwork,one) then + local one=lpegmatch(reslasher,one) + local two=lpegmatch(deslasher,three and concat({ two,three,... },"/") or two) + if lpegmatch(hasroot,two) then + return one..two + else + return one.."/"..two + end + elseif lpegmatch(isroot,one) then + local two=lpegmatch(deslasher,three and concat({ two,three,... },"/") or two) + if lpegmatch(hasroot,two) then + return two + else + return "/"..two + end + else + return lpegmatch(deslasher,concat({ one,two,three,... },"/")) + end +end +local drivespec=R("az","AZ")^1*colon +local anchors=fwslash+drivespec +local untouched=periods+(1-period)^1*P(-1) +local mswindrive=Cs(drivespec*(bwslash/"/"+fwslash)^0) +local mswinuncpath=(bwslash+fwslash)*(bwslash+fwslash)*Cc("//") +local splitstarter=(mswindrive+mswinuncpath+Cc(false))*Ct(lpeg.splitat(S("/\\")^1)) +local absolute=fwslash +function file.collapsepath(str,anchor) + if not str then + return + end + if anchor==true and not lpegmatch(anchors,str) then + str=getcurrentdir().."/"..str + end + if str=="" or str=="." then + return "." + elseif lpegmatch(untouched,str) then + return lpegmatch(reslasher,str) + end + local starter,oldelements=lpegmatch(splitstarter,str) + local newelements={} + local i=#oldelements + while i>0 do + local element=oldelements[i] + if element=='.' then + elseif element=='..' then + local n=i-1 + while n>0 do + local element=oldelements[n] + if element~='..' and element~='.' then + oldelements[n]='.' + break + else + n=n-1 + end + end + if n<1 then + insert(newelements,1,'..') + end + elseif element~="" then + insert(newelements,1,element) + end + i=i-1 + end + if #newelements==0 then + return starter or "." + elseif starter then + return starter..concat(newelements,'/') + elseif lpegmatch(absolute,str) then + return "/"..concat(newelements,'/') + else + newelements=concat(newelements,'/') + if anchor=="." and find(str,"^%./") then + return "./"..newelements + else + return newelements + end + end +end +local validchars=R("az","09","AZ","--","..") +local pattern_a=lpeg.replacer(1-validchars) +local pattern_a=Cs((validchars+P(1)/"-")^1) +local whatever=P("-")^0/"" +local pattern_b=Cs(whatever*(1-whatever*-1)^1) +function file.robustname(str,strict) + if str then + str=lpegmatch(pattern_a,str) or str + if strict then + return lpegmatch(pattern_b,str) or str + else + return str + end + end +end +local loaddata=io.loaddata +local savedata=io.savedata +file.readdata=loaddata +file.savedata=savedata +function file.copy(oldname,newname) + if oldname and newname then + local data=loaddata(oldname) + if data and data~="" then + savedata(newname,data) + end + end +end +local letter=R("az","AZ")+S("_-+") +local separator=P("://") +local qualified=period^0*fwslash+letter*colon+letter^1*separator+letter^1*fwslash +local rootbased=fwslash+letter*colon +lpeg.patterns.qualified=qualified +lpeg.patterns.rootbased=rootbased +function file.is_qualified_path(filename) + return filename and lpegmatch(qualified,filename)~=nil +end +function file.is_rootbased_path(filename) + return filename and lpegmatch(rootbased,filename)~=nil +end +function file.strip(name,dir) + if name then + local b,a=match(name,"^(.-)"..dir.."(.*)$") + return a~="" and a or name + end +end +function lfs.mkdirs(path) + local full="" + for sub in gmatch(path,"(/*[^\\/]+)") do + full=full..sub + lfs.mkdir(full) + end +end +function file.withinbase(path) + local l=0 + if not find(path,"^/") then + path="/"..path + end + for dir in gmatch(path,"/([^/]+)") do + if dir==".." then + l=l-1 + elseif dir~="." then + l=l+1 + end + if l<0 then + return false + end + end + return true +end +local symlinkattributes=lfs.symlinkattributes +function lfs.readlink(name) + return symlinkattributes(name,"target") or nil +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-gzip']={ + version=1.001, + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +if not gzip then + return +end +local suffix,suffixes=file.suffix,file.suffixes +function gzip.load(filename) + local f=io.open(filename,"rb") + if not f then + elseif suffix(filename)=="gz" then + f:close() + local g=gzip.open(filename,"rb") + if g then + local str=g:read("*all") + g:close() + return str + end + else + local str=f:read("*all") + f:close() + return str + end +end +function gzip.save(filename,data) + if suffix(filename)~="gz" then + filename=filename..".gz" + end + local f=io.open(filename,"wb") + if f then + local s=zlib.compress(data or "",9,nil,15+16) + f:write(s) + f:close() + return #s + end +end +function gzip.suffix(filename) + local suffix,extra=suffixes(filename) + local gzipped=extra=="gz" + return suffix,gzipped +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-md5']={ + version=1.001, + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +if not md5 then + md5=optionalrequire("md5") +end +if not md5 then + md5={ + sum=function(str) print("error: md5 is not loaded (sum ignored)") return str end, + sumhexa=function(str) print("error: md5 is not loaded (sumhexa ignored)") return str end, + } +end +local md5,file=md5,file +local gsub=string.gsub +do + local patterns=lpeg and lpeg.patterns + if patterns then + local bytestoHEX=patterns.bytestoHEX + local bytestohex=patterns.bytestohex + local bytestodec=patterns.bytestodec + local lpegmatch=lpeg.match + local md5sum=md5.sum + if not md5.HEX then function md5.HEX(str) if str then return lpegmatch(bytestoHEX,md5sum(str)) end end end + if not md5.hex then function md5.hex(str) if str then return lpegmatch(bytestohex,md5sum(str)) end end end + if not md5.dec then function md5.dec(str) if str then return lpegmatch(bytestodec,md5sum(str)) end end end + md5.sumhexa=md5.hex + md5.sumHEXA=md5.HEX + end +end +function file.needsupdating(oldname,newname,threshold) + local oldtime=lfs.attributes(oldname,"modification") + if oldtime then + local newtime=lfs.attributes(newname,"modification") + if not newtime then + return true + elseif newtime>=oldtime then + return false + elseif oldtime-newtime<(threshold or 1) then + return false + else + return true + end + else + return false + end +end +file.needs_updating=file.needsupdating +function file.syncmtimes(oldname,newname) + local oldtime=lfs.attributes(oldname,"modification") + if oldtime and lfs.isfile(newname) then + lfs.touch(newname,oldtime,oldtime) + end +end +function file.checksum(name) + if md5 then + local data=io.loaddata(name) + if data then + return md5.HEX(data) + end + end + return nil +end +function file.loadchecksum(name) + if md5 then + local data=io.loaddata(name..".md5") + return data and (gsub(data,"%s","")) + end + return nil +end +function file.savechecksum(name,checksum) + if not checksum then checksum=file.checksum(name) end + if checksum then + io.savedata(name..".md5",checksum) + return checksum + end + return nil +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-dir']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local type,select=type,select +local find,gmatch,match,gsub,sub=string.find,string.gmatch,string.match,string.gsub,string.sub +local concat,insert,remove,unpack=table.concat,table.insert,table.remove,table.unpack +local lpegmatch=lpeg.match +local P,S,R,C,Cc,Cs,Ct,Cv,V=lpeg.P,lpeg.S,lpeg.R,lpeg.C,lpeg.Cc,lpeg.Cs,lpeg.Ct,lpeg.Cv,lpeg.V +dir=dir or {} +local dir=dir +local lfs=lfs +local attributes=lfs.attributes +local walkdir=lfs.dir +local isdir=lfs.isdir +local isfile=lfs.isfile +local currentdir=lfs.currentdir +local chdir=lfs.chdir +local mkdir=lfs.mkdir +local onwindows=os.type=="windows" or find(os.getenv("PATH"),";",1,true) +if onwindows then + local tricky=S("/\\")*P(-1) + isdir=function(name) + if lpegmatch(tricky,name) then + return attributes(name,"mode")=="directory" + else + return attributes(name.."/.","mode")=="directory" + end + end + isfile=function(name) + return attributes(name,"mode")=="file" + end + lfs.isdir=isdir + lfs.isfile=isfile +else + isdir=function(name) + return attributes(name,"mode")=="directory" + end + isfile=function(name) + return attributes(name,"mode")=="file" + end + lfs.isdir=isdir + lfs.isfile=isfile +end +function dir.current() + return (gsub(currentdir(),"\\","/")) +end +local function glob_pattern_function(path,patt,recurse,action) + if isdir(path) then + local usedpath + if path=="/" then + usedpath="/." + elseif not find(path,"/$") then + usedpath=path.."/." + path=path.."/" + else + usedpath=path + end + local dirs + local nofdirs=0 + for name,mode,size,time in walkdir(usedpath) do + if name~="." and name~=".." then + local full=path..name + if mode==nil then + mode=attributes(full,'mode') + end + if mode=='file' then + if not patt or find(full,patt) then + action(full,size,time) + end + elseif recurse and mode=="directory" then + if dirs then + nofdirs=nofdirs+1 + dirs[nofdirs]=full + else + nofdirs=1 + dirs={ full } + end + end + end + end + if dirs then + for i=1,nofdirs do + glob_pattern_function(dirs[i],patt,recurse,action) + end + end + end +end +local function glob_pattern_table(path,patt,recurse,result) + if not result then + result={} + end + local usedpath + if path=="/" then + usedpath="/." + elseif not find(path,"/$") then + usedpath=path.."/." + path=path.."/" + else + usedpath=path + end + local dirs + local nofdirs=0 + local noffiles=#result + for name,mode in walkdir(usedpath) do + if name~="." and name~=".." then + local full=path..name + if mode==nil then + mode=attributes(full,'mode') + end + if mode=='file' then + if not patt or find(full,patt) then + noffiles=noffiles+1 + result[noffiles]=full + end + elseif recurse and mode=="directory" then + if dirs then + nofdirs=nofdirs+1 + dirs[nofdirs]=full + else + nofdirs=1 + dirs={ full } + end + end + end + end + if dirs then + for i=1,nofdirs do + glob_pattern_table(dirs[i],patt,recurse,result) + end + end + return result +end +local function globpattern(path,patt,recurse,method) + local kind=type(method) + if patt and sub(patt,1,-3)==path then + patt=false + end + local okay=isdir(path) + if kind=="function" then + return okay and glob_pattern_function(path,patt,recurse,method) or {} + elseif kind=="table" then + return okay and glob_pattern_table(path,patt,recurse,method) or method + else + return okay and glob_pattern_table(path,patt,recurse,{}) or {} + end +end +dir.globpattern=globpattern +local function collectpattern(path,patt,recurse,result) + local ok,scanner + result=result or {} + if path=="/" then + ok,scanner,first=xpcall(function() return walkdir(path..".") end,function() end) + else + ok,scanner,first=xpcall(function() return walkdir(path) end,function() end) + end + if ok and type(scanner)=="function" then + if not find(path,"/$") then + path=path..'/' + end + for name in scanner,first do + if name=="." then + elseif name==".." then + else + local full=path..name + local attr=attributes(full) + local mode=attr.mode + if mode=='file' then + if find(full,patt) then + result[name]=attr + end + elseif recurse and mode=="directory" then + attr.list=collectpattern(full,patt,recurse) + result[name]=attr + end + end + end + end + return result +end +dir.collectpattern=collectpattern +local separator,pattern +if onwindows then + local slash=S("/\\")/"/" + pattern={ + [1]=(Cs(P(".")+slash^1)+Cs(R("az","AZ")*P(":")*slash^0)+Cc("./"))*V(2)*V(3), + [2]=Cs(((1-S("*?/\\"))^0*slash)^0), + [3]=Cs(P(1)^0) + } +else + pattern={ + [1]=(C(P(".")+P("/")^1)+Cc("./"))*V(2)*V(3), + [2]=C(((1-S("*?/"))^0*P("/"))^0), + [3]=C(P(1)^0) + } +end +local filter=Cs (( + P("**")/".*"+P("*")/"[^/]*"+P("?")/"[^/]"+P(".")/"%%."+P("+")/"%%+"+P("-")/"%%-"+P(1) +)^0 ) +local function glob(str,t) + if type(t)=="function" then + if type(str)=="table" then + for s=1,#str do + glob(str[s],t) + end + elseif isfile(str) then + t(str) + else + local root,path,base=lpegmatch(pattern,str) + if root and path and base then + local recurse=find(base,"**",1,true) + local start=root..path + local result=lpegmatch(filter,start..base) + globpattern(start,result,recurse,t) + end + end + else + if type(str)=="table" then + local t=t or {} + for s=1,#str do + glob(str[s],t) + end + return t + elseif isfile(str) then + if t then + t[#t+1]=str + return t + else + return { str } + end + else + local root,path,base=lpegmatch(pattern,str) + if root and path and base then + local recurse=find(base,"**",1,true) + local start=root..path + local result=lpegmatch(filter,start..base) + return globpattern(start,result,recurse,t) + else + return {} + end + end + end +end +dir.glob=glob +local function globfiles(path,recurse,func,files) + if type(func)=="string" then + local s=func + func=function(name) return find(name,s) end + end + files=files or {} + local noffiles=#files + for name,mode in walkdir(path) do + if find(name,"^%.") then + else + if mode==nil then + mode=attributes(name,'mode') + end + if mode=="directory" then + if recurse then + globfiles(path.."/"..name,recurse,func,files) + end + elseif mode=="file" then + if not func or func(name) then + noffiles=noffiles+1 + files[noffiles]=path.."/"..name + end + end + end + end + return files +end +dir.globfiles=globfiles +local function globdirs(path,recurse,func,files) + if type(func)=="string" then + local s=func + func=function(name) return find(name,s) end + end + files=files or {} + local noffiles=#files + for name,mode in walkdir(path) do + if find(name,"^%.") then + else + if mode==nil then + mode=attributes(name,'mode') + end + if mode=="directory" then + if not func or func(name) then + noffiles=noffiles+1 + files[noffiles]=path.."/"..name + if recurse then + globdirs(path.."/"..name,recurse,func,files) + end + end + end + end + end + return files +end +dir.globdirs=globdirs +function dir.ls(pattern) + return concat(glob(pattern),"\n") +end +local make_indeed=true +if onwindows then + function dir.mkdirs(...) + local n=select("#",...) + local str + if n==1 then + str=select(1,...) + if isdir(str) then + return str,true + end + else + str="" + for i=1,n do + local s=select(i,...) + if s=="" then + elseif str=="" then + str=s + else + str=str.."/"..s + end + end + end + local pth="" + local drive=false + local first,middle,last=match(str,"^(//)(//*)(.*)$") + if first then + else + first,last=match(str,"^(//)/*(.-)$") + if first then + middle,last=match(str,"([^/]+)/+(.-)$") + if middle then + pth="//"..middle + else + pth="//"..last + last="" + end + else + first,middle,last=match(str,"^([a-zA-Z]:)(/*)(.-)$") + if first then + pth,drive=first..middle,true + else + middle,last=match(str,"^(/*)(.-)$") + if not middle then + last=str + end + end + end + end + for s in gmatch(last,"[^/]+") do + if pth=="" then + pth=s + elseif drive then + pth,drive=pth..s,false + else + pth=pth.."/"..s + end + if make_indeed and not isdir(pth) then + mkdir(pth) + end + end + return pth,(isdir(pth)==true) + end +else + function dir.mkdirs(...) + local n=select("#",...) + local str,pth + if n==1 then + str=select(1,...) + if isdir(str) then + return str,true + end + else + str="" + for i=1,n do + local s=select(i,...) + if s and s~="" then + if str~="" then + str=str.."/"..s + else + str=s + end + end + end + end + str=gsub(str,"/+","/") + if find(str,"^/") then + pth="/" + for s in gmatch(str,"[^/]+") do + local first=(pth=="/") + if first then + pth=pth..s + else + pth=pth.."/"..s + end + if make_indeed and not first and not isdir(pth) then + mkdir(pth) + end + end + else + pth="." + for s in gmatch(str,"[^/]+") do + pth=pth.."/"..s + if make_indeed and not isdir(pth) then + mkdir(pth) + end + end + end + return pth,(isdir(pth)==true) + end +end +dir.makedirs=dir.mkdirs +do + local chdir=sandbox and sandbox.original(chdir) or chdir + if onwindows then + local xcurrentdir=dir.current + function dir.expandname(str) + local first,nothing,last=match(str,"^(//)(//*)(.*)$") + if first then + first=xcurrentdir().."/" + end + if not first then + first,last=match(str,"^(//)/*(.*)$") + end + if not first then + first,last=match(str,"^([a-zA-Z]:)(.*)$") + if first and not find(last,"^/") then + local d=currentdir() + if chdir(first) then + first=xcurrentdir() + end + chdir(d) + end + end + if not first then + first,last=xcurrentdir(),str + end + last=gsub(last,"//","/") + last=gsub(last,"/%./","/") + last=gsub(last,"^/*","") + first=gsub(first,"/*$","") + if last=="" or last=="." then + return first + else + return first.."/"..last + end + end + else + function dir.expandname(str) + if not find(str,"^/") then + str=currentdir().."/"..str + end + str=gsub(str,"//","/") + str=gsub(str,"/%./","/") + str=gsub(str,"(.)/%.$","%1") + return str + end + end +end +file.expandname=dir.expandname +local stack={} +function dir.push(newdir) + local curdir=currentdir() + insert(stack,curdir) + if newdir and newdir~="" and chdir(newdir) then + return newdir + else + return curdir + end +end +function dir.pop() + local d=remove(stack) + if d then + chdir(d) + end + return d +end +local function found(...) + for i=1,select("#",...) do + local path=select(i,...) + local kind=type(path) + if kind=="string" then + if isdir(path) then + return path + end + elseif kind=="table" then + local path=found(unpack(path)) + if path then + return path + end + end + end +end +dir.found=found + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-unicode']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +utf=utf or {} +if not string.utfcharacters then + local gmatch=string.gmatch + function string.characters(str) + return gmatch(str,".[\128-\191]*") + end +end +utf.characters=string.utfcharacters +local type=type +local char,byte,format,sub,gmatch=string.char,string.byte,string.format,string.sub,string.gmatch +local concat=table.concat +local P,C,R,Cs,Ct,Cmt,Cc,Carg,Cp=lpeg.P,lpeg.C,lpeg.R,lpeg.Cs,lpeg.Ct,lpeg.Cmt,lpeg.Cc,lpeg.Carg,lpeg.Cp +local lpegmatch=lpeg.match +local patterns=lpeg.patterns +local tabletopattern=lpeg.utfchartabletopattern +local bytepairs=string.bytepairs +local finder=lpeg.finder +local replacer=lpeg.replacer +local p_utftype=patterns.utftype +local p_utfstricttype=patterns.utfstricttype +local p_utfoffset=patterns.utfoffset +local p_utf8character=patterns.utf8character +local p_utf8char=patterns.utf8char +local p_utf8byte=patterns.utf8byte +local p_utfbom=patterns.utfbom +local p_newline=patterns.newline +local p_whitespace=patterns.whitespace +if not utf.char then + utf.char=string.utfcharacter or (utf8 and utf8.char) + if not utf.char then + local char=string.char + if bit32 then + local rshift=bit32.rshift + function utf.char(n) + if n<0x80 then + return char(n) + elseif n<0x800 then + return char( + 0xC0+rshift(n,6), + 0x80+(n%0x40) + ) + elseif n<0x10000 then + return char( + 0xE0+rshift(n,12), + 0x80+(rshift(n,6)%0x40), + 0x80+(n%0x40) + ) + elseif n<0x200000 then + return char( + 0xF0+rshift(n,18), + 0x80+(rshift(n,12)%0x40), + 0x80+(rshift(n,6)%0x40), + 0x80+(n%0x40) + ) + else + return "" + end + end + else + local floor=math.floor + function utf.char(n) + if n<0x80 then + return char(n) + elseif n<0x800 then + return char( + 0xC0+floor(n/0x40), + 0x80+(n%0x40) + ) + elseif n<0x10000 then + return char( + 0xE0+floor(n/0x1000), + 0x80+(floor(n/0x40)%0x40), + 0x80+(n%0x40) + ) + elseif n<0x200000 then + return char( + 0xF0+floor(n/0x40000), + 0x80+(floor(n/0x1000)%0x40), + 0x80+(floor(n/0x40)%0x40), + 0x80+(n%0x40) + ) + else + return "" + end + end + end + end +end +if not utf.byte then + utf.byte=string.utfvalue or (utf8 and utf8.codepoint) + if not utf.byte then + function utf.byte(c) + return lpegmatch(p_utf8byte,c) + end + end +end +local utfchar,utfbyte=utf.char,utf.byte +function utf.filetype(data) + return data and lpegmatch(p_utftype,data) or "unknown" +end +local toentities=Cs ( + ( + patterns.utf8one+( + patterns.utf8two+patterns.utf8three+patterns.utf8four + )/function(s) local b=utfbyte(s) if b<127 then return s else return format("&#%X;",b) end end + )^0 +) +patterns.toentities=toentities +function utf.toentities(str) + return lpegmatch(toentities,str) +end +local one=P(1) +local two=C(1)*C(1) +local four=C(R(utfchar(0xD8),utfchar(0xFF)))*C(1)*C(1)*C(1) +local pattern=P("\254\255")*Cs(( + four/function(a,b,c,d) + local ab=0xFF*byte(a)+byte(b) + local cd=0xFF*byte(c)+byte(d) + return utfchar((ab-0xD800)*0x400+(cd-0xDC00)+0x10000) + end+two/function(a,b) + return utfchar(byte(a)*256+byte(b)) + end+one + )^1 )+P("\255\254")*Cs(( + four/function(b,a,d,c) + local ab=0xFF*byte(a)+byte(b) + local cd=0xFF*byte(c)+byte(d) + return utfchar((ab-0xD800)*0x400+(cd-0xDC00)+0x10000) + end+two/function(b,a) + return utfchar(byte(a)*256+byte(b)) + end+one + )^1 ) +function string.toutf(s) + return lpegmatch(pattern,s) or s +end +local validatedutf=Cs ( + ( + patterns.utf8one+patterns.utf8two+patterns.utf8three+patterns.utf8four+P(1)/"�" + )^0 +) +patterns.validatedutf=validatedutf +function utf.is_valid(str) + return type(str)=="string" and lpegmatch(validatedutf,str) or false +end +if not utf.len then + utf.len=string.utflength or (utf8 and utf8.len) + if not utf.len then + local n,f=0,1 + local utfcharcounter=patterns.utfbom^-1*Cmt ( + Cc(1)*patterns.utf8one^1+Cc(2)*patterns.utf8two^1+Cc(3)*patterns.utf8three^1+Cc(4)*patterns.utf8four^1, + function(_,t,d) + n=n+(t-f)/d + f=t + return true + end + )^0 + function utf.len(str) + n,f=0,1 + lpegmatch(utfcharcounter,str or "") + return n + end + end +end +utf.length=utf.len +if not utf.sub then + local utflength=utf.length + local b,e,n,first,last=0,0,0,0,0 + local function slide_zero(s,p) + n=n+1 + if n>=last then + e=p-1 + else + return p + end + end + local function slide_one(s,p) + n=n+1 + if n==first then + b=p + end + if n>=last then + e=p-1 + else + return p + end + end + local function slide_two(s,p) + n=n+1 + if n==first then + b=p + else + return true + end + end + local pattern_zero=Cmt(p_utf8character,slide_zero)^0 + local pattern_one=Cmt(p_utf8character,slide_one )^0 + local pattern_two=Cmt(p_utf8character,slide_two )^0 + local pattern_first=C(p_utf8character) + function utf.sub(str,start,stop) + if not start then + return str + end + if start==0 then + start=1 + end + if not stop then + if start<0 then + local l=utflength(str) + start=l+start + else + start=start-1 + end + b,n,first=0,0,start + lpegmatch(pattern_two,str) + if n>=first then + return sub(str,b) + else + return "" + end + end + if start<0 or stop<0 then + local l=utf.length(str) + if start<0 then + start=l+start + if start<=0 then + start=1 + else + start=start+1 + end + end + if stop<0 then + stop=l+stop + if stop==0 then + stop=1 + else + stop=stop+1 + end + end + end + if start==1 and stop==1 then + return lpegmatch(pattern_first,str) or "" + elseif start>stop then + return "" + elseif start>1 then + b,e,n,first,last=0,0,0,start-1,stop + lpegmatch(pattern_one,str) + if n>=first and e==0 then + e=#str + end + return sub(str,b,e) + else + b,e,n,last=1,0,0,stop + lpegmatch(pattern_zero,str) + if e==0 then + e=#str + end + return sub(str,b,e) + end + end +end +function utf.remapper(mapping,option,action) + local variant=type(mapping) + if variant=="table" then + action=action or mapping + if option=="dynamic" then + local pattern=false + table.setmetatablenewindex(mapping,function(t,k,v) rawset(t,k,v) pattern=false end) + return function(str) + if not str or str=="" then + return "" + else + if not pattern then + pattern=Cs((tabletopattern(mapping)/action+p_utf8character)^0) + end + return lpegmatch(pattern,str) + end + end + elseif option=="pattern" then + return Cs((tabletopattern(mapping)/action+p_utf8character)^0) + else + local pattern=Cs((tabletopattern(mapping)/action+p_utf8character)^0) + return function(str) + if not str or str=="" then + return "" + else + return lpegmatch(pattern,str) + end + end,pattern + end + elseif variant=="function" then + if option=="pattern" then + return Cs((p_utf8character/mapping+p_utf8character)^0) + else + local pattern=Cs((p_utf8character/mapping+p_utf8character)^0) + return function(str) + if not str or str=="" then + return "" + else + return lpegmatch(pattern,str) + end + end,pattern + end + else + return function(str) + return str or "" + end + end +end +function utf.replacer(t) + local r=replacer(t,false,false,true) + return function(str) + return lpegmatch(r,str) + end +end +function utf.subtituter(t) + local f=finder (t) + local r=replacer(t,false,false,true) + return function(str) + local i=lpegmatch(f,str) + if not i then + return str + elseif i>#str then + return str + else + return lpegmatch(r,str) + end + end +end +local utflinesplitter=p_utfbom^-1*lpeg.tsplitat(p_newline) +local utfcharsplitter_ows=p_utfbom^-1*Ct(C(p_utf8character)^0) +local utfcharsplitter_iws=p_utfbom^-1*Ct((p_whitespace^1+C(p_utf8character))^0) +local utfcharsplitter_raw=Ct(C(p_utf8character)^0) +patterns.utflinesplitter=utflinesplitter +function utf.splitlines(str) + return lpegmatch(utflinesplitter,str or "") +end +function utf.split(str,ignorewhitespace) + if ignorewhitespace then + return lpegmatch(utfcharsplitter_iws,str or "") + else + return lpegmatch(utfcharsplitter_ows,str or "") + end +end +function utf.totable(str) + return lpegmatch(utfcharsplitter_raw,str) +end +function utf.magic(f) + local str=f:read(4) or "" + local off=lpegmatch(p_utfoffset,str) + if off<4 then + f:seek('set',off) + end + return lpegmatch(p_utftype,str) +end +local utf16_to_utf8_be,utf16_to_utf8_le +local utf32_to_utf8_be,utf32_to_utf8_le +local utf_16_be_getbom=patterns.utfbom_16_be^-1 +local utf_16_le_getbom=patterns.utfbom_16_le^-1 +local utf_32_be_getbom=patterns.utfbom_32_be^-1 +local utf_32_le_getbom=patterns.utfbom_32_le^-1 +local utf_16_be_linesplitter=utf_16_be_getbom*lpeg.tsplitat(patterns.utf_16_be_nl) +local utf_16_le_linesplitter=utf_16_le_getbom*lpeg.tsplitat(patterns.utf_16_le_nl) +local utf_32_be_linesplitter=utf_32_be_getbom*lpeg.tsplitat(patterns.utf_32_be_nl) +local utf_32_le_linesplitter=utf_32_le_getbom*lpeg.tsplitat(patterns.utf_32_le_nl) +local more=0 +local p_utf16_to_utf8_be=C(1)*C(1)/function(left,right) + local now=256*byte(left)+byte(right) + if more>0 then + now=(more-0xD800)*0x400+(now-0xDC00)+0x10000 + more=0 + return utfchar(now) + elseif now>=0xD800 and now<=0xDBFF then + more=now + return "" + else + return utfchar(now) + end +end +local p_utf16_to_utf8_le=C(1)*C(1)/function(right,left) + local now=256*byte(left)+byte(right) + if more>0 then + now=(more-0xD800)*0x400+(now-0xDC00)+0x10000 + more=0 + return utfchar(now) + elseif now>=0xD800 and now<=0xDBFF then + more=now + return "" + else + return utfchar(now) + end +end +local p_utf32_to_utf8_be=C(1)*C(1)*C(1)*C(1)/function(a,b,c,d) + return utfchar(256*256*256*byte(a)+256*256*byte(b)+256*byte(c)+byte(d)) +end +local p_utf32_to_utf8_le=C(1)*C(1)*C(1)*C(1)/function(a,b,c,d) + return utfchar(256*256*256*byte(d)+256*256*byte(c)+256*byte(b)+byte(a)) +end +p_utf16_to_utf8_be=P(true)/function() more=0 end*utf_16_be_getbom*Cs(p_utf16_to_utf8_be^0) +p_utf16_to_utf8_le=P(true)/function() more=0 end*utf_16_le_getbom*Cs(p_utf16_to_utf8_le^0) +p_utf32_to_utf8_be=P(true)/function() more=0 end*utf_32_be_getbom*Cs(p_utf32_to_utf8_be^0) +p_utf32_to_utf8_le=P(true)/function() more=0 end*utf_32_le_getbom*Cs(p_utf32_to_utf8_le^0) +patterns.utf16_to_utf8_be=p_utf16_to_utf8_be +patterns.utf16_to_utf8_le=p_utf16_to_utf8_le +patterns.utf32_to_utf8_be=p_utf32_to_utf8_be +patterns.utf32_to_utf8_le=p_utf32_to_utf8_le +utf16_to_utf8_be=function(s) + if s and s~="" then + return lpegmatch(p_utf16_to_utf8_be,s) + else + return s + end +end +local utf16_to_utf8_be_t=function(t) + if not t then + return nil + elseif type(t)=="string" then + t=lpegmatch(utf_16_be_linesplitter,t) + end + for i=1,#t do + local s=t[i] + if s~="" then + t[i]=lpegmatch(p_utf16_to_utf8_be,s) + end + end + return t +end +utf16_to_utf8_le=function(s) + if s and s~="" then + return lpegmatch(p_utf16_to_utf8_le,s) + else + return s + end +end +local utf16_to_utf8_le_t=function(t) + if not t then + return nil + elseif type(t)=="string" then + t=lpegmatch(utf_16_le_linesplitter,t) + end + for i=1,#t do + local s=t[i] + if s~="" then + t[i]=lpegmatch(p_utf16_to_utf8_le,s) + end + end + return t +end +utf32_to_utf8_be=function(s) + if s and s~="" then + return lpegmatch(p_utf32_to_utf8_be,s) + else + return s + end +end +local utf32_to_utf8_be_t=function(t) + if not t then + return nil + elseif type(t)=="string" then + t=lpegmatch(utf_32_be_linesplitter,t) + end + for i=1,#t do + local s=t[i] + if s~="" then + t[i]=lpegmatch(p_utf32_to_utf8_be,s) + end + end + return t +end +utf32_to_utf8_le=function(s) + if s and s~="" then + return lpegmatch(p_utf32_to_utf8_le,s) + else + return s + end +end +local utf32_to_utf8_le_t=function(t) + if not t then + return nil + elseif type(t)=="string" then + t=lpegmatch(utf_32_le_linesplitter,t) + end + for i=1,#t do + local s=t[i] + if s~="" then + t[i]=lpegmatch(p_utf32_to_utf8_le,s) + end + end + return t +end +utf.utf16_to_utf8_le_t=utf16_to_utf8_le_t +utf.utf16_to_utf8_be_t=utf16_to_utf8_be_t +utf.utf32_to_utf8_le_t=utf32_to_utf8_le_t +utf.utf32_to_utf8_be_t=utf32_to_utf8_be_t +utf.utf16_to_utf8_le=utf16_to_utf8_le +utf.utf16_to_utf8_be=utf16_to_utf8_be +utf.utf32_to_utf8_le=utf32_to_utf8_le +utf.utf32_to_utf8_be=utf32_to_utf8_be +function utf.utf8_to_utf8_t(t) + return type(t)=="string" and lpegmatch(utflinesplitter,t) or t +end +function utf.utf16_to_utf8_t(t,endian) + return endian and utf16_to_utf8_be_t(t) or utf16_to_utf8_le_t(t) or t +end +function utf.utf32_to_utf8_t(t,endian) + return endian and utf32_to_utf8_be_t(t) or utf32_to_utf8_le_t(t) or t +end +local function little(b) + if b<0x10000 then + return char(b%256,rshift(b,8)) + else + b=b-0x10000 + local b1=rshift(b,10)+0xD800 + local b2=b%1024+0xDC00 + return char(b1%256,rshift(b1,8),b2%256,rshift(b2,8)) + end +end +local function big(b) + if b<0x10000 then + return char(rshift(b,8),b%256) + else + b=b-0x10000 + local b1=rshift(b,10)+0xD800 + local b2=b%1024+0xDC00 + return char(rshift(b1,8),b1%256,rshift(b2,8),b2%256) + end +end +local l_remap=Cs((p_utf8byte/little+P(1)/"")^0) +local b_remap=Cs((p_utf8byte/big+P(1)/"")^0) +local function utf8_to_utf16_be(str,nobom) + if nobom then + return lpegmatch(b_remap,str) + else + return char(254,255)..lpegmatch(b_remap,str) + end +end +local function utf8_to_utf16_le(str,nobom) + if nobom then + return lpegmatch(l_remap,str) + else + return char(255,254)..lpegmatch(l_remap,str) + end +end +utf.utf8_to_utf16_be=utf8_to_utf16_be +utf.utf8_to_utf16_le=utf8_to_utf16_le +function utf.utf8_to_utf16(str,littleendian,nobom) + if littleendian then + return utf8_to_utf16_le(str,nobom) + else + return utf8_to_utf16_be(str,nobom) + end +end +local pattern=Cs ( + (p_utf8byte/function(unicode ) return format("0x%04X",unicode) end)*(p_utf8byte*Carg(1)/function(unicode,separator) return format("%s0x%04X",separator,unicode) end)^0 +) +function utf.tocodes(str,separator) + return lpegmatch(pattern,str,1,separator or " ") +end +function utf.ustring(s) + return format("U+%05X",type(s)=="number" and s or utfbyte(s)) +end +function utf.xstring(s) + return format("0x%05X",type(s)=="number" and s or utfbyte(s)) +end +function utf.toeight(str) + if not str or str=="" then + return nil + end + local utftype=lpegmatch(p_utfstricttype,str) + if utftype=="utf-8" then + return sub(str,4) + elseif utftype=="utf-16-be" then + return utf16_to_utf8_be(str) + elseif utftype=="utf-16-le" then + return utf16_to_utf8_le(str) + else + return str + end +end +do + local p_nany=p_utf8character/"" + local cache={} + function utf.count(str,what) + if type(what)=="string" then + local p=cache[what] + if not p then + p=Cs((P(what)/" "+p_nany)^0) + cache[p]=p + end + return #lpegmatch(p,str) + else + return #lpegmatch(Cs((P(what)/" "+p_nany)^0),str) + end + end +end +if not string.utfvalues then + local find=string.find + local dummy=function() + end + function string.utfvalues(str) + local n=#str + if n==0 then + return dummy + elseif n==1 then + return function() return utfbyte(str) end + else + local p=1 + return function() + local b,e=find(str,".[\128-\191]*",p) + if b then + p=e+1 + return utfbyte(sub(str,b,e)) + end + end + end + end +end +utf.values=string.utfvalues +function utf.chrlen(u) + return + (u<0x80 and 1) or + (u<0xE0 and 2) or + (u<0xF0 and 3) or + (u<0xF8 and 4) or + (u<0xFC and 5) or + (u<0xFE and 6) or 0 +end +if bit32 then + local extract=bit32.extract + local char=string.char + function utf.toutf32string(n) + if n<=0xFF then + return + char(n).."\000\000\000" + elseif n<=0xFFFF then + return + char(extract(n,0,8))..char(extract(n,8,8)).."\000\000" + elseif n<=0xFFFFFF then + return + char(extract(n,0,8))..char(extract(n,8,8))..char(extract(n,16,8)).."\000" + else + return + char(extract(n,0,8))..char(extract(n,8,8))..char(extract(n,16,8))..char(extract(n,24,8)) + end + end +end +local len=utf.len +local rep=rep +function string.utfpadd(s,n) + if n and n~=0 then + local l=len(s) + if n>0 then + local d=n-l + if d>0 then + return rep(c or " ",d)..s + end + else + local d=- n-l + if d>0 then + return s..rep(c or " ",d) + end + end + end + return s +end +do + local utfcharacters=utf.characters or string.utfcharacters + local utfchar=utf.char or string.utfcharacter + lpeg.UP=P + if utfcharacters then + function lpeg.US(str) + local p=P(false) + for uc in utfcharacters(str) do + p=p+P(uc) + end + return p + end + else + function lpeg.US(str) + local p=P(false) + local f=function(uc) + p=p+P(uc) + end + lpegmatch((p_utf8char/f)^0,str) + return p + end + end + local range=p_utf8byte*p_utf8byte+Cc(false) + function lpeg.UR(str,more) + local first,last + if type(str)=="number" then + first=str + last=more or first + else + first,last=lpegmatch(range,str) + if not last then + return P(str) + end + end + if first==last then + return P(str) + end + if not utfchar then + utfchar=utf.char + end + if utfchar and (last-first<8) then + local p=P(false) + for i=first,last do + p=p+P(utfchar(i)) + end + return p + else + local f=function(b) + return b>=first and b<=last + end + return p_utf8byte/f + end + end +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-url']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local char,format,byte=string.char,string.format,string.byte +local concat=table.concat +local tonumber,type,next=tonumber,type,next +local P,C,R,S,Cs,Cc,Ct,Cf,Cg,V=lpeg.P,lpeg.C,lpeg.R,lpeg.S,lpeg.Cs,lpeg.Cc,lpeg.Ct,lpeg.Cf,lpeg.Cg,lpeg.V +local lpegmatch,lpegpatterns,replacer=lpeg.match,lpeg.patterns,lpeg.replacer +local sortedhash=table.sortedhash +url=url or {} +local url=url +local unescapes={} +local escapes={} +setmetatable(unescapes,{ __index=function(t,k) + local v=char(tonumber(k,16)) + t[k]=v + return v +end }) +setmetatable(escapes,{ __index=function(t,k) + local v=format("%%%02X",byte(k)) + t[k]=v + return v +end }) +local colon=P(":") +local qmark=P("?") +local hash=P("#") +local slash=P("/") +local atsign=P("@") +local percent=P("%") +local endofstring=P(-1) +local hexdigit=R("09","AF","af") +local plus=P("+") +local nothing=Cc("") +local okay=R("09","AZ","az")+S("-_.,:=+*~!'()@&$") +local escapedchar=(percent*C(hexdigit*hexdigit))/unescapes +local unescapedchar=P(1)/escapes +local escaped=(plus/" ")+escapedchar +local noslash=P("/")/"" +local plustospace=P("+")/" " +local decoder=Cs(( + plustospace+escapedchar+P("\r\n")/"\n"+P(1) + )^0 ) +local encoder=Cs(( + R("09","AZ","az")^1+S("-./_")^1+P(" ")/"+"+P("\n")/"\r\n"+unescapedchar + )^0 ) +lpegpatterns.urldecoder=decoder +lpegpatterns.urlencoder=encoder +function url.decode (str) return str and lpegmatch(decoder,str) or str end +function url.encode (str) return str and lpegmatch(encoder,str) or str end +function url.unescape(str) return str and lpegmatch(unescaper,str) or str end +local schemestr=Cs((escaped+(1-colon-slash-qmark-hash))^2) +local authoritystr=Cs((escaped+(1- slash-qmark-hash))^0) +local pathstr=Cs((escaped+(1- qmark-hash))^0) +local querystr=Cs(((1- hash))^0) +local fragmentstr=Cs((escaped+(1- endofstring))^0) +local scheme=schemestr*colon+nothing +local authority=slash*slash*authoritystr+nothing +local path=slash*pathstr+nothing +local query=qmark*querystr+nothing +local fragment=hash*fragmentstr+nothing +local validurl=scheme*authority*path*query*fragment +local parser=Ct(validurl) +lpegpatterns.url=validurl +lpegpatterns.urlsplitter=parser +local escaper=Cs((R("09","AZ","az")^1+P(" ")/"%%20"+S("-./_:")^1+P(1)/escapes)^0) +local unescaper=Cs((escapedchar+1)^0) +local getcleaner=Cs((P("+++")/"%%2B"+P("+")/"%%20"+P(1))^1) +lpegpatterns.urlunescaped=escapedchar +lpegpatterns.urlescaper=escaper +lpegpatterns.urlunescaper=unescaper +lpegpatterns.urlgetcleaner=getcleaner +function url.unescapeget(str) + return lpegmatch(getcleaner,str) +end +local function split(str) + return (type(str)=="string" and lpegmatch(parser,str)) or str +end +local isscheme=schemestr*colon*slash*slash +local function hasscheme(str) + if str then + local scheme=lpegmatch(isscheme,str) + return scheme~="" and scheme or false + else + return false + end +end +local rootletter=R("az","AZ")+S("_-+") +local separator=P("://") +local qualified=P(".")^0*P("/")+rootletter*P(":")+rootletter^1*separator+rootletter^1*P("/") +local rootbased=P("/")+rootletter*P(":") +local barswapper=replacer("|",":") +local backslashswapper=replacer("\\","/") +local equal=P("=") +local amp=P("&") +local key=Cs(((plustospace+escapedchar+1)-equal )^0) +local value=Cs(((plustospace+escapedchar+1)-amp-endofstring)^0) +local splitquery=Cf (Ct("")*P { "sequence", + sequence=V("pair")*(amp*V("pair"))^0, + pair=Cg(key*equal*value), +},rawset) +local userpart=(1-atsign-colon)^1 +local serverpart=(1-colon)^1 +local splitauthority=((Cs(userpart)*colon*Cs(userpart)+Cs(userpart)*Cc(nil))*atsign+Cc(nil)*Cc(nil))*Cs(serverpart)*(colon*(serverpart/tonumber)+Cc(nil)) +local function hashed(str) + if not str or str=="" then + return { + scheme="invalid", + original=str, + } + end + local detailed=split(str) + local rawscheme="" + local rawquery="" + local somescheme=false + local somequery=false + if detailed then + rawscheme=detailed[1] + rawquery=detailed[4] + somescheme=rawscheme~="" + somequery=rawquery~="" + end + if not somescheme and not somequery then + return { + scheme="file", + authority="", + path=str, + query="", + fragment="", + original=str, + noscheme=true, + filename=str, + } + end + local authority=detailed[2] + local path=detailed[3] + local filename + local username + local password + local host + local port + if authority~="" then + username,password,host,port=lpegmatch(splitauthority,authority) + end + if authority=="" then + filename=path + elseif path=="" then + filename="" + else + filename=authority.."/"..path + end + return { + scheme=rawscheme, + authority=authority, + path=path, + query=lpegmatch(unescaper,rawquery), + queries=lpegmatch(splitquery,rawquery), + fragment=detailed[5], + original=str, + noscheme=false, + filename=filename, + host=host, + port=port, + } +end +url.split=split +url.hasscheme=hasscheme +url.hashed=hashed +function url.addscheme(str,scheme) + if hasscheme(str) then + return str + elseif not scheme then + return "file:///"..str + else + return scheme..":///"..str + end +end +function url.construct(hash) + local result,r={},0 + local scheme=hash.scheme + local authority=hash.authority + local path=hash.path + local queries=hash.queries + local fragment=hash.fragment + if scheme and scheme~="" then + r=r+1;result[r]=lpegmatch(escaper,scheme) + r=r+1;result[r]="://" + end + if authority and authority~="" then + r=r+1;result[r]=lpegmatch(escaper,authority) + end + if path and path~="" then + r=r+1;result[r]="/" + r=r+1;result[r]=lpegmatch(escaper,path) + end + if queries then + local done=false + for k,v in sortedhash(queries) do + r=r+1;result[r]=done and "&" or "?" + r=r+1;result[r]=lpegmatch(escaper,k) + r=r+1;result[r]="=" + r=r+1;result[r]=lpegmatch(escaper,v) + done=true + end + end + if fragment and fragment~="" then + r=r+1;result[r]="#" + r=r+1;result[r]=lpegmatch(escaper,fragment) + end + return concat(result) +end +local pattern=Cs(slash^-1/""*R("az","AZ")*((S(":|")/":")+P(":"))*slash*P(1)^0) +function url.filename(filename) + local spec=hashed(filename) + local path=spec.path + return (spec.scheme=="file" and path and lpegmatch(pattern,path)) or filename +end +local function escapestring(str) + return lpegmatch(escaper,str) +end +url.escape=escapestring +function url.query(str) + if type(str)=="string" then + return lpegmatch(splitquery,str) or "" + else + return str + end +end +function url.toquery(data) + local td=type(data) + if td=="string" then + return #str and escape(data) or nil + elseif td=="table" then + if next(data) then + local t={} + for k,v in next,data do + t[#t+1]=format("%s=%s",k,escapestring(v)) + end + return concat(t,"&") + end + else + end +end +local pattern=Cs(noslash^0*(1-noslash*P(-1))^0) +function url.barepath(path) + if not path or path=="" then + return "" + else + return lpegmatch(pattern,path) + end +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['l-set']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +set=set or {} +local nums={} +local tabs={} +local concat=table.concat +local next,type=next,type +set.create=table.tohash +function set.tonumber(t) + if next(t) then + local s="" + for k,v in next,t do + if v then + s=s.." "..k + end + end + local n=nums[s] + if not n then + n=#tabs+1 + tabs[n]=t + nums[s]=n + end + return n + else + return 0 + end +end +function set.totable(n) + if n==0 then + return {} + else + return tabs[n] or {} + end +end +function set.tolist(n) + if n==0 or not tabs[n] then + return "" + else + local t,n={},0 + for k,v in next,tabs[n] do + if v then + n=n+1 + t[n]=k + end + end + return concat(t," ") + end +end +function set.contains(n,s) + if type(n)=="table" then + return n[s] + elseif n==0 then + return false + else + local t=tabs[n] + return t and t[s] + end +end + +end -- closure diff --git a/macros/luatex/generic/lualibs/lualibs-boolean.lua b/macros/luatex/generic/lualibs/lualibs-boolean.lua new file mode 100644 index 0000000000..8f18d4c003 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-boolean.lua @@ -0,0 +1,69 @@ +if not modules then modules = { } end modules ['l-boolean'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +local type, tonumber = type, tonumber + +boolean = boolean or { } +local boolean = boolean + +function boolean.tonumber(b) + if b then return 1 else return 0 end -- test and return or return +end + +function toboolean(str,tolerant) -- global + if str == nil then + return false + elseif str == false then + return false + elseif str == true then + return true + elseif str == "true" then + return true + elseif str == "false" then + return false + elseif not tolerant then + return false + elseif str == 0 then + return false + elseif (tonumber(str) or 0) > 0 then + return true + else + return str == "yes" or str == "on" or str == "t" + end +end + +string.toboolean = toboolean + +function string.booleanstring(str) + if str == "0" then + return false + elseif str == "1" then + return true + elseif str == "" then + return false + elseif str == "false" then + return false + elseif str == "true" then + return true + elseif (tonumber(str) or 0) > 0 then + return true + else + return str == "yes" or str == "on" or str == "t" + end +end + +function string.is_boolean(str,default,strict) + if type(str) == "string" then + if str == "true" or str == "yes" or str == "on" or str == "t" or (not strict and str == "1") then + return true + elseif str == "false" or str == "no" or str == "off" or str == "f" or (not strict and str == "0") then + return false + end + end + return default +end diff --git a/macros/luatex/generic/lualibs/lualibs-compat.lua b/macros/luatex/generic/lualibs/lualibs-compat.lua new file mode 100644 index 0000000000..707f26b8b6 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-compat.lua @@ -0,0 +1,31 @@ +#!/usr/bin/env texlua + +lualibs = lualibs or { } + +local stringgsub = string.gsub +local stringlower = string.lower +local next = next +local Ct, splitat = lpeg.Ct, lpeg.splitat + +--[[doc +Needed by legacy luat-dum.lua. +--doc]]-- +table.reverse_hash = function (h) + local r = { } + for k,v in next, h do + r[v] = stringlower(stringgsub(k," ","")) + end + return r +end + +--[[doc +Needed by legacy font-otn.lua. +--doc]]-- +lpeg.splitters = { [" "] = Ct(splitat" ") } + +--[[doc +Needed by legacy font-nms.lua. +--doc]]-- + +file.split_path = file.splitpath +file.collapse_path = file.collapsepath diff --git a/macros/luatex/generic/lualibs/lualibs-dir.lua b/macros/luatex/generic/lualibs/lualibs-dir.lua new file mode 100644 index 0000000000..325039cb1e --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-dir.lua @@ -0,0 +1,642 @@ +if not modules then modules = { } end modules ['l-dir'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +-- todo: dir.expandname will be sped up and merged with cleanpath and collapsepath +-- todo: keep track of currentdir (chdir, pushdir, popdir) + +local type, select = type, select +local find, gmatch, match, gsub, sub = string.find, string.gmatch, string.match, string.gsub, string.sub +local concat, insert, remove, unpack = table.concat, table.insert, table.remove, table.unpack +local lpegmatch = lpeg.match + +local P, S, R, C, Cc, Cs, Ct, Cv, V = lpeg.P, lpeg.S, lpeg.R, lpeg.C, lpeg.Cc, lpeg.Cs, lpeg.Ct, lpeg.Cv, lpeg.V + +dir = dir or { } +local dir = dir +local lfs = lfs + +local attributes = lfs.attributes +local walkdir = lfs.dir +local isdir = lfs.isdir -- not robust, will be overloaded anyway +local isfile = lfs.isfile -- not robust, will be overloaded anyway +local currentdir = lfs.currentdir +local chdir = lfs.chdir +local mkdir = lfs.mkdir + +local onwindows = os.type == "windows" or find(os.getenv("PATH"),";",1,true) + +-- in case we load outside luatex + +if onwindows then + + -- lfs.isdir does not like trailing / + -- lfs.dir accepts trailing / + + local tricky = S("/\\") * P(-1) + + isdir = function(name) + if lpegmatch(tricky,name) then + return attributes(name,"mode") == "directory" + else + return attributes(name.."/.","mode") == "directory" + end + end + + isfile = function(name) + return attributes(name,"mode") == "file" + end + + lfs.isdir = isdir + lfs.isfile = isfile + +else + + isdir = function(name) + return attributes(name,"mode") == "directory" + end + + isfile = function(name) + return attributes(name,"mode") == "file" + end + + lfs.isdir = isdir + lfs.isfile = isfile + +end + +-- handy + +function dir.current() + return (gsub(currentdir(),"\\","/")) +end + +-- The next one is somewhat optimized but still slow but it's a pitty that the iterator +-- doesn't return a mode too. + +local function glob_pattern_function(path,patt,recurse,action) + if isdir(path) then + local usedpath + if path == "/" then + usedpath = "/." + elseif not find(path,"/$") then + usedpath = path .. "/." + path = path .. "/" + else + usedpath = path + end + local dirs + local nofdirs = 0 + for name, mode, size, time in walkdir(usedpath) do + if name ~= "." and name ~= ".." then + local full = path .. name + if mode == nil then + mode = attributes(full,'mode') + end + if mode == 'file' then + if not patt or find(full,patt) then + action(full,size,time) + end + elseif recurse and mode == "directory" then + if dirs then + nofdirs = nofdirs + 1 + dirs[nofdirs] = full + else + nofdirs = 1 + dirs = { full } + end + end + end + end + if dirs then + for i=1,nofdirs do + glob_pattern_function(dirs[i],patt,recurse,action) + end + end + end +end + +local function glob_pattern_table(path,patt,recurse,result) + if not result then + result = { } + end + local usedpath + if path == "/" then + usedpath = "/." + elseif not find(path,"/$") then + usedpath = path .. "/." + path = path .. "/" + else + usedpath = path + end + local dirs + local nofdirs = 0 + local noffiles = #result + for name, mode in walkdir(usedpath) do + if name ~= "." and name ~= ".." then + local full = path .. name + if mode == nil then + mode = attributes(full,'mode') + end + if mode == 'file' then + if not patt or find(full,patt) then + noffiles = noffiles + 1 + result[noffiles] = full + end + elseif recurse and mode == "directory" then + if dirs then + nofdirs = nofdirs + 1 + dirs[nofdirs] = full + else + nofdirs = 1 + dirs = { full } + end + end + end + end + if dirs then + for i=1,nofdirs do + glob_pattern_table(dirs[i],patt,recurse,result) + end + end + return result +end + +local function globpattern(path,patt,recurse,method) + local kind = type(method) + if patt and sub(patt,1,-3) == path then + patt = false + end + local okay = isdir(path) + if kind == "function" then + return okay and glob_pattern_function(path,patt,recurse,method) or { } + elseif kind == "table" then + return okay and glob_pattern_table(path,patt,recurse,method) or method + else + return okay and glob_pattern_table(path,patt,recurse,{ }) or { } + end +end + +dir.globpattern = globpattern + +-- never or seldom used so far: + +local function collectpattern(path,patt,recurse,result) + local ok, scanner + result = result or { } + if path == "/" then + ok, scanner, first = xpcall(function() return walkdir(path..".") end, function() end) -- kepler safe + else + ok, scanner, first = xpcall(function() return walkdir(path) end, function() end) -- kepler safe + end + if ok and type(scanner) == "function" then + if not find(path,"/$") then + path = path .. '/' + end + for name in scanner, first do -- cna be optimized + if name == "." then + -- skip + elseif name == ".." then + -- skip + else + local full = path .. name + local attr = attributes(full) + local mode = attr.mode + if mode == 'file' then + if find(full,patt) then + result[name] = attr + end + elseif recurse and mode == "directory" then + attr.list = collectpattern(full,patt,recurse) + result[name] = attr + end + end + end + end + return result +end + +dir.collectpattern = collectpattern + +local separator, pattern + +if onwindows then -- we could sanitize here + + local slash = S("/\\") / "/" + +-- pattern = Ct { + pattern = { + [1] = (Cs(P(".") + slash^1) + Cs(R("az","AZ") * P(":") * slash^0) + Cc("./")) * V(2) * V(3), + [2] = Cs(((1-S("*?/\\"))^0 * slash)^0), + [3] = Cs(P(1)^0) + } + +else -- assume unix + +-- pattern = Ct { + pattern = { + [1] = (C(P(".") + P("/")^1) + Cc("./")) * V(2) * V(3), + [2] = C(((1-S("*?/"))^0 * P("/"))^0), + [3] = C(P(1)^0) + } + +end + +local filter = Cs ( ( + P("**") / ".*" + + P("*") / "[^/]*" + + P("?") / "[^/]" + + P(".") / "%%." + + P("+") / "%%+" + + P("-") / "%%-" + + P(1) +)^0 ) + +local function glob(str,t) + if type(t) == "function" then + if type(str) == "table" then + for s=1,#str do + glob(str[s],t) + end + elseif isfile(str) then + t(str) + else + local root, path, base = lpegmatch(pattern,str) -- we could use the file splitter + if root and path and base then + local recurse = find(base,"**",1,true) -- find(base,"%*%*") + local start = root .. path + local result = lpegmatch(filter,start .. base) + globpattern(start,result,recurse,t) + end + end + else + if type(str) == "table" then + local t = t or { } + for s=1,#str do + glob(str[s],t) + end + return t + elseif isfile(str) then + if t then + t[#t+1] = str + return t + else + return { str } + end + else + local root, path, base = lpegmatch(pattern,str) -- we could use the file splitter + if root and path and base then + local recurse = find(base,"**",1,true) -- find(base,"%*%*") + local start = root .. path + local result = lpegmatch(filter,start .. base) + return globpattern(start,result,recurse,t) + else + return { } + end + end + end +end + +dir.glob = glob + +-- local c = os.clock() +-- local t = dir.glob("e:/**") +-- local t = dir.glob("t:/sources/**") +-- local t = dir.glob("t:/**") +-- print(os.clock()-c,#t) + +-- for i=1,3000 do print(t[i]) end +-- for i=1,10 do print(t[i]) end + +-- list = dir.glob("**/*.tif") +-- list = dir.glob("/**/*.tif") +-- list = dir.glob("./**/*.tif") +-- list = dir.glob("oeps/**/*.tif") +-- list = dir.glob("/oeps/**/*.tif") + +local function globfiles(path,recurse,func,files) -- func == pattern or function + if type(func) == "string" then + local s = func + func = function(name) return find(name,s) end + end + files = files or { } + local noffiles = #files + for name, mode in walkdir(path) do + if find(name,"^%.") then + --- skip + else + if mode == nil then + mode = attributes(name,'mode') + end + if mode == "directory" then + if recurse then + globfiles(path .. "/" .. name,recurse,func,files) + end + elseif mode == "file" then + if not func or func(name) then + noffiles = noffiles + 1 + files[noffiles] = path .. "/" .. name + end + end + end + end + return files +end + +dir.globfiles = globfiles + +local function globdirs(path,recurse,func,files) -- func == pattern or function + if type(func) == "string" then + local s = func + func = function(name) return find(name,s) end + end + files = files or { } + local noffiles = #files + for name, mode in walkdir(path) do + if find(name,"^%.") then + --- skip + else + if mode == nil then + mode = attributes(name,'mode') + end + if mode == "directory" then + if not func or func(name) then + noffiles = noffiles + 1 + files[noffiles] = path .. "/" .. name + if recurse then + globdirs(path .. "/" .. name,recurse,func,files) + end + end + end + end + end + return files +end + +dir.globdirs = globdirs + +-- inspect(globdirs("e:/tmp")) + +-- t = dir.glob("c:/data/develop/context/sources/**/????-*.tex") +-- t = dir.glob("c:/data/develop/tex/texmf/**/*.tex") +-- t = dir.glob("c:/data/develop/context/texmf/**/*.tex") +-- t = dir.glob("f:/minimal/tex/**/*") +-- print(dir.ls("f:/minimal/tex/**/*")) +-- print(dir.ls("*.tex")) + +function dir.ls(pattern) + return concat(glob(pattern),"\n") +end + +-- mkdirs("temp") +-- mkdirs("a/b/c") +-- mkdirs(".","/a/b/c") +-- mkdirs("a","b","c") + +local make_indeed = true -- false + +if onwindows then + + function dir.mkdirs(...) + local n = select("#",...) + local str + if n == 1 then + str = select(1,...) + if isdir(str) then + return str, true + end + else + str = "" + for i=1,n do + local s = select(i,...) + if s == "" then + -- skip + elseif str == "" then + str = s + else + str = str .. "/" .. s + end + end + end + local pth = "" + local drive = false + local first, middle, last = match(str,"^(//)(//*)(.*)$") + if first then + -- empty network path == local path + else + first, last = match(str,"^(//)/*(.-)$") + if first then + middle, last = match(str,"([^/]+)/+(.-)$") + if middle then + pth = "//" .. middle + else + pth = "//" .. last + last = "" + end + else + first, middle, last = match(str,"^([a-zA-Z]:)(/*)(.-)$") + if first then + pth, drive = first .. middle, true + else + middle, last = match(str,"^(/*)(.-)$") + if not middle then + last = str + end + end + end + end + for s in gmatch(last,"[^/]+") do + if pth == "" then + pth = s + elseif drive then + pth, drive = pth .. s, false + else + pth = pth .. "/" .. s + end + if make_indeed and not isdir(pth) then + mkdir(pth) + end + end + return pth, (isdir(pth) == true) + end + + -- print(dir.mkdirs("","","a","c")) + -- print(dir.mkdirs("a")) + -- print(dir.mkdirs("a:")) + -- print(dir.mkdirs("a:/b/c")) + -- print(dir.mkdirs("a:b/c")) + -- print(dir.mkdirs("a:/bbb/c")) + -- print(dir.mkdirs("/a/b/c")) + -- print(dir.mkdirs("/aaa/b/c")) + -- print(dir.mkdirs("//a/b/c")) + -- print(dir.mkdirs("///a/b/c")) + -- print(dir.mkdirs("a/bbb//ccc/")) + +else + + function dir.mkdirs(...) + local n = select("#",...) + local str, pth + if n == 1 then + str = select(1,...) + if isdir(str) then + return str, true + end + else + str = "" + for i=1,n do + local s = select(i,...) + if s and s ~= "" then -- we catch nil and false + if str ~= "" then + str = str .. "/" .. s + else + str = s + end + end + end + end + str = gsub(str,"/+","/") + if find(str,"^/") then + pth = "/" + for s in gmatch(str,"[^/]+") do + local first = (pth == "/") + if first then + pth = pth .. s + else + pth = pth .. "/" .. s + end + if make_indeed and not first and not isdir(pth) then + mkdir(pth) + end + end + else + pth = "." + for s in gmatch(str,"[^/]+") do + pth = pth .. "/" .. s + if make_indeed and not isdir(pth) then + mkdir(pth) + end + end + end + return pth, (isdir(pth) == true) + end + + -- print(dir.mkdirs("","","a","c")) + -- print(dir.mkdirs("a")) + -- print(dir.mkdirs("/a/b/c")) + -- print(dir.mkdirs("/aaa/b/c")) + -- print(dir.mkdirs("//a/b/c")) + -- print(dir.mkdirs("///a/b/c")) + -- print(dir.mkdirs("a/bbb//ccc/")) + +end + +dir.makedirs = dir.mkdirs + + +do + + -- we can only define it here as it uses dir.chdir and we also need to + -- make sure we use the non sandboxed variant because otherwise we get + -- into a recursive loop due to usage of expandname in the file resolver + + local chdir = sandbox and sandbox.original(chdir) or chdir + + if onwindows then + + local xcurrentdir = dir.current + + function dir.expandname(str) -- will be merged with cleanpath and collapsepath\ + local first, nothing, last = match(str,"^(//)(//*)(.*)$") + if first then + first = xcurrentdir() .. "/" -- xcurrentdir sanitizes + end + if not first then + first, last = match(str,"^(//)/*(.*)$") + end + if not first then + first, last = match(str,"^([a-zA-Z]:)(.*)$") + if first and not find(last,"^/") then + local d = currentdir() -- push / pop + if chdir(first) then + first = xcurrentdir() -- xcurrentdir sanitizes + end + chdir(d) + end + end + if not first then + first, last = xcurrentdir(), str + end + last = gsub(last,"//","/") + last = gsub(last,"/%./","/") + last = gsub(last,"^/*","") + first = gsub(first,"/*$","") + if last == "" or last == "." then + return first + else + return first .. "/" .. last + end + end + + else + + function dir.expandname(str) -- will be merged with cleanpath and collapsepath + if not find(str,"^/") then + str = currentdir() .. "/" .. str + end + str = gsub(str,"//","/") + str = gsub(str,"/%./","/") + str = gsub(str,"(.)/%.$","%1") + return str + end + + end + +end + +file.expandname = dir.expandname -- for convenience + +local stack = { } + +function dir.push(newdir) + local curdir = currentdir() + insert(stack,curdir) + if newdir and newdir ~= "" and chdir(newdir) then + return newdir + else + return curdir + end +end + +function dir.pop() + local d = remove(stack) + if d then + chdir(d) + end + return d +end + +local function found(...) -- can have nil entries + for i=1,select("#",...) do + local path = select(i,...) + local kind = type(path) + if kind == "string" then + if isdir(path) then + return path + end + elseif kind == "table" then + -- here we asume no holes, i.e. an indexed table + local path = found(unpack(path)) + if path then + return path + end + end + end + -- return nil -- if we want print("crappath") to show something +end + +dir.found = found diff --git a/macros/luatex/generic/lualibs/lualibs-extended-merged.lua b/macros/luatex/generic/lualibs/lualibs-extended-merged.lua new file mode 100644 index 0000000000..b3fecf2c3e --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-extended-merged.lua @@ -0,0 +1,4194 @@ +-- merged file : lualibs-extended-merged.lua +-- parent file : lualibs-extended.lua +-- merge date : Tue Aug 13 20:12:59 2019 + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['util-str']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +utilities=utilities or {} +utilities.strings=utilities.strings or {} +local strings=utilities.strings +local format,gsub,rep,sub,find=string.format,string.gsub,string.rep,string.sub,string.find +local load,dump=load,string.dump +local tonumber,type,tostring,next,setmetatable=tonumber,type,tostring,next,setmetatable +local unpack,concat=table.unpack,table.concat +local P,V,C,S,R,Ct,Cs,Cp,Carg,Cc=lpeg.P,lpeg.V,lpeg.C,lpeg.S,lpeg.R,lpeg.Ct,lpeg.Cs,lpeg.Cp,lpeg.Carg,lpeg.Cc +local patterns,lpegmatch=lpeg.patterns,lpeg.match +local utfchar,utfbyte,utflen=utf.char,utf.byte,utf.len +local loadstripped=function(str,shortcuts) + if shortcuts then + return load(dump(load(str),true),nil,nil,shortcuts) + else + return load(dump(load(str),true)) + end +end +if not number then number={} end +local stripzero=patterns.stripzero +local stripzeros=patterns.stripzeros +local newline=patterns.newline +local endofstring=patterns.endofstring +local anything=patterns.anything +local whitespace=patterns.whitespace +local space=patterns.space +local spacer=patterns.spacer +local spaceortab=patterns.spaceortab +local digit=patterns.digit +local sign=patterns.sign +local period=patterns.period +local ptf=1/65536 +local bpf=(7200/7227)/65536 +local function points(n) + if n==0 then + return "0pt" + end + n=tonumber(n) + if not n or n==0 then + return "0pt" + end + n=n*ptf + if n%1==0 then + return format("%ipt",n) + end + return lpegmatch(stripzeros,format("%.5fpt",n)) +end +local function basepoints(n) + if n==0 then + return "0pt" + end + n=tonumber(n) + if not n or n==0 then + return "0pt" + end + n=n*bpf + if n%1==0 then + return format("%ibp",n) + end + return lpegmatch(stripzeros,format("%.5fbp",n)) +end +number.points=points +number.basepoints=basepoints +local rubish=spaceortab^0*newline +local anyrubish=spaceortab+newline +local stripped=(spaceortab^1/"")*newline +local leading=rubish^0/"" +local trailing=(anyrubish^1*endofstring)/"" +local redundant=rubish^3/"\n" +local pattern=Cs(leading*(trailing+redundant+stripped+anything)^0) +function strings.collapsecrlf(str) + return lpegmatch(pattern,str) +end +local repeaters={} +function strings.newrepeater(str,offset) + offset=offset or 0 + local s=repeaters[str] + if not s then + s={} + repeaters[str]=s + end + local t=s[offset] + if t then + return t + end + t={} + setmetatable(t,{ __index=function(t,k) + if not k then + return "" + end + local n=k+offset + local s=n>0 and rep(str,n) or "" + t[k]=s + return s + end }) + s[offset]=t + return t +end +local extra,tab,start=0,0,4,0 +local nspaces=strings.newrepeater(" ") +string.nspaces=nspaces +local pattern=Carg(1)/function(t) + extra,tab,start=0,t or 7,1 + end*Cs(( + Cp()*patterns.tab/function(position) + local current=(position-start+1)+extra + local spaces=tab-(current-1)%tab + if spaces>0 then + extra=extra+spaces-1 + return nspaces[spaces] + else + return "" + end + end+newline*Cp()/function(position) + extra,start=0,position + end+anything + )^1) +function strings.tabtospace(str,tab) + return lpegmatch(pattern,str,1,tab or 7) +end +function string.utfpadding(s,n) + if not n or n==0 then + return "" + end + local l=utflen(s) + if n>0 then + return nspaces[n-l] + else + return nspaces[-n-l] + end +end +local optionalspace=spacer^0 +local nospace=optionalspace/"" +local endofline=nospace*newline +local stripend=(whitespace^1*endofstring)/"" +local normalline=(nospace*((1-optionalspace*(newline+endofstring))^1)*nospace) +local stripempty=endofline^1/"" +local normalempty=endofline^1 +local singleempty=endofline*(endofline^0/"") +local doubleempty=endofline*endofline^-1*(endofline^0/"") +local stripstart=stripempty^0 +local intospace=whitespace^1/" " +local noleading=whitespace^1/"" +local notrailing=noleading*endofstring +local p_prune_normal=Cs (stripstart*(stripend+normalline+normalempty )^0 ) +local p_prune_collapse=Cs (stripstart*(stripend+normalline+doubleempty )^0 ) +local p_prune_noempty=Cs (stripstart*(stripend+normalline+singleempty )^0 ) +local p_prune_intospace=Cs (noleading*(notrailing+intospace+1 )^0 ) +local p_retain_normal=Cs ((normalline+normalempty )^0 ) +local p_retain_collapse=Cs ((normalline+doubleempty )^0 ) +local p_retain_noempty=Cs ((normalline+singleempty )^0 ) +local striplinepatterns={ + ["prune"]=p_prune_normal, + ["prune and collapse"]=p_prune_collapse, + ["prune and no empty"]=p_prune_noempty, + ["prune and to space"]=p_prune_intospace, + ["retain"]=p_retain_normal, + ["retain and collapse"]=p_retain_collapse, + ["retain and no empty"]=p_retain_noempty, + ["collapse"]=patterns.collapser, +} +setmetatable(striplinepatterns,{ __index=function(t,k) return p_prune_collapse end }) +strings.striplinepatterns=striplinepatterns +function strings.striplines(str,how) + return str and lpegmatch(striplinepatterns[how],str) or str +end +function strings.collapse(str) + return str and lpegmatch(p_prune_intospace,str) or str +end +strings.striplong=strings.striplines +function strings.nice(str) + str=gsub(str,"[:%-+_]+"," ") + return str +end +local n=0 +local sequenced=table.sequenced +function string.autodouble(s,sep) + if s==nil then + return '""' + end + local t=type(s) + if t=="number" then + return tostring(s) + end + if t=="table" then + return ('"'..sequenced(s,sep or ",")..'"') + end + return ('"'..tostring(s)..'"') +end +function string.autosingle(s,sep) + if s==nil then + return "''" + end + local t=type(s) + if t=="number" then + return tostring(s) + end + if t=="table" then + return ("'"..sequenced(s,sep or ",").."'") + end + return ("'"..tostring(s).."'") +end +local tracedchars={ [0]= + "[null]","[soh]","[stx]","[etx]","[eot]","[enq]","[ack]","[bel]", + "[bs]","[ht]","[lf]","[vt]","[ff]","[cr]","[so]","[si]", + "[dle]","[dc1]","[dc2]","[dc3]","[dc4]","[nak]","[syn]","[etb]", + "[can]","[em]","[sub]","[esc]","[fs]","[gs]","[rs]","[us]", + "[space]", +} +string.tracedchars=tracedchars +strings.tracers=tracedchars +function string.tracedchar(b) + if type(b)=="number" then + return tracedchars[b] or (utfchar(b).." (U+"..format("%05X",b)..")") + else + local c=utfbyte(b) + return tracedchars[c] or (b.." (U+"..(c and format("%05X",c) or "?????")..")") + end +end +function number.signed(i) + if i>0 then + return "+",i + else + return "-",-i + end +end +local two=digit*digit +local three=two*digit +local prefix=(Carg(1)*three)^1 +local splitter=Cs ( + (((1-(three^1*period))^1+C(three))*prefix+C((1-period)^1))*(anything/""*Carg(2))*C(2) +) +local splitter3=Cs ( + three*prefix*endofstring+two*prefix*endofstring+digit*prefix*endofstring+three+two+digit +) +patterns.formattednumber=splitter +function number.formatted(n,sep1,sep2) + if sep1==false then + if type(n)=="number" then + n=tostring(n) + end + return lpegmatch(splitter3,n,1,sep2 or ".") + else + if type(n)=="number" then + n=format("%0.2f",n) + end + if sep1==true then + return lpegmatch(splitter,n,1,".",",") + elseif sep1=="." then + return lpegmatch(splitter,n,1,sep1,sep2 or ",") + elseif sep1=="," then + return lpegmatch(splitter,n,1,sep1,sep2 or ".") + else + return lpegmatch(splitter,n,1,sep1 or ",",sep2 or ".") + end + end +end +local p=Cs( + P("-")^0*(P("0")^1/"")^0*(1-period)^0*(period*P("0")^1*endofstring/""+period^0)*P(1-P("0")^1*endofstring)^0 + ) +function number.compactfloat(n,fmt) + if n==0 then + return "0" + elseif n==1 then + return "1" + end + n=lpegmatch(p,format(fmt or "%0.3f",n)) + if n=="." or n=="" or n=="-" then + return "0" + end + return n +end +local zero=P("0")^1/"" +local plus=P("+")/"" +local minus=P("-") +local separator=period +local trailing=zero^1*#S("eE") +local exponent=(S("eE")*(plus+Cs((minus*zero^0*endofstring)/"")+minus)*zero^0*(endofstring*Cc("0")+anything^1)) +local pattern_a=Cs(minus^0*digit^1*(separator/""*trailing+separator*(trailing+digit)^0)*exponent) +local pattern_b=Cs((exponent+anything)^0) +function number.sparseexponent(f,n) + if not n then + n=f + f="%e" + end + local tn=type(n) + if tn=="string" then + local m=tonumber(n) + if m then + return lpegmatch((f=="%e" or f=="%E") and pattern_a or pattern_b,format(f,m)) + end + elseif tn=="number" then + return lpegmatch((f=="%e" or f=="%E") and pattern_a or pattern_b,format(f,n)) + end + return tostring(n) +end +local hf={} +local hs={} +setmetatable(hf,{ __index=function(t,k) + local v="%."..k.."f" + t[k]=v + return v +end } ) +setmetatable(hs,{ __index=function(t,k) + local v="%"..k.."s" + t[k]=v + return v +end } ) +function number.formattedfloat(n,b,a) + local s=format(hf[a],n) + local l=(b or 0)+(a or 0)+1 + if #s<l then + return format(hs[l],s) + else + return s + end +end +local template=[[ +%s +%s +return function(%s) return %s end +]] +local pattern=Cs(Cc('"')*( + (1-S('"\\\n\r'))^1+P('"')/'\\"'+P('\\')/'\\\\'+P('\n')/'\\n'+P('\r')/'\\r' +)^0*Cc('"')) +patterns.escapedquotes=pattern +function string.escapedquotes(s) + return lpegmatch(pattern,s) +end +local preamble="" +local environment={ + global=global or _G, + lpeg=lpeg, + type=type, + tostring=tostring, + tonumber=tonumber, + format=string.format, + concat=table.concat, + signed=number.signed, + points=number.points, + basepoints=number.basepoints, + utfchar=utf.char, + utfbyte=utf.byte, + lpegmatch=lpeg.match, + nspaces=string.nspaces, + utfpadding=string.utfpadding, + tracedchar=string.tracedchar, + autosingle=string.autosingle, + autodouble=string.autodouble, + sequenced=table.sequenced, + formattednumber=number.formatted, + sparseexponent=number.sparseexponent, + formattedfloat=number.formattedfloat, + stripzero=patterns.stripzero, + stripzeros=patterns.stripzeros, + escapedquotes=string.escapedquotes, + FORMAT=string.f9, +} +local arguments={ "a1" } +setmetatable(arguments,{ __index=function(t,k) + local v=t[k-1]..",a"..k + t[k]=v + return v + end +}) +local prefix_any=C((sign+space+period+digit)^0) +local prefix_sub=(C((sign+digit)^0)+Cc(0))*period*(C((sign+digit)^0)+Cc(0)) +local prefix_tab=P("{")*C((1-P("}"))^0)*P("}")+C((1-R("az","AZ","09","%%"))^0) +local format_s=function(f) + n=n+1 + if f and f~="" then + return format("format('%%%ss',a%s)",f,n) + else + return format("(a%s or '')",n) + end +end +local format_S=function(f) + n=n+1 + if f and f~="" then + return format("format('%%%ss',tostring(a%s))",f,n) + else + return format("tostring(a%s)",n) + end +end +local format_right=function(f) + n=n+1 + f=tonumber(f) + if not f or f==0 then + return format("(a%s or '')",n) + elseif f>0 then + return format("utfpadding(a%s,%i)..a%s",n,f,n) + else + return format("a%s..utfpadding(a%s,%i)",n,n,f) + end +end +local format_left=function(f) + n=n+1 + f=tonumber(f) + if not f or f==0 then + return format("(a%s or '')",n) + end + if f<0 then + return format("utfpadding(a%s,%i)..a%s",n,-f,n) + else + return format("a%s..utfpadding(a%s,%i)",n,n,-f) + end +end +local format_q=function() + n=n+1 + return format("(a%s ~= nil and format('%%q',tostring(a%s)) or '')",n,n) +end +local format_Q=function() + n=n+1 + return format("escapedquotes(tostring(a%s))",n) +end +local format_i=function(f) + n=n+1 + if f and f~="" then + return format("format('%%%si',a%s)",f,n) + else + return format("format('%%i',a%s)",n) + end +end +local format_d=format_i +local format_I=function(f) + n=n+1 + return format("format('%%s%%%si',signed(a%s))",f,n) +end +local format_f=function(f) + n=n+1 + return format("format('%%%sf',a%s)",f,n) +end +local format_F=function(f) + n=n+1 + if not f or f=="" then + return format("(((a%s > -0.0000000005 and a%s < 0.0000000005) and '0') or format((a%s %% 1 == 0) and '%%i' or '%%.9f',a%s))",n,n,n,n) + else + return format("format((a%s %% 1 == 0) and '%%i' or '%%%sf',a%s)",n,f,n) + end +end +local format_k=function(b,a) + n=n+1 + return format("formattedfloat(a%s,%s,%s)",n,b or 0,a or 0) +end +local format_g=function(f) + n=n+1 + return format("format('%%%sg',a%s)",f,n) +end +local format_G=function(f) + n=n+1 + return format("format('%%%sG',a%s)",f,n) +end +local format_e=function(f) + n=n+1 + return format("format('%%%se',a%s)",f,n) +end +local format_E=function(f) + n=n+1 + return format("format('%%%sE',a%s)",f,n) +end +local format_j=function(f) + n=n+1 + return format("sparseexponent('%%%se',a%s)",f,n) +end +local format_J=function(f) + n=n+1 + return format("sparseexponent('%%%sE',a%s)",f,n) +end +local format_x=function(f) + n=n+1 + return format("format('%%%sx',a%s)",f,n) +end +local format_X=function(f) + n=n+1 + return format("format('%%%sX',a%s)",f,n) +end +local format_o=function(f) + n=n+1 + return format("format('%%%so',a%s)",f,n) +end +local format_c=function() + n=n+1 + return format("utfchar(a%s)",n) +end +local format_C=function() + n=n+1 + return format("tracedchar(a%s)",n) +end +local format_r=function(f) + n=n+1 + return format("format('%%%s.0f',a%s)",f,n) +end +local format_h=function(f) + n=n+1 + if f=="-" then + f=sub(f,2) + return format("format('%%%sx',type(a%s) == 'number' and a%s or utfbyte(a%s))",f=="" and "05" or f,n,n,n) + else + return format("format('0x%%%sx',type(a%s) == 'number' and a%s or utfbyte(a%s))",f=="" and "05" or f,n,n,n) + end +end +local format_H=function(f) + n=n+1 + if f=="-" then + f=sub(f,2) + return format("format('%%%sX',type(a%s) == 'number' and a%s or utfbyte(a%s))",f=="" and "05" or f,n,n,n) + else + return format("format('0x%%%sX',type(a%s) == 'number' and a%s or utfbyte(a%s))",f=="" and "05" or f,n,n,n) + end +end +local format_u=function(f) + n=n+1 + if f=="-" then + f=sub(f,2) + return format("format('%%%sx',type(a%s) == 'number' and a%s or utfbyte(a%s))",f=="" and "05" or f,n,n,n) + else + return format("format('u+%%%sx',type(a%s) == 'number' and a%s or utfbyte(a%s))",f=="" and "05" or f,n,n,n) + end +end +local format_U=function(f) + n=n+1 + if f=="-" then + f=sub(f,2) + return format("format('%%%sX',type(a%s) == 'number' and a%s or utfbyte(a%s))",f=="" and "05" or f,n,n,n) + else + return format("format('U+%%%sX',type(a%s) == 'number' and a%s or utfbyte(a%s))",f=="" and "05" or f,n,n,n) + end +end +local format_p=function() + n=n+1 + return format("points(a%s)",n) +end +local format_b=function() + n=n+1 + return format("basepoints(a%s)",n) +end +local format_t=function(f) + n=n+1 + if f and f~="" then + return format("concat(a%s,%q)",n,f) + else + return format("concat(a%s)",n) + end +end +local format_T=function(f) + n=n+1 + if f and f~="" then + return format("sequenced(a%s,%q)",n,f) + else + return format("sequenced(a%s)",n) + end +end +local format_l=function() + n=n+1 + return format("(a%s and 'true' or 'false')",n) +end +local format_L=function() + n=n+1 + return format("(a%s and 'TRUE' or 'FALSE')",n) +end +local format_n=function() + n=n+1 + return format("((a%s %% 1 == 0) and format('%%i',a%s) or tostring(a%s))",n,n,n) +end +local format_N=function(f) + n=n+1 + if not f or f=="" then + f=".9" + end + return format("(((a%s %% 1 == 0) and format('%%i',a%s)) or lpegmatch(stripzero,format('%%%sf',a%s)))",n,n,f,n) +end +local format_a=function(f) + n=n+1 + if f and f~="" then + return format("autosingle(a%s,%q)",n,f) + else + return format("autosingle(a%s)",n) + end +end +local format_A=function(f) + n=n+1 + if f and f~="" then + return format("autodouble(a%s,%q)",n,f) + else + return format("autodouble(a%s)",n) + end +end +local format_w=function(f) + n=n+1 + f=tonumber(f) + if f then + return format("nspaces[%s+a%s]",f,n) + else + return format("nspaces[a%s]",n) + end +end +local format_W=function(f) + return format("nspaces[%s]",tonumber(f) or 0) +end +local format_m=function(f) + n=n+1 + if not f or f=="" then + f="," + end + if f=="0" then + return format([[formattednumber(a%s,false)]],n) + else + return format([[formattednumber(a%s,%q,".")]],n,f) + end +end +local format_M=function(f) + n=n+1 + if not f or f=="" then + f="." + end + if f=="0" then + return format([[formattednumber(a%s,false)]],n) + else + return format([[formattednumber(a%s,%q,",")]],n,f) + end +end +local format_z=function(f) + n=n+(tonumber(f) or 1) + return "''" +end +local format_rest=function(s) + return format("%q",s) +end +local format_extension=function(extensions,f,name) + local extension=extensions[name] or "tostring(%s)" + local f=tonumber(f) or 1 + local w=find(extension,"%.%.%.") + if w then + if f==0 then + extension=gsub(extension,"%.%.%.","") + return extension + elseif f==1 then + extension=gsub(extension,"%.%.%.","%%s") + n=n+1 + local a="a"..n + return format(extension,a,a) + elseif f<0 then + local a="a"..(n+f+1) + return format(extension,a,a) + else + extension=gsub(extension,"%.%.%.",rep("%%s,",f-1).."%%s") + local t={} + for i=1,f do + n=n+1 + t[i]="a"..n + end + return format(extension,unpack(t)) + end + else + extension=gsub(extension,"%%s",function() + n=n+1 + return "a"..n + end) + return extension + end +end +local builder=Cs { "start", + start=( + ( + P("%")/""*( + V("!") ++V("s")+V("q")+V("i")+V("d")+V("f")+V("F")+V("g")+V("G")+V("e")+V("E")+V("x")+V("X")+V("o") ++V("c")+V("C")+V("S") ++V("Q") ++V("n") ++V("N") ++V("k") ++V("r")+V("h")+V("H")+V("u")+V("U")+V("p")+V("b")+V("t")+V("T")+V("l")+V("L")+V("I")+V("w") ++V("W") ++V("a") ++V("A") ++V("j")+V("J") ++V("m")+V("M") ++V("z") ++V(">") ++V("<") + )+V("*") + )*(endofstring+Carg(1)) + )^0, + ["s"]=(prefix_any*P("s"))/format_s, + ["q"]=(prefix_any*P("q"))/format_q, + ["i"]=(prefix_any*P("i"))/format_i, + ["d"]=(prefix_any*P("d"))/format_d, + ["f"]=(prefix_any*P("f"))/format_f, + ["F"]=(prefix_any*P("F"))/format_F, + ["g"]=(prefix_any*P("g"))/format_g, + ["G"]=(prefix_any*P("G"))/format_G, + ["e"]=(prefix_any*P("e"))/format_e, + ["E"]=(prefix_any*P("E"))/format_E, + ["x"]=(prefix_any*P("x"))/format_x, + ["X"]=(prefix_any*P("X"))/format_X, + ["o"]=(prefix_any*P("o"))/format_o, + ["S"]=(prefix_any*P("S"))/format_S, + ["Q"]=(prefix_any*P("Q"))/format_Q, + ["n"]=(prefix_any*P("n"))/format_n, + ["N"]=(prefix_any*P("N"))/format_N, + ["k"]=(prefix_sub*P("k"))/format_k, + ["c"]=(prefix_any*P("c"))/format_c, + ["C"]=(prefix_any*P("C"))/format_C, + ["r"]=(prefix_any*P("r"))/format_r, + ["h"]=(prefix_any*P("h"))/format_h, + ["H"]=(prefix_any*P("H"))/format_H, + ["u"]=(prefix_any*P("u"))/format_u, + ["U"]=(prefix_any*P("U"))/format_U, + ["p"]=(prefix_any*P("p"))/format_p, + ["b"]=(prefix_any*P("b"))/format_b, + ["t"]=(prefix_tab*P("t"))/format_t, + ["T"]=(prefix_tab*P("T"))/format_T, + ["l"]=(prefix_any*P("l"))/format_l, + ["L"]=(prefix_any*P("L"))/format_L, + ["I"]=(prefix_any*P("I"))/format_I, + ["w"]=(prefix_any*P("w"))/format_w, + ["W"]=(prefix_any*P("W"))/format_W, + ["j"]=(prefix_any*P("j"))/format_j, + ["J"]=(prefix_any*P("J"))/format_J, + ["m"]=(prefix_any*P("m"))/format_m, + ["M"]=(prefix_any*P("M"))/format_M, + ["z"]=(prefix_any*P("z"))/format_z, + ["a"]=(prefix_any*P("a"))/format_a, + ["A"]=(prefix_any*P("A"))/format_A, + ["<"]=(prefix_any*P("<"))/format_left, + [">"]=(prefix_any*P(">"))/format_right, + ["*"]=Cs(((1-P("%"))^1+P("%%")/"%%")^1)/format_rest, + ["?"]=Cs(((1-P("%"))^1 )^1)/format_rest, + ["!"]=Carg(2)*prefix_any*P("!")*C((1-P("!"))^1)*P("!")/format_extension, +} +local xx=setmetatable({},{ __index=function(t,k) local v=format("%02x",k) t[k]=v return v end }) +local XX=setmetatable({},{ __index=function(t,k) local v=format("%02X",k) t[k]=v return v end }) +local preset={ + ["%02x"]=function(n) return xx[n] end, + ["%02X"]=function(n) return XX[n] end, +} +local direct=P("%")*(sign+space+period+digit)^0*S("sqidfgGeExXo")*endofstring/[[local format = string.format return function(str) return format("%0",str) end]] +local function make(t,str) + local f=preset[str] + if f then + return f + end + local p=lpegmatch(direct,str) + if p then + f=loadstripped(p)() + else + n=0 + p=lpegmatch(builder,str,1,t._connector_,t._extensions_) + if n>0 then + p=format(template,preamble,t._preamble_,arguments[n],p) + f=loadstripped(p,t._environment_)() + else + f=function() return str end + end + end + t[str]=f + return f +end +local function use(t,fmt,...) + return t[fmt](...) +end +strings.formatters={} +function strings.formatters.new(noconcat) + local e={} + for k,v in next,environment do + e[k]=v + end + local t={ + _type_="formatter", + _connector_=noconcat and "," or "..", + _extensions_={}, + _preamble_="", + _environment_=e, + } + setmetatable(t,{ __index=make,__call=use }) + return t +end +local formatters=strings.formatters.new() +string.formatters=formatters +string.formatter=function(str,...) return formatters[str](...) end +local function add(t,name,template,preamble) + if type(t)=="table" and t._type_=="formatter" then + t._extensions_[name]=template or "%s" + if type(preamble)=="string" then + t._preamble_=preamble.."\n"..t._preamble_ + elseif type(preamble)=="table" then + for k,v in next,preamble do + t._environment_[k]=v + end + end + end +end +strings.formatters.add=add +patterns.xmlescape=Cs((P("<")/"<"+P(">")/">"+P("&")/"&"+P('"')/"""+anything)^0) +patterns.texescape=Cs((C(S("#$%\\{}"))/"\\%1"+anything)^0) +patterns.luaescape=Cs(((1-S('"\n'))^1+P('"')/'\\"'+P('\n')/'\\n"')^0) +patterns.luaquoted=Cs(Cc('"')*((1-S('"\n'))^1+P('"')/'\\"'+P('\n')/'\\n"')^0*Cc('"')) +add(formatters,"xml",[[lpegmatch(xmlescape,%s)]],{ xmlescape=patterns.xmlescape }) +add(formatters,"tex",[[lpegmatch(texescape,%s)]],{ texescape=patterns.texescape }) +add(formatters,"lua",[[lpegmatch(luaescape,%s)]],{ luaescape=patterns.luaescape }) +local dquote=patterns.dquote +local equote=patterns.escaped+dquote/'\\"'+1 +local cquote=Cc('"') +local pattern=Cs(dquote*(equote-P(-2))^0*dquote) ++Cs(cquote*(equote-space)^0*space*equote^0*cquote) +function string.optionalquoted(str) + return lpegmatch(pattern,str) or str +end +local pattern=Cs((newline/(os.newline or "\r")+1)^0) +function string.replacenewlines(str) + return lpegmatch(pattern,str) +end +function strings.newcollector() + local result,r={},0 + return + function(fmt,str,...) + r=r+1 + result[r]=str==nil and fmt or formatters[fmt](str,...) + end, + function(connector) + if result then + local str=concat(result,connector) + result,r={},0 + return str + end + end +end +local f_16_16=formatters["%0.5N"] +function number.to16dot16(n) + return f_16_16(n/65536.0) +end +if not string.explode then + local tsplitat=lpeg.tsplitat + local p_utf=patterns.utf8character + local p_check=C(p_utf)*(P("+")*Cc(true))^0 + local p_split=Ct(C(p_utf)^0) + local p_space=Ct((C(1-P(" ")^1)+P(" ")^1)^0) + function string.explode(str,symbol) + if symbol=="" then + return lpegmatch(p_split,str) + elseif symbol then + local a,b=lpegmatch(p_check,symbol) + if b then + return lpegmatch(tsplitat(P(a)^1),str) + else + return lpegmatch(tsplitat(a),str) + end + else + return lpegmatch(p_space,str) + end + end +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['util-fil']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local byte=string.byte +local char=string.char +utilities=utilities or {} +local files={} +utilities.files=files +local zerobased={} +function files.open(filename,zb) + local f=io.open(filename,"rb") + if f then + zerobased[f]=zb or false + end + return f +end +function files.close(f) + zerobased[f]=nil + f:close() +end +function files.size(f) + local current=f:seek() + local size=f:seek("end") + f:seek("set",current) + return size +end +files.getsize=files.size +function files.setposition(f,n) + if zerobased[f] then + f:seek("set",n) + else + f:seek("set",n-1) + end +end +function files.getposition(f) + if zerobased[f] then + return f:seek() + else + return f:seek()+1 + end +end +function files.look(f,n,chars) + local p=f:seek() + local s=f:read(n) + f:seek("set",p) + if chars then + return s + else + return byte(s,1,#s) + end +end +function files.skip(f,n) + if n==1 then + f:read(n) + else + f:seek("set",f:seek()+n) + end +end +function files.readbyte(f) + return byte(f:read(1)) +end +function files.readbytes(f,n) + return byte(f:read(n),1,n) +end +function files.readbytetable(f,n) + local s=f:read(n or 1) + return { byte(s,1,#s) } +end +function files.readchar(f) + return f:read(1) +end +function files.readstring(f,n) + return f:read(n or 1) +end +function files.readinteger1(f) + local n=byte(f:read(1)) + if n>=0x80 then + return n-0x100 + else + return n + end +end +files.readcardinal1=files.readbyte +files.readcardinal=files.readcardinal1 +files.readinteger=files.readinteger1 +files.readsignedbyte=files.readinteger1 +function files.readcardinal2(f) + local a,b=byte(f:read(2),1,2) + return 0x100*a+b +end +function files.readcardinal2le(f) + local b,a=byte(f:read(2),1,2) + return 0x100*a+b +end +function files.readinteger2(f) + local a,b=byte(f:read(2),1,2) + if a>=0x80 then + return 0x100*a+b-0x10000 + else + return 0x100*a+b + end +end +function files.readinteger2le(f) + local b,a=byte(f:read(2),1,2) + if a>=0x80 then + return 0x100*a+b-0x10000 + else + return 0x100*a+b + end +end +function files.readcardinal3(f) + local a,b,c=byte(f:read(3),1,3) + return 0x10000*a+0x100*b+c +end +function files.readcardinal3le(f) + local c,b,a=byte(f:read(3),1,3) + return 0x10000*a+0x100*b+c +end +function files.readinteger3(f) + local a,b,c=byte(f:read(3),1,3) + if a>=0x80 then + return 0x10000*a+0x100*b+c-0x1000000 + else + return 0x10000*a+0x100*b+c + end +end +function files.readinteger3le(f) + local c,b,a=byte(f:read(3),1,3) + if a>=0x80 then + return 0x10000*a+0x100*b+c-0x1000000 + else + return 0x10000*a+0x100*b+c + end +end +function files.readcardinal4(f) + local a,b,c,d=byte(f:read(4),1,4) + return 0x1000000*a+0x10000*b+0x100*c+d +end +function files.readcardinal4le(f) + local d,c,b,a=byte(f:read(4),1,4) + return 0x1000000*a+0x10000*b+0x100*c+d +end +function files.readinteger4(f) + local a,b,c,d=byte(f:read(4),1,4) + if a>=0x80 then + return 0x1000000*a+0x10000*b+0x100*c+d-0x100000000 + else + return 0x1000000*a+0x10000*b+0x100*c+d + end +end +function files.readinteger4le(f) + local d,c,b,a=byte(f:read(4),1,4) + if a>=0x80 then + return 0x1000000*a+0x10000*b+0x100*c+d-0x100000000 + else + return 0x1000000*a+0x10000*b+0x100*c+d + end +end +function files.readfixed2(f) + local a,b=byte(f:read(2),1,2) + if a>=0x80 then + return (a-0x100)+b/0x100 + else + return (a )+b/0x100 + end +end +function files.readfixed4(f) + local a,b,c,d=byte(f:read(4),1,4) + if a>=0x80 then + return (0x100*a+b-0x10000)+(0x100*c+d)/0x10000 + else + return (0x100*a+b )+(0x100*c+d)/0x10000 + end +end +if bit32 then + local extract=bit32.extract + local band=bit32.band + function files.read2dot14(f) + local a,b=byte(f:read(2),1,2) + if a>=0x80 then + local n=-(0x100*a+b) + return-(extract(n,14,2)+(band(n,0x3FFF)/16384.0)) + else + local n=0x100*a+b + return (extract(n,14,2)+(band(n,0x3FFF)/16384.0)) + end + end +end +function files.skipshort(f,n) + f:read(2*(n or 1)) +end +function files.skiplong(f,n) + f:read(4*(n or 1)) +end +if bit32 then + local rshift=bit32.rshift + function files.writecardinal2(f,n) + local a=char(n%256) + n=rshift(n,8) + local b=char(n%256) + f:write(b,a) + end +else + local floor=math.floor + function files.writecardinal2(f,n) + local a=char(n%256) + n=floor(n/256) + local b=char(n%256) + f:write(b,a) + end +end +function files.writecardinal4(f,n) + local a=char(n%256) + n=rshift(n,8) + local b=char(n%256) + n=rshift(n,8) + local c=char(n%256) + n=rshift(n,8) + local d=char(n%256) + f:write(d,c,b,a) +end +function files.writestring(f,s) + f:write(char(byte(s,1,#s))) +end +function files.writebyte(f,b) + f:write(char(b)) +end +if fio and fio.readcardinal1 then + files.readcardinal1=fio.readcardinal1 + files.readcardinal2=fio.readcardinal2 + files.readcardinal3=fio.readcardinal3 + files.readcardinal4=fio.readcardinal4 + files.readinteger1=fio.readinteger1 + files.readinteger2=fio.readinteger2 + files.readinteger3=fio.readinteger3 + files.readinteger4=fio.readinteger4 + files.readfixed2=fio.readfixed2 + files.readfixed4=fio.readfixed4 + files.read2dot14=fio.read2dot14 + files.setposition=fio.setposition + files.getposition=fio.getposition + files.readbyte=files.readcardinal1 + files.readsignedbyte=files.readinteger1 + files.readcardinal=files.readcardinal1 + files.readinteger=files.readinteger1 + local skipposition=fio.skipposition + files.skipposition=skipposition + files.readbytes=fio.readbytes + files.readbytetable=fio.readbytetable + function files.skipshort(f,n) + skipposition(f,2*(n or 1)) + end + function files.skiplong(f,n) + skipposition(f,4*(n or 1)) + end +end +if fio and fio.readcardinaltable then + files.readcardinaltable=fio.readcardinaltable + files.readintegertable=fio.readintegertable +else + local readcardinal1=files.readcardinal1 + local readcardinal2=files.readcardinal2 + local readcardinal3=files.readcardinal3 + local readcardinal4=files.readcardinal4 + function files.readcardinaltable(f,n,b) + local t={} + if b==1 then for i=1,n do t[i]=readcardinal1(f) end + elseif b==2 then for i=1,n do t[i]=readcardinal2(f) end + elseif b==3 then for i=1,n do t[i]=readcardinal3(f) end + elseif b==4 then for i=1,n do t[i]=readcardinal4(f) end end + return t + end + local readinteger1=files.readinteger1 + local readinteger2=files.readinteger2 + local readinteger3=files.readinteger3 + local readinteger4=files.readinteger4 + function files.readintegertable(f,n,b) + local t={} + if b==1 then for i=1,n do t[i]=readinteger1(f) end + elseif b==2 then for i=1,n do t[i]=readinteger2(f) end + elseif b==3 then for i=1,n do t[i]=readinteger3(f) end + elseif b==4 then for i=1,n do t[i]=readinteger4(f) end end + return t + end +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['util-tab']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +utilities=utilities or {} +utilities.tables=utilities.tables or {} +local tables=utilities.tables +local format,gmatch,gsub,sub=string.format,string.gmatch,string.gsub,string.sub +local concat,insert,remove,sort=table.concat,table.insert,table.remove,table.sort +local setmetatable,getmetatable,tonumber,tostring,rawget=setmetatable,getmetatable,tonumber,tostring,rawget +local type,next,rawset,tonumber,tostring,load,select=type,next,rawset,tonumber,tostring,load,select +local lpegmatch,P,Cs,Cc=lpeg.match,lpeg.P,lpeg.Cs,lpeg.Cc +local sortedkeys,sortedpairs=table.sortedkeys,table.sortedpairs +local formatters=string.formatters +local utftoeight=utf.toeight +local splitter=lpeg.tsplitat(".") +function utilities.tables.definetable(target,nofirst,nolast) + local composed=nil + local t={} + local snippets=lpegmatch(splitter,target) + for i=1,#snippets-(nolast and 1 or 0) do + local name=snippets[i] + if composed then + composed=composed.."."..name + t[#t+1]=formatters["if not %s then %s = { } end"](composed,composed) + else + composed=name + if not nofirst then + t[#t+1]=formatters["%s = %s or { }"](composed,composed) + end + end + end + if composed then + if nolast then + composed=composed.."."..snippets[#snippets] + end + return concat(t,"\n"),composed + else + return "",target + end +end +function tables.definedtable(...) + local t=_G + for i=1,select("#",...) do + local li=select(i,...) + local tl=t[li] + if not tl then + tl={} + t[li]=tl + end + t=tl + end + return t +end +function tables.accesstable(target,root) + local t=root or _G + for name in gmatch(target,"([^%.]+)") do + t=t[name] + if not t then + return + end + end + return t +end +function tables.migratetable(target,v,root) + local t=root or _G + local names=lpegmatch(splitter,target) + for i=1,#names-1 do + local name=names[i] + t[name]=t[name] or {} + t=t[name] + if not t then + return + end + end + t[names[#names]]=v +end +function tables.removevalue(t,value) + if value then + for i=1,#t do + if t[i]==value then + remove(t,i) + end + end + end +end +function tables.replacevalue(t,oldvalue,newvalue) + if oldvalue and newvalue then + for i=1,#t do + if t[i]==oldvalue then + t[i]=newvalue + end + end + end +end +function tables.insertbeforevalue(t,value,extra) + for i=1,#t do + if t[i]==extra then + remove(t,i) + end + end + for i=1,#t do + if t[i]==value then + insert(t,i,extra) + return + end + end + insert(t,1,extra) +end +function tables.insertaftervalue(t,value,extra) + for i=1,#t do + if t[i]==extra then + remove(t,i) + end + end + for i=1,#t do + if t[i]==value then + insert(t,i+1,extra) + return + end + end + insert(t,#t+1,extra) +end +local escape=Cs(Cc('"')*((P('"')/'""'+P(1))^0)*Cc('"')) +function table.tocsv(t,specification) + if t and #t>0 then + local result={} + local r={} + specification=specification or {} + local fields=specification.fields + if type(fields)~="string" then + fields=sortedkeys(t[1]) + end + local separator=specification.separator or "," + local noffields=#fields + if specification.preamble==true then + for f=1,noffields do + r[f]=lpegmatch(escape,tostring(fields[f])) + end + result[1]=concat(r,separator) + end + for i=1,#t do + local ti=t[i] + for f=1,noffields do + local field=ti[fields[f]] + if type(field)=="string" then + r[f]=lpegmatch(escape,field) + else + r[f]=tostring(field) + end + end + result[i+1]=concat(r,separator) + end + return concat(result,"\n") + else + return "" + end +end +local nspaces=utilities.strings.newrepeater(" ") +local function toxml(t,d,result,step) + local r=#result + for k,v in sortedpairs(t) do + local s=nspaces[d] + local tk=type(k) + local tv=type(v) + if tv=="table" then + if tk=="number" then + r=r+1 result[r]=formatters["%s<entry n='%s'>"](s,k) + toxml(v,d+step,result,step) + r=r+1 result[r]=formatters["%s</entry>"](s,k) + else + r=r+1 result[r]=formatters["%s<%s>"](s,k) + toxml(v,d+step,result,step) + r=r+1 result[r]=formatters["%s</%s>"](s,k) + end + elseif tv=="string" then + if tk=="number" then + r=r+1 result[r]=formatters["%s<entry n='%s'>%!xml!</entry>"](s,k,v,k) + else + r=r+1 result[r]=formatters["%s<%s>%!xml!</%s>"](s,k,v,k) + end + elseif tk=="number" then + r=r+1 result[r]=formatters["%s<entry n='%s'>%S</entry>"](s,k,v,k) + else + r=r+1 result[r]=formatters["%s<%s>%S</%s>"](s,k,v,k) + end + end +end +function table.toxml(t,specification) + specification=specification or {} + local name=specification.name + local noroot=name==false + local result=(specification.nobanner or noroot) and {} or { "<?xml version='1.0' standalone='yes' ?>" } + local indent=specification.indent or 0 + local spaces=specification.spaces or 1 + if noroot then + toxml(t,indent,result,spaces) + else + toxml({ [name or "data"]=t },indent,result,spaces) + end + return concat(result,"\n") +end +function tables.encapsulate(core,capsule,protect) + if type(capsule)~="table" then + protect=true + capsule={} + end + for key,value in next,core do + if capsule[key] then + print(formatters["\ninvalid %s %a in %a"]("inheritance",key,core)) + os.exit() + else + capsule[key]=value + end + end + if protect then + for key,value in next,core do + core[key]=nil + end + setmetatable(core,{ + __index=capsule, + __newindex=function(t,key,value) + if capsule[key] then + print(formatters["\ninvalid %s %a' in %a"]("overload",key,core)) + os.exit() + else + rawset(t,key,value) + end + end + } ) + end +end +local f_hashed_string=formatters["[%Q]=%Q,"] +local f_hashed_number=formatters["[%Q]=%s,"] +local f_hashed_boolean=formatters["[%Q]=%l,"] +local f_hashed_table=formatters["[%Q]="] +local f_indexed_string=formatters["[%s]=%Q,"] +local f_indexed_number=formatters["[%s]=%s,"] +local f_indexed_boolean=formatters["[%s]=%l,"] +local f_indexed_table=formatters["[%s]="] +local f_ordered_string=formatters["%Q,"] +local f_ordered_number=formatters["%s,"] +local f_ordered_boolean=formatters["%l,"] +function table.fastserialize(t,prefix) + local r={ type(prefix)=="string" and prefix or "return" } + local m=1 + local function fastserialize(t,outer) + local n=#t + m=m+1 + r[m]="{" + if n>0 then + for i=0,n do + local v=t[i] + local tv=type(v) + if tv=="string" then + m=m+1 r[m]=f_ordered_string(v) + elseif tv=="number" then + m=m+1 r[m]=f_ordered_number(v) + elseif tv=="table" then + fastserialize(v) + elseif tv=="boolean" then + m=m+1 r[m]=f_ordered_boolean(v) + end + end + end + for k,v in next,t do + local tk=type(k) + if tk=="number" then + if k>n or k<0 then + local tv=type(v) + if tv=="string" then + m=m+1 r[m]=f_indexed_string(k,v) + elseif tv=="number" then + m=m+1 r[m]=f_indexed_number(k,v) + elseif tv=="table" then + m=m+1 r[m]=f_indexed_table(k) + fastserialize(v) + elseif tv=="boolean" then + m=m+1 r[m]=f_indexed_boolean(k,v) + end + end + else + local tv=type(v) + if tv=="string" then + m=m+1 r[m]=f_hashed_string(k,v) + elseif tv=="number" then + m=m+1 r[m]=f_hashed_number(k,v) + elseif tv=="table" then + m=m+1 r[m]=f_hashed_table(k) + fastserialize(v) + elseif tv=="boolean" then + m=m+1 r[m]=f_hashed_boolean(k,v) + end + end + end + m=m+1 + if outer then + r[m]="}" + else + r[m]="}," + end + return r + end + return concat(fastserialize(t,true)) +end +function table.deserialize(str) + if not str or str=="" then + return + end + local code=load(str) + if not code then + return + end + code=code() + if not code then + return + end + return code +end +function table.load(filename,loader) + if filename then + local t=(loader or io.loaddata)(filename) + if t and t~="" then + local t=utftoeight(t) + t=load(t) + if type(t)=="function" then + t=t() + if type(t)=="table" then + return t + end + end + end + end +end +function table.save(filename,t,n,...) + io.savedata(filename,table.serialize(t,n==nil and true or n,...)) +end +local f_key_value=formatters["%s=%q"] +local f_add_table=formatters[" {%t},\n"] +local f_return_table=formatters["return {\n%t}"] +local function slowdrop(t) + local r={} + local l={} + for i=1,#t do + local ti=t[i] + local j=0 + for k,v in next,ti do + j=j+1 + l[j]=f_key_value(k,v) + end + r[i]=f_add_table(l) + end + return f_return_table(r) +end +local function fastdrop(t) + local r={ "return {\n" } + local m=1 + for i=1,#t do + local ti=t[i] + m=m+1 r[m]=" {" + for k,v in next,ti do + m=m+1 r[m]=f_key_value(k,v) + end + m=m+1 r[m]="},\n" + end + m=m+1 + r[m]="}" + return concat(r) +end +function table.drop(t,slow) + if #t==0 then + return "return { }" + elseif slow==true then + return slowdrop(t) + else + return fastdrop(t) + end +end +local selfmapper={ __index=function(t,k) t[k]=k return k end } +function table.twowaymapper(t) + if not t then + t={} + else + local zero=rawget(t,0) + for i=zero and 0 or 1,#t do + local ti=t[i] + if ti then + local i=tostring(i) + t[i]=ti + t[ti]=i + end + end + end + setmetatable(t,selfmapper) + return t +end +local f_start_key_idx=formatters["%w{"] +local f_start_key_num=formatters["%w[%s]={"] +local f_start_key_str=formatters["%w[%q]={"] +local f_start_key_boo=formatters["%w[%l]={"] +local f_start_key_nop=formatters["%w{"] +local f_stop=formatters["%w},"] +local f_key_num_value_num=formatters["%w[%s]=%s,"] +local f_key_str_value_num=formatters["%w[%Q]=%s,"] +local f_key_boo_value_num=formatters["%w[%l]=%s,"] +local f_key_num_value_str=formatters["%w[%s]=%Q,"] +local f_key_str_value_str=formatters["%w[%Q]=%Q,"] +local f_key_boo_value_str=formatters["%w[%l]=%Q,"] +local f_key_num_value_boo=formatters["%w[%s]=%l,"] +local f_key_str_value_boo=formatters["%w[%Q]=%l,"] +local f_key_boo_value_boo=formatters["%w[%l]=%l,"] +local f_key_num_value_not=formatters["%w[%s]={},"] +local f_key_str_value_not=formatters["%w[%Q]={},"] +local f_key_boo_value_not=formatters["%w[%l]={},"] +local f_key_num_value_seq=formatters["%w[%s]={ %, t },"] +local f_key_str_value_seq=formatters["%w[%Q]={ %, t },"] +local f_key_boo_value_seq=formatters["%w[%l]={ %, t },"] +local f_val_num=formatters["%w%s,"] +local f_val_str=formatters["%w%Q,"] +local f_val_boo=formatters["%w%l,"] +local f_val_not=formatters["%w{},"] +local f_val_seq=formatters["%w{ %, t },"] +local f_fin_seq=formatters[" %, t }"] +local f_table_return=formatters["return {"] +local f_table_name=formatters["%s={"] +local f_table_direct=formatters["{"] +local f_table_entry=formatters["[%Q]={"] +local f_table_finish=formatters["}"] +local spaces=utilities.strings.newrepeater(" ") +local original_serialize=table.serialize +local is_simple_table=table.is_simple_table +local function serialize(root,name,specification) + if type(specification)=="table" then + return original_serialize(root,name,specification) + end + local t + local n=1 + local unknown=false + local function do_serialize(root,name,depth,level,indexed) + if level>0 then + n=n+1 + if indexed then + t[n]=f_start_key_idx(depth) + else + local tn=type(name) + if tn=="number" then + t[n]=f_start_key_num(depth,name) + elseif tn=="string" then + t[n]=f_start_key_str(depth,name) + elseif tn=="boolean" then + t[n]=f_start_key_boo(depth,name) + else + t[n]=f_start_key_nop(depth) + end + end + depth=depth+1 + end + if root and next(root)~=nil then + local first=nil + local last=#root + if last>0 then + for k=1,last do + if rawget(root,k)==nil then + last=k-1 + break + end + end + if last>0 then + first=1 + end + end + local sk=sortedkeys(root) + for i=1,#sk do + local k=sk[i] + local v=root[k] + local tv=type(v) + local tk=type(k) + if first and tk=="number" and k<=last and k>=first then + if tv=="number" then + n=n+1 t[n]=f_val_num(depth,v) + elseif tv=="string" then + n=n+1 t[n]=f_val_str(depth,v) + elseif tv=="table" then + if next(v)==nil then + n=n+1 t[n]=f_val_not(depth) + else + local st=is_simple_table(v) + if st then + n=n+1 t[n]=f_val_seq(depth,st) + else + do_serialize(v,k,depth,level+1,true) + end + end + elseif tv=="boolean" then + n=n+1 t[n]=f_val_boo(depth,v) + elseif unknown then + n=n+1 t[n]=f_val_str(depth,tostring(v)) + end + elseif tv=="number" then + if tk=="number" then + n=n+1 t[n]=f_key_num_value_num(depth,k,v) + elseif tk=="string" then + n=n+1 t[n]=f_key_str_value_num(depth,k,v) + elseif tk=="boolean" then + n=n+1 t[n]=f_key_boo_value_num(depth,k,v) + elseif unknown then + n=n+1 t[n]=f_key_str_value_num(depth,tostring(k),v) + end + elseif tv=="string" then + if tk=="number" then + n=n+1 t[n]=f_key_num_value_str(depth,k,v) + elseif tk=="string" then + n=n+1 t[n]=f_key_str_value_str(depth,k,v) + elseif tk=="boolean" then + n=n+1 t[n]=f_key_boo_value_str(depth,k,v) + elseif unknown then + n=n+1 t[n]=f_key_str_value_str(depth,tostring(k),v) + end + elseif tv=="table" then + if next(v)==nil then + if tk=="number" then + n=n+1 t[n]=f_key_num_value_not(depth,k) + elseif tk=="string" then + n=n+1 t[n]=f_key_str_value_not(depth,k) + elseif tk=="boolean" then + n=n+1 t[n]=f_key_boo_value_not(depth,k) + elseif unknown then + n=n+1 t[n]=f_key_str_value_not(depth,tostring(k)) + end + else + local st=is_simple_table(v) + if not st then + do_serialize(v,k,depth,level+1) + elseif tk=="number" then + n=n+1 t[n]=f_key_num_value_seq(depth,k,st) + elseif tk=="string" then + n=n+1 t[n]=f_key_str_value_seq(depth,k,st) + elseif tk=="boolean" then + n=n+1 t[n]=f_key_boo_value_seq(depth,k,st) + elseif unknown then + n=n+1 t[n]=f_key_str_value_seq(depth,tostring(k),st) + end + end + elseif tv=="boolean" then + if tk=="number" then + n=n+1 t[n]=f_key_num_value_boo(depth,k,v) + elseif tk=="string" then + n=n+1 t[n]=f_key_str_value_boo(depth,k,v) + elseif tk=="boolean" then + n=n+1 t[n]=f_key_boo_value_boo(depth,k,v) + elseif unknown then + n=n+1 t[n]=f_key_str_value_boo(depth,tostring(k),v) + end + else + if tk=="number" then + n=n+1 t[n]=f_key_num_value_str(depth,k,tostring(v)) + elseif tk=="string" then + n=n+1 t[n]=f_key_str_value_str(depth,k,tostring(v)) + elseif tk=="boolean" then + n=n+1 t[n]=f_key_boo_value_str(depth,k,tostring(v)) + elseif unknown then + n=n+1 t[n]=f_key_str_value_str(depth,tostring(k),tostring(v)) + end + end + end + end + if level>0 then + n=n+1 t[n]=f_stop(depth-1) + end + end + local tname=type(name) + if tname=="string" then + if name=="return" then + t={ f_table_return() } + else + t={ f_table_name(name) } + end + elseif tname=="number" then + t={ f_table_entry(name) } + elseif tname=="boolean" then + if name then + t={ f_table_return() } + else + t={ f_table_direct() } + end + else + t={ f_table_name("t") } + end + if root then + if getmetatable(root) then + local dummy=root._w_h_a_t_e_v_e_r_ + root._w_h_a_t_e_v_e_r_=nil + end + if next(root)~=nil then + local st=is_simple_table(root) + if st then + return t[1]..f_fin_seq(st) + else + do_serialize(root,name,1,0) + end + end + end + n=n+1 + t[n]=f_table_finish() + return concat(t,"\n") +end +table.serialize=serialize +if setinspector then + setinspector("table",function(v) + if type(v)=="table" then + print(serialize(v,"table",{ metacheck=false })) + return true + end + end) +end +local mt={ + __newindex=function(t,k,v) + local n=t.last+1 + t.last=n + t.list[n]=k + t.hash[k]=v + end, + __index=function(t,k) + return t.hash[k] + end, + __len=function(t) + return t.last + end, +} +function table.orderedhash() + return setmetatable({ list={},hash={},last=0 },mt) +end +function table.ordered(t) + local n=t.last + if n>0 then + local l=t.list + local i=1 + local h=t.hash + local f=function() + if i<=n then + local k=i + local v=h[l[k]] + i=i+1 + return k,v + end + end + return f,1,h[l[1]] + else + return function() end + end +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['util-sto']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local setmetatable,getmetatable,rawset,type=setmetatable,getmetatable,rawset,type +utilities=utilities or {} +utilities.storage=utilities.storage or {} +local storage=utilities.storage +function storage.mark(t) + if not t then + print("\nfatal error: storage cannot be marked\n") + os.exit() + return + end + local m=getmetatable(t) + if not m then + m={} + setmetatable(t,m) + end + m.__storage__=true + return t +end +function storage.allocate(t) + t=t or {} + local m=getmetatable(t) + if not m then + m={} + setmetatable(t,m) + end + m.__storage__=true + return t +end +function storage.marked(t) + local m=getmetatable(t) + return m and m.__storage__ +end +function storage.checked(t) + if not t then + report("\nfatal error: storage has not been allocated\n") + os.exit() + return + end + return t +end +function storage.setinitializer(data,initialize) + local m=getmetatable(data) or {} + m.__index=function(data,k) + m.__index=nil + initialize() + return data[k] + end + setmetatable(data,m) +end +local keyisvalue={ __index=function(t,k) + t[k]=k + return k +end } +function storage.sparse(t) + t=t or {} + setmetatable(t,keyisvalue) + return t +end +local function f_empty () return "" end +local function f_self (t,k) t[k]=k return k end +local function f_table (t,k) local v={} t[k]=v return v end +local function f_number(t,k) t[k]=0 return 0 end +local function f_ignore() end +local f_index={ + ["empty"]=f_empty, + ["self"]=f_self, + ["table"]=f_table, + ["number"]=f_number, +} +function table.setmetatableindex(t,f) + if type(t)~="table" then + f,t=t,{} + end + local m=getmetatable(t) + local i=f_index[f] or f + if m then + m.__index=i + else + setmetatable(t,{ __index=i }) + end + return t +end +local f_index={ + ["ignore"]=f_ignore, +} +function table.setmetatablenewindex(t,f) + if type(t)~="table" then + f,t=t,{} + end + local m=getmetatable(t) + local i=f_index[f] or f + if m then + m.__newindex=i + else + setmetatable(t,{ __newindex=i }) + end + return t +end +function table.setmetatablecall(t,f) + if type(t)~="table" then + f,t=t,{} + end + local m=getmetatable(t) + if m then + m.__call=f + else + setmetatable(t,{ __call=f }) + end + return t +end +function table.setmetatableindices(t,f,n,c) + if type(t)~="table" then + f,t=t,{} + end + local m=getmetatable(t) + local i=f_index[f] or f + if m then + m.__index=i + m.__newindex=n + m.__call=c + else + setmetatable(t,{ + __index=i, + __newindex=n, + __call=c, + }) + end + return t +end +function table.setmetatablekey(t,key,value) + local m=getmetatable(t) + if not m then + m={} + setmetatable(t,m) + end + m[key]=value + return t +end +function table.getmetatablekey(t,key,value) + local m=getmetatable(t) + return m and m[key] +end +function table.makeweak(t) + if not t then + t={} + end + local m=getmetatable(t) + if m then + m.__mode="v" + else + setmetatable(t,{ __mode="v" }) + end + return t +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['util-prs']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local lpeg,table,string=lpeg,table,string +local P,R,V,S,C,Ct,Cs,Carg,Cc,Cg,Cf,Cp=lpeg.P,lpeg.R,lpeg.V,lpeg.S,lpeg.C,lpeg.Ct,lpeg.Cs,lpeg.Carg,lpeg.Cc,lpeg.Cg,lpeg.Cf,lpeg.Cp +local lpegmatch,lpegpatterns=lpeg.match,lpeg.patterns +local concat,gmatch,find=table.concat,string.gmatch,string.find +local tostring,type,next,rawset=tostring,type,next,rawset +local mod,div=math.mod,math.div +utilities=utilities or {} +local parsers=utilities.parsers or {} +utilities.parsers=parsers +local patterns=parsers.patterns or {} +parsers.patterns=patterns +local setmetatableindex=table.setmetatableindex +local sortedhash=table.sortedhash +local sortedkeys=table.sortedkeys +local tohash=table.tohash +local hashes={} +parsers.hashes=hashes +local digit=R("09") +local space=P(' ') +local equal=P("=") +local colon=P(":") +local comma=P(",") +local lbrace=P("{") +local rbrace=P("}") +local lparent=P("(") +local rparent=P(")") +local lbracket=P("[") +local rbracket=P("]") +local period=S(".") +local punctuation=S(".,:;") +local spacer=lpegpatterns.spacer +local whitespace=lpegpatterns.whitespace +local newline=lpegpatterns.newline +local anything=lpegpatterns.anything +local endofstring=lpegpatterns.endofstring +local nobrace=1-(lbrace+rbrace ) +local noparent=1-(lparent+rparent) +local nobracket=1-(lbracket+rbracket) +local escape,left,right=P("\\"),P('{'),P('}') +lpegpatterns.balanced=P { + [1]=((escape*(left+right))+(1-(left+right))+V(2))^0, + [2]=left*V(1)*right +} +local nestedbraces=P { lbrace*(nobrace+V(1))^0*rbrace } +local nestedparents=P { lparent*(noparent+V(1))^0*rparent } +local nestedbrackets=P { lbracket*(nobracket+V(1))^0*rbracket } +local spaces=space^0 +local argument=Cs((lbrace/"")*((nobrace+nestedbraces)^0)*(rbrace/"")) +local content=(1-endofstring)^0 +lpegpatterns.nestedbraces=nestedbraces +lpegpatterns.nestedparents=nestedparents +lpegpatterns.nested=nestedbraces +lpegpatterns.argument=argument +lpegpatterns.content=content +local value=lbrace*C((nobrace+nestedbraces)^0)*rbrace+C((nestedbraces+(1-comma))^0) +local key=C((1-equal-comma)^1) +local pattern_a=(space+comma)^0*(key*equal*value+key*C("")) +local pattern_c=(space+comma)^0*(key*equal*value) +local pattern_d=(space+comma)^0*(key*(equal+colon)*value+key*C("")) +local key=C((1-space-equal-comma)^1) +local pattern_b=spaces*comma^0*spaces*(key*((spaces*equal*spaces*value)+C(""))) +local hash={} +local function set(key,value) + hash[key]=value +end +local pattern_a_s=(pattern_a/set)^1 +local pattern_b_s=(pattern_b/set)^1 +local pattern_c_s=(pattern_c/set)^1 +local pattern_d_s=(pattern_d/set)^1 +patterns.settings_to_hash_a=pattern_a_s +patterns.settings_to_hash_b=pattern_b_s +patterns.settings_to_hash_c=pattern_c_s +patterns.settings_to_hash_d=pattern_d_s +function parsers.make_settings_to_hash_pattern(set,how) + if how=="strict" then + return (pattern_c/set)^1 + elseif how=="tolerant" then + return (pattern_b/set)^1 + else + return (pattern_a/set)^1 + end +end +function parsers.settings_to_hash(str,existing) + if not str or str=="" then + return {} + elseif type(str)=="table" then + if existing then + for k,v in next,str do + existing[k]=v + end + return exiting + else + return str + end + else + hash=existing or {} + lpegmatch(pattern_a_s,str) + return hash + end +end +function parsers.settings_to_hash_colon_too(str) + if not str or str=="" then + return {} + elseif type(str)=="table" then + return str + else + hash={} + lpegmatch(pattern_d_s,str) + return hash + end +end +function parsers.settings_to_hash_tolerant(str,existing) + if not str or str=="" then + return {} + elseif type(str)=="table" then + if existing then + for k,v in next,str do + existing[k]=v + end + return exiting + else + return str + end + else + hash=existing or {} + lpegmatch(pattern_b_s,str) + return hash + end +end +function parsers.settings_to_hash_strict(str,existing) + if not str or str=="" then + return nil + elseif type(str)=="table" then + if existing then + for k,v in next,str do + existing[k]=v + end + return exiting + else + return str + end + elseif str and str~="" then + hash=existing or {} + lpegmatch(pattern_c_s,str) + return next(hash) and hash + end +end +local separator=comma*space^0 +local value=lbrace*C((nobrace+nestedbraces)^0)*rbrace+C((nestedbraces+(1-comma))^0) +local pattern=spaces*Ct(value*(separator*value)^0) +patterns.settings_to_array=pattern +function parsers.settings_to_array(str,strict) + if not str or str=="" then + return {} + elseif type(str)=="table" then + return str + elseif strict then + if find(str,"{",1,true) then + return lpegmatch(pattern,str) + else + return { str } + end + elseif find(str,",",1,true) then + return lpegmatch(pattern,str) + else + return { str } + end +end +function parsers.settings_to_numbers(str) + if not str or str=="" then + return {} + end + if type(str)=="table" then + elseif find(str,",",1,true) then + str=lpegmatch(pattern,str) + else + return { tonumber(str) } + end + for i=1,#str do + str[i]=tonumber(str[i]) + end + return str +end +local value=lbrace*C((nobrace+nestedbraces)^0)*rbrace+C((nestedbraces+nestedbrackets+nestedparents+(1-comma))^0) +local pattern=spaces*Ct(value*(separator*value)^0) +function parsers.settings_to_array_obey_fences(str) + return lpegmatch(pattern,str) +end +local cache_a={} +local cache_b={} +function parsers.groupedsplitat(symbol,withaction) + if not symbol then + symbol="," + end + local pattern=(withaction and cache_b or cache_a)[symbol] + if not pattern then + local symbols=S(symbol) + local separator=space^0*symbols*space^0 + local value=lbrace*C((nobrace+nestedbraces)^0)*rbrace+C((nestedbraces+(1-(space^0*(symbols+P(-1)))))^0) + if withaction then + local withvalue=Carg(1)*value/function(f,s) return f(s) end + pattern=spaces*withvalue*(separator*withvalue)^0 + cache_b[symbol]=pattern + else + pattern=spaces*Ct(value*(separator*value)^0) + cache_a[symbol]=pattern + end + end + return pattern +end +local pattern_a=parsers.groupedsplitat(",",false) +local pattern_b=parsers.groupedsplitat(",",true) +function parsers.stripped_settings_to_array(str) + if not str or str=="" then + return {} + else + return lpegmatch(pattern_a,str) + end +end +function parsers.process_stripped_settings(str,action) + if not str or str=="" then + return {} + else + return lpegmatch(pattern_b,str,1,action) + end +end +local function set(t,v) + t[#t+1]=v +end +local value=P(Carg(1)*value)/set +local pattern=value*(separator*value)^0*Carg(1) +function parsers.add_settings_to_array(t,str) + return lpegmatch(pattern,str,nil,t) +end +function parsers.hash_to_string(h,separator,yes,no,strict,omit) + if h then + local t={} + local tn=0 + local s=sortedkeys(h) + omit=omit and tohash(omit) + for i=1,#s do + local key=s[i] + if not omit or not omit[key] then + local value=h[key] + if type(value)=="boolean" then + if yes and no then + if value then + tn=tn+1 + t[tn]=key..'='..yes + elseif not strict then + tn=tn+1 + t[tn]=key..'='..no + end + elseif value or not strict then + tn=tn+1 + t[tn]=key..'='..tostring(value) + end + else + tn=tn+1 + t[tn]=key..'='..value + end + end + end + return concat(t,separator or ",") + else + return "" + end +end +function parsers.array_to_string(a,separator) + if a then + return concat(a,separator or ",") + else + return "" + end +end +local pattern=Cf(Ct("")*Cg(C((1-S(", "))^1)*S(", ")^0*Cc(true))^1,rawset) +function parsers.settings_to_set(str) + return str and lpegmatch(pattern,str) or {} +end +hashes.settings_to_set=table.setmetatableindex(function(t,k) + local v=k and lpegmatch(pattern,k) or {} + t[k]=v + return v +end) +getmetatable(hashes.settings_to_set).__mode="kv" +function parsers.simple_hash_to_string(h,separator) + local t={} + local tn=0 + for k,v in sortedhash(h) do + if v then + tn=tn+1 + t[tn]=k + end + end + return concat(t,separator or ",") +end +local str=Cs(lpegpatterns.unquoted)+C((1-whitespace-equal)^1) +local setting=Cf(Carg(1)*(whitespace^0*Cg(str*whitespace^0*(equal*whitespace^0*str+Cc(""))))^1,rawset) +local splitter=setting^1 +function parsers.options_to_hash(str,target) + return str and lpegmatch(splitter,str,1,target or {}) or {} +end +local splitter=lpeg.tsplitat(" ") +function parsers.options_to_array(str) + return str and lpegmatch(splitter,str) or {} +end +local value=P(lbrace*C((nobrace+nestedbraces)^0)*rbrace)+C(digit^1*lparent*(noparent+nestedparents)^1*rparent)+C((nestedbraces+(1-comma))^1) +local pattern_a=spaces*Ct(value*(separator*value)^0) +local function repeater(n,str) + if not n then + return str + else + local s=lpegmatch(pattern_a,str) + if n==1 then + return unpack(s) + else + local t={} + local tn=0 + for i=1,n do + for j=1,#s do + tn=tn+1 + t[tn]=s[j] + end + end + return unpack(t) + end + end +end +local value=P(lbrace*C((nobrace+nestedbraces)^0)*rbrace)+(C(digit^1)/tonumber*lparent*Cs((noparent+nestedparents)^1)*rparent)/repeater+C((nestedbraces+(1-comma))^1) +local pattern_b=spaces*Ct(value*(separator*value)^0) +function parsers.settings_to_array_with_repeat(str,expand) + if expand then + return lpegmatch(pattern_b,str) or {} + else + return lpegmatch(pattern_a,str) or {} + end +end +local value=lbrace*C((nobrace+nestedbraces)^0)*rbrace +local pattern=Ct((space+value)^0) +function parsers.arguments_to_table(str) + return lpegmatch(pattern,str) +end +function parsers.getparameters(self,class,parentclass,settings) + local sc=self[class] + if not sc then + sc={} + self[class]=sc + if parentclass then + local sp=self[parentclass] + if not sp then + sp={} + self[parentclass]=sp + end + setmetatableindex(sc,sp) + end + end + parsers.settings_to_hash(settings,sc) +end +function parsers.listitem(str) + return gmatch(str,"[^, ]+") +end +local pattern=Cs { "start", + start=V("one")+V("two")+V("three"), + rest=(Cc(",")*V("thousand"))^0*(P(".")+endofstring)*anything^0, + thousand=digit*digit*digit, + one=digit*V("rest"), + two=digit*digit*V("rest"), + three=V("thousand")*V("rest"), +} +lpegpatterns.splitthousands=pattern +function parsers.splitthousands(str) + return lpegmatch(pattern,str) or str +end +local optionalwhitespace=whitespace^0 +lpegpatterns.words=Ct((Cs((1-punctuation-whitespace)^1)+anything)^1) +lpegpatterns.sentences=Ct((optionalwhitespace*Cs((1-period)^0*period))^1) +lpegpatterns.paragraphs=Ct((optionalwhitespace*Cs((whitespace^1*endofstring/""+1-(spacer^0*newline*newline))^1))^1) +local dquote=P('"') +local equal=P('=') +local escape=P('\\') +local separator=S(' ,') +local key=C((1-equal)^1) +local value=dquote*C((1-dquote-escape*dquote)^0)*dquote +local pattern=Cf(Ct("")*(Cg(key*equal*value)*separator^0)^1,rawset)^0*P(-1) +function parsers.keq_to_hash(str) + if str and str~="" then + return lpegmatch(pattern,str) + else + return {} + end +end +local defaultspecification={ separator=",",quote='"' } +function parsers.csvsplitter(specification) + specification=specification and setmetatableindex(specification,defaultspecification) or defaultspecification + local separator=specification.separator + local quotechar=specification.quote + local separator=S(separator~="" and separator or ",") + local whatever=C((1-separator-newline)^0) + if quotechar and quotechar~="" then + local quotedata=nil + for chr in gmatch(quotechar,".") do + local quotechar=P(chr) + local quoteword=quotechar*C((1-quotechar)^0)*quotechar + if quotedata then + quotedata=quotedata+quoteword + else + quotedata=quoteword + end + end + whatever=quotedata+whatever + end + local parser=Ct((Ct(whatever*(separator*whatever)^0)*S("\n\r")^1)^0 ) + return function(data) + return lpegmatch(parser,data) + end +end +function parsers.rfc4180splitter(specification) + specification=specification and setmetatableindex(specification,defaultspecification) or defaultspecification + local separator=specification.separator + local quotechar=P(specification.quote) + local dquotechar=quotechar*quotechar +/specification.quote + local separator=S(separator~="" and separator or ",") + local escaped=quotechar*Cs((dquotechar+(1-quotechar))^0)*quotechar + local non_escaped=C((1-quotechar-newline-separator)^1) + local field=escaped+non_escaped+Cc("") + local record=Ct(field*(separator*field)^1) + local headerline=record*Cp() + local morerecords=(newline^(specification.strict and -1 or 1)*record)^0 + local headeryes=Ct(morerecords) + local headernop=Ct(record*morerecords) + return function(data,getheader) + if getheader then + local header,position=lpegmatch(headerline,data) + local data=lpegmatch(headeryes,data,position) + return data,header + else + return lpegmatch(headernop,data) + end + end +end +local function ranger(first,last,n,action) + if not first then + elseif last==true then + for i=first,n or first do + action(i) + end + elseif last then + for i=first,last do + action(i) + end + else + action(first) + end +end +local cardinal=lpegpatterns.cardinal/tonumber +local spacers=lpegpatterns.spacer^0 +local endofstring=lpegpatterns.endofstring +local stepper=spacers*(cardinal*(spacers*S(":-")*spacers*(cardinal+Cc(true) )+Cc(false) )*Carg(1)*Carg(2)/ranger*S(", ")^0 )^1 +local stepper=spacers*(cardinal*(spacers*S(":-")*spacers*(cardinal+(P("*")+endofstring)*Cc(true) )+Cc(false) )*Carg(1)*Carg(2)/ranger*S(", ")^0 )^1*endofstring +function parsers.stepper(str,n,action) + if type(n)=="function" then + lpegmatch(stepper,str,1,false,n or print) + else + lpegmatch(stepper,str,1,n,action or print) + end +end +local pattern_math=Cs((P("%")/"\\percent "+P("^")*Cc("{")*lpegpatterns.integer*Cc("}")+anything)^0) +local pattern_text=Cs((P("%")/"\\percent "+(P("^")/"\\high")*Cc("{")*lpegpatterns.integer*Cc("}")+anything)^0) +patterns.unittotex=pattern +function parsers.unittotex(str,textmode) + return lpegmatch(textmode and pattern_text or pattern_math,str) +end +local pattern=Cs((P("^")/"<sup>"*lpegpatterns.integer*Cc("</sup>")+anything)^0) +function parsers.unittoxml(str) + return lpegmatch(pattern,str) +end +local cache={} +local spaces=lpegpatterns.space^0 +local dummy=function() end +setmetatableindex(cache,function(t,k) + local separator=P(k) + local value=(1-separator)^0 + local pattern=spaces*C(value)*separator^0*Cp() + t[k]=pattern + return pattern +end) +local commalistiterator=cache[","] +function parsers.iterator(str,separator) + local n=#str + if n==0 then + return dummy + else + local pattern=separator and cache[separator] or commalistiterator + local p=1 + return function() + if p<=n then + local s,e=lpegmatch(pattern,str,p) + if e then + p=e + return s + end + end + end + end +end +local function initialize(t,name) + local source=t[name] + if source then + local result={} + for k,v in next,t[name] do + result[k]=v + end + return result + else + return {} + end +end +local function fetch(t,name) + return t[name] or {} +end +local function process(result,more) + for k,v in next,more do + result[k]=v + end + return result +end +local name=C((1-S(", "))^1) +local parser=(Carg(1)*name/initialize)*(S(", ")^1*(Carg(1)*name/fetch))^0 +local merge=Cf(parser,process) +function parsers.mergehashes(hash,list) + return lpegmatch(merge,list,1,hash) +end +function parsers.runtime(time) + if not time then + time=os.runtime() + end + local days=div(time,24*60*60) + time=mod(time,24*60*60) + local hours=div(time,60*60) + time=mod(time,60*60) + local minutes=div(time,60) + local seconds=mod(time,60) + return days,hours,minutes,seconds +end +local spacing=whitespace^0 +local apply=P("->") +local method=C((1-apply)^1) +local token=lbrace*C((1-rbrace)^1)*rbrace+C(anything^1) +local pattern=spacing*(method*spacing*apply+Carg(1))*spacing*token +function parsers.splitmethod(str,default) + if str then + return lpegmatch(pattern,str,1,default or false) + else + return default or false,"" + end +end +local p_year=lpegpatterns.digit^4/tonumber +local pattern=Cf(Ct("")*( + (Cg(Cc("year")*p_year)*S("-/")*Cg(Cc("month")*cardinal)*S("-/")*Cg(Cc("day")*cardinal) + )+(Cg(Cc("day")*cardinal)*S("-/")*Cg(Cc("month")*cardinal)*S("-/")*Cg(Cc("year")*p_year) + ) + )*P(" ")*Cg(Cc("hour")*cardinal)*P(":")*Cg(Cc("min")*cardinal)*(P(":")*Cg(Cc("sec")*cardinal))^-1 +,rawset) +lpegpatterns.splittime=pattern +function parsers.totime(str) + return lpegmatch(pattern,str) +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['util-dim']={ + version=1.001, + comment="support for dimensions", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local format,match,gsub,type,setmetatable=string.format,string.match,string.gsub,type,setmetatable +local P,S,R,Cc,C,lpegmatch=lpeg.P,lpeg.S,lpeg.R,lpeg.Cc,lpeg.C,lpeg.match +local allocate=utilities.storage.allocate +local setmetatableindex=table.setmetatableindex +local formatters=string.formatters +local texget=tex and tex.get or function() return 65536*10*100 end +local p_stripzeros=lpeg.patterns.stripzeros +number=number or {} +local number=number +number.tonumberf=function(n) return lpegmatch(p_stripzeros,format("%.20f",n)) end +number.tonumberg=function(n) return format("%.20g",n) end +local dimenfactors=allocate { + ["pt"]=1/65536, + ["in"]=(100/7227)/65536, + ["cm"]=(254/7227)/65536, + ["mm"]=(2540/7227)/65536, + ["sp"]=1, + ["bp"]=(7200/7227)/65536, + ["pc"]=(1/12)/65536, + ["dd"]=(1157/1238)/65536, + ["cc"]=(1157/14856)/65536, + ["nd"]=(20320/21681)/65536, + ["nc"]=(5080/65043)/65536 +} +local f_none=formatters["%s%s"] +local f_true=formatters["%0.5F%s"] +local function numbertodimen(n,unit,fmt) + if type(n)=='string' then + return n + else + unit=unit or 'pt' + n=n*dimenfactors[unit] + if not fmt then + fmt=f_none(n,unit) + elseif fmt==true then + fmt=f_true(n,unit) + else + return formatters[fmt](n,unit) + end + end +end +number.maxdimen=1073741823 +number.todimen=numbertodimen +number.dimenfactors=dimenfactors +function number.topoints (n,fmt) return numbertodimen(n,"pt",fmt) end +function number.toinches (n,fmt) return numbertodimen(n,"in",fmt) end +function number.tocentimeters (n,fmt) return numbertodimen(n,"cm",fmt) end +function number.tomillimeters (n,fmt) return numbertodimen(n,"mm",fmt) end +function number.toscaledpoints(n,fmt) return numbertodimen(n,"sp",fmt) end +function number.toscaledpoints(n) return n.."sp" end +function number.tobasepoints (n,fmt) return numbertodimen(n,"bp",fmt) end +function number.topicas (n,fmt) return numbertodimen(n "pc",fmt) end +function number.todidots (n,fmt) return numbertodimen(n,"dd",fmt) end +function number.tociceros (n,fmt) return numbertodimen(n,"cc",fmt) end +function number.tonewdidots (n,fmt) return numbertodimen(n,"nd",fmt) end +function number.tonewciceros (n,fmt) return numbertodimen(n,"nc",fmt) end +local amount=(S("+-")^0*R("09")^0*P(".")^0*R("09")^0)+Cc("0") +local unit=R("az")^1+P("%") +local dimenpair=amount/tonumber*(unit^1/dimenfactors+Cc(1)) +lpeg.patterns.dimenpair=dimenpair +local splitter=amount/tonumber*C(unit^1) +function number.splitdimen(str) + return lpegmatch(splitter,str) +end +setmetatableindex(dimenfactors,function(t,s) + return false +end) +local stringtodimen +local amount=S("+-")^0*R("09")^0*S(".,")^0*R("09")^0 +local unit=P("pt")+P("cm")+P("mm")+P("sp")+P("bp")+P("in")+P("pc")+P("dd")+P("cc")+P("nd")+P("nc") +local validdimen=amount*unit +lpeg.patterns.validdimen=validdimen +local dimensions={} +function dimensions.__add(a,b) + local ta,tb=type(a),type(b) + if ta=="string" then a=stringtodimen(a) elseif ta=="table" then a=a[1] end + if tb=="string" then b=stringtodimen(b) elseif tb=="table" then b=b[1] end + return setmetatable({ a+b },dimensions) +end +function dimensions.__sub(a,b) + local ta,tb=type(a),type(b) + if ta=="string" then a=stringtodimen(a) elseif ta=="table" then a=a[1] end + if tb=="string" then b=stringtodimen(b) elseif tb=="table" then b=b[1] end + return setmetatable({ a-b },dimensions) +end +function dimensions.__mul(a,b) + local ta,tb=type(a),type(b) + if ta=="string" then a=stringtodimen(a) elseif ta=="table" then a=a[1] end + if tb=="string" then b=stringtodimen(b) elseif tb=="table" then b=b[1] end + return setmetatable({ a*b },dimensions) +end +function dimensions.__div(a,b) + local ta,tb=type(a),type(b) + if ta=="string" then a=stringtodimen(a) elseif ta=="table" then a=a[1] end + if tb=="string" then b=stringtodimen(b) elseif tb=="table" then b=b[1] end + return setmetatable({ a/b },dimensions) +end +function dimensions.__unm(a) + local ta=type(a) + if ta=="string" then a=stringtodimen(a) elseif ta=="table" then a=a[1] end + return setmetatable({-a },dimensions) +end +function dimensions.__lt(a,b) + return a[1]<b[1] +end +function dimensions.__eq(a,b) + return a[1]==b[1] +end +function dimensions.__tostring(a) + return a[1]/65536 .."pt" +end +function dimensions.__index(tab,key) + local d=dimenfactors[key] + if not d then + error("illegal property of dimen: "..key) + d=1 + end + return 1/d +end + dimenfactors["ex"]=4*1/65536 + dimenfactors["em"]=10*1/65536 +local known={} setmetatable(known,{ __mode="v" }) +function dimen(a) + if a then + local ta=type(a) + if ta=="string" then + local k=known[a] + if k then + a=k + else + local value,unit=lpegmatch(dimenpair,a) + if value and unit then + k=value/unit + else + k=0 + end + known[a]=k + a=k + end + elseif ta=="table" then + a=a[1] + end + return setmetatable({ a },dimensions) + else + return setmetatable({ 0 },dimensions) + end +end +function string.todimen(str) + local t=type(str) + if t=="number" then + return str + else + local k=known[str] + if not k then + if t=="string" then + local value,unit=lpegmatch(dimenpair,str) + if value and unit then + k=value/unit + else + k=0 + end + else + k=0 + end + known[str]=k + end + return k + end +end +stringtodimen=string.todimen +function number.toscaled(d) + return format("%0.5f",d/0x10000) +end +function number.percent(n,d) + d=d or texget("hsize") + if type(d)=="string" then + d=stringtodimen(d) + end + return (n/100)*d +end +number["%"]=number.percent + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['util-jsn']={ + version=1.001, + comment="companion to m-json.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local P,V,R,S,C,Cc,Cs,Ct,Cf,Cg=lpeg.P,lpeg.V,lpeg.R,lpeg.S,lpeg.C,lpeg.Cc,lpeg.Cs,lpeg.Ct,lpeg.Cf,lpeg.Cg +local lpegmatch=lpeg.match +local format,gsub=string.format,string.gsub +local formatters=string.formatters +local utfchar=utf.char +local concat,sortedkeys=table.concat,table.sortedkeys +local tonumber,tostring,rawset,type,next=tonumber,tostring,rawset,type,next +local json=utilities.json or {} +utilities.json=json +do + local lbrace=P("{") + local rbrace=P("}") + local lparent=P("[") + local rparent=P("]") + local comma=P(",") + local colon=P(":") + local dquote=P('"') + local whitespace=lpeg.patterns.whitespace + local optionalws=whitespace^0 + local escapes={ + ["b"]="\010", + ["f"]="\014", + ["n"]="\n", + ["r"]="\r", + ["t"]="\t", + } + local escape_un=P("\\u")/""*(C(R("09","AF","af")^-4)/function(s) + return utfchar(tonumber(s,16)) + end) + local escape_bs=P([[\]])/""*(P(1)/escapes) + local jstring=dquote*Cs((escape_un+escape_bs+(1-dquote))^0)*dquote + local jtrue=P("true")*Cc(true) + local jfalse=P("false")*Cc(false) + local jnull=P("null")*Cc(nil) + local jnumber=(1-whitespace-rparent-rbrace-comma)^1/tonumber + local key=jstring + local jsonconverter={ "value", + hash=lbrace*Cf(Ct("")*(V("pair")*(comma*V("pair"))^0+optionalws),rawset)*rbrace, + pair=Cg(optionalws*key*optionalws*colon*V("value")), + array=Ct(lparent*(V("value")*(comma*V("value"))^0+optionalws)*rparent), + value=optionalws*(jstring+V("hash")+V("array")+jtrue+jfalse+jnull+jnumber)*optionalws, + } + function json.tolua(str) + return lpegmatch(jsonconverter,str) + end + function json.load(filename) + local data=io.loaddata(filename) + if data then + return lpegmatch(jsonconverter,data) + end + end +end +do + local escaper + local f_start_hash=formatters[ '%w{' ] + local f_start_array=formatters[ '%w[' ] + local f_start_hash_new=formatters[ "\n"..'%w{' ] + local f_start_array_new=formatters[ "\n"..'%w[' ] + local f_start_hash_key=formatters[ "\n"..'%w"%s" : {' ] + local f_start_array_key=formatters[ "\n"..'%w"%s" : [' ] + local f_stop_hash=formatters[ "\n"..'%w}' ] + local f_stop_array=formatters[ "\n"..'%w]' ] + local f_key_val_seq=formatters[ "\n"..'%w"%s" : %s' ] + local f_key_val_str=formatters[ "\n"..'%w"%s" : "%s"' ] + local f_key_val_num=f_key_val_seq + local f_key_val_yes=formatters[ "\n"..'%w"%s" : true' ] + local f_key_val_nop=formatters[ "\n"..'%w"%s" : false' ] + local f_key_val_null=formatters[ "\n"..'%w"%s" : null' ] + local f_val_num=formatters[ "\n"..'%w%s' ] + local f_val_str=formatters[ "\n"..'%w"%s"' ] + local f_val_yes=formatters[ "\n"..'%wtrue' ] + local f_val_nop=formatters[ "\n"..'%wfalse' ] + local f_val_null=formatters[ "\n"..'%wnull' ] + local f_val_empty=formatters[ "\n"..'%w{ }' ] + local f_val_seq=f_val_num + local t={} + local n=0 + local function is_simple_table(tt) + local l=#tt + if l>0 then + for i=1,l do + if type(tt[i])=="table" then + return false + end + end + local nn=n + n=n+1 t[n]="[ " + for i=1,l do + if i>1 then + n=n+1 t[n]=", " + end + local v=tt[i] + local tv=type(v) + if tv=="number" then + n=n+1 t[n]=v + elseif tv=="string" then + n=n+1 t[n]='"' + n=n+1 t[n]=lpegmatch(escaper,v) or v + n=n+1 t[n]='"' + elseif tv=="boolean" then + n=n+1 t[n]=v and "true" or "false" + elseif v then + n=n+1 t[n]=tostring(v) + else + n=n+1 t[n]="null" + end + end + n=n+1 t[n]=" ]" + local s=concat(t,"",nn+1,n) + n=nn + return s + end + return false + end + local function tojsonpp(root,name,depth,level,size) + if root then + local indexed=size>0 + n=n+1 + if level==0 then + if indexed then + t[n]=f_start_array(depth) + else + t[n]=f_start_hash(depth) + end + elseif name then + if tn=="string" then + name=lpegmatch(escaper,name) or name + elseif tn~="number" then + name=tostring(name) + end + if indexed then + t[n]=f_start_array_key(depth,name) + else + t[n]=f_start_hash_key(depth,name) + end + else + if indexed then + t[n]=f_start_array_new(depth) + else + t[n]=f_start_hash_new(depth) + end + end + depth=depth+1 + if indexed then + for i=1,size do + if i>1 then + n=n+1 t[n]="," + end + local v=root[i] + local tv=type(v) + if tv=="number" then + n=n+1 t[n]=f_val_num(depth,v) + elseif tv=="string" then + v=lpegmatch(escaper,v) or v + n=n+1 t[n]=f_val_str(depth,v) + elseif tv=="table" then + if next(v) then + local st=is_simple_table(v) + if st then + n=n+1 t[n]=f_val_seq(depth,st) + else + tojsonpp(v,nil,depth,level+1,#v) + end + else + n=n+1 + t[n]=f_val_empty(depth) + end + elseif tv=="boolean" then + n=n+1 + if v then + t[n]=f_val_yes(depth,v) + else + t[n]=f_val_nop(depth,v) + end + else + n=n+1 + t[n]=f_val_null(depth) + end + end + elseif next(root) then + local sk=sortedkeys(root) + for i=1,#sk do + if i>1 then + n=n+1 t[n]="," + end + local k=sk[i] + local v=root[k] + local tv=type(v) + local tk=type(k) + if tv=="number" then + if tk=="number" then + n=n+1 t[n]=f_key_val_num(depth,k,v) + elseif tk=="string" then + k=lpegmatch(escaper,k) or k + n=n+1 t[n]=f_key_val_num(depth,k,v) + end + elseif tv=="string" then + if tk=="number" then + v=lpegmatch(escaper,v) or v + n=n+1 t[n]=f_key_val_str(depth,k,v) + elseif tk=="string" then + k=lpegmatch(escaper,k) or k + v=lpegmatch(escaper,v) or v + n=n+1 t[n]=f_key_val_str(depth,k,v) + end + elseif tv=="table" then + local l=#v + if l>0 then + local st=is_simple_table(v) + if not st then + tojsonpp(v,k,depth,level+1,l) + elseif tk=="number" then + n=n+1 t[n]=f_key_val_seq(depth,k,st) + elseif tk=="string" then + k=lpegmatch(escaper,k) or k + n=n+1 t[n]=f_key_val_seq(depth,k,st) + end + elseif next(v) then + tojsonpp(v,k,depth,level+1,0) + end + elseif tv=="boolean" then + if tk=="number" then + n=n+1 + if v then + t[n]=f_key_val_yes(depth,k) + else + t[n]=f_key_val_nop(depth,k) + end + elseif tk=="string" then + k=lpegmatch(escaper,k) or k + n=n+1 + if v then + t[n]=f_key_val_yes(depth,k) + else + t[n]=f_key_val_nop(depth,k) + end + end + else + if tk=="number" then + n=n+1 + t[n]=f_key_val_null(depth,k) + elseif tk=="string" then + k=lpegmatch(escaper,k) or k + n=n+1 + t[n]=f_key_val_null(depth,k) + end + end + end + end + n=n+1 + if indexed then + t[n]=f_stop_array(depth-1) + else + t[n]=f_stop_hash(depth-1) + end + end + end + local function tojson(value,n) + local kind=type(value) + if kind=="table" then + local done=false + local size=#value + if size==0 then + for k,v in next,value do + if done then + n=n+1;t[n]=',"' + else + n=n+1;t[n]='{"' + done=true + end + n=n+1;t[n]=lpegmatch(escaper,k) or k + n=n+1;t[n]='":' + t,n=tojson(v,n) + end + if done then + n=n+1;t[n]="}" + else + n=n+1;t[n]="{}" + end + elseif size==1 then + n=n+1;t[n]="[" + t,n=tojson(value[1],n) + n=n+1;t[n]="]" + else + for i=1,size do + if done then + n=n+1;t[n]="," + else + n=n+1;t[n]="[" + done=true + end + t,n=tojson(value[i],n) + end + n=n+1;t[n]="]" + end + elseif kind=="string" then + n=n+1;t[n]='"' + n=n+1;t[n]=lpegmatch(escaper,value) or value + n=n+1;t[n]='"' + elseif kind=="number" then + n=n+1;t[n]=value + elseif kind=="boolean" then + n=n+1;t[n]=tostring(value) + else + n=n+1;t[n]="null" + end + return t,n + end + local function jsontostring(value,pretty) + local kind=type(value) + if kind=="table" then + if not escaper then + local escapes={ + ["\\"]="\\u005C", + ["\""]="\\u0022", + } + for i=0,0x1F do + escapes[utfchar(i)]=format("\\u%04X",i) + end + escaper=Cs(( + (R('\0\x20')+S('\"\\'))/escapes+P(1) + )^1 ) + end + t={} + n=0 + if pretty then + tojsonpp(value,name,0,0,#value) + value=concat(t,"",1,n) + else + t,n=tojson(value,0) + value=concat(t,"",1,n) + end + t=nil + n=0 + return value + elseif kind=="string" or kind=="number" then + return lpegmatch(escaper,value) or value + else + return tostring(value) + end + end + json.tostring=jsontostring + function json.tojson(value) + return jsontostring(value,true) + end +end +return json + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['trac-inf']={ + version=1.001, + comment="companion to trac-inf.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local type,tonumber,select=type,tonumber,select +local format,lower,find=string.format,string.lower,string.find +local concat=table.concat +local clock=os.gettimeofday or os.clock +local setmetatableindex=table.setmetatableindex +local serialize=table.serialize +local formatters=string.formatters +statistics=statistics or {} +local statistics=statistics +statistics.enable=true +statistics.threshold=0.01 +local statusinfo,n,registered,timers={},0,{},{} +setmetatableindex(timers,function(t,k) + local v={ timing=0,loadtime=0,offset=0 } + t[k]=v + return v +end) +local function hastiming(instance) + return instance and timers[instance] +end +local function resettiming(instance) + timers[instance or "notimer"]={ timing=0,loadtime=0,offset=0 } +end +local ticks=clock +local seconds=function(n) return n or 0 end +local function starttiming(instance,reset) + local timer=timers[instance or "notimer"] + local it=timer.timing + if reset then + it=0 + timer.loadtime=0 + end + if it==0 then + timer.starttime=ticks() + if not timer.loadtime then + timer.loadtime=0 + end + end + timer.timing=it+1 +end +local function stoptiming(instance) + local timer=timers[instance or "notimer"] + local it=timer.timing + if it>1 then + timer.timing=it-1 + else + local starttime=timer.starttime + if starttime and starttime>0 then + local stoptime=ticks() + local loadtime=stoptime-starttime + timer.stoptime=stoptime + timer.loadtime=timer.loadtime+loadtime + timer.timing=0 + timer.starttime=0 + return loadtime + end + end + return 0 +end +local function benchmarktimer(instance) + local timer=timers[instance or "notimer"] + local it=timer.timing + if it>1 then + timer.timing=it-1 + else + local starttime=timer.starttime + if starttime and starttime>0 then + timer.offset=ticks()-starttime + else + timer.offset=0 + end + end +end +local function elapsed(instance) + if type(instance)=="number" then + return instance + else + local timer=timers[instance or "notimer"] + return timer and seconds(timer.loadtime-2*(timer.offset or 0)) or 0 + end +end +local function currenttime(instance) + if type(instance)=="number" then + return instance + else + local timer=timers[instance or "notimer"] + local it=timer.timing + if it>1 then + else + local starttime=timer.starttime + if starttime and starttime>0 then + return seconds(timer.loadtime+ticks()-starttime-2*(timer.offset or 0)) + end + end + return 0 + end +end +local function elapsedtime(instance) + return format("%0.3f",elapsed(instance)) +end +local function elapsedindeed(instance) + return elapsed(instance)>statistics.threshold +end +local function elapsedseconds(instance,rest) + if elapsedindeed(instance) then + return format("%0.3f seconds %s",elapsed(instance),rest or "") + end +end +statistics.hastiming=hastiming +statistics.resettiming=resettiming +statistics.starttiming=starttiming +statistics.stoptiming=stoptiming +statistics.currenttime=currenttime +statistics.elapsed=elapsed +statistics.elapsedtime=elapsedtime +statistics.elapsedindeed=elapsedindeed +statistics.elapsedseconds=elapsedseconds +statistics.benchmarktimer=benchmarktimer +function statistics.register(tag,fnc) + if statistics.enable and type(fnc)=="function" then + local rt=registered[tag] or (#statusinfo+1) + statusinfo[rt]={ tag,fnc } + registered[tag]=rt + if #tag>n then n=#tag end + end +end +local report=logs.reporter("mkiv lua stats") +function statistics.show() + if statistics.enable then + local register=statistics.register + register("used platform",function() + return format("%s, type: %s, binary subtree: %s", + os.platform or "unknown",os.type or "unknown",environment.texos or "unknown") + end) + if LUATEXENGINE=="luametatex" then + register("used engine",function() + return format("%s version %s, functionality level %s, format id %s", + LUATEXENGINE,LUATEXVERSION,LUATEXFUNCTIONALITY,LUATEXFORMATID) + end) + else + register("used engine",function() + return format("%s version %s with functionality level %s, banner: %s", + LUATEXENGINE,LUATEXVERSION,LUATEXFUNCTIONALITY,lower(status.banner)) + end) + end + register("control sequences",function() + return format("%s of %s + %s",status.cs_count,status.hash_size,status.hash_extra) + end) + register("callbacks",statistics.callbacks) + if TEXENGINE=="luajittex" and JITSUPPORTED then + local jitstatus=jit.status + if jitstatus then + local jitstatus={ jitstatus() } + if jitstatus[1] then + register("luajit options",concat(jitstatus," ",2)) + end + end + end + register("lua properties",function() + local hashchar=tonumber(status.luatex_hashchars) + local mask=lua.mask or "ascii" + return format("engine: %s %s, used memory: %s, hash chars: min(%i,40), symbol mask: %s (%s)", + jit and "luajit" or "lua", + LUAVERSION, + statistics.memused(), + hashchar and 2^hashchar or "unknown", + mask, + mask=="utf" and "τεχ" or "tex") + end) + register("runtime",statistics.runtime) + logs.newline() + for i=1,#statusinfo do + local s=statusinfo[i] + local r=s[2]() + if r then + report("%s: %s",s[1],r) + end + end + statistics.enable=false + end +end +function statistics.memused() + local round=math.round or math.floor + return format("%s MB, ctx: %s MB, max: %s MB)", + round(collectgarbage("count")/1000), + round(status.luastate_bytes/1000000), + status.luastate_bytes_max and round(status.luastate_bytes_max/1000000) or "unknown" + ) +end +starttiming(statistics) +function statistics.formatruntime(runtime) + return format("%s seconds",runtime) +end +function statistics.runtime() + stoptiming(statistics) + local runtime=lua.getruntime and lua.getruntime() or elapsedtime(statistics) + return statistics.formatruntime(runtime) +end +local report=logs.reporter("system") +function statistics.timed(action,all) + starttiming("run") + action() + stoptiming("run") + local runtime=tonumber(elapsedtime("run")) + if all then + local alltime=tonumber(lua.getruntime and lua.getruntime() or elapsedtime(statistics)) + if alltime and alltime>0 then + report("total runtime: %0.3f seconds of %0.3f seconds",runtime,alltime) + return + end + end + report("total runtime: %0.3f seconds",runtime) +end +function statistics.tracefunction(base,tag,...) + for i=1,select("#",...) do + local name=select(i,...) + local stat={} + local func=base[name] + setmetatableindex(stat,function(t,k) t[k]=0 return 0 end) + base[name]=function(n,k,v) stat[k]=stat[k]+1 return func(n,k,v) end + statistics.register(formatters["%s.%s"](tag,name),function() return serialize(stat,"calls") end) + end +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['util-lua']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + comment="the strip code is written by Peter Cawley", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local rep,sub,byte,dump,format=string.rep,string.sub,string.byte,string.dump,string.format +local load,loadfile,type,collectgarbage=load,loadfile,type,collectgarbage +utilities=utilities or {} +utilities.lua=utilities.lua or {} +local luautilities=utilities.lua +local report_lua=logs.reporter("system","lua") +local report_mem=logs.reporter("system","lua memory") +local tracestripping=false +local tracememory=false +luautilities.stripcode=true +luautilities.alwaysstripcode=false +luautilities.nofstrippedchunks=0 +luautilities.nofstrippedbytes=0 +local strippedchunks={} +luautilities.strippedchunks=strippedchunks +luautilities.suffixes={ + tma="tma", + tmc=(CONTEXTLMTXMODE and CONTEXTLMTXMODE>0 and "tmd") or (jit and "tmb") or "tmc", + lua="lua", + luc=(CONTEXTLMTXMODE and CONTEXTLMTXMODE>0 and "lud") or (jit and "lub") or "luc", + lui="lui", + luv="luv", + luj="luj", + tua="tua", + tuc="tuc", +} +local function register(name) + if tracestripping then + report_lua("stripped bytecode from %a",name or "unknown") + end + strippedchunks[#strippedchunks+1]=name + luautilities.nofstrippedchunks=luautilities.nofstrippedchunks+1 +end +local function stupidcompile(luafile,lucfile,strip) + local code=io.loaddata(luafile) + if code and code~="" then + code=load(code) + if code then + code=dump(code,strip and luautilities.stripcode or luautilities.alwaysstripcode) + if code and code~="" then + register(name) + io.savedata(lucfile,code) + return true,0 + end + else + report_lua("fatal error %a in file %a",1,luafile) + end + else + report_lua("fatal error %a in file %a",2,luafile) + end + return false,0 +end +function luautilities.loadedluacode(fullname,forcestrip,name,macros) + name=name or fullname + if macros then + macros=lua.macros + end + local code,message + if macros then + code,message=macros.loaded(fullname,true,false) + else + code,message=loadfile(fullname) + end + if code then + code() + else + report_lua("loading of file %a failed:\n\t%s",fullname,message or "no message") + code,message=loadfile(fullname) + end + if forcestrip and luautilities.stripcode then + if type(forcestrip)=="function" then + forcestrip=forcestrip(fullname) + end + if forcestrip or luautilities.alwaysstripcode then + register(name) + return load(dump(code,true)),0 + else + return code,0 + end + elseif luautilities.alwaysstripcode then + register(name) + return load(dump(code,true)),0 + else + return code,0 + end +end +function luautilities.strippedloadstring(code,name,forcestrip) + local code,message=load(code) + if not code then + report_lua("loading of file %a failed:\n\t%s",name,message or "no message") + end + if forcestrip and luautilities.stripcode or luautilities.alwaysstripcode then + register(name) + return load(dump(code,true)),0 + else + return code,0 + end +end +function luautilities.loadstring(code,name) + local code,message=load(code) + if not code then + report_lua("loading of file %a failed:\n\t%s",name,message or "no message") + end + return code,0 +end +function luautilities.compile(luafile,lucfile,cleanup,strip,fallback) + report_lua("compiling %a into %a",luafile,lucfile) + os.remove(lucfile) + local done=stupidcompile(luafile,lucfile,strip~=false) + if done then + report_lua("dumping %a into %a stripped",luafile,lucfile) + if cleanup==true and lfs.isfile(lucfile) and lfs.isfile(luafile) then + report_lua("removing %a",luafile) + os.remove(luafile) + end + end + return done +end +function luautilities.loadstripped(...) + local l=load(...) + if l then + return load(dump(l,true)) + end +end +local finalizers={} +setmetatable(finalizers,{ + __gc=function(t) + for i=1,#t do + pcall(t[i]) + end + end +} ) +function luautilities.registerfinalizer(f) + finalizers[#finalizers+1]=f +end +function luautilities.checkmemory(previous,threshold,trace) + local current=collectgarbage("count") + if previous then + local checked=(threshold or 64)*1024 + local delta=current-previous + if current-previous>checked then + collectgarbage("collect") + local afterwards=collectgarbage("count") + if trace or tracememory then + report_mem("previous %r MB, current %r MB, delta %r MB, threshold %r MB, afterwards %r MB", + previous/1024,current/1024,delta/1024,threshold,afterwards) + end + return afterwards + elseif trace or tracememory then + report_mem("previous %r MB, current %r MB, delta %r MB, threshold %r MB", + previous/1024,current/1024,delta/1024,threshold) + end + end + return current +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['util-deb']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local type,next,tostring,tonumber=type,next,tostring,tonumber +local format,find,sub,gsub=string.format,string.find,string.sub,string.gsub +local insert,remove,sort=table.insert,table.remove,table.sort +local setmetatableindex=table.setmetatableindex +utilities=utilities or {} +local debugger=utilities.debugger or {} +utilities.debugger=debugger +local report=logs.reporter("debugger") +local ticks=os.gettimeofday or os.clock +local seconds=function(n) return n or 0 end +local overhead=0 +local dummycalls=10*1000 +local nesting=0 +local names={} +local initialize=false +if lua.getpreciseticks then + initialize=function() + ticks=lua.getpreciseticks + seconds=lua.getpreciseseconds + initialize=false + end +elseif not (FFISUPPORTED and ffi) then +elseif os.type=="windows" then + initialize=function() + local kernel=ffilib("kernel32","system") + if kernel then + local tonumber=ffi.number or tonumber + ffi.cdef[[ + int QueryPerformanceFrequency(int64_t *lpFrequency); + int QueryPerformanceCounter(int64_t *lpPerformanceCount); + ]] + local target=ffi.new("__int64[1]") + ticks=function() + if kernel.QueryPerformanceCounter(target)==1 then + return tonumber(target[0]) + else + return 0 + end + end + local target=ffi.new("__int64[1]") + seconds=function(ticks) + if kernel.QueryPerformanceFrequency(target)==1 then + return ticks/tonumber(target[0]) + else + return 0 + end + end + end + initialize=false + end +elseif os.type=="unix" then + initialize=function() + local C=ffi.C + local tonumber=ffi.number or tonumber + ffi.cdef [[ + /* what a mess */ + typedef int clk_id_t; + typedef enum { CLOCK_REALTIME, CLOCK_MONOTONIC, CLOCK_PROCESS_CPUTIME_ID } clk_id; + typedef struct timespec { long sec; long nsec; } ctx_timespec; + int clock_gettime(clk_id_t timerid, struct timespec *t); + ]] + local target=ffi.new("ctx_timespec[?]",1) + local clock=C.CLOCK_PROCESS_CPUTIME_ID + ticks=function () + C.clock_gettime(clock,target) + return tonumber(target[0].sec*1000000000+target[0].nsec) + end + seconds=function(ticks) + return ticks/1000000000 + end + initialize=false + end +end +setmetatableindex(names,function(t,name) + local v=setmetatableindex(function(t,source) + local v=setmetatableindex(function(t,line) + local v={ total=0,count=0,nesting=0 } + t[line]=v + return v + end) + t[source]=v + return v + end) + t[name]=v + return v +end) +local getinfo=nil +local sethook=nil +local function hook(where) + local f=getinfo(2,"nSl") + if f then + local source=f.short_src + if not source then + return + end + local line=f.linedefined or 0 + local name=f.name + if not name then + local what=f.what + if what=="C" then + name="<anonymous>" + else + name=f.namewhat or what or "<unknown>" + end + end + local data=names[name][source][line] + if where=="call" then + local nesting=data.nesting + if nesting==0 then + data.count=data.count+1 + insert(data,ticks()) + data.nesting=1 + else + data.nesting=nesting+1 + end + elseif where=="return" then + local nesting=data.nesting + if nesting==1 then + local t=remove(data) + if t then + data.total=data.total+ticks()-t + end + data.nesting=0 + else + data.nesting=nesting-1 + end + end + end +end +function debugger.showstats(printer,threshold) + local printer=printer or report + local calls=0 + local functions=0 + local dataset={} + local length=0 + local realtime=0 + local totaltime=0 + local threshold=threshold or 0 + for name,sources in next,names do + for source,lines in next,sources do + for line,data in next,lines do + local count=data.count + if count>threshold then + if #name>length then + length=#name + end + local total=data.total + local real=total + if real>0 then + real=total-(count*overhead/dummycalls) + if real<0 then + real=0 + end + realtime=realtime+real + end + totaltime=totaltime+total + if line<0 then + line=0 + end + dataset[#dataset+1]={ real,total,count,name,source,line } + end + end + end + end + sort(dataset,function(a,b) + if a[1]==b[1] then + if a[2]==b[2] then + if a[3]==b[3] then + if a[4]==b[4] then + if a[5]==b[5] then + return a[6]<b[6] + else + return a[5]<b[5] + end + else + return a[4]<b[4] + end + else + return b[3]<a[3] + end + else + return b[2]<a[2] + end + else + return b[1]<a[1] + end + end) + if length>50 then + length=50 + end + local fmt=string.formatters["%4.9k s %3.3k %% %4.9k s %3.3k %% %8i # %-"..length.."s %4i %s"] + for i=1,#dataset do + local data=dataset[i] + local real=data[1] + local total=data[2] + local count=data[3] + local name=data[4] + local source=data[5] + local line=data[6] + calls=calls+count + functions=functions+1 + name=gsub(name,"%s+"," ") + if #name>length then + name=sub(name,1,length) + end + printer(fmt(seconds(total),100*total/totaltime,seconds(real),100*real/realtime,count,name,line,source)) + end + printer("") + printer(format("functions : %i",functions)) + printer(format("calls : %i",calls)) + printer(format("overhead : %f",seconds(overhead/1000))) +end +local function getdebug() + if sethook and getinfo then + return + end + if not debug then + local okay + okay,debug=pcall(require,"debug") + end + if type(debug)~="table" then + return + end + getinfo=debug.getinfo + sethook=debug.sethook + if type(getinfo)~="function" then + getinfo=nil + end + if type(sethook)~="function" then + sethook=nil + end +end +function debugger.savestats(filename,threshold) + local f=io.open(filename,'w') + if f then + debugger.showstats(function(str) f:write(str,"\n") end,threshold) + f:close() + end +end +function debugger.enable() + getdebug() + if sethook and getinfo and nesting==0 then + running=true + if initialize then + initialize() + end + sethook(hook,"cr") + local function dummy() end + local t=ticks() + for i=1,dummycalls do + dummy() + end + overhead=ticks()-t + end + if nesting>0 then + nesting=nesting+1 + end +end +function debugger.disable() + if nesting>0 then + nesting=nesting-1 + end + if sethook and getinfo and nesting==0 then + sethook() + end +end +local function showtraceback(rep) + getdebug() + if getinfo then + local level=2 + local reporter=rep or report + while true do + local info=getinfo(level,"Sl") + if not info then + break + elseif info.what=="C" then + reporter("%2i : %s",level-1,"C function") + else + reporter("%2i : %s : %s",level-1,info.short_src,info.currentline) + end + level=level+1 + end + end +end +debugger.showtraceback=showtraceback + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['util-tpl']={ + version=1.001, + comment="companion to luat-lib.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +utilities.templates=utilities.templates or {} +local templates=utilities.templates +local trace_template=false trackers.register("templates.trace",function(v) trace_template=v end) +local report_template=logs.reporter("template") +local tostring,next=tostring,next +local format,sub,byte=string.format,string.sub,string.byte +local P,C,R,Cs,Cc,Carg,lpegmatch,lpegpatterns=lpeg.P,lpeg.C,lpeg.R,lpeg.Cs,lpeg.Cc,lpeg.Carg,lpeg.match,lpeg.patterns +local replacer +local function replacekey(k,t,how,recursive) + local v=t[k] + if not v then + if trace_template then + report_template("unknown key %a",k) + end + return "" + else + v=tostring(v) + if trace_template then + report_template("setting key %a to value %a",k,v) + end + if recursive then + return lpegmatch(replacer,v,1,t,how,recursive) + else + return v + end + end +end +local sqlescape=lpeg.replacer { + { "'","''" }, + { "\\","\\\\" }, + { "\r\n","\\n" }, + { "\r","\\n" }, +} +local sqlquoted=Cs(Cc("'")*sqlescape*Cc("'")) +lpegpatterns.sqlescape=sqlescape +lpegpatterns.sqlquoted=sqlquoted +local luaescape=lpegpatterns.luaescape +local escapers={ + lua=function(s) + return lpegmatch(luaescape,s) + end, + sql=function(s) + return lpegmatch(sqlescape,s) + end, +} +local quotedescapers={ + lua=function(s) + return format("%q",s) + end, + sql=function(s) + return lpegmatch(sqlquoted,s) + end, +} +local luaescaper=escapers.lua +local quotedluaescaper=quotedescapers.lua +local function replacekeyunquoted(s,t,how,recurse) + if how==false then + return replacekey(s,t,how,recurse) + else + local escaper=how and escapers[how] or luaescaper + return escaper(replacekey(s,t,how,recurse)) + end +end +local function replacekeyquoted(s,t,how,recurse) + if how==false then + return replacekey(s,t,how,recurse) + else + local escaper=how and quotedescapers[how] or quotedluaescaper + return escaper(replacekey(s,t,how,recurse)) + end +end +local function replaceoptional(l,m,r,t,how,recurse) + local v=t[l] + return v and v~="" and lpegmatch(replacer,r,1,t,how or "lua",recurse or false) or "" +end +local single=P("%") +local double=P("%%") +local lquoted=P("%[") +local rquoted=P("]%") +local lquotedq=P("%(") +local rquotedq=P(")%") +local escape=double/'%%' +local nosingle=single/'' +local nodouble=double/'' +local nolquoted=lquoted/'' +local norquoted=rquoted/'' +local nolquotedq=lquotedq/'' +local norquotedq=rquotedq/'' +local noloptional=P("%?")/'' +local noroptional=P("?%")/'' +local nomoptional=P(":")/'' +local args=Carg(1)*Carg(2)*Carg(3) +local key=nosingle*((C((1-nosingle )^1)*args)/replacekey )*nosingle +local quoted=nolquotedq*((C((1-norquotedq )^1)*args)/replacekeyquoted )*norquotedq +local unquoted=nolquoted*((C((1-norquoted )^1)*args)/replacekeyunquoted)*norquoted +local optional=noloptional*((C((1-nomoptional)^1)*nomoptional*C((1-noroptional)^1)*args)/replaceoptional)*noroptional +local any=P(1) + replacer=Cs((unquoted+quoted+escape+optional+key+any)^0) +local function replace(str,mapping,how,recurse) + if mapping and str then + return lpegmatch(replacer,str,1,mapping,how or "lua",recurse or false) or str + else + return str + end +end +templates.replace=replace +function templates.replacer(str,how,recurse) + return function(mapping) + return lpegmatch(replacer,str,1,mapping,how or "lua",recurse or false) or str + end +end +function templates.load(filename,mapping,how,recurse) + local data=io.loaddata(filename) or "" + if mapping and next(mapping) then + return replace(data,mapping,how,recurse) + else + return data + end +end +function templates.resolve(t,mapping,how,recurse) + if not mapping then + mapping=t + end + for k,v in next,t do + t[k]=replace(v,mapping,how,recurse) + end + return t +end + +end -- closure + +do -- begin closure to overcome local limits and interference + +if not modules then modules={} end modules ['util-sta']={ + version=1.001, + comment="companion to util-ini.mkiv", + author="Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright="PRAGMA ADE / ConTeXt Development Team", + license="see context related readme files" +} +local insert,remove,fastcopy,concat=table.insert,table.remove,table.fastcopy,table.concat +local format=string.format +local select,tostring=select,tostring +local trace_stacker=false trackers.register("stacker.resolve",function(v) trace_stacker=v end) +local stacker=stacker or {} +utilities.stacker=stacker +local function start(s,t,first,last) + if s.mode=="switch" then + local n=tostring(t[last]) + if trace_stacker then + s.report("start: %s",n) + end + return n + else + local r={} + for i=first,last do + r[#r+1]=tostring(t[i]) + end + local n=concat(r," ") + if trace_stacker then + s.report("start: %s",n) + end + return n + end +end +local function stop(s,t,first,last) + if s.mode=="switch" then + local n=tostring(false) + if trace_stacker then + s.report("stop: %s",n) + end + return n + else + local r={} + for i=last,first,-1 do + r[#r+1]=tostring(false) + end + local n=concat(r," ") + if trace_stacker then + s.report("stop: %s",n) + end + return n + end +end +local function change(s,t1,first1,last1,t2,first2,last2) + if s.mode=="switch" then + local n=tostring(t2[last2]) + if trace_stacker then + s.report("change: %s",n) + end + return n + else + local r={} + for i=last1,first1,-1 do + r[#r+1]=tostring(false) + end + local n=concat(r," ") + for i=first2,last2 do + r[#r+1]=tostring(t2[i]) + end + if trace_stacker then + s.report("change: %s",n) + end + return n + end +end +function stacker.new(name) + local report=logs.reporter("stacker",name or nil) + local s + local stack={} + local list={} + local ids={} + local hash={} + local hashing=true + local function push(...) + for i=1,select("#",...) do + insert(stack,(select(i,...))) + end + if hashing then + local c=concat(stack,"|") + local n=hash[c] + if not n then + n=#list+1 + hash[c]=n + list[n]=fastcopy(stack) + end + insert(ids,n) + return n + else + local n=#list+1 + list[n]=fastcopy(stack) + insert(ids,n) + return n + end + end + local function pop() + remove(stack) + remove(ids) + return ids[#ids] or s.unset or -1 + end + local function clean() + if #stack==0 then + if trace_stacker then + s.report("%s list entries, %s stack entries",#list,#stack) + end + end + end + local tops={} + local top=nil + local switch=nil + local function resolve_reset(mode) + if #tops>0 then + report("resetting %s left-over states of %a",#tops,name) + end + tops={} + top=nil + switch=nil + end + local function resolve_begin(mode) + if mode then + switch=mode=="switch" + else + switch=s.mode=="switch" + end + top={ switch=switch } + insert(tops,top) + end + local function resolve_step(ti) + local result=nil + local noftop=top and #top or 0 + if ti>0 then + local current=list[ti] + if current then + local noflist=#current + local nofsame=0 + if noflist>noftop then + for i=1,noflist do + if current[i]==top[i] then + nofsame=i + else + break + end + end + else + for i=1,noflist do + if current[i]==top[i] then + nofsame=i + else + break + end + end + end + local plus=nofsame+1 + if plus<=noftop then + if plus<=noflist then + if switch then + result=s.change(s,top,plus,noftop,current,nofsame,noflist) + else + result=s.change(s,top,plus,noftop,current,plus,noflist) + end + else + if switch then + result=s.change(s,top,plus,noftop,current,nofsame,noflist) + else + result=s.stop(s,top,plus,noftop) + end + end + elseif plus<=noflist then + if switch then + result=s.start(s,current,nofsame,noflist) + else + result=s.start(s,current,plus,noflist) + end + end + top=current + else + if 1<=noftop then + result=s.stop(s,top,1,noftop) + end + top={} + end + return result + else + if 1<=noftop then + result=s.stop(s,top,1,noftop) + end + top={} + return result + end + end + local function resolve_end() + if #tops>0 then + local result=s.stop(s,top,1,#top) + remove(tops) + top=tops[#tops] + switch=top and top.switch + return result + end + end + local function resolve(t) + resolve_begin() + for i=1,#t do + resolve_step(t[i]) + end + resolve_end() + end + s={ + name=name or "unknown", + unset=-1, + report=report, + start=start, + stop=stop, + change=change, + push=push, + pop=pop, + clean=clean, + resolve=resolve, + resolve_begin=resolve_begin, + resolve_step=resolve_step, + resolve_end=resolve_end, + resolve_reset=resolve_reset, + } + return s +end + +end -- closure diff --git a/macros/luatex/generic/lualibs/lualibs-file.lua b/macros/luatex/generic/lualibs/lualibs-file.lua new file mode 100644 index 0000000000..1b039a4386 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-file.lua @@ -0,0 +1,738 @@ +if not modules then modules = { } end modules ['l-file'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +-- needs a cleanup + +file = file or { } +local file = file + +if not lfs then + lfs = optionalrequire("lfs") +end + +-- -- see later +-- +-- if not lfs then +-- +-- lfs = { +-- getcurrentdir = function() +-- return "." +-- end, +-- attributes = function() +-- return nil +-- end, +-- isfile = function(name) +-- local f = io.open(name,'rb') +-- if f then +-- f:close() +-- return true +-- end +-- end, +-- isdir = function(name) +-- print("you need to load lfs") +-- return false +-- end +-- } +-- +-- elseif not lfs.isfile then +-- +-- local attributes = lfs.attributes +-- +-- function lfs.isdir(name) +-- return attributes(name,"mode") == "directory" +-- end +-- +-- function lfs.isfile(name) +-- return attributes(name,"mode") == "file" +-- end +-- +-- -- function lfs.isdir(name) +-- -- local a = attributes(name) +-- -- return a and a.mode == "directory" +-- -- end +-- +-- -- function lfs.isfile(name) +-- -- local a = attributes(name) +-- -- return a and a.mode == "file" +-- -- end +-- +-- end + +local insert, concat = table.insert, table.concat +local match, find, gmatch = string.match, string.find, string.gmatch +local lpegmatch = lpeg.match +local getcurrentdir, attributes = lfs.currentdir, lfs.attributes +local checkedsplit = string.checkedsplit + +local P, R, S, C, Cs, Cp, Cc, Ct = lpeg.P, lpeg.R, lpeg.S, lpeg.C, lpeg.Cs, lpeg.Cp, lpeg.Cc, lpeg.Ct + +-- better this way: + +local attributes = lfs.attributes + +function lfs.isdir(name) + return attributes(name,"mode") == "directory" +end + +function lfs.isfile(name) + local a = attributes(name,"mode") + return a == "file" or a == "link" or nil +end + +function lfs.isfound(name) + local a = attributes(name,"mode") + return (a == "file" or a == "link") and name or nil +end + +if sandbox then + sandbox.redefine(lfs.isfile,"lfs.isfile") + sandbox.redefine(lfs.isdir, "lfs.isdir") + sandbox.redefine(lfs.isfound, "lfs.isfound") +end + +local colon = P(":") +local period = P(".") +local periods = P("..") +local fwslash = P("/") +local bwslash = P("\\") +local slashes = S("\\/") +local noperiod = 1-period +local noslashes = 1-slashes +local name = noperiod^1 +local suffix = period/"" * (1-period-slashes)^1 * -1 + +----- pattern = C((noslashes^0 * slashes^1)^1) +local pattern = C((1 - (slashes^1 * noslashes^1 * -1))^1) * P(1) -- there must be a more efficient way + +local function pathpart(name,default) + return name and lpegmatch(pattern,name) or default or "" +end + +local pattern = (noslashes^0 * slashes)^1 * C(noslashes^1) * -1 + +local function basename(name) + return name and lpegmatch(pattern,name) or name +end + +-- print(pathpart("file")) +-- print(pathpart("dir/file")) +-- print(pathpart("/dir/file")) +-- print(basename("file")) +-- print(basename("dir/file")) +-- print(basename("/dir/file")) + +local pattern = (noslashes^0 * slashes^1)^0 * Cs((1-suffix)^1) * suffix^0 + +local function nameonly(name) + return name and lpegmatch(pattern,name) or name +end + +local pattern = (noslashes^0 * slashes)^0 * (noperiod^1 * period)^1 * C(noperiod^1) * -1 + +local function suffixonly(name) + return name and lpegmatch(pattern,name) or "" +end + +local pattern = (noslashes^0 * slashes)^0 * noperiod^1 * ((period * C(noperiod^1))^1) * -1 + Cc("") + +local function suffixesonly(name) + if name then + return lpegmatch(pattern,name) + else + return "" + end +end + +file.pathpart = pathpart +file.basename = basename +file.nameonly = nameonly +file.suffixonly = suffixonly +file.suffix = suffixonly +file.suffixesonly = suffixesonly +file.suffixes = suffixesonly + +file.dirname = pathpart -- obsolete +file.extname = suffixonly -- obsolete + +-- actually these are schemes + +local drive = C(R("az","AZ")) * colon +local path = C((noslashes^0 * slashes)^0) +local suffix = period * C(P(1-period)^0 * P(-1)) +local base = C((1-suffix)^0) +local rest = C(P(1)^0) + +drive = drive + Cc("") +path = path + Cc("") +base = base + Cc("") +suffix = suffix + Cc("") + +local pattern_a = drive * path * base * suffix +local pattern_b = path * base * suffix +local pattern_c = C(drive * path) * C(base * suffix) -- trick: two extra captures +local pattern_d = path * rest + +function file.splitname(str,splitdrive) + if not str then + -- error + elseif splitdrive then + return lpegmatch(pattern_a,str) -- returns drive, path, base, suffix + else + return lpegmatch(pattern_b,str) -- returns path, base, suffix + end +end + +function file.splitbase(str) + if str then + return lpegmatch(pattern_d,str) -- returns path, base+suffix (path has / appended, might change at some point) + else + return "", str -- assume no path + end +end + +---- stripslash = C((1 - P("/")^1*P(-1))^0) + +function file.nametotable(str,splitdrive) + if str then + local path, drive, subpath, name, base, suffix = lpegmatch(pattern_c,str) + -- if path ~= "" then + -- path = lpegmatch(stripslash,path) -- unfortunate hack, maybe this becomes default + -- end + if splitdrive then + return { + path = path, + drive = drive, + subpath = subpath, + name = name, + base = base, + suffix = suffix, + } + else + return { + path = path, + name = name, + base = base, + suffix = suffix, + } + end + end +end + +-- print(file.splitname("file")) +-- print(file.splitname("dir/file")) +-- print(file.splitname("/dir/file")) +-- print(file.splitname("file")) +-- print(file.splitname("dir/file")) +-- print(file.splitname("/dir/file")) + +-- inspect(file.nametotable("file.ext")) +-- inspect(file.nametotable("dir/file.ext")) +-- inspect(file.nametotable("/dir/file.ext")) +-- inspect(file.nametotable("file.ext")) +-- inspect(file.nametotable("dir/file.ext")) +-- inspect(file.nametotable("/dir/file.ext")) + +----- pattern = Cs(((period * noperiod^1 * -1) / "" + 1)^1) +local pattern = Cs(((period * (1-period-slashes)^1 * -1) / "" + 1)^1) + +function file.removesuffix(name) + return name and lpegmatch(pattern,name) +end + +-- local pattern = (noslashes^0 * slashes)^0 * (noperiod^1 * period)^1 * Cp() * noperiod^1 * -1 +-- +-- function file.addsuffix(name, suffix) +-- local p = lpegmatch(pattern,name) +-- if p then +-- return name +-- else +-- return name .. "." .. suffix +-- end +-- end + +local suffix = period/"" * (1-period-slashes)^1 * -1 +local pattern = Cs((noslashes^0 * slashes^1)^0 * ((1-suffix)^1)) * Cs(suffix) + +function file.addsuffix(filename,suffix,criterium) + if not filename or not suffix or suffix == "" then + return filename + elseif criterium == true then + return filename .. "." .. suffix + elseif not criterium then + local n, s = lpegmatch(pattern,filename) + if not s or s == "" then + return filename .. "." .. suffix + else + return filename + end + else + local n, s = lpegmatch(pattern,filename) + if s and s ~= "" then + local t = type(criterium) + if t == "table" then + -- keep if in criterium + for i=1,#criterium do + if s == criterium[i] then + return filename + end + end + elseif t == "string" then + -- keep if criterium + if s == criterium then + return filename + end + end + end + return (n or filename) .. "." .. suffix + end +end + +-- print("1 " .. file.addsuffix("name","new") .. " -> name.new") +-- print("2 " .. file.addsuffix("name.old","new") .. " -> name.old") +-- print("3 " .. file.addsuffix("name.old","new",true) .. " -> name.old.new") +-- print("4 " .. file.addsuffix("name.old","new","new") .. " -> name.new") +-- print("5 " .. file.addsuffix("name.old","new","old") .. " -> name.old") +-- print("6 " .. file.addsuffix("name.old","new","foo") .. " -> name.new") +-- print("7 " .. file.addsuffix("name.old","new",{"foo","bar"}) .. " -> name.new") +-- print("8 " .. file.addsuffix("name.old","new",{"old","bar"}) .. " -> name.old") + +local suffix = period * (1-period-slashes)^1 * -1 +local pattern = Cs((1-suffix)^0) + +function file.replacesuffix(name,suffix) + if name and suffix and suffix ~= "" then + return lpegmatch(pattern,name) .. "." .. suffix + else + return name + end +end + +-- + +local reslasher = lpeg.replacer(P("\\"),"/") + +function file.reslash(str) + return str and lpegmatch(reslasher,str) +end + +-- We should be able to use: +-- +-- local writable = P(1) * P("w") * Cc(true) +-- +-- function file.is_writable(name) +-- local a = attributes(name) or attributes(pathpart(name,".")) +-- return a and lpegmatch(writable,a.permissions) or false +-- end +-- +-- But after some testing Taco and I came up with the more robust +-- variant: + +if lfs.isreadablefile and lfs.iswritablefile then + + file.is_readable = lfs.isreadablefile + file.is_writable = lfs.iswritablefile + +else + + function file.is_writable(name) + if not name then + -- error + elseif lfs.isdir(name) then + name = name .. "/m_t_x_t_e_s_t.tmp" + local f = io.open(name,"wb") + if f then + f:close() + os.remove(name) + return true + end + elseif lfs.isfile(name) then + local f = io.open(name,"ab") + if f then + f:close() + return true + end + else + local f = io.open(name,"ab") + if f then + f:close() + os.remove(name) + return true + end + end + return false + end + + local readable = P("r") * Cc(true) + + function file.is_readable(name) + if name then + local a = attributes(name) + return a and lpegmatch(readable,a.permissions) or false + else + return false + end + end + +end + +file.isreadable = file.is_readable -- depricated +file.iswritable = file.is_writable -- depricated + +function file.size(name) + if name then + local a = attributes(name) + return a and a.size or 0 + else + return 0 + end +end + +function file.splitpath(str,separator) -- string .. reslash is a bonus (we could do a direct split) + return str and checkedsplit(lpegmatch(reslasher,str),separator or io.pathseparator) +end + +function file.joinpath(tab,separator) -- table + return tab and concat(tab,separator or io.pathseparator) -- can have trailing // +end + +local someslash = S("\\/") +local stripper = Cs(P(fwslash)^0/"" * reslasher) +local isnetwork = someslash * someslash * (1-someslash) + + (1-fwslash-colon)^1 * colon +local isroot = fwslash^1 * -1 +local hasroot = fwslash^1 + +local reslasher = lpeg.replacer(S("\\/"),"/") +local deslasher = lpeg.replacer(S("\\/")^1,"/") + +-- If we have a network or prefix then there is a change that we end up with two +-- // in the middle ... we could prevent this if we (1) expand prefixes: and (2) +-- split and rebuild as url. Of course we could assume no network paths (which +-- makes sense) adn assume either mapped drives (windows) or mounts (unix) but +-- then we still have to deal with urls ... anyhow, multiple // are never a real +-- problem but just ugly. + +-- function file.join(...) +-- local lst = { ... } +-- local one = lst[1] +-- if lpegmatch(isnetwork,one) then +-- local one = lpegmatch(reslasher,one) +-- local two = lpegmatch(deslasher,concat(lst,"/",2)) +-- if lpegmatch(hasroot,two) then +-- return one .. two +-- else +-- return one .. "/" .. two +-- end +-- elseif lpegmatch(isroot,one) then +-- local two = lpegmatch(deslasher,concat(lst,"/",2)) +-- if lpegmatch(hasroot,two) then +-- return two +-- else +-- return "/" .. two +-- end +-- elseif one == "" then +-- return lpegmatch(stripper,concat(lst,"/",2)) +-- else +-- return lpegmatch(deslasher,concat(lst,"/")) +-- end +-- end + +function file.join(one, two, three, ...) + if not two then + return one == "" and one or lpegmatch(reslasher,one) + end + if one == "" then + return lpegmatch(stripper,three and concat({ two, three, ... },"/") or two) + end + if lpegmatch(isnetwork,one) then + local one = lpegmatch(reslasher,one) + local two = lpegmatch(deslasher,three and concat({ two, three, ... },"/") or two) + if lpegmatch(hasroot,two) then + return one .. two + else + return one .. "/" .. two + end + elseif lpegmatch(isroot,one) then + local two = lpegmatch(deslasher,three and concat({ two, three, ... },"/") or two) + if lpegmatch(hasroot,two) then + return two + else + return "/" .. two + end + else + return lpegmatch(deslasher,concat({ one, two, three, ... },"/")) + end +end + +-- or we can use this: +-- +-- function file.join(...) +-- local n = select("#",...) +-- local one = select(1,...) +-- if n == 1 then +-- return one == "" and one or lpegmatch(stripper,one) +-- end +-- if one == "" then +-- return lpegmatch(stripper,n > 2 and concat({ ... },"/",2) or select(2,...)) +-- end +-- if lpegmatch(isnetwork,one) then +-- local one = lpegmatch(reslasher,one) +-- local two = lpegmatch(deslasher,n > 2 and concat({ ... },"/",2) or select(2,...)) +-- if lpegmatch(hasroot,two) then +-- return one .. two +-- else +-- return one .. "/" .. two +-- end +-- elseif lpegmatch(isroot,one) then +-- local two = lpegmatch(deslasher,n > 2 and concat({ ... },"/",2) or select(2,...)) +-- if lpegmatch(hasroot,two) then +-- return two +-- else +-- return "/" .. two +-- end +-- else +-- return lpegmatch(deslasher,concat({ ... },"/")) +-- end +-- end + +-- print(file.join("c:/whatever")) +-- print(file.join("c:/whatever","name")) +-- print(file.join("//","/y")) +-- print(file.join("/","/y")) +-- print(file.join("","/y")) +-- print(file.join("/x/","/y")) +-- print(file.join("x/","/y")) +-- print(file.join("http://","/y")) +-- print(file.join("http://a","/y")) +-- print(file.join("http:///a","/y")) +-- print(file.join("//nas-1","/y")) +-- print(file.join("//nas-1/a/b/c","/y")) +-- print(file.join("\\\\nas-1\\a\\b\\c","\\y")) + +-- The previous one fails on "a.b/c" so Taco came up with a split based +-- variant. After some skyping we got it sort of compatible with the old +-- one. After that the anchoring to currentdir was added in a better way. +-- Of course there are some optimizations too. Finally we had to deal with +-- windows drive prefixes and things like sys://. Eventually gsubs and +-- finds were replaced by lpegs. + +local drivespec = R("az","AZ")^1 * colon +local anchors = fwslash + + drivespec +local untouched = periods + + (1-period)^1 * P(-1) +local mswindrive = Cs(drivespec * (bwslash/"/" + fwslash)^0) +local mswinuncpath = (bwslash + fwslash) * (bwslash + fwslash) * Cc("//") +local splitstarter = (mswindrive + mswinuncpath + Cc(false)) + * Ct(lpeg.splitat(S("/\\")^1)) +local absolute = fwslash + +function file.collapsepath(str,anchor) -- anchor: false|nil, true, "." + if not str then + return + end + if anchor == true and not lpegmatch(anchors,str) then + str = getcurrentdir() .. "/" .. str + end + if str == "" or str =="." then + return "." + elseif lpegmatch(untouched,str) then + return lpegmatch(reslasher,str) + end + local starter, oldelements = lpegmatch(splitstarter,str) + local newelements = { } + local i = #oldelements + while i > 0 do + local element = oldelements[i] + if element == '.' then + -- do nothing + elseif element == '..' then + local n = i - 1 + while n > 0 do + local element = oldelements[n] + if element ~= '..' and element ~= '.' then + oldelements[n] = '.' + break + else + n = n - 1 + end + end + if n < 1 then + insert(newelements,1,'..') + end + elseif element ~= "" then + insert(newelements,1,element) + end + i = i - 1 + end + if #newelements == 0 then + return starter or "." + elseif starter then + return starter .. concat(newelements, '/') + elseif lpegmatch(absolute,str) then + return "/" .. concat(newelements,'/') + else + newelements = concat(newelements, '/') + if anchor == "." and find(str,"^%./") then + return "./" .. newelements + else + return newelements + end + end +end + +-- local function test(str,...) +-- print(string.format("%-20s %-15s %-30s %-20s",str,file.collapsepath(str),file.collapsepath(str,true),file.collapsepath(str,"."))) +-- end +-- test("a/b.c/d") test("b.c/d") test("b.c/..") +-- test("/") test("c:/..") test("sys://..") +-- test("") test("./") test(".") test("..") test("./..") test("../..") +-- test("a") test("./a") test("/a") test("a/../..") +-- test("a/./b/..") test("a/aa/../b/bb") test("a/.././././b/..") test("a/./././b/..") +-- test("a/b/c/../..") test("./a/b/c/../..") test("a/b/c/../..") +-- test("./a") +-- test([[\\a.b.c\d\e]]) + +local validchars = R("az","09","AZ","--","..") +local pattern_a = lpeg.replacer(1-validchars) +local pattern_a = Cs((validchars + P(1)/"-")^1) +local whatever = P("-")^0 / "" +local pattern_b = Cs(whatever * (1 - whatever * -1)^1) + +function file.robustname(str,strict) + if str then + str = lpegmatch(pattern_a,str) or str + if strict then + return lpegmatch(pattern_b,str) or str -- two step is cleaner (less backtracking) + else + return str + end + end +end + +local loaddata = io.loaddata +local savedata = io.savedata + +file.readdata = loaddata +file.savedata = savedata + +function file.copy(oldname,newname) + if oldname and newname then + local data = loaddata(oldname) + if data and data ~= "" then + savedata(newname,data) + end + end +end + +-- also rewrite previous + +local letter = R("az","AZ") + S("_-+") +local separator = P("://") + +local qualified = period^0 * fwslash + + letter * colon + + letter^1 * separator + + letter^1 * fwslash +local rootbased = fwslash + + letter * colon + +lpeg.patterns.qualified = qualified +lpeg.patterns.rootbased = rootbased + +-- ./name ../name /name c: :// name/name + +function file.is_qualified_path(filename) + return filename and lpegmatch(qualified,filename) ~= nil +end + +function file.is_rootbased_path(filename) + return filename and lpegmatch(rootbased,filename) ~= nil +end + +-- function test(t) for k, v in next, t do print(v, "=>", file.splitname(v)) end end +-- +-- test { "c:", "c:/aa", "c:/aa/bb", "c:/aa/bb/cc", "c:/aa/bb/cc.dd", "c:/aa/bb/cc.dd.ee" } +-- test { "c:", "c:aa", "c:aa/bb", "c:aa/bb/cc", "c:aa/bb/cc.dd", "c:aa/bb/cc.dd.ee" } +-- test { "/aa", "/aa/bb", "/aa/bb/cc", "/aa/bb/cc.dd", "/aa/bb/cc.dd.ee" } +-- test { "aa", "aa/bb", "aa/bb/cc", "aa/bb/cc.dd", "aa/bb/cc.dd.ee" } + +-- -- maybe: +-- +-- if os.type == "windows" then +-- local currentdir = getcurrentdir +-- function getcurrentdir() +-- return lpegmatch(reslasher,currentdir()) +-- end +-- end + +-- for myself: + +function file.strip(name,dir) + if name then + local b, a = match(name,"^(.-)" .. dir .. "(.*)$") + return a ~= "" and a or name + end +end + +-- local debuglist = { +-- "pathpart", "basename", "nameonly", "suffixonly", "suffix", "dirname", "extname", +-- "addsuffix", "removesuffix", "replacesuffix", "join", +-- "strip","collapsepath", "joinpath", "splitpath", +-- } + +-- for i=1,#debuglist do +-- local name = debuglist[i] +-- local f = file[name] +-- file[name] = function(...) +-- print(name,f(...)) +-- return f(...) +-- end +-- end + +-- a goodie: a dumb version of mkdirs (not used in context itself, only +-- in generic usage) + +function lfs.mkdirs(path) + local full = "" + for sub in gmatch(path,"(/*[^\\/]+)") do -- accepts leading c: and / + full = full .. sub + -- lfs.isdir("/foo") mistakenly returns true on windows so + -- so we don't test and just make as that one is not too picky + lfs.mkdir(full) + end +end + +-- here is oen i ran into when messign around with xavante code (keppler project) +-- where it's called in_base .. no gain in using lpeg here + +function file.withinbase(path) -- don't go beyond root + local l = 0 + if not find(path,"^/") then + path = "/" .. path + end + for dir in gmatch(path,"/([^/]+)") do + if dir == ".." then + l = l - 1 + elseif dir ~= "." then + l = l + 1 + end + if l < 0 then + return false + end + end + return true +end + +-- not used in context but was in luatex once: + +local symlinkattributes = lfs.symlinkattributes + +function lfs.readlink(name) + return symlinkattributes(name,"target") or nil +end diff --git a/macros/luatex/generic/lualibs/lualibs-function.lua b/macros/luatex/generic/lualibs/lualibs-function.lua new file mode 100644 index 0000000000..7ded8ceecd --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-function.lua @@ -0,0 +1,11 @@ +if not modules then modules = { } end modules ['l-functions'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +functions = functions or { } + +function functions.dummy() end diff --git a/macros/luatex/generic/lualibs/lualibs-gzip.lua b/macros/luatex/generic/lualibs/lualibs-gzip.lua new file mode 100644 index 0000000000..5100e47222 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-gzip.lua @@ -0,0 +1,54 @@ +if not modules then modules = { } end modules ['l-gzip'] = { + version = 1.001, + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +if not gzip then + + -- no fallback yet + + return + +end + +local suffix, suffixes = file.suffix, file.suffixes + +function gzip.load(filename) + local f = io.open(filename,"rb") + if not f then + -- invalid file + elseif suffix(filename) == "gz" then + f:close() + local g = gzip.open(filename,"rb") + if g then + local str = g:read("*all") + g:close() + return str + end + else + local str = f:read("*all") + f:close() + return str + end +end + +function gzip.save(filename,data) + if suffix(filename) ~= "gz" then + filename = filename .. ".gz" + end + local f = io.open(filename,"wb") + if f then + local s = zlib.compress(data or "",9,nil,15+16) + f:write(s) + f:close() + return #s + end +end + +function gzip.suffix(filename) + local suffix, extra = suffixes(filename) + local gzipped = extra == "gz" + return suffix, gzipped +end diff --git a/macros/luatex/generic/lualibs/lualibs-io.lua b/macros/luatex/generic/lualibs/lualibs-io.lua new file mode 100644 index 0000000000..fa3ad3c4cd --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-io.lua @@ -0,0 +1,473 @@ +if not modules then modules = { } end modules ['l-io'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +local io = io +local open, flush, write, read = io.open, io.flush, io.write, io.read +local byte, find, gsub, format = string.byte, string.find, string.gsub, string.format +local concat = table.concat +----- floor = math.floor +local type = type + +if string.find(os.getenv("PATH"),";",1,true) then + io.fileseparator, io.pathseparator = "\\", ";" +else + io.fileseparator, io.pathseparator = "/" , ":" +end + +-- local function readall(f) +-- return f:read("*all") +-- end + +-- The next one is upto 50% faster on large files and less memory consumption due +-- to less intermediate large allocations. This phenomena was discussed on the +-- luatex dev list. + +local large = 0x01000000 -- 2^24 16.777.216 +local medium = 0x00100000 -- 2^20 1.048.576 +local small = 0x00020000 -- 2^17 131.072 + +-- local function readall(f) +-- local size = f:seek("end") +-- if size == 0 then +-- return "" +-- end +-- f:seek("set",0) +-- if size < medium then +-- return f:read('*all') +-- else +-- local step = (size > large) and large or (floor(size/(medium)) * small) +-- local data = { } +-- while true do +-- local r = f:read(step) +-- if not r then +-- return concat(data) +-- else +-- data[#data+1] = r +-- end +-- end +-- end +-- end + +local function readall(f) + -- return f:read("*all") + local size = f:seek("end") + if size > 0 then + f:seek("set",0) + return f:read(size) + else + return "" + end +end + +io.readall = readall + +function io.loaddata(filename,textmode) -- return nil if empty + local f = open(filename,(textmode and 'r') or 'rb') + if f then + local size = f:seek("end") + local data = nil + if size > 0 then + -- data = f:read("*all") + f:seek("set",0) + data = f:read(size) + end + f:close() + return data + end +end + +-- function io.copydata(source,target,action) +-- local f = open(source,"rb") +-- if f then +-- local g = open(target,"wb") +-- if g then +-- local size = f:seek("end") +-- if size == 0 then +-- -- empty +-- else +-- f:seek("set",0) +-- if size < medium then +-- local data = f:read('*all') +-- if action then +-- data = action(data) +-- end +-- if data then +-- g:write(data) +-- end +-- else +-- local step = (size > large) and large or (floor(size/(medium)) * small) +-- while true do +-- local data = f:read(step) +-- if data then +-- if action then +-- data = action(data) +-- end +-- if data then +-- g:write(data) +-- end +-- else +-- break +-- end +-- end +-- end +-- end +-- g:close() +-- end +-- f:close() +-- flush() +-- end +-- end + +function io.copydata(source,target,action) + local f = open(source,"rb") + if f then + local g = open(target,"wb") + if g then + local size = f:seek("end") + if size > 0 then + -- local data = f:read('*all') + f:seek("set",0) + local data = f:read(size) + if action then + data = action(data) + end + if data then + g:write(data) + end + end + g:close() + end + f:close() + flush() + end +end + +function io.savedata(filename,data,joiner) + local f = open(filename,"wb") + if f then + if type(data) == "table" then + f:write(concat(data,joiner or "")) + elseif type(data) == "function" then + data(f) + else + f:write(data or "") + end + f:close() + flush() + return true + else + return false + end +end + +-- we can also chunk this one if needed: io.lines(filename,chunksize,"*l") + +-- ffi.readline + +if fio and fio.readline then + + local readline = fio.readline + + function io.loadlines(filename,n) -- return nil if empty + local f = open(filename,'r') + if not f then + -- no file + elseif n then + local lines = { } + for i=1,n do + local line = readline(f) + if line then + lines[i] = line + else + break + end + end + f:close() + lines = concat(lines,"\n") + if #lines > 0 then + return lines + end + else + local line = readline(f) + f:close() + if line and #line > 0 then + return line + end + end + end + +else + + function io.loadlines(filename,n) -- return nil if empty + local f = open(filename,'r') + if not f then + -- no file + elseif n then + local lines = { } + for i=1,n do + local line = f:read("*lines") + if line then + lines[i] = line + else + break + end + end + f:close() + lines = concat(lines,"\n") + if #lines > 0 then + return lines + end + else + local line = f:read("*line") or "" + f:close() + if #line > 0 then + return line + end + end + end + +end + +function io.loadchunk(filename,n) + local f = open(filename,'rb') + if f then + local data = f:read(n or 1024) + f:close() + if #data > 0 then + return data + end + end +end + +function io.exists(filename) + local f = open(filename) + if f == nil then + return false + else + f:close() + return true + end +end + +function io.size(filename) + local f = open(filename) + if f == nil then + return 0 + else + local s = f:seek("end") + f:close() + return s + end +end + +local function noflines(f) + if type(f) == "string" then + local f = open(filename) + if f then + local n = f and noflines(f) or 0 + f:close() + return n + else + return 0 + end + else + -- todo: load and lpeg + local n = 0 + for _ in f:lines() do + n = n + 1 + end + f:seek('set',0) + return n + end +end + +io.noflines = noflines + +-- inlined is faster ... beware, better use util-fil + +local nextchar = { + [ 4] = function(f) + return f:read(1,1,1,1) + end, + [ 2] = function(f) + return f:read(1,1) + end, + [ 1] = function(f) + return f:read(1) + end, + [-2] = function(f) + local a, b = f:read(1,1) + return b, a + end, + [-4] = function(f) + local a, b, c, d = f:read(1,1,1,1) + return d, c, b, a + end +} + +function io.characters(f,n) + if f then + return nextchar[n or 1], f + end +end + +local nextbyte = { + [4] = function(f) + local a, b, c, d = f:read(1,1,1,1) + if d then + return byte(a), byte(b), byte(c), byte(d) + end + end, + [3] = function(f) + local a, b, c = f:read(1,1,1) + if b then + return byte(a), byte(b), byte(c) + end + end, + [2] = function(f) + local a, b = f:read(1,1) + if b then + return byte(a), byte(b) + end + end, + [1] = function (f) + local a = f:read(1) + if a then + return byte(a) + end + end, + [-2] = function (f) + local a, b = f:read(1,1) + if b then + return byte(b), byte(a) + end + end, + [-3] = function(f) + local a, b, c = f:read(1,1,1) + if b then + return byte(c), byte(b), byte(a) + end + end, + [-4] = function(f) + local a, b, c, d = f:read(1,1,1,1) + if d then + return byte(d), byte(c), byte(b), byte(a) + end + end +} + +function io.bytes(f,n) + if f then + return nextbyte[n or 1], f + else + return nil, nil + end +end + +function io.ask(question,default,options) + while true do + write(question) + if options then + write(format(" [%s]",concat(options,"|"))) + end + if default then + write(format(" [%s]",default)) + end + write(format(" ")) + flush() + local answer = read() + answer = gsub(answer,"^%s*(.*)%s*$","%1") + if answer == "" and default then + return default + elseif not options then + return answer + else + for k=1,#options do + if options[k] == answer then + return answer + end + end + local pattern = "^" .. answer + for k=1,#options do + local v = options[k] + if find(v,pattern) then + return v + end + end + end + end +end + +local function readnumber(f,n,m) -- to be replaced + if m then + f:seek("set",n) + n = m + end + if n == 1 then + return byte(f:read(1)) + elseif n == 2 then + local a, b = byte(f:read(2),1,2) + return 0x100 * a + b + elseif n == 3 then + local a, b, c = byte(f:read(3),1,3) + return 0x10000 * a + 0x100 * b + c + elseif n == 4 then + local a, b, c, d = byte(f:read(4),1,4) + return 0x1000000 * a + 0x10000 * b + 0x100 * c + d + elseif n == 8 then + local a, b = readnumber(f,4), readnumber(f,4) + return 0x100 * a + b + elseif n == 12 then + local a, b, c = readnumber(f,4), readnumber(f,4), readnumber(f,4) + return 0x10000 * a + 0x100 * b + c + elseif n == -2 then + local b, a = byte(f:read(2),1,2) + return 0x100 * a + b + elseif n == -3 then + local c, b, a = byte(f:read(3),1,3) + return 0x10000 * a + 0x100 * b + c + elseif n == -4 then + local d, c, b, a = byte(f:read(4),1,4) + return 0x1000000 * a + 0x10000 * b + 0x100*c + d + elseif n == -8 then + local h, g, f, e, d, c, b, a = byte(f:read(8),1,8) + return 0x100000000000000 * a + 0x1000000000000 * b + 0x10000000000 * c + 0x100000000 * d + + 0x1000000 * e + 0x10000 * f + 0x100 * g + h + else + return 0 + end +end + +io.readnumber = readnumber + +function io.readstring(f,n,m) + if m then + f:seek("set",n) + n = m + end + local str = gsub(f:read(n),"\000","") + return str +end + +-- This works quite ok: +-- +-- function io.piped(command,writer) +-- local pipe = io.popen(command) +-- -- for line in pipe:lines() do +-- -- print(line) +-- -- end +-- while true do +-- local line = pipe:read(1) +-- if not line then +-- break +-- elseif line ~= "\n" then +-- writer(line) +-- end +-- end +-- return pipe:close() -- ok, status, (error)code +-- end diff --git a/macros/luatex/generic/lualibs/lualibs-lpeg.lua b/macros/luatex/generic/lualibs/lualibs-lpeg.lua new file mode 100644 index 0000000000..51bc1d3df3 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-lpeg.lua @@ -0,0 +1,1196 @@ +if not modules then modules = { } end modules ['l-lpeg'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +-- we can get too many captures (e.g. on largexml files) which makes me wonder +-- if P(foo)/"" can't be simplfied to N(foo) i.e. some direct instruction to the +-- lpeg virtual machine to ignore it + +-- lpeg 12 vs lpeg 10: slower compilation, similar parsing speed (i need to check +-- if i can use new features like capture / 2 and .B (at first sight the xml +-- parser is some 5% slower) + +-- lpeg.P("abc") is faster than lpeg.P("a") * lpeg.P("b") * lpeg.P("c") + +-- a new lpeg fails on a #(1-P(":")) test and really needs a + P(-1) + +-- move utf -> l-unicode +-- move string -> l-string or keep it here + +-- lpeg.B : backward without consumption +-- lpeg.F = getmetatable(lpeg.P(1)).__len : forward without consumption + + +lpeg = require("lpeg") -- does lpeg register itself global? + +local lpeg = lpeg + +-- The latest lpeg doesn't have print any more, and even the new ones are not +-- available by default (only when debug mode is enabled), which is a pitty as +-- as it helps nailing down bottlenecks. Performance seems comparable: some 10% +-- slower pattern compilation, same parsing speed, although, +-- +-- local p = lpeg.C(lpeg.P(1)^0 * lpeg.P(-1)) +-- local a = string.rep("123",100) +-- lpeg.match(p,a) +-- +-- seems slower and is also still suboptimal (i.e. a match that runs from begin +-- to end, one of the cases where string matchers win). + +if not lpeg.print then function lpeg.print(...) print(lpeg.pcode(...)) end end + +-- tracing (only used when we encounter a problem in integration of lpeg in luatex) + +-- some code will move to unicode and string + +-- local lpmatch = lpeg.match +-- local lpprint = lpeg.print +-- local lpp = lpeg.P +-- local lpr = lpeg.R +-- local lps = lpeg.S +-- local lpc = lpeg.C +-- local lpb = lpeg.B +-- local lpv = lpeg.V +-- local lpcf = lpeg.Cf +-- local lpcb = lpeg.Cb +-- local lpcg = lpeg.Cg +-- local lpct = lpeg.Ct +-- local lpcs = lpeg.Cs +-- local lpcc = lpeg.Cc +-- local lpcmt = lpeg.Cmt +-- local lpcarg = lpeg.Carg + +-- function lpeg.match(l,...) print("LPEG MATCH") lpprint(l) return lpmatch(l,...) end + +-- function lpeg.P (l) local p = lpp (l) print("LPEG P =") lpprint(l) return p end +-- function lpeg.R (l) local p = lpr (l) print("LPEG R =") lpprint(l) return p end +-- function lpeg.S (l) local p = lps (l) print("LPEG S =") lpprint(l) return p end +-- function lpeg.C (l) local p = lpc (l) print("LPEG C =") lpprint(l) return p end +-- function lpeg.B (l) local p = lpb (l) print("LPEG B =") lpprint(l) return p end +-- function lpeg.V (l) local p = lpv (l) print("LPEG V =") lpprint(l) return p end +-- function lpeg.Cf (l) local p = lpcf (l) print("LPEG Cf =") lpprint(l) return p end +-- function lpeg.Cb (l) local p = lpcb (l) print("LPEG Cb =") lpprint(l) return p end +-- function lpeg.Cg (l) local p = lpcg (l) print("LPEG Cg =") lpprint(l) return p end +-- function lpeg.Ct (l) local p = lpct (l) print("LPEG Ct =") lpprint(l) return p end +-- function lpeg.Cs (l) local p = lpcs (l) print("LPEG Cs =") lpprint(l) return p end +-- function lpeg.Cc (l) local p = lpcc (l) print("LPEG Cc =") lpprint(l) return p end +-- function lpeg.Cmt (l) local p = lpcmt (l) print("LPEG Cmt =") lpprint(l) return p end +-- function lpeg.Carg (l) local p = lpcarg(l) print("LPEG Carg =") lpprint(l) return p end + +local type, next, tostring = type, next, tostring +local byte, char, gmatch, format = string.byte, string.char, string.gmatch, string.format +----- mod, div = math.mod, math.div +local floor = math.floor + +local P, R, S, V, Ct, C, Cs, Cc, Cp, Cmt = lpeg.P, lpeg.R, lpeg.S, lpeg.V, lpeg.Ct, lpeg.C, lpeg.Cs, lpeg.Cc, lpeg.Cp, lpeg.Cmt +local lpegtype, lpegmatch, lpegprint = lpeg.type, lpeg.match, lpeg.print + +-- let's start with an inspector: + +if setinspector then + setinspector("lpeg",function(v) if lpegtype(v) then lpegprint(v) return true end end) +end + +-- Beware, we predefine a bunch of patterns here and one reason for doing so +-- is that we get consistent behaviour in some of the visualizers. + +lpeg.patterns = lpeg.patterns or { } -- so that we can share +local patterns = lpeg.patterns + +local anything = P(1) +local endofstring = P(-1) +local alwaysmatched = P(true) + +patterns.anything = anything +patterns.endofstring = endofstring +patterns.beginofstring = alwaysmatched +patterns.alwaysmatched = alwaysmatched + +local sign = S('+-') +local zero = P('0') +local digit = R('09') +local digits = digit^1 +local octdigit = R("07") +local octdigits = octdigit^1 +local lowercase = R("az") +local uppercase = R("AZ") +local underscore = P("_") +local hexdigit = digit + lowercase + uppercase +local hexdigits = hexdigit^1 +local cr, lf, crlf = P("\r"), P("\n"), P("\r\n") +----- newline = crlf + S("\r\n") -- cr + lf +local newline = P("\r") * (P("\n") + P(true)) + P("\n") -- P("\r")^-1 * P("\n")^-1 +local escaped = P("\\") * anything +local squote = P("'") +local dquote = P('"') +local space = P(" ") +local period = P(".") +local comma = P(",") + +local utfbom_32_be = P('\000\000\254\255') -- 00 00 FE FF +local utfbom_32_le = P('\255\254\000\000') -- FF FE 00 00 +local utfbom_16_be = P('\254\255') -- FE FF +local utfbom_16_le = P('\255\254') -- FF FE +local utfbom_8 = P('\239\187\191') -- EF BB BF +local utfbom = utfbom_32_be + utfbom_32_le + + utfbom_16_be + utfbom_16_le + + utfbom_8 +local utftype = utfbom_32_be * Cc("utf-32-be") + utfbom_32_le * Cc("utf-32-le") + + utfbom_16_be * Cc("utf-16-be") + utfbom_16_le * Cc("utf-16-le") + + utfbom_8 * Cc("utf-8") + alwaysmatched * Cc("utf-8") -- assume utf8 +local utfstricttype = utfbom_32_be * Cc("utf-32-be") + utfbom_32_le * Cc("utf-32-le") + + utfbom_16_be * Cc("utf-16-be") + utfbom_16_le * Cc("utf-16-le") + + utfbom_8 * Cc("utf-8") +local utfoffset = utfbom_32_be * Cc(4) + utfbom_32_le * Cc(4) + + utfbom_16_be * Cc(2) + utfbom_16_le * Cc(2) + + utfbom_8 * Cc(3) + Cc(0) + +local utf8next = R("\128\191") + +patterns.utfbom_32_be = utfbom_32_be +patterns.utfbom_32_le = utfbom_32_le +patterns.utfbom_16_be = utfbom_16_be +patterns.utfbom_16_le = utfbom_16_le +patterns.utfbom_8 = utfbom_8 + +patterns.utf_16_be_nl = P("\000\r\000\n") + P("\000\r") + P("\000\n") -- P("\000\r") * (P("\000\n") + P(true)) + P("\000\n") +patterns.utf_16_le_nl = P("\r\000\n\000") + P("\r\000") + P("\n\000") -- P("\r\000") * (P("\n\000") + P(true)) + P("\n\000") + +patterns.utf_32_be_nl = P("\000\000\000\r\000\000\000\n") + P("\000\000\000\r") + P("\000\000\000\n") +patterns.utf_32_le_nl = P("\r\000\000\000\n\000\000\000") + P("\r\000\000\000") + P("\n\000\000\000") + +patterns.utf8one = R("\000\127") +patterns.utf8two = R("\194\223") * utf8next +patterns.utf8three = R("\224\239") * utf8next * utf8next +patterns.utf8four = R("\240\244") * utf8next * utf8next * utf8next +patterns.utfbom = utfbom +patterns.utftype = utftype +patterns.utfstricttype = utfstricttype +patterns.utfoffset = utfoffset + +local utf8char = patterns.utf8one + patterns.utf8two + patterns.utf8three + patterns.utf8four +local validutf8char = utf8char^0 * endofstring * Cc(true) + Cc(false) + +local utf8character = P(1) * R("\128\191")^0 -- unchecked but fast + +patterns.utf8 = utf8char +patterns.utf8char = utf8char +patterns.utf8character = utf8character -- this one can be used in most cases so we might use that one +patterns.validutf8 = validutf8char +patterns.validutf8char = validutf8char + +local eol = S("\n\r") +local spacer = S(" \t\f\v") -- + char(0xc2, 0xa0) if we want utf (cf mail roberto) +local whitespace = eol + spacer +local nonspacer = 1 - spacer +local nonwhitespace = 1 - whitespace + +patterns.eol = eol +patterns.spacer = spacer +patterns.whitespace = whitespace +patterns.nonspacer = nonspacer +patterns.nonwhitespace = nonwhitespace + +local stripper = spacer ^0 * C((spacer ^0 * nonspacer ^1)^0) -- from example by roberto +local fullstripper = whitespace^0 * C((whitespace^0 * nonwhitespace^1)^0) + +----- collapser = Cs(spacer^0/"" * ((spacer^1 * endofstring / "") + (spacer^1/" ") + P(1))^0) +local collapser = Cs(spacer^0/"" * nonspacer^0 * ((spacer^0/" " * nonspacer^1)^0)) +local nospacer = Cs((whitespace^1/"" + nonwhitespace^1)^0) + +local b_collapser = Cs( whitespace^0 /"" * (nonwhitespace^1 + whitespace^1/" ")^0) +local e_collapser = Cs((whitespace^1 * endofstring/"" + nonwhitespace^1 + whitespace^1/" ")^0) +local m_collapser = Cs( (nonwhitespace^1 + whitespace^1/" ")^0) + +local b_stripper = Cs( spacer^0 /"" * (nonspacer^1 + spacer^1/" ")^0) +local e_stripper = Cs((spacer^1 * endofstring/"" + nonspacer^1 + spacer^1/" ")^0) +local m_stripper = Cs( (nonspacer^1 + spacer^1/" ")^0) + +patterns.stripper = stripper +patterns.fullstripper = fullstripper +patterns.collapser = collapser +patterns.nospacer = nospacer + +patterns.b_collapser = b_collapser +patterns.m_collapser = m_collapser +patterns.e_collapser = e_collapser + +patterns.b_stripper = b_stripper +patterns.m_stripper = m_stripper +patterns.e_stripper = e_stripper + +patterns.lowercase = lowercase +patterns.uppercase = uppercase +patterns.letter = patterns.lowercase + patterns.uppercase +patterns.space = space +patterns.tab = P("\t") +patterns.spaceortab = patterns.space + patterns.tab +patterns.newline = newline +patterns.emptyline = newline^1 +patterns.equal = P("=") +patterns.comma = comma +patterns.commaspacer = comma * spacer^0 +patterns.period = period +patterns.colon = P(":") +patterns.semicolon = P(";") +patterns.underscore = underscore +patterns.escaped = escaped +patterns.squote = squote +patterns.dquote = dquote +patterns.nosquote = (escaped + (1-squote))^0 +patterns.nodquote = (escaped + (1-dquote))^0 +patterns.unsingle = (squote/"") * patterns.nosquote * (squote/"") -- will change to C in the middle +patterns.undouble = (dquote/"") * patterns.nodquote * (dquote/"") -- will change to C in the middle +patterns.unquoted = patterns.undouble + patterns.unsingle -- more often undouble +patterns.unspacer = ((patterns.spacer^1)/"")^0 + +patterns.singlequoted = squote * patterns.nosquote * squote +patterns.doublequoted = dquote * patterns.nodquote * dquote +patterns.quoted = patterns.doublequoted + patterns.singlequoted + +patterns.digit = digit +patterns.digits = digits +patterns.octdigit = octdigit +patterns.octdigits = octdigits +patterns.hexdigit = hexdigit +patterns.hexdigits = hexdigits +patterns.sign = sign +patterns.cardinal = digits +patterns.integer = sign^-1 * digits +patterns.unsigned = digit^0 * period * digits +patterns.float = sign^-1 * patterns.unsigned +patterns.cunsigned = digit^0 * comma * digits +patterns.cpunsigned = digit^0 * (period + comma) * digits +patterns.cfloat = sign^-1 * patterns.cunsigned +patterns.cpfloat = sign^-1 * patterns.cpunsigned +patterns.number = patterns.float + patterns.integer +patterns.cnumber = patterns.cfloat + patterns.integer +patterns.cpnumber = patterns.cpfloat + patterns.integer +patterns.oct = zero * octdigits -- hm is this ok +patterns.octal = patterns.oct +patterns.HEX = zero * P("X") * (digit+uppercase)^1 +patterns.hex = zero * P("x") * (digit+lowercase)^1 +patterns.hexadecimal = zero * S("xX") * hexdigits + +patterns.hexafloat = sign^-1 + * zero * S("xX") + * (hexdigit^0 * period * hexdigits + hexdigits * period * hexdigit^0 + hexdigits) + * (S("pP") * sign^-1 * hexdigits)^-1 +patterns.decafloat = sign^-1 + * (digit^0 * period * digits + digits * period * digit^0 + digits) + * S("eE") * sign^-1 * digits + +patterns.propername = (uppercase + lowercase + underscore) * (uppercase + lowercase + underscore + digit)^0 * endofstring + +patterns.somecontent = (anything - newline - space)^1 -- (utf8char - newline - space)^1 +patterns.beginline = #(1-newline) + +patterns.longtostring = Cs(whitespace^0/"" * ((patterns.quoted + nonwhitespace^1 + whitespace^1/"" * (endofstring + Cc(" ")))^0)) + +-- local function anywhere(pattern) -- slightly adapted from website +-- return P { P(pattern) + 1 * V(1) } +-- end + +function anywhere(pattern) -- faster + return (1-P(pattern))^0 * P(pattern) +end + +lpeg.anywhere = anywhere + +function lpeg.instringchecker(p) + p = anywhere(p) + return function(str) + return lpegmatch(p,str) and true or false + end +end + +-- function lpeg.splitter(pattern, action) +-- return (((1-P(pattern))^1)/action+1)^0 +-- end + +-- function lpeg.tsplitter(pattern, action) +-- return Ct((((1-P(pattern))^1)/action+1)^0) +-- end + +function lpeg.splitter(pattern, action) + if action then + return (((1-P(pattern))^1)/action+1)^0 + else + return (Cs((1-P(pattern))^1)+1)^0 + end +end + +function lpeg.tsplitter(pattern, action) + if action then + return Ct((((1-P(pattern))^1)/action+1)^0) + else + return Ct((Cs((1-P(pattern))^1)+1)^0) + end +end + +-- probleem: separator can be lpeg and that does not hash too well, but +-- it's quite okay as the key is then not garbage collected + +local splitters_s, splitters_m, splitters_t = { }, { }, { } + +local function splitat(separator,single) + local splitter = (single and splitters_s[separator]) or splitters_m[separator] + if not splitter then + separator = P(separator) + local other = C((1 - separator)^0) + if single then + local any = anything + splitter = other * (separator * C(any^0) + "") -- ? + splitters_s[separator] = splitter + else + splitter = other * (separator * other)^0 + splitters_m[separator] = splitter + end + end + return splitter +end + +local function tsplitat(separator) + local splitter = splitters_t[separator] + if not splitter then + splitter = Ct(splitat(separator)) + splitters_t[separator] = splitter + end + return splitter +end + +lpeg.splitat = splitat +lpeg.tsplitat = tsplitat + +function string.splitup(str,separator) + if not separator then + separator = "," + end + return lpegmatch(splitters_m[separator] or splitat(separator),str) +end + +-- local p = splitat("->",false) print(lpegmatch(p,"oeps->what->more")) -- oeps what more +-- local p = splitat("->",true) print(lpegmatch(p,"oeps->what->more")) -- oeps what->more +-- local p = splitat("->",false) print(lpegmatch(p,"oeps")) -- oeps +-- local p = splitat("->",true) print(lpegmatch(p,"oeps")) -- oeps + +local cache = { } + +function lpeg.split(separator,str) + local c = cache[separator] + if not c then + c = tsplitat(separator) + cache[separator] = c + end + return lpegmatch(c,str) +end + +function string.split(str,separator) + if separator then + local c = cache[separator] + if not c then + c = tsplitat(separator) + cache[separator] = c + end + return lpegmatch(c,str) + else + return { str } + end +end + +local spacing = patterns.spacer^0 * newline -- sort of strip +local empty = spacing * Cc("") +local nonempty = Cs((1-spacing)^1) * spacing^-1 +local content = (empty + nonempty)^1 + +patterns.textline = content + +local linesplitter = tsplitat(newline) + +patterns.linesplitter = linesplitter + +function string.splitlines(str) + return lpegmatch(linesplitter,str) +end + +-- lpeg.splitters = cache -- no longer public + +local cache = { } + +function lpeg.checkedsplit(separator,str) + local c = cache[separator] + if not c then + separator = P(separator) + local other = C((1 - separator)^1) + c = Ct(separator^0 * other * (separator^1 * other)^0) + cache[separator] = c + end + return lpegmatch(c,str) +end + +function string.checkedsplit(str,separator) + local c = cache[separator] + if not c then + separator = P(separator) + local other = C((1 - separator)^1) + c = Ct(separator^0 * other * (separator^1 * other)^0) + cache[separator] = c + end + return lpegmatch(c,str) +end + +-- from roberto's site: + +local function f2(s) local c1, c2 = byte(s,1,2) return c1 * 64 + c2 - 12416 end +local function f3(s) local c1, c2, c3 = byte(s,1,3) return (c1 * 64 + c2) * 64 + c3 - 925824 end +local function f4(s) local c1, c2, c3, c4 = byte(s,1,4) return ((c1 * 64 + c2) * 64 + c3) * 64 + c4 - 63447168 end + +local utf8byte = patterns.utf8one/byte + patterns.utf8two/f2 + patterns.utf8three/f3 + patterns.utf8four/f4 + +patterns.utf8byte = utf8byte + +--~ local str = " a b c d " + +--~ local s = lpeg.stripper(lpeg.R("az")) print("["..lpegmatch(s,str).."]") +--~ local s = lpeg.keeper(lpeg.R("az")) print("["..lpegmatch(s,str).."]") +--~ local s = lpeg.stripper("ab") print("["..lpegmatch(s,str).."]") +--~ local s = lpeg.keeper("ab") print("["..lpegmatch(s,str).."]") + +local cache = { } + +function lpeg.stripper(str) + if type(str) == "string" then + local s = cache[str] + if not s then + s = Cs(((S(str)^1)/"" + 1)^0) + cache[str] = s + end + return s + else + return Cs(((str^1)/"" + 1)^0) + end +end + +local cache = { } + +function lpeg.keeper(str) + if type(str) == "string" then + local s = cache[str] + if not s then + s = Cs((((1-S(str))^1)/"" + 1)^0) + cache[str] = s + end + return s + else + return Cs((((1-str)^1)/"" + 1)^0) + end +end + +function lpeg.frontstripper(str) -- or pattern (yet undocumented) + return (P(str) + P(true)) * Cs(anything^0) +end + +function lpeg.endstripper(str) -- or pattern (yet undocumented) + return Cs((1 - P(str) * endofstring)^0) +end + +-- Just for fun I looked at the used bytecode and +-- p = (p and p + pp) or pp gets one more (testset). + +-- todo: cache when string + +function lpeg.replacer(one,two,makefunction,isutf) -- in principle we should sort the keys + local pattern + local u = isutf and utf8char or 1 + if type(one) == "table" then + local no = #one + local p = P(false) + if no == 0 then + for k, v in next, one do + p = p + P(k) / v + end + pattern = Cs((p + u)^0) + elseif no == 1 then + local o = one[1] + one, two = P(o[1]), o[2] + -- pattern = Cs(((1-one)^1 + one/two)^0) + pattern = Cs((one/two + u)^0) + else + for i=1,no do + local o = one[i] + p = p + P(o[1]) / o[2] + end + pattern = Cs((p + u)^0) + end + else + pattern = Cs((P(one)/(two or "") + u)^0) + end + if makefunction then + return function(str) + return lpegmatch(pattern,str) + end + else + return pattern + end +end + +-- local pattern1 = P(1-P(pattern))^0 * P(pattern) : test for not nil +-- local pattern2 = (P(pattern) * Cc(true) + P(1))^0 : test for true (could be faster, but not much) + +function lpeg.finder(lst,makefunction,isutf) -- beware: slower than find with 'patternless finds' + local pattern + if type(lst) == "table" then + pattern = P(false) + if #lst == 0 then + for k, v in next, lst do + pattern = pattern + P(k) -- ignore key, so we can use a replacer table + end + else + for i=1,#lst do + pattern = pattern + P(lst[i]) + end + end + else + pattern = P(lst) + end + if isutf then + pattern = ((utf8char or 1)-pattern)^0 * pattern + else + pattern = (1-pattern)^0 * pattern + end + if makefunction then + return function(str) + return lpegmatch(pattern,str) + end + else + return pattern + end +end + +-- print(lpeg.match(lpeg.replacer("e","a"),"test test")) +-- print(lpeg.match(lpeg.replacer{{"e","a"}},"test test")) +-- print(lpeg.match(lpeg.replacer({ e = "a", t = "x" }),"test test")) + +local splitters_f, splitters_s = { }, { } + +function lpeg.firstofsplit(separator) -- always return value + local splitter = splitters_f[separator] + if not splitter then + local pattern = P(separator) + splitter = C((1 - pattern)^0) + splitters_f[separator] = splitter + end + return splitter +end + +function lpeg.secondofsplit(separator) -- nil if not split + local splitter = splitters_s[separator] + if not splitter then + local pattern = P(separator) + splitter = (1 - pattern)^0 * pattern * C(anything^0) + splitters_s[separator] = splitter + end + return splitter +end + +local splitters_s, splitters_p = { }, { } + +function lpeg.beforesuffix(separator) -- nil if nothing but empty is ok + local splitter = splitters_s[separator] + if not splitter then + local pattern = P(separator) + splitter = C((1 - pattern)^0) * pattern * endofstring + splitters_s[separator] = splitter + end + return splitter +end + +function lpeg.afterprefix(separator) -- nil if nothing but empty is ok + local splitter = splitters_p[separator] + if not splitter then + local pattern = P(separator) + splitter = pattern * C(anything^0) + splitters_p[separator] = splitter + end + return splitter +end + +function lpeg.balancer(left,right) + left, right = P(left), P(right) + return P { left * ((1 - left - right) + V(1))^0 * right } +end + +-- print(1,lpegmatch(lpeg.firstofsplit(":"),"bc:de")) +-- print(2,lpegmatch(lpeg.firstofsplit(":"),":de")) -- empty +-- print(3,lpegmatch(lpeg.firstofsplit(":"),"bc")) +-- print(4,lpegmatch(lpeg.secondofsplit(":"),"bc:de")) +-- print(5,lpegmatch(lpeg.secondofsplit(":"),"bc:")) -- empty +-- print(6,lpegmatch(lpeg.secondofsplit(":",""),"bc")) +-- print(7,lpegmatch(lpeg.secondofsplit(":"),"bc")) +-- print(9,lpegmatch(lpeg.secondofsplit(":","123"),"bc")) + +-- this was slower but lpeg has been sped up in the meantime, so we no longer +-- use this (still seems somewhat faster on long strings) +-- +-- local nany = utf8char/"" +-- +-- function lpeg.counter(pattern) +-- pattern = Cs((P(pattern)/" " + nany)^0) +-- return function(str) +-- return #lpegmatch(pattern,str) +-- end +-- end + +function lpeg.counter(pattern,action) + local n = 0 + local pattern = (P(pattern) / function() n = n + 1 end + anything)^0 + ----- pattern = (P(pattern) * (P(true) / function() n = n + 1 end) + anything)^0 + ----- pattern = (P(pattern) * P(function() n = n + 1 end) + anything)^0 + if action then + return function(str) n = 0 ; lpegmatch(pattern,str) ; action(n) end + else + return function(str) n = 0 ; lpegmatch(pattern,str) ; return n end + end +end + +-- lpeg.print(lpeg.R("ab","cd","gh")) +-- lpeg.print(lpeg.P("a","b","c")) +-- lpeg.print(lpeg.S("a","b","c")) + +-- print(lpeg.count("äáàa",lpeg.P("á") + lpeg.P("à"))) +-- print(lpeg.count("äáàa",lpeg.UP("áà"))) +-- print(lpeg.count("äáàa",lpeg.US("àá"))) +-- print(lpeg.count("äáàa",lpeg.UR("aá"))) +-- print(lpeg.count("äáàa",lpeg.UR("àá"))) +-- print(lpeg.count("äáàa",lpeg.UR(0x0000,0xFFFF))) + +function lpeg.is_lpeg(p) + return p and lpegtype(p) == "pattern" +end + +function lpeg.oneof(list,...) -- lpeg.oneof("elseif","else","if","then") -- assume proper order + if type(list) ~= "table" then + list = { list, ... } + end + -- table.sort(list) -- longest match first + local p = P(list[1]) + for l=2,#list do + p = p + P(list[l]) + end + return p +end + +-- For the moment here, but it might move to utilities. Beware, we need to +-- have the longest keyword first, so 'aaa' comes beforte 'aa' which is why we +-- loop back from the end cq. prepend. + +local sort = table.sort + +local function copyindexed(old) + local new = { } + for i=1,#old do + new[i] = old + end + return new +end + +local function sortedkeys(tab) + local keys, s = { }, 0 + for key,_ in next, tab do + s = s + 1 + keys[s] = key + end + sort(keys) + return keys +end + +function lpeg.append(list,pp,delayed,checked) + local p = pp + if #list > 0 then + local keys = copyindexed(list) + sort(keys) + for i=#keys,1,-1 do + local k = keys[i] + if p then + p = P(k) + p + else + p = P(k) + end + end + elseif delayed then -- hm, it looks like the lpeg parser resolves anyway + local keys = sortedkeys(list) + if p then + for i=1,#keys,1 do + local k = keys[i] + local v = list[k] + p = P(k)/list + p + end + else + for i=1,#keys do + local k = keys[i] + local v = list[k] + if p then + p = P(k) + p + else + p = P(k) + end + end + if p then + p = p / list + end + end + elseif checked then + -- problem: substitution gives a capture + local keys = sortedkeys(list) + for i=1,#keys do + local k = keys[i] + local v = list[k] + if p then + if k == v then + p = P(k) + p + else + p = P(k)/v + p + end + else + if k == v then + p = P(k) + else + p = P(k)/v + end + end + end + else + local keys = sortedkeys(list) + for i=1,#keys do + local k = keys[i] + local v = list[k] + if p then + p = P(k)/v + p + else + p = P(k)/v + end + end + end + return p +end + +-- inspect(lpeg.append({ a = "1", aa = "1", aaa = "1" } ,nil,true)) +-- inspect(lpeg.append({ ["degree celsius"] = "1", celsius = "1", degree = "1" } ,nil,true)) + +-- function lpeg.exact_match(words,case_insensitive) +-- local pattern = concat(words) +-- if case_insensitive then +-- local pattern = S(upper(characters)) + S(lower(characters)) +-- local list = { } +-- for i=1,#words do +-- list[lower(words[i])] = true +-- end +-- return Cmt(pattern^1, function(_,i,s) +-- return list[lower(s)] and i +-- end) +-- else +-- local pattern = S(concat(words)) +-- local list = { } +-- for i=1,#words do +-- list[words[i]] = true +-- end +-- return Cmt(pattern^1, function(_,i,s) +-- return list[s] and i +-- end) +-- end +-- end + +-- experiment: + +local p_false = P(false) +local p_true = P(true) + +-- local function collapse(t,x) +-- if type(t) ~= "table" then +-- return t, x +-- else +-- local n = next(t) +-- if n == nil then +-- return t, x +-- elseif next(t,n) == nil then +-- -- one entry +-- local k = n +-- local v = t[k] +-- if type(v) == "table" then +-- return collapse(v,x..k) +-- else +-- return v, x .. k +-- end +-- else +-- local tt = { } +-- for k, v in next, t do +-- local vv, kk = collapse(v,k) +-- tt[kk] = vv +-- end +-- return tt, x +-- end +-- end +-- end + +local lower = utf and utf.lower or string.lower +local upper = utf and utf.upper or string.upper + +function lpeg.setutfcasers(l,u) + lower = l or lower + upper = u or upper +end + +local function make1(t,rest) + local p = p_false + local keys = sortedkeys(t) + for i=1,#keys do + local k = keys[i] + if k ~= "" then + local v = t[k] + if v == true then + p = p + P(k) * p_true + elseif v == false then + -- can't happen + else + p = p + P(k) * make1(v,v[""]) + end + end + end + if rest then + p = p + p_true + end + return p +end + +local function make2(t,rest) -- only ascii + local p = p_false + local keys = sortedkeys(t) + for i=1,#keys do + local k = keys[i] + if k ~= "" then + local v = t[k] + if v == true then + p = p + (P(lower(k))+P(upper(k))) * p_true + elseif v == false then + -- can't happen + else + p = p + (P(lower(k))+P(upper(k))) * make2(v,v[""]) + end + end + end + if rest then + p = p + p_true + end + return p +end + +local function utfchartabletopattern(list,insensitive) -- goes to util-lpg + local tree = { } + local n = #list + if n == 0 then + for s in next, list do + local t = tree + local p, pk + for c in gmatch(s,".") do + if t == true then + t = { [c] = true, [""] = true } + p[pk] = t + p = t + t = false + elseif t == false then + t = { [c] = false } + p[pk] = t + p = t + t = false + else + local tc = t[c] + if not tc then + tc = false + t[c] = false + end + p = t + t = tc + end + pk = c + end + if t == false then + p[pk] = true + elseif t == true then + -- okay + else + t[""] = true + end + end + else + for i=1,n do + local s = list[i] + local t = tree + local p, pk + for c in gmatch(s,".") do + if t == true then + t = { [c] = true, [""] = true } + p[pk] = t + p = t + t = false + elseif t == false then + t = { [c] = false } + p[pk] = t + p = t + t = false + else + local tc = t[c] + if not tc then + tc = false + t[c] = false + end + p = t + t = tc + end + pk = c + end + if t == false then + p[pk] = true + elseif t == true then + -- okay + else + t[""] = true + end + end + end + -- collapse(tree,"") -- needs testing, maybe optional, slightly faster because P("x")*P("X") seems slower than P"(xX") (why) + -- inspect(tree) + return (insensitive and make2 or make1)(tree) +end + +lpeg.utfchartabletopattern = utfchartabletopattern + +function lpeg.utfreplacer(list,insensitive) + local pattern = Cs((utfchartabletopattern(list,insensitive)/list + utf8character)^0) + return function(str) + return lpegmatch(pattern,str) or str + end +end + +-- local t = { "start", "stoep", "staart", "paard" } +-- local p = lpeg.Cs((lpeg.utfchartabletopattern(t)/string.upper + 1)^1) + +-- local t = { "a", "abc", "ac", "abe", "abxyz", "xy", "bef","aa" } +-- local p = lpeg.Cs((lpeg.utfchartabletopattern(t)/string.upper + 1)^1) + +-- inspect(lpegmatch(p,"a")=="A") +-- inspect(lpegmatch(p,"aa")=="AA") +-- inspect(lpegmatch(p,"aaaa")=="AAAA") +-- inspect(lpegmatch(p,"ac")=="AC") +-- inspect(lpegmatch(p,"bc")=="bc") +-- inspect(lpegmatch(p,"zzbczz")=="zzbczz") +-- inspect(lpegmatch(p,"zzabezz")=="zzABEzz") +-- inspect(lpegmatch(p,"ab")=="Ab") +-- inspect(lpegmatch(p,"abc")=="ABC") +-- inspect(lpegmatch(p,"abe")=="ABE") +-- inspect(lpegmatch(p,"xa")=="xA") +-- inspect(lpegmatch(p,"bx")=="bx") +-- inspect(lpegmatch(p,"bax")=="bAx") +-- inspect(lpegmatch(p,"abxyz")=="ABXYZ") +-- inspect(lpegmatch(p,"foobarbefcrap")=="foobArBEFcrAp") + +-- local t = { ["^"] = 1, ["^^"] = 2, ["^^^"] = 3, ["^^^^"] = 4 } +-- local p = lpeg.Cs((lpeg.utfchartabletopattern(t)/t + 1)^1) +-- inspect(lpegmatch(p," ^ ^^ ^^^ ^^^^ ^^^^^ ^^^^^^ ^^^^^^^ ")) + +-- local t = { ["^^"] = 2, ["^^^"] = 3, ["^^^^"] = 4 } +-- local p = lpeg.Cs((lpeg.utfchartabletopattern(t)/t + 1)^1) +-- inspect(lpegmatch(p," ^ ^^ ^^^ ^^^^ ^^^^^ ^^^^^^ ^^^^^^^ ")) + +-- lpeg.utfchartabletopattern { +-- utfchar(0x00A0), -- nbsp +-- utfchar(0x2000), -- enquad +-- utfchar(0x2001), -- emquad +-- utfchar(0x2002), -- enspace +-- utfchar(0x2003), -- emspace +-- utfchar(0x2004), -- threeperemspace +-- utfchar(0x2005), -- fourperemspace +-- utfchar(0x2006), -- sixperemspace +-- utfchar(0x2007), -- figurespace +-- utfchar(0x2008), -- punctuationspace +-- utfchar(0x2009), -- breakablethinspace +-- utfchar(0x200A), -- hairspace +-- utfchar(0x200B), -- zerowidthspace +-- utfchar(0x202F), -- narrownobreakspace +-- utfchar(0x205F), -- math thinspace +-- } + +-- a few handy ones: +-- +-- faster than find(str,"[\n\r]") when match and # > 7 and always faster when # > 3 + +patterns.containseol = lpeg.finder(eol) -- (1-eol)^0 * eol + +-- The next pattern^n variant is based on an approach suggested +-- by Roberto: constructing a big repetition in chunks. +-- +-- Being sparse is not needed, and only complicate matters and +-- the number of redundant entries is not that large. + +local function nextstep(n,step,result) + local m = n % step -- mod(n,step) + local d = floor(n/step) -- div(n,step) + if d > 0 then + local v = V(tostring(step)) + local s = result.start + for i=1,d do + if s then + s = v * s + else + s = v + end + end + result.start = s + end + if step > 1 and result.start then + local v = V(tostring(step/2)) + result[tostring(step)] = v * v + end + if step > 0 then + return nextstep(m,step/2,result) + else + return result + end +end + +function lpeg.times(pattern,n) + return P(nextstep(n,2^16,{ "start", ["1"] = pattern })) +end + +-- local p = lpeg.Cs((1 - lpeg.times(lpeg.P("AB"),25))^1) +-- local s = "12" .. string.rep("AB",20) .. "34" .. string.rep("AB",30) .. "56" +-- inspect(p) +-- print(lpeg.match(p,s)) + +-- moved here (before util-str) + +do + + local trailingzeros = zero^0 * -digit -- suggested by Roberto + local stripper = Cs(( + digits * ( + period * trailingzeros / "" + + period * (digit - trailingzeros)^1 * (trailingzeros / "") + ) + 1 + )^0) + + lpeg.patterns.stripzeros = stripper -- multiple in string + + local nonzero = digit - zero + local trailingzeros = zero^1 * endofstring + local stripper = Cs( (1-period)^0 * ( + period * trailingzeros/"" + + period * (nonzero^1 + (trailingzeros/"") + zero^1)^0 + + endofstring + )) + + lpeg.patterns.stripzero = stripper -- slightly more efficient but expects a float ! + + -- local sample = "bla 11.00 bla 11 bla 0.1100 bla 1.00100 bla 0.00 bla 0.001 bla 1.1100 bla 0.100100100 bla 0.00100100100" + -- collectgarbage("collect") + -- str = string.rep(sample,10000) + -- local ts = os.clock() + -- lpegmatch(stripper,str) + -- print(#str, os.clock()-ts, lpegmatch(stripper,sample)) + +end + +-- for practical reasons we keep this here: + +local byte_to_HEX = { } +local byte_to_hex = { } +local byte_to_dec = { } -- for md5 +local hex_to_byte = { } + +for i=0,255 do + local H = format("%02X",i) + local h = format("%02x",i) + local d = format("%03i",i) + local c = char(i) + byte_to_HEX[c] = H + byte_to_hex[c] = h + byte_to_dec[c] = d + hex_to_byte[h] = c + hex_to_byte[H] = c +end + +local hextobyte = P(2)/hex_to_byte +local bytetoHEX = P(1)/byte_to_HEX +local bytetohex = P(1)/byte_to_hex +local bytetodec = P(1)/byte_to_dec +local hextobytes = Cs(hextobyte^0) +local bytestoHEX = Cs(bytetoHEX^0) +local bytestohex = Cs(bytetohex^0) +local bytestodec = Cs(bytetodec^0) + +patterns.hextobyte = hextobyte +patterns.bytetoHEX = bytetoHEX +patterns.bytetohex = bytetohex +patterns.bytetodec = bytetodec +patterns.hextobytes = hextobytes +patterns.bytestoHEX = bytestoHEX +patterns.bytestohex = bytestohex +patterns.bytestodec = bytestodec + +function string.toHEX(s) + if not s or s == "" then + return s + else + return lpegmatch(bytestoHEX,s) + end +end + +function string.tohex(s) + if not s or s == "" then + return s + else + return lpegmatch(bytestohex,s) + end +end + +function string.todec(s) + if not s or s == "" then + return s + else + return lpegmatch(bytestodec,s) + end +end + +function string.tobytes(s) + if not s or s == "" then + return s + else + return lpegmatch(hextobytes,s) + end +end + +-- local h = "ADFE0345" +-- local b = lpegmatch(patterns.hextobytes,h) +-- print(h,b,string.tohex(b),string.toHEX(b)) + +local patterns = { } -- can be made weak + +local function containsws(what) + local p = patterns[what] + if not p then + local p1 = P(what) * (whitespace + endofstring) * Cc(true) + local p2 = whitespace * P(p1) + p = P(p1) + P(1-p2)^0 * p2 + Cc(false) + patterns[what] = p + end + return p +end + +lpeg.containsws = containsws + +function string.containsws(str,what) + return lpegmatch(patterns[what] or containsws(what),str) +end diff --git a/macros/luatex/generic/lualibs/lualibs-lua.lua b/macros/luatex/generic/lualibs/lualibs-lua.lua new file mode 100644 index 0000000000..6e9606e849 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-lua.lua @@ -0,0 +1,259 @@ +if not modules then modules = { } end modules ['l-lua'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +-- potential issues with 5.3: + +-- i'm not sure yet if the int/float change is good for luatex + +-- math.min +-- math.max +-- tostring +-- tonumber +-- utf.* +-- bit32 + +local next, type, tonumber = next, type, tonumber + +-- compatibility hacks and helpers + +LUAMAJORVERSION, LUAMINORVERSION = string.match(_VERSION,"^[^%d]+(%d+)%.(%d+).*$") + +LUAMAJORVERSION = tonumber(LUAMAJORVERSION) or 5 +LUAMINORVERSION = tonumber(LUAMINORVERSION) or 1 +LUAVERSION = LUAMAJORVERSION + LUAMINORVERSION/10 + +if LUAVERSION < 5.2 and jit then + -- + -- we want loadstring cum suis to behave like 5.2 + -- + MINORVERSION = 2 + LUAVERSION = 5.2 +end + +-- lpeg + +if not lpeg then + lpeg = require("lpeg") +end + +-- if utf8 then +-- utf8lua = utf8 +-- utf8 = nil +-- end + +-- basics: + +if loadstring then + + local loadnormal = load + + function load(first,...) + if type(first) == "string" then + return loadstring(first,...) + else + return loadnormal(first,...) + end + end + +else + + loadstring = load + +end + +-- table: + +-- At some point it was announced that i[pairs would be dropped, which makes +-- sense. As we already used the for loop and # in most places the impact on +-- ConTeXt was not that large; the remaining ipairs already have been replaced. +-- Hm, actually ipairs was retained, but we no longer use it anyway (nor +-- pairs). +-- +-- Just in case, we provide the fallbacks as discussed in Programming +-- in Lua (http://www.lua.org/pil/7.3.html): + +if not ipairs then + + -- for k, v in ipairs(t) do ... end + -- for k=1,#t do local v = t[k] ... end + + local function iterate(a,i) + i = i + 1 + local v = a[i] + if v ~= nil then + return i, v --, nil + end + end + + function ipairs(a) + return iterate, a, 0 + end + +end + +if not pairs then + + -- for k, v in pairs(t) do ... end + -- for k, v in next, t do ... end + + function pairs(t) + return next, t -- , nil + end + +end + +-- The unpack function has been moved to the table table, and for compatiility +-- reasons we provide both now. + +if not table.unpack then + + table.unpack = _G.unpack + +elseif not unpack then + + _G.unpack = table.unpack + +end + +-- package: + +-- if not package.seachers then +-- +-- package.searchers = package.loaders -- 5.2 +-- +-- elseif not package.loaders then +-- +-- package.loaders = package.searchers +-- +-- end + +if not package.loaders then -- brr, searchers is a special "loadlib function" userdata type + + package.loaders = package.searchers + +end + +-- moved from util-deb to here: + +local print, select, tostring = print, select, tostring + +local inspectors = { } + +function setinspector(kind,inspector) -- global function + inspectors[kind] = inspector +end + +function inspect(...) -- global function + for s=1,select("#",...) do + local value = select(s,...) + if value == nil then + print("nil") + else + local done = false + -- type driven (table) + local kind = type(value) + local inspector = inspectors[kind] + if inspector then + done = inspector(value) + if done then + break + end + end + -- whatever driven (token, node, ...) + for kind, inspector in next, inspectors do + done = inspector(value) + if done then + break + end + end + if not done then + print(tostring(value)) + end + end + end +end + +-- + +local dummy = function() end + +function optionalrequire(...) + local ok, result = xpcall(require,dummy,...) + if ok then + return result + end +end + +-- nice for non ascii scripts (this might move): + +if lua then + lua.mask = load([[τεχ = 1]]) and "utf" or "ascii" +end + +local flush = io.flush + +if flush then + + local execute = os.execute if execute then function os.execute(...) flush() return execute(...) end end + local exec = os.exec if exec then function os.exec (...) flush() return exec (...) end end + local spawn = os.spawn if spawn then function os.spawn (...) flush() return spawn (...) end end + local popen = io.popen if popen then function io.popen (...) flush() return popen (...) end end + +end + +-- new + +FFISUPPORTED = type(ffi) == "table" and ffi.os ~= "" and ffi.arch ~= "" and ffi.load + +if not FFISUPPORTED then + + -- Maybe we should check for LUATEXENGINE but that's also a bit tricky as we still + -- can have a weird ffi library laying around. Checking for presence of 'jit' is + -- also not robust. So for now we hope for the best. + + local okay ; okay, ffi = pcall(require,"ffi") + + FFISUPPORTED = type(ffi) == "table" and ffi.os ~= "" and ffi.arch ~= "" and ffi.load + +end + +if not FFISUPPORTED then + ffi = nil +elseif not ffi.number then + ffi.number = tonumber +end + +-- if not bit32 then -- and utf8 then +-- -- bit32 = load ( [[ -- replacement code with 5.3 syntax so that 5.2 doesn't bark on it ]] ) +-- bit32 = require("l-bit32") +-- end + +-- We need this due a bug in luatex socket loading: + +-- local loaded = package.loaded +-- +-- if not loaded["socket"] then loaded["socket"] = loaded["socket.core"] end +-- if not loaded["mime"] then loaded["mime"] = loaded["mime.core"] end +-- +-- if not socket.mime then socket.mime = package.loaded["mime"] end +-- +-- if not loaded["socket.mime"] then loaded["socket.mime"] = socket.mime end +-- if not loaded["socket.http"] then loaded["socket.http"] = socket.http end +-- if not loaded["socket.ftp"] then loaded["socket.ftp"] = socket.ftp end +-- if not loaded["socket.smtp"] then loaded["socket.smtp"] = socket.smtp end +-- if not loaded["socket.tp"] then loaded["socket.tp"] = socket.tp end +-- if not loaded["socket.url"] then loaded["socket.url"] = socket.url end + +if LUAVERSION > 5.3 then + -- collectgarbage("collect") + -- collectgarbage("generational") -- crashes on unix +end + +if status and os.setenv then + os.setenv("engine",string.lower(status.luatex_engine or "unknown")) +end diff --git a/macros/luatex/generic/lualibs/lualibs-math.lua b/macros/luatex/generic/lualibs/lualibs-math.lua new file mode 100644 index 0000000000..e5668a5db0 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-math.lua @@ -0,0 +1,149 @@ +if not modules then modules = { } end modules ['l-math'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +if not math.ceiling then + + math.ceiling = math.ceil + +end + +if not math.round then + + local floor = math.floor + + function math.round(x) return floor(x + 0.5) end + +end + +if not math.div then + + local floor = math.floor + + function math.div(n,m) return floor(n/m) end + +end + +if not math.mod then + + function math.mod(n,m) return n % m end + +end + +if not math.sind then + + local sin, cos, tan = math.sin, math.cos, math.tan + + local pipi = 2*math.pi/360 + + function math.sind(d) return sin(d*pipi) end + function math.cosd(d) return cos(d*pipi) end + function math.tand(d) return tan(d*pipi) end + +end + +if not math.odd then + + function math.odd (n) return n % 2 ~= 0 end + function math.even(n) return n % 2 == 0 end + +end + +if not math.cosh then + + local exp = math.exp + + function math.cosh(x) + local xx = exp(x) + return (xx+1/xx)/2 + end + function math.sinh(x) + local xx = exp(x) + return (xx-1/xx)/2 + end + function math.tanh(x) + local xx = exp(x) + return (xx-1/xx)/(xx+1/xx) + end + +end + +if not math.pow then + + function math.pow(x,y) + return x^y + end + +end + +if not math.atan2 then + + math.atan2 = math.atan + +end + +if not math.ldexp then + + function math.ldexp(x,e) + return x * 2.0^e + end + +end + +-- if not math.frexp then +-- +-- -- not a oneliner so use a math library instead +-- +-- function math.frexp(x,e) +-- -- returns m and e such that x = m2e, e is an integer and the absolute +-- -- value of m is in the range [0.5, 1) (or zero when x is zero) +-- end +-- +-- end + +if not math.log10 then + + local log = math.log + + function math.log10(x) + return log(x,10) + end + +end + +if not math.type then + + function math.type() + return "float" + end + +end + +if not math.tointeger then + + math.mininteger = -0x4FFFFFFFFFFF + math.maxinteger = 0x4FFFFFFFFFFF + + local floor = math.floor + + function math.tointeger(n) + local f = floor(n) + return f == n and f or nil + end + +end + +if not math.ult then + + local floor = math.floor + + function math.tointeger(m,n) + -- not ok but i'm not motivated to look into it now + return floor(m) < floor(n) -- unsigned comparison needed + end + +end diff --git a/macros/luatex/generic/lualibs/lualibs-md5.lua b/macros/luatex/generic/lualibs/lualibs-md5.lua new file mode 100644 index 0000000000..6758fa4444 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-md5.lua @@ -0,0 +1,110 @@ +if not modules then modules = { } end modules ['l-md5'] = { + version = 1.001, + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +-- This also provides file checksums and checkers. + +if not md5 then + md5 = optionalrequire("md5") +end + +if not md5 then + md5 = { + sum = function(str) print("error: md5 is not loaded (sum ignored)") return str end, + sumhexa = function(str) print("error: md5 is not loaded (sumhexa ignored)") return str end, + } +end + +local md5, file = md5, file +local gsub = string.gsub + +-- local gsub, format, byte = string.gsub, string.format, string.byte +-- +-- local function convert(str,fmt) +-- return (gsub(md5sum(str),".",function(chr) return format(fmt,byte(chr)) end)) +-- end +-- +-- if not md5.HEX then function md5.HEX(str) return convert(str,"%02X") end end +-- if not md5.hex then function md5.hex(str) return convert(str,"%02x") end end +-- if not md5.dec then function md5.dec(str) return convert(str,"%03i") end end + +do + + local patterns = lpeg and lpeg.patterns + + if patterns then + + local bytestoHEX = patterns.bytestoHEX + local bytestohex = patterns.bytestohex + local bytestodec = patterns.bytestodec + + local lpegmatch = lpeg.match + local md5sum = md5.sum + + if not md5.HEX then function md5.HEX(str) if str then return lpegmatch(bytestoHEX,md5sum(str)) end end end + if not md5.hex then function md5.hex(str) if str then return lpegmatch(bytestohex,md5sum(str)) end end end + if not md5.dec then function md5.dec(str) if str then return lpegmatch(bytestodec,md5sum(str)) end end end + + md5.sumhexa = md5.hex + md5.sumHEXA = md5.HEX + + end + +end + +function file.needsupdating(oldname,newname,threshold) -- size modification access change + local oldtime = lfs.attributes(oldname,"modification") + if oldtime then + local newtime = lfs.attributes(newname,"modification") + if not newtime then + return true -- no new file, so no updating needed + elseif newtime >= oldtime then + return false -- new file definitely needs updating + elseif oldtime - newtime < (threshold or 1) then + return false -- new file is probably still okay + else + return true -- new file has to be updated + end + else + return false -- no old file, so no updating needed + end +end + +file.needs_updating = file.needsupdating + +function file.syncmtimes(oldname,newname) + local oldtime = lfs.attributes(oldname,"modification") + if oldtime and lfs.isfile(newname) then + lfs.touch(newname,oldtime,oldtime) + end +end + +function file.checksum(name) + if md5 then + local data = io.loaddata(name) + if data then + return md5.HEX(data) + end + end + return nil +end + +function file.loadchecksum(name) + if md5 then + local data = io.loaddata(name .. ".md5") + return data and (gsub(data,"%s","")) + end + return nil +end + +function file.savechecksum(name,checksum) + if not checksum then checksum = file.checksum(name) end + if checksum then + io.savedata(name .. ".md5",checksum) + return checksum + end + return nil +end diff --git a/macros/luatex/generic/lualibs/lualibs-number.lua b/macros/luatex/generic/lualibs/lualibs-number.lua new file mode 100644 index 0000000000..9fd2f82f78 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-number.lua @@ -0,0 +1,241 @@ +if not modules then modules = { } end modules ['l-number'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +-- this module will be replaced when we have the bit library .. the number based sets +-- might go away + +local tostring, tonumber = tostring, tonumber +local format, floor, match, rep = string.format, math.floor, string.match, string.rep +local concat, insert = table.concat, table.insert +local lpegmatch = lpeg.match +local floor = math.floor + +number = number or { } +local number = number + +-- begin obsolete code -- + +-- if bit32 then +-- +-- local btest, bor = bit32.btest, bit32.bor +-- +-- function number.bit(p) +-- return 2 ^ (p - 1) -- 1-based indexing +-- end +-- +-- number.hasbit = btest +-- number.setbit = bor +-- +-- function number.setbit(x,p) -- why not bor? +-- return btest(x,p) and x or x + p +-- end +-- +-- function number.clearbit(x,p) +-- return btest(x,p) and x - p or x +-- end +-- +-- else +-- +-- -- http://ricilake.blogspot.com/2007/10/iterating-bits-in-lua.html +-- +-- function number.bit(p) +-- return 2 ^ (p - 1) -- 1-based indexing +-- end +-- +-- function number.hasbit(x, p) -- typical call: if hasbit(x, bit(3)) then ... +-- return x % (p + p) >= p +-- end +-- +-- function number.setbit(x, p) +-- return (x % (p + p) >= p) and x or x + p +-- end +-- +-- function number.clearbit(x, p) +-- return (x % (p + p) >= p) and x - p or x +-- end +-- +-- end + +-- end obsolete code -- + +-- print(number.tobitstring(8)) +-- print(number.tobitstring(14)) +-- print(number.tobitstring(66)) +-- print(number.tobitstring(0x00)) +-- print(number.tobitstring(0xFF)) +-- print(number.tobitstring(46260767936,4)) + +if bit32 then + + local bextract = bit32.extract + + local t = { + "0", "0", "0", "0", "0", "0", "0", "0", + "0", "0", "0", "0", "0", "0", "0", "0", + "0", "0", "0", "0", "0", "0", "0", "0", + "0", "0", "0", "0", "0", "0", "0", "0", + } + + function number.tobitstring(b,m,w) + if not w then + w = 32 + end + local n = w + for i=0,w-1 do + local v = bextract(b,i) + local k = w - i + if v == 1 then + n = k + t[k] = "1" + else + t[k] = "0" + end + end + if w then + return concat(t,"",1,w) + elseif m then + m = 33 - m * 8 + if m < 1 then + m = 1 + end + return concat(t,"",1,m) + elseif n < 8 then + return concat(t) + elseif n < 16 then + return concat(t,"",9) + elseif n < 24 then + return concat(t,"",17) + else + return concat(t,"",25) + end + end + +else + + function number.tobitstring(n,m) + if n > 0 then + local t = { } + while n > 0 do + insert(t,1,n % 2 > 0 and 1 or 0) + n = floor(n/2) + end + local nn = 8 - #t % 8 + if nn > 0 and nn < 8 then + for i=1,nn do + insert(t,1,0) + end + end + if m then + m = m * 8 - #t + if m > 0 then + insert(t,1,rep("0",m)) + end + end + return concat(t) + elseif m then + rep("00000000",m) + else + return "00000000" + end + end + +end + +function number.valid(str,default) + return tonumber(str) or default or nil +end + +function number.toevenhex(n) + local s = format("%X",n) + if #s % 2 == 0 then + return s + else + return "0" .. s + end +end + +-- -- a,b,c,d,e,f = number.toset(100101) +-- -- +-- -- function number.toset(n) +-- -- return match(tostring(n),"(.?)(.?)(.?)(.?)(.?)(.?)(.?)(.?)") +-- -- end +-- -- +-- -- -- the lpeg way is slower on 8 digits, but faster on 4 digits, some 7.5% +-- -- -- on +-- -- +-- -- for i=1,1000000 do +-- -- local a,b,c,d,e,f,g,h = number.toset(12345678) +-- -- local a,b,c,d = number.toset(1234) +-- -- local a,b,c = number.toset(123) +-- -- local a,b,c = number.toset("123") +-- -- end +-- +-- local one = lpeg.C(1-lpeg.S('')/tonumber)^1 +-- +-- function number.toset(n) +-- return lpegmatch(one,tostring(n)) +-- end +-- +-- -- function number.bits(n,zero) +-- -- local t, i = { }, (zero and 0) or 1 +-- -- while n > 0 do +-- -- local m = n % 2 +-- -- if m > 0 then +-- -- insert(t,1,i) +-- -- end +-- -- n = floor(n/2) +-- -- i = i + 1 +-- -- end +-- -- return t +-- -- end +-- -- +-- -- -- a bit faster +-- +-- local function bits(n,i,...) +-- if n > 0 then +-- local m = n % 2 +-- local n = floor(n/2) +-- if m > 0 then +-- return bits(n, i+1, i, ...) +-- else +-- return bits(n, i+1, ...) +-- end +-- else +-- return ... +-- end +-- end +-- +-- function number.bits(n) +-- return { bits(n,1) } +-- end + +function number.bytetodecimal(b) + local d = floor(b * 100 / 255 + 0.5) + if d > 100 then + return 100 + elseif d < -100 then + return -100 + else + return d + end +end + +function number.decimaltobyte(d) + local b = floor(d * 255 / 100 + 0.5) + if b > 255 then + return 255 + elseif b < -255 then + return -255 + else + return b + end +end + +function number.idiv(i,d) + return floor(i/d) -- i//d in 5.3 +end diff --git a/macros/luatex/generic/lualibs/lualibs-os.lua b/macros/luatex/generic/lualibs/lualibs-os.lua new file mode 100644 index 0000000000..aa04e254df --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-os.lua @@ -0,0 +1,655 @@ +if not modules then modules = { } end modules ['l-os'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +-- This file deals with some operating system issues. Please don't bother me +-- with the pros and cons of operating systems as they all have their flaws +-- and benefits. Bashing one of them won't help solving problems and fixing +-- bugs faster and is a waste of time and energy. +-- +-- path separators: / or \ ... we can use / everywhere +-- suffixes : dll so exe <none> ... no big deal +-- quotes : we can use "" in most cases +-- expansion : unless "" are used * might give side effects +-- piping/threads : somewhat different for each os +-- locations : specific user file locations and settings can change over time +-- +-- os.type : windows | unix (new, we already guessed os.platform) +-- os.name : windows | msdos | linux | macosx | solaris | .. | generic (new) +-- os.platform : extended os.name with architecture + +-- os.sleep() => socket.sleep() +-- math.randomseed(tonumber(string.sub(string.reverse(tostring(math.floor(socket.gettime()*10000))),1,6))) + +local os = os +local date, time = os.date, os.time +local find, format, gsub, upper, gmatch = string.find, string.format, string.gsub, string.upper, string.gmatch +local concat = table.concat +local random, ceil, randomseed = math.random, math.ceil, math.randomseed +local rawget, rawset, type, getmetatable, setmetatable, tonumber, tostring = rawget, rawset, type, getmetatable, setmetatable, tonumber, tostring + +-- This check needs to happen real early on. Todo: we can pick it up from the commandline +-- if we pass --binpath= (which is useful anyway) + +do + local selfdir = os.selfdir + if selfdir == "" then + selfdir = nil + end + if not selfdir then + -- We need a fallback plan so let's see what we get. + if arg then + -- passed by mtx-context ... saves network access + for i=1,#arg do + local a = arg[i] + if find(a,"^%-%-[c:]*texmfbinpath=") then + selfdir = gsub(a,"^.-=","") + break + end + end + end + if not selfdir then + selfdir = os.selfbin or "luatex" + if find(selfdir,"[/\\]") then + selfdir = gsub(selfdir,"[/\\][^/\\]*$","") + elseif os.getenv then + local path = os.getenv("PATH") + local name = gsub(selfdir,"^.*[/\\][^/\\]","") + local patt = "[^:]+" + if os.type == "windows" then + patt = "[^;]+" + name = name .. ".exe" + end + local isfile + if lfs then + -- we're okay as lfs is assumed present + local attributes = lfs.attributes + isfile = function(name) + local a = attributes(name,"mode") + return a == "file" or a == "link" or nil + end + else + -- we're not okay and much will not work as we miss lfs + local open = io.open + isfile = function(name) + local f = open(name) + if f then + f:close() + return true + end + end + end + for p in gmatch(path,patt) do + -- possible speedup: there must be tex in 'p' + if isfile(p .. "/" .. name) then + selfdir = p + break + end + end + end + end + -- let's hope we're okay now + os.selfdir = selfdir or "." + end +end +-- print(os.selfdir) os.exit() + +-- The following code permits traversing the environment table, at least in luatex. Internally all +-- environment names are uppercase. + +-- The randomseed in Lua is not that random, although this depends on the operating system as well +-- as the binary (Luatex is normally okay). But to be sure we set the seed anyway. It will be better +-- in Lua 5.4 (according to the announcements.) + +math.initialseed = tonumber(string.sub(string.reverse(tostring(ceil(socket and socket.gettime()*10000 or time()))),1,6)) + +randomseed(math.initialseed) + +if not os.__getenv__ then + + os.__getenv__ = os.getenv + os.__setenv__ = os.setenv + + if os.env then + + local osgetenv = os.getenv + local ossetenv = os.setenv + local osenv = os.env local _ = osenv.PATH -- initialize the table + + function os.setenv(k,v) + if v == nil then + v = "" + end + local K = upper(k) + osenv[K] = v + if type(v) == "table" then + v = concat(v,";") -- path + end + ossetenv(K,v) + end + + function os.getenv(k) + local K = upper(k) + local v = osenv[K] or osenv[k] or osgetenv(K) or osgetenv(k) + if v == "" then + return nil + else + return v + end + end + + else + + local ossetenv = os.setenv + local osgetenv = os.getenv + local osenv = { } + + function os.setenv(k,v) + if v == nil then + v = "" + end + local K = upper(k) + osenv[K] = v + end + + function os.getenv(k) + local K = upper(k) + local v = osenv[K] or osgetenv(K) or osgetenv(k) + if v == "" then + return nil + else + return v + end + end + + local function __index(t,k) + return os.getenv(k) + end + local function __newindex(t,k,v) + os.setenv(k,v) + end + + os.env = { } + + setmetatable(os.env, { __index = __index, __newindex = __newindex } ) + + end + +end + +-- end of environment hack + +local execute = os.execute +local iopopen = io.popen + +local function resultof(command) + local handle = iopopen(command,"r") -- already has flush + if handle then + local result = handle:read("*all") or "" + handle:close() + return result + else + return "" + end +end + +os.resultof = resultof + +function os.pipeto(command) + return iopopen(command,"w") -- already has flush +end + +if not io.fileseparator then + if find(os.getenv("PATH"),";",1,true) then + io.fileseparator, io.pathseparator, os.type = "\\", ";", os.type or "windows" + else + io.fileseparator, io.pathseparator, os.type = "/" , ":", os.type or "unix" + end +end + +os.type = os.type or (io.pathseparator == ";" and "windows") or "unix" +os.name = os.name or (os.type == "windows" and "mswin" ) or "linux" + +if os.type == "windows" then + os.libsuffix, os.binsuffix, os.binsuffixes = 'dll', 'exe', { 'exe', 'cmd', 'bat' } +else + os.libsuffix, os.binsuffix, os.binsuffixes = 'so', '', { '' } +end + +local launchers = { + windows = "start %s", + macosx = "open %s", + unix = "xdg-open %s &> /dev/null &", +} + +function os.launch(str) + local command = format(launchers[os.name] or launchers.unix,str) + -- todo: pcall +-- print(command) + execute(command) +end + +local gettimeofday = os.gettimeofday or os.clock +os.gettimeofday = gettimeofday + +local startuptime = gettimeofday() + +function os.runtime() + return gettimeofday() - startuptime +end + +-- print(os.gettimeofday()-os.time()) +-- os.sleep(1.234) +-- print (">>",os.runtime()) +-- print(os.date("%H:%M:%S",os.gettimeofday())) +-- print(os.date("%H:%M:%S",os.time())) + +-- no need for function anymore as we have more clever code and helpers now +-- this metatable trickery might as well disappear + +local resolvers = os.resolvers or { } +os.resolvers = resolvers + +setmetatable(os, { __index = function(t,k) + local r = resolvers[k] + return r and r(t,k) or nil -- no memoize +end }) + +-- we can use HOSTTYPE on some platforms + +local name, platform = os.name or "linux", os.getenv("MTX_PLATFORM") or "" + +-- local function guess() +-- local architecture = resultof("uname -m") or "" +-- if architecture ~= "" then +-- return architecture +-- end +-- architecture = os.getenv("HOSTTYPE") or "" +-- if architecture ~= "" then +-- return architecture +-- end +-- return resultof("echo $HOSTTYPE") or "" +-- end + +-- os.bits = 32 | 64 + +-- os.uname() +-- sysname +-- machine +-- release +-- version +-- nodename + +if platform ~= "" then + + os.platform = platform + +elseif os.type == "windows" then + + -- we could set the variable directly, no function needed here + + -- PROCESSOR_ARCHITECTURE : binary platform + -- PROCESSOR_ARCHITEW6432 : OS platform + + -- mswin-64 is now win64 + + function resolvers.platform(t,k) + local architecture = os.getenv("PROCESSOR_ARCHITECTURE") or "" + local platform = "" + if find(architecture,"AMD64",1,true) then + platform = "win64" + else + platform = "mswin" + end + os.setenv("MTX_PLATFORM",platform) + os.platform = platform + return platform + end + +elseif name == "linux" then + + function resolvers.platform(t,k) + -- we sometimes have HOSTTYPE set so let's check that first + local architecture = os.getenv("HOSTTYPE") or resultof("uname -m") or "" + local platform = os.getenv("MTX_PLATFORM") or "" + local musl = find(os.selfdir or "","linuxmusl") + if platform ~= "" then + -- we're done + elseif find(architecture,"x86_64",1,true) then + platform = musl and "linuxmusl" or "linux-64" + elseif find(architecture,"ppc",1,true) then + platform = "linux-ppc" + else + platform = musl and "linuxmusl" or "linux" + end + os.setenv("MTX_PLATFORM",platform) + os.platform = platform + return platform + end + +elseif name == "macosx" then + + --[[ + Identifying the architecture of OSX is quite a mess and this + is the best we can come up with. For some reason $HOSTTYPE is + a kind of pseudo environment variable, not known to the current + environment. And yes, uname cannot be trusted either, so there + is a change that you end up with a 32 bit run on a 64 bit system. + Also, some proper 64 bit intel macs are too cheap (low-end) and + therefore not permitted to run the 64 bit kernel. + ]]-- + + function resolvers.platform(t,k) + -- local platform = "" + -- local architecture = os.getenv("HOSTTYPE") or "" + -- if architecture == "" then + -- architecture = resultof("echo $HOSTTYPE") or "" + -- end + local architecture = resultof("echo $HOSTTYPE") or "" + local platform = "" + if architecture == "" then + -- print("\nI have no clue what kind of OSX you're running so let's assume an 32 bit intel.\n") + platform = "osx-intel" + elseif find(architecture,"i386",1,true) then + platform = "osx-intel" + elseif find(architecture,"x86_64",1,true) then + platform = "osx-64" + else + platform = "osx-ppc" + end + os.setenv("MTX_PLATFORM",platform) + os.platform = platform + return platform + end + +elseif name == "sunos" then + + function resolvers.platform(t,k) + local architecture = resultof("uname -m") or "" + local platform = "" + if find(architecture,"sparc",1,true) then + platform = "solaris-sparc" + else -- if architecture == 'i86pc' + platform = "solaris-intel" + end + os.setenv("MTX_PLATFORM",platform) + os.platform = platform + return platform + end + +elseif name == "freebsd" then + + function resolvers.platform(t,k) + local architecture = resultof("uname -m") or "" + local platform = "" + if find(architecture,"amd64",1,true) then + platform = "freebsd-amd64" + else + platform = "freebsd" + end + os.setenv("MTX_PLATFORM",platform) + os.platform = platform + return platform + end + +elseif name == "kfreebsd" then + + function resolvers.platform(t,k) + -- we sometimes have HOSTTYPE set so let's check that first + local architecture = os.getenv("HOSTTYPE") or resultof("uname -m") or "" + local platform = "" + if find(architecture,"x86_64",1,true) then + platform = "kfreebsd-amd64" + else + platform = "kfreebsd-i386" + end + os.setenv("MTX_PLATFORM",platform) + os.platform = platform + return platform + end + +else + + -- platform = "linux" + -- os.setenv("MTX_PLATFORM",platform) + -- os.platform = platform + + function resolvers.platform(t,k) + local platform = "linux" + os.setenv("MTX_PLATFORM",platform) + os.platform = platform + return platform + end + +end + +os.newline = name == "windows" and "\013\010" or "\010" -- crlf or lf + +function resolvers.bits(t,k) + local bits = find(os.platform,"64",1,true) and 64 or 32 + os.bits = bits + return bits +end + +-- beware, we set the randomseed + +-- from wikipedia: Version 4 UUIDs use a scheme relying only on random numbers. This algorithm sets the +-- version number as well as two reserved bits. All other bits are set using a random or pseudorandom +-- data source. Version 4 UUIDs have the form xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx with hexadecimal +-- digits x and hexadecimal digits 8, 9, A, or B for y. e.g. f47ac10b-58cc-4372-a567-0e02b2c3d479. +-- +-- as we don't call this function too often there is not so much risk on repetition + +local t = { 8, 9, "a", "b" } + +function os.uuid() + return format("%04x%04x-4%03x-%s%03x-%04x-%04x%04x%04x", + random(0xFFFF),random(0xFFFF), + random(0x0FFF), + t[ceil(random(4))] or 8,random(0x0FFF), + random(0xFFFF), + random(0xFFFF),random(0xFFFF),random(0xFFFF) + ) +end + +local d + +function os.timezone(delta) + d = d or tonumber(tonumber(date("%H")-date("!%H"))) + if delta then + if d > 0 then + return format("+%02i:00",d) + else + return format("-%02i:00",-d) + end + else + return 1 + end +end + +local timeformat = format("%%s%s",os.timezone(true)) +local dateformat = "!%Y-%m-%d %H:%M:%S" +local lasttime = nil +local lastdate = nil + +function os.fulltime(t,default) + t = t and tonumber(t) or 0 + if t > 0 then + -- valid time + elseif default then + return default + else + t = time() + end + if t ~= lasttime then + lasttime = t + lastdate = format(timeformat,date(dateformat)) + end + return lastdate +end + +local dateformat = "%Y-%m-%d %H:%M:%S" +local lasttime = nil +local lastdate = nil + +function os.localtime(t,default) + t = t and tonumber(t) or 0 + if t > 0 then + -- valid time + elseif default then + return default + else + t = time() + end + if t ~= lasttime then + lasttime = t + lastdate = date(dateformat,t) + end + return lastdate +end + +function os.converttime(t,default) + local t = tonumber(t) + if t and t > 0 then + return date(dateformat,t) + else + return default or "-" + end +end + +local memory = { } + +local function which(filename) + local fullname = memory[filename] + if fullname == nil then + local suffix = file.suffix(filename) + local suffixes = suffix == "" and os.binsuffixes or { suffix } + for directory in gmatch(os.getenv("PATH"),"[^" .. io.pathseparator .."]+") do + local df = file.join(directory,filename) + for i=1,#suffixes do + local dfs = file.addsuffix(df,suffixes[i]) + if io.exists(dfs) then + fullname = dfs + break + end + end + end + if not fullname then + fullname = false + end + memory[filename] = fullname + end + return fullname +end + +os.which = which +os.where = which + +function os.today() + return date("!*t") -- table with values +end + +function os.now() + return date("!%Y-%m-%d %H:%M:%S") -- 2011-12-04 14:59:12 +end + +-- if not os.sleep and socket then +-- os.sleep = socket.sleep +-- end + +if not os.sleep then + local socket = socket + function os.sleep(n) + if not socket then + -- so we delay ... if os.sleep is really needed then one should also + -- be sure that socket can be found + socket = require("socket") + end + socket.sleep(n) + end +end + +-- print(os.which("inkscape.exe")) +-- print(os.which("inkscape")) +-- print(os.which("gs.exe")) +-- print(os.which("ps2pdf")) + +-- These are moved from core-con.lua (as I needed them elsewhere). + +local function isleapyear(year) -- timed for bram's cs practicum + -- return (year % 400 == 0) or (year % 100 ~= 0 and year % 4 == 0) -- 3:4:1600:1900 = 9.9 : 8.2 : 5.0 : 6.8 (29.9) + return (year % 4 == 0) and (year % 100 ~= 0 or year % 400 == 0) -- 3:4:1600:1900 = 5.1 : 6.5 : 8.1 : 10.2 (29.9) + -- return (year % 4 == 0) and (year % 400 == 0 or year % 100 ~= 0) -- 3:4:1600:1900 = 5.2 : 8.5 : 6.8 : 10.1 (30.6) +end + +os.isleapyear = isleapyear + +-- nicer: +-- +-- local days = { +-- [false] = { 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 }, +-- [true] = { 31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 } +-- } +-- +-- local function nofdays(year,month) +-- return days[isleapyear(year)][month] +-- return month == 2 and isleapyear(year) and 29 or days[month] +-- end +-- +-- more efficient: + +local days = { 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 } + +local function nofdays(year,month) + if not month then + return isleapyear(year) and 365 or 364 + else + return month == 2 and isleapyear(year) and 29 or days[month] + end +end + +os.nofdays = nofdays + +function os.weekday(day,month,year) + return date("%w",time { year = year, month = month, day = day }) + 1 +end + +function os.validdate(year,month,day) + -- we assume that all three values are set + -- year is always ok, even if lua has a 1970 time limit + if month < 1 then + month = 1 + elseif month > 12 then + month = 12 + end + if day < 1 then + day = 1 + else + local max = nofdays(year,month) + if day > max then + day = max + end + end + return year, month, day +end + +local osexit = os.exit +local exitcode = nil + +function os.setexitcode(code) + exitcode = code +end + +function os.exit(c) + if exitcode ~= nil then + return osexit(exitcode) + end + if c ~= nil then + return osexit(c) + end + return osexit() +end diff --git a/macros/luatex/generic/lualibs/lualibs-package.lua b/macros/luatex/generic/lualibs/lualibs-package.lua new file mode 100644 index 0000000000..0dd71e5eca --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-package.lua @@ -0,0 +1,400 @@ +if not modules then modules = { } end modules ['l-package'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +-- Code moved from data-lua and changed into a plug-in. + +-- We overload the regular loader. We do so because we operate mostly in +-- tds and use our own loader code. Alternatively we could use a more +-- extensive definition of package.path and package.cpath but even then +-- we're not done. Also, we now have better tracing. +-- +-- -- local mylib = require("libtest") +-- -- local mysql = require("luasql.mysql") + +local type = type +local gsub, format, find = string.gsub, string.format, string.find +local insert, remove = table.insert, table.remove + +local P, S, Cs, lpegmatch = lpeg.P, lpeg.S, lpeg.Cs, lpeg.match + +local package = package +local searchers = package.searchers or package.loaders + +-------.loaders = nil -- old stuff that we don't want +-------.seeall = nil -- old stuff that we don't want + +-- dummies + +local filejoin = file and file.join or function(path,name) return path .. "/" .. name end +local isreadable = file and file.is_readable or function(name) local f = io.open(name) if f then f:close() return true end end +local addsuffix = file and file.addsuffix or function(name,suffix) return name .. "." .. suffix end + +-- local separator, concatinator, placeholder, pathofexecutable, ignorebefore = string.match(package.config,"(.-)\n(.-)\n(.-)\n(.-)\n(.-)\n") +-- +-- local config = { +-- separator = separator, -- \ or / +-- concatinator = concatinator, -- ; +-- placeholder = placeholder, -- ? becomes name +-- pathofexecutable = pathofexecutable, -- ! becomes executables dir (on windows) +-- ignorebefore = ignorebefore, -- - remove all before this when making lua_open +-- } + +local function cleanpath(path) -- hm, don't we have a helper for this? + return path +end + +local pattern = Cs((((1-S("\\/"))^0 * (S("\\/")^1/"/"))^0 * (P(".")^1/"/"+P(1))^1) * -1) + +local function lualibfile(name) + return lpegmatch(pattern,name) or name +end + +local offset = luarocks and 1 or 0 -- todo: also check other extras + +local helpers = package.helpers or { + cleanpath = cleanpath, + lualibfile = lualibfile, + trace = false, + report = function(...) print(format(...)) end, + builtin = { + ["preload table"] = searchers[1+offset], -- special case, built-in libs + ["path specification"] = searchers[2+offset], + ["cpath specification"] = searchers[3+offset], + ["all in one fallback"] = searchers[4+offset], -- special case, combined libs + }, + methods = { + }, + sequence = { + "already loaded", + "preload table", + "qualified path", -- beware, lua itself doesn't handle qualified paths (prepends ./) + "lua extra list", + "lib extra list", + "path specification", + "cpath specification", + "all in one fallback", + "not loaded", + } +} + +package.helpers = helpers + +local methods = helpers.methods +local builtin = helpers.builtin + +-- extra tds/ctx paths ... a bit of overhead for efficient tracing + +local extraluapaths = { } +local extralibpaths = { } +local luapaths = nil -- delayed +local libpaths = nil -- delayed +local oldluapath = nil +local oldlibpath = nil + +local nofextralua = -1 +local nofextralib = -1 +local nofpathlua = -1 +local nofpathlib = -1 + +local function listpaths(what,paths) + local nofpaths = #paths + if nofpaths > 0 then + for i=1,nofpaths do + helpers.report("using %s path %i: %s",what,i,paths[i]) + end + else + helpers.report("no %s paths defined",what) + end + return nofpaths +end + +local function getextraluapaths() + if helpers.trace and #extraluapaths ~= nofextralua then + nofextralua = listpaths("extra lua",extraluapaths) + end + return extraluapaths +end + +local function getextralibpaths() + if helpers.trace and #extralibpaths ~= nofextralib then + nofextralib = listpaths("extra lib",extralibpaths) + end + return extralibpaths +end + +local function getluapaths() + local luapath = package.path or "" + if oldluapath ~= luapath then + luapaths = file.splitpath(luapath,";") + oldluapath = luapath + nofpathlua = -1 + end + if helpers.trace and #luapaths ~= nofpathlua then + nofpathlua = listpaths("builtin lua",luapaths) + end + return luapaths +end + +local function getlibpaths() + local libpath = package.cpath or "" + if oldlibpath ~= libpath then + libpaths = file.splitpath(libpath,";") + oldlibpath = libpath + nofpathlib = -1 + end + if helpers.trace and #libpaths ~= nofpathlib then + nofpathlib = listpaths("builtin lib",libpaths) + end + return libpaths +end + +package.luapaths = getluapaths +package.libpaths = getlibpaths +package.extraluapaths = getextraluapaths +package.extralibpaths = getextralibpaths + +local hashes = { + lua = { }, + lib = { }, +} + +local function registerpath(tag,what,target,...) + local pathlist = { ... } + local cleanpath = helpers.cleanpath + local trace = helpers.trace + local report = helpers.report + local hash = hashes[what] + -- + local function add(path) + local path = cleanpath(path) + if not hash[path] then + target[#target+1] = path + hash[path] = true + if trace then + report("registered %s path %s: %s",tag,#target,path) + end + else + if trace then + report("duplicate %s path: %s",tag,path) + end + end + end + -- + for p=1,#pathlist do + local path = pathlist[p] + if type(path) == "table" then + for i=1,#path do + add(path[i]) + end + else + add(path) + end + end +end + +local function pushpath(tag,what,target,path) + local path = helpers.cleanpath(path) + insert(target,1,path) + if helpers.trace then + helpers.report("pushing %s path in front: %s",tag,path) + end +end + +local function poppath(tag,what,target) + local path = remove(target,1) + if helpers.trace then + if path then + helpers.report("popping %s path from front: %s",tag,path) + else + helpers.report("no %s path to pop",tag) + end + end +end + +helpers.registerpath = registerpath + +function package.extraluapath(...) + registerpath("extra lua","lua",extraluapaths,...) +end +function package.pushluapath(path) + pushpath("extra lua","lua",extraluapaths,path) +end +function package.popluapath() + poppath("extra lua","lua",extraluapaths) +end + +function package.extralibpath(...) + registerpath("extra lib","lib",extralibpaths,...) +end +function package.pushlibpath(path) + pushpath("extra lib","lib",extralibpaths,path) +end +function package.poplibpath() + poppath("extra lib","lua",extralibpaths) +end + +-- lib loader (used elsewhere) + +local function loadedaslib(resolved,rawname) -- todo: strip all before first - + local base = gsub(rawname,"%.","_") + -- so, we can do a require("foo/bar") and initialize bar + -- local base = gsub(file.basename(rawname),"%.","_") + local init = "luaopen_" .. gsub(base,"%.","_") + if helpers.trace then + helpers.report("calling loadlib with '%s' with init '%s'",resolved,init) + end + return package.loadlib(resolved,init) +end + +helpers.loadedaslib = loadedaslib + +-- wrapped and new loaders + +local function loadedbypath(name,rawname,paths,islib,what) + local trace = helpers.trace + for p=1,#paths do + local path = paths[p] + local resolved = filejoin(path,name) + if trace then + helpers.report("%s path, identifying '%s' on '%s'",what,name,path) + end + if isreadable(resolved) then + if trace then + helpers.report("%s path, '%s' found on '%s'",what,name,resolved) + end + if islib then + return loadedaslib(resolved,rawname) + else + return loadfile(resolved) + end + end + end +end + +helpers.loadedbypath = loadedbypath + +local function loadedbyname(name,rawname) + if find(name,"^/") or find(name,"^[a-zA-Z]:/") then + local trace=helpers.trace + if trace then + helpers.report("qualified name, identifying '%s'",what,name) + end + if isreadable(name) then + if trace then + helpers.report("qualified name, '%s' found",what,name) + end + return loadfile(name) + end + end +end + +helpers.loadedbyname = loadedbyname + +methods["already loaded"] = function(name) + return package.loaded[name] +end + +methods["preload table"] = function(name) + return builtin["preload table"](name) +end + +methods["qualified path"]=function(name) + return loadedbyname(addsuffix(lualibfile(name),"lua"),name) +end + +methods["lua extra list"] = function(name) + return loadedbypath(addsuffix(lualibfile(name),"lua"),name,getextraluapaths(),false,"lua") +end + +methods["lib extra list"] = function(name) + return loadedbypath(addsuffix(lualibfile(name),os.libsuffix),name,getextralibpaths(),true, "lib") +end + +methods["path specification"] = function(name) + getluapaths() -- triggers list building and tracing + return builtin["path specification"](name) +end + +methods["cpath specification"] = function(name) + getlibpaths() -- triggers list building and tracing + return builtin["cpath specification"](name) +end + +methods["all in one fallback"] = function(name) + return builtin["all in one fallback"](name) +end + +methods["not loaded"] = function(name) + if helpers.trace then + helpers.report("unable to locate '%s'",name or "?") + end + return nil +end + +local level = 0 +local used = { } + +helpers.traceused = false + +function helpers.loaded(name) + local sequence = helpers.sequence + level = level + 1 + for i=1,#sequence do + local method = sequence[i] + if helpers.trace then + helpers.report("%s, level '%s', method '%s', name '%s'","locating",level,method,name) + end + local result, rest = methods[method](name) + if type(result) == "function" then + if helpers.trace then + helpers.report("%s, level '%s', method '%s', name '%s'","found",level,method,name) + end + if helpers.traceused then + used[#used+1] = { level = level, name = name } + end + level = level - 1 + return result, rest + end + end + -- safeguard, we never come here + level = level - 1 + return nil +end + +function helpers.showused() + local n = #used + if n > 0 then + helpers.report("%s libraries loaded:",n) + helpers.report() + for i=1,n do + local u = used[i] + helpers.report("%i %a",u.level,u.name) + end + helpers.report() + end +end + +function helpers.unload(name) + if helpers.trace then + if package.loaded[name] then + helpers.report("unloading, name '%s', %s",name,"done") + else + helpers.report("unloading, name '%s', %s",name,"not loaded") + end + end + package.loaded[name] = nil +end + +-- overloading require does not work out well so we need to push it in +-- front .. + +table.insert(searchers,1,helpers.loaded) + +if context then + package.path = "" +end diff --git a/macros/luatex/generic/lualibs/lualibs-set.lua b/macros/luatex/generic/lualibs/lualibs-set.lua new file mode 100644 index 0000000000..2370f0139f --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-set.lua @@ -0,0 +1,87 @@ +if not modules then modules = { } end modules ['l-set'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +-- This will become obsolete when we have the bitset library embedded. + +set = set or { } + +local nums = { } +local tabs = { } +local concat = table.concat +local next, type = next, type + +set.create = table.tohash + +function set.tonumber(t) + if next(t) then + local s = "" + -- we could save mem by sorting, but it slows down + for k, v in next, t do + if v then + -- why bother about the leading space + s = s .. " " .. k + end + end + local n = nums[s] + if not n then + n = #tabs + 1 + tabs[n] = t + nums[s] = n + end + return n + else + return 0 + end +end + +function set.totable(n) + if n == 0 then + return { } + else + return tabs[n] or { } + end +end + +function set.tolist(n) + if n == 0 or not tabs[n] then + return "" + else + local t, n = { }, 0 + for k, v in next, tabs[n] do + if v then + n = n + 1 + t[n] = k + end + end + return concat(t," ") + end +end + +function set.contains(n,s) + if type(n) == "table" then + return n[s] + elseif n == 0 then + return false + else + local t = tabs[n] + return t and t[s] + end +end + +--~ local c = set.create{'aap','noot','mies'} +--~ local s = set.tonumber(c) +--~ local t = set.totable(s) +--~ print(t['aap']) +--~ local c = set.create{'zus','wim','jet'} +--~ local s = set.tonumber(c) +--~ local t = set.totable(s) +--~ print(t['aap']) +--~ print(t['jet']) +--~ print(set.contains(t,'jet')) +--~ print(set.contains(t,'aap')) + diff --git a/macros/luatex/generic/lualibs/lualibs-string.lua b/macros/luatex/generic/lualibs/lualibs-string.lua new file mode 100644 index 0000000000..1dee85e289 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-string.lua @@ -0,0 +1,244 @@ +if not modules then modules = { } end modules ['l-string'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +local string = string +local sub, gmatch, format, char, byte, rep, lower = string.sub, string.gmatch, string.format, string.char, string.byte, string.rep, string.lower +local lpegmatch, patterns = lpeg.match, lpeg.patterns +local P, S, C, Ct, Cc, Cs = lpeg.P, lpeg.S, lpeg.C, lpeg.Ct, lpeg.Cc, lpeg.Cs + +-- Some functions are already defined in l-lpeg and maybe some from here will +-- move there (unless we also expose caches). + +-- if not string.split then +-- +-- function string.split(str,pattern) +-- local t = { } +-- if str ~= "" then +-- local n = 1 +-- for s in gmatch(str..pattern,"(.-)"..pattern) do +-- t[n] = s +-- n = n + 1 +-- end +-- end +-- return t +-- end +-- +-- end + +-- function string.unquoted(str) +-- return (gsub(str,"^([\"\'])(.*)%1$","%2")) -- interesting pattern +-- end + +local unquoted = patterns.squote * C(patterns.nosquote) * patterns.squote + + patterns.dquote * C(patterns.nodquote) * patterns.dquote + +function string.unquoted(str) + return lpegmatch(unquoted,str) or str +end + +-- print(string.unquoted("test")) +-- print(string.unquoted([["t\"est"]])) +-- print(string.unquoted([["t\"est"x]])) +-- print(string.unquoted("\'test\'")) +-- print(string.unquoted('"test"')) +-- print(string.unquoted('"test"')) + +function string.quoted(str) + return format("%q",str) -- always double quote +end + +function string.count(str,pattern) -- variant 3 + local n = 0 + for _ in gmatch(str,pattern) do -- not for utf + n = n + 1 + end + return n +end + +function string.limit(str,n,sentinel) -- not utf proof + if #str > n then + sentinel = sentinel or "..." + return sub(str,1,(n-#sentinel)) .. sentinel + else + return str + end +end + +local stripper = patterns.stripper +local fullstripper = patterns.fullstripper +local collapser = patterns.collapser +local nospacer = patterns.nospacer +local longtostring = patterns.longtostring + +function string.strip(str) + return str and lpegmatch(stripper,str) or "" +end + +function string.fullstrip(str) + return str and lpegmatch(fullstripper,str) or "" +end + +function string.collapsespaces(str) + return str and lpegmatch(collapser,str) or "" +end + +function string.nospaces(str) + return str and lpegmatch(nospacer,str) or "" +end + +function string.longtostring(str) + return str and lpegmatch(longtostring,str) or "" +end + +-- function string.is_empty(str) +-- return not find(str,"%S") +-- end + +local pattern = P(" ")^0 * P(-1) -- maybe also newlines + +-- patterns.onlyspaces = pattern + +function string.is_empty(str) + if not str or str == "" then + return true + else + return lpegmatch(pattern,str) and true or false + end +end + +-- if not string.escapedpattern then +-- +-- local patterns_escapes = { +-- ["%"] = "%%", +-- ["."] = "%.", +-- ["+"] = "%+", ["-"] = "%-", ["*"] = "%*", +-- ["["] = "%[", ["]"] = "%]", +-- ["("] = "%(", [")"] = "%)", +-- -- ["{"] = "%{", ["}"] = "%}" +-- -- ["^"] = "%^", ["$"] = "%$", +-- } +-- +-- local simple_escapes = { +-- ["-"] = "%-", +-- ["."] = "%.", +-- ["?"] = ".", +-- ["*"] = ".*", +-- } +-- +-- function string.escapedpattern(str,simple) +-- return (gsub(str,".",simple and simple_escapes or patterns_escapes)) +-- end +-- +-- function string.topattern(str,lowercase,strict) +-- if str == "" then +-- return ".*" +-- else +-- str = gsub(str,".",simple_escapes) +-- if lowercase then +-- str = lower(str) +-- end +-- if strict then +-- return "^" .. str .. "$" +-- else +-- return str +-- end +-- end +-- end +-- +-- end + +--- needs checking + +local anything = patterns.anything +local moreescapes = Cc("%") * S(".-+%?()[]*$^{}") +local allescapes = Cc("%") * S(".-+%?()[]*") -- also {} and ^$ ? +local someescapes = Cc("%") * S(".-+%()[]") -- also {} and ^$ ? +local matchescapes = Cc(".") * S("*?") -- wildcard and single match + +local pattern_m = Cs ( ( moreescapes + anything )^0 ) +local pattern_a = Cs ( ( allescapes + anything )^0 ) +local pattern_b = Cs ( ( someescapes + matchescapes + anything )^0 ) +local pattern_c = Cs ( Cc("^") * ( someescapes + matchescapes + anything )^0 * Cc("$") ) + +function string.escapedpattern(str,simple) + return lpegmatch(simple and pattern_b or pattern_a,str) +end + +function string.topattern(str,lowercase,strict) + if str == "" or type(str) ~= "string" then + return ".*" + elseif strict == "all" then + str = lpegmatch(pattern_m,str) + elseif strict then + str = lpegmatch(pattern_c,str) + else + str = lpegmatch(pattern_b,str) + end + if lowercase then + return lower(str) + else + return str + end +end + +-- print(string.escapedpattern("abc*234",true)) +-- print(string.escapedpattern("12+34*.tex",false)) +-- print(string.escapedpattern("12+34*.tex",true)) +-- print(string.topattern ("12+34*.tex",false,false)) +-- print(string.topattern ("12+34*.tex",false,true)) + +function string.valid(str,default) + return (type(str) == "string" and str ~= "" and str) or default or nil +end + +-- handy fallback + +string.itself = function(s) return s end + +-- also handy (see utf variant) + +local pattern_c = Ct( C(1) ^0) -- string and not utf ! +local pattern_b = Ct((C(1)/byte)^0) + +function string.totable(str,bytes) + return lpegmatch(bytes and pattern_b or pattern_c,str) +end + +-- handy from within tex: + +local replacer = lpeg.replacer("@","%%") -- Watch the escaped % in lpeg! + +function string.tformat(fmt,...) + return format(lpegmatch(replacer,fmt),...) +end + +-- obsolete names: + +string.quote = string.quoted +string.unquote = string.unquoted + +-- new + +if not string.bytetable then -- used in font-cff.lua + + local limit = 5000 -- we can go to 8000 in luajit and much higher in lua if needed + + function string.bytetable(str) -- from a string + local n = #str + if n > limit then + local t = { byte(str,1,limit) } + for i=limit+1,n do + t[i] = byte(str,i) + end + return t + else + return { byte(str,1,n) } + end + end + +end diff --git a/macros/luatex/generic/lualibs/lualibs-table.lua b/macros/luatex/generic/lualibs/lualibs-table.lua new file mode 100644 index 0000000000..cffdcc23e2 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-table.lua @@ -0,0 +1,1471 @@ +if not modules then modules = { } end modules ['l-table'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +local type, next, tostring, tonumber, select = type, next, tostring, tonumber, select +local table, string = table, string +local concat, sort = table.concat, table.sort +local format, lower, dump = string.format, string.lower, string.dump +local getmetatable, setmetatable = getmetatable, setmetatable +local lpegmatch, patterns = lpeg.match, lpeg.patterns +local floor = math.floor + +-- extra functions, some might go (when not used) +-- +-- we could serialize using %a but that won't work well is in the code we mostly use +-- floats and as such we get unequality e.g. in version comparisons + +local stripper = patterns.stripper + +function table.getn(t) + return t and #t -- for very old times sake +end + +function table.strip(tab) + local lst = { } + local l = 0 + for i=1,#tab do + local s = lpegmatch(stripper,tab[i]) or "" + if s == "" then + -- skip this one + else + l = l + 1 + lst[l] = s + end + end + return lst +end + +function table.keys(t) + if t then + local keys = { } + local k = 0 + for key in next, t do + k = k + 1 + keys[k] = key + end + return keys + else + return { } + end +end + +-- local function compare(a,b) +-- local ta = type(a) -- needed, else 11 < 2 +-- local tb = type(b) -- needed, else 11 < 2 +-- if ta == tb and ta == "number" then +-- return a < b +-- else +-- return tostring(a) < tostring(b) -- not that efficient +-- end +-- end + +-- local function compare(a,b) +-- local ta = type(a) -- needed, else 11 < 2 +-- local tb = type(b) -- needed, else 11 < 2 +-- if ta == tb and (ta == "number" or ta == "string") then +-- return a < b +-- else +-- return tostring(a) < tostring(b) -- not that efficient +-- end +-- end + +-- local function sortedkeys(tab) +-- if tab then +-- local srt, category, s = { }, 0, 0 -- 0=unknown 1=string, 2=number 3=mixed +-- for key in next, tab do +-- s = s + 1 +-- srt[s] = key +-- if category == 3 then +-- -- no further check +-- else +-- local tkey = type(key) +-- if tkey == "string" then +-- category = (category == 2 and 3) or 1 +-- elseif tkey == "number" then +-- category = (category == 1 and 3) or 2 +-- else +-- category = 3 +-- end +-- end +-- end +-- if category == 0 or category == 3 then +-- sort(srt,compare) +-- else +-- sort(srt) +-- end +-- return srt +-- else +-- return { } +-- end +-- end + +-- local function compare(a,b) +-- local ta = type(a) -- needed, else 11 < 2 +-- local tb = type(b) -- needed, else 11 < 2 +-- if ta == tb and (ta == "number" or ta == "string") then +-- return a < b +-- else +-- return tostring(a) < tostring(b) -- not that efficient +-- end +-- end + +-- local function compare(a,b) +-- local ta = type(a) -- needed, else 11 < 2 +-- if ta == "number" or ta == "string" then +-- local tb = type(b) -- needed, else 11 < 2 +-- if ta == tb then +-- return a < b +-- end +-- end +-- return tostring(a) < tostring(b) -- not that efficient +-- end + +local function compare(a,b) + local ta = type(a) -- needed, else 11 < 2 + if ta == "number" then + local tb = type(b) -- needed, else 11 < 2 + if ta == tb then + return a < b + elseif tb == "string" then + return tostring(a) < b + end + elseif ta == "string" then + local tb = type(b) -- needed, else 11 < 2 + if ta == tb then + return a < b + else + return a < tostring(b) + end + end + return tostring(a) < tostring(b) -- not that efficient +end + +local function sortedkeys(tab) + if tab then + local srt = { } + local category = 0 -- 0=unknown 1=string, 2=number 3=mixed + local s = 0 + for key in next, tab do + s = s + 1 + srt[s] = key + if category ~= 3 then + local tkey = type(key) + if category == 1 then + if tkey ~= "string" then + category = 3 + end + elseif category == 2 then + if tkey ~= "number" then + category = 3 + end + else + if tkey == "string" then + category = 1 + elseif tkey == "number" then + category = 2 + else + category = 3 + end + end + end + end + if s < 2 then + -- nothing to sort + elseif category == 3 then + sort(srt,compare) + else + sort(srt) + end + return srt + else + return { } + end +end + +local function sortedhashonly(tab) + if tab then + local srt = { } + local s = 0 + for key in next, tab do + if type(key) == "string" then + s = s + 1 + srt[s] = key + end + end + if s > 1 then + sort(srt) + end + return srt + else + return { } + end +end + +local function sortedindexonly(tab) + if tab then + local srt = { } + local s = 0 + for key in next, tab do + if type(key) == "number" then + s = s + 1 + srt[s] = key + end + end + if s > 1 then + sort(srt) + end + return srt + else + return { } + end +end + +local function sortedhashkeys(tab,cmp) -- fast one + if tab then + local srt = { } + local s = 0 + for key in next, tab do + if key then + s= s + 1 + srt[s] = key + end + end + if s > 1 then + sort(srt,cmp) + end + return srt + else + return { } + end +end + +function table.allkeys(t) + local keys = { } + for k, v in next, t do + for k in next, v do + keys[k] = true + end + end + return sortedkeys(keys) +end + +table.sortedkeys = sortedkeys +table.sortedhashonly = sortedhashonly +table.sortedindexonly = sortedindexonly +table.sortedhashkeys = sortedhashkeys + +local function nothing() end + +local function sortedhash(t,cmp) + if t then + local s + if cmp then + -- it would be nice if the sort function would accept a third argument (or nicer, an optional first) + s = sortedhashkeys(t,function(a,b) return cmp(t,a,b) end) + else + s = sortedkeys(t) -- the robust one + end + local m = #s + if m == 1 then + return next, t + elseif m > 0 then + local n = 0 + return function() + if n < m then + n = n + 1 + local k = s[n] + return k, t[k] + end + end + end + end + return nothing +end + +table.sortedhash = sortedhash +table.sortedpairs = sortedhash -- obsolete + +function table.append(t,list) + local n = #t + for i=1,#list do + n = n + 1 + t[n] = list[i] + end + return t +end + +function table.prepend(t, list) + local nl = #list + local nt = nl + #t + for i=#t,1,-1 do + t[nt] = t[i] + nt = nt - 1 + end + for i=1,#list do + t[i] = list[i] + end + return t +end + +-- function table.merge(t, ...) -- first one is target +-- t = t or { } +-- local lst = { ... } +-- for i=1,#lst do +-- for k, v in next, lst[i] do +-- t[k] = v +-- end +-- end +-- return t +-- end + +function table.merge(t, ...) -- first one is target + if not t then + t = { } + end + for i=1,select("#",...) do + for k, v in next, (select(i,...)) do + t[k] = v + end + end + return t +end + +-- function table.merged(...) +-- local tmp, lst = { }, { ... } +-- for i=1,#lst do +-- for k, v in next, lst[i] do +-- tmp[k] = v +-- end +-- end +-- return tmp +-- end + +function table.merged(...) + local t = { } + for i=1,select("#",...) do + for k, v in next, (select(i,...)) do + t[k] = v + end + end + return t +end + +-- function table.imerge(t, ...) +-- local lst, nt = { ... }, #t +-- for i=1,#lst do +-- local nst = lst[i] +-- for j=1,#nst do +-- nt = nt + 1 +-- t[nt] = nst[j] +-- end +-- end +-- return t +-- end + +function table.imerge(t, ...) + local nt = #t + for i=1,select("#",...) do + local nst = select(i,...) + for j=1,#nst do + nt = nt + 1 + t[nt] = nst[j] + end + end + return t +end + +-- function table.imerged(...) +-- local tmp, ntmp, lst = { }, 0, {...} +-- for i=1,#lst do +-- local nst = lst[i] +-- for j=1,#nst do +-- ntmp = ntmp + 1 +-- tmp[ntmp] = nst[j] +-- end +-- end +-- return tmp +-- end + +function table.imerged(...) + local tmp = { } + local ntmp = 0 + for i=1,select("#",...) do + local nst = select(i,...) + for j=1,#nst do + ntmp = ntmp + 1 + tmp[ntmp] = nst[j] + end + end + return tmp +end + +local function fastcopy(old,metatabletoo) -- fast one + if old then + local new = { } + for k, v in next, old do + if type(v) == "table" then + new[k] = fastcopy(v,metatabletoo) -- was just table.copy + else + new[k] = v + end + end + if metatabletoo then + -- optional second arg + local mt = getmetatable(old) + if mt then + setmetatable(new,mt) + end + end + return new + else + return { } + end +end + +-- todo : copy without metatable + +local function copy(t,tables) -- taken from lua wiki, slightly adapted + if not tables then + tables = { } + end + local tcopy = { } + if not tables[t] then + tables[t] = tcopy + end + for i,v in next, t do -- brrr, what happens with sparse indexed + if type(i) == "table" then + if tables[i] then + i = tables[i] + else + i = copy(i,tables) + end + end + if type(v) ~= "table" then + tcopy[i] = v + elseif tables[v] then + tcopy[i] = tables[v] + else + tcopy[i] = copy(v,tables) + end + end + local mt = getmetatable(t) + if mt then + setmetatable(tcopy,mt) + end + return tcopy +end + +table.fastcopy = fastcopy +table.copy = copy + +function table.derive(parent) -- for the moment not public + local child = { } + if parent then + setmetatable(child,{ __index = parent }) + end + return child +end + +function table.tohash(t,value) + local h = { } + if t then + if value == nil then value = true end + for _, v in next, t do + h[v] = value + end + end + return h +end + +function table.fromhash(t) + local hsh = { } + local h = 0 + for k, v in next, t do + if v then + h = h + 1 + hsh[h] = k + end + end + return hsh +end + +local noquotes, hexify, handle, compact, inline, functions, metacheck + +local reserved = table.tohash { -- intercept a language inconvenience: no reserved words as key + 'and', 'break', 'do', 'else', 'elseif', 'end', 'false', 'for', 'function', 'if', + 'in', 'local', 'nil', 'not', 'or', 'repeat', 'return', 'then', 'true', 'until', 'while', + 'NaN', 'goto', +} + +-- local function is_simple_table(t) +-- if #t > 0 then +-- local n = 0 +-- for _,v in next, t do +-- n = n + 1 +-- end +-- if n == #t then +-- local tt, nt = { }, 0 +-- for i=1,#t do +-- local v = t[i] +-- local tv = type(v) +-- if tv == "number" then +-- nt = nt + 1 +-- if hexify then +-- tt[nt] = format("0x%X",v) +-- else +-- tt[nt] = tostring(v) -- tostring not needed +-- end +-- elseif tv == "string" then +-- nt = nt + 1 +-- tt[nt] = format("%q",v) +-- elseif tv == "boolean" then +-- nt = nt + 1 +-- tt[nt] = v and "true" or "false" +-- else +-- return nil +-- end +-- end +-- return tt +-- end +-- end +-- return nil +-- end + +-- local function is_simple_table(t) +-- local nt = #t +-- if nt > 0 then +-- local n = 0 +-- for _,v in next, t do +-- n = n + 1 +-- -- if type(v) == "table" then +-- -- return nil +-- -- end +-- end +-- if n == nt then +-- local tt = { } +-- for i=1,nt do +-- local v = t[i] +-- local tv = type(v) +-- if tv == "number" then +-- if hexify then +-- tt[i] = format("0x%X",v) +-- else +-- tt[i] = tostring(v) -- tostring not needed +-- end +-- elseif tv == "string" then +-- tt[i] = format("%q",v) +-- elseif tv == "boolean" then +-- tt[i] = v and "true" or "false" +-- else +-- return nil +-- end +-- end +-- return tt +-- end +-- end +-- return nil +-- end + +local function is_simple_table(t,hexify) -- also used in util-tab so maybe public + local nt = #t + if nt > 0 then + local n = 0 + for _, v in next, t do + n = n + 1 + if type(v) == "table" then + return nil + end + end + -- local haszero = t[0] + local haszero = rawget(t,0) -- don't trigger meta + if n == nt then + local tt = { } + for i=1,nt do + local v = t[i] + local tv = type(v) + if tv == "number" then + -- tt[i] = v -- not needed tostring(v) + if hexify then + tt[i] = format("0x%X",v) + else + tt[i] = v -- not needed tostring(v) + end + elseif tv == "string" then + tt[i] = format("%q",v) -- f_string(v) + elseif tv == "boolean" then + tt[i] = v and "true" or "false" + else + return nil + end + end + return tt + elseif haszero and (n == nt + 1) then + local tt = { } + for i=0,nt do + local v = t[i] + local tv = type(v) + if tv == "number" then + -- tt[i+1] = v -- not needed tostring(v) + if hexify then + tt[i+1] = format("0x%X",v) + else + tt[i+1] = v -- not needed tostring(v) + end + elseif tv == "string" then + tt[i+1] = format("%q",v) -- f_string(v) + elseif tv == "boolean" then + tt[i+1] = v and "true" or "false" + else + return nil + end + end + tt[1] = "[0] = " .. tt[1] + return tt + end + end + return nil +end + +table.is_simple_table = is_simple_table + +-- Because this is a core function of mkiv I moved some function calls +-- inline. +-- +-- twice as fast in a test: +-- +-- local propername = lpeg.P(lpeg.R("AZ","az","__") * lpeg.R("09","AZ","az", "__")^0 * lpeg.P(-1) ) + +-- problem: there no good number_to_string converter with the best resolution + +-- probably using .. is faster than format +-- maybe split in a few cases (yes/no hexify) + +-- todo: %g faster on numbers than %s + +-- we can speed this up with repeaters and formatters but we haven't defined them +-- yet + +local propername = patterns.propername -- was find(name,"^%a[%w%_]*$") + +local function dummy() end + +local function do_serialize(root,name,depth,level,indexed) + if level > 0 then + depth = depth .. " " + if indexed then + handle(format("%s{",depth)) + else + local tn = type(name) + if tn == "number" then + if hexify then + handle(format("%s[0x%X]={",depth,name)) + else + handle(format("%s[%s]={",depth,name)) + end + elseif tn == "string" then + if noquotes and not reserved[name] and lpegmatch(propername,name) then + handle(format("%s%s={",depth,name)) + else + handle(format("%s[%q]={",depth,name)) + end + elseif tn == "boolean" then + handle(format("%s[%s]={",depth,name and "true" or "false")) + else + handle(format("%s{",depth)) + end + end + end + -- we could check for k (index) being number (cardinal) + if root and next(root) ~= nil then + local first = nil + local last = 0 + if compact then + last = #root + for k=1,last do + -- if root[k] == nil then + if rawget(root,k) == nil then + last = k - 1 + break + end + end + if last > 0 then + first = 1 + end + end + local sk = sortedkeys(root) + for i=1,#sk do + local k = sk[i] + local v = root[k] + local tv = type(v) + local tk = type(k) + if compact and first and tk == "number" and k >= first and k <= last then + if tv == "number" then + if hexify then + handle(format("%s 0x%X,",depth,v)) + else + handle(format("%s %s,",depth,v)) -- %.99g + end + elseif tv == "string" then + handle(format("%s %q,",depth,v)) + elseif tv == "table" then + if next(v) == nil then + handle(format("%s {},",depth)) + elseif inline then -- and #t > 0 + local st = is_simple_table(v,hexify) + if st then + handle(format("%s { %s },",depth,concat(st,", "))) + else + do_serialize(v,k,depth,level+1,true) + end + else + do_serialize(v,k,depth,level+1,true) + end + elseif tv == "boolean" then + handle(format("%s %s,",depth,v and "true" or "false")) + elseif tv == "function" then + if functions then + handle(format('%s load(%q),',depth,dump(v))) -- maybe strip + else + handle(format('%s "function",',depth)) + end + else + handle(format("%s %q,",depth,tostring(v))) + end + elseif k == "__p__" then -- parent + if false then + handle(format("%s __p__=nil,",depth)) + end + elseif tv == "number" then + if tk == "number" then + if hexify then + handle(format("%s [0x%X]=0x%X,",depth,k,v)) + else + handle(format("%s [%s]=%s,",depth,k,v)) -- %.99g + end + elseif tk == "boolean" then + if hexify then + handle(format("%s [%s]=0x%X,",depth,k and "true" or "false",v)) + else + handle(format("%s [%s]=%s,",depth,k and "true" or "false",v)) -- %.99g + end + elseif tk ~= "string" then + -- ignore + elseif noquotes and not reserved[k] and lpegmatch(propername,k) then + if hexify then + handle(format("%s %s=0x%X,",depth,k,v)) + else + handle(format("%s %s=%s,",depth,k,v)) -- %.99g + end + else + if hexify then + handle(format("%s [%q]=0x%X,",depth,k,v)) + else + handle(format("%s [%q]=%s,",depth,k,v)) -- %.99g + end + end + elseif tv == "string" then + if tk == "number" then + if hexify then + handle(format("%s [0x%X]=%q,",depth,k,v)) + else + handle(format("%s [%s]=%q,",depth,k,v)) + end + elseif tk == "boolean" then + handle(format("%s [%s]=%q,",depth,k and "true" or "false",v)) + elseif tk ~= "string" then + -- ignore + elseif noquotes and not reserved[k] and lpegmatch(propername,k) then + handle(format("%s %s=%q,",depth,k,v)) + else + handle(format("%s [%q]=%q,",depth,k,v)) + end + elseif tv == "table" then + if next(v) == nil then + if tk == "number" then + if hexify then + handle(format("%s [0x%X]={},",depth,k)) + else + handle(format("%s [%s]={},",depth,k)) + end + elseif tk == "boolean" then + handle(format("%s [%s]={},",depth,k and "true" or "false")) + elseif tk ~= "string" then + -- ignore + elseif noquotes and not reserved[k] and lpegmatch(propername,k) then + handle(format("%s %s={},",depth,k)) + else + handle(format("%s [%q]={},",depth,k)) + end + elseif inline then + local st = is_simple_table(v,hexify) + if st then + if tk == "number" then + if hexify then + handle(format("%s [0x%X]={ %s },",depth,k,concat(st,", "))) + else + handle(format("%s [%s]={ %s },",depth,k,concat(st,", "))) + end + elseif tk == "boolean" then + handle(format("%s [%s]={ %s },",depth,k and "true" or "false",concat(st,", "))) + elseif tk ~= "string" then + -- ignore + elseif noquotes and not reserved[k] and lpegmatch(propername,k) then + handle(format("%s %s={ %s },",depth,k,concat(st,", "))) + else + handle(format("%s [%q]={ %s },",depth,k,concat(st,", "))) + end + else + do_serialize(v,k,depth,level+1) + end + else + do_serialize(v,k,depth,level+1) + end + elseif tv == "boolean" then + if tk == "number" then + if hexify then + handle(format("%s [0x%X]=%s,",depth,k,v and "true" or "false")) + else + handle(format("%s [%s]=%s,",depth,k,v and "true" or "false")) + end + elseif tk == "boolean" then + handle(format("%s [%s]=%s,",depth,tostring(k),v and "true" or "false")) + elseif tk ~= "string" then + -- ignore + elseif noquotes and not reserved[k] and lpegmatch(propername,k) then + handle(format("%s %s=%s,",depth,k,v and "true" or "false")) + else + handle(format("%s [%q]=%s,",depth,k,v and "true" or "false")) + end + elseif tv == "function" then + if functions then + local getinfo = debug and debug.getinfo + if getinfo then + local f = getinfo(v).what == "C" and dump(dummy) or dump(v) -- maybe strip + -- local f = getinfo(v).what == "C" and dump(function(...) return v(...) end) or dump(v) -- maybe strip + if tk == "number" then + if hexify then + handle(format("%s [0x%X]=load(%q),",depth,k,f)) + else + handle(format("%s [%s]=load(%q),",depth,k,f)) + end + elseif tk == "boolean" then + handle(format("%s [%s]=load(%q),",depth,k and "true" or "false",f)) + elseif tk ~= "string" then + -- ignore + elseif noquotes and not reserved[k] and lpegmatch(propername,k) then + handle(format("%s %s=load(%q),",depth,k,f)) + else + handle(format("%s [%q]=load(%q),",depth,k,f)) + end + end + end + else + if tk == "number" then + if hexify then + handle(format("%s [0x%X]=%q,",depth,k,tostring(v))) + else + handle(format("%s [%s]=%q,",depth,k,tostring(v))) + end + elseif tk == "boolean" then + handle(format("%s [%s]=%q,",depth,k and "true" or "false",tostring(v))) + elseif tk ~= "string" then + -- ignore + elseif noquotes and not reserved[k] and lpegmatch(propername,k) then + handle(format("%s %s=%q,",depth,k,tostring(v))) + else + handle(format("%s [%q]=%q,",depth,k,tostring(v))) + end + end + end + end + if level > 0 then + handle(format("%s},",depth)) + end +end + +-- replacing handle by a direct t[#t+1] = ... (plus test) is not much +-- faster (0.03 on 1.00 for zapfino.tma) + +local function serialize(_handle,root,name,specification) -- handle wins + local tname = type(name) + if type(specification) == "table" then + noquotes = specification.noquotes + hexify = specification.hexify + handle = _handle or specification.handle or print + functions = specification.functions + compact = specification.compact + inline = specification.inline and compact + metacheck = specification.metacheck + if functions == nil then + functions = true + end + if compact == nil then + compact = true + end + if inline == nil then + inline = compact + end + if metacheck == nil then + metacheck = true + end + else + noquotes = false + hexify = false + handle = _handle or print + compact = true + inline = true + functions = true + metacheck = true + end + if tname == "string" then + if name == "return" then + handle("return {") + else + handle(name .. "={") + end + elseif tname == "number" then + if hexify then + handle(format("[0x%X]={",name)) + else + handle("[" .. name .. "]={") + end + elseif tname == "boolean" then + if name then + handle("return {") + else + handle("{") + end + else + handle("t={") + end + if root then + -- The dummy access will initialize a table that has a delayed initialization + -- using a metatable. (maybe explicitly test for metatable). This can crash on + -- metatables that check the index against a number. + if metacheck and getmetatable(root) then + local dummy = root._w_h_a_t_e_v_e_r_ + root._w_h_a_t_e_v_e_r_ = nil + end + -- Let's forget about empty tables. + if next(root) ~= nil then + do_serialize(root,name,"",0) + end + end + handle("}") +end + +-- A version with formatters is some 20% faster than using format (because formatters are +-- much faster) but of course, inlining the format using .. is then again faster .. anyway, +-- as we do some pretty printing as well there is not that much to gain unless we make a +-- 'fast' ugly variant as well. But, we would have to move the formatter to l-string then. + +-- name: +-- +-- true : return { } +-- false : { } +-- nil : t = { } +-- string : string = { } +-- "return" : return { } +-- number : [number] = { } + +function table.serialize(root,name,specification) + local t = { } + local n = 0 + local function flush(s) + n = n + 1 + t[n] = s + end + serialize(flush,root,name,specification) + return concat(t,"\n") +end + +-- local a = { e = { 1,2,3,4,5,6}, a = 1, b = 2, c = "ccc", d = { a = 1, b = 2, c = "ccc", d = { a = 1, b = 2, c = "ccc" } } } +-- local t = os.clock() +-- for i=1,10000 do +-- table.serialize(a) +-- end +-- print(os.clock()-t,table.serialize(a)) + +table.tohandle = serialize + +local maxtab = 2*1024 + +function table.tofile(filename,root,name,specification) + local f = io.open(filename,'w') + if f then + if maxtab > 1 then + local t = { } + local n = 0 + local function flush(s) + n = n + 1 + t[n] = s + if n > maxtab then + f:write(concat(t,"\n"),"\n") -- hm, write(sometable) should be nice + t = { } -- we could recycle t if needed + n = 0 + end + end + serialize(flush,root,name,specification) + f:write(concat(t,"\n"),"\n") + else + local function flush(s) + f:write(s,"\n") + end + serialize(flush,root,name,specification) + end + f:close() + io.flush() + end +end + +local function flattened(t,f,depth) -- also handles { nil, 1, nil, 2 } + if f == nil then + f = { } + depth = 0xFFFF + elseif tonumber(f) then + -- assume that only two arguments are given + depth = f + f = { } + elseif not depth then + depth = 0xFFFF + end + for k, v in next, t do + if type(k) ~= "number" then + if depth > 0 and type(v) == "table" then + flattened(v,f,depth-1) + else + f[#f+1] = v + end + end + end + for k=1,#t do + local v = t[k] + if depth > 0 and type(v) == "table" then + flattened(v,f,depth-1) + else + f[#f+1] = v + end + end + return f +end + +table.flattened = flattened + +local function collapsed(t,f,h) + if f == nil then + f = { } + h = { } + end + for k=1,#t do + local v = t[k] + if type(v) == "table" then + collapsed(v,f,h) + elseif not h[v] then + f[#f+1] = v + h[v] = true + end + end + return f +end + +local function collapsedhash(t,h) + if h == nil then + h = { } + end + for k=1,#t do + local v = t[k] + if type(v) == "table" then + collapsedhash(v,h) + else + h[v] = true + end + end + return h +end + +table.collapsed = collapsed -- 20% faster than unique(collapsed(t)) +table.collapsedhash = collapsedhash + +local function unnest(t,f) -- only used in mk, for old times sake + if not f then -- and only relevant for token lists + f = { } -- this one can become obsolete + end + for i=1,#t do + local v = t[i] + if type(v) == "table" then + if type(v[1]) == "table" then + unnest(v,f) + else + f[#f+1] = v + end + else + f[#f+1] = v + end + end + return f +end + +function table.unnest(t) -- bad name + return unnest(t) +end + +local function are_equal(a,b,n,m) -- indexed + if a == b then + return true + elseif a and b and #a == #b then + if not n then + n = 1 + end + if not m then + m = #a + end + for i=n,m do + local ai, bi = a[i], b[i] + if ai==bi then + -- same + elseif type(ai) == "table" and type(bi) == "table" then + if not are_equal(ai,bi) then + return false + end + else + return false + end + end + return true + else + return false + end +end + +local function identical(a,b) -- assumes same structure + if a ~= b then + for ka, va in next, a do + local vb = b[ka] + if va == vb then + -- same + elseif type(va) == "table" and type(vb) == "table" then + if not identical(va,vb) then + return false + end + else + return false + end + end + end + return true +end + +table.identical = identical +table.are_equal = are_equal + +local function sparse(old,nest,keeptables) + local new = { } + for k, v in next, old do + if not (v == "" or v == false) then + if nest and type(v) == "table" then + v = sparse(v,nest) + if keeptables or next(v) ~= nil then + new[k] = v + end + else + new[k] = v + end + end + end + return new +end + +table.sparse = sparse + +function table.compact(t) + return sparse(t,true,true) +end + +function table.contains(t, v) + if t then + for i=1, #t do + if t[i] == v then + return i + end + end + end + return false +end + +function table.count(t) + local n = 0 + for k, v in next, t do + n = n + 1 + end + return n +end + +function table.swapped(t,s) -- hash, we need to make sure we don't mess up next + local n = { } + if s then + for k, v in next, s do + n[k] = v + end + end + for k, v in next, t do + n[v] = k + end + return n +end + +function table.hashed(t) -- list, add hash to index (save because we are not yet mixed + for i=1,#t do + t[t[i]] = i + end + return t +end + +function table.mirrored(t) -- hash, we need to make sure we don't mess up next + local n = { } + for k, v in next, t do + n[v] = k + n[k] = v + end + return n +end + +function table.reversed(t) + if t then + local tt = { } + local tn = #t + if tn > 0 then + local ttn = 0 + for i=tn,1,-1 do + ttn = ttn + 1 + tt[ttn] = t[i] + end + end + return tt + end +end + +function table.reverse(t) -- check with 5.3 ? + if t then + local n = #t + local m = n + 1 + for i=1,floor(n/2) do -- maybe just n//2 + local j = m - i + t[i], t[j] = t[j], t[i] + end + return t + end +end + +-- This one is for really simple cases where need a hash from a table. + +local function sequenced(t,sep,simple) + if not t then + return "" + elseif type(t) ~= "table" then + return t -- handy fallback + end + local n = #t + local s = { } + if n > 0 then + -- indexed + for i=1,n do + local v = t[i] + if type(v) == "table" then + s[i] = "{" .. sequenced(v,sep,simple) .. "}" + else + s[i] = tostring(t[i]) + end + end + else + -- hashed + n = 0 + for k, v in sortedhash(t) do + if simple then + if v == true then + n = n + 1 + s[n] = k + elseif v and v~= "" then + n = n + 1 + if type(v) == "table" then + s[n] = k .. "={" .. sequenced(v,sep,simple) .. "}" + else + s[n] = k .. "=" .. tostring(v) + end + end + else + n = n + 1 + if type(v) == "table" then + s[n] = k .. "={" .. sequenced(v,sep,simple) .. "}" + else + s[n] = k .. "=" .. tostring(v) + end + end + end + end + if sep == true then + return "{ " .. concat(s,", ") .. " }" + else + return concat(s,sep or " | ") + end +end + +table.sequenced = sequenced + +function table.print(t,...) + if type(t) ~= "table" then + print(tostring(t)) + else + serialize(print,t,...) + end +end + +if setinspector then + setinspector("table",function(v) if type(v) == "table" then serialize(print,v,"table") return true end end) +end + +-- -- -- obsolete but we keep them for a while and might comment them later -- -- -- + +-- roughly: copy-loop : unpack : sub == 0.9 : 0.4 : 0.45 (so in critical apps, use unpack) + +function table.sub(t,i,j) + return { unpack(t,i,j) } +end + +-- slower than #t on indexed tables (#t only returns the size of the numerically indexed slice) + +function table.is_empty(t) + return not t or next(t) == nil +end + +function table.has_one_entry(t) + return t and next(t,next(t)) == nil +end + +-- new (rather basic, not indexed and nested) + +function table.loweredkeys(t) -- maybe utf + local l = { } + for k, v in next, t do + l[lower(k)] = v + end + return l +end + +-- new, might move (maybe duplicate) + +function table.unique(old) + local hash = { } + local new = { } + local n = 0 + for i=1,#old do + local oi = old[i] + if not hash[oi] then + n = n + 1 + new[n] = oi + hash[oi] = true + end + end + return new +end + +function table.sorted(t,...) + sort(t,...) + return t -- still sorts in-place +end + +-- + +function table.values(t,s) -- optional sort flag + if t then + local values = { } + local keys = { } + local v = 0 + for key, value in next, t do + if not keys[value] then + v = v + 1 + values[v] = value + keys[k] = key + end + end + if s then + sort(values) + end + return values + else + return { } + end +end + +-- maybe this will move to util-tab.lua + +-- for k, v in table.filtered(t,pattern) do ... end +-- for k, v in table.filtered(t,pattern,true) do ... end +-- for k, v in table.filtered(t,pattern,true,cmp) do ... end + +function table.filtered(t,pattern,sort,cmp) + if t and type(pattern) == "string" then + if sort then + local s + if cmp then + -- it would be nice if the sort function would accept a third argument (or nicer, an optional first) + s = sortedhashkeys(t,function(a,b) return cmp(t,a,b) end) + else + s = sortedkeys(t) -- the robust one + end + local n = 0 + local m = #s + local function kv(s) + while n < m do + n = n + 1 + local k = s[n] + if find(k,pattern) then + return k, t[k] + end + end + end + return kv, s + else + local n = next(t) + local function iterator() + while n ~= nil do + local k = n + n = next(t,k) + if find(k,pattern) then + return k, t[k] + end + end + end + return iterator, t + end + else + return nothing + end +end + +-- lua 5.3: + +if not table.move then + + function table.move(a1,f,e,t,a2) + if a2 and a1 ~= a2 then + for i=f,e do + a2[t] = a1[i] + t = t + 1 + end + return a2 + else + t = t + e - f + for i=e,f,-1 do + a1[t] = a1[i] + t = t - 1 + end + return a1 + end + end + +end diff --git a/macros/luatex/generic/lualibs/lualibs-trac-inf.lua b/macros/luatex/generic/lualibs/lualibs-trac-inf.lua new file mode 100644 index 0000000000..24ebc52499 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-trac-inf.lua @@ -0,0 +1,313 @@ +if not modules then modules = { } end modules ['trac-inf'] = { + version = 1.001, + comment = "companion to trac-inf.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +-- As we want to protect the global tables, we no longer store the timing +-- in the tables themselves but in a hidden timers table so that we don't +-- get warnings about assignments. This is more efficient than using rawset +-- and rawget. + +local type, tonumber, select = type, tonumber, select +local format, lower, find = string.format, string.lower, string.find +local concat = table.concat +local clock = os.gettimeofday or os.clock -- should go in environment + +local setmetatableindex = table.setmetatableindex +local serialize = table.serialize +local formatters = string.formatters + +statistics = statistics or { } +local statistics = statistics + +statistics.enable = true +statistics.threshold = 0.01 + +local statusinfo, n, registered, timers = { }, 0, { }, { } + +setmetatableindex(timers,function(t,k) + local v = { timing = 0, loadtime = 0, offset = 0 } + t[k] = v + return v +end) + +local function hastiming(instance) + return instance and timers[instance] +end + +local function resettiming(instance) + timers[instance or "notimer"] = { timing = 0, loadtime = 0, offset = 0 } +end + +local ticks = clock +local seconds = function(n) return n or 0 end + +-- if FFISUPPORTED and ffi and os.type == "windows" then +-- +-- local okay, kernel = pcall(ffi.load,"kernel32") +-- +-- if kernel then +-- +-- local tonumber = ffi.number or tonumber +-- +-- ffi.cdef[[ +-- int QueryPerformanceFrequency(int64_t *lpFrequency); +-- int QueryPerformanceCounter(int64_t *lpPerformanceCount); +-- ]] +-- +-- local target = ffi.new("__int64[1]") +-- +-- ticks = function() +-- if kernel.QueryPerformanceCounter(target) == 1 then +-- return tonumber(target[0]) +-- else +-- return 0 +-- end +-- end +-- +-- local target = ffi.new("__int64[1]") +-- +-- seconds = function(ticks) +-- if kernel.QueryPerformanceFrequency(target) == 1 then +-- return ticks / tonumber(target[0]) +-- else +-- return 0 +-- end +-- end +-- +-- end +-- +-- end + +local function starttiming(instance,reset) + local timer = timers[instance or "notimer"] + local it = timer.timing + if reset then + it = 0 + timer.loadtime = 0 + end + if it == 0 then + timer.starttime = ticks() + if not timer.loadtime then + timer.loadtime = 0 + end + end + timer.timing = it + 1 +end + +local function stoptiming(instance) + local timer = timers[instance or "notimer"] + local it = timer.timing + if it > 1 then + timer.timing = it - 1 + else + local starttime = timer.starttime + if starttime and starttime > 0 then + local stoptime = ticks() + local loadtime = stoptime - starttime + timer.stoptime = stoptime + timer.loadtime = timer.loadtime + loadtime + timer.timing = 0 + timer.starttime = 0 + return loadtime + end + end + return 0 +end + +local function benchmarktimer(instance) + local timer = timers[instance or "notimer"] + local it = timer.timing + if it > 1 then + timer.timing = it - 1 + else + local starttime = timer.starttime + if starttime and starttime > 0 then + timer.offset = ticks() - starttime + else + timer.offset = 0 + end + end +end + +local function elapsed(instance) + if type(instance) == "number" then + return instance + else + local timer = timers[instance or "notimer"] + return timer and seconds(timer.loadtime - 2*(timer.offset or 0)) or 0 + end +end + +local function currenttime(instance) + if type(instance) == "number" then + return instance + else + local timer = timers[instance or "notimer"] + local it = timer.timing + if it > 1 then + -- whatever + else + local starttime = timer.starttime + if starttime and starttime > 0 then + return seconds(timer.loadtime + ticks() - starttime - 2*(timer.offset or 0)) + end + end + return 0 + end +end + +local function elapsedtime(instance) + return format("%0.3f",elapsed(instance)) +end + +local function elapsedindeed(instance) + return elapsed(instance) > statistics.threshold +end + +local function elapsedseconds(instance,rest) -- returns nil if 0 seconds + if elapsedindeed(instance) then + return format("%0.3f seconds %s", elapsed(instance),rest or "") + end +end + +statistics.hastiming = hastiming +statistics.resettiming = resettiming +statistics.starttiming = starttiming +statistics.stoptiming = stoptiming +statistics.currenttime = currenttime +statistics.elapsed = elapsed +statistics.elapsedtime = elapsedtime +statistics.elapsedindeed = elapsedindeed +statistics.elapsedseconds = elapsedseconds +statistics.benchmarktimer = benchmarktimer + +-- general function .. we might split this module + +function statistics.register(tag,fnc) + if statistics.enable and type(fnc) == "function" then + local rt = registered[tag] or (#statusinfo + 1) + statusinfo[rt] = { tag, fnc } + registered[tag] = rt + if #tag > n then n = #tag end + end +end + +local report = logs.reporter("mkiv lua stats") + +function statistics.show() + if statistics.enable then + -- this code will move + local register = statistics.register + register("used platform", function() + return format("%s, type: %s, binary subtree: %s", + os.platform or "unknown",os.type or "unknown", environment.texos or "unknown") + end) + -- register("luatex banner", function() + -- return lower(status.banner) + -- end) + if LUATEXENGINE == "luametatex" then + register("used engine", function() + return format("%s version %s, functionality level %s, format id %s", + LUATEXENGINE, LUATEXVERSION, LUATEXFUNCTIONALITY, LUATEXFORMATID) + end) + else + register("used engine", function() + return format("%s version %s with functionality level %s, banner: %s", + LUATEXENGINE, LUATEXVERSION, LUATEXFUNCTIONALITY, lower(status.banner)) + end) + end + register("control sequences", function() + return format("%s of %s + %s", status.cs_count, status.hash_size,status.hash_extra) + end) + register("callbacks", statistics.callbacks) + if TEXENGINE == "luajittex" and JITSUPPORTED then + local jitstatus = jit.status + if jitstatus then + local jitstatus = { jitstatus() } + if jitstatus[1] then + register("luajit options", concat(jitstatus," ",2)) + end + end + end + -- so far + -- collectgarbage("collect") + register("lua properties",function() + local hashchar = tonumber(status.luatex_hashchars) + local mask = lua.mask or "ascii" + return format("engine: %s %s, used memory: %s, hash chars: min(%i,40), symbol mask: %s (%s)", + jit and "luajit" or "lua", + LUAVERSION, + statistics.memused(), + hashchar and 2^hashchar or "unknown", + mask, + mask == "utf" and "τεχ" or "tex") + end) + register("runtime",statistics.runtime) + logs.newline() -- initial newline + for i=1,#statusinfo do + local s = statusinfo[i] + local r = s[2]() + if r then + report("%s: %s",s[1],r) + end + end + -- logs.newline() -- final newline + statistics.enable = false + end +end + +function statistics.memused() -- no math.round yet -) + local round = math.round or math.floor + return format("%s MB, ctx: %s MB, max: %s MB)", + round(collectgarbage("count")/1000), + round(status.luastate_bytes/1000000), + status.luastate_bytes_max and round(status.luastate_bytes_max/1000000) or "unknown" + ) +end + +starttiming(statistics) + +function statistics.formatruntime(runtime) -- indirect so it can be overloaded and + return format("%s seconds", runtime) -- indeed that happens in cure-uti.lua +end + +function statistics.runtime() + stoptiming(statistics) + -- stoptiming(statistics) -- somehow we can start the timer twice, but where + local runtime = lua.getruntime and lua.getruntime() or elapsedtime(statistics) + return statistics.formatruntime(runtime) +end + +local report = logs.reporter("system") + +function statistics.timed(action,all) + starttiming("run") + action() + stoptiming("run") + local runtime = tonumber(elapsedtime("run")) + if all then + local alltime = tonumber(lua.getruntime and lua.getruntime() or elapsedtime(statistics)) + if alltime and alltime > 0 then + report("total runtime: %0.3f seconds of %0.3f seconds",runtime,alltime) + return + end + end + report("total runtime: %0.3f seconds",runtime) +end + +-- goodie + +function statistics.tracefunction(base,tag,...) + for i=1,select("#",...) do + local name = select(i,...) + local stat = { } + local func = base[name] + setmetatableindex(stat,function(t,k) t[k] = 0 return 0 end) + base[name] = function(n,k,v) stat[k] = stat[k] + 1 return func(n,k,v) end + statistics.register(formatters["%s.%s"](tag,name),function() return serialize(stat,"calls") end) + end +end diff --git a/macros/luatex/generic/lualibs/lualibs-unicode.lua b/macros/luatex/generic/lualibs/lualibs-unicode.lua new file mode 100644 index 0000000000..73d1eb7660 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-unicode.lua @@ -0,0 +1,1405 @@ +if not modules then modules = { } end modules ['l-unicode'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +-- floor(b/256) => rshift(b, 8) +-- floor(b/1024) => rshift(b,10) + +-- in lua 5.3: + +-- utf8.char(···) : concatinated +-- utf8.charpatt : "[\0-\x7F\xC2-\xF4][\x80-\xBF]*" +-- utf8.codes(s) : for p, c in utf8.codes(s) do body end +-- utf8.codepoint(s [, i [, j]]) +-- utf8.len(s [, i]) +-- utf8.offset(s, n [, i]) + +-- todo: utf.sub replacement (used in syst-aux) +-- we put these in the utf namespace: + +-- used : byte char len lower sub upper +-- not used : dump find format gmatch gfind gsub match rep reverse + +-- utf = utf or (unicode and unicode.utf8) or { } + +-- not supported: +-- +-- dump, find, format, gfind, gmatch, gsub, lower, match, rep, reverse, upper + +utf = utf or { } +-- unicode = nil + +if not string.utfcharacters then + + -- New: this gmatch hack is taken from the Lua 5.2 book. It's about two times slower + -- than the built-in string.utfcharacters. + + local gmatch = string.gmatch + + function string.characters(str) + return gmatch(str,".[\128-\191]*") + end + + +end + +utf.characters = string.utfcharacters + +-- string.utfvalues +-- string.utfcharacters +-- string.characters +-- string.characterpairs +-- string.bytes +-- string.bytepairs +-- string.utflength +-- string.utfvalues +-- string.utfcharacters + +local type = type +local char, byte, format, sub, gmatch = string.char, string.byte, string.format, string.sub, string.gmatch +local concat = table.concat +local P, C, R, Cs, Ct, Cmt, Cc, Carg, Cp = lpeg.P, lpeg.C, lpeg.R, lpeg.Cs, lpeg.Ct, lpeg.Cmt, lpeg.Cc, lpeg.Carg, lpeg.Cp + +local lpegmatch = lpeg.match +local patterns = lpeg.patterns +local tabletopattern = lpeg.utfchartabletopattern + +local bytepairs = string.bytepairs + +local finder = lpeg.finder +local replacer = lpeg.replacer + +local p_utftype = patterns.utftype +local p_utfstricttype = patterns.utfstricttype +local p_utfoffset = patterns.utfoffset +local p_utf8character = patterns.utf8character +local p_utf8char = patterns.utf8char +local p_utf8byte = patterns.utf8byte +local p_utfbom = patterns.utfbom +local p_newline = patterns.newline +local p_whitespace = patterns.whitespace + +-- if not unicode then +-- unicode = { utf = utf } -- for a while +-- end + +if not utf.char then + + utf.char = string.utfcharacter or (utf8 and utf8.char) + + if not utf.char then + + -- no multiples + + local char = string.char + + if bit32 then + + local rshift = bit32.rshift + + function utf.char(n) + if n < 0x80 then + -- 0aaaaaaa : 0x80 + return char(n) + elseif n < 0x800 then + -- 110bbbaa : 0xC0 : n >> 6 + -- 10aaaaaa : 0x80 : n & 0x3F + return char( + 0xC0 + rshift(n,6), + 0x80 + (n % 0x40) + ) + elseif n < 0x10000 then + -- 1110bbbb : 0xE0 : n >> 12 + -- 10bbbbaa : 0x80 : (n >> 6) & 0x3F + -- 10aaaaaa : 0x80 : n & 0x3F + return char( + 0xE0 + rshift(n,12), + 0x80 + (rshift(n,6) % 0x40), + 0x80 + (n % 0x40) + ) + elseif n < 0x200000 then + -- 11110ccc : 0xF0 : n >> 18 + -- 10ccbbbb : 0x80 : (n >> 12) & 0x3F + -- 10bbbbaa : 0x80 : (n >> 6) & 0x3F + -- 10aaaaaa : 0x80 : n & 0x3F + -- dddd : ccccc - 1 + return char( + 0xF0 + rshift(n,18), + 0x80 + (rshift(n,12) % 0x40), + 0x80 + (rshift(n,6) % 0x40), + 0x80 + (n % 0x40) + ) + else + return "" + end + end + + else + + local floor = math.floor + + function utf.char(n) + if n < 0x80 then + return char(n) + elseif n < 0x800 then + return char( + 0xC0 + floor(n/0x40), + 0x80 + (n % 0x40) + ) + elseif n < 0x10000 then + return char( + 0xE0 + floor(n/0x1000), + 0x80 + (floor(n/0x40) % 0x40), + 0x80 + (n % 0x40) + ) + elseif n < 0x200000 then + return char( + 0xF0 + floor(n/0x40000), + 0x80 + (floor(n/0x1000) % 0x40), + 0x80 + (floor(n/0x40) % 0x40), + 0x80 + (n % 0x40) + ) + else + return "" + end + end + + end + + end + +end + +if not utf.byte then + + utf.byte = string.utfvalue or (utf8 and utf8.codepoint) + + if not utf.byte then + + function utf.byte(c) + return lpegmatch(p_utf8byte,c) + end + + end + +end + +local utfchar, utfbyte = utf.char, utf.byte + +-- As we want to get rid of the (unmaintained) utf library we implement our own +-- variants (in due time an independent module): + +function utf.filetype(data) + return data and lpegmatch(p_utftype,data) or "unknown" +end + +local toentities = Cs ( + ( + patterns.utf8one + + ( + patterns.utf8two + + patterns.utf8three + + patterns.utf8four + ) / function(s) local b = utfbyte(s) if b < 127 then return s else return format("&#%X;",b) end end + )^0 +) + +patterns.toentities = toentities + +function utf.toentities(str) + return lpegmatch(toentities,str) +end + +-- local utfchr = { } -- 60K -> 2.638 M extra mem but currently not called that often (on latin) +-- +-- setmetatable(utfchr, { __index = function(t,k) local v = utfchar(k) t[k] = v return v end } ) +-- +-- collectgarbage("collect") +-- local u = collectgarbage("count")*1024 +-- local t = os.clock() +-- for i=1,1000 do +-- for i=1,600 do +-- local a = utfchr[i] +-- end +-- end +-- print(os.clock()-t,collectgarbage("count")*1024-u) + +-- collectgarbage("collect") +-- local t = os.clock() +-- for i=1,1000 do +-- for i=1,600 do +-- local a = utfchar(i) +-- end +-- end +-- print(os.clock()-t,collectgarbage("count")*1024-u) + +-- local byte = string.byte +-- local utfchar = utf.char + +local one = P(1) +local two = C(1) * C(1) +local four = C(R(utfchar(0xD8),utfchar(0xFF))) * C(1) * C(1) * C(1) + +local pattern = P("\254\255") * Cs( ( + four / function(a,b,c,d) + local ab = 0xFF * byte(a) + byte(b) + local cd = 0xFF * byte(c) + byte(d) + return utfchar((ab-0xD800)*0x400 + (cd-0xDC00) + 0x10000) + end + + two / function(a,b) + return utfchar(byte(a)*256 + byte(b)) + end + + one + )^1 ) + + P("\255\254") * Cs( ( + four / function(b,a,d,c) + local ab = 0xFF * byte(a) + byte(b) + local cd = 0xFF * byte(c) + byte(d) + return utfchar((ab-0xD800)*0x400 + (cd-0xDC00) + 0x10000) + end + + two / function(b,a) + return utfchar(byte(a)*256 + byte(b)) + end + + one + )^1 ) + +function string.toutf(s) -- in string namespace + return lpegmatch(pattern,s) or s -- todo: utf32 +end + +local validatedutf = Cs ( + ( + patterns.utf8one + + patterns.utf8two + + patterns.utf8three + + patterns.utf8four + + P(1) / "�" + )^0 +) + +patterns.validatedutf = validatedutf + +function utf.is_valid(str) + return type(str) == "string" and lpegmatch(validatedutf,str) or false +end + +if not utf.len then + + utf.len = string.utflength or (utf8 and utf8.len) + + if not utf.len then + + -- -- alternative 1: 0.77 + -- + -- local utfcharcounter = utfbom^-1 * Cs((p_utf8character/'!')^0) + -- + -- function utf.len(str) + -- return #lpegmatch(utfcharcounter,str or "") + -- end + -- + -- -- alternative 2: 1.70 + -- + -- local n = 0 + -- + -- local utfcharcounter = utfbom^-1 * (p_utf8character/function() n = n + 1 end)^0 -- slow + -- + -- function utf.length(str) + -- n = 0 + -- lpegmatch(utfcharcounter,str or "") + -- return n + -- end + -- + -- -- alternative 3: 0.24 (native unicode.utf8.len: 0.047) + + -- local n = 0 + -- + -- -- local utfcharcounter = lpeg.patterns.utfbom^-1 * P ( ( Cp() * ( + -- -- patterns.utf8one ^1 * Cc(1) + -- -- + patterns.utf8two ^1 * Cc(2) + -- -- + patterns.utf8three^1 * Cc(3) + -- -- + patterns.utf8four ^1 * Cc(4) ) * Cp() / function(f,d,t) n = n + (t - f)/d end + -- -- )^0 ) -- just as many captures as below + -- + -- -- local utfcharcounter = lpeg.patterns.utfbom^-1 * P ( ( + -- -- (Cmt(patterns.utf8one ^1,function(_,_,s) n = n + #s return true end)) + -- -- + (Cmt(patterns.utf8two ^1,function(_,_,s) n = n + #s/2 return true end)) + -- -- + (Cmt(patterns.utf8three^1,function(_,_,s) n = n + #s/3 return true end)) + -- -- + (Cmt(patterns.utf8four ^1,function(_,_,s) n = n + #s/4 return true end)) + -- -- )^0 ) -- not interesting as it creates strings but sometimes faster + -- + -- -- The best so far: + -- + -- local utfcharcounter = utfbom^-1 * P ( ( + -- Cp() * (patterns.utf8one )^1 * Cp() / function(f,t) n = n + t - f end + -- + Cp() * (patterns.utf8two )^1 * Cp() / function(f,t) n = n + (t - f)/2 end + -- + Cp() * (patterns.utf8three)^1 * Cp() / function(f,t) n = n + (t - f)/3 end + -- + Cp() * (patterns.utf8four )^1 * Cp() / function(f,t) n = n + (t - f)/4 end + -- )^0 ) + + -- function utf.len(str) + -- n = 0 + -- lpegmatch(utfcharcounter,str or "") + -- return n + -- end + + local n, f = 0, 1 + + local utfcharcounter = patterns.utfbom^-1 * Cmt ( + Cc(1) * patterns.utf8one ^1 + + Cc(2) * patterns.utf8two ^1 + + Cc(3) * patterns.utf8three^1 + + Cc(4) * patterns.utf8four ^1, + function(_,t,d) -- due to Cc no string captures, so faster + n = n + (t - f)/d + f = t + return true + end + )^0 + + function utf.len(str) + n, f = 0, 1 + lpegmatch(utfcharcounter,str or "") + return n + end + + -- -- these are quite a bit slower: + + -- utfcharcounter = utfbom^-1 * (Cmt(P(1) * R("\128\191")^0, function() n = n + 1 return true end))^0 -- 50+ times slower + -- utfcharcounter = utfbom^-1 * (Cmt(P(1), function() n = n + 1 return true end) * R("\128\191")^0)^0 -- 50- times slower + + end + +end + +utf.length = utf.len + +if not utf.sub then + + -- inefficient as lpeg just copies ^n + + -- local function sub(str,start,stop) + -- local pattern = p_utf8character^-(start-1) * C(p_utf8character^-(stop-start+1)) + -- inspect(pattern) + -- return lpegmatch(pattern,str) or "" + -- end + + -- local b, e, n, first, last = 0, 0, 0, 0, 0 + -- + -- local function slide(s,p) + -- n = n + 1 + -- if n == first then + -- b = p + -- if not last then + -- return nil + -- end + -- end + -- if n == last then + -- e = p + -- return nil + -- else + -- return p + -- end + -- end + -- + -- local pattern = Cmt(p_utf8character,slide)^0 + -- + -- function utf.sub(str,start,stop) -- todo: from the end + -- if not start then + -- return str + -- end + -- b, e, n, first, last = 0, 0, 0, start, stop + -- lpegmatch(pattern,str) + -- if not stop then + -- return sub(str,b) + -- else + -- return sub(str,b,e-1) + -- end + -- end + + -- print(utf.sub("Hans Hagen is my name")) + -- print(utf.sub("Hans Hagen is my name",5)) + -- print(utf.sub("Hans Hagen is my name",5,10)) + + local utflength = utf.length + + -- also negative indices, upto 10 times slower than a c variant + + local b, e, n, first, last = 0, 0, 0, 0, 0 + + local function slide_zero(s,p) + n = n + 1 + if n >= last then + e = p - 1 + else + return p + end + end + + local function slide_one(s,p) + n = n + 1 + if n == first then + b = p + end + if n >= last then + e = p - 1 + else + return p + end + end + + local function slide_two(s,p) + n = n + 1 + if n == first then + b = p + else + return true + end + end + + local pattern_zero = Cmt(p_utf8character,slide_zero)^0 + local pattern_one = Cmt(p_utf8character,slide_one )^0 + local pattern_two = Cmt(p_utf8character,slide_two )^0 + + local pattern_first = C(p_utf8character) + + function utf.sub(str,start,stop) + if not start then + return str + end + if start == 0 then + start = 1 + end + if not stop then + if start < 0 then + local l = utflength(str) -- we can inline this function if needed + start = l + start + else + start = start - 1 + end + b, n, first = 0, 0, start + lpegmatch(pattern_two,str) + if n >= first then + return sub(str,b) + else + return "" + end + end + if start < 0 or stop < 0 then + local l = utf.length(str) + if start < 0 then + start = l + start + if start <= 0 then + start = 1 + else + start = start + 1 + end + end + if stop < 0 then + stop = l + stop + if stop == 0 then + stop = 1 + else + stop = stop + 1 + end + end + end + if start == 1 and stop == 1 then + return lpegmatch(pattern_first,str) or "" + elseif start > stop then + return "" + elseif start > 1 then + b, e, n, first, last = 0, 0, 0, start - 1, stop + lpegmatch(pattern_one,str) + if n >= first and e == 0 then + e = #str + end + return sub(str,b,e) + else + b, e, n, last = 1, 0, 0, stop + lpegmatch(pattern_zero,str) + if e == 0 then + e = #str + end + return sub(str,b,e) + end + end + + -- local n = 100000 + -- local str = string.rep("123456àáâãäå",100) + -- + -- for i=-15,15,1 do + -- for j=-15,15,1 do + -- if utf.xsub(str,i,j) ~= utf.sub(str,i,j) then + -- print("error",i,j,"l>"..utf.xsub(str,i,j),"s>"..utf.sub(str,i,j)) + -- end + -- end + -- if utf.xsub(str,i) ~= utf.sub(str,i) then + -- print("error",i,"l>"..utf.xsub(str,i),"s>"..utf.sub(str,i)) + -- end + -- end + + -- print(" 1, 7",utf.xsub(str, 1, 7),utf.sub(str, 1, 7)) + -- print(" 0, 7",utf.xsub(str, 0, 7),utf.sub(str, 0, 7)) + -- print(" 0, 9",utf.xsub(str, 0, 9),utf.sub(str, 0, 9)) + -- print(" 4 ",utf.xsub(str, 4 ),utf.sub(str, 4 )) + -- print(" 0 ",utf.xsub(str, 0 ),utf.sub(str, 0 )) + -- print(" 0, 0",utf.xsub(str, 0, 0),utf.sub(str, 0, 0)) + -- print(" 4, 4",utf.xsub(str, 4, 4),utf.sub(str, 4, 4)) + -- print(" 4, 0",utf.xsub(str, 4, 0),utf.sub(str, 4, 0)) + -- print("-3, 0",utf.xsub(str,-3, 0),utf.sub(str,-3, 0)) + -- print(" 0,-3",utf.xsub(str, 0,-3),utf.sub(str, 0,-3)) + -- print(" 5,-3",utf.xsub(str,-5,-3),utf.sub(str,-5,-3)) + -- print("-3 ",utf.xsub(str,-3 ),utf.sub(str,-3 )) + +end + +-- a replacement for simple gsubs: + +-- function utf.remapper(mapping) +-- local pattern = Cs((p_utf8character/mapping)^0) +-- return function(str) +-- if not str or str == "" then +-- return "" +-- else +-- return lpegmatch(pattern,str) +-- end +-- end, pattern +-- end + +function utf.remapper(mapping,option,action) -- static also returns a pattern + local variant = type(mapping) + if variant == "table" then + action = action or mapping + if option == "dynamic" then + local pattern = false + table.setmetatablenewindex(mapping,function(t,k,v) rawset(t,k,v) pattern = false end) + return function(str) + if not str or str == "" then + return "" + else + if not pattern then + pattern = Cs((tabletopattern(mapping)/action + p_utf8character)^0) + end + return lpegmatch(pattern,str) + end + end + elseif option == "pattern" then + return Cs((tabletopattern(mapping)/action + p_utf8character)^0) + -- elseif option == "static" then + else + local pattern = Cs((tabletopattern(mapping)/action + p_utf8character)^0) + return function(str) + if not str or str == "" then + return "" + else + return lpegmatch(pattern,str) + end + end, pattern + end + elseif variant == "function" then + if option == "pattern" then + return Cs((p_utf8character/mapping + p_utf8character)^0) + else + local pattern = Cs((p_utf8character/mapping + p_utf8character)^0) + return function(str) + if not str or str == "" then + return "" + else + return lpegmatch(pattern,str) + end + end, pattern + end + else + -- is actually an error + return function(str) + return str or "" + end + end +end + +-- local remap = utf.remapper { a = 'd', b = "c", c = "b", d = "a" } +-- print(remap("abcd 1234 abcd")) + +function utf.replacer(t) -- no precheck, always string builder + local r = replacer(t,false,false,true) + return function(str) + return lpegmatch(r,str) + end +end + +function utf.subtituter(t) -- with precheck and no building if no match + local f = finder (t) + local r = replacer(t,false,false,true) + return function(str) + local i = lpegmatch(f,str) + if not i then + return str + elseif i > #str then + return str + else + -- return sub(str,1,i-2) .. lpegmatch(r,str,i-1) -- slower + return lpegmatch(r,str) + end + end +end + +-- inspect(utf.split("a b c d")) +-- inspect(utf.split("a b c d",true)) + +local utflinesplitter = p_utfbom^-1 * lpeg.tsplitat(p_newline) +local utfcharsplitter_ows = p_utfbom^-1 * Ct(C(p_utf8character)^0) +local utfcharsplitter_iws = p_utfbom^-1 * Ct((p_whitespace^1 + C(p_utf8character))^0) +local utfcharsplitter_raw = Ct(C(p_utf8character)^0) + +patterns.utflinesplitter = utflinesplitter + +function utf.splitlines(str) + return lpegmatch(utflinesplitter,str or "") +end + +function utf.split(str,ignorewhitespace) -- new + if ignorewhitespace then + return lpegmatch(utfcharsplitter_iws,str or "") + else + return lpegmatch(utfcharsplitter_ows,str or "") + end +end + +function utf.totable(str) -- keeps bom + return lpegmatch(utfcharsplitter_raw,str) +end + +-- 0 EF BB BF UTF-8 +-- 1 FF FE UTF-16-little-endian +-- 2 FE FF UTF-16-big-endian +-- 3 FF FE 00 00 UTF-32-little-endian +-- 4 00 00 FE FF UTF-32-big-endian +-- +-- \000 fails in <= 5.0 but is valid in >=5.1 where %z is depricated + +-- utf.name = { +-- [0] = 'utf-8', +-- [1] = 'utf-16-le', +-- [2] = 'utf-16-be', +-- [3] = 'utf-32-le', +-- [4] = 'utf-32-be' +-- } +-- +-- function utf.magic(f) +-- local str = f:read(4) +-- if not str then +-- f:seek('set') +-- return 0 +-- -- elseif find(str,"^%z%z\254\255") then -- depricated +-- -- elseif find(str,"^\000\000\254\255") then -- not permitted and bugged +-- elseif find(str,"\000\000\254\255",1,true) then -- seems to work okay (TH) +-- return 4 +-- -- elseif find(str,"^\255\254%z%z") then -- depricated +-- -- elseif find(str,"^\255\254\000\000") then -- not permitted and bugged +-- elseif find(str,"\255\254\000\000",1,true) then -- seems to work okay (TH) +-- return 3 +-- elseif find(str,"^\254\255") then +-- f:seek('set',2) +-- return 2 +-- elseif find(str,"^\255\254") then +-- f:seek('set',2) +-- return 1 +-- elseif find(str,"^\239\187\191") then +-- f:seek('set',3) +-- return 0 +-- else +-- f:seek('set') +-- return 0 +-- end +-- end + +function utf.magic(f) -- not used + local str = f:read(4) or "" + local off = lpegmatch(p_utfoffset,str) + if off < 4 then + f:seek('set',off) + end + return lpegmatch(p_utftype,str) +end + +local utf16_to_utf8_be, utf16_to_utf8_le +local utf32_to_utf8_be, utf32_to_utf8_le + +local utf_16_be_getbom = patterns.utfbom_16_be^-1 +local utf_16_le_getbom = patterns.utfbom_16_le^-1 +local utf_32_be_getbom = patterns.utfbom_32_be^-1 +local utf_32_le_getbom = patterns.utfbom_32_le^-1 + +local utf_16_be_linesplitter = utf_16_be_getbom * lpeg.tsplitat(patterns.utf_16_be_nl) +local utf_16_le_linesplitter = utf_16_le_getbom * lpeg.tsplitat(patterns.utf_16_le_nl) +local utf_32_be_linesplitter = utf_32_be_getbom * lpeg.tsplitat(patterns.utf_32_be_nl) +local utf_32_le_linesplitter = utf_32_le_getbom * lpeg.tsplitat(patterns.utf_32_le_nl) + +-- we have three possibilities: bytepairs (using tables), gmatch (using tables), gsub and +-- lpeg. Bytepairs are the fastert but as soon as we need to remove bombs and so the gain +-- is less due to more testing. Also, we seldom have to convert utf16 so we don't care to +-- much about a few milliseconds more runtime. The lpeg variant is upto 20% slower but +-- still pretty fast. +-- +-- for historic resone we keep the bytepairs variants around .. beware they don't grab the +-- bom like the lpegs do so they're not dropins in the functions that follow +-- +-- utf16_to_utf8_be = function(s) +-- if not s then +-- return nil +-- elseif s == "" then +-- return "" +-- end +-- local result, r, more = { }, 0, 0 +-- for left, right in bytepairs(s) do +-- if right then +-- local now = 256*left + right +-- if more > 0 then +-- now = (more-0xD800)*0x400 + (now-0xDC00) + 0x10000 -- the 0x10000 smells wrong +-- more = 0 +-- r = r + 1 +-- result[r] = utfchar(now) +-- elseif now >= 0xD800 and now <= 0xDBFF then +-- more = now +-- else +-- r = r + 1 +-- result[r] = utfchar(now) +-- end +-- end +-- end +-- return concat(result) +-- end +-- +-- local utf16_to_utf8_be_t = function(t) +-- if not t then +-- return nil +-- elseif type(t) == "string" then +-- t = lpegmatch(utf_16_be_linesplitter,t) +-- end +-- local result = { } -- we reuse result +-- for i=1,#t do +-- local s = t[i] +-- if s ~= "" then +-- local r, more = 0, 0 +-- for left, right in bytepairs(s) do +-- if right then +-- local now = 256*left + right +-- if more > 0 then +-- now = (more-0xD800)*0x400 + (now-0xDC00) + 0x10000 +-- more = 0 +-- r = r + 1 +-- result[r] = utfchar(now) +-- elseif now >= 0xD800 and now <= 0xDBFF then +-- more = now +-- else +-- r = r + 1 +-- result[r] = utfchar(now) +-- end +-- end +-- end +-- t[i] = concat(result,"",1,r) -- we reused tmp, hence t +-- end +-- end +-- return t +-- end +-- +-- utf16_to_utf8_le = function(s) +-- if not s then +-- return nil +-- elseif s == "" then +-- return "" +-- end +-- local result, r, more = { }, 0, 0 +-- for left, right in bytepairs(s) do +-- if right then +-- local now = 256*right + left +-- if more > 0 then +-- now = (more-0xD800)*0x400 + (now-0xDC00) + 0x10000 +-- more = 0 +-- r = r + 1 +-- result[r] = utfchar(now) +-- elseif now >= 0xD800 and now <= 0xDBFF then +-- more = now +-- else +-- r = r + 1 +-- result[r] = utfchar(now) +-- end +-- end +-- end +-- return concat(result) +-- end +-- +-- local utf16_to_utf8_le_t = function(t) +-- if not t then +-- return nil +-- elseif type(t) == "string" then +-- t = lpegmatch(utf_16_le_linesplitter,t) +-- end +-- local result = { } -- we reuse result +-- for i=1,#t do +-- local s = t[i] +-- if s ~= "" then +-- local r, more = 0, 0 +-- for left, right in bytepairs(s) do +-- if right then +-- local now = 256*right + left +-- if more > 0 then +-- now = (more-0xD800)*0x400 + (now-0xDC00) + 0x10000 +-- more = 0 +-- r = r + 1 +-- result[r] = utfchar(now) +-- elseif now >= 0xD800 and now <= 0xDBFF then +-- more = now +-- else +-- r = r + 1 +-- result[r] = utfchar(now) +-- end +-- end +-- end +-- t[i] = concat(result,"",1,r) -- we reused tmp, hence t +-- end +-- end +-- return t +-- end +-- +-- local utf32_to_utf8_be_t = function(t) +-- if not t then +-- return nil +-- elseif type(t) == "string" then +-- t = lpegmatch(utflinesplitter,t) +-- end +-- local result = { } -- we reuse result +-- for i=1,#t do +-- local r, more = 0, -1 +-- for a,b in bytepairs(t[i]) do +-- if a and b then +-- if more < 0 then +-- more = 256*256*256*a + 256*256*b +-- else +-- r = r + 1 +-- result[t] = utfchar(more + 256*a + b) +-- more = -1 +-- end +-- else +-- break +-- end +-- end +-- t[i] = concat(result,"",1,r) +-- end +-- return t +-- end +-- +-- local utf32_to_utf8_le_t = function(t) +-- if not t then +-- return nil +-- elseif type(t) == "string" then +-- t = lpegmatch(utflinesplitter,t) +-- end +-- local result = { } -- we reuse result +-- for i=1,#t do +-- local r, more = 0, -1 +-- for a,b in bytepairs(t[i]) do +-- if a and b then +-- if more < 0 then +-- more = 256*b + a +-- else +-- r = r + 1 +-- result[t] = utfchar(more + 256*256*256*b + 256*256*a) +-- more = -1 +-- end +-- else +-- break +-- end +-- end +-- t[i] = concat(result,"",1,r) +-- end +-- return t +-- end + +local more = 0 + +local p_utf16_to_utf8_be = C(1) * C(1) /function(left,right) + local now = 256*byte(left) + byte(right) + if more > 0 then + now = (more-0xD800)*0x400 + (now-0xDC00) + 0x10000 + more = 0 + return utfchar(now) + elseif now >= 0xD800 and now <= 0xDBFF then + more = now + return "" -- else the c's end up in the stream + else + return utfchar(now) + end +end + +local p_utf16_to_utf8_le = C(1) * C(1) /function(right,left) + local now = 256*byte(left) + byte(right) + if more > 0 then + now = (more-0xD800)*0x400 + (now-0xDC00) + 0x10000 + more = 0 + return utfchar(now) + elseif now >= 0xD800 and now <= 0xDBFF then + more = now + return "" -- else the c's end up in the stream + else + return utfchar(now) + end +end +local p_utf32_to_utf8_be = C(1) * C(1) * C(1) * C(1) /function(a,b,c,d) + return utfchar(256*256*256*byte(a) + 256*256*byte(b) + 256*byte(c) + byte(d)) +end + +local p_utf32_to_utf8_le = C(1) * C(1) * C(1) * C(1) /function(a,b,c,d) + return utfchar(256*256*256*byte(d) + 256*256*byte(c) + 256*byte(b) + byte(a)) +end + +p_utf16_to_utf8_be = P(true) / function() more = 0 end * utf_16_be_getbom * Cs(p_utf16_to_utf8_be^0) +p_utf16_to_utf8_le = P(true) / function() more = 0 end * utf_16_le_getbom * Cs(p_utf16_to_utf8_le^0) +p_utf32_to_utf8_be = P(true) / function() more = 0 end * utf_32_be_getbom * Cs(p_utf32_to_utf8_be^0) +p_utf32_to_utf8_le = P(true) / function() more = 0 end * utf_32_le_getbom * Cs(p_utf32_to_utf8_le^0) + +patterns.utf16_to_utf8_be = p_utf16_to_utf8_be +patterns.utf16_to_utf8_le = p_utf16_to_utf8_le +patterns.utf32_to_utf8_be = p_utf32_to_utf8_be +patterns.utf32_to_utf8_le = p_utf32_to_utf8_le + +utf16_to_utf8_be = function(s) + if s and s ~= "" then + return lpegmatch(p_utf16_to_utf8_be,s) + else + return s + end +end + +local utf16_to_utf8_be_t = function(t) + if not t then + return nil + elseif type(t) == "string" then + t = lpegmatch(utf_16_be_linesplitter,t) + end + for i=1,#t do + local s = t[i] + if s ~= "" then + t[i] = lpegmatch(p_utf16_to_utf8_be,s) + end + end + return t +end + +utf16_to_utf8_le = function(s) + if s and s ~= "" then + return lpegmatch(p_utf16_to_utf8_le,s) + else + return s + end +end + +local utf16_to_utf8_le_t = function(t) + if not t then + return nil + elseif type(t) == "string" then + t = lpegmatch(utf_16_le_linesplitter,t) + end + for i=1,#t do + local s = t[i] + if s ~= "" then + t[i] = lpegmatch(p_utf16_to_utf8_le,s) + end + end + return t +end + +utf32_to_utf8_be = function(s) + if s and s ~= "" then + return lpegmatch(p_utf32_to_utf8_be,s) + else + return s + end +end + +local utf32_to_utf8_be_t = function(t) + if not t then + return nil + elseif type(t) == "string" then + t = lpegmatch(utf_32_be_linesplitter,t) + end + for i=1,#t do + local s = t[i] + if s ~= "" then + t[i] = lpegmatch(p_utf32_to_utf8_be,s) + end + end + return t +end + +utf32_to_utf8_le = function(s) + if s and s ~= "" then + return lpegmatch(p_utf32_to_utf8_le,s) + else + return s + end +end + +local utf32_to_utf8_le_t = function(t) + if not t then + return nil + elseif type(t) == "string" then + t = lpegmatch(utf_32_le_linesplitter,t) + end + for i=1,#t do + local s = t[i] + if s ~= "" then + t[i] = lpegmatch(p_utf32_to_utf8_le,s) + end + end + return t +end + +utf.utf16_to_utf8_le_t = utf16_to_utf8_le_t +utf.utf16_to_utf8_be_t = utf16_to_utf8_be_t +utf.utf32_to_utf8_le_t = utf32_to_utf8_le_t +utf.utf32_to_utf8_be_t = utf32_to_utf8_be_t + +utf.utf16_to_utf8_le = utf16_to_utf8_le +utf.utf16_to_utf8_be = utf16_to_utf8_be +utf.utf32_to_utf8_le = utf32_to_utf8_le +utf.utf32_to_utf8_be = utf32_to_utf8_be + +function utf.utf8_to_utf8_t(t) + return type(t) == "string" and lpegmatch(utflinesplitter,t) or t +end + +function utf.utf16_to_utf8_t(t,endian) + return endian and utf16_to_utf8_be_t(t) or utf16_to_utf8_le_t(t) or t +end + +function utf.utf32_to_utf8_t(t,endian) + return endian and utf32_to_utf8_be_t(t) or utf32_to_utf8_le_t(t) or t +end + +local function little(b) + if b < 0x10000 then + return char(b%256,rshift(b,8)) + else + b = b - 0x10000 + local b1 = rshift(b,10) + 0xD800 + local b2 = b%1024 + 0xDC00 + return char(b1%256,rshift(b1,8),b2%256,rshift(b2,8)) + end +end + +local function big(b) + if b < 0x10000 then + return char(rshift(b,8),b%256) + else + b = b - 0x10000 + local b1 = rshift(b,10) + 0xD800 + local b2 = b%1024 + 0xDC00 + return char(rshift(b1,8),b1%256,rshift(b2,8),b2%256) + end +end + +local l_remap = Cs((p_utf8byte/little+P(1)/"")^0) +local b_remap = Cs((p_utf8byte/big +P(1)/"")^0) + +local function utf8_to_utf16_be(str,nobom) + if nobom then + return lpegmatch(b_remap,str) + else + return char(254,255) .. lpegmatch(b_remap,str) + end +end + +local function utf8_to_utf16_le(str,nobom) + if nobom then + return lpegmatch(l_remap,str) + else + return char(255,254) .. lpegmatch(l_remap,str) + end +end + +utf.utf8_to_utf16_be = utf8_to_utf16_be +utf.utf8_to_utf16_le = utf8_to_utf16_le + +function utf.utf8_to_utf16(str,littleendian,nobom) + if littleendian then + return utf8_to_utf16_le(str,nobom) + else + return utf8_to_utf16_be(str,nobom) + end +end + +local pattern = Cs ( + (p_utf8byte / function(unicode ) return format( "0x%04X", unicode) end) * + (p_utf8byte * Carg(1) / function(unicode,separator) return format("%s0x%04X",separator,unicode) end)^0 +) + +function utf.tocodes(str,separator) + return lpegmatch(pattern,str,1,separator or " ") +end + +function utf.ustring(s) + return format("U+%05X",type(s) == "number" and s or utfbyte(s)) +end + +function utf.xstring(s) + return format("0x%05X",type(s) == "number" and s or utfbyte(s)) +end + +function utf.toeight(str) + if not str or str == "" then + return nil + end + local utftype = lpegmatch(p_utfstricttype,str) + if utftype == "utf-8" then + return sub(str,4) -- remove the bom + elseif utftype == "utf-16-be" then + return utf16_to_utf8_be(str) -- bom gets removed + elseif utftype == "utf-16-le" then + return utf16_to_utf8_le(str) -- bom gets removed + else + return str + end +end + +-- + +do + + local p_nany = p_utf8character / "" + local cache = { } + + function utf.count(str,what) + if type(what) == "string" then + local p = cache[what] + if not p then + p = Cs((P(what)/" " + p_nany)^0) + cache[p] = p + end + return #lpegmatch(p,str) + else -- 4 times slower but still faster than / function + return #lpegmatch(Cs((P(what)/" " + p_nany)^0),str) + end + end + +end + +if not string.utfvalues then + + -- So, a logical next step is to check for the values variant. It over five times + -- slower than the built-in string.utfvalues. I optimized it a bit for n=0,1. + + ----- wrap, yield, gmatch = coroutine.wrap, coroutine.yield, string.gmatch + local find = string.find + + local dummy = function() + -- we share this one + end + + -- function string.utfvalues(str) + -- local n = #str + -- if n == 0 then + -- return wrap(dummy) + -- elseif n == 1 then + -- return wrap(function() yield(utfbyte(str)) end) + -- else + -- return wrap(function() for s in gmatch(str,".[\128-\191]*") do + -- yield(utfbyte(s)) + -- end end) + -- end + -- end + -- + -- faster: + + function string.utfvalues(str) + local n = #str + if n == 0 then + return dummy + elseif n == 1 then + return function() return utfbyte(str) end + else + local p = 1 + -- local n = #str + return function() + -- if p <= n then -- slower than the last find + local b, e = find(str,".[\128-\191]*",p) + if b then + p = e + 1 + return utfbyte(sub(str,b,e)) + end + -- end + end + end + end + + -- slower: + -- + -- local pattern = C(p_utf8character) * Cp() + -- ----- pattern = p_utf8character/utfbyte * Cp() + -- ----- pattern = p_utf8byte * Cp() + -- + -- function string.utfvalues(str) -- one of the cases where a find is faster than an lpeg + -- local n = #str + -- if n == 0 then + -- return dummy + -- elseif n == 1 then + -- return function() return utfbyte(str) end + -- else + -- local p = 1 + -- return function() + -- local s, e = lpegmatch(pattern,str,p) + -- if e then + -- p = e + -- return utfbyte(s) + -- -- return s + -- end + -- end + -- end + -- end + +end + +utf.values = string.utfvalues + +function utf.chrlen(u) -- u is number + return + (u < 0x80 and 1) or + (u < 0xE0 and 2) or + (u < 0xF0 and 3) or + (u < 0xF8 and 4) or + (u < 0xFC and 5) or + (u < 0xFE and 6) or 0 +end + +-- hashing saves a little but not that much in practice +-- +-- local utf32 = table.setmetatableindex(function(t,k) local v = toutf32(k) t[k] = v return v end) + +if bit32 then + + local extract = bit32.extract + local char = string.char + + function utf.toutf32string(n) + if n <= 0xFF then + return + char(n) .. + "\000\000\000" + elseif n <= 0xFFFF then + return + char(extract(n, 0,8)) .. + char(extract(n, 8,8)) .. + "\000\000" + elseif n <= 0xFFFFFF then + return + char(extract(n, 0,8)) .. + char(extract(n, 8,8)) .. + char(extract(n,16,8)) .. + "\000" + else + return + char(extract(n, 0,8)) .. + char(extract(n, 8,8)) .. + char(extract(n,16,8)) .. + char(extract(n,24,8)) + end + end + +end + +-- goodie: + +local len = utf.len +local rep = rep + +function string.utfpadd(s,n) + if n and n ~= 0 then + local l = len(s) + if n > 0 then + local d = n - l + if d > 0 then + return rep(c or " ",d) .. s + end + else + local d = - n - l + if d > 0 then + return s .. rep(c or " ",d) + end + end + end + return s +end + +-- goodies + +do + + local utfcharacters = utf.characters or string.utfcharacters + local utfchar = utf.char or string.utfcharacter + + lpeg.UP = P + + if utfcharacters then + + function lpeg.US(str) + local p = P(false) + for uc in utfcharacters(str) do + p = p + P(uc) + end + return p + end + + else + + function lpeg.US(str) + local p = P(false) + local f = function(uc) + p = p + P(uc) + end + lpegmatch((p_utf8char/f)^0,str) + return p + end + + end + + local range = p_utf8byte * p_utf8byte + Cc(false) -- utf8byte is already a capture + + function lpeg.UR(str,more) + local first, last + if type(str) == "number" then + first = str + last = more or first + else + first, last = lpegmatch(range,str) + if not last then + return P(str) + end + end + if first == last then + return P(str) + end + if not utfchar then + utfchar = utf.char -- maybe delayed + end + if utfchar and (last - first < 8) then -- a somewhat arbitrary criterium + local p = P(false) + for i=first,last do + p = p + P(utfchar(i)) + end + return p -- nil when invalid range + else + local f = function(b) + return b >= first and b <= last + end + -- tricky, these nested captures + return p_utf8byte / f -- nil when invalid range + end + end + + -- print(lpeg.match(lpeg.Cs((C(lpeg.UR("αω"))/{ ["χ"] = "OEPS" })^0),"αωχαω")) + +end diff --git a/macros/luatex/generic/lualibs/lualibs-url.lua b/macros/luatex/generic/lualibs/lualibs-url.lua new file mode 100644 index 0000000000..6f834d778d --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-url.lua @@ -0,0 +1,431 @@ +if not modules then modules = { } end modules ['l-url'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +local char, format, byte = string.char, string.format, string.byte +local concat = table.concat +local tonumber, type, next = tonumber, type, next +local P, C, R, S, Cs, Cc, Ct, Cf, Cg, V = lpeg.P, lpeg.C, lpeg.R, lpeg.S, lpeg.Cs, lpeg.Cc, lpeg.Ct, lpeg.Cf, lpeg.Cg, lpeg.V +local lpegmatch, lpegpatterns, replacer = lpeg.match, lpeg.patterns, lpeg.replacer +local sortedhash = table.sortedhash + +-- from wikipedia: +-- +-- foo://username:password@example.com:8042/over/there/index.dtb?type=animal;name=narwhal#nose +-- \_/ \_______________/ \_________/ \__/ \___/ \_/ \______________________/ \__/ +-- | | | | | | | | +-- | userinfo hostname port | | query fragment +-- | \________________________________/\_____________|____|/ +-- scheme | | | | +-- | authority path | | +-- | | | +-- | path interpretable as filename +-- | ___________|____________ | +-- / \ / \ | +-- urn:example:animal:ferret:nose interpretable as extension +-- +-- also nice: http://url.spec.whatwg.org/ (maybe some day ...) + +url = url or { } +local url = url + +local unescapes = { } +local escapes = { } + +setmetatable(unescapes, { __index = function(t,k) + local v = char(tonumber(k,16)) + t[k] = v + return v +end }) + +setmetatable(escapes, { __index = function(t,k) + local v = format("%%%02X",byte(k)) + t[k] = v + return v +end }) + +-- okay: + +local colon = P(":") +local qmark = P("?") +local hash = P("#") +local slash = P("/") +local atsign = P("@") +local percent = P("%") +local endofstring = P(-1) +local hexdigit = R("09","AF","af") +local plus = P("+") +local nothing = Cc("") +local okay = R("09","AZ","az") + S("-_.,:=+*~!'()@&$") + +local escapedchar = (percent * C(hexdigit * hexdigit)) / unescapes +local unescapedchar = P(1) / escapes +local escaped = (plus / " ") + escapedchar -- so no loc://foo++.tex +local noslash = P("/") / "" +local plustospace = P("+")/" " + +local decoder = Cs( ( + plustospace + + escapedchar + + P("\r\n")/"\n" + + P(1) + )^0 ) +local encoder = Cs( ( + R("09","AZ","az")^1 + + S("-./_")^1 + + P(" ")/"+" + + P("\n")/"\r\n" + + unescapedchar + )^0 ) + +lpegpatterns.urldecoder = decoder +lpegpatterns.urlencoder = encoder + +function url.decode (str) return str and lpegmatch(decoder, str) or str end +function url.encode (str) return str and lpegmatch(encoder, str) or str end +function url.unescape(str) return str and lpegmatch(unescaper,str) or str end + +-- we assume schemes with more than 1 character (in order to avoid problems with windows disks) +-- we also assume that when we have a scheme, we also have an authority +-- +-- maybe we should already split the query (better for unescaping as = & can be part of a value + +local schemestr = Cs((escaped+(1-colon-slash-qmark-hash))^2) +local authoritystr = Cs((escaped+(1- slash-qmark-hash))^0) +local pathstr = Cs((escaped+(1- qmark-hash))^0) +----- querystr = Cs((escaped+(1- hash))^0) +local querystr = Cs(( (1- hash))^0) +local fragmentstr = Cs((escaped+(1- endofstring))^0) + +local scheme = schemestr * colon + nothing +local authority = slash * slash * authoritystr + nothing +local path = slash * pathstr + nothing +local query = qmark * querystr + nothing +local fragment = hash * fragmentstr + nothing + +local validurl = scheme * authority * path * query * fragment +local parser = Ct(validurl) + +lpegpatterns.url = validurl +lpegpatterns.urlsplitter = parser + +local escaper = Cs((R("09","AZ","az")^1 + P(" ")/"%%20" + S("-./_:")^1 + P(1) / escapes)^0) -- space happens most +local unescaper = Cs((escapedchar + 1)^0) +local getcleaner = Cs((P("+++")/"%%2B" + P("+")/"%%20" + P(1))^1) + +lpegpatterns.urlunescaped = escapedchar +lpegpatterns.urlescaper = escaper +lpegpatterns.urlunescaper = unescaper +lpegpatterns.urlgetcleaner = getcleaner + +function url.unescapeget(str) + return lpegmatch(getcleaner,str) +end + +-- todo: reconsider Ct as we can as well have five return values (saves a table) +-- so we can have two parsers, one with and one without + +local function split(str) + return (type(str) == "string" and lpegmatch(parser,str)) or str +end + +local isscheme = schemestr * colon * slash * slash -- this test also assumes authority + +local function hasscheme(str) + if str then + local scheme = lpegmatch(isscheme,str) -- at least one character + return scheme ~= "" and scheme or false + else + return false + end +end + +--~ print(hasscheme("home:")) +--~ print(hasscheme("home://")) + +-- todo: cache them + +local rootletter = R("az","AZ") + + S("_-+") +local separator = P("://") +local qualified = P(".")^0 * P("/") + + rootletter * P(":") + + rootletter^1 * separator + + rootletter^1 * P("/") +local rootbased = P("/") + + rootletter * P(":") + +local barswapper = replacer("|",":") +local backslashswapper = replacer("\\","/") + +-- queries: + +local equal = P("=") +local amp = P("&") +local key = Cs(((plustospace + escapedchar + 1) - equal )^0) +local value = Cs(((plustospace + escapedchar + 1) - amp - endofstring)^0) + +local splitquery = Cf ( Ct("") * P { "sequence", + sequence = V("pair") * (amp * V("pair"))^0, + pair = Cg(key * equal * value), +}, rawset) + +-- hasher + +local userpart = (1-atsign-colon)^1 +local serverpart = (1-colon)^1 +local splitauthority = ((Cs(userpart) * colon * Cs(userpart) + Cs(userpart) * Cc(nil)) * atsign + Cc(nil) * Cc(nil)) + * Cs(serverpart) * (colon * (serverpart/tonumber) + Cc(nil)) + +local function hashed(str) -- not yet ok (/test?test) + if not str or str == "" then + return { + scheme = "invalid", + original = str, + } + end + local detailed = split(str) + local rawscheme = "" + local rawquery = "" + local somescheme = false + local somequery = false + if detailed then + rawscheme = detailed[1] + rawquery = detailed[4] + somescheme = rawscheme ~= "" + somequery = rawquery ~= "" + end + if not somescheme and not somequery then + return { + scheme = "file", + authority = "", + path = str, + query = "", + fragment = "", + original = str, + noscheme = true, + filename = str, + } + end + -- not always a filename but handy anyway + local authority = detailed[2] + local path = detailed[3] + local filename -- = nil + local username -- = nil + local password -- = nil + local host -- = nil + local port -- = nil + if authority ~= "" then + -- these can be invalid + username, password, host, port = lpegmatch(splitauthority,authority) + end + if authority == "" then + filename = path + elseif path == "" then + filename = "" + else + -- this one can be can be invalid + filename = authority .. "/" .. path + end + return { + scheme = rawscheme, + authority = authority, + path = path, + query = lpegmatch(unescaper,rawquery), -- unescaped, but possible conflict with & and = + queries = lpegmatch(splitquery,rawquery), -- split first and then unescaped + fragment = detailed[5], + original = str, + noscheme = false, + filename = filename, + -- + host = host, + port = port, + -- usename = username, + -- password = password, + } +end + +-- inspect(hashed()) +-- inspect(hashed("")) +-- inspect(hashed("template:///test")) +-- inspect(hashed("template:///test++.whatever")) +-- inspect(hashed("template:///test%2B%2B.whatever")) +-- inspect(hashed("template:///test%x.whatever")) +-- inspect(hashed("tem%2Bplate:///test%x.whatever")) + +-- Here we assume: +-- +-- files: /// = relative +-- files: //// = absolute (!) + +--~ table.print(hashed("file://c:/opt/tex/texmf-local")) -- c:/opt/tex/texmf-local +--~ table.print(hashed("file://opt/tex/texmf-local" )) -- opt/tex/texmf-local +--~ table.print(hashed("file:///opt/tex/texmf-local" )) -- opt/tex/texmf-local +--~ table.print(hashed("file:////opt/tex/texmf-local" )) -- /opt/tex/texmf-local +--~ table.print(hashed("file:///./opt/tex/texmf-local" )) -- ./opt/tex/texmf-local + +--~ table.print(hashed("c:/opt/tex/texmf-local" )) -- c:/opt/tex/texmf-local +--~ table.print(hashed("opt/tex/texmf-local" )) -- opt/tex/texmf-local +--~ table.print(hashed("/opt/tex/texmf-local" )) -- /opt/tex/texmf-local + +url.split = split +url.hasscheme = hasscheme +url.hashed = hashed + +function url.addscheme(str,scheme) -- no authority + if hasscheme(str) then + return str + elseif not scheme then + return "file:///" .. str + else + return scheme .. ":///" .. str + end +end + +function url.construct(hash) -- dodo: we need to escape ! + local result, r = { }, 0 + local scheme = hash.scheme + local authority = hash.authority + local path = hash.path + local queries = hash.queries + local fragment = hash.fragment + if scheme and scheme ~= "" then + r = r + 1 ; result[r] = lpegmatch(escaper,scheme) + r = r + 1 ; result[r] = "://" + end + if authority and authority ~= "" then + r = r + 1 ; result[r] = lpegmatch(escaper,authority) + end + if path and path ~= "" then + r = r + 1 ; result[r] = "/" + r = r + 1 ; result[r] = lpegmatch(escaper,path) + end + if queries then + local done = false + for k, v in sortedhash(queries) do + r = r + 1 ; result[r] = done and "&" or "?" + r = r + 1 ; result[r] = lpegmatch(escaper,k) -- is this escaped + r = r + 1 ; result[r] = "=" + r = r + 1 ; result[r] = lpegmatch(escaper,v) -- is this escaped + done = true + end + end + if fragment and fragment ~= "" then + r = r + 1 ; result[r] = "#" + r = r + 1 ; result[r] = lpegmatch(escaper,fragment) + end + return concat(result) +end + +local pattern = Cs(slash^-1/"" * R("az","AZ") * ((S(":|")/":") + P(":")) * slash * P(1)^0) + +function url.filename(filename) + local spec = hashed(filename) + local path = spec.path + return (spec.scheme == "file" and path and lpegmatch(pattern,path)) or filename +end + +-- print(url.filename("/c|/test")) +-- print(url.filename("/c/test")) +-- print(url.filename("file:///t:/sources/cow.svg")) + +local function escapestring(str) + return lpegmatch(escaper,str) +end + +url.escape = escapestring + +function url.query(str) + if type(str) == "string" then + return lpegmatch(splitquery,str) or "" + else + return str + end +end + +function url.toquery(data) + local td = type(data) + if td == "string" then + return #str and escape(data) or nil -- beware of double escaping + elseif td == "table" then + if next(data) then + local t = { } + for k, v in next, data do + t[#t+1] = format("%s=%s",k,escapestring(v)) + end + return concat(t,"&") + end + else + -- nil is a signal that no query + end +end + +-- /test/ | /test | test/ | test => test + +local pattern = Cs(noslash^0 * (1 - noslash * P(-1))^0) + +function url.barepath(path) + if not path or path == "" then + return "" + else + return lpegmatch(pattern,path) + end +end + +-- print(url.barepath("/test"),url.barepath("test/"),url.barepath("/test/"),url.barepath("test")) +-- print(url.barepath("/x/yz"),url.barepath("x/yz/"),url.barepath("/x/yz/"),url.barepath("x/yz")) + +--~ print(url.filename("file:///c:/oeps.txt")) +--~ print(url.filename("c:/oeps.txt")) +--~ print(url.filename("file:///oeps.txt")) +--~ print(url.filename("file:///etc/test.txt")) +--~ print(url.filename("/oeps.txt")) + +--~ from the spec on the web (sort of): + +--~ local function test(str) +--~ local t = url.hashed(str) +--~ t.constructed = url.construct(t) +--~ print(table.serialize(t)) +--~ end + +--~ inspect(url.hashed("http://www.pragma-ade.com/test%20test?test=test%20test&x=123%3d45")) +--~ inspect(url.hashed("http://www.pragma-ade.com/test%20test?test=test%20test&x=123%3d45")) + +--~ test("sys:///./colo-rgb") + +--~ test("/data/site/output/q2p-develop/resources/ecaboperception4_res/topicresources/58313733/figuur-cow.jpg") +--~ test("file:///M:/q2p/develop/output/q2p-develop/resources/ecaboperception4_res/topicresources/58313733") +--~ test("M:/q2p/develop/output/q2p-develop/resources/ecaboperception4_res/topicresources/58313733") +--~ test("file:///q2p/develop/output/q2p-develop/resources/ecaboperception4_res/topicresources/58313733") +--~ test("/q2p/develop/output/q2p-develop/resources/ecaboperception4_res/topicresources/58313733") + +--~ test("file:///cow%20with%20spaces") +--~ test("file:///cow%20with%20spaces.pdf") +--~ test("cow%20with%20spaces.pdf") +--~ test("some%20file") +--~ test("/etc/passwords") +--~ test("http://www.myself.com/some%20words.html") +--~ test("file:///c:/oeps.txt") +--~ test("file:///c|/oeps.txt") +--~ test("file:///etc/oeps.txt") +--~ test("file://./etc/oeps.txt") +--~ test("file:////etc/oeps.txt") +--~ test("ftp://ftp.is.co.za/rfc/rfc1808.txt") +--~ test("http://www.ietf.org/rfc/rfc2396.txt") +--~ test("ldap://[2001:db8::7]/c=GB?objectClass?one#what") +--~ test("mailto:John.Doe@example.com") +--~ test("news:comp.infosystems.www.servers.unix") +--~ test("tel:+1-816-555-1212") +--~ test("telnet://192.0.2.16:80/") +--~ test("urn:oasis:names:specification:docbook:dtd:xml:4.1.2") +--~ test("http://www.pragma-ade.com/spaced%20name") + +--~ test("zip:///oeps/oeps.zip#bla/bla.tex") +--~ test("zip:///oeps/oeps.zip?bla/bla.tex") + +--~ table.print(url.hashed("/test?test")) diff --git a/macros/luatex/generic/lualibs/lualibs-util-deb.lua b/macros/luatex/generic/lualibs/lualibs-util-deb.lua new file mode 100644 index 0000000000..bd94b6d01e --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-util-deb.lua @@ -0,0 +1,348 @@ +if not modules then modules = { } end modules ['util-deb'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +-- the <anonymous> tag is kind of generic and used for functions that are not +-- bound to a variable, like node.new, node.copy etc (contrary to for instance +-- node.has_attribute which is bound to a has_attribute local variable in mkiv) + +local type, next, tostring, tonumber = type, next, tostring, tonumber +local format, find, sub, gsub = string.format, string.find, string.sub, string.gsub +local insert, remove, sort = table.insert, table.remove, table.sort +local setmetatableindex = table.setmetatableindex + +utilities = utilities or { } +local debugger = utilities.debugger or { } +utilities.debugger = debugger + +local report = logs.reporter("debugger") + +local ticks = os.gettimeofday or os.clock +local seconds = function(n) return n or 0 end +local overhead = 0 +local dummycalls = 10*1000 +local nesting = 0 +local names = { } + +local initialize = false + +if lua.getpreciseticks then + + initialize = function() + ticks = lua.getpreciseticks + seconds = lua.getpreciseseconds + initialize = false + end + +elseif not (FFISUPPORTED and ffi) then + + -- we have no precise timer + +elseif os.type == "windows" then + + initialize = function() + local kernel = ffilib("kernel32","system") -- no checking + if kernel then + local tonumber = ffi.number or tonumber + ffi.cdef[[ + int QueryPerformanceFrequency(int64_t *lpFrequency); + int QueryPerformanceCounter(int64_t *lpPerformanceCount); + ]] + local target = ffi.new("__int64[1]") + ticks = function() + if kernel.QueryPerformanceCounter(target) == 1 then + return tonumber(target[0]) + else + return 0 + end + end + local target = ffi.new("__int64[1]") + seconds = function(ticks) + if kernel.QueryPerformanceFrequency(target) == 1 then + return ticks / tonumber(target[0]) + else + return 0 + end + end + end + initialize = false + end + +elseif os.type == "unix" then + + -- for the values: echo '#include <time.h>' > foo.h; gcc -dM -E foo.h + + initialize = function() + local C = ffi.C + local tonumber = ffi.number or tonumber + ffi.cdef [[ + /* what a mess */ + typedef int clk_id_t; + typedef enum { CLOCK_REALTIME, CLOCK_MONOTONIC, CLOCK_PROCESS_CPUTIME_ID } clk_id; + typedef struct timespec { long sec; long nsec; } ctx_timespec; + int clock_gettime(clk_id_t timerid, struct timespec *t); + ]] + local target = ffi.new("ctx_timespec[?]",1) + local clock = C.CLOCK_PROCESS_CPUTIME_ID + ticks = function () + C.clock_gettime(clock,target) + return tonumber(target[0].sec*1000000000 + target[0].nsec) + end + seconds = function(ticks) + return ticks/1000000000 + end + initialize = false + end + +end + +setmetatableindex(names,function(t,name) + local v = setmetatableindex(function(t,source) + local v = setmetatableindex(function(t,line) + local v = { total = 0, count = 0, nesting = 0 } + t[line] = v + return v + end) + t[source] = v + return v + end) + t[name] = v + return v +end) + +local getinfo = nil +local sethook = nil + +local function hook(where) + local f = getinfo(2,"nSl") + if f then + local source = f.short_src + if not source then + return + end + local line = f.linedefined or 0 + local name = f.name + if not name then + local what = f.what + if what == "C" then + name = "<anonymous>" + else + name = f.namewhat or what or "<unknown>" + end + end + local data = names[name][source][line] + if where == "call" then + local nesting = data.nesting + if nesting == 0 then + data.count = data.count + 1 + insert(data,ticks()) + data.nesting = 1 + else + data.nesting = nesting + 1 + end + elseif where == "return" then + local nesting = data.nesting + if nesting == 1 then + local t = remove(data) + if t then + data.total = data.total + ticks() - t + end + data.nesting = 0 + else + data.nesting = nesting - 1 + end + end + end +end + +function debugger.showstats(printer,threshold) + local printer = printer or report + local calls = 0 + local functions = 0 + local dataset = { } + local length = 0 + local realtime = 0 + local totaltime = 0 + local threshold = threshold or 0 + for name, sources in next, names do + for source, lines in next, sources do + for line, data in next, lines do + local count = data.count + if count > threshold then + if #name > length then + length = #name + end + local total = data.total + local real = total + if real > 0 then + real = total - (count * overhead / dummycalls) + if real < 0 then + real = 0 + end + realtime = realtime + real + end + totaltime = totaltime + total + if line < 0 then + line = 0 + end + -- if name = "a" then + -- -- weird name + -- end + dataset[#dataset+1] = { real, total, count, name, source, line } + end + end + end + end + sort(dataset,function(a,b) + if a[1] == b[1] then + if a[2] == b[2] then + if a[3] == b[3] then + if a[4] == b[4] then + if a[5] == b[5] then + return a[6] < b[6] + else + return a[5] < b[5] + end + else + return a[4] < b[4] + end + else + return b[3] < a[3] + end + else + return b[2] < a[2] + end + else + return b[1] < a[1] + end + end) + if length > 50 then + length = 50 + end + local fmt = string.formatters["%4.9k s %3.3k %% %4.9k s %3.3k %% %8i # %-" .. length .. "s %4i %s"] + for i=1,#dataset do + local data = dataset[i] + local real = data[1] + local total = data[2] + local count = data[3] + local name = data[4] + local source = data[5] + local line = data[6] + calls = calls + count + functions = functions + 1 + name = gsub(name,"%s+"," ") + if #name > length then + name = sub(name,1,length) + end + printer(fmt(seconds(total),100*total/totaltime,seconds(real),100*real/realtime,count,name,line,source)) + end + printer("") + printer(format("functions : %i", functions)) + printer(format("calls : %i", calls)) + printer(format("overhead : %f", seconds(overhead/1000))) + + -- table.save("luatex-profile.lua",names) +end + +local function getdebug() + if sethook and getinfo then + return + end + if not debug then + local okay + okay, debug = pcall(require,"debug") + end + if type(debug) ~= "table" then + return + end + getinfo = debug.getinfo + sethook = debug.sethook + if type(getinfo) ~= "function" then + getinfo = nil + end + if type(sethook) ~= "function" then + sethook = nil + end +end + +function debugger.savestats(filename,threshold) + local f = io.open(filename,'w') + if f then + debugger.showstats(function(str) f:write(str,"\n") end,threshold) + f:close() + end +end + +function debugger.enable() + getdebug() + if sethook and getinfo and nesting == 0 then + running = true + if initialize then + initialize() + end + sethook(hook,"cr") + local function dummy() end + local t = ticks() + for i=1,dummycalls do + dummy() + end + overhead = ticks() - t + end + if nesting > 0 then + nesting = nesting + 1 + end +end + +function debugger.disable() + if nesting > 0 then + nesting = nesting - 1 + end + if sethook and getinfo and nesting == 0 then + sethook() + end +end + +-- debugger.enable() +-- +-- print(math.sin(1*.5)) +-- print(math.sin(1*.5)) +-- print(math.sin(1*.5)) +-- print(math.sin(1*.5)) +-- print(math.sin(1*.5)) +-- +-- debugger.disable() +-- +-- print("") +-- debugger.showstats() +-- print("") +-- debugger.showstats(print,3) +-- +-- from the lua book: + +local function showtraceback(rep) -- from lua site / adapted + getdebug() + if getinfo then + local level = 2 -- we don't want this function to be reported + local reporter = rep or report + while true do + local info = getinfo(level, "Sl") + if not info then + break + elseif info.what == "C" then + reporter("%2i : %s",level-1,"C function") + else + reporter("%2i : %s : %s",level-1,info.short_src,info.currentline) + end + level = level + 1 + end + end +end + +debugger.showtraceback = showtraceback +-- debug.showtraceback = showtraceback + +-- showtraceback() diff --git a/macros/luatex/generic/lualibs/lualibs-util-dim.lua b/macros/luatex/generic/lualibs/lualibs-util-dim.lua new file mode 100644 index 0000000000..76b5fd20bb --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-util-dim.lua @@ -0,0 +1,456 @@ +if not modules then modules = { } end modules ['util-dim'] = { + version = 1.001, + comment = "support for dimensions", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +--[[ldx-- +<p>Internally <l n='luatex'/> work with scaled point, which are +represented by integers. However, in practice, at east at the +<l n='tex'/> end we work with more generic units like points (pt). Going +from scaled points (numbers) to one of those units can be +done by using the conversion factors collected in the following +table.</p> +--ldx]]-- + +local format, match, gsub, type, setmetatable = string.format, string.match, string.gsub, type, setmetatable +local P, S, R, Cc, C, lpegmatch = lpeg.P, lpeg.S, lpeg.R, lpeg.Cc, lpeg.C, lpeg.match + +local allocate = utilities.storage.allocate +local setmetatableindex = table.setmetatableindex +local formatters = string.formatters + +local texget = tex and tex.get or function() return 65536*10*100 end + +local p_stripzeros = lpeg.patterns.stripzeros + +--this might become another namespace + +number = number or { } +local number = number + +number.tonumberf = function(n) return lpegmatch(p_stripzeros,format("%.20f",n)) end +number.tonumberg = function(n) return format("%.20g",n) end + +local dimenfactors = allocate { + ["pt"] = 1/65536, + ["in"] = ( 100/ 7227)/65536, + ["cm"] = ( 254/ 7227)/65536, + ["mm"] = ( 2540/ 7227)/65536, + ["sp"] = 1, -- 65536 sp in 1pt + ["bp"] = ( 7200/ 7227)/65536, + ["pc"] = ( 1/ 12)/65536, + ["dd"] = ( 1157/ 1238)/65536, + ["cc"] = ( 1157/14856)/65536, + ["nd"] = (20320/21681)/65536, + ["nc"] = ( 5080/65043)/65536 +} + +-- print(table.serialize(dimenfactors)) +-- +-- %.99g: +-- +-- t={ +-- ["bp"]=1.5201782378580324e-005, +-- ["cc"]=1.1883696112892098e-006, +-- ["cm"]=5.3628510057769479e-007, +-- ["dd"]=1.4260435335470516e-005, +-- ["em"]=0.000152587890625, +-- ["ex"]=6.103515625e-005, +-- ["in"]=2.1113586636917117e-007, +-- ["mm"]=5.3628510057769473e-008, +-- ["nc"]=1.1917446679504327e-006, +-- ["nd"]=1.4300936015405194e-005, +-- ["pc"]=1.2715657552083333e-006, +-- ["pt"]=1.52587890625e-005, +-- ["sp"]=1, +-- } +-- +-- patched %s and tonumber +-- +-- t={ +-- ["bp"]=0.00001520178238, +-- ["cc"]=0.00000118836961, +-- ["cm"]=0.0000005362851, +-- ["dd"]=0.00001426043534, +-- ["em"]=0.00015258789063, +-- ["ex"]=0.00006103515625, +-- ["in"]=0.00000021113587, +-- ["mm"]=0.00000005362851, +-- ["nc"]=0.00000119174467, +-- ["nd"]=0.00001430093602, +-- ["pc"]=0.00000127156576, +-- ["pt"]=0.00001525878906, +-- ["sp"]=1, +-- } + +--[[ldx-- +<p>A conversion function that takes a number, unit (string) and optional +format (string) is implemented using this table.</p> +--ldx]]-- + +local f_none = formatters["%s%s"] +local f_true = formatters["%0.5F%s"] + +local function numbertodimen(n,unit,fmt) -- will be redefined later ! + if type(n) == 'string' then + return n + else + unit = unit or 'pt' + n = n * dimenfactors[unit] + if not fmt then + fmt = f_none(n,unit) + elseif fmt == true then + fmt = f_true(n,unit) + else + return formatters[fmt](n,unit) + end + end +end + +--[[ldx-- +<p>We collect a bunch of converters in the <type>number</type> namespace.</p> +--ldx]]-- + +number.maxdimen = 1073741823 +number.todimen = numbertodimen +number.dimenfactors = dimenfactors + +function number.topoints (n,fmt) return numbertodimen(n,"pt",fmt) end +function number.toinches (n,fmt) return numbertodimen(n,"in",fmt) end +function number.tocentimeters (n,fmt) return numbertodimen(n,"cm",fmt) end +function number.tomillimeters (n,fmt) return numbertodimen(n,"mm",fmt) end +function number.toscaledpoints(n,fmt) return numbertodimen(n,"sp",fmt) end +function number.toscaledpoints(n) return n .. "sp" end +function number.tobasepoints (n,fmt) return numbertodimen(n,"bp",fmt) end +function number.topicas (n,fmt) return numbertodimen(n "pc",fmt) end +function number.todidots (n,fmt) return numbertodimen(n,"dd",fmt) end +function number.tociceros (n,fmt) return numbertodimen(n,"cc",fmt) end +function number.tonewdidots (n,fmt) return numbertodimen(n,"nd",fmt) end +function number.tonewciceros (n,fmt) return numbertodimen(n,"nc",fmt) end + +--[[ldx-- +<p>More interesting it to implement a (sort of) dimen datatype, one +that permits calculations too. First we define a function that +converts a string to scaledpoints. We use <l n='lpeg'/>. We capture +a number and optionally a unit. When no unit is given a constant +capture takes place.</p> +--ldx]]-- + +local amount = (S("+-")^0 * R("09")^0 * P(".")^0 * R("09")^0) + Cc("0") +local unit = R("az")^1 + P("%") + +local dimenpair = amount/tonumber * (unit^1/dimenfactors + Cc(1)) -- tonumber is new + +lpeg.patterns.dimenpair = dimenpair + +local splitter = amount/tonumber * C(unit^1) + +function number.splitdimen(str) + return lpegmatch(splitter,str) +end + +--[[ldx-- +<p>We use a metatable to intercept errors. When no key is found in +the table with factors, the metatable will be consulted for an +alternative index function.</p> +--ldx]]-- + +setmetatableindex(dimenfactors, function(t,s) + -- error("wrong dimension: " .. (s or "?")) -- better a message + return false +end) + +--[[ldx-- +<p>We redefine the following function later on, so we comment it +here (which saves us bytecodes.</p> +--ldx]]-- + +-- function string.todimen(str) +-- if type(str) == "number" then +-- return str +-- else +-- local value, unit = lpegmatch(dimenpair,str) +-- return value/unit +-- end +-- end +-- +-- local stringtodimen = string.todimen + +local stringtodimen -- assigned later (commenting saves bytecode) + +local amount = S("+-")^0 * R("09")^0 * S(".,")^0 * R("09")^0 +local unit = P("pt") + P("cm") + P("mm") + P("sp") + P("bp") + P("in") + + P("pc") + P("dd") + P("cc") + P("nd") + P("nc") + +local validdimen = amount * unit + +lpeg.patterns.validdimen = validdimen + +--[[ldx-- +<p>This converter accepts calls like:</p> + +<typing> +string.todimen("10") +string.todimen(".10") +string.todimen("10.0") +string.todimen("10.0pt") +string.todimen("10pt") +string.todimen("10.0pt") +</typing> + +<p>With this in place, we can now implement a proper datatype for dimensions, one +that permits us to do this:</p> + +<typing> +s = dimen "10pt" + dimen "20pt" + dimen "200pt" + - dimen "100sp" / 10 + "20pt" + "0pt" +</typing> + +<p>We create a local metatable for this new type:</p> +--ldx]]-- + +local dimensions = { } + +--[[ldx-- +<p>The main (and globally) visible representation of a dimen is defined next: it is +a one-element table. The unit that is returned from the match is normally a number +(one of the previously defined factors) but we also accept functions. Later we will +see why. This function is redefined later.</p> +--ldx]]-- + +-- function dimen(a) +-- if a then +-- local ta= type(a) +-- if ta == "string" then +-- local value, unit = lpegmatch(pattern,a) +-- if type(unit) == "function" then +-- k = value/unit() +-- else +-- k = value/unit +-- end +-- a = k +-- elseif ta == "table" then +-- a = a[1] +-- end +-- return setmetatable({ a }, dimensions) +-- else +-- return setmetatable({ 0 }, dimensions) +-- end +-- end + +--[[ldx-- +<p>This function return a small hash with a metatable attached. It is +through this metatable that we can do the calculations. We could have +shared some of the code but for reasons of speed we don't.</p> +--ldx]]-- + +function dimensions.__add(a, b) + local ta, tb = type(a), type(b) + if ta == "string" then a = stringtodimen(a) elseif ta == "table" then a = a[1] end + if tb == "string" then b = stringtodimen(b) elseif tb == "table" then b = b[1] end + return setmetatable({ a + b }, dimensions) +end + +function dimensions.__sub(a, b) + local ta, tb = type(a), type(b) + if ta == "string" then a = stringtodimen(a) elseif ta == "table" then a = a[1] end + if tb == "string" then b = stringtodimen(b) elseif tb == "table" then b = b[1] end + return setmetatable({ a - b }, dimensions) +end + +function dimensions.__mul(a, b) + local ta, tb = type(a), type(b) + if ta == "string" then a = stringtodimen(a) elseif ta == "table" then a = a[1] end + if tb == "string" then b = stringtodimen(b) elseif tb == "table" then b = b[1] end + return setmetatable({ a * b }, dimensions) +end + +function dimensions.__div(a, b) + local ta, tb = type(a), type(b) + if ta == "string" then a = stringtodimen(a) elseif ta == "table" then a = a[1] end + if tb == "string" then b = stringtodimen(b) elseif tb == "table" then b = b[1] end + return setmetatable({ a / b }, dimensions) +end + +function dimensions.__unm(a) + local ta = type(a) + if ta == "string" then a = stringtodimen(a) elseif ta == "table" then a = a[1] end + return setmetatable({ - a }, dimensions) +end + +--[[ldx-- +<p>It makes no sense to implement the power and modulo function but +the next two do make sense because they permits is code like:</p> + +<typing> +local a, b = dimen "10pt", dimen "11pt" +... +if a > b then + ... +end +</typing> +--ldx]]-- + +-- makes no sense: dimensions.__pow and dimensions.__mod + +function dimensions.__lt(a, b) + return a[1] < b[1] +end + +function dimensions.__eq(a, b) + return a[1] == b[1] +end + +--[[ldx-- +<p>We also need to provide a function for conversion to string (so that +we can print dimensions). We print them as points, just like <l n='tex'/>.</p> +--ldx]]-- + +function dimensions.__tostring(a) + return a[1]/65536 .. "pt" -- instead of todimen(a[1]) +end + +--[[ldx-- +<p>Since it does not take much code, we also provide a way to access +a few accessors</p> + +<typing> +print(dimen().pt) +print(dimen().sp) +</typing> +--ldx]]-- + +function dimensions.__index(tab,key) + local d = dimenfactors[key] + if not d then + error("illegal property of dimen: " .. key) + d = 1 + end + return 1/d +end + +--[[ldx-- +<p>In the converter from string to dimension we support functions as +factors. This is because in <l n='tex'/> we have a few more units: +<type>ex</type> and <type>em</type>. These are not constant factors but +depend on the current font. They are not defined by default, but need +an explicit function call. This is because at the moment that this code +is loaded, the relevant tables that hold the functions needed may not +yet be available.</p> +--ldx]]-- + + dimenfactors["ex"] = 4 * 1/65536 -- 4pt + dimenfactors["em"] = 10 * 1/65536 -- 10pt +-- dimenfactors["%"] = 4 * 1/65536 -- 400pt/100 + +--[[ldx-- +<p>The previous code is rather efficient (also thanks to <l n='lpeg'/>) but we +can speed it up by caching converted dimensions. On my machine (2008) the following +loop takes about 25.5 seconds.</p> + +<typing> +for i=1,1000000 do + local s = dimen "10pt" + dimen "20pt" + dimen "200pt" + - dimen "100sp" / 10 + "20pt" + "0pt" +end +</typing> + +<p>When we cache converted strings this becomes 16.3 seconds. In order not +to waste too much memory on it, we tag the values of the cache as being +week which mean that the garbage collector will collect them in a next +sweep. This means that in most cases the speed up is mostly affecting the +current couple of calculations and as such the speed penalty is small.</p> + +<p>We redefine two previous defined functions that can benefit from +this:</p> +--ldx]]-- + +local known = { } setmetatable(known, { __mode = "v" }) + +function dimen(a) + if a then + local ta= type(a) + if ta == "string" then + local k = known[a] + if k then + a = k + else + local value, unit = lpegmatch(dimenpair,a) + if value and unit then + k = value/unit -- to be considered: round + else + k = 0 + end + known[a] = k + a = k + end + elseif ta == "table" then + a = a[1] + end + return setmetatable({ a }, dimensions) + else + return setmetatable({ 0 }, dimensions) + end +end + +function string.todimen(str) -- maybe use tex.sp when available + local t = type(str) + if t == "number" then + return str + else + local k = known[str] + if not k then + if t == "string" then + local value, unit = lpegmatch(dimenpair,str) + if value and unit then + k = value/unit -- to be considered: round + else + k = 0 + end + else + k = 0 + end + known[str] = k + end + return k + end +end + +-- local known = { } +-- +-- function string.todimen(str) -- maybe use tex.sp +-- local k = known[str] +-- if not k then +-- k = tex.sp(str) +-- known[str] = k +-- end +-- return k +-- end + +stringtodimen = string.todimen -- local variable defined earlier + +function number.toscaled(d) + return format("%0.5f",d/0x10000) -- 2^16 +end + +--[[ldx-- +<p>In a similar fashion we can define a glue datatype. In that case we +probably use a hash instead of a one-element table.</p> +--ldx]]-- + +--[[ldx-- +<p>Goodie:s</p> +--ldx]]-- + +function number.percent(n,d) -- will be cleaned up once luatex 0.30 is out + d = d or texget("hsize") + if type(d) == "string" then + d = stringtodimen(d) + end + return (n/100) * d +end + +number["%"] = number.percent diff --git a/macros/luatex/generic/lualibs/lualibs-util-fil.lua b/macros/luatex/generic/lualibs/lualibs-util-fil.lua new file mode 100644 index 0000000000..d568a2e1fd --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-util-fil.lua @@ -0,0 +1,382 @@ +if not modules then modules = { } end modules ['util-fil'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +local byte = string.byte +local char = string.char + +-- Here are a few helpers (the starting point were old ones I used for parsing +-- flac files). In Lua 5.3 we can probably do this better. Some code will move +-- here. + +utilities = utilities or { } +local files = { } +utilities.files = files + +local zerobased = { } + +function files.open(filename,zb) + local f = io.open(filename,"rb") + if f then + zerobased[f] = zb or false + end + return f +end + +function files.close(f) + zerobased[f] = nil + f:close() +end + +function files.size(f) + local current = f:seek() + local size = f:seek("end") + f:seek("set",current) + return size +end + +files.getsize = files.size + +function files.setposition(f,n) + if zerobased[f] then + f:seek("set",n) + else + f:seek("set",n - 1) + end +end + +function files.getposition(f) + if zerobased[f] then + return f:seek() + else + return f:seek() + 1 + end +end + +function files.look(f,n,chars) + local p = f:seek() + local s = f:read(n) + f:seek("set",p) + if chars then + return s + else + return byte(s,1,#s) + end +end + +function files.skip(f,n) + if n == 1 then + f:read(n) + else + f:seek("set",f:seek()+n) + end +end + +function files.readbyte(f) + return byte(f:read(1)) +end + +function files.readbytes(f,n) + return byte(f:read(n),1,n) +end + +function files.readbytetable(f,n) + -- return { byte(f:read(n),1,n) } + local s = f:read(n or 1) + return { byte(s,1,#s) } -- best use the real length +end + +function files.readchar(f) + return f:read(1) +end + +function files.readstring(f,n) + return f:read(n or 1) +end + +function files.readinteger1(f) -- one byte + local n = byte(f:read(1)) + if n >= 0x80 then + return n - 0x100 + else + return n + end +end + +files.readcardinal1 = files.readbyte -- one byte +files.readcardinal = files.readcardinal1 +files.readinteger = files.readinteger1 +files.readsignedbyte = files.readinteger1 + +function files.readcardinal2(f) + local a, b = byte(f:read(2),1,2) + return 0x100 * a + b +end + +function files.readcardinal2le(f) + local b, a = byte(f:read(2),1,2) + return 0x100 * a + b +end + +function files.readinteger2(f) + local a, b = byte(f:read(2),1,2) + if a >= 0x80 then + return 0x100 * a + b - 0x10000 + else + return 0x100 * a + b + end +end + +function files.readinteger2le(f) + local b, a = byte(f:read(2),1,2) + if a >= 0x80 then + return 0x100 * a + b - 0x10000 + else + return 0x100 * a + b + end +end + +function files.readcardinal3(f) + local a, b, c = byte(f:read(3),1,3) + return 0x10000 * a + 0x100 * b + c +end + +function files.readcardinal3le(f) + local c, b, a = byte(f:read(3),1,3) + return 0x10000 * a + 0x100 * b + c +end + +function files.readinteger3(f) + local a, b, c = byte(f:read(3),1,3) + if a >= 0x80 then + return 0x10000 * a + 0x100 * b + c - 0x1000000 + else + return 0x10000 * a + 0x100 * b + c + end +end + +function files.readinteger3le(f) + local c, b, a = byte(f:read(3),1,3) + if a >= 0x80 then + return 0x10000 * a + 0x100 * b + c - 0x1000000 + else + return 0x10000 * a + 0x100 * b + c + end +end + +function files.readcardinal4(f) + local a, b, c, d = byte(f:read(4),1,4) + return 0x1000000 * a + 0x10000 * b + 0x100 * c + d +end + +function files.readcardinal4le(f) + local d, c, b, a = byte(f:read(4),1,4) + return 0x1000000 * a + 0x10000 * b + 0x100 * c + d +end + +function files.readinteger4(f) + local a, b, c, d = byte(f:read(4),1,4) + if a >= 0x80 then + return 0x1000000 * a + 0x10000 * b + 0x100 * c + d - 0x100000000 + else + return 0x1000000 * a + 0x10000 * b + 0x100 * c + d + end +end + +function files.readinteger4le(f) + local d, c, b, a = byte(f:read(4),1,4) + if a >= 0x80 then + return 0x1000000 * a + 0x10000 * b + 0x100 * c + d - 0x100000000 + else + return 0x1000000 * a + 0x10000 * b + 0x100 * c + d + end +end + +-- function files.readfixed2(f) +-- local a, b = byte(f:read(2),1,2) +-- if a >= 0x80 then +-- return (0x100 * a + b - 0x10000)/256.0 +-- else +-- return (0x100 * a + b)/256.0 +-- end +-- end + +function files.readfixed2(f) + local a, b = byte(f:read(2),1,2) + if a >= 0x80 then + return (a - 0x100) + b/0x100 + else + return (a ) + b/0x100 + end +end + +-- (real) (n>>16) + ((n&0xffff)/65536.0)) + +-- function files.readfixed4(f) +-- local a, b, c, d = byte(f:read(4),1,4) +-- if a >= 0x80 then +-- return (0x1000000 * a + 0x10000 * b + 0x100 * c + d - 0x100000000)/65536.0 +-- else +-- return (0x1000000 * a + 0x10000 * b + 0x100 * c + d)/65536.0 +-- end +-- end + +function files.readfixed4(f) + local a, b, c, d = byte(f:read(4),1,4) + if a >= 0x80 then + return (0x100 * a + b - 0x10000) + (0x100 * c + d)/0x10000 + else + return (0x100 * a + b ) + (0x100 * c + d)/0x10000 + end +end + +-- (real) ((n<<16)>>(16+14)) + ((n&0x3fff)/16384.0)) + +if bit32 then + + local extract = bit32.extract + local band = bit32.band + + function files.read2dot14(f) + local a, b = byte(f:read(2),1,2) + if a >= 0x80 then + local n = -(0x100 * a + b) + return - (extract(n,14,2) + (band(n,0x3FFF) / 16384.0)) + else + local n = 0x100 * a + b + return (extract(n,14,2) + (band(n,0x3FFF) / 16384.0)) + end + end + +end + +function files.skipshort(f,n) + f:read(2*(n or 1)) +end + +function files.skiplong(f,n) + f:read(4*(n or 1)) +end + +-- writers (kind of slow) + +if bit32 then + + local rshift = bit32.rshift + + function files.writecardinal2(f,n) + local a = char(n % 256) + n = rshift(n,8) + local b = char(n % 256) + f:write(b,a) + end + +else + + local floor = math.floor + + function files.writecardinal2(f,n) + local a = char(n % 256) + n = floor(n/256) + local b = char(n % 256) + f:write(b,a) + end + +end + +function files.writecardinal4(f,n) + local a = char(n % 256) + n = rshift(n,8) + local b = char(n % 256) + n = rshift(n,8) + local c = char(n % 256) + n = rshift(n,8) + local d = char(n % 256) + f:write(d,c,b,a) +end + +function files.writestring(f,s) + f:write(char(byte(s,1,#s))) +end + +function files.writebyte(f,b) + f:write(char(b)) +end + +if fio and fio.readcardinal1 then + + files.readcardinal1 = fio.readcardinal1 + files.readcardinal2 = fio.readcardinal2 + files.readcardinal3 = fio.readcardinal3 + files.readcardinal4 = fio.readcardinal4 + files.readinteger1 = fio.readinteger1 + files.readinteger2 = fio.readinteger2 + files.readinteger3 = fio.readinteger3 + files.readinteger4 = fio.readinteger4 + files.readfixed2 = fio.readfixed2 + files.readfixed4 = fio.readfixed4 + files.read2dot14 = fio.read2dot14 + files.setposition = fio.setposition + files.getposition = fio.getposition + + files.readbyte = files.readcardinal1 + files.readsignedbyte = files.readinteger1 + files.readcardinal = files.readcardinal1 + files.readinteger = files.readinteger1 + + local skipposition = fio.skipposition + files.skipposition = skipposition + + files.readbytes = fio.readbytes + files.readbytetable = fio.readbytetable + + function files.skipshort(f,n) + skipposition(f,2*(n or 1)) + end + + function files.skiplong(f,n) + skipposition(f,4*(n or 1)) + end + +end + +if fio and fio.readcardinaltable then + + files.readcardinaltable = fio.readcardinaltable + files.readintegertable = fio.readintegertable + +else + + local readcardinal1 = files.readcardinal1 + local readcardinal2 = files.readcardinal2 + local readcardinal3 = files.readcardinal3 + local readcardinal4 = files.readcardinal4 + + function files.readcardinaltable(f,n,b) + local t = { } + if b == 1 then for i=1,n do t[i] = readcardinal1(f) end + elseif b == 2 then for i=1,n do t[i] = readcardinal2(f) end + elseif b == 3 then for i=1,n do t[i] = readcardinal3(f) end + elseif b == 4 then for i=1,n do t[i] = readcardinal4(f) end end + return t + end + + local readinteger1 = files.readinteger1 + local readinteger2 = files.readinteger2 + local readinteger3 = files.readinteger3 + local readinteger4 = files.readinteger4 + + function files.readintegertable(f,n,b) + local t = { } + if b == 1 then for i=1,n do t[i] = readinteger1(f) end + elseif b == 2 then for i=1,n do t[i] = readinteger2(f) end + elseif b == 3 then for i=1,n do t[i] = readinteger3(f) end + elseif b == 4 then for i=1,n do t[i] = readinteger4(f) end end + return t + end + +end diff --git a/macros/luatex/generic/lualibs/lualibs-util-jsn.lua b/macros/luatex/generic/lualibs/lualibs-util-jsn.lua new file mode 100644 index 0000000000..8da3518979 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-util-jsn.lua @@ -0,0 +1,443 @@ +if not modules then modules = { } end modules ['util-jsn'] = { + version = 1.001, + comment = "companion to m-json.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +-- Of course we could make a nice complete parser with proper error messages but +-- as json is generated programmatically errors are systematic and we can assume +-- a correct stream. If not, we have some fatal error anyway. So, we can just rely +-- on strings being strings (apart from the unicode escape which is not in 5.1) and +-- as we first catch known types we just assume that anything else is a number. +-- +-- Reminder for me: check usage in framework and extend when needed. Also document +-- it in the cld lib documentation. +-- +-- Upgraded for handling the somewhat more fax server templates. + +local P, V, R, S, C, Cc, Cs, Ct, Cf, Cg = lpeg.P, lpeg.V, lpeg.R, lpeg.S, lpeg.C, lpeg.Cc, lpeg.Cs, lpeg.Ct, lpeg.Cf, lpeg.Cg +local lpegmatch = lpeg.match +local format, gsub = string.format, string.gsub +local formatters = string.formatters +local utfchar = utf.char +local concat, sortedkeys = table.concat, table.sortedkeys + +local tonumber, tostring, rawset, type, next = tonumber, tostring, rawset, type, next + +local json = utilities.json or { } +utilities.json = json + +do + + -- \\ \/ \b \f \n \r \t \uHHHH + + local lbrace = P("{") + local rbrace = P("}") + local lparent = P("[") + local rparent = P("]") + local comma = P(",") + local colon = P(":") + local dquote = P('"') + + local whitespace = lpeg.patterns.whitespace + local optionalws = whitespace^0 + + local escapes = { + ["b"] = "\010", + ["f"] = "\014", + ["n"] = "\n", + ["r"] = "\r", + ["t"] = "\t", + } + + -- todo: also handle larger utf16 + + local escape_un = P("\\u")/"" * (C(R("09","AF","af")^-4) / function(s) + return utfchar(tonumber(s,16)) + end) + + local escape_bs = P([[\]]) / "" * (P(1) / escapes) -- if not found then P(1) is returned i.e. the to be escaped char + + local jstring = dquote * Cs((escape_un + escape_bs + (1-dquote))^0) * dquote + local jtrue = P("true") * Cc(true) + local jfalse = P("false") * Cc(false) + local jnull = P("null") * Cc(nil) + local jnumber = (1-whitespace-rparent-rbrace-comma)^1 / tonumber + + local key = jstring + + local jsonconverter = { "value", + hash = lbrace * Cf(Ct("") * (V("pair") * (comma * V("pair"))^0 + optionalws),rawset) * rbrace, + pair = Cg(optionalws * key * optionalws * colon * V("value")), + array = Ct(lparent * (V("value") * (comma * V("value"))^0 + optionalws) * rparent), + -- value = optionalws * (jstring + V("hash") + V("array") + jtrue + jfalse + jnull + jnumber + #rparent) * optionalws, + value = optionalws * (jstring + V("hash") + V("array") + jtrue + jfalse + jnull + jnumber) * optionalws, + } + + -- local jsonconverter = { "value", + -- hash = lbrace * Cf(Ct("") * (V("pair") * (comma * V("pair"))^0 + optionalws),rawset) * rbrace, + -- pair = Cg(optionalws * V("string") * optionalws * colon * V("value")), + -- array = Ct(lparent * (V("value") * (comma * V("value"))^0 + optionalws) * rparent), + -- string = jstring, + -- value = optionalws * (V("string") + V("hash") + V("array") + jtrue + jfalse + jnull + jnumber) * optionalws, + -- } + + -- lpeg.print(jsonconverter) -- size 181 + + function json.tolua(str) + return lpegmatch(jsonconverter,str) + end + + function json.load(filename) + local data = io.loaddata(filename) + if data then + return lpegmatch(jsonconverter,data) + end + end + +end + +do + + -- It's pretty bad that JSON doesn't allow the trailing comma ... it's a + -- typical example of a spec that then forces all generators to check for + -- this. It's a way to make sure programmers keep jobs. + + local escaper + + local f_start_hash = formatters[ '%w{' ] + local f_start_array = formatters[ '%w[' ] + local f_start_hash_new = formatters[ "\n" .. '%w{' ] + local f_start_array_new = formatters[ "\n" .. '%w[' ] + local f_start_hash_key = formatters[ "\n" .. '%w"%s" : {' ] + local f_start_array_key = formatters[ "\n" .. '%w"%s" : [' ] + + local f_stop_hash = formatters[ "\n" .. '%w}' ] + local f_stop_array = formatters[ "\n" .. '%w]' ] + + local f_key_val_seq = formatters[ "\n" .. '%w"%s" : %s' ] + local f_key_val_str = formatters[ "\n" .. '%w"%s" : "%s"' ] + local f_key_val_num = f_key_val_seq + local f_key_val_yes = formatters[ "\n" .. '%w"%s" : true' ] + local f_key_val_nop = formatters[ "\n" .. '%w"%s" : false' ] + local f_key_val_null = formatters[ "\n" .. '%w"%s" : null' ] + + local f_val_num = formatters[ "\n" .. '%w%s' ] + local f_val_str = formatters[ "\n" .. '%w"%s"' ] + local f_val_yes = formatters[ "\n" .. '%wtrue' ] + local f_val_nop = formatters[ "\n" .. '%wfalse' ] + local f_val_null = formatters[ "\n" .. '%wnull' ] + local f_val_empty = formatters[ "\n" .. '%w{ }' ] + local f_val_seq = f_val_num + + -- no empty tables because unknown if table or hash + + local t = { } + local n = 0 + + local function is_simple_table(tt) -- also used in util-tab so maybe public + local l = #tt + if l > 0 then + for i=1,l do + if type(tt[i]) == "table" then + return false + end + end + local nn = n + n = n + 1 t[n] = "[ " + for i=1,l do + if i > 1 then + n = n + 1 t[n] = ", " + end + local v = tt[i] + local tv = type(v) + if tv == "number" then + n = n + 1 t[n] = v + elseif tv == "string" then + n = n + 1 t[n] = '"' + n = n + 1 t[n] = lpegmatch(escaper,v) or v + n = n + 1 t[n] = '"' + elseif tv == "boolean" then + n = n + 1 t[n] = v and "true" or "false" + elseif v then + n = n + 1 t[n] = tostring(v) + else + n = n + 1 t[n] = "null" + end + end + n = n + 1 t[n] = " ]" + local s = concat(t,"",nn+1,n) + n = nn + return s + end + return false + end + + local function tojsonpp(root,name,depth,level,size) + if root then + local indexed = size > 0 + n = n + 1 + if level == 0 then + if indexed then + t[n] = f_start_array(depth) + else + t[n] = f_start_hash(depth) + end + elseif name then + if tn == "string" then + name = lpegmatch(escaper,name) or name + elseif tn ~= "number" then + name = tostring(name) + end + if indexed then + t[n] = f_start_array_key(depth,name) + else + t[n] = f_start_hash_key(depth,name) + end + else + if indexed then + t[n] = f_start_array_new(depth) + else + t[n] = f_start_hash_new(depth) + end + end + depth = depth + 1 + if indexed then -- indexed + for i=1,size do + if i > 1 then + n = n + 1 t[n] = "," + end + local v = root[i] + local tv = type(v) + if tv == "number" then + n = n + 1 t[n] = f_val_num(depth,v) + elseif tv == "string" then + v = lpegmatch(escaper,v) or v + n = n + 1 t[n] = f_val_str(depth,v) + elseif tv == "table" then + if next(v) then + local st = is_simple_table(v) + if st then + n = n + 1 t[n] = f_val_seq(depth,st) + else + tojsonpp(v,nil,depth,level+1,#v) + end + else + n = n + 1 + t[n] = f_val_empty(depth) + end + elseif tv == "boolean" then + n = n + 1 + if v then + t[n] = f_val_yes(depth,v) + else + t[n] = f_val_nop(depth,v) + end + else + n = n + 1 + t[n] = f_val_null(depth) + end + end + elseif next(root) then + local sk = sortedkeys(root) + for i=1,#sk do + if i > 1 then + n = n + 1 t[n] = "," + end + local k = sk[i] + local v = root[k] + local tv = type(v) + local tk = type(k) + if tv == "number" then + if tk == "number" then + n = n + 1 t[n] = f_key_val_num(depth,k,v) + elseif tk == "string" then + k = lpegmatch(escaper,k) or k + n = n + 1 t[n] = f_key_val_num(depth,k,v) + end + elseif tv == "string" then + if tk == "number" then + v = lpegmatch(escaper,v) or v + n = n + 1 t[n] = f_key_val_str(depth,k,v) + elseif tk == "string" then + k = lpegmatch(escaper,k) or k + v = lpegmatch(escaper,v) or v + n = n + 1 t[n] = f_key_val_str(depth,k,v) + end + elseif tv == "table" then + local l = #v + if l > 0 then + local st = is_simple_table(v) + if not st then + tojsonpp(v,k,depth,level+1,l) + elseif tk == "number" then + n = n + 1 t[n] = f_key_val_seq(depth,k,st) + elseif tk == "string" then + k = lpegmatch(escaper,k) or k + n = n + 1 t[n] = f_key_val_seq(depth,k,st) + end + elseif next(v) then + tojsonpp(v,k,depth,level+1,0) + end + elseif tv == "boolean" then + if tk == "number" then + n = n + 1 + if v then + t[n] = f_key_val_yes(depth,k) + else + t[n] = f_key_val_nop(depth,k) + end + elseif tk == "string" then + k = lpegmatch(escaper,k) or k + n = n + 1 + if v then + t[n] = f_key_val_yes(depth,k) + else + t[n] = f_key_val_nop(depth,k) + end + end + else + if tk == "number" then + n = n + 1 + t[n] = f_key_val_null(depth,k) + elseif tk == "string" then + k = lpegmatch(escaper,k) or k + n = n + 1 + t[n] = f_key_val_null(depth,k) + end + end + end + end + n = n + 1 + if indexed then + t[n] = f_stop_array(depth-1) + else + t[n] = f_stop_hash(depth-1) + end + end + end + + local function tojson(value,n) + local kind = type(value) + if kind == "table" then + local done = false + local size = #value + if size == 0 then + for k, v in next, value do + if done then + -- n = n + 1 ; t[n] = "," + n = n + 1 ; t[n] = ',"' + else + -- n = n + 1 ; t[n] = "{" + n = n + 1 ; t[n] = '{"' + done = true + end + n = n + 1 ; t[n] = lpegmatch(escaper,k) or k + n = n + 1 ; t[n] = '":' + t, n = tojson(v,n) + end + if done then + n = n + 1 ; t[n] = "}" + else + n = n + 1 ; t[n] = "{}" + end + elseif size == 1 then + -- we can optimize for non tables + n = n + 1 ; t[n] = "[" + t, n = tojson(value[1],n) + n = n + 1 ; t[n] = "]" + else + for i=1,size do + if done then + n = n + 1 ; t[n] = "," + else + n = n + 1 ; t[n] = "[" + done = true + end + t, n = tojson(value[i],n) + end + n = n + 1 ; t[n] = "]" + end + elseif kind == "string" then + n = n + 1 ; t[n] = '"' + n = n + 1 ; t[n] = lpegmatch(escaper,value) or value + n = n + 1 ; t[n] = '"' + elseif kind == "number" then + n = n + 1 ; t[n] = value + elseif kind == "boolean" then + n = n + 1 ; t[n] = tostring(value) + else + n = n + 1 ; t[n] = "null" + end + return t, n + end + + -- escaping keys can become an option + + local function jsontostring(value,pretty) + -- todo optimize for non table + local kind = type(value) + if kind == "table" then + if not escaper then + local escapes = { + ["\\"] = "\\u005C", + ["\""] = "\\u0022", + } + for i=0,0x1F do + escapes[utfchar(i)] = format("\\u%04X",i) + end + escaper = Cs( ( + (R('\0\x20') + S('\"\\')) / escapes + + P(1) + )^1 ) + + end + -- local to the closure (saves wrapping and local functions) + t = { } + n = 0 + if pretty then + tojsonpp(value,name,0,0,#value) + value = concat(t,"",1,n) + else + t, n = tojson(value,0) + value = concat(t,"",1,n) + end + t = nil + n = 0 + return value + elseif kind == "string" or kind == "number" then + return lpegmatch(escaper,value) or value + else + return tostring(value) + end + end + + json.tostring = jsontostring + + function json.tojson(value) + return jsontostring(value,true) + end + +end + +-- local tmp = [[ { "t\nt t" : "foo bar", "a" : true, "b" : [ 123 , 456E-10, { "a" : true, "b" : [ 123 , 456 ] } ] } ]] +-- tmp = json.tolua(tmp) +-- inspect(tmp) +-- tmp = json.tostring(tmp,true) +-- inspect(tmp) +-- tmp = json.tolua(tmp) +-- inspect(tmp) +-- tmp = json.tostring(tmp) +-- inspect(tmp) +-- inspect(json.tostring(true)) + +-- local s = [[\foo"bar"]] +-- local j = json.tostring { s = s } +-- local l = json.tolua(j) +-- inspect(j) +-- inspect(l) +-- print(s==l.s) + +return json diff --git a/macros/luatex/generic/lualibs/lualibs-util-lua.lua b/macros/luatex/generic/lualibs/lualibs-util-lua.lua new file mode 100644 index 0000000000..f7e6e4f313 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-util-lua.lua @@ -0,0 +1,212 @@ +if not modules then modules = { } end modules ['util-lua'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + comment = "the strip code is written by Peter Cawley", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +-- we will remove the 5.1 code some day soon + +local rep, sub, byte, dump, format = string.rep, string.sub, string.byte, string.dump, string.format +local load, loadfile, type, collectgarbage = load, loadfile, type, collectgarbage + +utilities = utilities or {} +utilities.lua = utilities.lua or { } +local luautilities = utilities.lua + +local report_lua = logs.reporter("system","lua") +local report_mem = logs.reporter("system","lua memory") + +local tracestripping = false +local tracememory = false +luautilities.stripcode = true -- support stripping when asked for +luautilities.alwaysstripcode = false -- saves 1 meg on 7 meg compressed format file (2012.08.12) +luautilities.nofstrippedchunks = 0 +luautilities.nofstrippedbytes = 0 +local strippedchunks = { } -- allocate() +luautilities.strippedchunks = strippedchunks + +luautilities.suffixes = { + tma = "tma", + tmc = (CONTEXTLMTXMODE and CONTEXTLMTXMODE > 0 and "tmd") or (jit and "tmb") or "tmc", + lua = "lua", + luc = (CONTEXTLMTXMODE and CONTEXTLMTXMODE > 0 and "lud") or (jit and "lub") or "luc", + lui = "lui", + luv = "luv", + luj = "luj", + tua = "tua", + tuc = "tuc", +} + +-- environment.loadpreprocessedfile can be set to a preprocessor + +local function register(name) -- makes no sense runtime + if tracestripping then + report_lua("stripped bytecode from %a",name or "unknown") + end + strippedchunks[#strippedchunks+1] = name + luautilities.nofstrippedchunks = luautilities.nofstrippedchunks + 1 +end + +local function stupidcompile(luafile,lucfile,strip) + local code = io.loaddata(luafile) + if code and code ~= "" then + code = load(code) + if code then + code = dump(code,strip and luautilities.stripcode or luautilities.alwaysstripcode) + if code and code ~= "" then + register(name) + io.savedata(lucfile,code) + return true, 0 + end + else + report_lua("fatal error %a in file %a",1,luafile) + end + else + report_lua("fatal error %a in file %a",2,luafile) + end + return false, 0 +end + +-- quite subtle ... doing this wrong incidentally can give more bytes + +function luautilities.loadedluacode(fullname,forcestrip,name,macros) + -- quite subtle ... doing this wrong incidentally can give more bytes + name = name or fullname + if macros then + macros = lua.macros + end + local code, message + if macros then + code, message = macros.loaded(fullname,true,false) + else + code, message = loadfile(fullname) + end + if code then + code() + else + report_lua("loading of file %a failed:\n\t%s",fullname,message or "no message") + code, message = loadfile(fullname) + end + if forcestrip and luautilities.stripcode then + if type(forcestrip) == "function" then + forcestrip = forcestrip(fullname) + end + if forcestrip or luautilities.alwaysstripcode then + register(name) + return load(dump(code,true)), 0 + else + return code, 0 + end + elseif luautilities.alwaysstripcode then + register(name) + return load(dump(code,true)), 0 + else + return code, 0 + end +end + +function luautilities.strippedloadstring(code,name,forcestrip) -- not executed + local code, message = load(code) + if not code then + report_lua("loading of file %a failed:\n\t%s",name,message or "no message") + end + if forcestrip and luautilities.stripcode or luautilities.alwaysstripcode then + register(name) + return load(dump(code,true)), 0 -- not yet executed + else + return code, 0 + end +end + +function luautilities.loadstring(code,name) -- not executed + local code, message = load(code) + if not code then + report_lua("loading of file %a failed:\n\t%s",name,message or "no message") + end + return code, 0 +end + +function luautilities.compile(luafile,lucfile,cleanup,strip,fallback) -- defaults: cleanup=false strip=true + report_lua("compiling %a into %a",luafile,lucfile) + os.remove(lucfile) + local done = stupidcompile(luafile,lucfile,strip ~= false) + if done then + report_lua("dumping %a into %a stripped",luafile,lucfile) + if cleanup == true and lfs.isfile(lucfile) and lfs.isfile(luafile) then + report_lua("removing %a",luafile) + os.remove(luafile) + end + end + return done +end + +function luautilities.loadstripped(...) + local l = load(...) + if l then + return load(dump(l,true)) + end +end + +-- local getmetatable, type = getmetatable, type +-- +-- local types = { } +-- +-- function luautilities.registerdatatype(d,name) +-- types[getmetatable(d)] = name +-- end +-- +-- function luautilities.datatype(d) +-- local t = type(d) +-- if t == "userdata" then +-- local m = getmetatable(d) +-- return m and types[m] or "userdata" +-- else +-- return t +-- end +-- end +-- +-- luautilities.registerdatatype(lpeg.P("!"),"lpeg") +-- +-- print(luautilities.datatype(lpeg.P("oeps"))) + +-- These finalizers will only be invoked when we have a proper lua_close +-- call (which is not happening in luatex tex node yes) or finish with an +-- os.exit(n,true). + +local finalizers = { } + +setmetatable(finalizers, { + __gc = function(t) + for i=1,#t do + pcall(t[i]) -- let's not crash + end + end +} ) + +function luautilities.registerfinalizer(f) + finalizers[#finalizers+1] = f +end + +function luautilities.checkmemory(previous,threshold,trace) -- threshold in MB + local current = collectgarbage("count") + if previous then + local checked = (threshold or 64)*1024 + local delta = current - previous + if current - previous > checked then + collectgarbage("collect") + local afterwards = collectgarbage("count") + if trace or tracememory then + report_mem("previous %r MB, current %r MB, delta %r MB, threshold %r MB, afterwards %r MB", + previous/1024,current/1024,delta/1024,threshold,afterwards) + end + return afterwards + elseif trace or tracememory then + report_mem("previous %r MB, current %r MB, delta %r MB, threshold %r MB", + previous/1024,current/1024,delta/1024,threshold) + end + end + return current +end diff --git a/macros/luatex/generic/lualibs/lualibs-util-prs.lua b/macros/luatex/generic/lualibs/lualibs-util-prs.lua new file mode 100644 index 0000000000..3154bf6c28 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-util-prs.lua @@ -0,0 +1,804 @@ +if not modules then modules = { } end modules ['util-prs'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +local lpeg, table, string = lpeg, table, string +local P, R, V, S, C, Ct, Cs, Carg, Cc, Cg, Cf, Cp = lpeg.P, lpeg.R, lpeg.V, lpeg.S, lpeg.C, lpeg.Ct, lpeg.Cs, lpeg.Carg, lpeg.Cc, lpeg.Cg, lpeg.Cf, lpeg.Cp +local lpegmatch, lpegpatterns = lpeg.match, lpeg.patterns +local concat, gmatch, find = table.concat, string.gmatch, string.find +local tostring, type, next, rawset = tostring, type, next, rawset +local mod, div = math.mod, math.div + +utilities = utilities or {} +local parsers = utilities.parsers or { } +utilities.parsers = parsers +local patterns = parsers.patterns or { } +parsers.patterns = patterns + +local setmetatableindex = table.setmetatableindex +local sortedhash = table.sortedhash +local sortedkeys = table.sortedkeys +local tohash = table.tohash + +local hashes = { } +parsers.hashes = hashes +-- we share some patterns + +local digit = R("09") +local space = P(' ') +local equal = P("=") +local colon = P(":") +local comma = P(",") +local lbrace = P("{") +local rbrace = P("}") +local lparent = P("(") +local rparent = P(")") +local lbracket = P("[") +local rbracket = P("]") +local period = S(".") +local punctuation = S(".,:;") +local spacer = lpegpatterns.spacer +local whitespace = lpegpatterns.whitespace +local newline = lpegpatterns.newline +local anything = lpegpatterns.anything +local endofstring = lpegpatterns.endofstring + +local nobrace = 1 - (lbrace + rbrace ) +local noparent = 1 - (lparent + rparent) +local nobracket = 1 - (lbracket + rbracket) + +-- we could use a Cf Cg construct + +local escape, left, right = P("\\"), P('{'), P('}') + +lpegpatterns.balanced = P { + [1] = ((escape * (left+right)) + (1 - (left+right)) + V(2))^0, + [2] = left * V(1) * right +} + +local nestedbraces = P { lbrace * (nobrace + V(1))^0 * rbrace } +local nestedparents = P { lparent * (noparent + V(1))^0 * rparent } +local nestedbrackets = P { lbracket * (nobracket + V(1))^0 * rbracket } +local spaces = space^0 +local argument = Cs((lbrace/"") * ((nobrace + nestedbraces)^0) * (rbrace/"")) +local content = (1-endofstring)^0 + +lpegpatterns.nestedbraces = nestedbraces -- no capture +lpegpatterns.nestedparents = nestedparents -- no capture +lpegpatterns.nested = nestedbraces -- no capture +lpegpatterns.argument = argument -- argument after e.g. = +lpegpatterns.content = content -- rest after e.g = + +local value = lbrace * C((nobrace + nestedbraces)^0) * rbrace + + C((nestedbraces + (1-comma))^0) + +local key = C((1-equal-comma)^1) +local pattern_a = (space+comma)^0 * (key * equal * value + key * C("")) +local pattern_c = (space+comma)^0 * (key * equal * value) +local pattern_d = (space+comma)^0 * (key * (equal+colon) * value + key * C("")) + +local key = C((1-space-equal-comma)^1) +local pattern_b = spaces * comma^0 * spaces * (key * ((spaces * equal * spaces * value) + C(""))) + +-- "a=1, b=2, c=3, d={a{b,c}d}, e=12345, f=xx{a{b,c}d}xx, g={}" : outer {} removes, leading spaces ignored + +local hash = { } + +local function set(key,value) + hash[key] = value +end + +local pattern_a_s = (pattern_a/set)^1 +local pattern_b_s = (pattern_b/set)^1 +local pattern_c_s = (pattern_c/set)^1 +local pattern_d_s = (pattern_d/set)^1 + +patterns.settings_to_hash_a = pattern_a_s +patterns.settings_to_hash_b = pattern_b_s +patterns.settings_to_hash_c = pattern_c_s +patterns.settings_to_hash_d = pattern_d_s + +function parsers.make_settings_to_hash_pattern(set,how) + if how == "strict" then + return (pattern_c/set)^1 + elseif how == "tolerant" then + return (pattern_b/set)^1 + else + return (pattern_a/set)^1 + end +end + +function parsers.settings_to_hash(str,existing) + if not str or str == "" then + return { } + elseif type(str) == "table" then + if existing then + for k, v in next, str do + existing[k] = v + end + return exiting + else + return str + end + else + hash = existing or { } + lpegmatch(pattern_a_s,str) + return hash + end +end + +function parsers.settings_to_hash_colon_too(str) + if not str or str == "" then + return { } + elseif type(str) == "table" then + return str + else + hash = { } + lpegmatch(pattern_d_s,str) + return hash + end +end + +function parsers.settings_to_hash_tolerant(str,existing) + if not str or str == "" then + return { } + elseif type(str) == "table" then + if existing then + for k, v in next, str do + existing[k] = v + end + return exiting + else + return str + end + else + hash = existing or { } + lpegmatch(pattern_b_s,str) + return hash + end +end + +function parsers.settings_to_hash_strict(str,existing) + if not str or str == "" then + return nil + elseif type(str) == "table" then + if existing then + for k, v in next, str do + existing[k] = v + end + return exiting + else + return str + end + elseif str and str ~= "" then + hash = existing or { } + lpegmatch(pattern_c_s,str) + return next(hash) and hash + end +end + +local separator = comma * space^0 +local value = lbrace * C((nobrace + nestedbraces)^0) * rbrace + + C((nestedbraces + (1-comma))^0) +local pattern = spaces * Ct(value*(separator*value)^0) + +-- "aap, {noot}, mies" : outer {} removed, leading spaces ignored + +patterns.settings_to_array = pattern + +-- we could use a weak table as cache + +function parsers.settings_to_array(str,strict) + if not str or str == "" then + return { } + elseif type(str) == "table" then + return str + elseif strict then + if find(str,"{",1,true) then + return lpegmatch(pattern,str) + else + return { str } + end + elseif find(str,",",1,true) then + return lpegmatch(pattern,str) + else + return { str } + end +end + +function parsers.settings_to_numbers(str) + if not str or str == "" then + return { } + end + if type(str) == "table" then + -- fall through + elseif find(str,",",1,true) then + str = lpegmatch(pattern,str) + else + return { tonumber(str) } + end + for i=1,#str do + str[i] = tonumber(str[i]) + end + return str +end + +local value = lbrace * C((nobrace + nestedbraces)^0) * rbrace + + C((nestedbraces + nestedbrackets + nestedparents + (1-comma))^0) +local pattern = spaces * Ct(value*(separator*value)^0) + +function parsers.settings_to_array_obey_fences(str) + return lpegmatch(pattern,str) +end + +-- inspect(parsers.settings_to_array_obey_fences("url(http://a,b.c)")) + +-- this one also strips end spaces before separators +-- +-- "{123} , 456 " -> "123" "456" + +-- local separator = space^0 * comma * space^0 +-- local value = P(lbrace * C((nobrace + nestedbraces)^0) * rbrace) +-- + C((nestedbraces + (1-(space^0*(comma+P(-1)))))^0) +-- local withvalue = Carg(1) * value / function(f,s) return f(s) end +-- local pattern_a = spaces * Ct(value*(separator*value)^0) +-- local pattern_b = spaces * withvalue * (separator*withvalue)^0 + +local cache_a = { } +local cache_b = { } + +function parsers.groupedsplitat(symbol,withaction) + if not symbol then + symbol = "," + end + local pattern = (withaction and cache_b or cache_a)[symbol] + if not pattern then + local symbols = S(symbol) + local separator = space^0 * symbols * space^0 + local value = lbrace * C((nobrace + nestedbraces)^0) * rbrace + + C((nestedbraces + (1-(space^0*(symbols+P(-1)))))^0) + if withaction then + local withvalue = Carg(1) * value / function(f,s) return f(s) end + pattern = spaces * withvalue * (separator*withvalue)^0 + cache_b[symbol] = pattern + else + pattern = spaces * Ct(value*(separator*value)^0) + cache_a[symbol] = pattern + end + end + return pattern +end + +local pattern_a = parsers.groupedsplitat(",",false) +local pattern_b = parsers.groupedsplitat(",",true) + +function parsers.stripped_settings_to_array(str) + if not str or str == "" then + return { } + else + return lpegmatch(pattern_a,str) + end +end + +function parsers.process_stripped_settings(str,action) + if not str or str == "" then + return { } + else + return lpegmatch(pattern_b,str,1,action) + end +end + +-- parsers.process_stripped_settings("{123} , 456 ",function(s) print("["..s.."]") end) +-- parsers.process_stripped_settings("123 , 456 ",function(s) print("["..s.."]") end) + +local function set(t,v) + t[#t+1] = v +end + +local value = P(Carg(1)*value) / set +local pattern = value*(separator*value)^0 * Carg(1) + +function parsers.add_settings_to_array(t,str) + return lpegmatch(pattern,str,nil,t) +end + +function parsers.hash_to_string(h,separator,yes,no,strict,omit) + if h then + local t = { } + local tn = 0 + local s = sortedkeys(h) + omit = omit and tohash(omit) + for i=1,#s do + local key = s[i] + if not omit or not omit[key] then + local value = h[key] + if type(value) == "boolean" then + if yes and no then + if value then + tn = tn + 1 + t[tn] = key .. '=' .. yes + elseif not strict then + tn = tn + 1 + t[tn] = key .. '=' .. no + end + elseif value or not strict then + tn = tn + 1 + t[tn] = key .. '=' .. tostring(value) + end + else + tn = tn + 1 + t[tn] = key .. '=' .. value + end + end + end + return concat(t,separator or ",") + else + return "" + end +end + +function parsers.array_to_string(a,separator) + if a then + return concat(a,separator or ",") + else + return "" + end +end + +-- function parsers.settings_to_set(str,t) -- tohash? -- todo: lpeg -- duplicate anyway +-- if str then +-- t = t or { } +-- for s in gmatch(str,"[^, ]+") do -- space added +-- t[s] = true +-- end +-- return t +-- else +-- return { } +-- end +-- end + +local pattern = Cf(Ct("") * Cg(C((1-S(", "))^1) * S(", ")^0 * Cc(true))^1,rawset) + +function parsers.settings_to_set(str) + return str and lpegmatch(pattern,str) or { } +end + +hashes.settings_to_set = table.setmetatableindex(function(t,k) -- experiment, not public + local v = k and lpegmatch(pattern,k) or { } + t[k] = v + return v +end) + +getmetatable(hashes.settings_to_set).__mode = "kv" -- could be an option (maybe sharing makes sense) + +function parsers.simple_hash_to_string(h, separator) + local t = { } + local tn = 0 + for k, v in sortedhash(h) do + if v then + tn = tn + 1 + t[tn] = k + end + end + return concat(t,separator or ",") +end + +-- for mtx-context etc: aaaa bbbb cccc=dddd eeee=ffff + +local str = Cs(lpegpatterns.unquoted) + C((1-whitespace-equal)^1) +local setting = Cf( Carg(1) * (whitespace^0 * Cg(str * whitespace^0 * (equal * whitespace^0 * str + Cc(""))))^1,rawset) +local splitter = setting^1 + +function parsers.options_to_hash(str,target) + return str and lpegmatch(splitter,str,1,target or { }) or { } +end + +local splitter = lpeg.tsplitat(" ") + +function parsers.options_to_array(str) + return str and lpegmatch(splitter,str) or { } +end + +-- for chem (currently one level) + +local value = P(lbrace * C((nobrace + nestedbraces)^0) * rbrace) + + C(digit^1 * lparent * (noparent + nestedparents)^1 * rparent) + + C((nestedbraces + (1-comma))^1) +local pattern_a = spaces * Ct(value*(separator*value)^0) + +local function repeater(n,str) + if not n then + return str + else + local s = lpegmatch(pattern_a,str) + if n == 1 then + return unpack(s) + else + local t = { } + local tn = 0 + for i=1,n do + for j=1,#s do + tn = tn + 1 + t[tn] = s[j] + end + end + return unpack(t) + end + end +end + +local value = P(lbrace * C((nobrace + nestedbraces)^0) * rbrace) + + (C(digit^1)/tonumber * lparent * Cs((noparent + nestedparents)^1) * rparent) / repeater + + C((nestedbraces + (1-comma))^1) +local pattern_b = spaces * Ct(value*(separator*value)^0) + +function parsers.settings_to_array_with_repeat(str,expand) -- beware: "" => { } + if expand then + return lpegmatch(pattern_b,str) or { } + else + return lpegmatch(pattern_a,str) or { } + end +end + +-- + +local value = lbrace * C((nobrace + nestedbraces)^0) * rbrace +local pattern = Ct((space + value)^0) + +function parsers.arguments_to_table(str) + return lpegmatch(pattern,str) +end + +-- temporary here (unoptimized) + +function parsers.getparameters(self,class,parentclass,settings) + local sc = self[class] + if not sc then + sc = { } + self[class] = sc + if parentclass then + local sp = self[parentclass] + if not sp then + sp = { } + self[parentclass] = sp + end + setmetatableindex(sc,sp) + end + end + parsers.settings_to_hash(settings,sc) +end + +function parsers.listitem(str) + return gmatch(str,"[^, ]+") +end + +-- + +local pattern = Cs { "start", + start = V("one") + V("two") + V("three"), + rest = (Cc(",") * V("thousand"))^0 * (P(".") + endofstring) * anything^0, + thousand = digit * digit * digit, + one = digit * V("rest"), + two = digit * digit * V("rest"), + three = V("thousand") * V("rest"), +} + +lpegpatterns.splitthousands = pattern -- maybe better in the parsers namespace ? + +function parsers.splitthousands(str) + return lpegmatch(pattern,str) or str +end + +-- print(parsers.splitthousands("11111111111.11")) + +local optionalwhitespace = whitespace^0 + +lpegpatterns.words = Ct((Cs((1-punctuation-whitespace)^1) + anything)^1) +lpegpatterns.sentences = Ct((optionalwhitespace * Cs((1-period)^0 * period))^1) +lpegpatterns.paragraphs = Ct((optionalwhitespace * Cs((whitespace^1*endofstring/"" + 1 - (spacer^0*newline*newline))^1))^1) + +-- local str = " Word1 word2. \n Word3 word4. \n\n Word5 word6.\n " +-- inspect(lpegmatch(lpegpatterns.paragraphs,str)) +-- inspect(lpegmatch(lpegpatterns.sentences,str)) +-- inspect(lpegmatch(lpegpatterns.words,str)) + +-- handy for k="v" [, ] k="v" + +local dquote = P('"') +local equal = P('=') +local escape = P('\\') +local separator = S(' ,') + +local key = C((1-equal)^1) +local value = dquote * C((1-dquote-escape*dquote)^0) * dquote + +----- pattern = Cf(Ct("") * Cg(key * equal * value) * separator^0,rawset)^0 * P(-1) -- was wrong +local pattern = Cf(Ct("") * (Cg(key * equal * value) * separator^0)^1,rawset)^0 * P(-1) + +function parsers.keq_to_hash(str) + if str and str ~= "" then + return lpegmatch(pattern,str) + else + return { } + end +end + +-- inspect(lpeg.match(pattern,[[key="value" foo="bar"]])) + +local defaultspecification = { separator = ",", quote = '"' } + +-- this version accepts multiple separators and quotes as used in the +-- database module + +function parsers.csvsplitter(specification) + specification = specification and setmetatableindex(specification,defaultspecification) or defaultspecification + local separator = specification.separator + local quotechar = specification.quote + local separator = S(separator ~= "" and separator or ",") + local whatever = C((1 - separator - newline)^0) + if quotechar and quotechar ~= "" then + local quotedata = nil + for chr in gmatch(quotechar,".") do + local quotechar = P(chr) + local quoteword = quotechar * C((1 - quotechar)^0) * quotechar + if quotedata then + quotedata = quotedata + quoteword + else + quotedata = quoteword + end + end + whatever = quotedata + whatever + end + local parser = Ct((Ct(whatever * (separator * whatever)^0) * S("\n\r")^1)^0 ) + return function(data) + return lpegmatch(parser,data) + end +end + +-- and this is a slightly patched version of a version posted by Philipp Gesang + +-- local mycsvsplitter = parsers.rfc4180splitter() + +-- local crap = [[ +-- first,second,third,fourth +-- "1","2","3","4" +-- "a","b","c","d" +-- "foo","bar""baz","boogie","xyzzy" +-- ]] + +-- local list, names = mycsvsplitter(crap,true) inspect(list) inspect(names) +-- local list, names = mycsvsplitter(crap) inspect(list) inspect(names) + +function parsers.rfc4180splitter(specification) + specification = specification and setmetatableindex(specification,defaultspecification) or defaultspecification + local separator = specification.separator --> rfc: COMMA + local quotechar = P(specification.quote) --> DQUOTE + local dquotechar = quotechar * quotechar --> 2DQUOTE + / specification.quote + local separator = S(separator ~= "" and separator or ",") + local escaped = quotechar + * Cs((dquotechar + (1 - quotechar))^0) + * quotechar + local non_escaped = C((1 - quotechar - newline - separator)^1) + local field = escaped + non_escaped + Cc("") + local record = Ct(field * (separator * field)^1) + local headerline = record * Cp() + local morerecords = (newline^(specification.strict and -1 or 1) * record)^0 + local headeryes = Ct(morerecords) + local headernop = Ct(record * morerecords) + return function(data,getheader) + if getheader then + local header, position = lpegmatch(headerline,data) + local data = lpegmatch(headeryes,data,position) + return data, header + else + return lpegmatch(headernop,data) + end + end +end + +-- parsers.stepper("1,7-",9,function(i) print(">>>",i) end) +-- parsers.stepper("1-3,7,8,9") +-- parsers.stepper("1-3,6,7",function(i) print(">>>",i) end) +-- parsers.stepper(" 1 : 3, ,7 ") +-- parsers.stepper("1:4,9:13,24:*",30) + +local function ranger(first,last,n,action) + if not first then + -- forget about it + elseif last == true then + for i=first,n or first do + action(i) + end + elseif last then + for i=first,last do + action(i) + end + else + action(first) + end +end + +local cardinal = lpegpatterns.cardinal / tonumber +local spacers = lpegpatterns.spacer^0 +local endofstring = lpegpatterns.endofstring + +local stepper = spacers * ( cardinal * ( spacers * S(":-") * spacers * ( cardinal + Cc(true) ) + Cc(false) ) + * Carg(1) * Carg(2) / ranger * S(", ")^0 )^1 + +local stepper = spacers * ( cardinal * ( spacers * S(":-") * spacers * ( cardinal + (P("*") + endofstring) * Cc(true) ) + Cc(false) ) + * Carg(1) * Carg(2) / ranger * S(", ")^0 )^1 * endofstring -- we're sort of strict (could do without endofstring) + +function parsers.stepper(str,n,action) + if type(n) == "function" then + lpegmatch(stepper,str,1,false,n or print) + else + lpegmatch(stepper,str,1,n,action or print) + end +end + +-- + +local pattern_math = Cs((P("%")/"\\percent " + P("^") * Cc("{") * lpegpatterns.integer * Cc("}") + anything)^0) +local pattern_text = Cs((P("%")/"\\percent " + (P("^")/"\\high") * Cc("{") * lpegpatterns.integer * Cc("}") + anything)^0) + +patterns.unittotex = pattern + +function parsers.unittotex(str,textmode) + return lpegmatch(textmode and pattern_text or pattern_math,str) +end + +local pattern = Cs((P("^") / "<sup>" * lpegpatterns.integer * Cc("</sup>") + anything)^0) + +function parsers.unittoxml(str) + return lpegmatch(pattern,str) +end + +-- print(parsers.unittotex("10^-32 %"),utilities.parsers.unittoxml("10^32 %")) + +local cache = { } +local spaces = lpegpatterns.space^0 +local dummy = function() end + +setmetatableindex(cache,function(t,k) + local separator = P(k) + local value = (1-separator)^0 + local pattern = spaces * C(value) * separator^0 * Cp() + t[k] = pattern + return pattern +end) + +local commalistiterator = cache[","] + +function parsers.iterator(str,separator) + local n = #str + if n == 0 then + return dummy + else + local pattern = separator and cache[separator] or commalistiterator + local p = 1 + return function() + if p <= n then + local s, e = lpegmatch(pattern,str,p) + if e then + p = e + return s + end + end + end + end +end + +-- for s in parsers.iterator("a b c,b,c") do +-- print(s) +-- end + +local function initialize(t,name) + local source = t[name] + if source then + local result = { } + for k, v in next, t[name] do + result[k] = v + end + return result + else + return { } + end +end + +local function fetch(t,name) + return t[name] or { } +end + +local function process(result,more) + for k, v in next, more do + result[k] = v + end + return result +end + +local name = C((1-S(", "))^1) +local parser = (Carg(1) * name / initialize) * (S(", ")^1 * (Carg(1) * name / fetch))^0 +local merge = Cf(parser,process) + +function parsers.mergehashes(hash,list) + return lpegmatch(merge,list,1,hash) +end + +-- local t = { +-- aa = { alpha = 1, beta = 2, gamma = 3, }, +-- bb = { alpha = 4, beta = 5, delta = 6, }, +-- cc = { epsilon = 3 }, +-- } +-- +-- inspect(parsers.mergehashes(t,"aa, bb, cc")) + +function parsers.runtime(time) + if not time then + time = os.runtime() + end + local days = div(time,24*60*60) + time = mod(time,24*60*60) + local hours = div(time,60*60) + time = mod(time,60*60) + local minutes = div(time,60) + local seconds = mod(time,60) + return days, hours, minutes, seconds +end + +-- + +local spacing = whitespace^0 +local apply = P("->") +local method = C((1-apply)^1) +local token = lbrace * C((1-rbrace)^1) * rbrace + C(anything^1) + +local pattern = spacing * (method * spacing * apply + Carg(1)) * spacing * token + +function parsers.splitmethod(str,default) + if str then + return lpegmatch(pattern,str,1,default or false) + else + return default or false, "" + end +end + +-- print(parsers.splitmethod(" foo -> {bar} ")) +-- print(parsers.splitmethod("foo->{bar}")) +-- print(parsers.splitmethod("foo->bar")) +-- print(parsers.splitmethod("foo")) +-- print(parsers.splitmethod("{foo}")) +-- print(parsers.splitmethod()) + +local p_year = lpegpatterns.digit^4 / tonumber + +local pattern = Cf( Ct("") * + ( + ( Cg(Cc("year") * p_year) + * S("-/") * Cg(Cc("month") * cardinal) + * S("-/") * Cg(Cc("day") * cardinal) + ) + + ( Cg(Cc("day") * cardinal) + * S("-/") * Cg(Cc("month") * cardinal) + * S("-/") * Cg(Cc("year") * p_year) + ) + ) + * P(" ") * Cg(Cc("hour") * cardinal) + * P(":") * Cg(Cc("min") * cardinal) + * (P(":") * Cg(Cc("sec") * cardinal))^-1 +, rawset) + +lpegpatterns.splittime = pattern + +function parsers.totime(str) + return lpegmatch(pattern,str) +end + +-- print(os.time(parsers.totime("2019-03-05 12:12:12"))) +-- print(os.time(parsers.totime("2019/03/05 12:12:12"))) +-- print(os.time(parsers.totime("05-03-2019 12:12:12"))) +-- print(os.time(parsers.totime("05/03/2019 12:12:12"))) diff --git a/macros/luatex/generic/lualibs/lualibs-util-sta.lua b/macros/luatex/generic/lualibs/lualibs-util-sta.lua new file mode 100644 index 0000000000..7819395f69 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-util-sta.lua @@ -0,0 +1,352 @@ +if not modules then modules = { } end modules ['util-sta'] = { + version = 1.001, + comment = "companion to util-ini.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +local insert, remove, fastcopy, concat = table.insert, table.remove, table.fastcopy, table.concat +local format = string.format +local select, tostring = select, tostring + +local trace_stacker = false trackers.register("stacker.resolve", function(v) trace_stacker = v end) + +local stacker = stacker or { } + +utilities.stacker = stacker + +local function start(s,t,first,last) + if s.mode == "switch" then + local n = tostring(t[last]) + if trace_stacker then + s.report("start: %s",n) + end + return n + else + local r = { } + for i=first,last do + r[#r+1] = tostring(t[i]) + end + local n = concat(r," ") + if trace_stacker then + s.report("start: %s",n) + end + return n + end +end + +local function stop(s,t,first,last) + if s.mode == "switch" then + local n = tostring(false) + if trace_stacker then + s.report("stop: %s",n) + end + return n + else + local r = { } + for i=last,first,-1 do + r[#r+1] = tostring(false) + end + local n = concat(r," ") + if trace_stacker then + s.report("stop: %s",n) + end + return n + end +end + +local function change(s,t1,first1,last1,t2,first2,last2) + if s.mode == "switch" then + local n = tostring(t2[last2]) + if trace_stacker then + s.report("change: %s",n) + end + return n + else + local r = { } + for i=last1,first1,-1 do + r[#r+1] = tostring(false) + end + local n = concat(r," ") + for i=first2,last2 do + r[#r+1] = tostring(t2[i]) + end + if trace_stacker then + s.report("change: %s",n) + end + return n + end +end + +function stacker.new(name) + + local report = logs.reporter("stacker",name or nil) + + local s + + local stack = { } + local list = { } + local ids = { } + local hash = { } + + local hashing = true + + local function push(...) + for i=1,select("#",...) do + insert(stack,(select(i,...))) -- watch the () + end + if hashing then + local c = concat(stack,"|") + local n = hash[c] + if not n then + n = #list+1 + hash[c] = n + list[n] = fastcopy(stack) + end + insert(ids,n) + return n + else + local n = #list+1 + list[n] = fastcopy(stack) + insert(ids,n) + return n + end + end + + local function pop() + remove(stack) + remove(ids) + return ids[#ids] or s.unset or -1 + end + + local function clean() + if #stack == 0 then + if trace_stacker then + s.report("%s list entries, %s stack entries",#list,#stack) + end + end + end + + local tops = { } + local top = nil + local switch = nil + + local function resolve_reset(mode) + if #tops > 0 then + report("resetting %s left-over states of %a",#tops,name) + end + tops = { } + top = nil + switch = nil + end + + local function resolve_begin(mode) + if mode then + switch = mode == "switch" + else + switch = s.mode == "switch" + end + top = { switch = switch } + insert(tops,top) + end + + local function resolve_step(ti) -- keep track of changes outside function ! + -- todo: optimize for n=1 etc + local result = nil + local noftop = top and #top or 0 + if ti > 0 then + local current = list[ti] + if current then + local noflist = #current + local nofsame = 0 + if noflist > noftop then + for i=1,noflist do + if current[i] == top[i] then + nofsame = i + else + break + end + end + else + for i=1,noflist do + if current[i] == top[i] then + nofsame = i + else + break + end + end + end + local plus = nofsame + 1 + if plus <= noftop then + if plus <= noflist then + if switch then + result = s.change(s,top,plus,noftop,current,nofsame,noflist) + else + result = s.change(s,top,plus,noftop,current,plus,noflist) + end + else + if switch then + result = s.change(s,top,plus,noftop,current,nofsame,noflist) + else + result = s.stop(s,top,plus,noftop) + end + end + elseif plus <= noflist then + if switch then + result = s.start(s,current,nofsame,noflist) + else + result = s.start(s,current,plus,noflist) + end + end + top = current + else + if 1 <= noftop then + result = s.stop(s,top,1,noftop) + end + top = { } + end + return result + else + if 1 <= noftop then + result = s.stop(s,top,1,noftop) + end + top = { } + return result + end + end + + local function resolve_end() + -- resolve_step(s.unset) + if #tops > 0 then -- was #top brrr + local result = s.stop(s,top,1,#top) + remove(tops) + top = tops[#tops] + switch = top and top.switch + return result + end + end + + local function resolve(t) + resolve_begin() + for i=1,#t do + resolve_step(t[i]) + end + resolve_end() + end + + s = { + name = name or "unknown", + unset = -1, + report = report, + start = start, + stop = stop, + change = change, + push = push, + pop = pop, + clean = clean, + resolve = resolve, + resolve_begin = resolve_begin, + resolve_step = resolve_step, + resolve_end = resolve_end, + resolve_reset = resolve_reset, + } + + return s -- we can overload functions + +end + +-- local s = utilities.stacker.new("demo") +-- +-- local unset = s.unset +-- local push = s.push +-- local pop = s.pop +-- +-- local t = { +-- unset, +-- unset, +-- push("a"), -- a +-- push("b","c"), -- a b c +-- pop(), -- a b +-- push("d"), -- a b d +-- pop(), -- a b +-- unset, +-- pop(), -- a +-- pop(), -- b +-- unset, +-- unset, +-- } +-- +-- s.resolve(t) + +-- demostacker = utilities.stacker.new("demos") +-- +-- local whatever = { +-- one = "1 0 0 RG 1 0 0 rg", +-- two = "1 1 0 RG 1 1 0 rg", +-- [false] = "0 G 0 g", +-- } +-- +-- local concat = table.concat +-- +-- local pageliteral = nodes.pool.pageliteral +-- +-- function demostacker.start(s,t,first,last) +-- local n = whatever[t[last]] +-- -- s.report("start: %s",n) +-- return pageliteral(n) +-- end +-- +-- function demostacker.stop(s,t,first,last) +-- local n = whatever[false] +-- -- s.report("stop: %s",n) +-- return pageliteral(n) +-- end +-- +-- function demostacker.change(s,t1,first1,last1,t2,first2,last2) +-- local n = whatever[t2[last2]] +-- -- s.report("change: %s",n) +-- return pageliteral(n) +-- end +-- +-- demostacker.mode = "switch" +-- +-- local whatever = { +-- one = "/OC /test1 BDC", +-- two = "/OC /test2 BDC", +-- [false] = "EMC", +-- } +-- +-- demostacker = utilities.stacker.new("demos") +-- +-- function demostacker.start(s,t,first,last) +-- local r = { } +-- for i=first,last do +-- r[#r+1] = whatever[t[i]] +-- end +-- -- s.report("start: %s",concat(r," ")) +-- return pageliteral(concat(r," ")) +-- end +-- +-- function demostacker.stop(s,t,first,last) +-- local r = { } +-- for i=last,first,-1 do +-- r[#r+1] = whatever[false] +-- end +-- -- s.report("stop: %s",concat(r," ")) +-- return pageliteral(concat(r," ")) +-- end +-- +-- function demostacker.change(s,t1,first1,last1,t2,first2,last2) +-- local r = { } +-- for i=last1,first1,-1 do +-- r[#r+1] = whatever[false] +-- end +-- for i=first2,last2 do +-- r[#r+1] = whatever[t2[i]] +-- end +-- -- s.report("change: %s",concat(r," ")) +-- return pageliteral(concat(r," ")) +-- end +-- +-- demostacker.mode = "stack" diff --git a/macros/luatex/generic/lualibs/lualibs-util-sto.lua b/macros/luatex/generic/lualibs/lualibs-util-sto.lua new file mode 100644 index 0000000000..a08d25ced1 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-util-sto.lua @@ -0,0 +1,282 @@ +if not modules then modules = { } end modules ['util-sto'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +local setmetatable, getmetatable, rawset, type = setmetatable, getmetatable, rawset, type + +utilities = utilities or { } +utilities.storage = utilities.storage or { } +local storage = utilities.storage + +function storage.mark(t) + if not t then + print("\nfatal error: storage cannot be marked\n") + os.exit() + return + end + local m = getmetatable(t) + if not m then + m = { } + setmetatable(t,m) + end + m.__storage__ = true + return t +end + +function storage.allocate(t) + t = t or { } + local m = getmetatable(t) + if not m then + m = { } + setmetatable(t,m) + end + m.__storage__ = true + return t +end + +function storage.marked(t) + local m = getmetatable(t) + return m and m.__storage__ +end + +function storage.checked(t) + if not t then + report("\nfatal error: storage has not been allocated\n") + os.exit() + return + end + return t +end + +-- function utilities.storage.delay(parent,name,filename) +-- local m = getmetatable(parent) +-- m.__list[name] = filename +-- end +-- +-- function utilities.storage.predefine(parent) +-- local list = { } +-- local m = getmetatable(parent) or { +-- __list = list, +-- __index = function(t,k) +-- local l = require(list[k]) +-- t[k] = l +-- return l +-- end +-- } +-- setmetatable(parent,m) +-- end +-- +-- bla = { } +-- utilities.storage.predefine(bla) +-- utilities.storage.delay(bla,"test","oepsoeps") +-- local t = bla.test +-- table.print(t) +-- print(t.a) + +function storage.setinitializer(data,initialize) + local m = getmetatable(data) or { } + m.__index = function(data,k) + m.__index = nil -- so that we can access the entries during initializing + initialize() + return data[k] + end + setmetatable(data, m) +end + +local keyisvalue = { __index = function(t,k) + t[k] = k + return k +end } + +function storage.sparse(t) + t = t or { } + setmetatable(t,keyisvalue) + return t +end + +-- table namespace ? + +local function f_empty () return "" end -- t,k +local function f_self (t,k) t[k] = k return k end +local function f_table (t,k) local v = { } t[k] = v return v end +local function f_number(t,k) t[k] = 0 return 0 end -- t,k,v +local function f_ignore() end -- t,k,v + +local f_index = { + ["empty"] = f_empty, + ["self"] = f_self, + ["table"] = f_table, + ["number"] = f_number, +} + +function table.setmetatableindex(t,f) + if type(t) ~= "table" then + f, t = t, { } + end + local m = getmetatable(t) + local i = f_index[f] or f + if m then + m.__index = i + else + setmetatable(t,{ __index = i }) + end + return t +end + +local f_index = { + ["ignore"] = f_ignore, +} + +function table.setmetatablenewindex(t,f) + if type(t) ~= "table" then + f, t = t, { } + end + local m = getmetatable(t) + local i = f_index[f] or f + if m then + m.__newindex = i + else + setmetatable(t,{ __newindex = i }) + end + return t +end + +function table.setmetatablecall(t,f) + if type(t) ~= "table" then + f, t = t, { } + end + local m = getmetatable(t) + if m then + m.__call = f + else + setmetatable(t,{ __call = f }) + end + return t +end + +-- the manual is somewhat fuzzy about this but suggests that one can best +-- set all fields before assigning a metatable + +function table.setmetatableindices(t,f,n,c) + if type(t) ~= "table" then + f, t = t, { } + end + local m = getmetatable(t) + local i = f_index[f] or f + if m then + m.__index = i + m.__newindex = n + m.__call = c + else + setmetatable(t,{ + __index = i, + __newindex = n, + __call = c, + }) + end + return t +end + +function table.setmetatablekey(t,key,value) + local m = getmetatable(t) + if not m then + m = { } + setmetatable(t,m) + end + m[key] = value + return t +end + +function table.getmetatablekey(t,key,value) + local m = getmetatable(t) + return m and m[key] +end + +function table.makeweak(t) + if not t then + t = { } + end + local m = getmetatable(t) + if m then + m.__mode = "v" + else + setmetatable(t,{ __mode = "v" }) + end + return t +end + +-- Problem: we have no __next (which is ok as it would probably slow down lua) so +-- we cannot loop over the keys. + +-- local parametersets = table.autokeys() +-- +-- parametersets.foo.bar = function(t,k) return "OEPS" end +-- parametersets.foo.foo = "SPEO" +-- parametersets.crap = { a = "a", b = table.autokey { function() return "b" end } } +-- +-- print(parametersets.foo.bar) +-- print(parametersets.foo.foo) +-- print(parametersets.crap.b) +-- print(parametersets.crap.b[1]) + +-- function table.autotables(t) +-- local t = t or { } +-- local m = getmetatable(t) +-- if not m then +-- m = { } +-- setmetatable(t,m) +-- end +-- m.__newindex = function(t,k,p) +-- local v = { } +-- local m = { +-- __index = function(t,k) +-- local v = p[k] +-- if type(v) == "function" then +-- return v(t,k) -- so we can have multiple arguments +-- else +-- return v +-- end +-- end, +-- __newindex = function(t,k,v) +-- p[k] = v +-- end, +-- __len = function(t) +-- return #p +-- end, +-- } +-- setmetatable(v,m) +-- rawset(t,k,v) +-- return v +-- end +-- m.__index = function(t,k) +-- local v = { } +-- t[k] = v -- calls newindex +-- return v +-- end +-- return t +-- end +-- +-- function table.autokeys(p) +-- local t = { } +-- setmetatable(t, { +-- __newindex = function(t,k,v) +-- p[k] = v +-- end, +-- __index = function(t,k) +-- local v = p[k] +-- if type(v) == "function" then +-- return v(t,k) -- so we can have multiple arguments +-- else +-- return v +-- end +-- end, +-- __len = function(t) +-- return #p +-- end, +-- }) +-- return t +-- end diff --git a/macros/luatex/generic/lualibs/lualibs-util-str.lua b/macros/luatex/generic/lualibs/lualibs-util-str.lua new file mode 100644 index 0000000000..ad22302df0 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-util-str.lua @@ -0,0 +1,1461 @@ +if not modules then modules = { } end modules ['util-str'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +utilities = utilities or { } +utilities.strings = utilities.strings or { } +local strings = utilities.strings + +local format, gsub, rep, sub, find = string.format, string.gsub, string.rep, string.sub, string.find +local load, dump = load, string.dump +local tonumber, type, tostring, next, setmetatable = tonumber, type, tostring, next, setmetatable +local unpack, concat = table.unpack, table.concat +local P, V, C, S, R, Ct, Cs, Cp, Carg, Cc = lpeg.P, lpeg.V, lpeg.C, lpeg.S, lpeg.R, lpeg.Ct, lpeg.Cs, lpeg.Cp, lpeg.Carg, lpeg.Cc +local patterns, lpegmatch = lpeg.patterns, lpeg.match +local utfchar, utfbyte, utflen = utf.char, utf.byte, utf.len + +----- loadstripped = utilities.lua.loadstripped +----- setmetatableindex = table.setmetatableindex + +local loadstripped = function(str,shortcuts) + if shortcuts then + return load(dump(load(str),true),nil,nil,shortcuts) + else + return load(dump(load(str),true)) + end +end + +-- todo: make a special namespace for the formatter + +if not number then number = { } end -- temp hack for luatex-fonts + +local stripzero = patterns.stripzero +local stripzeros = patterns.stripzeros +local newline = patterns.newline +local endofstring = patterns.endofstring +local anything = patterns.anything +local whitespace = patterns.whitespace +local space = patterns.space +local spacer = patterns.spacer +local spaceortab = patterns.spaceortab +local digit = patterns.digit +local sign = patterns.sign +local period = patterns.period + +-- local function points(n) +-- n = tonumber(n) +-- return (not n or n == 0) and "0pt" or lpegmatch(stripzeros,format("%.5fpt",n/65536)) +-- end + +-- local function basepoints(n) +-- n = tonumber(n) +-- return (not n or n == 0) and "0bp" or lpegmatch(stripzeros,format("%.5fbp", n*(7200/7227)/65536)) +-- end + +local ptf = 1 / 65536 +local bpf = (7200/7227) / 65536 + +local function points(n) + if n == 0 then + return "0pt" + end + n = tonumber(n) + if not n or n == 0 then + return "0pt" + end + n = n * ptf + if n % 1 == 0 then + return format("%ipt",n) + end + return lpegmatch(stripzeros,format("%.5fpt",n)) -- plural as we need to keep the pt +end + +local function basepoints(n) + if n == 0 then + return "0pt" + end + n = tonumber(n) + if not n or n == 0 then + return "0pt" + end + n = n * bpf + if n % 1 == 0 then + return format("%ibp",n) + end + return lpegmatch(stripzeros,format("%.5fbp",n)) -- plural as we need to keep the pt +end + +number.points = points +number.basepoints = basepoints + +-- str = " \n \ntest \n test\ntest " +-- print("["..string.gsub(string.collapsecrlf(str),"\n","+").."]") + +local rubish = spaceortab^0 * newline +local anyrubish = spaceortab + newline +local stripped = (spaceortab^1 / "") * newline +local leading = rubish^0 / "" +local trailing = (anyrubish^1 * endofstring) / "" +local redundant = rubish^3 / "\n" + +local pattern = Cs(leading * (trailing + redundant + stripped + anything)^0) + +function strings.collapsecrlf(str) + return lpegmatch(pattern,str) +end + +-- The following functions might end up in another namespace. + +local repeaters = { } -- watch how we also moved the -1 in depth-1 to the creator + +function strings.newrepeater(str,offset) + offset = offset or 0 + local s = repeaters[str] + if not s then + s = { } + repeaters[str] = s + end + local t = s[offset] + if t then + return t + end + t = { } + setmetatable(t, { __index = function(t,k) + if not k then + return "" + end + local n = k + offset + local s = n > 0 and rep(str,n) or "" + t[k] = s + return s + end }) + s[offset] = t + return t +end + +-- local dashes = strings.newrepeater("--",-1) +-- print(dashes[2],dashes[3],dashes[1]) + +local extra, tab, start = 0, 0, 4, 0 + +local nspaces = strings.newrepeater(" ") + +string.nspaces = nspaces + +local pattern = + Carg(1) / function(t) + extra, tab, start = 0, t or 7, 1 + end + * Cs(( + Cp() * patterns.tab / function(position) + local current = (position - start + 1) + extra + local spaces = tab-(current-1) % tab + if spaces > 0 then + extra = extra + spaces - 1 + return nspaces[spaces] -- rep(" ",spaces) + else + return "" + end + end + + newline * Cp() / function(position) + extra, start = 0, position + end + + anything + )^1) + +function strings.tabtospace(str,tab) + -- no real gain in first checking if a \t is there + return lpegmatch(pattern,str,1,tab or 7) +end + +function string.utfpadding(s,n) + if not n or n == 0 then + return "" + end + local l = utflen(s) + if n > 0 then + return nspaces[n-l] + else + return nspaces[-n-l] + end +end + +-- local t = { +-- "1234567123456712345671234567", +-- "\tb\tc", +-- "a\tb\tc", +-- "aa\tbb\tcc", +-- "aaa\tbbb\tccc", +-- "aaaa\tbbbb\tcccc", +-- "aaaaa\tbbbbb\tccccc", +-- "aaaaaa\tbbbbbb\tcccccc\n aaaaaa\tbbbbbb\tcccccc", +-- "one\n two\nxxx three\nxx four\nx five\nsix", +-- } +-- for k=1,#t do +-- print(strings.tabtospace(t[k])) +-- end + +-- todo: lpeg + +-- function strings.striplong(str) -- strips all leading spaces +-- str = gsub(str,"^%s*","") +-- str = gsub(str,"[\n\r]+ *","\n") +-- return str +-- end + +local optionalspace = spacer^0 +local nospace = optionalspace/"" +local endofline = nospace * newline + +local stripend = (whitespace^1 * endofstring)/"" + +local normalline = (nospace * ((1-optionalspace*(newline+endofstring))^1) * nospace) + +local stripempty = endofline^1/"" +local normalempty = endofline^1 +local singleempty = endofline * (endofline^0/"") +local doubleempty = endofline * endofline^-1 * (endofline^0/"") +local stripstart = stripempty^0 + +local intospace = whitespace^1/" " +local noleading = whitespace^1/"" +local notrailing = noleading * endofstring + +local p_prune_normal = Cs ( stripstart * ( stripend + normalline + normalempty )^0 ) +local p_prune_collapse = Cs ( stripstart * ( stripend + normalline + doubleempty )^0 ) +local p_prune_noempty = Cs ( stripstart * ( stripend + normalline + singleempty )^0 ) +local p_prune_intospace = Cs ( noleading * ( notrailing + intospace + 1 )^0 ) +local p_retain_normal = Cs ( ( normalline + normalempty )^0 ) +local p_retain_collapse = Cs ( ( normalline + doubleempty )^0 ) +local p_retain_noempty = Cs ( ( normalline + singleempty )^0 ) + +-- function striplines(str,prune,collapse,noempty) +-- if prune then +-- if noempty then +-- return lpegmatch(p_prune_noempty,str) or str +-- elseif collapse then +-- return lpegmatch(p_prune_collapse,str) or str +-- else +-- return lpegmatch(p_prune_normal,str) or str +-- end +-- else +-- if noempty then +-- return lpegmatch(p_retain_noempty,str) or str +-- elseif collapse then +-- return lpegmatch(p_retain_collapse,str) or str +-- else +-- return lpegmatch(p_retain_normal,str) or str +-- end +-- end +-- end + +local striplinepatterns = { + ["prune"] = p_prune_normal, + ["prune and collapse"] = p_prune_collapse, -- default + ["prune and no empty"] = p_prune_noempty, + ["prune and to space"] = p_prune_intospace, + ["retain"] = p_retain_normal, + ["retain and collapse"] = p_retain_collapse, + ["retain and no empty"] = p_retain_noempty, + ["collapse"] = patterns.collapser, +} + +setmetatable(striplinepatterns,{ __index = function(t,k) return p_prune_collapse end }) + +strings.striplinepatterns = striplinepatterns + +function strings.striplines(str,how) + return str and lpegmatch(striplinepatterns[how],str) or str +end + +function strings.collapse(str) -- maybe also in strings + return str and lpegmatch(p_prune_intospace,str) or str +end + +-- also see: string.collapsespaces + +strings.striplong = strings.striplines -- for old times sake + +-- local str = table.concat( { +-- " ", +-- " aap", +-- " noot mies", +-- " ", +-- " ", +-- " zus wim jet", +-- "zus wim jet", +-- " zus wim jet", +-- " ", +-- }, "\n") +-- +-- local str = table.concat( { +-- " aaaa", +-- " bb", +-- " cccccc", +-- " ", +-- }, "\n") +-- +-- for k, v in table.sortedhash(utilities.strings.striplinepatterns) do +-- logs.report("stripper","method: %s, result: [[%s]]",k,utilities.strings.striplines(str,k)) +-- end + +-- inspect(strings.striplong([[ +-- aaaa +-- bb +-- cccccc +-- ]])) + +function strings.nice(str) + str = gsub(str,"[:%-+_]+"," ") -- maybe more + return str +end + +-- Work in progress. Interesting is that compared to the built-in this is faster in +-- luatex than in luajittex where we have a comparable speed. It only makes sense +-- to use the formatter when a (somewhat) complex format is used a lot. Each formatter +-- is a function so there is some overhead and not all formatted output is worth that +-- overhead. Keep in mind that there is an extra function call involved. In principle +-- we end up with a string concatination so one could inline such a sequence but often +-- at the cost of less readabinity. So, it's a sort of (visual) compromise. Of course +-- there is the benefit of more variants. (Concerning the speed: a simple format like +-- %05fpt is better off with format than with a formatter, but as soon as you put +-- something in front formatters become faster. Passing the pt as extra argument makes +-- formatters behave better. Of course this is rather implementation dependent. Also, +-- when a specific format is only used a few times the overhead in creating it is not +-- compensated by speed.) +-- +-- More info can be found in cld-mkiv.pdf so here I stick to a simple list. +-- +-- integer %...i number +-- integer %...d number +-- unsigned %...u number -- not used +-- character %...c number +-- hexadecimal %...x number +-- HEXADECIMAL %...X number +-- octal %...o number +-- string %...s string number +-- float %...f number +-- checked float %...F number +-- exponential %...e number +-- exponential %...E number +-- stripped e %...j number +-- stripped E %...J number +-- autofloat %...g number +-- autofloat %...G number +-- utf character %...c number +-- force tostring %...S any +-- force tostring %Q any +-- force tonumber %N number (strip leading zeros) +-- signed number %I number +-- rounded number %r number +-- 0xhexadecimal %...h character number +-- 0xHEXADECIMAL %...H character number +-- U+hexadecimal %...u character number +-- U+HEXADECIMAL %...U character number +-- points %p number (scaled points) +-- basepoints %b number (scaled points) +-- table concat %...t table +-- table concat %{.}t table +-- serialize %...T sequenced (no nested tables) +-- serialize %{.}T sequenced (no nested tables) +-- boolean (logic) %l boolean +-- BOOLEAN %L boolean +-- whitespace %...w number +-- whitespace %...W (fixed) +-- automatic %...a 'whatever' (string, table, ...) +-- automatic %...A "whatever" (string, table, ...) +-- zap %...z skip +-- stripped %...N %...N +-- comma/period real %...m +-- period/comma real %...M +-- formatted float %...k n.m + +local n = 0 + +-- we are somewhat sloppy in parsing prefixes as it's not that critical + +-- hard to avoid but we can collect them in a private namespace if needed + +-- inline the next two makes no sense as we only use this in logging + +local sequenced = table.sequenced + +function string.autodouble(s,sep) + if s == nil then + return '""' + end + local t = type(s) + if t == "number" then + return tostring(s) -- tostring not really needed + end + if t == "table" then + return ('"' .. sequenced(s,sep or ",") .. '"') + end + return ('"' .. tostring(s) .. '"') +end + +function string.autosingle(s,sep) + if s == nil then + return "''" + end + local t = type(s) + if t == "number" then + return tostring(s) -- tostring not really needed + end + if t == "table" then + return ("'" .. sequenced(s,sep or ",") .. "'") + end + return ("'" .. tostring(s) .. "'") +end + +local tracedchars = { [0] = + -- the regular bunch + "[null]", "[soh]", "[stx]", "[etx]", "[eot]", "[enq]", "[ack]", "[bel]", + "[bs]", "[ht]", "[lf]", "[vt]", "[ff]", "[cr]", "[so]", "[si]", + "[dle]", "[dc1]", "[dc2]", "[dc3]", "[dc4]", "[nak]", "[syn]", "[etb]", + "[can]", "[em]", "[sub]", "[esc]", "[fs]", "[gs]", "[rs]", "[us]", + -- plus space + "[space]", -- 0x20 +} + +string.tracedchars = tracedchars +strings.tracers = tracedchars + +function string.tracedchar(b) + -- todo: table + if type(b) == "number" then + return tracedchars[b] or (utfchar(b) .. " (U+" .. format("%05X",b) .. ")") + else + local c = utfbyte(b) + return tracedchars[c] or (b .. " (U+" .. (c and format("%05X",c) or "?????") .. ")") + end +end + +function number.signed(i) + if i > 0 then + return "+", i + else + return "-", -i + end +end + +-- maybe to util-num + +local two = digit * digit +local three = two * digit +local prefix = (Carg(1) * three)^1 + +local splitter = Cs ( + (((1 - (three^1 * period))^1 + C(three)) * prefix + C((1-period)^1)) + * (anything/"" * Carg(2)) * C(2) +) + +local splitter3 = Cs ( + three * prefix * endofstring + + two * prefix * endofstring + + digit * prefix * endofstring + + three + + two + + digit +) + +patterns.formattednumber = splitter + +function number.formatted(n,sep1,sep2) + if sep1 == false then + if type(n) == "number" then + n = tostring(n) + end + return lpegmatch(splitter3,n,1,sep2 or ".") + else + if type(n) == "number" then + n = format("%0.2f",n) + end + if sep1 == true then + return lpegmatch(splitter,n,1,".",",") + elseif sep1 == "." then + return lpegmatch(splitter,n,1,sep1,sep2 or ",") + elseif sep1 == "," then + return lpegmatch(splitter,n,1,sep1,sep2 or ".") + else + return lpegmatch(splitter,n,1,sep1 or ",",sep2 or ".") + end + end +end + +-- print(number.formatted(1)) +-- print(number.formatted(12)) +-- print(number.formatted(123)) +-- print(number.formatted(1234)) +-- print(number.formatted(12345)) +-- print(number.formatted(123456)) +-- print(number.formatted(1234567)) +-- print(number.formatted(12345678)) +-- print(number.formatted(12345678,true)) +-- print(number.formatted(1,false)) +-- print(number.formatted(12,false)) +-- print(number.formatted(123,false)) +-- print(number.formatted(1234,false)) +-- print(number.formatted(12345,false)) +-- print(number.formatted(123456,false)) +-- print(number.formatted(1234567,false)) +-- print(number.formatted(12345678,false)) +-- print(number.formatted(1234.56,"!","?")) + +local p = Cs( + P("-")^0 + * (P("0")^1/"")^0 + * (1-period)^0 + * (period * P("0")^1 * endofstring/"" + period^0) + * P(1-P("0")^1*endofstring)^0 + ) + +function number.compactfloat(n,fmt) + if n == 0 then + return "0" + elseif n == 1 then + return "1" + end + n = lpegmatch(p,format(fmt or "%0.3f",n)) + if n == "." or n == "" or n == "-" then + return "0" + end + return n +end + +local zero = P("0")^1 / "" +local plus = P("+") / "" +local minus = P("-") +local separator = period +local trailing = zero^1 * #S("eE") +local exponent = (S("eE") * (plus + Cs((minus * zero^0 * endofstring)/"") + minus) * zero^0 * (endofstring * Cc("0") + anything^1)) +local pattern_a = Cs(minus^0 * digit^1 * (separator/"" * trailing + separator * (trailing + digit)^0) * exponent) +local pattern_b = Cs((exponent + anything)^0) + +function number.sparseexponent(f,n) + if not n then + n = f + f = "%e" + end + local tn = type(n) + if tn == "string" then -- cast to number + local m = tonumber(n) + if m then + return lpegmatch((f == "%e" or f == "%E") and pattern_a or pattern_b,format(f,m)) + end + elseif tn == "number" then + return lpegmatch((f == "%e" or f == "%E") and pattern_a or pattern_b,format(f,n)) + end + return tostring(n) +end + +local hf = { } +local hs = { } + +setmetatable(hf, { __index = function(t,k) + local v = "%." .. k .. "f" + t[k] = v + return v +end } ) + +setmetatable(hs, { __index = function(t,k) + local v = "%" .. k .. "s" + t[k] = v + return v +end } ) + +function number.formattedfloat(n,b,a) + local s = format(hf[a],n) + local l = (b or 0) + (a or 0) + 1 + if #s < l then + return format(hs[l],s) + else + return s + end +end + +local template = [[ +%s +%s +return function(%s) return %s end +]] + +-- this might move + +local pattern = Cs(Cc('"') * ( + (1-S('"\\\n\r'))^1 + + P('"') / '\\"' + + P('\\') / '\\\\' + + P('\n') / '\\n' + + P('\r') / '\\r' +)^0 * Cc('"')) + +patterns.escapedquotes = pattern + +function string.escapedquotes(s) + return lpegmatch(pattern,s) +end + +-- print(string.escapedquotes('1\\23\n"')) + +-- but for now here + +local preamble = "" + +local environment = { + global = global or _G, + lpeg = lpeg, + type = type, + tostring = tostring, + tonumber = tonumber, + format = string.format, + concat = table.concat, + signed = number.signed, + points = number.points, + basepoints = number.basepoints, + utfchar = utf.char, + utfbyte = utf.byte, + lpegmatch = lpeg.match, + nspaces = string.nspaces, + utfpadding = string.utfpadding, + tracedchar = string.tracedchar, + autosingle = string.autosingle, + autodouble = string.autodouble, + sequenced = table.sequenced, + formattednumber = number.formatted, + sparseexponent = number.sparseexponent, + formattedfloat = number.formattedfloat, + stripzero = patterns.stripzero, + stripzeros = patterns.stripzeros, + escapedquotes = string.escapedquotes, + + FORMAT = string.f9, +} + +-- -- -- + +local arguments = { "a1" } -- faster than previously used (select(n,...)) + +setmetatable(arguments, { __index = + function(t,k) + local v = t[k-1] .. ",a" .. k + t[k] = v + return v + end +}) + +local prefix_any = C((sign + space + period + digit)^0) +local prefix_sub = (C((sign + digit)^0) + Cc(0)) + * period + * (C((sign + digit)^0) + Cc(0)) +local prefix_tab = P("{") * C((1-P("}"))^0) * P("}") + C((1-R("az","AZ","09","%%"))^0) + +-- we've split all cases as then we can optimize them (let's omit the fuzzy u) + +-- todo: replace outer formats in next by .. + +local format_s = function(f) + n = n + 1 + if f and f ~= "" then + return format("format('%%%ss',a%s)",f,n) + else -- best no tostring in order to stay compatible (.. does a selective tostring too) + return format("(a%s or '')",n) -- goodie: nil check + end +end + +local format_S = function(f) -- can be optimized + n = n + 1 + if f and f ~= "" then + return format("format('%%%ss',tostring(a%s))",f,n) + else + return format("tostring(a%s)",n) + end +end + +local format_right = function(f) + n = n + 1 + f = tonumber(f) + if not f or f == 0 then + return format("(a%s or '')",n) + elseif f > 0 then + return format("utfpadding(a%s,%i)..a%s",n,f,n) + else + return format("a%s..utfpadding(a%s,%i)",n,n,f) + end +end + +local format_left = function(f) + n = n + 1 + f = tonumber(f) + if not f or f == 0 then + return format("(a%s or '')",n) + end + if f < 0 then + return format("utfpadding(a%s,%i)..a%s",n,-f,n) + else + return format("a%s..utfpadding(a%s,%i)",n,n,-f) + end +end + +local format_q = function() + n = n + 1 + -- lua 5.3 has a different q than lua 5.2 (which does a tostring on numbers) + -- return format("(a%s ~= nil and format('%%q',a%s) or '')",n,n) + return format("(a%s ~= nil and format('%%q',tostring(a%s)) or '')",n,n) + -- return format("(a%s ~= nil and escapedquotes(tostring(a%s)) or '')",n,n) +end + +local format_Q = function() -- fast escaping + n = n + 1 +-- return format("format('%%q',tostring(a%s))",n) + return format("escapedquotes(tostring(a%s))",n) +end + +local format_i = function(f) + n = n + 1 + if f and f ~= "" then + return format("format('%%%si',a%s)",f,n) + else + return format("format('%%i',a%s)",n) -- why not just tostring() + end +end + +local format_d = format_i + +local format_I = function(f) + n = n + 1 + return format("format('%%s%%%si',signed(a%s))",f,n) +end + +local format_f = function(f) + n = n + 1 + return format("format('%%%sf',a%s)",f,n) +end + +-- The next one formats an integer as integer and very small values as zero. This is needed +-- for pdf backend code. +-- +-- 1.23 % 1 : 0.23 +-- - 1.23 % 1 : 0.77 +-- +-- We could probably use just %s with integers but who knows what Lua 5.3 will do? So let's +-- for the moment use %i. + +local format_F = function(f) -- beware, no cast to number + n = n + 1 + if not f or f == "" then + return format("(((a%s > -0.0000000005 and a%s < 0.0000000005) and '0') or format((a%s %% 1 == 0) and '%%i' or '%%.9f',a%s))",n,n,n,n) + else + return format("format((a%s %% 1 == 0) and '%%i' or '%%%sf',a%s)",n,f,n) + end +end + +-- if string.f9 then +-- format_F = function(f) -- beware, no cast to number +-- n = n + 1 +-- if not f or f == "" then +-- return format("(((a%s > -0.0000000005 and a%s < 0.0000000005) and '0') or FORMAT(a%s))",n,n,n,n,n) +-- else +-- return format("((a%s %% 1 == 0) and format('%%i',a%s) or FORMAT(a%s,'%%%sf'))",n,n,n,f) +-- end +-- end +-- end + +local format_k = function(b,a) -- slow + n = n + 1 + return format("formattedfloat(a%s,%s,%s)",n,b or 0,a or 0) +end + +local format_g = function(f) + n = n + 1 + return format("format('%%%sg',a%s)",f,n) +end + +local format_G = function(f) + n = n + 1 + return format("format('%%%sG',a%s)",f,n) +end + +local format_e = function(f) + n = n + 1 + return format("format('%%%se',a%s)",f,n) +end + +local format_E = function(f) + n = n + 1 + return format("format('%%%sE',a%s)",f,n) +end + +local format_j = function(f) + n = n + 1 + return format("sparseexponent('%%%se',a%s)",f,n) +end + +local format_J = function(f) + n = n + 1 + return format("sparseexponent('%%%sE',a%s)",f,n) +end + +local format_x = function(f) + n = n + 1 + return format("format('%%%sx',a%s)",f,n) +end + +local format_X = function(f) + n = n + 1 + return format("format('%%%sX',a%s)",f,n) +end + +local format_o = function(f) + n = n + 1 + return format("format('%%%so',a%s)",f,n) +end + +local format_c = function() + n = n + 1 + return format("utfchar(a%s)",n) +end + +local format_C = function() + n = n + 1 + return format("tracedchar(a%s)",n) +end + +local format_r = function(f) + n = n + 1 + return format("format('%%%s.0f',a%s)",f,n) +end + +local format_h = function(f) + n = n + 1 + if f == "-" then + f = sub(f,2) + return format("format('%%%sx',type(a%s) == 'number' and a%s or utfbyte(a%s))",f == "" and "05" or f,n,n,n) + else + return format("format('0x%%%sx',type(a%s) == 'number' and a%s or utfbyte(a%s))",f == "" and "05" or f,n,n,n) + end +end + +local format_H = function(f) + n = n + 1 + if f == "-" then + f = sub(f,2) + return format("format('%%%sX',type(a%s) == 'number' and a%s or utfbyte(a%s))",f == "" and "05" or f,n,n,n) + else + return format("format('0x%%%sX',type(a%s) == 'number' and a%s or utfbyte(a%s))",f == "" and "05" or f,n,n,n) + end +end + +local format_u = function(f) + n = n + 1 + if f == "-" then + f = sub(f,2) + return format("format('%%%sx',type(a%s) == 'number' and a%s or utfbyte(a%s))",f == "" and "05" or f,n,n,n) + else + return format("format('u+%%%sx',type(a%s) == 'number' and a%s or utfbyte(a%s))",f == "" and "05" or f,n,n,n) + end +end + +local format_U = function(f) + n = n + 1 + if f == "-" then + f = sub(f,2) + return format("format('%%%sX',type(a%s) == 'number' and a%s or utfbyte(a%s))",f == "" and "05" or f,n,n,n) + else + return format("format('U+%%%sX',type(a%s) == 'number' and a%s or utfbyte(a%s))",f == "" and "05" or f,n,n,n) + end +end + +local format_p = function() + n = n + 1 + return format("points(a%s)",n) +end + +local format_b = function() + n = n + 1 + return format("basepoints(a%s)",n) +end + +local format_t = function(f) + n = n + 1 + if f and f ~= "" then + return format("concat(a%s,%q)",n,f) + else + return format("concat(a%s)",n) + end +end + +local format_T = function(f) + n = n + 1 + if f and f ~= "" then + return format("sequenced(a%s,%q)",n,f) + else + return format("sequenced(a%s)",n) + end +end + +local format_l = function() + n = n + 1 + return format("(a%s and 'true' or 'false')",n) +end + +local format_L = function() + n = n + 1 + return format("(a%s and 'TRUE' or 'FALSE')",n) +end + +local format_n = function() -- strips leading and trailing zeros and removes .0 + n = n + 1 + return format("((a%s %% 1 == 0) and format('%%i',a%s) or tostring(a%s))",n,n,n) +end + +-- local format_N = function() -- strips leading and trailing zeros (also accepts string) +-- n = n + 1 +-- return format("tostring(tonumber(a%s) or a%s)",n,n) +-- end + +-- local format_N = function(f) -- strips leading and trailing zeros +-- n = n + 1 +-- -- stripzero (singular) as we only have a number +-- if not f or f == "" then +-- return format("(((a%s > -0.0000000005 and a%s < 0.0000000005) and '0') or ((a%s %% 1 == 0) and format('%%i',a%s)) or lpegmatch(stripzero,format('%%.9f',a%s)))",n,n,n,n,n) +-- else +-- return format("(((a%s %% 1 == 0) and format('%%i',a%s)) or lpegmatch(stripzero,format('%%%sf',a%s)))",n,n,f,n) +-- end +-- end + +-- local format_N = function(f) -- strips leading and trailing zeros +-- n = n + 1 +-- -- stripzero (singular) as we only have a number +-- if not f or f == "" then +-- return format("(((a%s %% 1 == 0) and format('%%i',a%s)) or ((a%s > -0.0000000005 and a%s < 0.0000000005) and '0') or lpegmatch(stripzero,format('%%.9f',a%s)))",n,n,n,n,n) +-- else +-- return format("(((a%s %% 1 == 0) and format('%%i',a%s)) or lpegmatch(stripzero,format('%%%sf',a%s)))",n,n,f,n) +-- end +-- end + +local format_N = function(f) -- strips leading and trailing zeros + n = n + 1 + -- stripzero (singular) as we only have a number + if not f or f == "" then + f = ".9" + end -- always a leading number ! + return format("(((a%s %% 1 == 0) and format('%%i',a%s)) or lpegmatch(stripzero,format('%%%sf',a%s)))",n,n,f,n) +end + +local format_a = function(f) + n = n + 1 + if f and f ~= "" then + return format("autosingle(a%s,%q)",n,f) + else + return format("autosingle(a%s)",n) + end +end + +local format_A = function(f) + n = n + 1 + if f and f ~= "" then + return format("autodouble(a%s,%q)",n,f) + else + return format("autodouble(a%s)",n) + end +end + +local format_w = function(f) -- handy when doing depth related indent + n = n + 1 + f = tonumber(f) + if f then -- not that useful + return format("nspaces[%s+a%s]",f,n) -- no real need for tonumber + else + return format("nspaces[a%s]",n) -- no real need for tonumber + end +end + +local format_W = function(f) -- handy when doing depth related indent + return format("nspaces[%s]",tonumber(f) or 0) +end + +local format_m = function(f) + n = n + 1 + if not f or f == "" then + f = "," + end + if f == "0" then + return format([[formattednumber(a%s,false)]],n) + else + return format([[formattednumber(a%s,%q,".")]],n,f) + end +end + +local format_M = function(f) + n = n + 1 + if not f or f == "" then + f = "." + end + if f == "0" then + return format([[formattednumber(a%s,false)]],n) + else + return format([[formattednumber(a%s,%q,",")]],n,f) + end +end + +-- + +local format_z = function(f) + n = n + (tonumber(f) or 1) + return "''" -- okay, not that efficient to append '' but a special case anyway +end + +-- + +-- local strip +-- +-- local format_Z = function(f) +-- n = n + 1 +-- if not f or f == "" then +-- f = ".9" +-- end +-- return format("(((a%s %% 1 == 0) and format('%%i',a%s)) or (strip and lpegmatch(stripzero,format('%%%sf',a%s))) or format('%%%sf',a%s))",n,n,f,n,f,n) +-- end +-- +-- function strings.stripformatterzeros() +-- strip = true +-- end + +-- add(formatters,"texexp", [[texexp(...)]], "local texexp = metapost.texexp") +-- +-- add(formatters,"foo:bar",[[foo(...)]], { foo = function(...) print(...) return "!" end }) +-- print(string.formatters["foo %3!foo:bar! bar"](1,2,3)) + + +local format_rest = function(s) + return format("%q",s) -- catches " and \n and such +end + +-- local format_extension = function(extensions,f,name) +-- local extension = extensions[name] or "tostring(%s)" +-- local f = tonumber(f) or 1 +-- local w = find(extension,"%.%.%.") +-- if f == 0 then +-- if w then +-- extension = gsub(extension,"%.%.%.","") +-- end +-- return extension +-- elseif f == 1 then +-- if w then +-- extension = gsub(extension,"%.%.%.","%%s") +-- end +-- n = n + 1 +-- local a = "a" .. n +-- return format(extension,a,a) -- maybe more times? +-- elseif f < 0 then +-- local a = "a" .. (n + f + 1) +-- return format(extension,a,a) +-- else +-- if w then +-- extension = gsub(extension,"%.%.%.",rep("%%s,",f-1).."%%s") +-- end +-- -- we could fill an array and then n = n + 1 unpack(t,n,n+f) but as we +-- -- cache we don't save much and there are hardly any extensions anyway +-- local t = { } +-- for i=1,f do +-- n = n + 1 +-- -- t[#t+1] = "a" .. n +-- t[i] = "a" .. n +-- end +-- return format(extension,unpack(t)) +-- end +-- end + +local format_extension = function(extensions,f,name) + local extension = extensions[name] or "tostring(%s)" + local f = tonumber(f) or 1 + local w = find(extension,"%.%.%.") + if w then + -- we have a wildcard + if f == 0 then + extension = gsub(extension,"%.%.%.","") + return extension + elseif f == 1 then + extension = gsub(extension,"%.%.%.","%%s") + n = n + 1 + local a = "a" .. n + return format(extension,a,a) -- maybe more times? + elseif f < 0 then + local a = "a" .. (n + f + 1) + return format(extension,a,a) + else + extension = gsub(extension,"%.%.%.",rep("%%s,",f-1).."%%s") + -- we could fill an array and then n = n + 1 unpack(t,n,n+f) but as we + -- cache we don't save much and there are hardly any extensions anyway + local t = { } + for i=1,f do + n = n + 1 + -- t[#t+1] = "a" .. n + t[i] = "a" .. n + end + return format(extension,unpack(t)) + end + else + extension = gsub(extension,"%%s",function() + n = n + 1 + return "a" .. n + end) + return extension + end +end + +-- aA b cC d eE f gG hH iI jJ lL mM N o p qQ r sS tT uU wW xX z + +-- extensions : %!tag! + +-- can be made faster but not called that often + +local builder = Cs { "start", + start = ( + ( + P("%") / "" + * ( + V("!") -- new + + V("s") + V("q") + + V("i") + V("d") + + V("f") + V("F") + V("g") + V("G") + V("e") + V("E") + + V("x") + V("X") + V("o") + -- + + V("c") + + V("C") + + V("S") -- new + + V("Q") -- new + + V("n") -- new + + V("N") -- new + + V("k") -- new + -- + + V("r") + + V("h") + V("H") + V("u") + V("U") + + V("p") + V("b") + + V("t") + V("T") + + V("l") + V("L") + + V("I") + + V("w") -- new + + V("W") -- new + + V("a") -- new + + V("A") -- new + + V("j") + V("J") -- stripped e E + + V("m") + V("M") -- new (formatted number) + + V("z") -- new + -- + + V(">") -- left padding + + V("<") -- right padding + -- + -- + V("?") -- ignored, probably messed up % + ) + + V("*") + ) + * (endofstring + Carg(1)) + )^0, + -- + ["s"] = (prefix_any * P("s")) / format_s, -- %s => regular %s (string) + ["q"] = (prefix_any * P("q")) / format_q, -- %q => regular %q (quoted string) + ["i"] = (prefix_any * P("i")) / format_i, -- %i => regular %i (integer) + ["d"] = (prefix_any * P("d")) / format_d, -- %d => regular %d (integer) + ["f"] = (prefix_any * P("f")) / format_f, -- %f => regular %f (float) + ["F"] = (prefix_any * P("F")) / format_F, -- %F => regular %f (float) but 0/1 check + ["g"] = (prefix_any * P("g")) / format_g, -- %g => regular %g (float) + ["G"] = (prefix_any * P("G")) / format_G, -- %G => regular %G (float) + ["e"] = (prefix_any * P("e")) / format_e, -- %e => regular %e (float) + ["E"] = (prefix_any * P("E")) / format_E, -- %E => regular %E (float) + ["x"] = (prefix_any * P("x")) / format_x, -- %x => regular %x (hexadecimal) + ["X"] = (prefix_any * P("X")) / format_X, -- %X => regular %X (HEXADECIMAL) + ["o"] = (prefix_any * P("o")) / format_o, -- %o => regular %o (octal) + -- + ["S"] = (prefix_any * P("S")) / format_S, -- %S => %s (tostring) + ["Q"] = (prefix_any * P("Q")) / format_Q, -- %Q => %q (tostring) + ["n"] = (prefix_any * P("n")) / format_n, -- %n => tonumber (strips leading and trailing zeros, as well as .0, expects number) + ["N"] = (prefix_any * P("N")) / format_N, -- %N => tonumber (strips leading and trailing zeros, also takes string) + ["k"] = (prefix_sub * P("k")) / format_k, -- %k => like f but with n.m + ["c"] = (prefix_any * P("c")) / format_c, -- %c => utf character (extension to regular) + ["C"] = (prefix_any * P("C")) / format_C, -- %c => U+.... utf character + -- + ["r"] = (prefix_any * P("r")) / format_r, -- %r => round + ["h"] = (prefix_any * P("h")) / format_h, -- %h => 0x0a1b2 (when - no 0x) was v + ["H"] = (prefix_any * P("H")) / format_H, -- %H => 0x0A1B2 (when - no 0x) was V + ["u"] = (prefix_any * P("u")) / format_u, -- %u => u+0a1b2 (when - no u+) + ["U"] = (prefix_any * P("U")) / format_U, -- %U => U+0A1B2 (when - no U+) + ["p"] = (prefix_any * P("p")) / format_p, -- %p => 12.345pt / maybe: P (and more units) + ["b"] = (prefix_any * P("b")) / format_b, -- %b => 12.342bp / maybe: B (and more units) + ["t"] = (prefix_tab * P("t")) / format_t, -- %t => concat + ["T"] = (prefix_tab * P("T")) / format_T, -- %t => sequenced + ["l"] = (prefix_any * P("l")) / format_l, -- %l => boolean + ["L"] = (prefix_any * P("L")) / format_L, -- %L => BOOLEAN + ["I"] = (prefix_any * P("I")) / format_I, -- %I => signed integer + -- + ["w"] = (prefix_any * P("w")) / format_w, -- %w => n spaces (optional prefix is added) + ["W"] = (prefix_any * P("W")) / format_W, -- %W => mandate prefix, no specifier + -- + ["j"] = (prefix_any * P("j")) / format_j, -- %j => %e (float) stripped exponent (irrational) + ["J"] = (prefix_any * P("J")) / format_J, -- %J => %E (float) stripped exponent (irrational) + -- + ["m"] = (prefix_any * P("m")) / format_m, -- %m => xxx.xxx.xxx,xx (optional prefix instead of .) + ["M"] = (prefix_any * P("M")) / format_M, -- %M => xxx,xxx,xxx.xx (optional prefix instead of ,) + -- + ["z"] = (prefix_any * P("z")) / format_z, -- %z => skip n arguments + -- ["Z"] = (prefix_any * P("Z")) / format_Z, -- %Z => optionally strip zeros + -- + ["a"] = (prefix_any * P("a")) / format_a, -- %a => '...' (forces tostring) + ["A"] = (prefix_any * P("A")) / format_A, -- %A => "..." (forces tostring) + -- + ["<"] = (prefix_any * P("<")) / format_left, + [">"] = (prefix_any * P(">")) / format_right, + -- + ["*"] = Cs(((1-P("%"))^1 + P("%%")/"%%")^1) / format_rest, -- rest (including %%) + ["?"] = Cs(((1-P("%"))^1 )^1) / format_rest, -- rest (including %%) + -- + ["!"] = Carg(2) * prefix_any * P("!") * C((1-P("!"))^1) * P("!") / format_extension, +} + +-- We can be clever and only alias what is needed: + +local xx = setmetatable({ }, { __index = function(t,k) local v = format("%02x",k) t[k] = v return v end }) +local XX = setmetatable({ }, { __index = function(t,k) local v = format("%02X",k) t[k] = v return v end }) + +local preset = { + ["%02x"] = function(n) return xx[n] end, + ["%02X"] = function(n) return XX[n] end, +} + +local direct = + P("%") * (sign + space + period + digit)^0 * S("sqidfgGeExXo") * endofstring + / [[local format = string.format return function(str) return format("%0",str) end]] + +local function make(t,str) + local f = preset[str] + if f then + return f + end + local p = lpegmatch(direct,str) + if p then + -- print("builder 1 >",p) + f = loadstripped(p)() + else + n = 0 -- used in patterns + -- p = lpegmatch(builder,str,1,"..",t._extensions_) -- after this we know n + p = lpegmatch(builder,str,1,t._connector_,t._extensions_) -- after this we know n + if n > 0 then + p = format(template,preamble,t._preamble_,arguments[n],p) + -- print("builder 2 >",p) + f = loadstripped(p,t._environment_)() -- t._environment is not populated (was experiment) + else + f = function() return str end + end + end + -- if jit then jit.on(f,true) end + t[str] = f + return f +end + +-- -- collect periodically +-- +-- local threshold = 1000 -- max nof cached formats +-- +-- local function make(t,str) +-- local f = rawget(t,str) +-- if f then +-- return f +-- end +-- local parent = t._t_ +-- if parent._n_ > threshold then +-- local m = { _t_ = parent } +-- getmetatable(parent).__index = m +-- setmetatable(m, { __index = make }) +-- else +-- parent._n_ = parent._n_ + 1 +-- end +-- local f +-- local p = lpegmatch(direct,str) +-- if p then +-- f = loadstripped(p)() +-- else +-- n = 0 +-- p = lpegmatch(builder,str,1,"..",parent._extensions_) -- after this we know n +-- if n > 0 then +-- p = format(template,preamble,parent._preamble_,arguments[n],p) +-- -- print("builder>",p) +-- f = loadstripped(p)() +-- else +-- f = function() return str end +-- end +-- end +-- t[str] = f +-- return f +-- end + +local function use(t,fmt,...) + return t[fmt](...) +end + +strings.formatters = { } + +-- we cannot make these tables weak, unless we start using an indirect +-- table (metatable) in which case we could better keep a count and +-- clear that table when a threshold is reached + +-- _connector_ is an experiment + +function strings.formatters.new(noconcat) + local e = { } -- better make a copy as we can overload + for k, v in next, environment do + e[k] = v + end + local t = { + _type_ = "formatter", + _connector_ = noconcat and "," or "..", + _extensions_ = { }, + _preamble_ = "", + _environment_ = e, + } + setmetatable(t, { __index = make, __call = use }) + return t +end + +local formatters = strings.formatters.new() -- the default instance + +string.formatters = formatters -- in the main string namespace +string.formatter = function(str,...) return formatters[str](...) end -- sometimes nicer name + +local function add(t,name,template,preamble) + if type(t) == "table" and t._type_ == "formatter" then + t._extensions_[name] = template or "%s" + if type(preamble) == "string" then + t._preamble_ = preamble .. "\n" .. t._preamble_ -- so no overload ! + elseif type(preamble) == "table" then + for k, v in next, preamble do + t._environment_[k] = v + end + end + end +end + +strings.formatters.add = add + +-- registered in the default instance (should we fall back on this one?) + +patterns.xmlescape = Cs((P("<")/"<" + P(">")/">" + P("&")/"&" + P('"')/""" + anything)^0) +patterns.texescape = Cs((C(S("#$%\\{}"))/"\\%1" + anything)^0) +patterns.luaescape = Cs(((1-S('"\n'))^1 + P('"')/'\\"' + P('\n')/'\\n"')^0) -- maybe also \0 +patterns.luaquoted = Cs(Cc('"') * ((1-S('"\n'))^1 + P('"')/'\\"' + P('\n')/'\\n"')^0 * Cc('"')) + +-- escaping by lpeg is faster for strings without quotes, slower on a string with quotes, but +-- faster again when other q-escapables are found (the ones we don't need to escape) + +add(formatters,"xml",[[lpegmatch(xmlescape,%s)]],{ xmlescape = patterns.xmlescape }) +add(formatters,"tex",[[lpegmatch(texescape,%s)]],{ texescape = patterns.texescape }) +add(formatters,"lua",[[lpegmatch(luaescape,%s)]],{ luaescape = patterns.luaescape }) + +-- -- yes or no: +-- +-- local function make(t,str) +-- local f +-- local p = lpegmatch(direct,str) +-- if p then +-- f = loadstripped(p)() +-- else +-- n = 0 +-- p = lpegmatch(builder,str,1,",") -- after this we know n +-- if n > 0 then +-- p = format(template,template_shortcuts,arguments[n],p) +-- f = loadstripped(p)() +-- else +-- f = function() return str end +-- end +-- end +-- t[str] = f +-- return f +-- end +-- +-- local formatteds = string.formatteds or { } +-- string.formatteds = formatteds +-- +-- setmetatable(formatteds, { __index = make, __call = use }) + +-- This is a somewhat silly one used in commandline reconstruction but the older +-- method, using a combination of fine, gsub, quoted and unquoted was not that +-- reliable. +-- +-- '"foo"bar \"and " whatever"' => "foo\"bar \"and \" whatever" +-- 'foo"bar \"and " whatever' => "foo\"bar \"and \" whatever" + +local dquote = patterns.dquote -- P('"') +local equote = patterns.escaped + dquote / '\\"' + 1 +local cquote = Cc('"') + +local pattern = + Cs(dquote * (equote - P(-2))^0 * dquote) -- we keep the outer but escape unescaped ones + + Cs(cquote * (equote - space)^0 * space * equote^0 * cquote) -- we escape unescaped ones + +function string.optionalquoted(str) + return lpegmatch(pattern,str) or str +end + +local pattern = Cs((newline / (os.newline or "\r") + 1)^0) + +function string.replacenewlines(str) + return lpegmatch(pattern,str) +end + +-- + +function strings.newcollector() + local result, r = { }, 0 + return + function(fmt,str,...) -- write + r = r + 1 + result[r] = str == nil and fmt or formatters[fmt](str,...) + end, + function(connector) -- flush + if result then + local str = concat(result,connector) + result, r = { }, 0 + return str + end + end +end + +-- + +local f_16_16 = formatters["%0.5N"] + +function number.to16dot16(n) + return f_16_16(n/65536.0) +end + +-- + +if not string.explode then + + local tsplitat = lpeg.tsplitat + + local p_utf = patterns.utf8character + local p_check = C(p_utf) * (P("+") * Cc(true))^0 + local p_split = Ct(C(p_utf)^0) + local p_space = Ct((C(1-P(" ")^1) + P(" ")^1)^0) + + function string.explode(str,symbol) + if symbol == "" then + return lpegmatch(p_split,str) + elseif symbol then + local a, b = lpegmatch(p_check,symbol) + if b then + return lpegmatch(tsplitat(P(a)^1),str) + else + return lpegmatch(tsplitat(a),str) + end + else + return lpegmatch(p_space,str) + end + end + +end diff --git a/macros/luatex/generic/lualibs/lualibs-util-tab.lua b/macros/luatex/generic/lualibs/lualibs-util-tab.lua new file mode 100644 index 0000000000..f970d8ace0 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-util-tab.lua @@ -0,0 +1,874 @@ +if not modules then modules = { } end modules ['util-tab'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +utilities = utilities or {} +utilities.tables = utilities.tables or { } +local tables = utilities.tables + +local format, gmatch, gsub, sub = string.format, string.gmatch, string.gsub, string.sub +local concat, insert, remove, sort = table.concat, table.insert, table.remove, table.sort +local setmetatable, getmetatable, tonumber, tostring, rawget = setmetatable, getmetatable, tonumber, tostring, rawget +local type, next, rawset, tonumber, tostring, load, select = type, next, rawset, tonumber, tostring, load, select +local lpegmatch, P, Cs, Cc = lpeg.match, lpeg.P, lpeg.Cs, lpeg.Cc +local sortedkeys, sortedpairs = table.sortedkeys, table.sortedpairs +local formatters = string.formatters +local utftoeight = utf.toeight + +local splitter = lpeg.tsplitat(".") + +function utilities.tables.definetable(target,nofirst,nolast) -- defines undefined tables + local composed = nil + local t = { } + local snippets = lpegmatch(splitter,target) + for i=1,#snippets - (nolast and 1 or 0) do + local name = snippets[i] + if composed then + composed = composed .. "." .. name + t[#t+1] = formatters["if not %s then %s = { } end"](composed,composed) + else + composed = name + if not nofirst then + t[#t+1] = formatters["%s = %s or { }"](composed,composed) + end + end + end + if composed then + if nolast then + composed = composed .. "." .. snippets[#snippets] + end + return concat(t,"\n"), composed -- could be shortcut + else + return "", target + end +end + +-- local t = tables.definedtable("a","b","c","d") + +function tables.definedtable(...) + local t = _G + for i=1,select("#",...) do + local li = select(i,...) + local tl = t[li] + if not tl then + tl = { } + t[li] = tl + end + t = tl + end + return t +end + +function tables.accesstable(target,root) + local t = root or _G + for name in gmatch(target,"([^%.]+)") do + t = t[name] + if not t then + return + end + end + return t +end + +function tables.migratetable(target,v,root) + local t = root or _G + local names = lpegmatch(splitter,target) + for i=1,#names-1 do + local name = names[i] + t[name] = t[name] or { } + t = t[name] + if not t then + return + end + end + t[names[#names]] = v +end + +function tables.removevalue(t,value) -- todo: n + if value then + for i=1,#t do + if t[i] == value then + remove(t,i) + -- remove all, so no: return + end + end + end +end + +function tables.replacevalue(t,oldvalue,newvalue) + if oldvalue and newvalue then + for i=1,#t do + if t[i] == oldvalue then + t[i] = newvalue + -- replace all, so no: return + end + end + end +end + +function tables.insertbeforevalue(t,value,extra) + for i=1,#t do + if t[i] == extra then + remove(t,i) + end + end + for i=1,#t do + if t[i] == value then + insert(t,i,extra) + return + end + end + insert(t,1,extra) +end + +function tables.insertaftervalue(t,value,extra) + for i=1,#t do + if t[i] == extra then + remove(t,i) + end + end + for i=1,#t do + if t[i] == value then + insert(t,i+1,extra) + return + end + end + insert(t,#t+1,extra) +end + +-- experimental + +local escape = Cs(Cc('"') * ((P('"')/'""' + P(1))^0) * Cc('"')) + +function table.tocsv(t,specification) + if t and #t > 0 then + local result = { } + local r = { } + specification = specification or { } + local fields = specification.fields + if type(fields) ~= "string" then + fields = sortedkeys(t[1]) + end + local separator = specification.separator or "," + local noffields = #fields + if specification.preamble == true then + for f=1,noffields do + r[f] = lpegmatch(escape,tostring(fields[f])) + end + result[1] = concat(r,separator) + end + for i=1,#t do + local ti = t[i] + for f=1,noffields do + local field = ti[fields[f]] + if type(field) == "string" then + r[f] = lpegmatch(escape,field) + else + r[f] = tostring(field) + end + end + -- result[#result+1] = concat(r,separator) + result[i+1] = concat(r,separator) + end + return concat(result,"\n") + else + return "" + end +end + +-- local nspaces = utilities.strings.newrepeater(" ") +-- local escape = Cs((P("<")/"<" + P(">")/">" + P("&")/"&" + P(1))^0) +-- +-- local function toxml(t,d,result,step) +-- for k, v in sortedpairs(t) do +-- local s = nspaces[d] +-- local tk = type(k) +-- local tv = type(v) +-- if tv == "table" then +-- if tk == "number" then +-- result[#result+1] = format("%s<entry n='%s'>",s,k) +-- toxml(v,d+step,result,step) +-- result[#result+1] = format("%s</entry>",s,k) +-- else +-- result[#result+1] = format("%s<%s>",s,k) +-- toxml(v,d+step,result,step) +-- result[#result+1] = format("%s</%s>",s,k) +-- end +-- elseif tv == "string" then +-- if tk == "number" then +-- result[#result+1] = format("%s<entry n='%s'>%s</entry>",s,k,lpegmatch(escape,v),k) +-- else +-- result[#result+1] = format("%s<%s>%s</%s>",s,k,lpegmatch(escape,v),k) +-- end +-- elseif tk == "number" then +-- result[#result+1] = format("%s<entry n='%s'>%s</entry>",s,k,tostring(v),k) +-- else +-- result[#result+1] = format("%s<%s>%s</%s>",s,k,tostring(v),k) +-- end +-- end +-- end +-- +-- much faster + +local nspaces = utilities.strings.newrepeater(" ") + +local function toxml(t,d,result,step) + local r = #result + for k, v in sortedpairs(t) do + local s = nspaces[d] -- inlining this is somewhat faster but gives more formatters + local tk = type(k) + local tv = type(v) + if tv == "table" then + if tk == "number" then + r = r + 1 result[r] = formatters["%s<entry n='%s'>"](s,k) + toxml(v,d+step,result,step) + r = r + 1 result[r] = formatters["%s</entry>"](s,k) + else + r = r + 1 result[r] = formatters["%s<%s>"](s,k) + toxml(v,d+step,result,step) + r = r + 1 result[r] = formatters["%s</%s>"](s,k) + end + elseif tv == "string" then + if tk == "number" then + r = r + 1 result[r] = formatters["%s<entry n='%s'>%!xml!</entry>"](s,k,v,k) + else + r = r + 1 result[r] = formatters["%s<%s>%!xml!</%s>"](s,k,v,k) + end + elseif tk == "number" then + r = r + 1 result[r] = formatters["%s<entry n='%s'>%S</entry>"](s,k,v,k) + else + r = r + 1 result[r] = formatters["%s<%s>%S</%s>"](s,k,v,k) + end + end +end + +-- function table.toxml(t,name,nobanner,indent,spaces) +-- local noroot = name == false +-- local result = (nobanner or noroot) and { } or { "<?xml version='1.0' standalone='yes' ?>" } +-- local indent = rep(" ",indent or 0) +-- local spaces = rep(" ",spaces or 1) +-- if noroot then +-- toxml( t, inndent, result, spaces) +-- else +-- toxml( { [name or "root"] = t }, indent, result, spaces) +-- end +-- return concat(result,"\n") +-- end + +function table.toxml(t,specification) + specification = specification or { } + local name = specification.name + local noroot = name == false + local result = (specification.nobanner or noroot) and { } or { "<?xml version='1.0' standalone='yes' ?>" } + local indent = specification.indent or 0 + local spaces = specification.spaces or 1 + if noroot then + toxml( t, indent, result, spaces) + else + toxml( { [name or "data"] = t }, indent, result, spaces) + end + return concat(result,"\n") +end + +-- also experimental + +-- encapsulate(table,utilities.tables) +-- encapsulate(table,utilities.tables,true) +-- encapsulate(table,true) + +function tables.encapsulate(core,capsule,protect) + if type(capsule) ~= "table" then + protect = true + capsule = { } + end + for key, value in next, core do + if capsule[key] then + print(formatters["\ninvalid %s %a in %a"]("inheritance",key,core)) + os.exit() + else + capsule[key] = value + end + end + if protect then + for key, value in next, core do + core[key] = nil + end + setmetatable(core, { + __index = capsule, + __newindex = function(t,key,value) + if capsule[key] then + print(formatters["\ninvalid %s %a' in %a"]("overload",key,core)) + os.exit() + else + rawset(t,key,value) + end + end + } ) + end +end + +-- best keep [%q] keys (as we have some in older applications i.e. saving user data (otherwise +-- we also need to check for reserved words) + +local f_hashed_string = formatters["[%Q]=%Q,"] +local f_hashed_number = formatters["[%Q]=%s,"] +local f_hashed_boolean = formatters["[%Q]=%l,"] +local f_hashed_table = formatters["[%Q]="] + +local f_indexed_string = formatters["[%s]=%Q,"] +local f_indexed_number = formatters["[%s]=%s,"] +local f_indexed_boolean = formatters["[%s]=%l,"] +local f_indexed_table = formatters["[%s]="] + +local f_ordered_string = formatters["%Q,"] +local f_ordered_number = formatters["%s,"] +local f_ordered_boolean = formatters["%l,"] + +function table.fastserialize(t,prefix) -- todo, move local function out + + -- prefix should contain the = + -- not sorted + -- only number and string indices (currently) + + local r = { type(prefix) == "string" and prefix or "return" } + local m = 1 + local function fastserialize(t,outer) -- no mixes + local n = #t + m = m + 1 + r[m] = "{" + if n > 0 then + for i=0,n do + local v = t[i] + local tv = type(v) + if tv == "string" then + m = m + 1 r[m] = f_ordered_string(v) + elseif tv == "number" then + m = m + 1 r[m] = f_ordered_number(v) + elseif tv == "table" then + fastserialize(v) + elseif tv == "boolean" then + m = m + 1 r[m] = f_ordered_boolean(v) + end + end + end + for k, v in next, t do + local tk = type(k) + if tk == "number" then + if k > n or k < 0 then + local tv = type(v) + if tv == "string" then + m = m + 1 r[m] = f_indexed_string(k,v) + elseif tv == "number" then + m = m + 1 r[m] = f_indexed_number(k,v) + elseif tv == "table" then + m = m + 1 r[m] = f_indexed_table(k) + fastserialize(v) + elseif tv == "boolean" then + m = m + 1 r[m] = f_indexed_boolean(k,v) + end + end + else + local tv = type(v) + if tv == "string" then + m = m + 1 r[m] = f_hashed_string(k,v) + elseif tv == "number" then + m = m + 1 r[m] = f_hashed_number(k,v) + elseif tv == "table" then + m = m + 1 r[m] = f_hashed_table(k) + fastserialize(v) + elseif tv == "boolean" then + m = m + 1 r[m] = f_hashed_boolean(k,v) + end + end + end + m = m + 1 + if outer then + r[m] = "}" + else + r[m] = "}," + end + return r + end + return concat(fastserialize(t,true)) +end + +function table.deserialize(str) + if not str or str == "" then + return + end + local code = load(str) + if not code then + return + end + code = code() + if not code then + return + end + return code +end + +-- inspect(table.fastserialize { a = 1, b = { 4, { 5, 6 } }, c = { d = 7, e = 'f"g\nh' } }) + +function table.load(filename,loader) + if filename then + local t = (loader or io.loaddata)(filename) + if t and t ~= "" then + local t = utftoeight(t) + t = load(t) + if type(t) == "function" then + t = t() + if type(t) == "table" then + return t + end + end + end + end +end + +function table.save(filename,t,n,...) + io.savedata(filename,table.serialize(t,n == nil and true or n,...)) -- no frozen table.serialize +end + +local f_key_value = formatters["%s=%q"] +local f_add_table = formatters[" {%t},\n"] +local f_return_table = formatters["return {\n%t}"] + +local function slowdrop(t) -- maybe less memory (intermediate concat) + local r = { } + local l = { } + for i=1,#t do + local ti = t[i] + local j = 0 + for k, v in next, ti do + j = j + 1 + l[j] = f_key_value(k,v) + end + r[i] = f_add_table(l) + end + return f_return_table(r) +end + +local function fastdrop(t) + local r = { "return {\n" } + local m = 1 + for i=1,#t do + local ti = t[i] + m = m + 1 r[m] = " {" + for k, v in next, ti do + m = m + 1 r[m] = f_key_value(k,v) + end + m = m + 1 r[m] = "},\n" + end + m = m + 1 + r[m] = "}" + return concat(r) +end + +function table.drop(t,slow) -- only { { a=2 }, {a=3} } -- for special cases + if #t == 0 then + return "return { }" + elseif slow == true then + return slowdrop(t) -- less memory + else + return fastdrop(t) -- some 15% faster + end +end + +-- inspect(table.drop({ { a=2 }, {a=3} })) +-- inspect(table.drop({ { a=2 }, {a=3} },true)) + +-- function table.autokey(t,k) -- replaced +-- local v = { } +-- t[k] = v +-- return v +-- end + +local selfmapper = { __index = function(t,k) t[k] = k return k end } + +function table.twowaymapper(t) -- takes a 0/1 .. n indexed table and returns + if not t then -- it with string-numbers as indices + reverse + t = { } -- mapping (all strings) .. used in cvs etc but + else -- typically a helper that one forgets about + local zero = rawget(t,0) -- so it might move someplace else + for i=zero and 0 or 1,#t do + local ti = t[i] -- t[1] = "one" + if ti then + local i = tostring(i) + t[i] = ti -- t["1"] = "one" + t[ti] = i -- t["one"] = "1" + end + end + end + -- setmetatableindex(t,"key") + setmetatable(t,selfmapper) + return t +end + +-- The next version is somewhat faster, although in practice one will seldom +-- serialize a lot using this one. Often the above variants are more efficient. +-- If we would really need this a lot, we could hash q keys, or just not used +-- indented code. + +-- char-def.lua : 0.53 -> 0.38 +-- husayni.tma : 0.28 -> 0.19 + +local f_start_key_idx = formatters["%w{"] +local f_start_key_num = formatters["%w[%s]={"] +local f_start_key_str = formatters["%w[%q]={"] +local f_start_key_boo = formatters["%w[%l]={"] +local f_start_key_nop = formatters["%w{"] + +local f_stop = formatters["%w},"] + +local f_key_num_value_num = formatters["%w[%s]=%s,"] +local f_key_str_value_num = formatters["%w[%Q]=%s,"] +local f_key_boo_value_num = formatters["%w[%l]=%s,"] + +local f_key_num_value_str = formatters["%w[%s]=%Q,"] +local f_key_str_value_str = formatters["%w[%Q]=%Q,"] +local f_key_boo_value_str = formatters["%w[%l]=%Q,"] + +local f_key_num_value_boo = formatters["%w[%s]=%l,"] +local f_key_str_value_boo = formatters["%w[%Q]=%l,"] +local f_key_boo_value_boo = formatters["%w[%l]=%l,"] + +local f_key_num_value_not = formatters["%w[%s]={},"] +local f_key_str_value_not = formatters["%w[%Q]={},"] +local f_key_boo_value_not = formatters["%w[%l]={},"] + +local f_key_num_value_seq = formatters["%w[%s]={ %, t },"] +local f_key_str_value_seq = formatters["%w[%Q]={ %, t },"] +local f_key_boo_value_seq = formatters["%w[%l]={ %, t },"] + +local f_val_num = formatters["%w%s,"] +local f_val_str = formatters["%w%Q,"] +local f_val_boo = formatters["%w%l,"] +local f_val_not = formatters["%w{},"] +local f_val_seq = formatters["%w{ %, t },"] +local f_fin_seq = formatters[" %, t }"] + +local f_table_return = formatters["return {"] +local f_table_name = formatters["%s={"] +local f_table_direct = formatters["{"] +local f_table_entry = formatters["[%Q]={"] +local f_table_finish = formatters["}"] + +----- f_string = formatters["%q"] + +local spaces = utilities.strings.newrepeater(" ") + +local original_serialize = table.serialize -- the extensive one, the one we started with + +-- there is still room for optimization: index run, key run, but i need to check with the +-- latest lua for the value of #n (with holes) .. anyway for tracing purposes we want +-- indices / keys being sorted, so it will never be real fast + +local is_simple_table = table.is_simple_table + +-- local function is_simple_table(t) +-- local nt = #t +-- if nt > 0 then +-- local n = 0 +-- for _, v in next, t do +-- n = n + 1 +-- if type(v) == "table" then +-- return nil +-- end +-- end +-- -- local haszero = t[0] +-- local haszero = rawget(t,0) -- don't trigger meta +-- if n == nt then +-- local tt = { } +-- for i=1,nt do +-- local v = t[i] +-- local tv = type(v) +-- if tv == "number" then +-- tt[i] = v -- not needed tostring(v) +-- elseif tv == "string" then +-- tt[i] = format("%q",v) -- f_string(v) +-- elseif tv == "boolean" then +-- tt[i] = v and "true" or "false" +-- else +-- return nil +-- end +-- end +-- return tt +-- elseif haszero and (n == nt + 1) then +-- local tt = { } +-- for i=0,nt do +-- local v = t[i] +-- local tv = type(v) +-- if tv == "number" then +-- tt[i+1] = v -- not needed tostring(v) +-- elseif tv == "string" then +-- tt[i+1] = format("%q",v) -- f_string(v) +-- elseif tv == "boolean" then +-- tt[i+1] = v and "true" or "false" +-- else +-- return nil +-- end +-- end +-- tt[1] = "[0] = " .. tt[1] +-- return tt +-- end +-- end +-- return nil +-- end + +-- In order to overcome the luajit (65K constant) limitation I tried a split approach, +-- i.e. outputting the first level tables as locals but that failed with large cjk +-- fonts too so I removed that ... just use luatex instead. + +local function serialize(root,name,specification) + + if type(specification) == "table" then + return original_serialize(root,name,specification) -- the original one + end + + local t -- = { } + local n = 1 + local unknown = false + + local function do_serialize(root,name,depth,level,indexed) + if level > 0 then + n = n + 1 + if indexed then + t[n] = f_start_key_idx(depth) + else + local tn = type(name) + if tn == "number" then + t[n] = f_start_key_num(depth,name) + elseif tn == "string" then + t[n] = f_start_key_str(depth,name) + elseif tn == "boolean" then + t[n] = f_start_key_boo(depth,name) + else + t[n] = f_start_key_nop(depth) + end + end + depth = depth + 1 + end + -- we could check for k (index) being number (cardinal) + if root and next(root) ~= nil then + local first = nil + local last = #root + if last > 0 then + for k=1,last do + if rawget(root,k) == nil then + -- if root[k] == nil then + last = k - 1 + break + end + end + if last > 0 then + first = 1 + end + end + local sk = sortedkeys(root) + for i=1,#sk do + local k = sk[i] + local v = root[k] + local tv = type(v) + local tk = type(k) + if first and tk == "number" and k <= last and k >= first then + if tv == "number" then + n = n + 1 t[n] = f_val_num(depth,v) + elseif tv == "string" then + n = n + 1 t[n] = f_val_str(depth,v) + elseif tv == "table" then + if next(v) == nil then -- tricky as next is unpredictable in a hash + n = n + 1 t[n] = f_val_not(depth) + else + local st = is_simple_table(v) + if st then + n = n + 1 t[n] = f_val_seq(depth,st) + else + do_serialize(v,k,depth,level+1,true) + end + end + elseif tv == "boolean" then + n = n + 1 t[n] = f_val_boo(depth,v) + elseif unknown then + n = n + 1 t[n] = f_val_str(depth,tostring(v)) + end + elseif tv == "number" then + if tk == "number" then + n = n + 1 t[n] = f_key_num_value_num(depth,k,v) + elseif tk == "string" then + n = n + 1 t[n] = f_key_str_value_num(depth,k,v) + elseif tk == "boolean" then + n = n + 1 t[n] = f_key_boo_value_num(depth,k,v) + elseif unknown then + n = n + 1 t[n] = f_key_str_value_num(depth,tostring(k),v) + end + elseif tv == "string" then + if tk == "number" then + n = n + 1 t[n] = f_key_num_value_str(depth,k,v) + elseif tk == "string" then + n = n + 1 t[n] = f_key_str_value_str(depth,k,v) + elseif tk == "boolean" then + n = n + 1 t[n] = f_key_boo_value_str(depth,k,v) + elseif unknown then + n = n + 1 t[n] = f_key_str_value_str(depth,tostring(k),v) + end + elseif tv == "table" then + if next(v) == nil then + if tk == "number" then + n = n + 1 t[n] = f_key_num_value_not(depth,k) + elseif tk == "string" then + n = n + 1 t[n] = f_key_str_value_not(depth,k) + elseif tk == "boolean" then + n = n + 1 t[n] = f_key_boo_value_not(depth,k) + elseif unknown then + n = n + 1 t[n] = f_key_str_value_not(depth,tostring(k)) + end + else + local st = is_simple_table(v) + if not st then + do_serialize(v,k,depth,level+1) + elseif tk == "number" then + n = n + 1 t[n] = f_key_num_value_seq(depth,k,st) + elseif tk == "string" then + n = n + 1 t[n] = f_key_str_value_seq(depth,k,st) + elseif tk == "boolean" then + n = n + 1 t[n] = f_key_boo_value_seq(depth,k,st) + elseif unknown then + n = n + 1 t[n] = f_key_str_value_seq(depth,tostring(k),st) + end + end + elseif tv == "boolean" then + if tk == "number" then + n = n + 1 t[n] = f_key_num_value_boo(depth,k,v) + elseif tk == "string" then + n = n + 1 t[n] = f_key_str_value_boo(depth,k,v) + elseif tk == "boolean" then + n = n + 1 t[n] = f_key_boo_value_boo(depth,k,v) + elseif unknown then + n = n + 1 t[n] = f_key_str_value_boo(depth,tostring(k),v) + end + else + if tk == "number" then + n = n + 1 t[n] = f_key_num_value_str(depth,k,tostring(v)) + elseif tk == "string" then + n = n + 1 t[n] = f_key_str_value_str(depth,k,tostring(v)) + elseif tk == "boolean" then + n = n + 1 t[n] = f_key_boo_value_str(depth,k,tostring(v)) + elseif unknown then + n = n + 1 t[n] = f_key_str_value_str(depth,tostring(k),tostring(v)) + end + end + end + end + if level > 0 then + n = n + 1 t[n] = f_stop(depth-1) + end + end + + local tname = type(name) + + if tname == "string" then + if name == "return" then + t = { f_table_return() } + else + t = { f_table_name(name) } + end + elseif tname == "number" then + t = { f_table_entry(name) } + elseif tname == "boolean" then + if name then + t = { f_table_return() } + else + t = { f_table_direct() } + end + else + t = { f_table_name("t") } + end + + if root then + -- The dummy access will initialize a table that has a delayed initialization + -- using a metatable. (maybe explicitly test for metatable). This can crash on + -- metatables that check the index against a number. + if getmetatable(root) then -- todo: make this an option, maybe even per subtable + local dummy = root._w_h_a_t_e_v_e_r_ -- needed + root._w_h_a_t_e_v_e_r_ = nil + end + -- Let's forget about empty tables. + if next(root) ~= nil then + local st = is_simple_table(root) + if st then + return t[1] .. f_fin_seq(st) -- todo: move up and in one go + else + do_serialize(root,name,1,0) + end + end + end + n = n + 1 + t[n] = f_table_finish() + return concat(t,"\n") +end + +table.serialize = serialize + +if setinspector then + setinspector("table",function(v) + if type(v) == "table" then + print(serialize(v,"table",{ metacheck = false })) + return true + end + end) +end + +-- ordered hashes (for now here but in the table namespace): + +-- local t = table.orderedhash() +-- +-- t["1"] = { "a", "b" } +-- t["2"] = { } +-- t["2a"] = { "a", "c", "d" } +-- +-- for k, v in table.ordered(t) do +-- ... +-- end + +local mt = { + __newindex = function(t,k,v) + local n = t.last + 1 + t.last = n + t.list[n] = k + t.hash[k] = v + end, + __index = function(t,k) + return t.hash[k] + end, + __len = function(t) + return t.last + end, +} + +function table.orderedhash() + return setmetatable({ list = { }, hash = { }, last = 0 }, mt) +end + +function table.ordered(t) + local n = t.last + if n > 0 then + local l = t.list + local i = 1 + local h = t.hash + local f = function() + if i <= n then + local k = i + local v = h[l[k]] + i = i + 1 + return k, v + end + end + return f, 1, h[l[1]] + else + return function() end + end +end diff --git a/macros/luatex/generic/lualibs/lualibs-util-tpl.lua b/macros/luatex/generic/lualibs/lualibs-util-tpl.lua new file mode 100644 index 0000000000..57d7df3a91 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs-util-tpl.lua @@ -0,0 +1,215 @@ +if not modules then modules = { } end modules ['util-tpl'] = { + version = 1.001, + comment = "companion to luat-lib.mkiv", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files" +} + +-- This is experimental code. Coming from dos and windows, I've always used %whatever% +-- as template variables so let's stick to it. After all, it's easy to parse and stands +-- out well. A double %% is turned into a regular %. + +utilities.templates = utilities.templates or { } +local templates = utilities.templates + +local trace_template = false trackers.register("templates.trace",function(v) trace_template = v end) +local report_template = logs.reporter("template") + +local tostring, next = tostring, next +local format, sub, byte = string.format, string.sub, string.byte +local P, C, R, Cs, Cc, Carg, lpegmatch, lpegpatterns = lpeg.P, lpeg.C, lpeg.R, lpeg.Cs, lpeg.Cc, lpeg.Carg, lpeg.match, lpeg.patterns + +-- todo: make installable template.new + +local replacer + +local function replacekey(k,t,how,recursive) + local v = t[k] + if not v then + if trace_template then + report_template("unknown key %a",k) + end + return "" + else + v = tostring(v) + if trace_template then + report_template("setting key %a to value %a",k,v) + end + if recursive then + return lpegmatch(replacer,v,1,t,how,recursive) + else + return v + end + end +end + +local sqlescape = lpeg.replacer { + { "'", "''" }, + { "\\", "\\\\" }, + { "\r\n", "\\n" }, + { "\r", "\\n" }, + -- { "\t", "\\t" }, +} + +local sqlquoted = Cs(Cc("'") * sqlescape * Cc("'")) + +lpegpatterns.sqlescape = sqlescape +lpegpatterns.sqlquoted = sqlquoted + +-- escapeset : \0\1\2\3\4\5\6\7\8\9\10\11\12\13\14\15\16\17\18\19\20\21\22\23\24\25\26\27\28\29\30\31\"\\\127 +-- test string: [[1\0\31test23"\\]] .. string.char(19) .. "23" +-- +-- slow: +-- +-- local luaescape = lpeg.replacer { +-- { '"', [[\"]] }, +-- { '\\', [[\\]] }, +-- { R("\0\9") * #R("09"), function(s) return "\\00" .. byte(s) end }, +-- { R("\10\31") * #R("09"), function(s) return "\\0" .. byte(s) end }, +-- { R("\0\31") , function(s) return "\\" .. byte(s) end }, +-- } +-- +-- slightly faster: + +-- local luaescape = Cs (( +-- P('"' ) / [[\"]] + +-- P('\\') / [[\\]] + +-- Cc("\\00") * (R("\0\9") / byte) * #R("09") + +-- Cc("\\0") * (R("\10\31") / byte) * #R("09") + +-- Cc("\\") * (R("\0\31") / byte) + +-- P(1) +-- )^0) + +----- xmlescape = lpegpatterns.xmlescape +----- texescape = lpegpatterns.texescape +local luaescape = lpegpatterns.luaescape +----- sqlquoted = lpegpatterns.sqlquoted +----- luaquoted = lpegpatterns.luaquoted + +local escapers = { + lua = function(s) + -- return sub(format("%q",s),2,-2) + return lpegmatch(luaescape,s) + end, + sql = function(s) + return lpegmatch(sqlescape,s) + end, +} + +local quotedescapers = { + lua = function(s) + -- return lpegmatch(luaquoted,s) + return format("%q",s) + end, + sql = function(s) + return lpegmatch(sqlquoted,s) + end, +} + +local luaescaper = escapers.lua +local quotedluaescaper = quotedescapers.lua + +local function replacekeyunquoted(s,t,how,recurse) -- ".. \" " + if how == false then + return replacekey(s,t,how,recurse) + else + local escaper = how and escapers[how] or luaescaper + return escaper(replacekey(s,t,how,recurse)) + end +end + +local function replacekeyquoted(s,t,how,recurse) -- ".. \" " + if how == false then + return replacekey(s,t,how,recurse) + else + local escaper = how and quotedescapers[how] or quotedluaescaper + return escaper(replacekey(s,t,how,recurse)) + end +end + +local function replaceoptional(l,m,r,t,how,recurse) + local v = t[l] + return v and v ~= "" and lpegmatch(replacer,r,1,t,how or "lua",recurse or false) or "" +end + +local single = P("%") -- test %test% test : resolves test +local double = P("%%") -- test 10%% test : %% becomes % +local lquoted = P("%[") -- test '%[test]%' test : resolves to test with escaped "'s +local rquoted = P("]%") -- +local lquotedq = P("%(") -- test %(test)% test : resolves to 'test' with escaped "'s +local rquotedq = P(")%") -- + +local escape = double / '%%' +local nosingle = single / '' +local nodouble = double / '' +local nolquoted = lquoted / '' +local norquoted = rquoted / '' +local nolquotedq = lquotedq / '' +local norquotedq = rquotedq / '' + +local noloptional = P("%?") / '' +local noroptional = P("?%") / '' +local nomoptional = P(":") / '' + + +local args = Carg(1) * Carg(2) * Carg(3) +local key = nosingle * ((C((1-nosingle )^1) * args) / replacekey ) * nosingle +local quoted = nolquotedq * ((C((1-norquotedq )^1) * args) / replacekeyquoted ) * norquotedq +local unquoted = nolquoted * ((C((1-norquoted )^1) * args) / replacekeyunquoted) * norquoted +local optional = noloptional * ((C((1-nomoptional)^1) * nomoptional * C((1-noroptional)^1) * args) / replaceoptional) * noroptional +local any = P(1) + + replacer = Cs((unquoted + quoted + escape + optional + key + any)^0) + +local function replace(str,mapping,how,recurse) + if mapping and str then + return lpegmatch(replacer,str,1,mapping,how or "lua",recurse or false) or str + else + return str + end +end + +-- print(replace("test '%[x]%' test",{ x = [[a 'x' a]] })) +-- print(replace("test '%x%' test",{ x = [[a "x" a]] })) +-- print(replace([[test "%[x]%" test]],{ x = [[a "x" a]] })) +-- print(replace("test '%[x]%' test",{ x = true })) +-- print(replace("test '%[x]%' test",{ x = [[a 'x' a]], y = "oeps" },'sql')) +-- print(replace("test '%[x]%' test",{ x = [[a '%y%' a]], y = "oeps" },'sql',true)) +-- print(replace([[test %[x]% test]],{ x = [[a "x" a]]})) +-- print(replace([[test %(x)% test]],{ x = [[a "x" a]]})) +-- print(replace([[convert %?x: -x "%x%" ?% %?y: -y "%y%" ?%]],{ x = "yes" })) + +templates.replace = replace + +function templates.replacer(str,how,recurse) -- reads nicer + return function(mapping) + return lpegmatch(replacer,str,1,mapping,how or "lua",recurse or false) or str + end +end + +-- local cmd = templates.replacer([[foo %bar%]]) print(cmd { bar = "foo" }) + +function templates.load(filename,mapping,how,recurse) + local data = io.loaddata(filename) or "" + if mapping and next(mapping) then + return replace(data,mapping,how,recurse) + else + return data + end +end + +function templates.resolve(t,mapping,how,recurse) + if not mapping then + mapping = t + end + for k, v in next, t do + t[k] = replace(v,mapping,how,recurse) + end + return t +end + +-- inspect(utilities.templates.replace("test %one% test", { one = "%two%", two = "two" })) +-- inspect(utilities.templates.resolve({ one = "%two%", two = "two", three = "%three%" })) +-- inspect(utilities.templates.replace("test %one% test", { one = "%two%", two = "two" },false,true)) +-- inspect(utilities.templates.replace("test %one% test", { one = "%two%", two = "two" },false)) diff --git a/macros/luatex/generic/lualibs/lualibs.dtx b/macros/luatex/generic/lualibs/lualibs.dtx new file mode 100644 index 0000000000..8c302400f1 --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs.dtx @@ -0,0 +1,813 @@ +% \iffalse meta-comment +% +% Copyright (C) 2009--2019 by +% +% PRAGMA ADE / ConTeXt Development Team +% The LuaLaTeX Dev Team +% +% See ConTeXt's mreadme.pdf for the license. +% +% This work consists of the main source file lualibs.dtx +% and the derived files lualibs.lua, lualibs-basic.lua, +% and lualibs-extended.lua. +% +% Unpacking: +% tex lualibs.dtx +% +% Documentation: +% lualatex lualibs.dtx +% +% The class ltxdoc loads the configuration file ltxdoc.cfg +% if available. Here you can specify further options, e.g. +% use A4 as paper format: +% \PassOptionsToClass{a4paper}{article} +% +% +% +%<*ignore> +\begingroup + \def\x{LaTeX2e}% +\expandafter\endgroup +\ifcase 0\ifx\install y1\fi\expandafter + \ifx\csname processbatchFile\endcsname\relax\else1\fi + \ifx\fmtname\x\else 1\fi\relax +\else\csname fi\endcsname +%</ignore> +%<*install> +\input docstrip.tex +\Msg{************************************************************************} +\Msg{* Installation} +\Msg{* Package: lualibs 2019-08-11 v2.67 Lua additional functions.} +\Msg{************************************************************************} + +\keepsilent +\askforoverwritefalse + +\let\MetaPrefix\relax + +\preamble +This is a generated file. + +Copyright (C) 2009--2018 by + PRAGMA ADE / ConTeXt Development Team + The LuaLaTeX Dev Team + +See ConTeXt's mreadme.pdf for the license. + +This work consists of the main source file lualibs.dtx +and the derived files lualibs.lua, lualibs-basic.lua, +and lualibs-extended.lua. + +\endpreamble + +% The following hacks are to generate a lua file with lua comments starting by +% -- instead of %% + +\def\MetaPrefix{-- } + +\def\luapostamble{% + \MetaPrefix^^J% + \MetaPrefix\space End of File `\outFileName'.% +} + +\def\currentpostamble{\luapostamble}% + +\generate{% + \usedir{tex/luatex/lualibs}% + \file{lualibs.lua}{\from{lualibs.dtx}{lualibs}}% +} + +\generate{% + \usedir{tex/luatex/lualibs}% + \file{lualibs-basic.lua}{\from{lualibs.dtx}{basic}}% +} + +\generate{% + \usedir{tex/luatex/lualibs}% + \file{lualibs-extended.lua}{\from{lualibs.dtx}{extended}}% +} + +\obeyspaces +\Msg{************************************************************************} +\Msg{*} +\Msg{* To finish the installation you have to move the following} +\Msg{* files into a directory searched by TeX:} +\Msg{*} +\Msg{* lualibs.lua, lualibs-basic.lua, lualibs-extended.lua} +\Msg{*} +\Msg{* Happy TeXing!} +\Msg{*} +\Msg{************************************************************************} + +\endbatchfile +%</install> +%<*ignore> +\fi +%</ignore> +%<*driver> +\NeedsTeXFormat{LaTeX2e} +\ProvidesFile{lualibs.drv} + [2019/08/11 v2.67 Lua Libraries.] +\documentclass{ltxdoc} +\usepackage{fancyvrb,xspace} +\usepackage[x11names]{xcolor} +% +\def\primarycolor{DodgerBlue4} %%-> rgb 16 78 139 | #104e8b +\def\secondarycolor{Goldenrod4} %%-> rgb 139 105 200 | #8b6914 +% +\usepackage[ + bookmarks=true, + colorlinks=true, + linkcolor=\primarycolor, + urlcolor=\secondarycolor, + citecolor=\primarycolor, + pdftitle={The lualibs package}, + pdfsubject={Port of the ConTeXt Lua libraries}, + pdfauthor={Elie Roux & Philipp Gesang}, + pdfkeywords={luatex, lualatex, unicode, opentype} +]{hyperref} +\newcommand*\email[1]{\href{mailto:#1}{#1}} +\usepackage{fontspec} +\setmainfont[ + Numbers=OldStyle, + Ligatures=TeX, +]{Linux Libertine O} +\setmonofont [Ligatures=TeX,Scale=MatchLowercase] {Liberation Mono} +\setsansfont [Ligatures=TeX,Scale=MatchLowercase] {Iwona Medium} +\usepackage{metalogo} +\EnableCrossrefs +\CodelineIndex +\newcommand\TEX {\TeX\xspace} +\newcommand\LATEX {\LaTeX\xspace} +\newcommand\LUA {Lua\xspace} +\newcommand\LUATEX {\LuaTeX\xspace} +\newcommand\CONTEXT {Con\TeX t\xspace} + +\def\definehighlight[#1][#2]% + {\ifcsname #1\endcsname\else + \expandafter\def\csname #1\endcsname% + {\bgroup#2\csname #1_indeed\endcsname} + \expandafter\def\csname #1_indeed\endcsname##1% + {##1\egroup}% + \fi} + +\def\restoreunderscore{\catcode`\_=12\relax} + +\definehighlight [fileent][\ttfamily\restoreunderscore] %% files, dirs +\definehighlight [texmacro][\sffamily\itshape\textbackslash] %% cs +\definehighlight [luaident][\sffamily\itshape\restoreunderscore] %% lua identifiers +\definehighlight [identifier][\sffamily] %% names +\definehighlight [abbrev][\rmfamily\scshape] %% acronyms +\definehighlight [emphasis][\rmfamily\slshape] %% level 1 emph + +\usepackage{titlesec} +\def\movecountertomargin#1{\llap{\rmfamily\upshape#1\hskip2em}} +\def\zeropoint{0pt} +%% we don’t have to grab attention like a child +\titleformat \part + {\normalsize\bfseries} + {\movecountertomargin\thepart} \zeropoint {} +\titleformat \section + {\normalsize\scshape}%% no \word; life is full of disappointments + {\movecountertomargin\thesection} \zeropoint {} +\titleformat \subsection + {\small\itshape} + {\movecountertomargin\thesubsection} \zeropoint {} +\renewcommand\partname{Part}%% gets rid of the stupid “file” heading + +\usepackage{tocloft} +\renewcommand \cftpartfont {\rmfamily\upshape} +\renewcommand \cftsecfont {\rmfamily\upshape} +\renewcommand \cftsubsecfont {\rmfamily\upshape} +\setlength \cftbeforepartskip {1ex} +\setlength \cftbeforesecskip {1ex} + +\begin{document} + \DocInput{lualibs.dtx}% +\end{document} +%</driver> +% \fi +% \CheckSum{0} +% +% \CharacterTable +% {Upper-case \A\B\C\D\E\F\G\H\I\J\K\L\M\N\O\P\Q\R\S\T\U\V\W\X\Y\Z +% Lower-case \a\b\c\d\e\f\g\h\i\j\k\l\m\n\o\p\q\r\s\t\u\v\w\x\y\z +% Digits \0\1\2\3\4\5\6\7\8\9 +% Exclamation \! Double quote \" Hash (number) \# +% Dollar \$ Percent \% Ampersand \& +% Acute accent \' Left paren \( Right paren \) +% Asterisk \* Plus \+ Comma \, +% Minus \- Point \. Solidus \/ +% Colon \: Semicolon \; Less than \< +% Equals \= Greater than \> Question mark \? +% Commercial at \@ Left bracket \[ Backslash \\ +% Right bracket \] Circumflex \^ Underscore \_ +% Grave accent \` Left brace \{ Vertical bar \| +% Right brace \} Tilde \~} +% +% \GetFileInfo{lualibs.drv} +% +% \title{The \identifier{lualibs} package} +% \date{2019/08/11 v2.67} +% \author{Élie Roux · \email{elie.roux@telecom-bretagne.eu}\\ +% Philipp Gesang · \email{phg@phi-gamma.net}\\ +% Ulrike Fischer · \email{fischer@troubleshooting-tex.de}\\ +% } +% +% \maketitle +% +% \begin{abstract} +% Additional \LUA functions taken from the \verb|l-*| and \verb|util-*| files +% of \CONTEXT. +% For an introduction on this package (among others), please refer +% to the document \fileent{lualatex-doc.pdf}. +% \end{abstract} +% +% \tableofcontents +% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% \part{Package Description} +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% \section{Overview} +% +% \LUA is a very minimal language, and it does only have a minimal standard +% library. The aim of this package is to provide an extended standard library, +% to be used by various \LUATEX packages. The code is specific to \LUATEX +% and depends on \LUATEX functions and modules not available in regular lua. +% +% \noindent The code is derived from \CONTEXT libraries. +% +% \section{Usage} +% +% You can either load the \identifier{lualibs} module, which will in turn load +% one of two sets of libraries provided by this package: |require("lualibs")|, or +% explicitly load the modules you need, e.g.: |require("lualibs-table")|, +% please note that some modules depend on others. +% +% \subsection{Loading Library Collections} +% +% The libraries are split into a \identifier{basic} and an +% \identifier{extended} collection. +% Though neither initialization time nor memory consumption will be noticably +% impacted,\footnote{% +% Note that in terms of code this is only a small fraction of what +% \CONTEXT loads with every run. +% } +% the \identifier{lualibs} package can skip loading of the latter on demand. +% The \luaident{config} table needs to be present prior to loading the +% package for this to work (in the future this may be achieved by an option of +% \texmacro{usepackage}) for \LATEX users). +% In the \verb|lualibs| field, set \verb|load_extended| to false: +% +% \begin{quote}\begin{verbatim} +% \directlua{ +% --- 1) create config table safely +% config = config or { } +% config.lualibs = config.lualibs or { } +% +% --- 2) unset the load_extended entry +% config.lualibs.load_extended = false +% +% --- 3) load the lualibs +% require "lualibs" +% } +% \end{verbatim}\end{quote} +% +% However, there is no guarantee that the \identifier{extended} set remains +% permanently excluded. +% Re-loading the package at a later point will cancel this option and possibly +% add the missing libraries. +% +% \subsection{Loading libraries Individually} +% In order to load the \fileent{table} module you would actually have to prepare +% it as follows: +% +% \begin{quote}\begin{verbatim} +% require"lualibs-lua" +% require"lualibs-lpeg" +% require"lualibs-table" +% \end{verbatim}\end{quote} +% +% \noindent +% If your code is run by the \fileent{texlua} intepreter, you will need to +% initialize \luaident{kpse} library so that |require()| can find files +% under \abbrev{texmf} tree: |kpse.set_program_name("luatex")|. +% +% \section{Files} +% +% The \identifier{lualibs} bundle contains files from two \CONTEXT \LUA +% library categories: The generic auxiliary functions (original file prefix: +% |l-|) together form something close to a standard libary. Most of these are +% extensions of an existing namespace, like for instance |l-table.lua| which +% adds full-fledged serialization capabilities to the \LUA table library. +% They were imported under the \identifier{lualibs}-prefix and are contained +% in the \identifier{basic} collection. +% (For a list see table~\ref{tab:basic}.) +% +% \begin{table}[t] +% \hrule +% \centering +% \caption{The \identifier{basic} set.} +% \vskip1em +% \begin{tabular}{l l l} +% \identifier{lualibs} name & \CONTEXT name & primary purpose \\[1ex] +% lualibs-lua.lua & l-lua.lua & compatibility \\ +% lualibs-package.lua & l-package.lua & \LUA file loaders \\ +% lualibs-lpeg.lua & l-lpeg.lua & patterns \\ +% lualibs-function.lua & l-function.lua & defines a dummy function \\ +% lualibs-string.lua & l-string.lua & string manipulation \\ +% lualibs-table.lua & l-table.lua & serialization, conversion \\ +% lualibs-boolean.lua & l-boolean.lua & boolean converter \\ +% lualibs-number.lua & l-number.lua & bit operations \\ +% lualibs-math.lua & l-math.lua & math functions \\ +% lualibs-io.lua & l-io.lua & reading and writing files \\ +% lualibs-os.lua & l-os.lua & platform specific code \\ +% lualibs-file.lua & l-file.lua & filesystem operations \\ +% lualibs-gzip.lua & l-gzip.lua & wrapper for \identifier{lgzip} \\ +% lualibs-md5.lua & l-md5.lua & checksum functions \\ +% lualibs-dir.lua & l-dir.lua & directory handling \\ +% lualibs-unicode.lua & l-unicode.lua & utf and unicode \\ +% lualibs-url.lua & l-url.lua & url handling \\ +% lualibs-set.lua & l-set.lua & sets \\[1ex] +% \end{tabular} +% \label{tab:basic} +% \hrule +% \end{table} +% +% The \identifier{extended} category comprises a selection of files mostly from +% the utilities namespace (|util-|; cf. table~\ref{tab:extended}). +% Their purpose is more specific and at times quite low-level. +% Additionally, the file \fileent{trac-inf.lua} has been included because it is +% essential to some of the code loaded subsequently. +% +% \begin{table}[t] +% \hrule +% \centering +% \caption{The \identifier{extended} set.} +% \vskip1em +% \begin{tabular}{l l l} +% \identifier{lualibs} name & \CONTEXT name & primary purpose \\[1ex] +% lualibs-util-str.lua & util-str.lua & extra |string| functions \\ +% lualibs-util-fil.lua & util-fil.lua & extra |file| functions \\ +% lualibs-util-tab.lua & util-tab.lua & extra |table| functions \\ +% lualibs-util-sto.lua & util-sto.lua & table allocation \\ +% lualibs-util-prs.lua & util-sto.lua & miscellaneous parsers \\ +% lualibs-util-dim.lua & util-dim.lua & conversion between dimensions \\ +% lualibs-trac-inf.lua & trac-inf.lua & timing, statistics \\ +% lualibs-util-lua.lua & util-lua.lua & operations on bytecode \\ +% lualibs-util-deb.lua & util-deb.lua & extra |debug| functionality \\ +% lualibs-util-tpl.lua & util-tpl.lua & templating \\ +% lualibs-util-sta.lua & util-sta.lua & stacker (e.~g. for \abbrev{pdf}) \\ +% lualibs-util-jsn.lua & util-jsn.lua & conversion to and from json \\[1ex] +% \end{tabular} +% \label{tab:extended} +% \hrule +% \end{table} +% +% \section{Packaging} +% +% By default, \identifier{lualibs} will not load the libraries individually. +% Instead, it includes two \emphasis{merged packages} that have been compiled +% from the original files. +% This is achieved by means of \identifier{mtx-package}, a script for bundling +% \LUA code shipped with \CONTEXT. +% This concatenates the code of several \LUA files into a single file that is +% both easier to distribute and loading marginally faster. +% \identifier{mtx-package} ensures that the code from each file gets its +% own closure and strips newlines and comments, resulting in a smaller payload. +% Another package that relies on it heavily is the font loader as contained in +% \identifier{luaotfload} and \identifier{luatex-fonts}. +% \identifier{Luaotfload}, a port of the \CONTEXT fontloader for Plain and +% \LATEX, has a hard dependency on the functionality provided by the % +% \identifier{Lualibs} package. The packages should not be updated independently. +% +% If \CONTEXT is installed on the system, the merge files can be created +% by running: +% +% \begin{quote}\begin{verbatim} +% mtxrun --script package --merge lualibs-basic.lua +% mtxrun --script package --merge lualibs-extended.lua +% \end{verbatim}\end{quote} +% +% \noindent +% Of course there is a make target for that: +% +% \begin{quote}\begin{verbatim} +% make merge +% \end{verbatim}\end{quote} +% \noindent +% will take care of assembling the packages from the files distributed with +% \identifier{lualibs}. +% +% For this to work, the syntax of the \LUA file needs to be well-formed: +% files that should be merged must be included via a function +% \verb|loadmodule()|. +% It doesn’t matter if the function actually does something; a dummy will +% suffice. +% Also, the argument to \verb|loadmodule()| must be wrapped in parentheses. +% This rule is quite convenient, actually, since it allows excluding files +% from the merge while still using \verb|loadmodule()| consistently. +% +% \begin{quote}\begin{verbatim} +% ... +% loadmodule("my-lua-file.lua") -- <= will be merged +% loadmodule('my-2nd-file.lua') -- <= will be merged +% loadmodule "my-3rd-file.lua" -- <= will be ignored +% ... +% \end{verbatim}\end{quote} +% +% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% \part{\fileent{lualibs.lua}} +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% +% \iffalse +%<*lualibs> +% \fi +% \begin{macrocode} +lualibs = lualibs or { } + +lualibs.module_info = { + name = "lualibs", + version = "2.67", --TAGVERSION + date = "2019-08-11", --TAGDATE + description = "ConTeXt Lua standard libraries.", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL & Elie Roux & Philipp Gesang", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "See ConTeXt's mreadme.pdf for the license", +} + +% \end{macrocode} +% The behavior of the lualibs can be configured to some extent. +% \begin{itemize} +% \item Based on the parameter \verb|lualibs.prefer_merged|, the +% libraries can be loaded via the included merged packages or +% the individual files. +% \item Two classes of libraries are distinguished, mainly because +% of a similar distinction in \CONTEXT, but also to make +% loading of the less fundamental functionality optional. +% While the “basic” collection is always loaded, the +% configuration setting \verb|lualibs.load_extended| triggers +% inclusion of the extended collection. +% \item Verbosity can be increased via the \verb|verbose| switch. +% \end{itemize} +% +% \begin{macrocode} + +config = config or { } +config.lualibs = config.lualibs or { } + +if config.lualibs.prefer_merged ~= nil then + lualibs.prefer_merged = config.lualibs.prefer_merged +else + lualibs.prefer_merged = true +end + +if config.lualibs.load_extended ~= nil then + lualibs.load_extended = config.lualibs.load_extended +else + lualibs.load_extended = true +end + +if config.lualibs.verbose ~= nil then + config.lualibs.verbose = config.lualibs.verbose +else + config.lualibs.verbose = false +end + +% \end{macrocode} +% The lualibs may be loaded in scripts. +% To account for the different environment, fallbacks for +% the luatexbase facilities are installed. +% +% \begin{macrocode} + +local dofile = dofile +local kpsefind_file = kpse.find_file +local stringformat = string.format +local texiowrite_nl = texio.write_nl + +local find_file, error, warn, info +do + local _error, _warn, _info + if luatexbase and luatexbase.provides_module then + _error, _warn, _info = luatexbase.provides_module(lualibs.module_info) + else + _error, _warn, _info = texiowrite_nl, texiowrite_nl, texiowrite_nl + end + + if lualibs.verbose then + error, warn, info = _error, _warn, _info + else + local dummylogger = function ( ) end + error, warn, info = _error, dummylogger, dummylogger + end + lualibs.error, lualibs.warn, lualibs.info = error, warn, info +end + +if status.kpse_used == 0 then + kpse.set_program_name("luatex") +end + +find_file = kpsefind_file + +% \end{macrocode} +% The lualibs load a merged package by default. +% In order to create one of these, the meta file that includes the +% libraries must satisfy certain assumptions \verb|mtx-package| makes +% about the coding style. +% Most important is that the functions that indicates which files +% to include must go by the name \verb|loadmodule()|. +% For this reason we define a \verb|loadmodule()| function as a +% wrapper around \verb|dofile()|. +% +% \begin{macrocode} + +local loadmodule = loadmodule or function (name, t) + if not t then t = "library" end + local filepath = find_file(name, "lua") + if not filepath or filepath == "" then + warn(stringformat("Could not locate %s “%s”.", t, name)) + return false + end + dofile(filepath) + return true +end + +lualibs.loadmodule = loadmodule + +% \end{macrocode} +% The separation of the “basic” from the “extended” sets coincides +% with the split into luat-bas.mkiv and luat-lib.mkiv. +% +% \begin{macrocode} + +if lualibs.basic_loaded ~= true +or config.lualibs.force_reload == true +then + loadmodule"lualibs-basic.lua" + loadmodule"lualibs-compat.lua" --- restore stuff gone since v1.* +end + +if lualibs.load_extended == true +and lualibs.extended_loaded ~= true +or config.lualibs.force_reload == true +then + loadmodule"lualibs-extended.lua" +end + +--- This restores the default of loading everything should a package +--- have requested otherwise. Will be gone once there is a canonical +--- interface for parameterized loading of libraries. +config.lualibs.load_extended = true + +-- vim:tw=71:sw=2:ts=2:expandtab + +% \end{macrocode} +% +% \iffalse +%</lualibs> +% \fi +% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% \part{\fileent{lualibs-basic.lua}} +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% +% \iffalse +%<*basic> +% \fi +% \begin{macrocode} +lualibs = lualibs or { } +local info = lualibs.info +local loadmodule = lualibs.loadmodule + +local lualibs_basic_module = { + name = "lualibs-basic", + version = "2.67", --TAGVERSION + date = "2019-08-11", --TAGDATE + description = "ConTeXt Lua libraries -- basic collection.", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL & Elie Roux & Philipp Gesang", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "See ConTeXt's mreadme.pdf for the license", +} + +local loaded = false --- track success of package loading + +if lualibs.prefer_merged then + info"Loading merged package for collection “basic”." + loaded = loadmodule('lualibs-basic-merged.lua') +else + info"Ignoring merged packages." + info"Falling back to individual libraries from collection “basic”." +end + +% \end{macrocode} +% \verb|mtx-package| expects the files to be included by +% \verb|loadmodule|. +% If run on this file, it will create \verb|lualibs-basic-merged.lua| +% from all the files mentioned in the next block. +% +% \begin{macrocode} + +if loaded == false then + loadmodule("lualibs-lua.lua") + loadmodule("lualibs-package.lua") + loadmodule("lualibs-lpeg.lua") + loadmodule("lualibs-function.lua") + loadmodule("lualibs-string.lua") + loadmodule("lualibs-table.lua") + loadmodule("lualibs-boolean.lua") + loadmodule("lualibs-number.lua") + loadmodule("lualibs-math.lua") + loadmodule("lualibs-io.lua") + loadmodule("lualibs-os.lua") + loadmodule("lualibs-file.lua") + loadmodule("lualibs-gzip.lua") + loadmodule("lualibs-md5.lua") + loadmodule("lualibs-dir.lua") + loadmodule("lualibs-unicode.lua") + loadmodule("lualibs-url.lua") + loadmodule("lualibs-set.lua") +end + +lualibs.basic_loaded = true +-- vim:tw=71:sw=2:ts=2:expandtab + +% \end{macrocode} +% +% \iffalse +%</basic> +% \fi +% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% \part{\fileent{lualibs-extended.lua}} +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% +% \iffalse +%<*extended> +% \fi +% \begin{macrocode} +lualibs = lualibs or { } + +% \end{macrocode} +% Loading the \emph{extended} set requires a tad more effort, but +% it’s well invested. +% +% Since we only want the functionality, we have to simulate parts +% of a running \CONTEXT environment, above all logging, that some +% of the more involved libraries cannot be loaded without. +% Also, one utility file cannot be packaged because it returns a +% table which would preclude loading of later code. +% Thus, we remove it from the natural loading chain (it is not +% critical) and append it at the end. +% +% \begin{macrocode} + +local lualibs_extended_module = { + name = "lualibs-extended", + version = "2.67", --TAGVERSION + date = "2019-08-11", --TAGDATE + description = "ConTeXt Lua libraries -- extended collection.", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL & Elie Roux & Philipp Gesang", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "See ConTeXt's mreadme.pdf for the license", +} + + +local stringformat = string.format +local loadmodule = lualibs.loadmodule +local texiowrite = texio.write +local texiowrite_nl = texio.write_nl + +% \end{macrocode} +% Here we define some functions that fake the elaborate +% logging/tracking mechanism Context provides. +% +% \begin{macrocode} + +local error, logger, mklog +if luatexbase and luatexbase.provides_module then + --- TODO test how those work out when running tex + local __error,_,_,__logger = + luatexbase.provides_module(lualibs_extended_module) + error = __error + logger = __logger + mklog = function ( ) return logger end +else + mklog = function (t) + local prefix = stringformat("[%s] ", t) + return function (...) + texiowrite_nl(prefix) + texiowrite (stringformat(...)) + end + end + error = mklog"ERROR" + logger = mklog"INFO" +end + +local info = lualibs.info + +% \end{macrocode} +% We temporarily put our own global table in place and restore +% whatever we overloaded afterwards. +% +% \CONTEXT\ modules each have a custom logging mechanism that can be +% enabled for debugging. +% In order to fake the presence of this facility we need to define at +% least the function \verb|logs.reporter|. +% For now it’s sufficient to make it a reference to \verb|mklog| as +% defined above. +% +% \begin{macrocode} + +local dummy_function = function ( ) end +local newline = function ( ) texiowrite_nl"" end + +local fake_logs = function (name) + return { + name = name, + enable = dummy_function, + disable = dummy_function, + reporter = mklog, + newline = newline + } +end + +local fake_trackers = function (name) + return { + name = name, + enable = dummy_function, + disable = dummy_function, + register = mklog, + newline = newline, + } +end + +local backup_store = { } + +local fake_context = function ( ) + if logs then backup_store.logs = logs end + if trackers then backup_store.trackers = trackers end + logs = fake_logs"logs" + trackers = fake_trackers"trackers" +end + + +% \end{macrocode} +% Restore a backed up logger if appropriate. +% +% \begin{macrocode} +local unfake_context = function ( ) + if backup_store then + local bl, bt = backup_store.logs, backup_store.trackers + if bl then logs = bl end + if bt then trackers = bt end + end +end + +fake_context() + +local loaded = false +if lualibs.prefer_merged then + info"Loading merged package for collection “extended”." + loaded = loadmodule('lualibs-extended-merged.lua') +else + info"Ignoring merged packages." + info"Falling back to individual libraries from collection “extended”." +end + +if loaded == false then + loadmodule("lualibs-util-str.lua")--- string formatters (fast) + loadmodule("lualibs-util-fil.lua")--- extra file helpers + loadmodule("lualibs-util-tab.lua")--- extended table operations + loadmodule("lualibs-util-sto.lua")--- storage (hash allocation) + ----------("lualibs-util-pck.lua")---!packers; necessary? + ----------("lualibs-util-seq.lua")---!sequencers (function chaining) + ----------("lualibs-util-mrg.lua")---!only relevant in mtx-package + loadmodule("lualibs-util-prs.lua")--- miscellaneous parsers; cool. cool cool cool + ----------("lualibs-util-fmt.lua")---!column formatter (rarely used) + loadmodule("lualibs-util-dim.lua")--- conversions between dimensions + loadmodule("lualibs-util-jsn.lua")--- JSON parser + + ----------("lualibs-trac-set.lua")---!generalization of trackers + ----------("lualibs-trac-log.lua")---!logging + loadmodule("lualibs-trac-inf.lua")--- timing/statistics + loadmodule("lualibs-util-lua.lua")--- operations on lua bytecode + loadmodule("lualibs-util-deb.lua")--- extra debugging + loadmodule("lualibs-util-tpl.lua")--- templating + loadmodule("lualibs-util-sta.lua")--- stacker (for writing pdf) +end + +unfake_context() --- TODO check if this works at runtime + +lualibs.extended_loaded = true +-- vim:tw=71:sw=2:ts=2:expandtab + +% \end{macrocode} +% +% \iffalse +%</extended> +% \fi +% +% \Finale +\endinput diff --git a/macros/luatex/generic/lualibs/lualibs.pdf b/macros/luatex/generic/lualibs/lualibs.pdf Binary files differnew file mode 100644 index 0000000000..ad1ee3061b --- /dev/null +++ b/macros/luatex/generic/lualibs/lualibs.pdf |