diff options
author | Karl Berry <karl@freefriends.org> | 2013-06-23 23:38:05 +0000 |
---|---|---|
committer | Karl Berry <karl@freefriends.org> | 2013-06-23 23:38:05 +0000 |
commit | 60f28f85f2707b6a48c3b5aec0ea68b8ee21e466 (patch) | |
tree | ab34d8d573457bea6a804ee3d5de48dd500bf028 /Master/texmf-dist/context/data/scite | |
parent | ac5882d0cb115e11b5cc2e62107ef5822f3507f5 (diff) |
context
git-svn-id: svn://tug.org/texlive/trunk@30865 c570f23f-e606-0410-a88d-b1316a301751
Diffstat (limited to 'Master/texmf-dist/context/data/scite')
5 files changed, 1258 insertions, 411 deletions
diff --git a/Master/texmf-dist/context/data/scite/lexers/archive/scite-context-lexer-pre-3-3-1.lua b/Master/texmf-dist/context/data/scite/lexers/archive/scite-context-lexer-pre-3-3-1.lua new file mode 100644 index 00000000000..7883177b41e --- /dev/null +++ b/Master/texmf-dist/context/data/scite/lexers/archive/scite-context-lexer-pre-3-3-1.lua @@ -0,0 +1,1100 @@ +local info = { + version = 1.324, + comment = "basics for scintilla lpeg lexer for context/metafun", + author = "Hans Hagen, PRAGMA-ADE, Hasselt NL", + copyright = "PRAGMA ADE / ConTeXt Development Team", + license = "see context related readme files", + comment = "contains copyrighted code from mitchell.att.foicica.com", + +} + +-- todo: move all code here +-- todo: explore adapted dll ... properties + init + +-- The fold and lex functions are copied and patched from original code by Mitchell (see +-- lexer.lua). All errors are mine. +-- +-- Starting with SciTE version 3.20 there is an issue with coloring. As we still lack +-- a connection with scite itself (properties as well as printing to the log pane) we +-- cannot trace this (on windows). As far as I can see, there are no fundamental +-- changes in lexer.lua or LexLPeg.cxx so it must be in scintilla itself. So for the +-- moment I stick to 3.10. Indicators are: no lexing of 'next' and 'goto <label>' in the +-- Lua lexer and no brace highlighting either. Interesting is that it does work ok in +-- the cld lexer (so the Lua code is okay). Also the fact that char-def.lua lexes fast +-- is a signal that the lexer quits somewhere halfway. +-- +-- After checking 3.24 and adapting to the new lexer tables things are okay again. So, +-- this version assumes 3.24 or higher. In 3.24 we have a different token result, i.e. no +-- longer a { tag, pattern } but just two return values. I didn't check other changes but +-- will do that when I run into issues. +-- +-- I've considered making a whole copy and patch the other functions too as we need +-- an extra nesting model. However, I don't want to maintain too much. An unfortunate +-- change in 3.03 is that no longer a script can be specified. This means that instead +-- of loading the extensions via the properties file, we now need to load them in our +-- own lexers, unless of course we replace lexer.lua completely (which adds another +-- installation issue). +-- +-- Another change has been that _LEXERHOME is no longer available. It looks like more and +-- more functionality gets dropped so maybe at some point we need to ship our own dll/so +-- files. For instance, I'd like to have access to the current filename and other scite +-- properties. For instance, we could cache some info with each file, if only we had +-- knowledge of what file we're dealing with. +-- +-- For huge files folding can be pretty slow and I do have some large ones that I keep +-- open all the time. Loading is normally no ussue, unless one has remembered the status +-- and the cursor is at the last line of a 200K line file. Optimizing the fold function +-- brought down loading of char-def.lua from 14 sec => 8 sec. Replacing the word_match +-- function and optimizing the lex function gained another 2+ seconds. A 6 second load +-- is quite ok for me. The changed lexer table structure (no subtables) brings loading +-- down to a few seconds. +-- +-- When the lexer path is copied to the textadept lexer path, and the theme definition to +-- theme path (as lexer.lua), the lexer works there as well. When I have time and motive +-- I will make a proper setup file to tune the look and feel a bit and associate suffixes +-- with the context lexer. The textadept editor has a nice style tracing option but lacks +-- the tabs for selecting files that scite has. It also has no integrated run that pipes +-- to the log pane (I wonder if it could borrow code from the console2 project). Interesting +-- is that the jit version of textadept crashes on lexing large files (and does not feel +-- faster either). +-- +-- Function load(lexer_name) starts with _M.WHITESPACE = lexer_name..'_whitespace' which +-- means that we need to have it frozen at the moment we load another lexer. Because spacing +-- is used to revert to a parent lexer we need to make sure that we load children as late +-- as possible in order not to get the wrong whitespace trigger. This took me quite a while +-- to figure out (not being that familiar with the internals). The lex and fold functions +-- have been optimized. It is a pitty that there is no proper print available. Another thing +-- needed is a default style in ourown theme style definition, as otherwise we get wrong +-- nested lexers, especially if they are larger than a view. This is the hardest part of +-- getting things right. +-- +-- Eventually it might be safer to copy the other methods from lexer.lua here as well so +-- that we have no dependencies, apart from the c library (for which at some point the api +-- will be stable I hope). +-- +-- It's a pitty that there is no scintillua library for the OSX version of scite. Even +-- better would be to have the scintillua library as integral part of scite as that way I +-- could use OSX alongside windows and linux (depending on needs). Also nice would be to +-- have a proper interface to scite then because currently the lexer is rather isolated and the +-- lua version does not provide all standard libraries. It would also be good to have lpeg +-- support in the regular scite lua extension (currently you need to pick it up from someplace +-- else). + +local lpeg = require 'lpeg' + +local R, P, S, C, V, Cp, Cs, Ct, Cmt, Cc, Cf, Cg, Carg = lpeg.R, lpeg.P, lpeg.S, lpeg.C, lpeg.V, lpeg.Cp, lpeg.Cs, lpeg.Ct, lpeg.Cmt, lpeg.Cc, lpeg.Cf, lpeg.Cg, lpeg.Carg +local lpegmatch = lpeg.match +local find, gmatch, match, lower, upper, gsub = string.find, string.gmatch, string.match, string.lower, string.upper, string.gsub +local concat = table.concat +local global = _G +local type, next, setmetatable, rawset = type, next, setmetatable, rawset + +if lexer then + -- in recent c++ code the lexername and loading is hard coded +elseif _LEXERHOME then + dofile(_LEXERHOME .. '/lexer.lua') -- pre 3.03 situation +else + dofile('lexer.lua') -- whatever +end + +lexer.context = lexer.context or { } +local context = lexer.context + +context.patterns = context.patterns or { } +local patterns = context.patterns + +lexer._CONTEXTEXTENSIONS = true + +local locations = { + -- lexer.context.path, + "data", -- optional data directory + "..", -- regular scite directory +} + +local function collect(name) +-- local definitions = loadfile(name .. ".luc") or loadfile(name .. ".lua") + local okay, definitions = pcall(function () return require(name) end) + if okay then + if type(definitions) == "function" then + definitions = definitions() + end + if type(definitions) == "table" then + return definitions + end + end +end + +function context.loaddefinitions(name) + for i=1,#locations do + local data = collect(locations[i] .. "/" .. name) + if data then + return data + end + end +end + +-- maybe more efficient: + +function context.word_match(words,word_chars,case_insensitive) + local chars = '%w_' -- maybe just "" when word_chars + if word_chars then + chars = '^([' .. chars .. gsub(word_chars,'([%^%]%-])', '%%%1') ..']+)' + else + chars = '^([' .. chars ..']+)' + end + if case_insensitive then + local word_list = { } + for i=1,#words do + word_list[lower(words[i])] = true + end + return P(function(input, index) + local s, e, word = find(input,chars,index) + return word and word_list[lower(word)] and e + 1 or nil + end) + else + local word_list = { } + for i=1,#words do + word_list[words[i]] = true + end + return P(function(input, index) + local s, e, word = find(input,chars,index) + return word and word_list[word] and e + 1 or nil + end) + end +end + +local idtoken = R("az","AZ","\127\255","__") +local digit = R("09") +local sign = S("+-") +local period = P(".") +local space = S(" \n\r\t\f\v") + +patterns.idtoken = idtoken + +patterns.digit = digit +patterns.sign = sign +patterns.period = period + +patterns.cardinal = digit^1 +patterns.integer = sign^-1 * digit^1 + +patterns.real = + sign^-1 * ( -- at most one + digit^1 * period * digit^0 -- 10.0 10. + + digit^0 * period * digit^1 -- 0.10 .10 + + digit^1 -- 10 + ) + +patterns.restofline = (1-S("\n\r"))^1 +patterns.space = space +patterns.spacing = space^1 +patterns.nospacing = (1-space)^1 +patterns.anything = P(1) + +local endof = S("\n\r\f") + +patterns.startofline = P(function(input,index) + return (index == 1 or lpegmatch(endof,input,index-1)) and index +end) + +function context.exact_match(words,word_chars,case_insensitive) + local characters = concat(words) + local pattern -- the concat catches _ etc + if word_chars == true or word_chars == false or word_chars == nil then + word_chars = "" + end + if type(word_chars) == "string" then + pattern = S(characters) + idtoken + if case_insensitive then + pattern = pattern + S(upper(characters)) + S(lower(characters)) + end + if word_chars ~= "" then + pattern = pattern + S(word_chars) + end + elseif word_chars then + pattern = word_chars + end + if case_insensitive then + local list = { } + for i=1,#words do + list[lower(words[i])] = true + end + return Cmt(pattern^1, function(_,i,s) + return list[lower(s)] -- and i or nil + end) + else + local list = { } + for i=1,#words do + list[words[i]] = true + end + return Cmt(pattern^1, function(_,i,s) + return list[s] -- and i or nil + end) + end +end + +-- spell checking (we can only load lua files) +-- +-- return { +-- min = 3, +-- max = 40, +-- n = 12345, +-- words = { +-- ["someword"] = "someword", +-- ["anotherword"] = "Anotherword", +-- }, +-- } + +local lists = { } + +function context.setwordlist(tag,limit) -- returns hash (lowercase keys and original values) + if not tag or tag == "" then + return false, 3 + end + local list = lists[tag] + if not list then + list = context.loaddefinitions("spell-" .. tag) + if not list or type(list) ~= "table" then + list = { words = false, min = 3 } + else + list.words = list.words or false + list.min = list.min or 3 + end + lists[tag] = list + end + return list.words, list.min +end + +patterns.wordtoken = R("az","AZ","\127\255") +patterns.wordpattern = patterns.wordtoken^3 -- todo: if limit and #s < limit then + +-- -- pre 3.24: +-- +-- function context.checkedword(validwords,validminimum,s,i) -- ,limit +-- if not validwords then -- or #s < validminimum then +-- return true, { "text", i } -- { "default", i } +-- else +-- -- keys are lower +-- local word = validwords[s] +-- if word == s then +-- return true, { "okay", i } -- exact match +-- elseif word then +-- return true, { "warning", i } -- case issue +-- else +-- local word = validwords[lower(s)] +-- if word == s then +-- return true, { "okay", i } -- exact match +-- elseif word then +-- return true, { "warning", i } -- case issue +-- elseif upper(s) == s then +-- return true, { "warning", i } -- probably a logo or acronym +-- else +-- return true, { "error", i } +-- end +-- end +-- end +-- end + +function context.checkedword(validwords,validminimum,s,i) -- ,limit + if not validwords then -- or #s < validminimum then + return true, "text", i -- { "default", i } + else + -- keys are lower + local word = validwords[s] + if word == s then + return true, "okay", i -- exact match + elseif word then + return true, "warning", i -- case issue + else + local word = validwords[lower(s)] + if word == s then + return true, "okay", i -- exact match + elseif word then + return true, "warning", i -- case issue + elseif upper(s) == s then + return true, "warning", i -- probably a logo or acronym + else + return true, "error", i + end + end + end +end + +function context.styleofword(validwords,validminimum,s) -- ,limit + if not validwords or #s < validminimum then + return "text" + else + -- keys are lower + local word = validwords[s] + if word == s then + return "okay" -- exact match + elseif word then + return "warning" -- case issue + else + local word = validwords[lower(s)] + if word == s then + return "okay" -- exact match + elseif word then + return "warning" -- case issue + elseif upper(s) == s then + return "warning" -- probably a logo or acronym + else + return "error" + end + end + end +end + +-- overloaded functions + +local FOLD_BASE = SC_FOLDLEVELBASE +local FOLD_HEADER = SC_FOLDLEVELHEADERFLAG +local FOLD_BLANK = SC_FOLDLEVELWHITEFLAG + +local get_style_at = GetStyleAt +local get_property = GetProperty +local get_indent_amount = GetIndentAmount + +local h_table, b_table, n_table = { }, { }, { } + +setmetatable(h_table, { __index = function(t,level) local v = { level, FOLD_HEADER } t[level] = v return v end }) +setmetatable(b_table, { __index = function(t,level) local v = { level, FOLD_BLANK } t[level] = v return v end }) +setmetatable(n_table, { __index = function(t,level) local v = { level } t[level] = v return v end }) + +-- -- todo: move the local functions outside (see below) .. old variant < 3.24 +-- +-- local newline = P("\r\n") + S("\r\n") +-- local p_yes = Cp() * Cs((1-newline)^1) * newline^-1 +-- local p_nop = newline +-- +-- local function fold_by_parsing(text,start_pos,start_line,start_level,lexer) +-- local foldsymbols = lexer._foldsymbols +-- if not foldsymbols then +-- return { } +-- end +-- local patterns = foldsymbols._patterns +-- if not patterns then +-- return { } +-- end +-- local nofpatterns = #patterns +-- if nofpatterns == 0 then +-- return { } +-- end +-- local folds = { } +-- local line_num = start_line +-- local prev_level = start_level +-- local current_level = prev_level +-- local validmatches = foldsymbols._validmatches +-- if not validmatches then +-- validmatches = { } +-- for symbol, matches in next, foldsymbols do -- whatever = { start = 1, stop = -1 } +-- if not find(symbol,"^_") then -- brrr +-- for s, _ in next, matches do +-- validmatches[s] = true +-- end +-- end +-- end +-- foldsymbols._validmatches = validmatches +-- end +-- -- of course we could instead build a nice lpeg checker .. something for +-- -- a rainy day with a stack of new cd's at hand +-- local function action_y(pos,line) +-- for i=1,nofpatterns do +-- for s, m in gmatch(line,patterns[i]) do +-- if validmatches[m] then +-- local symbols = foldsymbols[get_style_at(start_pos + pos + s - 1)] +-- if symbols then +-- local action = symbols[m] +-- if action then +-- if type(action) == 'number' then -- we could store this in validmatches if there was only one symbol category +-- current_level = current_level + action +-- else +-- current_level = current_level + action(text,pos,line,s,m) +-- end +-- if current_level < FOLD_BASE then +-- current_level = FOLD_BASE +-- end +-- end +-- end +-- end +-- end +-- end +-- if current_level > prev_level then +-- folds[line_num] = h_table[prev_level] -- { prev_level, FOLD_HEADER } +-- else +-- folds[line_num] = n_table[prev_level] -- { prev_level } +-- end +-- prev_level = current_level +-- line_num = line_num + 1 +-- end +-- local function action_n() +-- folds[line_num] = b_table[prev_level] -- { prev_level, FOLD_BLANK } +-- line_num = line_num + 1 +-- end +-- if lexer._reset_parser then +-- lexer._reset_parser() +-- end +-- local lpegpattern = (p_yes/action_y + p_nop/action_n)^0 -- not too efficient but indirect function calls are neither but +-- lpegmatch(lpegpattern,text) -- keys are not pressed that fast ... large files are slow anyway +-- return folds +-- end + +-- The 3.24 variant; no longer subtable optimization is needed: + +local newline = P("\r\n") + S("\r\n") +local p_yes = Cp() * Cs((1-newline)^1) * newline^-1 +local p_nop = newline + +local folders = { } + +local function fold_by_parsing(text,start_pos,start_line,start_level,lexer) + local folder = folders[lexer] + if not folder then + -- + local pattern, folds, text, start_pos, line_num, prev_level, current_level + -- + local fold_symbols = lexer._foldsymbols + local fold_pattern = lexer._foldpattern -- use lpeg instead (context extension) + -- + if fold_pattern then + -- if no functions are found then we could have a faster one + + -- fold_pattern = Cp() * C(fold_pattern) * Carg(1) / function(s,match,pos) + -- local symbols = fold_symbols[get_style_at(start_pos + pos + s - 1)] + -- local l = symbols and symbols[match] + -- if l then + -- local t = type(l) + -- if t == 'number' then + -- current_level = current_level + l + -- elseif t == 'function' then + -- current_level = current_level + l(text, pos, line, s, match) + -- end + -- end + -- end + -- fold_pattern = (fold_pattern + P(1))^0 + -- local action_y = function(pos,line) + -- lpegmatch(fold_pattern,line,1,pos) + -- folds[line_num] = prev_level + -- if current_level > prev_level then + -- folds[line_num] = prev_level + FOLD_HEADER + -- end + -- if current_level < FOLD_BASE then + -- current_level = FOLD_BASE + -- end + -- prev_level = current_level + -- line_num = line_num + 1 + -- end + -- local action_n = function() + -- folds[line_num] = prev_level + FOLD_BLANK + -- line_num = line_num + 1 + -- end + -- pattern = (p_yes/action_y + p_nop/action_n)^0 + + fold_pattern = Cp() * C(fold_pattern) / function(s,match) + local symbols = fold_symbols[get_style_at(start_pos + s)] + if symbols then + local l = symbols[match] + if l then + current_level = current_level + l + end + end + end + local action_y = function() + folds[line_num] = prev_level + if current_level > prev_level then + folds[line_num] = prev_level + FOLD_HEADER + end + if current_level < FOLD_BASE then + current_level = FOLD_BASE + end + prev_level = current_level + line_num = line_num + 1 + end + local action_n = function() + folds[line_num] = prev_level + FOLD_BLANK + line_num = line_num + 1 + end + pattern = ((fold_pattern + (1-newline))^1 * newline / action_y + newline/action_n)^0 + + else + -- the traditional one but a bit optimized + local fold_symbols_patterns = fold_symbols._patterns + local action_y = function(pos,line) + for j = 1, #fold_symbols_patterns do + for s, match in gmatch(line,fold_symbols_patterns[j]) do -- '()('..patterns[i]..')' + local symbols = fold_symbols[get_style_at(start_pos + pos + s - 1)] + local l = symbols and symbols[match] + local t = type(l) + if t == 'number' then + current_level = current_level + l + elseif t == 'function' then + current_level = current_level + l(text, pos, line, s, match) + end + end + end + folds[line_num] = prev_level + if current_level > prev_level then + folds[line_num] = prev_level + FOLD_HEADER + end + if current_level < FOLD_BASE then + current_level = FOLD_BASE + end + prev_level = current_level + line_num = line_num + 1 + end + local action_n = function() + folds[line_num] = prev_level + FOLD_BLANK + line_num = line_num + 1 + end + pattern = (p_yes/action_y + p_nop/action_n)^0 + end + -- + local reset_parser = lexer._reset_parser + -- + folder = function(_text_,_start_pos_,_start_line_,_start_level_) + if reset_parser then + reset_parser() + end + folds = { } + text = _text_ + start_pos = _start_pos_ + line_num = _start_line_ + prev_level = _start_level_ + current_level = prev_level + lpegmatch(pattern,text) +-- return folds +local t = folds +folds = nil +return t -- so folds can be collected + end + folders[lexer] = folder + end + return folder(text,start_pos,start_line,start_level,lexer) +end + +-- local function fold_by_indentation(text,start_pos,start_line,start_level) +-- local folds = { } +-- local current_line = start_line +-- local prev_level = start_level +-- for line in gmatch(text,'[\t ]*(.-)\r?\n') do +-- if line ~= "" then +-- local current_level = FOLD_BASE + get_indent_amount(current_line) +-- if current_level > prev_level then -- next level +-- local i = current_line - 1 +-- while true do +-- local f = folds[i] +-- if f and f[2] == FOLD_BLANK then +-- i = i - 1 +-- else +-- break +-- end +-- end +-- local f = folds[i] +-- if f then +-- f[2] = FOLD_HEADER +-- end -- low indent +-- folds[current_line] = n_table[current_level] -- { current_level } -- high indent +-- elseif current_level < prev_level then -- prev level +-- local f = folds[current_line - 1] +-- if f then +-- f[1] = prev_level -- high indent +-- end +-- folds[current_line] = n_table[current_level] -- { current_level } -- low indent +-- else -- same level +-- folds[current_line] = n_table[prev_level] -- { prev_level } +-- end +-- prev_level = current_level +-- else +-- folds[current_line] = b_table[prev_level] -- { prev_level, FOLD_BLANK } +-- end +-- current_line = current_line + 1 +-- end +-- return folds +-- end + +-- local function fold_by_indentation(text,start_pos,start_line,start_level) +-- local folds = { } +-- local current_line = start_line +-- local prev_level = start_level +-- for line in gmatch(text,'[\t ]*(.-)\r?\n') do +-- if line ~= '' then +-- local current_level = FOLD_BASE + get_indent_amount(current_line) +-- if current_level > prev_level then -- next level +-- local i = current_line - 1 +-- local f +-- while true do +-- f = folds[i] +-- if not f then +-- break +-- elseif f[2] == FOLD_BLANK then +-- i = i - 1 +-- else +-- f[2] = FOLD_HEADER -- low indent +-- break +-- end +-- end +-- folds[current_line] = { current_level } -- high indent +-- elseif current_level < prev_level then -- prev level +-- local f = folds[current_line - 1] +-- if f then +-- f[1] = prev_level -- high indent +-- end +-- folds[current_line] = { current_level } -- low indent +-- else -- same level +-- folds[current_line] = { prev_level } +-- end +-- prev_level = current_level +-- else +-- folds[current_line] = { prev_level, FOLD_BLANK } +-- end +-- current_line = current_line + 1 +-- end +-- for line, level in next, folds do +-- folds[line] = level[1] + (level[2] or 0) +-- end +-- return folds +-- end + +local folds, current_line, prev_level + +local function action_y() + local current_level = FOLD_BASE + get_indent_amount(current_line) + if current_level > prev_level then -- next level + local i = current_line - 1 + local f + while true do + f = folds[i] + if not f then + break + elseif f[2] == FOLD_BLANK then + i = i - 1 + else + f[2] = FOLD_HEADER -- low indent + break + end + end + folds[current_line] = { current_level } -- high indent + elseif current_level < prev_level then -- prev level + local f = folds[current_line - 1] + if f then + f[1] = prev_level -- high indent + end + folds[current_line] = { current_level } -- low indent + else -- same level + folds[current_line] = { prev_level } + end + prev_level = current_level + current_line = current_line + 1 +end + +local function action_n() + folds[current_line] = { prev_level, FOLD_BLANK } + current_line = current_line + 1 +end + +local pattern = ( S("\t ")^0 * ( (1-S("\n\r"))^1 / action_y + P(true) / action_n) * newline )^0 + +local function fold_by_indentation(text,start_pos,start_line,start_level) + -- initialize + folds = { } + current_line = start_line + prev_level = start_level + -- define + -- -- not here .. pattern binds and local functions are not frozen + -- analyze + lpegmatch(pattern,text) + -- flatten + for line, level in next, folds do + folds[line] = level[1] + (level[2] or 0) + end + -- done +-- return folds +local t = folds +folds = nil +return t -- so folds can be collected +end + +local function fold_by_line(text,start_pos,start_line,start_level) + local folds = { } + -- can also be lpeg'd + for _ in gmatch(text,".-\r?\n") do + folds[start_line] = n_table[start_level] -- { start_level } + start_line = start_line + 1 + end + return folds +end + +local threshold_by_lexer = 512 * 1024 -- we don't know the filesize yet +local threshold_by_parsing = 512 * 1024 -- we don't know the filesize yet +local threshold_by_indentation = 512 * 1024 -- we don't know the filesize yet +local threshold_by_line = 512 * 1024 -- we don't know the filesize yet + +function context.fold(text,start_pos,start_line,start_level) -- hm, we had size thresholds .. where did they go + if text == '' then + return { } + end + local lexer = global._LEXER + local fold_by_lexer = lexer._fold + local fold_by_symbols = lexer._foldsymbols + local filesize = 0 -- we don't know that + if fold_by_lexer then + if filesize <= threshold_by_lexer then + return fold_by_lexer(text,start_pos,start_line,start_level,lexer) + end + elseif fold_by_symbols then -- and get_property('fold.by.parsing',1) > 0 then + if filesize <= threshold_by_parsing then + return fold_by_parsing(text,start_pos,start_line,start_level,lexer) + end + elseif get_property('fold.by.indentation',1) > 0 then + if filesize <= threshold_by_indentation then + return fold_by_indentation(text,start_pos,start_line,start_level,lexer) + end + elseif get_property('fold.by.line',1) > 0 then + if filesize <= threshold_by_line then + return fold_by_line(text,start_pos,start_line,start_level,lexer) + end + end + return { } +end + +-- The following code is mostly unchanged: + +local function add_rule(lexer, id, rule) + if not lexer._RULES then + lexer._RULES = {} + lexer._RULEORDER = {} + end + lexer._RULES[id] = rule + lexer._RULEORDER[#lexer._RULEORDER + 1] = id +end + +local function add_style(lexer, token_name, style) + local len = lexer._STYLES.len + if len == 32 then + len = len + 8 + end + if len >= 128 then + print('Too many styles defined (128 MAX)') + end + lexer._TOKENS[token_name] = len + lexer._STYLES[len] = style + lexer._STYLES.len = len + 1 +end + +local function join_tokens(lexer) + local patterns, order = lexer._RULES, lexer._RULEORDER + local token_rule = patterns[order[1]] + for i=2,#order do + token_rule = token_rule + patterns[order[i]] + end + lexer._TOKENRULE = token_rule + return lexer._TOKENRULE +end + +local function add_lexer(grammar, lexer, token_rule) + local token_rule = join_tokens(lexer) + local lexer_name = lexer._NAME + local children = lexer._CHILDREN + for i=1,#children do + local child = children[i] + if child._CHILDREN then + add_lexer(grammar, child) + end + local child_name = child._NAME + local rules = child._EMBEDDEDRULES[lexer_name] + local rules_token_rule = grammar['__'..child_name] or rules.token_rule + grammar[child_name] = (-rules.end_rule * rules_token_rule)^0 * rules.end_rule^-1 * V(lexer_name) + local embedded_child = '_' .. child_name + grammar[embedded_child] = rules.start_rule * (-rules.end_rule * rules_token_rule)^0 * rules.end_rule^-1 + token_rule = V(embedded_child) + token_rule + end + grammar['__' .. lexer_name] = token_rule + grammar[lexer_name] = token_rule^0 +end + +local function build_grammar(lexer, initial_rule) + local children = lexer._CHILDREN + if children then + local lexer_name = lexer._NAME + if not initial_rule then + initial_rule = lexer_name + end + local grammar = { initial_rule } + add_lexer(grammar, lexer) + lexer._INITIALRULE = initial_rule + lexer._GRAMMAR = Ct(P(grammar)) + else + lexer._GRAMMAR = Ct(join_tokens(lexer)^0) + end +end + +-- so far. We need these local functions in the next one. +-- +-- Before 3.24 we had tokens[..] = { category, position }, now it's a two values. + +local lineparsers = { } + +function context.lex(text,init_style) + local lexer = global._LEXER + local grammar = lexer._GRAMMAR + if not grammar then + return { } + elseif lexer._LEXBYLINE then -- we could keep token + local tokens = { } + local offset = 0 + local noftokens = 0 + -- -- pre 3.24 + -- + -- for line in gmatch(text,'[^\r\n]*\r?\n?') do -- could be an lpeg + -- local line_tokens = lpegmatch(grammar,line) + -- if line_tokens then + -- for i=1,#line_tokens do + -- local token = line_tokens[i] + -- token[2] = token[2] + offset + -- noftokens = noftokens + 1 + -- tokens[noftokens] = token + -- end + -- end + -- offset = offset + #line + -- if noftokens > 0 and tokens[noftokens][2] ~= offset then + -- noftokens = noftokens + 1 + -- tokens[noftokens] = { 'default', offset + 1 } + -- end + -- end + + -- for line in gmatch(text,'[^\r\n]*\r?\n?') do + -- local line_tokens = lpegmatch(grammar,line) + -- if line_tokens then + -- for i=1,#line_tokens,2 do + -- noftokens = noftokens + 1 + -- tokens[noftokens] = line_tokens[i] + -- noftokens = noftokens + 1 + -- tokens[noftokens] = line_tokens[i + 1] + offset + -- end + -- end + -- offset = offset + #line + -- if noftokens > 0 and tokens[noftokens] ~= offset then + -- noftokens = noftokens + 1 + -- tokens[noftokens] = 'default' + -- noftokens = noftokens + 1 + -- tokens[noftokens] = offset + 1 + -- end + -- end + + local lineparser = lineparsers[lexer] + if not lineparser then -- probably a cmt is more efficient + lineparser = C((1-newline)^0 * newline) / function(line) + local length = #line + local line_tokens = length > 0 and lpegmatch(grammar,line) + if line_tokens then + for i=1,#line_tokens,2 do + noftokens = noftokens + 1 + tokens[noftokens] = line_tokens[i] + noftokens = noftokens + 1 + tokens[noftokens] = line_tokens[i + 1] + offset + end + end + offset = offset + length + if noftokens > 0 and tokens[noftokens] ~= offset then + noftokens = noftokens + 1 + tokens[noftokens] = 'default' + noftokens = noftokens + 1 + tokens[noftokens] = offset + 1 + end + end + lineparser = lineparser^0 + lineparsers[lexer] = lineparser + end + lpegmatch(lineparser,text) + return tokens + + elseif lexer._CHILDREN then + -- as we cannot print, tracing is not possible ... this might change as we can as well + -- generate them all in one go (sharing as much as possible) + local hash = lexer._HASH -- hm, was _hash + if not hash then + hash = { } + lexer._HASH = hash + end + grammar = hash[init_style] + if grammar then + lexer._GRAMMAR = grammar + else + for style, style_num in next, lexer._TOKENS do + if style_num == init_style then + -- the name of the lexers is filtered from the whitespace + -- specification + local lexer_name = match(style,'^(.+)_whitespace') or lexer._NAME + if lexer._INITIALRULE ~= lexer_name then + grammar = hash[lexer_name] + if not grammar then + build_grammar(lexer,lexer_name) + grammar = lexer._GRAMMAR + hash[lexer_name] = grammar + end + end + break + end + end + grammar = grammar or lexer._GRAMMAR + hash[init_style] = grammar + end + return lpegmatch(grammar,text) + else + return lpegmatch(grammar,text) + end +end + +-- todo: keywords: one lookup and multiple matches + +-- function context.token(name, patt) +-- return Ct(patt * Cc(name) * Cp()) +-- end +-- +-- -- hm, changed in 3.24 .. no longer a table + +function context.token(name, patt) + return patt * Cc(name) * Cp() +end + +lexer.fold = context.fold +lexer.lex = context.lex +lexer.token = context.token +lexer.exact_match = context.exact_match + +-- helper .. alas ... the lexer's lua instance is rather crippled .. not even +-- math is part of it + +local floor = math and math.floor +local char = string.char + +if not floor then + + floor = function(n) + return tonumber(string.format("%d",n)) + end + + math = math or { } + + math.floor = floor + +end + +local function utfchar(n) + if n < 0x80 then + return char(n) + elseif n < 0x800 then + return char( + 0xC0 + floor(n/0x40), + 0x80 + (n % 0x40) + ) + elseif n < 0x10000 then + return char( + 0xE0 + floor(n/0x1000), + 0x80 + (floor(n/0x40) % 0x40), + 0x80 + (n % 0x40) + ) + elseif n < 0x40000 then + return char( + 0xF0 + floor(n/0x40000), + 0x80 + floor(n/0x1000), + 0x80 + (floor(n/0x40) % 0x40), + 0x80 + (n % 0x40) + ) + else + -- return char( + -- 0xF1 + floor(n/0x1000000), + -- 0x80 + floor(n/0x40000), + -- 0x80 + floor(n/0x1000), + -- 0x80 + (floor(n/0x40) % 0x40), + -- 0x80 + (n % 0x40) + -- ) + return "?" + end +end + +context.utfchar = utfchar + +-- a helper from l-lpeg: + +local gmatch = string.gmatch + +local function make(t) + local p + for k, v in next, t do + if not p then + if next(v) then + p = P(k) * make(v) + else + p = P(k) + end + else + if next(v) then + p = p + P(k) * make(v) + else + p = p + P(k) + end + end + end + return p +end + +function lpeg.utfchartabletopattern(list) + local tree = { } + for i=1,#list do + local t = tree + for c in gmatch(list[i],".") do + if not t[c] then + t[c] = { } + end + t = t[c] + end + end + return make(tree) +end + +-- patterns.invisibles = +-- P(utfchar(0x00A0)) -- nbsp +-- + P(utfchar(0x2000)) -- enquad +-- + P(utfchar(0x2001)) -- emquad +-- + P(utfchar(0x2002)) -- enspace +-- + P(utfchar(0x2003)) -- emspace +-- + P(utfchar(0x2004)) -- threeperemspace +-- + P(utfchar(0x2005)) -- fourperemspace +-- + P(utfchar(0x2006)) -- sixperemspace +-- + P(utfchar(0x2007)) -- figurespace +-- + P(utfchar(0x2008)) -- punctuationspace +-- + P(utfchar(0x2009)) -- breakablethinspace +-- + P(utfchar(0x200A)) -- hairspace +-- + P(utfchar(0x200B)) -- zerowidthspace +-- + P(utfchar(0x202F)) -- narrownobreakspace +-- + P(utfchar(0x205F)) -- math thinspace + +patterns.invisibles = lpeg.utfchartabletopattern { + utfchar(0x00A0), -- nbsp + utfchar(0x2000), -- enquad + utfchar(0x2001), -- emquad + utfchar(0x2002), -- enspace + utfchar(0x2003), -- emspace + utfchar(0x2004), -- threeperemspace + utfchar(0x2005), -- fourperemspace + utfchar(0x2006), -- sixperemspace + utfchar(0x2007), -- figurespace + utfchar(0x2008), -- punctuationspace + utfchar(0x2009), -- breakablethinspace + utfchar(0x200A), -- hairspace + utfchar(0x200B), -- zerowidthspace + utfchar(0x202F), -- narrownobreakspace + utfchar(0x205F), -- math thinspace +} + +-- now we can make: + +patterns.iwordtoken = patterns.wordtoken - patterns.invisibles +patterns.iwordpattern = patterns.iwordtoken^3 + +-- require("themes/scite-context-theme") + +-- In order to deal with some bug in additional styles (I have no cue what is +-- wrong, but additional styles get ignored and clash somehow) I just copy the +-- original lexer code ... see original for comments. diff --git a/Master/texmf-dist/context/data/scite/lexers/data/scite-context-data-context.lua b/Master/texmf-dist/context/data/scite/lexers/data/scite-context-data-context.lua index 208a2679ace..0894e86884f 100644 --- a/Master/texmf-dist/context/data/scite/lexers/data/scite-context-data-context.lua +++ b/Master/texmf-dist/context/data/scite/lexers/data/scite-context-data-context.lua @@ -1,4 +1,4 @@ return { ["constants"]={ "zerocount", "minusone", "minustwo", "plusone", "plustwo", "plusthree", "plusfour", "plusfive", "plussix", "plusseven", "pluseight", "plusnine", "plusten", "plussixteen", "plushundred", "plusthousand", "plustenthousand", "plustwentythousand", "medcard", "maxcard", "zeropoint", "onepoint", "halfapoint", "onebasepoint", "maxdimen", "scaledpoint", "thousandpoint", "points", "halfpoint", "zeroskip", "zeromuskip", "onemuskip", "pluscxxvii", "pluscxxviii", "pluscclv", "pluscclvi", "normalpagebox", "endoflinetoken", "outputnewlinechar", "emptytoks", "empty", "undefined", "voidbox", "emptybox", "emptyvbox", "emptyhbox", "bigskipamount", "medskipamount", "smallskipamount", "fmtname", "fmtversion", "texengine", "texenginename", "texengineversion", "luatexengine", "pdftexengine", "xetexengine", "unknownengine", "etexversion", "pdftexversion", "xetexversion", "xetexrevision", "activecatcode", "bgroup", "egroup", "endline", "conditionaltrue", "conditionalfalse", "attributeunsetvalue", "uprotationangle", "rightrotationangle", "downrotationangle", "leftrotationangle", "inicatcodes", "ctxcatcodes", "texcatcodes", "notcatcodes", "txtcatcodes", "vrbcatcodes", "prtcatcodes", "nilcatcodes", "luacatcodes", "tpacatcodes", "tpbcatcodes", "xmlcatcodes", "escapecatcode", "begingroupcatcode", "endgroupcatcode", "mathshiftcatcode", "alignmentcatcode", "endoflinecatcode", "parametercatcode", "superscriptcatcode", "subscriptcatcode", "ignorecatcode", "spacecatcode", "lettercatcode", "othercatcode", "activecatcode", "commentcatcode", "invalidcatcode", "tabasciicode", "newlineasciicode", "formfeedasciicode", "endoflineasciicode", "endoffileasciicode", "spaceasciicode", "hashasciicode", "dollarasciicode", "commentasciicode", "ampersandasciicode", "colonasciicode", "backslashasciicode", "circumflexasciicode", "underscoreasciicode", "leftbraceasciicode", "barasciicode", "rightbraceasciicode", "tildeasciicode", "delasciicode", "lessthanasciicode", "morethanasciicode", "doublecommentsignal", "atsignasciicode", "exclamationmarkasciicode", "questionmarkasciicode", "doublequoteasciicode", "singlequoteasciicode", "forwardslashasciicode", "primeasciicode", "activemathcharcode", "activetabtoken", "activeformfeedtoken", "activeendoflinetoken", "batchmodecode", "nonstopmodecode", "scrollmodecode", "errorstopmodecode", "bottomlevelgroupcode", "simplegroupcode", "hboxgroupcode", "adjustedhboxgroupcode", "vboxgroupcode", "vtopgroupcode", "aligngroupcode", "noaligngroupcode", "outputgroupcode", "mathgroupcode", "discretionarygroupcode", "insertgroupcode", "vcentergroupcode", "mathchoicegroupcode", "semisimplegroupcode", "mathshiftgroupcode", "mathleftgroupcode", "vadjustgroupcode", "charnodecode", "hlistnodecode", "vlistnodecode", "rulenodecode", "insertnodecode", "marknodecode", "adjustnodecode", "ligaturenodecode", "discretionarynodecode", "whatsitnodecode", "mathnodecode", "gluenodecode", "kernnodecode", "penaltynodecode", "unsetnodecode", "mathsnodecode", "charifcode", "catifcode", "numifcode", "dimifcode", "oddifcode", "vmodeifcode", "hmodeifcode", "mmodeifcode", "innerifcode", "voidifcode", "hboxifcode", "vboxifcode", "xifcode", "eofifcode", "trueifcode", "falseifcode", "caseifcode", "definedifcode", "csnameifcode", "fontcharifcode", "fontslantperpoint", "fontinterwordspace", "fontinterwordstretch", "fontinterwordshrink", "fontexheight", "fontemwidth", "fontextraspace", "slantperpoint", "interwordspace", "interwordstretch", "interwordshrink", "exheight", "emwidth", "extraspace", "mathsupdisplay", "mathsupnormal", "mathsupcramped", "mathsubnormal", "mathsubcombined", "mathaxisheight", "startmode", "stopmode", "startnotmode", "stopnotmode", "startmodeset", "stopmodeset", "doifmode", "doifmodeelse", "doifnotmode", "startallmodes", "stopallmodes", "startnotallmodes", "stopnotallmodes", "doifallmodes", "doifallmodeselse", "doifnotallmodes", "startenvironment", "stopenvironment", "environment", "startcomponent", "stopcomponent", "component", "startproduct", "stopproduct", "product", "startproject", "stopproject", "project", "starttext", "stoptext", "startnotext", "stopnotext", "startdocument", "stopdocument", "documentvariable", "setupdocument", "startmodule", "stopmodule", "usemodule", "usetexmodule", "useluamodule", "startTEXpage", "stopTEXpage", "enablemode", "disablemode", "preventmode", "globalenablemode", "globaldisablemode", "globalpreventmode", "pushmode", "popmode", "typescriptone", "typescripttwo", "typescriptthree", "mathsizesuffix", "mathordcode", "mathopcode", "mathbincode", "mathrelcode", "mathopencode", "mathclosecode", "mathpunctcode", "mathalphacode", "mathinnercode", "mathnothingcode", "mathlimopcode", "mathnolopcode", "mathboxcode", "mathchoicecode", "mathaccentcode", "mathradicalcode", "constantnumber", "constantnumberargument", "constantdimen", "constantdimenargument", "constantemptyargument", "continueifinputfile", "luastringsep", "!!bs", "!!es" }, - ["helpers"]={ "startsetups", "stopsetups", "startxmlsetups", "stopxmlsetups", "startluasetups", "stopluasetups", "starttexsetups", "stoptexsetups", "startrawsetups", "stoprawsetups", "startlocalsetups", "stoplocalsetups", "starttexdefinition", "stoptexdefinition", "starttexcode", "stoptexcode", "startcontextcode", "stopcontextcode", "doifsetupselse", "doifsetups", "doifnotsetups", "setup", "setups", "texsetup", "xmlsetup", "luasetup", "directsetup", "doifelsecommandhandler", "doifnotcommandhandler", "doifcommandhandler", "newmode", "setmode", "resetmode", "newsystemmode", "setsystemmode", "resetsystemmode", "pushsystemmode", "popsystemmode", "booleanmodevalue", "newcount", "newdimen", "newskip", "newmuskip", "newbox", "newtoks", "newread", "newwrite", "newmarks", "newinsert", "newattribute", "newif", "newlanguage", "newfamily", "newfam", "newhelp", "then", "firstargumentfalse", "firstargumenttrue", "secondargumentfalse", "secondargumenttrue", "thirdargumentfalse", "thirdargumenttrue", "fourthargumentfalse", "fourthargumenttrue", "fifthargumentfalse", "fifthsargumenttrue", "sixthargumentfalse", "sixtsargumenttrue", "doglobal", "dodoglobal", "redoglobal", "resetglobal", "donothing", "dontcomplain", "forgetall", "donetrue", "donefalse", "htdp", "unvoidbox", "hfilll", "vfilll", "mathbox", "mathlimop", "mathnolop", "mathnothing", "mathalpha", "currentcatcodetable", "defaultcatcodetable", "catcodetablename", "newcatcodetable", "startcatcodetable", "stopcatcodetable", "startextendcatcodetable", "stopextendcatcodetable", "pushcatcodetable", "popcatcodetable", "restorecatcodes", "setcatcodetable", "letcatcodecommand", "defcatcodecommand", "uedcatcodecommand", "hglue", "vglue", "hfillneg", "vfillneg", "hfilllneg", "vfilllneg", "ruledhss", "ruledhfil", "ruledhfill", "ruledhfilneg", "ruledhfillneg", "normalhfillneg", "ruledvss", "ruledvfil", "ruledvfill", "ruledvfilneg", "ruledvfillneg", "normalvfillneg", "ruledhbox", "ruledvbox", "ruledvtop", "ruledvcenter", "ruledhskip", "ruledvskip", "ruledkern", "ruledmskip", "ruledmkern", "ruledhglue", "ruledvglue", "normalhglue", "normalvglue", "ruledpenalty", "filledhboxb", "filledhboxr", "filledhboxg", "filledhboxc", "filledhboxm", "filledhboxy", "filledhboxk", "scratchcounter", "globalscratchcounter", "scratchdimen", "globalscratchdimen", "scratchskip", "globalscratchskip", "scratchmuskip", "globalscratchmuskip", "scratchtoks", "globalscratchtoks", "scratchbox", "globalscratchbox", "availablehsize", "localhsize", "setlocalhsize", "nextbox", "dowithnextbox", "dowithnextboxcs", "dowithnextboxcontent", "dowithnextboxcontentcs", "scratchwidth", "scratchheight", "scratchdepth", "scratchoffset", "scratchdistance", "scratchhsize", "scratchvsize", "scratchxoffset", "scratchyoffset", "scratchhoffset", "scratchvoffset", "scratchxposition", "scratchyposition", "scratchtopoffset", "scratchbottomoffset", "scratchleftoffset", "scratchrightoffset", "scratchcounterone", "scratchcountertwo", "scratchcounterthree", "scratchdimenone", "scratchdimentwo", "scratchdimenthree", "scratchskipone", "scratchskiptwo", "scratchskipthree", "scratchmuskipone", "scratchmuskiptwo", "scratchmuskipthree", "scratchtoksone", "scratchtokstwo", "scratchtoksthree", "scratchboxone", "scratchboxtwo", "scratchboxthree", "scratchnx", "scratchny", "scratchmx", "scratchmy", "scratchunicode", "scratchleftskip", "scratchrightskip", "scratchtopskip", "scratchbottomskip", "doif", "doifnot", "doifelse", "doifinset", "doifnotinset", "doifinsetelse", "doifnextcharelse", "doifnextoptionalelse", "doifnextbgroupelse", "doifnextparenthesiselse", "doiffastoptionalcheckelse", "doifundefinedelse", "doifdefinedelse", "doifundefined", "doifdefined", "doifelsevalue", "doifvalue", "doifnotvalue", "doifnothing", "doifsomething", "doifelsenothing", "doifsomethingelse", "doifvaluenothing", "doifvaluesomething", "doifelsevaluenothing", "doifdimensionelse", "doifnumberelse", "doifnumber", "doifnotnumber", "doifcommonelse", "doifcommon", "doifnotcommon", "doifinstring", "doifnotinstring", "doifinstringelse", "doifassignmentelse", "docheckassignment", "tracingall", "tracingnone", "loggingall", "removetoks", "appendtoks", "prependtoks", "appendtotoks", "prependtotoks", "to", "endgraf", "endpar", "everyendpar", "reseteverypar", "finishpar", "empty", "null", "space", "quad", "enspace", "obeyspaces", "obeylines", "obeyedspace", "obeyedline", "normalspace", "executeifdefined", "singleexpandafter", "doubleexpandafter", "tripleexpandafter", "dontleavehmode", "removelastspace", "removeunwantedspaces", "keepunwantedspaces", "wait", "writestatus", "define", "defineexpandable", "redefine", "setmeasure", "setemeasure", "setgmeasure", "setxmeasure", "definemeasure", "freezemeasure", "measure", "installcorenamespace", "getvalue", "setvalue", "setevalue", "setgvalue", "setxvalue", "letvalue", "letgvalue", "resetvalue", "undefinevalue", "ignorevalue", "setuvalue", "setuevalue", "setugvalue", "setuxvalue", "globallet", "glet", "udef", "ugdef", "uedef", "uxdef", "checked", "unique", "getparameters", "geteparameters", "getgparameters", "getxparameters", "forgetparameters", "copyparameters", "getdummyparameters", "dummyparameter", "directdummyparameter", "setdummyparameter", "letdummyparameter", "usedummystyleandcolor", "usedummystyleparameter", "usedummycolorparameter", "processcommalist", "processcommacommand", "quitcommalist", "quitprevcommalist", "processaction", "processallactions", "processfirstactioninset", "processallactionsinset", "unexpanded", "expanded", "startexpanded", "stopexpanded", "protected", "protect", "unprotect", "firstofoneargument", "firstoftwoarguments", "secondoftwoarguments", "firstofthreearguments", "secondofthreearguments", "thirdofthreearguments", "firstoffourarguments", "secondoffourarguments", "thirdoffourarguments", "fourthoffourarguments", "firstoffivearguments", "secondoffivearguments", "thirdoffivearguments", "fourthoffivearguments", "fifthoffivearguments", "firstofsixarguments", "secondofsixarguments", "thirdofsixarguments", "fourthofsixarguments", "fifthofsixarguments", "sixthofsixarguments", "firstofoneunexpanded", "gobbleoneargument", "gobbletwoarguments", "gobblethreearguments", "gobblefourarguments", "gobblefivearguments", "gobblesixarguments", "gobblesevenarguments", "gobbleeightarguments", "gobbleninearguments", "gobbletenarguments", "gobbleoneoptional", "gobbletwooptionals", "gobblethreeoptionals", "gobblefouroptionals", "gobblefiveoptionals", "dorecurse", "doloop", "exitloop", "dostepwiserecurse", "recurselevel", "recursedepth", "dofastloopcs", "newconstant", "setnewconstant", "newconditional", "settrue", "setfalse", "setconstant", "newmacro", "setnewmacro", "newfraction", "newsignal", "dosingleempty", "dodoubleempty", "dotripleempty", "doquadrupleempty", "doquintupleempty", "dosixtupleempty", "doseventupleempty", "dosingleargument", "dodoubleargument", "dotripleargument", "doquadrupleargument", "doquintupleargument", "dosixtupleargument", "doseventupleargument", "dosinglegroupempty", "dodoublegroupempty", "dotriplegroupempty", "doquadruplegroupempty", "doquintuplegroupempty", "permitspacesbetweengroups", "dontpermitspacesbetweengroups", "nopdfcompression", "maximumpdfcompression", "normalpdfcompression", "modulonumber", "dividenumber", "getfirstcharacter", "doiffirstcharelse", "startnointerference", "stopnointerference", "twodigits", "threedigits", "strut", "setstrut", "strutbox", "strutht", "strutdp", "strutwd", "struthtdp", "begstrut", "endstrut", "lineheight", "ordordspacing", "ordopspacing", "ordbinspacing", "ordrelspacing", "ordopenspacing", "ordclosespacing", "ordpunctspacing", "ordinnerspacing", "opordspacing", "opopspacing", "opbinspacing", "oprelspacing", "opopenspacing", "opclosespacing", "oppunctspacing", "opinnerspacing", "binordspacing", "binopspacing", "binbinspacing", "binrelspacing", "binopenspacing", "binclosespacing", "binpunctspacing", "bininnerspacing", "relordspacing", "relopspacing", "relbinspacing", "relrelspacing", "relopenspacing", "relclosespacing", "relpunctspacing", "relinnerspacing", "openordspacing", "openopspacing", "openbinspacing", "openrelspacing", "openopenspacing", "openclosespacing", "openpunctspacing", "openinnerspacing", "closeordspacing", "closeopspacing", "closebinspacing", "closerelspacing", "closeopenspacing", "closeclosespacing", "closepunctspacing", "closeinnerspacing", "punctordspacing", "punctopspacing", "punctbinspacing", "punctrelspacing", "punctopenspacing", "punctclosespacing", "punctpunctspacing", "punctinnerspacing", "innerordspacing", "inneropspacing", "innerbinspacing", "innerrelspacing", "inneropenspacing", "innerclosespacing", "innerpunctspacing", "innerinnerspacing", "normalreqno", "startimath", "stopimath", "normalstartimath", "normalstopimath", "startdmath", "stopdmath", "normalstartdmath", "normalstopdmath", "uncramped", "cramped", "triggermathstyle", "mathstylefont", "mathsmallstylefont", "mathstyleface", "mathsmallstyleface", "mathstylecommand", "mathpalette", "mathstylehbox", "mathstylevbox", "mathstylevcenter", "mathstylevcenteredhbox", "mathstylevcenteredvbox", "mathtext", "setmathsmalltextbox", "setmathtextbox", "triggerdisplaystyle", "triggertextstyle", "triggerscriptstyle", "triggerscriptscriptstyle", "triggeruncrampedstyle", "triggercrampedstyle", "triggersmallstyle", "triggeruncrampedsmallstyle", "triggercrampedsmallstyle", "triggerbigstyle", "triggeruncrampedbigstyle", "triggercrampedbigstyle", "luaexpr", "expdoifelse", "expdoif", "expdoifnot", "expdoifcommonelse", "expdoifinsetelse", "ctxdirectlua", "ctxlatelua", "ctxsprint", "ctxwrite", "ctxcommand", "ctxdirectcommand", "ctxlatecommand", "ctxreport", "ctxlua", "luacode", "lateluacode", "directluacode", "registerctxluafile", "ctxloadluafile", "luaversion", "luamajorversion", "luaminorversion", "ctxluacode", "luaconditional", "luaexpanded", "startluaparameterset", "stopluaparameterset", "luaparameterset", "definenamedlua", "obeylualines", "obeyluatokens", "startluacode", "stopluacode", "startlua", "stoplua", "carryoverpar", "Umathbotaccent" }, + ["helpers"]={ "startsetups", "stopsetups", "startxmlsetups", "stopxmlsetups", "startluasetups", "stopluasetups", "starttexsetups", "stoptexsetups", "startrawsetups", "stoprawsetups", "startlocalsetups", "stoplocalsetups", "starttexdefinition", "stoptexdefinition", "starttexcode", "stoptexcode", "startcontextcode", "stopcontextcode", "doifsetupselse", "doifsetups", "doifnotsetups", "setup", "setups", "texsetup", "xmlsetup", "luasetup", "directsetup", "doifelsecommandhandler", "doifnotcommandhandler", "doifcommandhandler", "newmode", "setmode", "resetmode", "newsystemmode", "setsystemmode", "resetsystemmode", "pushsystemmode", "popsystemmode", "booleanmodevalue", "newcount", "newdimen", "newskip", "newmuskip", "newbox", "newtoks", "newread", "newwrite", "newmarks", "newinsert", "newattribute", "newif", "newlanguage", "newfamily", "newfam", "newhelp", "then", "begcsname", "strippedcsname", "firstargumentfalse", "firstargumenttrue", "secondargumentfalse", "secondargumenttrue", "thirdargumentfalse", "thirdargumenttrue", "fourthargumentfalse", "fourthargumenttrue", "fifthargumentfalse", "fifthsargumenttrue", "sixthargumentfalse", "sixtsargumenttrue", "doglobal", "dodoglobal", "redoglobal", "resetglobal", "donothing", "dontcomplain", "forgetall", "donetrue", "donefalse", "htdp", "unvoidbox", "hfilll", "vfilll", "mathbox", "mathlimop", "mathnolop", "mathnothing", "mathalpha", "currentcatcodetable", "defaultcatcodetable", "catcodetablename", "newcatcodetable", "startcatcodetable", "stopcatcodetable", "startextendcatcodetable", "stopextendcatcodetable", "pushcatcodetable", "popcatcodetable", "restorecatcodes", "setcatcodetable", "letcatcodecommand", "defcatcodecommand", "uedcatcodecommand", "hglue", "vglue", "hfillneg", "vfillneg", "hfilllneg", "vfilllneg", "ruledhss", "ruledhfil", "ruledhfill", "ruledhfilneg", "ruledhfillneg", "normalhfillneg", "ruledvss", "ruledvfil", "ruledvfill", "ruledvfilneg", "ruledvfillneg", "normalvfillneg", "ruledhbox", "ruledvbox", "ruledvtop", "ruledvcenter", "ruledhskip", "ruledvskip", "ruledkern", "ruledmskip", "ruledmkern", "ruledhglue", "ruledvglue", "normalhglue", "normalvglue", "ruledpenalty", "filledhboxb", "filledhboxr", "filledhboxg", "filledhboxc", "filledhboxm", "filledhboxy", "filledhboxk", "scratchcounter", "globalscratchcounter", "scratchdimen", "globalscratchdimen", "scratchskip", "globalscratchskip", "scratchmuskip", "globalscratchmuskip", "scratchtoks", "globalscratchtoks", "scratchbox", "globalscratchbox", "availablehsize", "localhsize", "setlocalhsize", "nextbox", "dowithnextbox", "dowithnextboxcs", "dowithnextboxcontent", "dowithnextboxcontentcs", "scratchwidth", "scratchheight", "scratchdepth", "scratchoffset", "scratchdistance", "scratchhsize", "scratchvsize", "scratchxoffset", "scratchyoffset", "scratchhoffset", "scratchvoffset", "scratchxposition", "scratchyposition", "scratchtopoffset", "scratchbottomoffset", "scratchleftoffset", "scratchrightoffset", "scratchcounterone", "scratchcountertwo", "scratchcounterthree", "scratchdimenone", "scratchdimentwo", "scratchdimenthree", "scratchskipone", "scratchskiptwo", "scratchskipthree", "scratchmuskipone", "scratchmuskiptwo", "scratchmuskipthree", "scratchtoksone", "scratchtokstwo", "scratchtoksthree", "scratchboxone", "scratchboxtwo", "scratchboxthree", "scratchnx", "scratchny", "scratchmx", "scratchmy", "scratchunicode", "scratchleftskip", "scratchrightskip", "scratchtopskip", "scratchbottomskip", "doif", "doifnot", "doifelse", "doifinset", "doifnotinset", "doifinsetelse", "doifnextcharelse", "doifnextoptionalelse", "doifnextbgroupelse", "doifnextparenthesiselse", "doiffastoptionalcheckelse", "doifundefinedelse", "doifdefinedelse", "doifundefined", "doifdefined", "doifelsevalue", "doifvalue", "doifnotvalue", "doifnothing", "doifsomething", "doifelsenothing", "doifsomethingelse", "doifvaluenothing", "doifvaluesomething", "doifelsevaluenothing", "doifdimensionelse", "doifnumberelse", "doifnumber", "doifnotnumber", "doifcommonelse", "doifcommon", "doifnotcommon", "doifinstring", "doifnotinstring", "doifinstringelse", "doifassignmentelse", "docheckassignment", "tracingall", "tracingnone", "loggingall", "removetoks", "appendtoks", "prependtoks", "appendtotoks", "prependtotoks", "to", "endgraf", "endpar", "everyendpar", "reseteverypar", "finishpar", "empty", "null", "space", "quad", "enspace", "obeyspaces", "obeylines", "obeyedspace", "obeyedline", "normalspace", "executeifdefined", "singleexpandafter", "doubleexpandafter", "tripleexpandafter", "dontleavehmode", "removelastspace", "removeunwantedspaces", "keepunwantedspaces", "wait", "writestatus", "define", "defineexpandable", "redefine", "setmeasure", "setemeasure", "setgmeasure", "setxmeasure", "definemeasure", "freezemeasure", "measure", "measured", "installcorenamespace", "getvalue", "getuvalue", "setvalue", "setevalue", "setgvalue", "setxvalue", "letvalue", "letgvalue", "resetvalue", "undefinevalue", "ignorevalue", "setuvalue", "setuevalue", "setugvalue", "setuxvalue", "globallet", "glet", "udef", "ugdef", "uedef", "uxdef", "checked", "unique", "getparameters", "geteparameters", "getgparameters", "getxparameters", "forgetparameters", "copyparameters", "getdummyparameters", "dummyparameter", "directdummyparameter", "setdummyparameter", "letdummyparameter", "usedummystyleandcolor", "usedummystyleparameter", "usedummycolorparameter", "processcommalist", "processcommacommand", "quitcommalist", "quitprevcommalist", "processaction", "processallactions", "processfirstactioninset", "processallactionsinset", "unexpanded", "expanded", "startexpanded", "stopexpanded", "protected", "protect", "unprotect", "firstofoneargument", "firstoftwoarguments", "secondoftwoarguments", "firstofthreearguments", "secondofthreearguments", "thirdofthreearguments", "firstoffourarguments", "secondoffourarguments", "thirdoffourarguments", "fourthoffourarguments", "firstoffivearguments", "secondoffivearguments", "thirdoffivearguments", "fourthoffivearguments", "fifthoffivearguments", "firstofsixarguments", "secondofsixarguments", "thirdofsixarguments", "fourthofsixarguments", "fifthofsixarguments", "sixthofsixarguments", "firstofoneunexpanded", "gobbleoneargument", "gobbletwoarguments", "gobblethreearguments", "gobblefourarguments", "gobblefivearguments", "gobblesixarguments", "gobblesevenarguments", "gobbleeightarguments", "gobbleninearguments", "gobbletenarguments", "gobbleoneoptional", "gobbletwooptionals", "gobblethreeoptionals", "gobblefouroptionals", "gobblefiveoptionals", "dorecurse", "doloop", "exitloop", "dostepwiserecurse", "recurselevel", "recursedepth", "dofastloopcs", "dowith", "newconstant", "setnewconstant", "newconditional", "settrue", "setfalse", "setconstant", "newmacro", "setnewmacro", "newfraction", "newsignal", "dosingleempty", "dodoubleempty", "dotripleempty", "doquadrupleempty", "doquintupleempty", "dosixtupleempty", "doseventupleempty", "dosingleargument", "dodoubleargument", "dotripleargument", "doquadrupleargument", "doquintupleargument", "dosixtupleargument", "doseventupleargument", "dosinglegroupempty", "dodoublegroupempty", "dotriplegroupempty", "doquadruplegroupempty", "doquintuplegroupempty", "permitspacesbetweengroups", "dontpermitspacesbetweengroups", "nopdfcompression", "maximumpdfcompression", "normalpdfcompression", "modulonumber", "dividenumber", "getfirstcharacter", "doiffirstcharelse", "startnointerference", "stopnointerference", "twodigits", "threedigits", "strut", "setstrut", "strutbox", "strutht", "strutdp", "strutwd", "struthtdp", "begstrut", "endstrut", "lineheight", "ordordspacing", "ordopspacing", "ordbinspacing", "ordrelspacing", "ordopenspacing", "ordclosespacing", "ordpunctspacing", "ordinnerspacing", "opordspacing", "opopspacing", "opbinspacing", "oprelspacing", "opopenspacing", "opclosespacing", "oppunctspacing", "opinnerspacing", "binordspacing", "binopspacing", "binbinspacing", "binrelspacing", "binopenspacing", "binclosespacing", "binpunctspacing", "bininnerspacing", "relordspacing", "relopspacing", "relbinspacing", "relrelspacing", "relopenspacing", "relclosespacing", "relpunctspacing", "relinnerspacing", "openordspacing", "openopspacing", "openbinspacing", "openrelspacing", "openopenspacing", "openclosespacing", "openpunctspacing", "openinnerspacing", "closeordspacing", "closeopspacing", "closebinspacing", "closerelspacing", "closeopenspacing", "closeclosespacing", "closepunctspacing", "closeinnerspacing", "punctordspacing", "punctopspacing", "punctbinspacing", "punctrelspacing", "punctopenspacing", "punctclosespacing", "punctpunctspacing", "punctinnerspacing", "innerordspacing", "inneropspacing", "innerbinspacing", "innerrelspacing", "inneropenspacing", "innerclosespacing", "innerpunctspacing", "innerinnerspacing", "normalreqno", "startimath", "stopimath", "normalstartimath", "normalstopimath", "startdmath", "stopdmath", "normalstartdmath", "normalstopdmath", "uncramped", "cramped", "triggermathstyle", "mathstylefont", "mathsmallstylefont", "mathstyleface", "mathsmallstyleface", "mathstylecommand", "mathpalette", "mathstylehbox", "mathstylevbox", "mathstylevcenter", "mathstylevcenteredhbox", "mathstylevcenteredvbox", "mathtext", "setmathsmalltextbox", "setmathtextbox", "triggerdisplaystyle", "triggertextstyle", "triggerscriptstyle", "triggerscriptscriptstyle", "triggeruncrampedstyle", "triggercrampedstyle", "triggersmallstyle", "triggeruncrampedsmallstyle", "triggercrampedsmallstyle", "triggerbigstyle", "triggeruncrampedbigstyle", "triggercrampedbigstyle", "luaexpr", "expdoifelse", "expdoif", "expdoifnot", "expdoifcommonelse", "expdoifinsetelse", "ctxdirectlua", "ctxlatelua", "ctxsprint", "ctxwrite", "ctxcommand", "ctxdirectcommand", "ctxlatecommand", "ctxreport", "ctxlua", "luacode", "lateluacode", "directluacode", "registerctxluafile", "ctxloadluafile", "luaversion", "luamajorversion", "luaminorversion", "ctxluacode", "luaconditional", "luaexpanded", "startluaparameterset", "stopluaparameterset", "luaparameterset", "definenamedlua", "obeylualines", "obeyluatokens", "startluacode", "stopluacode", "startlua", "stoplua", "carryoverpar", "Umathbotaccent" }, }
\ No newline at end of file diff --git a/Master/texmf-dist/context/data/scite/lexers/scite-context-lexer.lua b/Master/texmf-dist/context/data/scite/lexers/scite-context-lexer.lua index 7883177b41e..816d9583bca 100644 --- a/Master/texmf-dist/context/data/scite/lexers/scite-context-lexer.lua +++ b/Master/texmf-dist/context/data/scite/lexers/scite-context-lexer.lua @@ -10,9 +10,15 @@ local info = { -- todo: move all code here -- todo: explore adapted dll ... properties + init +-- todo: play with hotspot and other properties -- The fold and lex functions are copied and patched from original code by Mitchell (see --- lexer.lua). All errors are mine. +-- lexer.lua). All errors are mine. The ability to use lpeg is a real nice adition and a +-- brilliant move. The code is a byproduct of the (mainly Lua based) textadept (still a +-- rapidly moving target) that unfortunately misses a realtime output pane. On the other +-- hand, SciTE is somewhat crippled by the fact that we cannot pop in our own (language +-- dependent) lexer into the output pane (somehow the errorlist lexer is hard coded into +-- the editor). Hopefully that will change some day. -- -- Starting with SciTE version 3.20 there is an issue with coloring. As we still lack -- a connection with scite itself (properties as well as printing to the log pane) we @@ -26,7 +32,12 @@ local info = { -- After checking 3.24 and adapting to the new lexer tables things are okay again. So, -- this version assumes 3.24 or higher. In 3.24 we have a different token result, i.e. no -- longer a { tag, pattern } but just two return values. I didn't check other changes but --- will do that when I run into issues. +-- will do that when I run into issues. I had optimized these small tables by hashing which +-- was more efficient but this is no longer needed. +-- +-- In 3.3.1 another major change took place: some helper constants (maybe they're no +-- longer constants) and functions were moved into the lexer modules namespace but the +-- functions are assigned to the Lua module afterward so we cannot alias them beforehand. -- -- I've considered making a whole copy and patch the other functions too as we need -- an extra nesting model. However, I don't want to maintain too much. An unfortunate @@ -89,24 +100,38 @@ local concat = table.concat local global = _G local type, next, setmetatable, rawset = type, next, setmetatable, rawset -if lexer then - -- in recent c++ code the lexername and loading is hard coded -elseif _LEXERHOME then - dofile(_LEXERHOME .. '/lexer.lua') -- pre 3.03 situation -else - dofile('lexer.lua') -- whatever +-- less confusing as we also use lexer for the current lexer and local _M = lexer is just ugly + +local lexers = lexer + +-- these helpers are set afterwards so we delay their initialization ... there is no need to alias each time again + +local get_style_at, get_indent_amount, get_property, get_fold_level, FOLD_BASE, FOLD_HEADER, FOLD_BLANK, initialize + +initialize = function() + FOLD_BASE = lexers.FOLD_BASE or SC_FOLDLEVELBASE + FOLD_HEADER = lexers.FOLD_HEADER or SC_FOLDLEVELHEADERFLAG + FOLD_BLANK = lexers.FOLD_BLANK or SC_FOLDLEVELWHITEFLAG + get_style_at = lexers.get_style_at or GetStyleAt + get_indent_amount = lexers.get_indent_amount or GetIndentAmount + get_property = lexers.get_property or GetProperty + get_fold_level = lexers.get_fold_level or GetFoldLevel + -- + initialize = nil end -lexer.context = lexer.context or { } -local context = lexer.context +-- we create our own extra namespace for extensions and helpers + +lexers.context = lexers.context or { } +local context = lexers.context context.patterns = context.patterns or { } local patterns = context.patterns -lexer._CONTEXTEXTENSIONS = true +lexers._CONTEXTEXTENSIONS = true local locations = { - -- lexer.context.path, + -- lexers.context.path, "data", -- optional data directory "..", -- regular scite directory } @@ -133,8 +158,6 @@ function context.loaddefinitions(name) end end --- maybe more efficient: - function context.word_match(words,word_chars,case_insensitive) local chars = '%w_' -- maybe just "" when word_chars if word_chars then @@ -268,36 +291,9 @@ end patterns.wordtoken = R("az","AZ","\127\255") patterns.wordpattern = patterns.wordtoken^3 -- todo: if limit and #s < limit then --- -- pre 3.24: --- --- function context.checkedword(validwords,validminimum,s,i) -- ,limit --- if not validwords then -- or #s < validminimum then --- return true, { "text", i } -- { "default", i } --- else --- -- keys are lower --- local word = validwords[s] --- if word == s then --- return true, { "okay", i } -- exact match --- elseif word then --- return true, { "warning", i } -- case issue --- else --- local word = validwords[lower(s)] --- if word == s then --- return true, { "okay", i } -- exact match --- elseif word then --- return true, { "warning", i } -- case issue --- elseif upper(s) == s then --- return true, { "warning", i } -- probably a logo or acronym --- else --- return true, { "error", i } --- end --- end --- end --- end - function context.checkedword(validwords,validminimum,s,i) -- ,limit if not validwords then -- or #s < validminimum then - return true, "text", i -- { "default", i } + return true, "text", i -- true, "default", i else -- keys are lower local word = validwords[s] @@ -347,100 +343,12 @@ end -- overloaded functions -local FOLD_BASE = SC_FOLDLEVELBASE -local FOLD_HEADER = SC_FOLDLEVELHEADERFLAG -local FOLD_BLANK = SC_FOLDLEVELWHITEFLAG - -local get_style_at = GetStyleAt -local get_property = GetProperty -local get_indent_amount = GetIndentAmount - -local h_table, b_table, n_table = { }, { }, { } +local h_table, b_table, n_table = { }, { }, { } -- from the time small tables were used (optimization) setmetatable(h_table, { __index = function(t,level) local v = { level, FOLD_HEADER } t[level] = v return v end }) setmetatable(b_table, { __index = function(t,level) local v = { level, FOLD_BLANK } t[level] = v return v end }) setmetatable(n_table, { __index = function(t,level) local v = { level } t[level] = v return v end }) --- -- todo: move the local functions outside (see below) .. old variant < 3.24 --- --- local newline = P("\r\n") + S("\r\n") --- local p_yes = Cp() * Cs((1-newline)^1) * newline^-1 --- local p_nop = newline --- --- local function fold_by_parsing(text,start_pos,start_line,start_level,lexer) --- local foldsymbols = lexer._foldsymbols --- if not foldsymbols then --- return { } --- end --- local patterns = foldsymbols._patterns --- if not patterns then --- return { } --- end --- local nofpatterns = #patterns --- if nofpatterns == 0 then --- return { } --- end --- local folds = { } --- local line_num = start_line --- local prev_level = start_level --- local current_level = prev_level --- local validmatches = foldsymbols._validmatches --- if not validmatches then --- validmatches = { } --- for symbol, matches in next, foldsymbols do -- whatever = { start = 1, stop = -1 } --- if not find(symbol,"^_") then -- brrr --- for s, _ in next, matches do --- validmatches[s] = true --- end --- end --- end --- foldsymbols._validmatches = validmatches --- end --- -- of course we could instead build a nice lpeg checker .. something for --- -- a rainy day with a stack of new cd's at hand --- local function action_y(pos,line) --- for i=1,nofpatterns do --- for s, m in gmatch(line,patterns[i]) do --- if validmatches[m] then --- local symbols = foldsymbols[get_style_at(start_pos + pos + s - 1)] --- if symbols then --- local action = symbols[m] --- if action then --- if type(action) == 'number' then -- we could store this in validmatches if there was only one symbol category --- current_level = current_level + action --- else --- current_level = current_level + action(text,pos,line,s,m) --- end --- if current_level < FOLD_BASE then --- current_level = FOLD_BASE --- end --- end --- end --- end --- end --- end --- if current_level > prev_level then --- folds[line_num] = h_table[prev_level] -- { prev_level, FOLD_HEADER } --- else --- folds[line_num] = n_table[prev_level] -- { prev_level } --- end --- prev_level = current_level --- line_num = line_num + 1 --- end --- local function action_n() --- folds[line_num] = b_table[prev_level] -- { prev_level, FOLD_BLANK } --- line_num = line_num + 1 --- end --- if lexer._reset_parser then --- lexer._reset_parser() --- end --- local lpegpattern = (p_yes/action_y + p_nop/action_n)^0 -- not too efficient but indirect function calls are neither but --- lpegmatch(lpegpattern,text) -- keys are not pressed that fast ... large files are slow anyway --- return folds --- end - --- The 3.24 variant; no longer subtable optimization is needed: - local newline = P("\r\n") + S("\r\n") local p_yes = Cp() * Cs((1-newline)^1) * newline^-1 local p_nop = newline @@ -458,38 +366,6 @@ local function fold_by_parsing(text,start_pos,start_line,start_level,lexer) -- if fold_pattern then -- if no functions are found then we could have a faster one - - -- fold_pattern = Cp() * C(fold_pattern) * Carg(1) / function(s,match,pos) - -- local symbols = fold_symbols[get_style_at(start_pos + pos + s - 1)] - -- local l = symbols and symbols[match] - -- if l then - -- local t = type(l) - -- if t == 'number' then - -- current_level = current_level + l - -- elseif t == 'function' then - -- current_level = current_level + l(text, pos, line, s, match) - -- end - -- end - -- end - -- fold_pattern = (fold_pattern + P(1))^0 - -- local action_y = function(pos,line) - -- lpegmatch(fold_pattern,line,1,pos) - -- folds[line_num] = prev_level - -- if current_level > prev_level then - -- folds[line_num] = prev_level + FOLD_HEADER - -- end - -- if current_level < FOLD_BASE then - -- current_level = FOLD_BASE - -- end - -- prev_level = current_level - -- line_num = line_num + 1 - -- end - -- local action_n = function() - -- folds[line_num] = prev_level + FOLD_BLANK - -- line_num = line_num + 1 - -- end - -- pattern = (p_yes/action_y + p_nop/action_n)^0 - fold_pattern = Cp() * C(fold_pattern) / function(s,match) local symbols = fold_symbols[get_style_at(start_pos + s)] if symbols then @@ -562,99 +438,16 @@ local function fold_by_parsing(text,start_pos,start_line,start_level,lexer) prev_level = _start_level_ current_level = prev_level lpegmatch(pattern,text) --- return folds -local t = folds -folds = nil -return t -- so folds can be collected + -- make folds collectable + local t = folds + folds = nil + return t end folders[lexer] = folder end return folder(text,start_pos,start_line,start_level,lexer) end --- local function fold_by_indentation(text,start_pos,start_line,start_level) --- local folds = { } --- local current_line = start_line --- local prev_level = start_level --- for line in gmatch(text,'[\t ]*(.-)\r?\n') do --- if line ~= "" then --- local current_level = FOLD_BASE + get_indent_amount(current_line) --- if current_level > prev_level then -- next level --- local i = current_line - 1 --- while true do --- local f = folds[i] --- if f and f[2] == FOLD_BLANK then --- i = i - 1 --- else --- break --- end --- end --- local f = folds[i] --- if f then --- f[2] = FOLD_HEADER --- end -- low indent --- folds[current_line] = n_table[current_level] -- { current_level } -- high indent --- elseif current_level < prev_level then -- prev level --- local f = folds[current_line - 1] --- if f then --- f[1] = prev_level -- high indent --- end --- folds[current_line] = n_table[current_level] -- { current_level } -- low indent --- else -- same level --- folds[current_line] = n_table[prev_level] -- { prev_level } --- end --- prev_level = current_level --- else --- folds[current_line] = b_table[prev_level] -- { prev_level, FOLD_BLANK } --- end --- current_line = current_line + 1 --- end --- return folds --- end - --- local function fold_by_indentation(text,start_pos,start_line,start_level) --- local folds = { } --- local current_line = start_line --- local prev_level = start_level --- for line in gmatch(text,'[\t ]*(.-)\r?\n') do --- if line ~= '' then --- local current_level = FOLD_BASE + get_indent_amount(current_line) --- if current_level > prev_level then -- next level --- local i = current_line - 1 --- local f --- while true do --- f = folds[i] --- if not f then --- break --- elseif f[2] == FOLD_BLANK then --- i = i - 1 --- else --- f[2] = FOLD_HEADER -- low indent --- break --- end --- end --- folds[current_line] = { current_level } -- high indent --- elseif current_level < prev_level then -- prev level --- local f = folds[current_line - 1] --- if f then --- f[1] = prev_level -- high indent --- end --- folds[current_line] = { current_level } -- low indent --- else -- same level --- folds[current_line] = { prev_level } --- end --- prev_level = current_level --- else --- folds[current_line] = { prev_level, FOLD_BLANK } --- end --- current_line = current_line + 1 --- end --- for line, level in next, folds do --- folds[line] = level[1] + (level[2] or 0) --- end --- return folds --- end - local folds, current_line, prev_level local function action_y() @@ -707,18 +500,17 @@ local function fold_by_indentation(text,start_pos,start_line,start_level) for line, level in next, folds do folds[line] = level[1] + (level[2] or 0) end - -- done --- return folds -local t = folds -folds = nil -return t -- so folds can be collected + -- done, make folds collectable + local t = folds + folds = nil + return t end local function fold_by_line(text,start_pos,start_line,start_level) local folds = { } -- can also be lpeg'd for _ in gmatch(text,".-\r?\n") do - folds[start_line] = n_table[start_level] -- { start_level } + folds[start_line] = n_table[start_level] -- { start_level } -- stile tables ? needs checking start_line = start_line + 1 end return folds @@ -733,10 +525,13 @@ function context.fold(text,start_pos,start_line,start_level) -- hm, we had size if text == '' then return { } end - local lexer = global._LEXER - local fold_by_lexer = lexer._fold + if initialize then + initialize() + end + local lexer = global._LEXER + local fold_by_lexer = lexer._fold local fold_by_symbols = lexer._foldsymbols - local filesize = 0 -- we don't know that + local filesize = 0 -- we don't know that if fold_by_lexer then if filesize <= threshold_by_lexer then return fold_by_lexer(text,start_pos,start_line,start_level,lexer) @@ -759,16 +554,16 @@ end -- The following code is mostly unchanged: -local function add_rule(lexer, id, rule) +local function add_rule(lexer,id,rule) if not lexer._RULES then - lexer._RULES = {} - lexer._RULEORDER = {} + lexer._RULES = { } + lexer._RULEORDER = { } end lexer._RULES[id] = rule lexer._RULEORDER[#lexer._RULEORDER + 1] = id end -local function add_style(lexer, token_name, style) +local function add_style(lexer,token_name,style) local len = lexer._STYLES.len if len == 32 then len = len + 8 @@ -777,39 +572,40 @@ local function add_style(lexer, token_name, style) print('Too many styles defined (128 MAX)') end lexer._TOKENS[token_name] = len - lexer._STYLES[len] = style - lexer._STYLES.len = len + 1 + lexer._STYLES[len] = style + lexer._STYLES.len = len + 1 end local function join_tokens(lexer) - local patterns, order = lexer._RULES, lexer._RULEORDER + local patterns = lexer._RULES + local order = lexer._RULEORDER local token_rule = patterns[order[1]] for i=2,#order do token_rule = token_rule + patterns[order[i]] end lexer._TOKENRULE = token_rule - return lexer._TOKENRULE + return token_rule end local function add_lexer(grammar, lexer, token_rule) local token_rule = join_tokens(lexer) local lexer_name = lexer._NAME - local children = lexer._CHILDREN + local children = lexer._CHILDREN for i=1,#children do local child = children[i] if child._CHILDREN then add_lexer(grammar, child) end - local child_name = child._NAME - local rules = child._EMBEDDEDRULES[lexer_name] - local rules_token_rule = grammar['__'..child_name] or rules.token_rule - grammar[child_name] = (-rules.end_rule * rules_token_rule)^0 * rules.end_rule^-1 * V(lexer_name) - local embedded_child = '_' .. child_name + local child_name = child._NAME + local rules = child._EMBEDDEDRULES[lexer_name] + local rules_token_rule = grammar['__'..child_name] or rules.token_rule + grammar[child_name] = (-rules.end_rule * rules_token_rule)^0 * rules.end_rule^-1 * V(lexer_name) + local embedded_child = '_' .. child_name grammar[embedded_child] = rules.start_rule * (-rules.end_rule * rules_token_rule)^0 * rules.end_rule^-1 - token_rule = V(embedded_child) + token_rule + token_rule = V(embedded_child) + token_rule end grammar['__' .. lexer_name] = token_rule - grammar[lexer_name] = token_rule^0 + grammar[lexer_name] = token_rule^0 end local function build_grammar(lexer, initial_rule) @@ -829,58 +625,21 @@ local function build_grammar(lexer, initial_rule) end -- so far. We need these local functions in the next one. --- --- Before 3.24 we had tokens[..] = { category, position }, now it's a two values. local lineparsers = { } function context.lex(text,init_style) local lexer = global._LEXER local grammar = lexer._GRAMMAR + if initialize then + initialize() + end if not grammar then return { } elseif lexer._LEXBYLINE then -- we could keep token local tokens = { } local offset = 0 local noftokens = 0 - -- -- pre 3.24 - -- - -- for line in gmatch(text,'[^\r\n]*\r?\n?') do -- could be an lpeg - -- local line_tokens = lpegmatch(grammar,line) - -- if line_tokens then - -- for i=1,#line_tokens do - -- local token = line_tokens[i] - -- token[2] = token[2] + offset - -- noftokens = noftokens + 1 - -- tokens[noftokens] = token - -- end - -- end - -- offset = offset + #line - -- if noftokens > 0 and tokens[noftokens][2] ~= offset then - -- noftokens = noftokens + 1 - -- tokens[noftokens] = { 'default', offset + 1 } - -- end - -- end - - -- for line in gmatch(text,'[^\r\n]*\r?\n?') do - -- local line_tokens = lpegmatch(grammar,line) - -- if line_tokens then - -- for i=1,#line_tokens,2 do - -- noftokens = noftokens + 1 - -- tokens[noftokens] = line_tokens[i] - -- noftokens = noftokens + 1 - -- tokens[noftokens] = line_tokens[i + 1] + offset - -- end - -- end - -- offset = offset + #line - -- if noftokens > 0 and tokens[noftokens] ~= offset then - -- noftokens = noftokens + 1 - -- tokens[noftokens] = 'default' - -- noftokens = noftokens + 1 - -- tokens[noftokens] = offset + 1 - -- end - -- end - local lineparser = lineparsers[lexer] if not lineparser then -- probably a cmt is more efficient lineparser = C((1-newline)^0 * newline) / function(line) @@ -957,10 +716,10 @@ function context.token(name, patt) return patt * Cc(name) * Cp() end -lexer.fold = context.fold -lexer.lex = context.lex -lexer.token = context.token -lexer.exact_match = context.exact_match +lexers.fold = context.fold +lexers.lex = context.lex +lexers.token = context.token +lexers.exact_match = context.exact_match -- helper .. alas ... the lexer's lua instance is rather crippled .. not even -- math is part of it @@ -1053,23 +812,6 @@ function lpeg.utfchartabletopattern(list) return make(tree) end --- patterns.invisibles = --- P(utfchar(0x00A0)) -- nbsp --- + P(utfchar(0x2000)) -- enquad --- + P(utfchar(0x2001)) -- emquad --- + P(utfchar(0x2002)) -- enspace --- + P(utfchar(0x2003)) -- emspace --- + P(utfchar(0x2004)) -- threeperemspace --- + P(utfchar(0x2005)) -- fourperemspace --- + P(utfchar(0x2006)) -- sixperemspace --- + P(utfchar(0x2007)) -- figurespace --- + P(utfchar(0x2008)) -- punctuationspace --- + P(utfchar(0x2009)) -- breakablethinspace --- + P(utfchar(0x200A)) -- hairspace --- + P(utfchar(0x200B)) -- zerowidthspace --- + P(utfchar(0x202F)) -- narrownobreakspace --- + P(utfchar(0x205F)) -- math thinspace - patterns.invisibles = lpeg.utfchartabletopattern { utfchar(0x00A0), -- nbsp utfchar(0x2000), -- enquad @@ -1098,3 +840,5 @@ patterns.iwordpattern = patterns.iwordtoken^3 -- In order to deal with some bug in additional styles (I have no cue what is -- wrong, but additional styles get ignored and clash somehow) I just copy the -- original lexer code ... see original for comments. + +return lexers diff --git a/Master/texmf-dist/context/data/scite/scite-context-data-context.properties b/Master/texmf-dist/context/data/scite/scite-context-data-context.properties index db9324666a4..4a08810ab76 100644 --- a/Master/texmf-dist/context/data/scite/scite-context-data-context.properties +++ b/Master/texmf-dist/context/data/scite/scite-context-data-context.properties @@ -71,72 +71,73 @@ setsystemmode resetsystemmode pushsystemmode popsystemmode booleanmodevalue \ newcount newdimen newskip newmuskip newbox \ newtoks newread newwrite newmarks newinsert \ newattribute newif newlanguage newfamily newfam \ -newhelp then firstargumentfalse firstargumenttrue secondargumentfalse \ -secondargumenttrue thirdargumentfalse thirdargumenttrue fourthargumentfalse fourthargumenttrue \ -fifthargumentfalse fifthsargumenttrue sixthargumentfalse sixtsargumenttrue doglobal \ -dodoglobal redoglobal resetglobal donothing dontcomplain \ -forgetall donetrue donefalse htdp unvoidbox \ -hfilll vfilll mathbox mathlimop mathnolop \ -mathnothing mathalpha currentcatcodetable defaultcatcodetable catcodetablename \ -newcatcodetable startcatcodetable stopcatcodetable startextendcatcodetable stopextendcatcodetable \ -pushcatcodetable popcatcodetable restorecatcodes setcatcodetable letcatcodecommand \ -defcatcodecommand uedcatcodecommand hglue vglue hfillneg \ -vfillneg hfilllneg vfilllneg ruledhss ruledhfil \ -ruledhfill ruledhfilneg ruledhfillneg normalhfillneg ruledvss \ -ruledvfil ruledvfill ruledvfilneg ruledvfillneg normalvfillneg \ -ruledhbox ruledvbox ruledvtop ruledvcenter ruledhskip \ -ruledvskip ruledkern ruledmskip ruledmkern ruledhglue \ -ruledvglue normalhglue normalvglue ruledpenalty filledhboxb \ -filledhboxr filledhboxg filledhboxc filledhboxm filledhboxy \ -filledhboxk scratchcounter globalscratchcounter scratchdimen globalscratchdimen \ -scratchskip globalscratchskip scratchmuskip globalscratchmuskip scratchtoks \ -globalscratchtoks scratchbox globalscratchbox availablehsize localhsize \ -setlocalhsize nextbox dowithnextbox dowithnextboxcs dowithnextboxcontent \ -dowithnextboxcontentcs scratchwidth scratchheight scratchdepth scratchoffset \ -scratchdistance scratchhsize scratchvsize scratchxoffset scratchyoffset \ -scratchhoffset scratchvoffset scratchxposition scratchyposition scratchtopoffset \ -scratchbottomoffset scratchleftoffset scratchrightoffset scratchcounterone scratchcountertwo \ -scratchcounterthree scratchdimenone scratchdimentwo scratchdimenthree scratchskipone \ -scratchskiptwo scratchskipthree scratchmuskipone scratchmuskiptwo scratchmuskipthree \ -scratchtoksone scratchtokstwo scratchtoksthree scratchboxone scratchboxtwo \ -scratchboxthree scratchnx scratchny scratchmx scratchmy \ -scratchunicode scratchleftskip scratchrightskip scratchtopskip scratchbottomskip \ -doif doifnot doifelse doifinset doifnotinset \ -doifinsetelse doifnextcharelse doifnextoptionalelse doifnextbgroupelse doifnextparenthesiselse \ -doiffastoptionalcheckelse doifundefinedelse doifdefinedelse doifundefined doifdefined \ -doifelsevalue doifvalue doifnotvalue doifnothing doifsomething \ -doifelsenothing doifsomethingelse doifvaluenothing doifvaluesomething doifelsevaluenothing \ -doifdimensionelse doifnumberelse doifnumber doifnotnumber doifcommonelse \ -doifcommon doifnotcommon doifinstring doifnotinstring doifinstringelse \ -doifassignmentelse docheckassignment tracingall tracingnone loggingall \ -removetoks appendtoks prependtoks appendtotoks prependtotoks \ -to endgraf endpar everyendpar reseteverypar \ -finishpar empty null space quad \ -enspace obeyspaces obeylines obeyedspace obeyedline \ -normalspace executeifdefined singleexpandafter doubleexpandafter tripleexpandafter \ -dontleavehmode removelastspace removeunwantedspaces keepunwantedspaces wait \ -writestatus define defineexpandable redefine setmeasure \ -setemeasure setgmeasure setxmeasure definemeasure freezemeasure \ -measure installcorenamespace getvalue setvalue setevalue \ -setgvalue setxvalue letvalue letgvalue resetvalue \ -undefinevalue ignorevalue setuvalue setuevalue setugvalue \ -setuxvalue globallet glet udef ugdef \ -uedef uxdef checked unique getparameters \ -geteparameters getgparameters getxparameters forgetparameters copyparameters \ -getdummyparameters dummyparameter directdummyparameter setdummyparameter letdummyparameter \ -usedummystyleandcolor usedummystyleparameter usedummycolorparameter processcommalist processcommacommand \ -quitcommalist quitprevcommalist processaction processallactions processfirstactioninset \ -processallactionsinset unexpanded expanded startexpanded stopexpanded \ -protected protect unprotect firstofoneargument firstoftwoarguments \ -secondoftwoarguments firstofthreearguments secondofthreearguments thirdofthreearguments firstoffourarguments \ -secondoffourarguments thirdoffourarguments fourthoffourarguments firstoffivearguments secondoffivearguments \ -thirdoffivearguments fourthoffivearguments fifthoffivearguments firstofsixarguments secondofsixarguments \ -thirdofsixarguments fourthofsixarguments fifthofsixarguments sixthofsixarguments firstofoneunexpanded \ -gobbleoneargument gobbletwoarguments gobblethreearguments gobblefourarguments gobblefivearguments \ -gobblesixarguments gobblesevenarguments gobbleeightarguments gobbleninearguments gobbletenarguments \ -gobbleoneoptional gobbletwooptionals gobblethreeoptionals gobblefouroptionals gobblefiveoptionals \ -dorecurse doloop exitloop dostepwiserecurse recurselevel \ -recursedepth dofastloopcs newconstant setnewconstant newconditional \ +newhelp then begcsname strippedcsname firstargumentfalse \ +firstargumenttrue secondargumentfalse secondargumenttrue thirdargumentfalse thirdargumenttrue \ +fourthargumentfalse fourthargumenttrue fifthargumentfalse fifthsargumenttrue sixthargumentfalse \ +sixtsargumenttrue doglobal dodoglobal redoglobal resetglobal \ +donothing dontcomplain forgetall donetrue donefalse \ +htdp unvoidbox hfilll vfilll mathbox \ +mathlimop mathnolop mathnothing mathalpha currentcatcodetable \ +defaultcatcodetable catcodetablename newcatcodetable startcatcodetable stopcatcodetable \ +startextendcatcodetable stopextendcatcodetable pushcatcodetable popcatcodetable restorecatcodes \ +setcatcodetable letcatcodecommand defcatcodecommand uedcatcodecommand hglue \ +vglue hfillneg vfillneg hfilllneg vfilllneg \ +ruledhss ruledhfil ruledhfill ruledhfilneg ruledhfillneg \ +normalhfillneg ruledvss ruledvfil ruledvfill ruledvfilneg \ +ruledvfillneg normalvfillneg ruledhbox ruledvbox ruledvtop \ +ruledvcenter ruledhskip ruledvskip ruledkern ruledmskip \ +ruledmkern ruledhglue ruledvglue normalhglue normalvglue \ +ruledpenalty filledhboxb filledhboxr filledhboxg filledhboxc \ +filledhboxm filledhboxy filledhboxk scratchcounter globalscratchcounter \ +scratchdimen globalscratchdimen scratchskip globalscratchskip scratchmuskip \ +globalscratchmuskip scratchtoks globalscratchtoks scratchbox globalscratchbox \ +availablehsize localhsize setlocalhsize nextbox dowithnextbox \ +dowithnextboxcs dowithnextboxcontent dowithnextboxcontentcs scratchwidth scratchheight \ +scratchdepth scratchoffset scratchdistance scratchhsize scratchvsize \ +scratchxoffset scratchyoffset scratchhoffset scratchvoffset scratchxposition \ +scratchyposition scratchtopoffset scratchbottomoffset scratchleftoffset scratchrightoffset \ +scratchcounterone scratchcountertwo scratchcounterthree scratchdimenone scratchdimentwo \ +scratchdimenthree scratchskipone scratchskiptwo scratchskipthree scratchmuskipone \ +scratchmuskiptwo scratchmuskipthree scratchtoksone scratchtokstwo scratchtoksthree \ +scratchboxone scratchboxtwo scratchboxthree scratchnx scratchny \ +scratchmx scratchmy scratchunicode scratchleftskip scratchrightskip \ +scratchtopskip scratchbottomskip doif doifnot doifelse \ +doifinset doifnotinset doifinsetelse doifnextcharelse doifnextoptionalelse \ +doifnextbgroupelse doifnextparenthesiselse doiffastoptionalcheckelse doifundefinedelse doifdefinedelse \ +doifundefined doifdefined doifelsevalue doifvalue doifnotvalue \ +doifnothing doifsomething doifelsenothing doifsomethingelse doifvaluenothing \ +doifvaluesomething doifelsevaluenothing doifdimensionelse doifnumberelse doifnumber \ +doifnotnumber doifcommonelse doifcommon doifnotcommon doifinstring \ +doifnotinstring doifinstringelse doifassignmentelse docheckassignment tracingall \ +tracingnone loggingall removetoks appendtoks prependtoks \ +appendtotoks prependtotoks to endgraf endpar \ +everyendpar reseteverypar finishpar empty null \ +space quad enspace obeyspaces obeylines \ +obeyedspace obeyedline normalspace executeifdefined singleexpandafter \ +doubleexpandafter tripleexpandafter dontleavehmode removelastspace removeunwantedspaces \ +keepunwantedspaces wait writestatus define defineexpandable \ +redefine setmeasure setemeasure setgmeasure setxmeasure \ +definemeasure freezemeasure measure measured installcorenamespace \ +getvalue getuvalue setvalue setevalue setgvalue \ +setxvalue letvalue letgvalue resetvalue undefinevalue \ +ignorevalue setuvalue setuevalue setugvalue setuxvalue \ +globallet glet udef ugdef uedef \ +uxdef checked unique getparameters geteparameters \ +getgparameters getxparameters forgetparameters copyparameters getdummyparameters \ +dummyparameter directdummyparameter setdummyparameter letdummyparameter usedummystyleandcolor \ +usedummystyleparameter usedummycolorparameter processcommalist processcommacommand quitcommalist \ +quitprevcommalist processaction processallactions processfirstactioninset processallactionsinset \ +unexpanded expanded startexpanded stopexpanded protected \ +protect unprotect firstofoneargument firstoftwoarguments secondoftwoarguments \ +firstofthreearguments secondofthreearguments thirdofthreearguments firstoffourarguments secondoffourarguments \ +thirdoffourarguments fourthoffourarguments firstoffivearguments secondoffivearguments thirdoffivearguments \ +fourthoffivearguments fifthoffivearguments firstofsixarguments secondofsixarguments thirdofsixarguments \ +fourthofsixarguments fifthofsixarguments sixthofsixarguments firstofoneunexpanded gobbleoneargument \ +gobbletwoarguments gobblethreearguments gobblefourarguments gobblefivearguments gobblesixarguments \ +gobblesevenarguments gobbleeightarguments gobbleninearguments gobbletenarguments gobbleoneoptional \ +gobbletwooptionals gobblethreeoptionals gobblefouroptionals gobblefiveoptionals dorecurse \ +doloop exitloop dostepwiserecurse recurselevel recursedepth \ +dofastloopcs dowith newconstant setnewconstant newconditional \ settrue setfalse setconstant newmacro setnewmacro \ newfraction newsignal dosingleempty dodoubleempty dotripleempty \ doquadrupleempty doquintupleempty dosixtupleempty doseventupleempty dosingleargument \ diff --git a/Master/texmf-dist/context/data/scite/scite-context.properties b/Master/texmf-dist/context/data/scite/scite-context.properties index 571396b58a6..ce56084c07d 100644 --- a/Master/texmf-dist/context/data/scite/scite-context.properties +++ b/Master/texmf-dist/context/data/scite/scite-context.properties @@ -27,13 +27,14 @@ # if PLAT_WIN # find.command=fgrep -G -n $(find.what) $(find.files) -# bugged: bad cursor -# -# technology=0 -# +# 1: better anti-aliasing on windows, also slightly diffent lineheights + +technology=1 + # not much difference # # buffered.draw=0 +# two.phase.draw=0 # # no auto save: # @@ -44,13 +45,13 @@ code.page=65001 output.code.page=65001 -# caret gets weird: -# -# technology=1 +position.maximize=1 +virtual.space=1 textwrapper.margin=4 textwrapper.length=68 -#~ textwrapper.length=80 + +# xml.auto.close.tags=1 # ConTeXt: suffixes (really needed) @@ -99,6 +100,7 @@ if PLAT_WIN name.context.gsview=gsview32.exe name.context.acrobat= name.flag.pdfopen=--autopdf + name.flag.pdfopen=--autopdf=sumatra # consoles: rxvt,konsole | acrobat: acroread,xpdfopen # |