diff options
Diffstat (limited to 'Master/texmf-dist/tex/luatex/luaotfload/fontloader-font-ots.lua')
-rw-r--r-- | Master/texmf-dist/tex/luatex/luaotfload/fontloader-font-ots.lua | 4351 |
1 files changed, 2475 insertions, 1876 deletions
diff --git a/Master/texmf-dist/tex/luatex/luaotfload/fontloader-font-ots.lua b/Master/texmf-dist/tex/luatex/luaotfload/fontloader-font-ots.lua index 1f84214812f..e3389b9daf7 100644 --- a/Master/texmf-dist/tex/luatex/luaotfload/fontloader-font-ots.lua +++ b/Master/texmf-dist/tex/luatex/luaotfload/fontloader-font-ots.lua @@ -6,73 +6,45 @@ if not modules then modules = { } end modules ['font-ots'] = { -- sequences license = "see context related readme files", } --- to be checked: discrun doesn't seem to do something useful now (except run the --- check again) so if we need it again we'll do a zwnjrun or so - --- components will go away and be replaced by a property table which simplifies --- code (also more efficient) - --- beware, on my development machine we test a slightly a more optimized version - --- assumptions: --- --- cursives don't cross discretionaries --- marks precede bases --- --- pitfalls: --- --- when we append to a dics field we need to set the field in order to update tail --- --- This is a version of font-otn.lua adapted to the new font loader code. It --- is a context version which can contain experimental code, but when we --- have serious patches we will backport to the font-otn files. The plain --- loader that ships with context also uses this now. --- --- todo: looks like we have a leak somewhere (probably in ligatures) --- todo: copy attributes to disc --- todo: get rid of components, better use the tounicode entry if needed (at all) --- --- we do some disc juggling where we need to keep in mind that the --- pre, post and replace fields can have prev pointers to a nesting --- node ... i wonder if that is still needed --- --- not possible: --- --- \discretionary {alpha-} {betagammadelta} --- {\discretionary {alphabeta-} {gammadelta} --- {\discretionary {alphabetagamma-} {delta} --- {alphabetagammadelta}}} - --[[ldx-- <p>This module is a bit more split up that I'd like but since we also want to test with plain <l n='tex'/> it has to be so. This module is part of <l n='context'/> and discussion about improvements and functionality mostly happens on the <l n='context'/> mailing list.</p> -<p>The specification of OpenType is kind of vague. Apart from a lack of a proper -free specifications there's also the problem that Microsoft and Adobe -may have their own interpretation of how and in what order to apply features. -In general the Microsoft website has more detailed specifications and is a -better reference. There is also some information in the FontForge help files.</p> +<p>The specification of OpenType is (or at least a decade ago was) kind of vague. +Apart from a lack of a proper free specifications there's also the problem that +Microsoft and Adobe may have their own interpretation of how and in what order to +apply features. In general the Microsoft website has more detailed specifications +and is a better reference. There is also some information in the FontForge help +files. In the end we rely most on the Microsoft specification.</p> <p>Because there is so much possible, fonts might contain bugs and/or be made to work with certain rederers. These may evolve over time which may have the side -effect that suddenly fonts behave differently.</p> +effect that suddenly fonts behave differently. We don't want to catch all font +issues.</p> -<p>After a lot of experiments (mostly by Taco, me and Idris) we're now at yet another -implementation. Of course all errors are mine and of course the code can be -improved. There are quite some optimizations going on here and processing speed -is currently acceptable. Not all functions are implemented yet, often because I -lack the fonts for testing. Many scripts are not yet supported either, but I will -look into them as soon as <l n='context'/> users ask for it.</p> +<p>After a lot of experiments (mostly by Taco, me and Idris) the first implementation +becaus quite useful. When it did most of what we wanted, a more optimized version +evolved. Of course all errors are mine and of course the code can be improved. There +are quite some optimizations going on here and processing speed is currently quite +acceptable and has been improved over time. Many complex scripts are not yet supported +yet, but I will look into them as soon as <l n='context'/> users ask for it.</p> -<p>The specification leaves room for interpretation. In case of doubt the microsoft +<p>The specification leaves room for interpretation. In case of doubt the Microsoft implementation is the reference as it is the most complete one. As they deal with lots of scripts and fonts, Kai and Ivo did a lot of testing of the generic code and their suggestions help improve the code. I'm aware that not all border cases can be taken care of, unless we accept excessive runtime, and even then the interference with other mechanisms (like hyphenation) are not trivial.</p> +<p>Especially discretionary handling has been improved much by Kai Eigner who uses complex +(latin) fonts. The current implementation is a compromis between his patches and my code +and in the meantime performance is quite ok. We cannot check all border cases without +compromising speed but so far we're okay. Given good test cases we can probably improve +it here and there. Especially chain lookups are non trivial with discretionaries but +things got much better over time thanks to Kai.</p> + <p>Glyphs are indexed not by unicode but in their own way. This is because there is no relationship with unicode at all, apart from the fact that a font might cover certain ranges of characters. One character can have multiple shapes. However, at the @@ -84,81 +56,128 @@ then in the output eventually.</p> that different from the one produced by <l n='fontforge'/> but we uses hashes instead. In <l n='context'/> that table is packed (similar tables are shared) and cached on disk so that successive runs can use the optimized table (after loading the table is -unpacked). The flattening code used later is a prelude to an even more compact table -format (and as such it keeps evolving).</p> +unpacked).</p> -<p>This module is sparsely documented because it is a moving target. The table format -of the reader changes and we experiment a lot with different methods for supporting -features.</p> - -<p>As with the <l n='afm'/> code, we may decide to store more information in the -<l n='otf'/> table.</p> +<p>This module is sparsely documented because it is has been a moving target. The +table format of the reader changed a bit over time and we experiment a lot with +different methods for supporting features. By now the structures are quite stable</p> <p>Incrementing the version number will force a re-cache. We jump the number by one -when there's a fix in the <l n='fontforge'/> library or <l n='lua'/> code that -results in different tables.</p> +when there's a fix in the reader or processing code that can result in different +results.</p> + +<p>This code is also used outside context but in context it has to work with other +mechanisms. Both put some constraints on the code here.</p> + --ldx]]-- +-- Remark: We assume that cursives don't cross discretionaries which is okay because it +-- is only used in semitic scripts. +-- +-- Remark: We assume that marks precede base characters. +-- +-- Remark: When complex ligatures extend into discs nodes we can get side effects. Normally +-- this doesn't happen; ff\d{l}{l}{l} in lm works but ff\d{f}{f}{f}. +-- +-- Todo: check if we copy attributes to disc nodes if needed. +-- +-- Todo: it would be nice if we could get rid of components. In other places we can use +-- the unicode properties. +-- +-- Remark: We do some disc juggling where we need to keep in mind that the pre, post and +-- replace fields can have prev pointers to a nesting node ... I wonder if that is still +-- needed. +-- +-- Remark: This is not possible: +-- +-- \discretionary {alpha-} {betagammadelta} +-- {\discretionary {alphabeta-} {gammadelta} +-- {\discretionary {alphabetagamma-} {delta} +-- {alphabetagammadelta}}} +-- +-- Remark: Something is messed up: we have two mark / ligature indices, one at the +-- injection end and one here ... this is based on KE's patches but there is something +-- fishy there as I'm pretty sure that for husayni we need some connection (as it's much +-- more complex than an average font) but I need proper examples of all cases, not of +-- only some. +-- +-- Remark: I wonder if indexed would be faster than unicoded. It would be a major +-- rewrite to have char being unicode + an index field in glyph nodes. Also more +-- assignments have to be made in order to keep things in sync. So, it's a no-go. +-- +-- Remark: We can provide a fast loop when there are no disc nodes (tests show a 1% +-- gain). Smaller functions might perform better cache-wise. But ... memory becomes +-- faster anyway, so ... +-- +-- Remark: Some optimizations made sense for 5.2 but seem less important for 5.3 but +-- anyway served their purpose. +-- +-- Todo: just (0=l2r and 1=r2l) or maybe (r2l = true) + local type, next, tonumber = type, next, tonumber local random = math.random local formatters = string.formatters local insert = table.insert -local logs, trackers, nodes, attributes = logs, trackers, nodes, attributes - -local registertracker = trackers.register -local registerdirective = directives.register - -local fonts = fonts -local otf = fonts.handlers.otf - -local trace_lookups = false registertracker("otf.lookups", function(v) trace_lookups = v end) -local trace_singles = false registertracker("otf.singles", function(v) trace_singles = v end) -local trace_multiples = false registertracker("otf.multiples", function(v) trace_multiples = v end) -local trace_alternatives = false registertracker("otf.alternatives", function(v) trace_alternatives = v end) -local trace_ligatures = false registertracker("otf.ligatures", function(v) trace_ligatures = v end) -local trace_contexts = false registertracker("otf.contexts", function(v) trace_contexts = v end) -local trace_marks = false registertracker("otf.marks", function(v) trace_marks = v end) -local trace_kerns = false registertracker("otf.kerns", function(v) trace_kerns = v end) -local trace_cursive = false registertracker("otf.cursive", function(v) trace_cursive = v end) -local trace_preparing = false registertracker("otf.preparing", function(v) trace_preparing = v end) -local trace_bugs = false registertracker("otf.bugs", function(v) trace_bugs = v end) -local trace_details = false registertracker("otf.details", function(v) trace_details = v end) ------ trace_applied = false registertracker("otf.applied", function(v) trace_applied = v end) -local trace_steps = false registertracker("otf.steps", function(v) trace_steps = v end) -local trace_skips = false registertracker("otf.skips", function(v) trace_skips = v end) -local trace_directions = false registertracker("otf.directions", function(v) trace_directions = v end) -local trace_plugins = false registertracker("otf.plugins", function(v) trace_plugins = v end) - -local trace_kernruns = false registertracker("otf.kernruns", function(v) trace_kernruns = v end) -local trace_discruns = false registertracker("otf.discruns", function(v) trace_discruns = v end) -local trace_compruns = false registertracker("otf.compruns", function(v) trace_compruns = v end) -local trace_testruns = false registertracker("otf.testruns", function(v) trace_testruns = v end) - ------ zwnjruns = true registerdirective("otf.zwnjruns", function(v) zwnjruns = v end) -local optimizekerns = true -local alwaysdisc = true registerdirective("otf.alwaysdisc", function(v) alwaysdisc = v end) - -local report_direct = logs.reporter("fonts","otf direct") -local report_subchain = logs.reporter("fonts","otf subchain") -local report_chain = logs.reporter("fonts","otf chain") -local report_process = logs.reporter("fonts","otf process") -local report_warning = logs.reporter("fonts","otf warning") -local report_run = logs.reporter("fonts","otf run") - -registertracker("otf.replacements", "otf.singles,otf.multiples,otf.alternatives,otf.ligatures") -registertracker("otf.positions","otf.marks,otf.kerns,otf.cursive") -registertracker("otf.actions","otf.replacements,otf.positions") -registertracker("otf.injections","nodes.injections") - -registertracker("*otf.sample","otf.steps,otf.actions,otf.analyzing") +local registertracker = trackers.register + +local logs = logs +local trackers = trackers +local nodes = nodes +local attributes = attributes +local fonts = fonts + +local otf = fonts.handlers.otf +local tracers = nodes.tracers + +local trace_singles = false registertracker("otf.singles", function(v) trace_singles = v end) +local trace_multiples = false registertracker("otf.multiples", function(v) trace_multiples = v end) +local trace_alternatives = false registertracker("otf.alternatives", function(v) trace_alternatives = v end) +local trace_ligatures = false registertracker("otf.ligatures", function(v) trace_ligatures = v end) +local trace_contexts = false registertracker("otf.contexts", function(v) trace_contexts = v end) +local trace_marks = false registertracker("otf.marks", function(v) trace_marks = v end) +local trace_kerns = false registertracker("otf.kerns", function(v) trace_kerns = v end) +local trace_cursive = false registertracker("otf.cursive", function(v) trace_cursive = v end) +local trace_preparing = false registertracker("otf.preparing", function(v) trace_preparing = v end) +local trace_bugs = false registertracker("otf.bugs", function(v) trace_bugs = v end) +local trace_details = false registertracker("otf.details", function(v) trace_details = v end) +local trace_steps = false registertracker("otf.steps", function(v) trace_steps = v end) +local trace_skips = false registertracker("otf.skips", function(v) trace_skips = v end) +local trace_plugins = false registertracker("otf.plugins", function(v) trace_plugins = v end) +local trace_chains = false registertracker("otf.chains", function(v) trace_chains = v end) + +local trace_kernruns = false registertracker("otf.kernruns", function(v) trace_kernruns = v end) +----- trace_discruns = false registertracker("otf.discruns", function(v) trace_discruns = v end) +local trace_compruns = false registertracker("otf.compruns", function(v) trace_compruns = v end) +local trace_testruns = false registertracker("otf.testruns", function(v) trace_testruns = v end) + +local forcediscretionaries = false +local forcepairadvance = false -- for testing + +directives.register("otf.forcediscretionaries",function(v) + forcediscretionaries = v +end) + +directives.register("otf.forcepairadvance",function(v) + forcepairadvance = v +end) + +local report_direct = logs.reporter("fonts","otf direct") +local report_subchain = logs.reporter("fonts","otf subchain") +local report_chain = logs.reporter("fonts","otf chain") +local report_process = logs.reporter("fonts","otf process") +local report_warning = logs.reporter("fonts","otf warning") +local report_run = logs.reporter("fonts","otf run") + +registertracker("otf.substitutions", "otf.singles","otf.multiples","otf.alternatives","otf.ligatures") +registertracker("otf.positions", "otf.marks","otf.kerns","otf.cursive") +registertracker("otf.actions", "otf.substitutions","otf.positions") +registertracker("otf.sample", "otf.steps","otf.substitutions","otf.positions","otf.analyzing") +registertracker("otf.sample.silent", "otf.steps=silent","otf.substitutions","otf.positions","otf.analyzing") local nuts = nodes.nuts -local tonode = nuts.tonode -local tonut = nuts.tonut local getfield = nuts.getfield -local setfield = nuts.setfield local getnext = nuts.getnext local setnext = nuts.setnext local getprev = nuts.getprev @@ -178,26 +197,37 @@ local setchar = nuts.setchar local getdisc = nuts.getdisc local setdisc = nuts.setdisc local setlink = nuts.setlink +local getcomponents = nuts.getcomponents -- the original one, not yet node-aux +local setcomponents = nuts.setcomponents -- the original one, not yet node-aux +local getdir = nuts.getdir +local getwidth = nuts.getwidth local ischar = nuts.is_char +local usesfont = nuts.uses_font local insert_node_after = nuts.insert_after local copy_node = nuts.copy local copy_node_list = nuts.copy_list +local remove_node = nuts.remove local find_node_tail = nuts.tail local flush_node_list = nuts.flush_list local flush_node = nuts.flush_node local end_of_math = nuts.end_of_math -local traverse_nodes = nuts.traverse -local traverse_id = nuts.traverse_id -local remove_node = nuts.remove - +----- traverse_nodes = nuts.traverse +----- traverse_id = nuts.traverse_id +local set_components = nuts.set_components +local take_components = nuts.take_components +local count_components = nuts.count_components +local copy_no_components = nuts.copy_no_components +local copy_only_glyphs = nuts.copy_only_glyphs + +local setmetatable = setmetatable local setmetatableindex = table.setmetatableindex -local zwnj = 0x200C -local zwj = 0x200D -local wildcard = "*" -local default = "dflt" +local nextnode = nuts.traversers.node + +----- zwnj = 0x200C +----- zwj = 0x200D local nodecodes = nodes.nodecodes local glyphcodes = nodes.glyphcodes @@ -213,31 +243,22 @@ local localpar_code = nodecodes.localpar local discretionary_code = disccodes.discretionary local ligature_code = glyphcodes.ligature -local privateattribute = attributes.private - --- Something is messed up: we have two mark / ligature indices, one at the injection --- end and one here ... this is based on KE's patches but there is something fishy --- there as I'm pretty sure that for husayni we need some connection (as it's much --- more complex than an average font) but I need proper examples of all cases, not --- of only some. - -local a_state = privateattribute('state') +local a_state = attributes.private('state') +local a_noligature = attributes.private("noligature") local injections = nodes.injections local setmark = injections.setmark local setcursive = injections.setcursive local setkern = injections.setkern -local setpair = injections.setpair +local setmove = injections.setmove +local setposition = injections.setposition local resetinjection = injections.reset local copyinjection = injections.copy local setligaindex = injections.setligaindex local getligaindex = injections.getligaindex -local cursonce = true - -local fonthashes = fonts.hashes -local fontdata = fonthashes.identifiers -local fontfeatures = fonthashes.features +local fontdata = fonts.hashes.identifiers +local fontfeatures = fonts.hashes.features local otffeatures = fonts.constructors.features.otf local registerotffeature = otffeatures.register @@ -248,48 +269,51 @@ local getrandom = utilities and utilities.randomizer and utilities.rand otf.defaultnodealternate = "none" -- first last --- We use a few global variables. The handler can be called nested but this assumes that the --- same font is used. Nested calls are normally not needed (only for devanagari). +-- We use a few semi-global variables. The handler can be called nested but this assumes +-- that the same font is used. -local tfmdata = false -local characters = false -local descriptions = false -local marks = false -local currentfont = false -local factor = 0 -local threshold = 0 -local checkmarks = false +local tfmdata = false +local characters = false +local descriptions = false +local marks = false +local classes = false +local currentfont = false +local factor = 0 +local threshold = 0 +local checkmarks = false -local sweepnode = nil -local sweepprev = nil -local sweepnext = nil -local sweephead = { } +local discs = false +local spaces = false -local notmatchpre = { } -local notmatchpost = { } -local notmatchreplace = { } +local sweepnode = nil +local sweephead = { } -- we don't nil entries but false them (no collection and such) -local handlers = { } +local notmatchpre = { } -- to be checked: can we use false instead of nil / what if a == b tests +local notmatchpost = { } -- to be checked: can we use false instead of nil / what if a == b tests +local notmatchreplace = { } -- to be checked: can we use false instead of nil / what if a == b tests -local isspace = injections.isspace -local getthreshold = injections.getthreshold +local handlers = { } --- we use this for special testing and documentation +local isspace = injections.isspace +local getthreshold = injections.getthreshold -local checkstep = (nodes and nodes.tracers and nodes.tracers.steppers.check) or function() end -local registerstep = (nodes and nodes.tracers and nodes.tracers.steppers.register) or function() end -local registermessage = (nodes and nodes.tracers and nodes.tracers.steppers.message) or function() end +local checkstep = (tracers and tracers.steppers.check) or function() end +local registerstep = (tracers and tracers.steppers.register) or function() end +local registermessage = (tracers and tracers.steppers.message) or function() end -local function checkdisccontent(d) - local pre, post, replace = getdisc(d) - if pre then for n in traverse_id(glue_code,pre) do print("pre",nodes.idstostring(pre)) break end end - if post then for n in traverse_id(glue_code,post) do print("pos",nodes.idstostring(post)) break end end - if replace then for n in traverse_id(glue_code,replace) do print("rep",nodes.idstostring(replace)) break end end -end +-- local function checkdisccontent(d) +-- local pre, post, replace = getdisc(d) +-- if pre then for n in traverse_id(glue_code,pre) do report("pre: %s",nodes.idstostring(pre)) break end end +-- if post then for n in traverse_id(glue_code,post) do report("pos: %s",nodes.idstostring(post)) break end end +-- if replace then for n in traverse_id(glue_code,replace) do report("rep: %s",nodes.idstostring(replace)) break end end +-- end local function logprocess(...) if trace_steps then registermessage(...) + if trace_steps == "silent" then + return + end end report_direct(...) end @@ -298,59 +322,77 @@ local function logwarning(...) report_direct(...) end -local f_unicode = formatters["%U"] -local f_uniname = formatters["%U (%s)"] -local f_unilist = formatters["% t (% t)"] +local gref do -local function gref(n) -- currently the same as in font-otb - if type(n) == "number" then - local description = descriptions[n] - local name = description and description.name - if name then - return f_uniname(n,name) - else - return f_unicode(n) - end - elseif n then - local num, nam = { }, { } - for i=1,#n do - local ni = n[i] - if tonumber(ni) then -- later we will start at 2 - local di = descriptions[ni] - num[i] = f_unicode(ni) - nam[i] = di and di.name or "-" + local f_unicode = formatters["U+%X"] -- was ["%U"] + local f_uniname = formatters["U+%X (%s)"] -- was ["%U (%s)"] + local f_unilist = formatters["% t"] + + gref = function(n) -- currently the same as in font-otb + if type(n) == "number" then + local description = descriptions[n] + local name = description and description.name + if name then + return f_uniname(n,name) + else + return f_unicode(n) + end + elseif n then + local t = { } + for i=1,#n do + local ni = n[i] + if tonumber(ni) then -- later we will start at 2 + local di = descriptions[ni] + local nn = di and di.name + if nn then + t[#t+1] = f_uniname(ni,nn) + else + t[#t+1] = f_unicode(ni) + end + end end + return f_unilist(t) + else + return "<error in node mode tracing>" end - return f_unilist(num,nam) - else - return "<error in node mode tracing>" end + end local function cref(dataset,sequence,index) if not dataset then return "no valid dataset" - elseif index then - return formatters["feature %a, type %a, chain lookup %a, index %a"](dataset[4],sequence.type,sequence.name,index) + end + local merged = sequence.merged and "merged " or "" + if index then + return formatters["feature %a, type %a, %schain lookup %a, index %a"]( + dataset[4],sequence.type,merged,sequence.name,index) else - return formatters["feature %a, type %a, chain lookup %a"](dataset[4],sequence.type,sequence.name) + return formatters["feature %a, type %a, %schain lookup %a"]( + dataset[4],sequence.type,merged,sequence.name) end end local function pref(dataset,sequence) - return formatters["feature %a, type %a, lookup %a"](dataset[4],sequence.type,sequence.name) + return formatters["feature %a, type %a, %slookup %a"]( + dataset[4],sequence.type,sequence.merged and "merged " or "",sequence.name) end local function mref(rlmode) - if not rlmode or rlmode == 0 then - return "---" - elseif rlmode == -1 or rlmode == "+TRT" then - return "r2l" - else + if not rlmode or rlmode >= 0 then return "l2r" + else + return "r2l" end end +-- The next code is somewhat complicated by the fact that some fonts can have ligatures made +-- from ligatures that themselves have marks. This was identified by Kai in for instance +-- arabtype: KAF LAM SHADDA ALEF FATHA (0x0643 0x0644 0x0651 0x0627 0x064E). This becomes +-- KAF LAM-ALEF with a SHADDA on the first and a FATHA op de second component. In a next +-- iteration this becomes a KAF-LAM-ALEF with a SHADDA on the second and a FATHA on the +-- third component. + -- We can assume that languages that use marks are not hyphenated. We can also assume -- that at most one discretionary is present. @@ -360,21 +402,6 @@ end -- However, for arabic we need to keep them around for the sake of mark placement -- and indices. -local function copy_glyph(g) -- next and prev are untouched ! - local components = getfield(g,"components") - if components then - setfield(g,"components") - local n = copy_node(g) - copyinjection(n,g) -- we need to preserve the lig indices - setfield(g,"components",components) - return n - else - local n = copy_node(g) - copyinjection(n,g) -- we need to preserve the lig indices - return n - end -end - local function flattendisk(head,disc) local pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true) local prev, next = getboth(disc) @@ -396,7 +423,7 @@ local function flattendisk(head,disc) elseif next then return next, next else - return -- maybe warning + -- return -- maybe warning end else if replace then @@ -419,18 +446,40 @@ local function appenddisc(disc,list) if post then setlink(posttail,posthead) else - post = phead + post = posthead end if replace then setlink(replacetail,replacehead) else - replace = rhead + replace = replacehead end setdisc(disc,pre,post,replace) end -- start is a mark and we need to keep that one +local take_components = getcomponents -- we overload here (for now) +local set_components = setcomponents -- we overload here (for now) +----- get_components = getcomponents -- we overload here (for now) + +local function count_components(start,marks) + if getid(start) ~= glyph_code then + return 0 + elseif getsubtype(start) == ligature_code then + local i = 0 + local components = getcomponents(start) + while components do + i = i + count_components(components,marks) + components = getnext(components) + end + return i + elseif not marks[getchar(start)] then + return 1 + else + return 0 + end +end + local function markstoligature(head,start,stop,char) if start == stop and getchar(start) == char then return head, start @@ -439,48 +488,28 @@ local function markstoligature(head,start,stop,char) local next = getnext(stop) setprev(start) setnext(stop) - local base = copy_glyph(start) + local base = copy_no_components(start,copyinjection) if head == start then head = base end resetinjection(base) setchar(base,char) setsubtype(base,ligature_code) - setfield(base,"components",start) - setlink(prev,base) - setlink(base,next) + set_components(base,start) + setlink(prev,base,next) return head, base end end --- The next code is somewhat complicated by the fact that some fonts can have ligatures made --- from ligatures that themselves have marks. This was identified by Kai in for instance --- arabtype: KAF LAM SHADDA ALEF FATHA (0x0643 0x0644 0x0651 0x0627 0x064E). This becomes --- KAF LAM-ALEF with a SHADDA on the first and a FATHA op de second component. In a next --- iteration this becomes a KAF-LAM-ALEF with a SHADDA on the second and a FATHA on the --- third component. - -local function getcomponentindex(start) -- we could store this offset in the glyph (nofcomponents) - if getid(start) ~= glyph_code then -- and then get rid of all components - return 0 - elseif getsubtype(start) == ligature_code then - local i = 0 - local components = getfield(start,"components") - while components do - i = i + getcomponentindex(components) - components = getnext(components) - end - return i - elseif not marks[getchar(start)] then - return 1 - else - return 0 - end -end - -local a_noligature = attributes.private("noligature") +-- Remark for Kai: (some arabic fonts do mark + mark = other mark and such) +-- +-- The hasmarks is needed for ligatures of marks that are part of a ligature in +-- which case we assume that we can delete the marks anyway (we can always become +-- more clever if needed) .. in fact the whole logic here should be redone. We're +-- in the not discfound branch then. We now have skiphash too so we can be more +-- selective if needed (todo). -local function toligature(head,start,stop,char,dataset,sequence,markflag,discfound) -- brr head +local function toligature(head,start,stop,char,dataset,sequence,skiphash,discfound,hasmarks) -- brr head if getattr(start,a_noligature) == 1 then -- so we can do: e\noligature{ff}e e\noligature{f}fie (we only look at the first) return head, start @@ -490,53 +519,43 @@ local function toligature(head,start,stop,char,dataset,sequence,markflag,discfou setchar(start,char) return head, start end - -- needs testing (side effects): - local components = getfield(start,"components") - if components then - -- we get a double free .. needs checking - -- flush_node_list(components) - end - -- local prev = getprev(start) local next = getnext(stop) local comp = start setprev(start) setnext(stop) - local base = copy_glyph(start) + local base = copy_no_components(start,copyinjection) if start == head then head = base end resetinjection(base) setchar(base,char) setsubtype(base,ligature_code) - setfield(base,"components",comp) -- start can have components ... do we need to flush? - if prev then - setnext(prev,base) - end - if next then - setprev(next,base) - end - setboth(base,prev,next) + set_components(base,comp) + setlink(prev,base,next) if not discfound then - local deletemarks = markflag ~= "mark" + local deletemarks = not skiphash or hasmarks local components = start local baseindex = 0 local componentindex = 0 local head = base local current = base - -- first we loop over the glyphs in start .. stop + -- first we loop over the glyphs in start ... stop while start do local char = getchar(start) if not marks[char] then baseindex = baseindex + componentindex - componentindex = getcomponentindex(start) - elseif not deletemarks then -- quite fishy + componentindex = count_components(start,marks) + -- we can be more clever here: "not deletemarks or (skiphash and not skiphash[char])" + -- and such: + elseif not deletemarks then + -- we can get a loop when the font expects otherwise (i.e. unexpected deletemarks) setligaindex(start,baseindex + getligaindex(start,componentindex)) if trace_marks then logwarning("%s: keep mark %s, gets index %s",pref(dataset,sequence),gref(char),getligaindex(start)) end local n = copy_node(start) - copyinjection(n,start) + copyinjection(n,start) -- is this ok ? we position later anyway head, current = insert_node_after(head,current,n) -- unlikely that mark has components elseif trace_marks then logwarning("%s: delete mark %s",pref(dataset,sequence),gref(char)) @@ -544,11 +563,11 @@ local function toligature(head,start,stop,char,dataset,sequence,markflag,discfou start = getnext(start) end -- we can have one accent as part of a lookup and another following - -- local start = components -- was wrong (component scanning was introduced when more complex ligs in devanagari was added) local start = getnext(current) while start do local char = ischar(start) if char then + -- also something skiphash here? if marks[char] then setligaindex(start,baseindex + getligaindex(start,componentindex)) if trace_marks then @@ -570,27 +589,14 @@ local function toligature(head,start,stop,char,dataset,sequence,markflag,discfou -- \- can give problems as there we can have a negative char but that won't match -- anyway local pre, post, replace, pretail, posttail, replacetail = getdisc(discfound,true) - if not replace then -- todo: signal simple hyphen - local prev = getprev(base) - local current = comp - local previous = nil - local copied = nil - while current do - if getid(current) == glyph_code then - local n = copy_node(current) - if copied then - setlink(previous,n) - else - copied = n - end - previous = n - end - current = getnext(current) - end - setprev(discnext) -- also blocks funny assignments - setnext(discprev) -- also blocks funny assignments + if not replace then + local prev = getprev(base) + local comp = take_components(base) + local copied = copy_only_glyphs(comp) if pre then setlink(discprev,pre) + else + setnext(discprev) -- also blocks funny assignments end pre = comp if post then @@ -598,20 +604,26 @@ local function toligature(head,start,stop,char,dataset,sequence,markflag,discfou setprev(post) else post = discnext + setprev(discnext) -- also blocks funny assignments end - setlink(prev,discfound) - setlink(discfound,next) + setlink(prev,discfound,next) setboth(base) - setfield(base,"components",copied) - setdisc(discfound,pre,post,base) -- was discretionary_code - base = prev -- restart + -- here components have a pointer so we can't free it! + set_components(base,copied) + replace = base + if forcediscretionaries then + setdisc(discfound,pre,post,replace,discretionary_code) + else + setdisc(discfound,pre,post,replace) + end + base = prev end end end return head, base end -local function multiple_glyphs(head,start,multiple,ignoremarks,what) +local function multiple_glyphs(head,start,multiple,skiphash,what,stop) -- what to do with skiphash matches here local nofmultiples = #multiple if nofmultiples > 0 then resetinjection(start) @@ -654,7 +666,11 @@ end local function get_alternative_glyph(start,alternatives,value) local n = #alternatives - if value == "random" then + if n == 1 then + -- we could actually change that into a gsub and save some memory in the + -- font loader but it makes tracing more messy + return alternatives[1], trace_alternatives and "1 (only one present)" + elseif value == "random" then local r = getrandom and getrandom("glyph",1,n) or random(1,n) return alternatives[r], trace_alternatives and formatters["value %a, taking %a"](value,r) elseif value == "first" then @@ -719,21 +735,25 @@ function handlers.gsub_alternate(head,start,dataset,sequence,alternative) return head, start, true end -function handlers.gsub_multiple(head,start,dataset,sequence,multiple) +function handlers.gsub_multiple(head,start,dataset,sequence,multiple,rlmode,skiphash) if trace_multiples then logprocess("%s: replacing %s by multiple %s",pref(dataset,sequence),gref(getchar(start)),gref(multiple)) end - return multiple_glyphs(head,start,multiple,sequence.flags[1],dataset[1]) + return multiple_glyphs(head,start,multiple,skiphash,dataset[1]) end -function handlers.gsub_ligature(head,start,dataset,sequence,ligature) +-- Don't we deal with disc otherwise now? I need to check if the next one can be +-- simplified. Anyway, it can be way messier: marks that get removed as well as +-- marks that are kept. + +function handlers.gsub_ligature(head,start,dataset,sequence,ligature,rlmode,skiphash) local current = getnext(start) if not current then return head, start, false, nil end local stop = nil local startchar = getchar(start) - if marks[startchar] then + if skiphash and skiphash[startchar] then while current do local char = ischar(current,currentfont) if char then @@ -764,15 +784,16 @@ function handlers.gsub_ligature(head,start,dataset,sequence,ligature) -- ok, goto next lookup end end - else - local skipmark = sequence.flags[1] + else -- is the check for disc still valid here ? and why only replace then local discfound = false local lastdisc = nil + local hasmarks = marks[startchar] while current do local char, id = ischar(current,currentfont) if char then - if skipmark and marks[char] then + if skiphash and skiphash[char] then current = getnext(current) + -- if stop then stop = current end -- ? else -- ligature is a tree local lg = ligature[char] -- can there be multiple in a row? maybe in a bad font if lg then @@ -780,6 +801,9 @@ function handlers.gsub_ligature(head,start,dataset,sequence,ligature) discfound = lastdisc lastdisc = nil end + if marks[char] then + hasmarks = true + end stop = current -- needed for fake so outside then ligature = lg current = getnext(current) @@ -791,8 +815,20 @@ function handlers.gsub_ligature(head,start,dataset,sequence,ligature) -- kind of weird break elseif id == disc_code then - -- tricky .. we also need to do pre here - local replace = getfield(current,"replace") + -- + -- Kai: see chainprocs, we probably could do the same here or was there a reason + -- why we kept the replace check here. + -- + -- if not discfound then + -- discfound = current + -- end + -- if current == stop then + -- break -- okay? or before the disc + -- else + -- current = getnext(current) + -- end + -- + local replace = getfield(current,"replace") -- hm: pre and post if replace then -- of{f-}{}{f}e o{f-}{}{f}fe o{-}{}{ff}e (oe and ff ligature) -- we can end up here when we have a start run .. testruns start at a disc but @@ -802,6 +838,9 @@ function handlers.gsub_ligature(head,start,dataset,sequence,ligature) if char then local lg = ligature[char] -- can there be multiple in a row? maybe in a bad font if lg then + if marks[char] then + hasmarks = true -- very unlikely + end ligature = lg replace = getnext(replace) else @@ -824,10 +863,10 @@ function handlers.gsub_ligature(head,start,dataset,sequence,ligature) if stop then if trace_ligatures then local stopchar = getchar(stop) - head, start = toligature(head,start,stop,lig,dataset,sequence,skipmark,discfound) + head, start = toligature(head,start,stop,lig,dataset,sequence,skiphash,discfound,hasmarks) logprocess("%s: replacing %s upto %s by ligature %s case 2",pref(dataset,sequence),gref(startchar),gref(stopchar),gref(lig)) else - head, start = toligature(head,start,stop,lig,dataset,sequence,skipmark,discfound) + head, start = toligature(head,start,stop,lig,dataset,sequence,skiphash,discfound,hasmarks) end else -- weird but happens (in some arabic font) @@ -845,84 +884,81 @@ function handlers.gsub_ligature(head,start,dataset,sequence,ligature) return head, start, false, discfound end -function handlers.gpos_single(head,start,dataset,sequence,kerns,rlmode,step,i,injection) +function handlers.gpos_single(head,start,dataset,sequence,kerns,rlmode,skiphash,step,injection) local startchar = getchar(start) - if step.format == "pair" then - local dx, dy, w, h = setpair(start,factor,rlmode,sequence.flags[4],kerns,injection) + local format = step.format + if format == "single" or type(kerns) == "table" then -- the table check can go + local dx, dy, w, h = setposition(0,start,factor,rlmode,kerns,injection) if trace_kerns then - logprocess("%s: shifting single %s by (%p,%p) and correction (%p,%p)",pref(dataset,sequence),gref(startchar),dx,dy,w,h) + logprocess("%s: shifting single %s by %s xy (%p,%p) and wh (%p,%p)",pref(dataset,sequence),gref(startchar),format,dx,dy,w,h) end else - -- needs checking .. maybe no kerns format for single - local k = setkern(start,factor,rlmode,kerns,injection) + local k = (format == "move" and setmove or setkern)(start,factor,rlmode,kerns,injection) if trace_kerns then - logprocess("%s: shifting single %s by %p",pref(dataset,sequence),gref(startchar),k) + logprocess("%s: shifting single %s by %s %p",pref(dataset,sequence),gref(startchar),format,k) end end - return head, start, false + return head, start, true end -function handlers.gpos_pair(head,start,dataset,sequence,kerns,rlmode,step,i,injection) +function handlers.gpos_pair(head,start,dataset,sequence,kerns,rlmode,skiphash,step,injection) local snext = getnext(start) if not snext then return head, start, false else local prev = start - local done = false while snext do local nextchar = ischar(snext,currentfont) if nextchar then - local krn = kerns[nextchar] - if not krn and marks[nextchar] then - prev = snext + if skiphash and skiphash[nextchar] then -- includes marks too when flag + prev = snext snext = getnext(snext) - elseif not krn then - break - elseif step.format == "pair" then - local a, b = krn[1], krn[2] - if optimizekerns then - -- this permits a mixed table, but we could also decide to optimize this - -- in the loader and use format 'kern' - if not b and a[1] == 0 and a[2] == 0 and a[4] == 0 then - local k = setkern(snext,factor,rlmode,a[3],injection) + else + local krn = kerns[nextchar] + if not krn then + break + end + local format = step.format + if format == "pair" then + local a, b = krn[1], krn[2] + if a == true then + -- zero + elseif a then -- #a > 0 + local x, y, w, h = setposition(1,start,factor,rlmode,a,injection) if trace_kerns then - logprocess("%s: shifting single %s by %p",pref(dataset,sequence),gref(nextchar),k) + local startchar = getchar(start) + logprocess("%s: shifting first of pair %s and %s by xy (%p,%p) and wh (%p,%p) as %s",pref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h,injection or "injections") end - done = true - break end - end - if a and #a > 0 then - local x, y, w, h = setpair(start,factor,rlmode,sequence.flags[4],a,injection) - if trace_kerns then - local startchar = getchar(start) - logprocess("%s: shifting first of pair %s and %s by (%p,%p) and correction (%p,%p) as %s",pref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h,injection or "injections") + if b == true then + -- zero + start = snext -- cf spec + elseif b then -- #b > 0 + local x, y, w, h = setposition(2,snext,factor,rlmode,b,injection) + if trace_kerns then + local startchar = getchar(start) + logprocess("%s: shifting second of pair %s and %s by xy (%p,%p) and wh (%p,%p) as %s",pref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h,injection or "injections") + end + start = snext -- cf spec + elseif forcepairadvance then + start = snext -- for testing, not cf spec end - end - if b and #b > 0 then - local x, y, w, h = setpair(snext,factor,rlmode,sequence.flags[4],b,injection) + return head, start, true + elseif krn ~= 0 then + local k = (format == "move" and setmove or setkern)(snext,factor,rlmode,krn,injection) if trace_kerns then - local startchar = getchar(snext) - logprocess("%s: shifting second of pair %s and %s by (%p,%p) and correction (%p,%p) as %s",pref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h,injection or "injections") + logprocess("%s: inserting %s %p between %s and %s as %s",pref(dataset,sequence),format,k,gref(getchar(prev)),gref(nextchar),injection or "injections") end + return head, start, true + else -- can't happen + break end - done = true - break - elseif krn ~= 0 then - local k = setkern(snext,factor,rlmode,krn,injection) - if trace_kerns then - logprocess("%s: inserting kern %p between %s and %s as %s",pref(dataset,sequence),k,gref(getchar(prev)),gref(nextchar),injection or "injections") - end - done = true - break - else -- can't happen - break end else break end end - return head, start, done + return head, start, false end end @@ -931,7 +967,7 @@ end we need to explicitly test for basechar, baselig and basemark entries.</p> --ldx]]-- -function handlers.gpos_mark2base(head,start,dataset,sequence,markanchors,rlmode) +function handlers.gpos_mark2base(head,start,dataset,sequence,markanchors,rlmode,skiphash) local markchar = getchar(start) if marks[markchar] then local base = getprev(start) -- [glyph] [start=mark] @@ -966,10 +1002,13 @@ function handlers.gpos_mark2base(head,start,dataset,sequence,markanchors,rlmode) local ma = markanchors[2] local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks) if trace_marks then - logprocess("%s, anchor %s, bound %s: anchoring mark %s to basechar %s => (%p,%p)", - pref(dataset,sequence),anchor,bound,gref(markchar),gref(basechar),dx,dy) + logprocess("%s, bound %s, anchoring mark %s to basechar %s => (%p,%p)", + pref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy) end return head, start, true + elseif trace_bugs then + -- onetimemessage(currentfont,basechar,"no base anchors",report_fonts) + logwarning("%s: mark %s is not anchored to %s",pref(dataset,sequence),gref(markchar),gref(basechar)) end elseif trace_bugs then logwarning("%s: nothing preceding, case %i",pref(dataset,sequence),1) @@ -983,7 +1022,7 @@ function handlers.gpos_mark2base(head,start,dataset,sequence,markanchors,rlmode) return head, start, false end -function handlers.gpos_mark2ligature(head,start,dataset,sequence,markanchors,rlmode) +function handlers.gpos_mark2ligature(head,start,dataset,sequence,markanchors,rlmode,skiphash) local markchar = getchar(start) if marks[markchar] then local base = getprev(start) -- [glyph] [optional marks] [start=mark] @@ -1022,8 +1061,8 @@ function handlers.gpos_mark2ligature(head,start,dataset,sequence,markanchors,rlm if ba then local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks) if trace_marks then - logprocess("%s, anchor %s, index %s, bound %s: anchoring mark %s to baselig %s at index %s => (%p,%p)", - pref(dataset,sequence),anchor,index,bound,gref(markchar),gref(basechar),index,dx,dy) + logprocess("%s, index %s, bound %s, anchoring mark %s to baselig %s at index %s => (%p,%p)", + pref(dataset,sequence),index,bound,gref(markchar),gref(basechar),index,dx,dy) end return head, start, true else @@ -1048,7 +1087,7 @@ function handlers.gpos_mark2ligature(head,start,dataset,sequence,markanchors,rlm return head, start, false end -function handlers.gpos_mark2mark(head,start,dataset,sequence,markanchors,rlmode) +function handlers.gpos_mark2mark(head,start,dataset,sequence,markanchors,rlmode,skiphash) local markchar = getchar(start) if marks[markchar] then local base = getprev(start) -- [glyph] [basemark] [start=mark] @@ -1071,8 +1110,8 @@ function handlers.gpos_mark2mark(head,start,dataset,sequence,markanchors,rlmode) local ma = markanchors[2] local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],true,checkmarks) if trace_marks then - logprocess("%s, anchor %s, bound %s: anchoring mark %s to basemark %s => (%p,%p)", - pref(dataset,sequence),anchor,bound,gref(markchar),gref(basechar),dx,dy) + logprocess("%s, bound %s, anchoring mark %s to basemark %s => (%p,%p)", + pref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy) end return head, start, true end @@ -1084,8 +1123,7 @@ function handlers.gpos_mark2mark(head,start,dataset,sequence,markanchors,rlmode) return head, start, false end -function handlers.gpos_cursive(head,start,dataset,sequence,exitanchors,rlmode,step,i) -- to be checked - local done = false +function handlers.gpos_cursive(head,start,dataset,sequence,exitanchors,rlmode,skiphash,step) -- to be checked local startchar = getchar(start) if marks[startchar] then if trace_cursive then @@ -1093,12 +1131,11 @@ function handlers.gpos_cursive(head,start,dataset,sequence,exitanchors,rlmode,st end else local nxt = getnext(start) - while not done and nxt do + while nxt do local nextchar = ischar(nxt,currentfont) if not nextchar then break - elseif marks[nextchar] then - -- should not happen (maybe warning) + elseif marks[nextchar] then -- always sequence.flags[1] nxt = getnext(nxt) else local exit = exitanchors[3] @@ -1107,11 +1144,12 @@ function handlers.gpos_cursive(head,start,dataset,sequence,exitanchors,rlmode,st if entry then entry = entry[2] if entry then - local dx, dy, bound = setcursive(start,nxt,factor,rlmode,exit,entry,characters[startchar],characters[nextchar]) + local r2lflag = sequence.flags[4] -- mentioned in the standard + local dx, dy, bound = setcursive(start,nxt,factor,rlmode,exit,entry,characters[startchar],characters[nextchar],r2lflag) if trace_cursive then - logprocess("%s: moving %s to %s cursive (%p,%p) using anchor %s and bound %s in %s mode",pref(dataset,sequence),gref(startchar),gref(nextchar),dx,dy,anchor,bound,mref(rlmode)) + logprocess("%s: moving %s to %s cursive (%p,%p) using bound %s in %s mode",pref(dataset,sequence),gref(startchar),gref(nextchar),dx,dy,bound,mref(rlmode)) end - done = true + return head, start, true end end end @@ -1119,7 +1157,7 @@ function handlers.gpos_cursive(head,start,dataset,sequence,exitanchors,rlmode,st end end end - return head, start, done + return head, start, false end --[[ldx-- @@ -1132,6 +1170,9 @@ local chainprocs = { } local function logprocess(...) if trace_steps then registermessage(...) + if trace_steps == "silent" then + return + end end report_subchain(...) end @@ -1141,6 +1182,9 @@ local logwarning = report_subchain local function logprocess(...) if trace_steps then registermessage(...) + if trace_steps == "silent" then + return + end end report_chain(...) end @@ -1154,7 +1198,7 @@ local logwarning = report_chain -- in a bit weird way. There is no lookup and the replacement comes from the lookup -- itself. It is meant mostly for dealing with Urdu. -local function reversesub(head,start,stop,dataset,sequence,replacements,rlmode) +local function reversesub(head,start,stop,dataset,sequence,replacements,rlmode,skiphash) local char = getchar(start) local replacement = replacements[char] if replacement then @@ -1196,66 +1240,76 @@ as less as needed but that would also make the code even more messy.</p> -- this is messy: do we need this disc checking also in alternaties? +local function reportzerosteps(dataset,sequence) + logwarning("%s: no steps",cref(dataset,sequence)) +end + local function reportmoresteps(dataset,sequence) logwarning("%s: more than 1 step",cref(dataset,sequence)) end -function chainprocs.gsub_single(head,start,stop,dataset,sequence,currentlookup,chainindex) +-- local function reportbadsteps(dataset,sequence) +-- logwarning("%s: bad step, no proper return values",cref(dataset,sequence)) +-- end + +local function getmapping(dataset,sequence,currentlookup) local steps = currentlookup.steps local nofsteps = currentlookup.nofsteps - if nofsteps > 1 then - reportmoresteps(dataset,sequence) - end - local current = start - while current do - local currentchar = ischar(current) - if currentchar then - local replacement = steps[1].coverage[currentchar] - if not replacement or replacement == "" then - if trace_bugs then - logwarning("%s: no single for %s",cref(dataset,sequence,chainindex),gref(currentchar)) - end - else - if trace_singles then - logprocess("%s: replacing single %s by %s",cref(dataset,sequence,chainindex),gref(currentchar),gref(replacement)) - end - resetinjection(current) - setchar(current,replacement) - end - return head, start, true - elseif currentchar == false then - -- can't happen - break - elseif current == stop then - break - else - current = getnext(current) + if nofsteps == 0 then + reportzerosteps(dataset,sequence) + currentlookup.mapping = false + return false + else + if nofsteps > 1 then + reportmoresteps(dataset,sequence) end + local mapping = steps[1].coverage + currentlookup.mapping = mapping + currentlookup.format = steps[1].format + return mapping end - return head, start, false end ---[[ldx-- -<p>Here we replace start by a sequence of new glyphs.</p> ---ldx]]-- +function chainprocs.gsub_remove(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex) + if trace_chains then + logprocess("%s: removing character %s",cref(dataset,sequence,chainindex),gref(getchar(start))) + end + head, start = remove_node(head,start,true) + return head, getprev(start), true +end -function chainprocs.gsub_multiple(head,start,stop,dataset,sequence,currentlookup) - local steps = currentlookup.steps - local nofsteps = currentlookup.nofsteps - if nofsteps > 1 then - reportmoresteps(dataset,sequence) +function chainprocs.gsub_single(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex) + local mapping = currentlookup.mapping + if mapping == nil then + mapping = getmapping(dataset,sequence,currentlookup) end - local startchar = getchar(start) - local replacement = steps[1].coverage[startchar] - if not replacement or replacement == "" then - if trace_bugs then - logwarning("%s: no multiple for %s",cref(dataset,sequence),gref(startchar)) - end - else - if trace_multiples then - logprocess("%s: replacing %s by multiple characters %s",cref(dataset,sequence),gref(startchar),gref(replacement)) + if mapping then + local current = start + while current do + local currentchar = ischar(current) + if currentchar then + local replacement = mapping[currentchar] + if not replacement or replacement == "" then + if trace_bugs then + logwarning("%s: no single for %s",cref(dataset,sequence,chainindex),gref(currentchar)) + end + else + if trace_singles then + logprocess("%s: replacing single %s by %s",cref(dataset,sequence,chainindex),gref(currentchar),gref(replacement)) + end + resetinjection(current) + setchar(current,replacement) + end + return head, start, true + elseif currentchar == false then + -- can't happen + break + elseif current == stop then + break + else + current = getnext(current) + end end - return multiple_glyphs(head,start,replacement,sequence.flags[1],dataset[1]) end return head, start, false end @@ -1272,42 +1326,69 @@ end -- marks come last anyway -- are there cases where we need to delete the mark -function chainprocs.gsub_alternate(head,start,stop,dataset,sequence,currentlookup) - local steps = currentlookup.steps - local nofsteps = currentlookup.nofsteps - if nofsteps > 1 then - reportmoresteps(dataset,sequence) +function chainprocs.gsub_alternate(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex) + local mapping = currentlookup.mapping + if mapping == nil then + mapping = getmapping(dataset,sequence,currentlookup) end - local kind = dataset[4] - local what = dataset[1] - local value = what == true and tfmdata.shared.features[kind] or what -- todo: optimize in ctx - local current = start - while current do - local currentchar = ischar(current) - if currentchar then - local alternatives = steps[1].coverage[currentchar] - if alternatives then - local choice, comment = get_alternative_glyph(current,alternatives,value) - if choice then - if trace_alternatives then - logprocess("%s: replacing %s by alternative %a to %s, %s",cref(dataset,sequence),gref(char),choice,gref(choice),comment) - end - resetinjection(start) - setchar(start,choice) - else - if trace_alternatives then - logwarning("%s: no variant %a for %s, %s",cref(dataset,sequence),value,gref(char),comment) + if mapping then + local kind = dataset[4] + local what = dataset[1] + local value = what == true and tfmdata.shared.features[kind] or what -- todo: optimize in ctx + local current = start + while current do + local currentchar = ischar(current) + if currentchar then + local alternatives = mapping[currentchar] + if alternatives then + local choice, comment = get_alternative_glyph(current,alternatives,value) + if choice then + if trace_alternatives then + logprocess("%s: replacing %s by alternative %a to %s, %s",cref(dataset,sequence),gref(currentchar),choice,gref(choice),comment) + end + resetinjection(start) + setchar(start,choice) + else + if trace_alternatives then + logwarning("%s: no variant %a for %s, %s",cref(dataset,sequence),value,gref(currentchar),comment) + end end end + return head, start, true + elseif currentchar == false then + -- can't happen + break + elseif current == stop then + break + else + current = getnext(current) + end + end + end + return head, start, false +end + +--[[ldx-- +<p>Here we replace start by a sequence of new glyphs.</p> +--ldx]]-- + +function chainprocs.gsub_multiple(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex) + local mapping = currentlookup.mapping + if mapping == nil then + mapping = getmapping(dataset,sequence,currentlookup) + end + if mapping then + local startchar = getchar(start) + local replacement = mapping[startchar] + if not replacement or replacement == "" then + if trace_bugs then + logwarning("%s: no multiple for %s",cref(dataset,sequence),gref(startchar)) end - return head, start, true - elseif currentchar == false then - -- can't happen - break - elseif current == stop then - break else - current = getnext(current) + if trace_multiples then + logprocess("%s: replacing %s by multiple characters %s",cref(dataset,sequence),gref(startchar),gref(replacement)) + end + return multiple_glyphs(head,start,replacement,skiphash,dataset[1],stop) end end return head, start, false @@ -1319,419 +1400,423 @@ this function (move code inline and handle the marks by a separate function). We assume rather stupid ligatures (no complex disc nodes).</p> --ldx]]-- -function chainprocs.gsub_ligature(head,start,stop,dataset,sequence,currentlookup,chainindex) - local steps = currentlookup.steps - local nofsteps = currentlookup.nofsteps - if nofsteps > 1 then - reportmoresteps(dataset,sequence) +-- compare to handlers.gsub_ligature which is more complex ... why + +function chainprocs.gsub_ligature(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex) + local mapping = currentlookup.mapping + if mapping == nil then + mapping = getmapping(dataset,sequence,currentlookup) end - local startchar = getchar(start) - local ligatures = steps[1].coverage[startchar] - if not ligatures then - if trace_bugs then - logwarning("%s: no ligatures starting with %s",cref(dataset,sequence,chainindex),gref(startchar)) - end - else - local current = getnext(start) - local discfound = false - local last = stop - local nofreplacements = 1 - local skipmark = currentlookup.flags[1] -- sequence.flags? - while current do - -- todo: ischar ... can there really be disc nodes here? - local id = getid(current) - if id == disc_code then - if not discfound then - discfound = current - end - if current == stop then - break -- okay? or before the disc - else - current = getnext(current) - end - else - local schar = getchar(current) - if skipmark and marks[schar] then -- marks - -- if current == stop then -- maybe add this - -- break - -- else + if mapping then + local startchar = getchar(start) + local ligatures = mapping[startchar] + if not ligatures then + if trace_bugs then + logwarning("%s: no ligatures starting with %s",cref(dataset,sequence,chainindex),gref(startchar)) + end + else + local hasmarks = marks[startchar] + local current = getnext(start) + local discfound = false + local last = stop + local nofreplacements = 1 + while current do + -- todo: ischar ... can there really be disc nodes here? + local id = getid(current) + if id == disc_code then + if not discfound then + discfound = current + end + if current == stop then + break -- okay? or before the disc + else current = getnext(current) - -- end + end else - local lg = ligatures[schar] - if lg then - ligatures = lg - last = current - nofreplacements = nofreplacements + 1 - if current == stop then - break - else + local schar = getchar(current) + if skiphash and skiphash[schar] then -- marks + -- if current == stop then -- maybe add this + -- break + -- else current = getnext(current) - end + -- end else - break + local lg = ligatures[schar] + if lg then + ligatures = lg + last = current + nofreplacements = nofreplacements + 1 + if marks[char] then + hasmarks = true + end + if current == stop then + break + else + current = getnext(current) + end + else + break + end end end end - end - local ligature = ligatures.ligature - if ligature then - if chainindex then - stop = last - end - if trace_ligatures then + local ligature = ligatures.ligature + if ligature then + if chainindex then + stop = last + end + if trace_ligatures then + if start == stop then + logprocess("%s: replacing character %s by ligature %s case 3",cref(dataset,sequence,chainindex),gref(startchar),gref(ligature)) + else + logprocess("%s: replacing character %s upto %s by ligature %s case 4",cref(dataset,sequence,chainindex),gref(startchar),gref(getchar(stop)),gref(ligature)) + end + end + head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,discfound,hasmarks) + return head, start, true, nofreplacements, discfound + elseif trace_bugs then if start == stop then - logprocess("%s: replacing character %s by ligature %s case 3",cref(dataset,sequence,chainindex),gref(startchar),gref(ligature)) + logwarning("%s: replacing character %s by ligature fails",cref(dataset,sequence,chainindex),gref(startchar)) else - logprocess("%s: replacing character %s upto %s by ligature %s case 4",cref(dataset,sequence,chainindex),gref(startchar),gref(getchar(stop)),gref(ligature)) + logwarning("%s: replacing character %s upto %s by ligature fails",cref(dataset,sequence,chainindex),gref(startchar),gref(getchar(stop))) end end - head, start = toligature(head,start,stop,ligature,dataset,sequence,skipmark,discfound) - return head, start, true, nofreplacements, discfound - elseif trace_bugs then - if start == stop then - logwarning("%s: replacing character %s by ligature fails",cref(dataset,sequence,chainindex),gref(startchar)) - else - logwarning("%s: replacing character %s upto %s by ligature fails",cref(dataset,sequence,chainindex),gref(startchar),gref(getchar(stop))) - end end end return head, start, false, 0, false end -function chainprocs.gpos_single(head,start,stop,dataset,sequence,currentlookup,rlmode,chainindex) - local steps = currentlookup.steps - local nofsteps = currentlookup.nofsteps - if nofsteps > 1 then - reportmoresteps(dataset,sequence) +function chainprocs.gpos_single(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex) + local mapping = currentlookup.mapping + if mapping == nil then + mapping = getmapping(dataset,sequence,currentlookup) end - local startchar = getchar(start) - local step = steps[1] - local kerns = step.coverage[startchar] - if not kerns then - -- skip - elseif step.format == "pair" then - local dx, dy, w, h = setpair(start,factor,rlmode,sequence.flags[4],kerns) -- currentlookup.flags ? - if trace_kerns then - logprocess("%s: shifting single %s by (%p,%p) and correction (%p,%p)",cref(dataset,sequence),gref(startchar),dx,dy,w,h) - end - else -- needs checking .. maybe no kerns format for single - local k = setkern(start,factor,rlmode,kerns,injection) - if trace_kerns then - logprocess("%s: shifting single %s by %p",cref(dataset,sequence),gref(startchar),k) + if mapping then + local startchar = getchar(start) + local kerns = mapping[startchar] + if kerns then + local format = currentlookup.format + if format == "single" then + local dx, dy, w, h = setposition(0,start,factor,rlmode,kerns) -- currentlookup.flags ? + if trace_kerns then + logprocess("%s: shifting single %s by %s (%p,%p) and correction (%p,%p)",cref(dataset,sequence),gref(startchar),format,dx,dy,w,h) + end + else -- needs checking .. maybe no kerns format for single + local k = (format == "move" and setmove or setkern)(start,factor,rlmode,kerns,injection) + if trace_kerns then + logprocess("%s: shifting single %s by %s %p",cref(dataset,sequence),gref(startchar),format,k) + end + end + return head, start, true end end return head, start, false end -function chainprocs.gpos_pair(head,start,stop,dataset,sequence,currentlookup,rlmode,chainindex) -- todo: injections ? - local steps = currentlookup.steps - local nofsteps = currentlookup.nofsteps - if nofsteps > 1 then - reportmoresteps(dataset,sequence) +function chainprocs.gpos_pair(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex) -- todo: injections ? + local mapping = currentlookup.mapping + if mapping == nil then + mapping = getmapping(dataset,sequence,currentlookup) end - local snext = getnext(start) - if snext then - local startchar = getchar(start) - local step = steps[1] - local kerns = step.coverage[startchar] -- always 1 step - if kerns then - local prev = start - local done = false - while snext do - local nextchar = ischar(snext,currentfont) - if not nextchar then - break - end - local krn = kerns[nextchar] - if not krn and marks[nextchar] then - prev = snext - snext = getnext(snext) - elseif not krn then - break - elseif step.format == "pair" then - local a, b = krn[1], krn[2] - if optimizekerns then - -- this permits a mixed table, but we could also decide to optimize this - -- in the loader and use format 'kern' - if not b and a[1] == 0 and a[2] == 0 and a[4] == 0 then - local k = setkern(snext,factor,rlmode,a[3],"injections") + if mapping then + local snext = getnext(start) + if snext then + local startchar = getchar(start) + local kerns = mapping[startchar] -- always 1 step + if kerns then + local prev = start + while snext do + local nextchar = ischar(snext,currentfont) + if not nextchar then + break + end + if skiphash and skiphash[nextchar] then + prev = snext + snext = getnext(snext) + else + local krn = kerns[nextchar] + if not krn then + break + end + local format = currentlookup.format + if format == "pair" then + local a, b = krn[1], krn[2] + if a == true then + -- zero + elseif a then + local x, y, w, h = setposition(1,start,factor,rlmode,a,"injections") -- currentlookups flags? + if trace_kerns then + local startchar = getchar(start) + logprocess("%s: shifting first of pair %s and %s by (%p,%p) and correction (%p,%p)",cref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h) + end + end + if b == true then + -- zero + start = snext -- cf spec + elseif b then -- #b > 0 + local x, y, w, h = setposition(2,snext,factor,rlmode,b,"injections") + if trace_kerns then + local startchar = getchar(start) + logprocess("%s: shifting second of pair %s and %s by (%p,%p) and correction (%p,%p)",cref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h) + end + start = snext -- cf spec + elseif forcepairadvance then + start = snext -- for testing, not cf spec + end + return head, start, true + elseif krn ~= 0 then + local k = (format == "move" and setmove or setkern)(snext,factor,rlmode,krn) if trace_kerns then - logprocess("%s: shifting single %s by %p",cref(dataset,sequence),gref(startchar),k) + logprocess("%s: inserting %s %p between %s and %s",cref(dataset,sequence),format,k,gref(getchar(prev)),gref(nextchar)) end - done = true + return head, start, true + else break end end - if a and #a > 0 then - local startchar = getchar(start) - local x, y, w, h = setpair(start,factor,rlmode,sequence.flags[4],a,"injections") -- currentlookups flags? - if trace_kerns then - logprocess("%s: shifting first of pair %s and %s by (%p,%p) and correction (%p,%p)",cref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h) - end - end - if b and #b > 0 then - local startchar = getchar(start) - local x, y, w, h = setpair(snext,factor,rlmode,sequence.flags[4],b,"injections") - if trace_kerns then - logprocess("%s: shifting second of pair %s and %s by (%p,%p) and correction (%p,%p)",cref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h) - end - end - done = true - break - elseif krn ~= 0 then - local k = setkern(snext,factor,rlmode,krn) - if trace_kerns then - logprocess("%s: inserting kern %s between %s and %s",cref(dataset,sequence),k,gref(getchar(prev)),gref(nextchar)) - end - done = true - break - else - break end end - return head, start, done end end return head, start, false end -function chainprocs.gpos_mark2base(head,start,stop,dataset,sequence,currentlookup,rlmode) - local steps = currentlookup.steps - local nofsteps = currentlookup.nofsteps - if nofsteps > 1 then - reportmoresteps(dataset,sequence) +function chainprocs.gpos_mark2base(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex) + local mapping = currentlookup.mapping + if mapping == nil then + mapping = getmapping(dataset,sequence,currentlookup) end - local markchar = getchar(start) - if marks[markchar] then - local markanchors = steps[1].coverage[markchar] -- always 1 step - if markanchors then - local base = getprev(start) -- [glyph] [start=mark] - if base then - local basechar = ischar(base,currentfont) - if basechar then - if marks[basechar] then - while base do - base = getprev(base) - if base then - local basechar = ischar(base,currentfont) - if basechar then - if not marks[basechar] then - break + if mapping then + local markchar = getchar(start) + if marks[markchar] then + local markanchors = mapping[markchar] -- always 1 step + if markanchors then + local base = getprev(start) -- [glyph] [start=mark] + if base then + local basechar = ischar(base,currentfont) + if basechar then + if marks[basechar] then + while base do + base = getprev(base) + if base then + local basechar = ischar(base,currentfont) + if basechar then + if not marks[basechar] then + break + end + else + if trace_bugs then + logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),1) + end + return head, start, false end else if trace_bugs then - logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),1) + logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),2) end return head, start, false end - else - if trace_bugs then - logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),2) - end - return head, start, false end end - end - local ba = markanchors[1][basechar] - if ba then - local ma = markanchors[2] - if ma then - local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks) - if trace_marks then - logprocess("%s, anchor %s, bound %s: anchoring mark %s to basechar %s => (%p,%p)", - cref(dataset,sequence),anchor,bound,gref(markchar),gref(basechar),dx,dy) + local ba = markanchors[1][basechar] + if ba then + local ma = markanchors[2] + if ma then + local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks) + if trace_marks then + logprocess("%s, bound %s, anchoring mark %s to basechar %s => (%p,%p)", + cref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy) + end + return head, start, true end - return head, start, true end + elseif trace_bugs then + logwarning("%s: prev node is no char, case %i",cref(dataset,sequence),1) end elseif trace_bugs then - logwarning("%s: prev node is no char, case %i",cref(dataset,sequence),1) + logwarning("%s: prev node is no char, case %i",cref(dataset,sequence),2) end elseif trace_bugs then - logwarning("%s: prev node is no char, case %i",cref(dataset,sequence),2) + logwarning("%s: mark %s has no anchors",cref(dataset,sequence),gref(markchar)) end elseif trace_bugs then - logwarning("%s: mark %s has no anchors",cref(dataset,sequence),gref(markchar)) + logwarning("%s: mark %s is no mark",cref(dataset,sequence),gref(markchar)) end - elseif trace_bugs then - logwarning("%s: mark %s is no mark",cref(dataset,sequence),gref(markchar)) end return head, start, false end -function chainprocs.gpos_mark2ligature(head,start,stop,dataset,sequence,currentlookup,rlmode) - local steps = currentlookup.steps - local nofsteps = currentlookup.nofsteps - if nofsteps > 1 then - reportmoresteps(dataset,sequence) +function chainprocs.gpos_mark2ligature(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex) + local mapping = currentlookup.mapping + if mapping == nil then + mapping = getmapping(dataset,sequence,currentlookup) end - local markchar = getchar(start) - if marks[markchar] then - local markanchors = steps[1].coverage[markchar] -- always 1 step - if markanchors then - local base = getprev(start) -- [glyph] [optional marks] [start=mark] - if base then - local basechar = ischar(base,currentfont) - if basechar then - if marks[basechar] then - while base do - base = getprev(base) - if base then - local basechar = ischar(base,currentfont) - if basechar then - if not marks[basechar] then - break + if mapping then + local markchar = getchar(start) + if marks[markchar] then + local markanchors = mapping[markchar] -- always 1 step + if markanchors then + local base = getprev(start) -- [glyph] [optional marks] [start=mark] + if base then + local basechar = ischar(base,currentfont) + if basechar then + if marks[basechar] then + while base do + base = getprev(base) + if base then + local basechar = ischar(base,currentfont) + if basechar then + if not marks[basechar] then + break + end + else + if trace_bugs then + logwarning("%s: no base for mark %s, case %i",cref(dataset,sequence),markchar,1) + end + return head, start, false end else if trace_bugs then - logwarning("%s: no base for mark %s, case %i",cref(dataset,sequence),markchar,1) + logwarning("%s: no base for mark %s, case %i",cref(dataset,sequence),markchar,2) end return head, start, false end - else - if trace_bugs then - logwarning("%s: no base for mark %s, case %i",cref(dataset,sequence),markchar,2) - end - return head, start, false end end - end - local ba = markanchors[1][basechar] - if ba then - local ma = markanchors[2] - if ma then - local index = getligaindex(start) - ba = ba[index] - if ba then - local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks) - if trace_marks then - logprocess("%s, anchor %s, bound %s: anchoring mark %s to baselig %s at index %s => (%p,%p)", - cref(dataset,sequence),anchor,a or bound,gref(markchar),gref(basechar),index,dx,dy) + local ba = markanchors[1][basechar] + if ba then + local ma = markanchors[2] + if ma then + local index = getligaindex(start) + ba = ba[index] + if ba then + local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks) + if trace_marks then + logprocess("%s, bound %s, anchoring mark %s to baselig %s at index %s => (%p,%p)", + cref(dataset,sequence),a or bound,gref(markchar),gref(basechar),index,dx,dy) + end + return head, start, true end - return head, start, true end end + elseif trace_bugs then + logwarning("%s, prev node is no char, case %i",cref(dataset,sequence),1) end elseif trace_bugs then - logwarning("%s, prev node is no char, case %i",cref(dataset,sequence),1) + logwarning("%s, prev node is no char, case %i",cref(dataset,sequence),2) end elseif trace_bugs then - logwarning("%s, prev node is no char, case %i",cref(dataset,sequence),2) + logwarning("%s, mark %s has no anchors",cref(dataset,sequence),gref(markchar)) end elseif trace_bugs then - logwarning("%s, mark %s has no anchors",cref(dataset,sequence),gref(markchar)) + logwarning("%s, mark %s is no mark",cref(dataset,sequence),gref(markchar)) end - elseif trace_bugs then - logwarning("%s, mark %s is no mark",cref(dataset,sequence),gref(markchar)) end return head, start, false end -function chainprocs.gpos_mark2mark(head,start,stop,dataset,sequence,currentlookup,rlmode) - local steps = currentlookup.steps - local nofsteps = currentlookup.nofsteps - if nofsteps > 1 then - reportmoresteps(dataset,sequence) +function chainprocs.gpos_mark2mark(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex) + local mapping = currentlookup.mapping + if mapping == nil then + mapping = getmapping(dataset,sequence,currentlookup) end - local markchar = getchar(start) - if marks[markchar] then - local markanchors = steps[1].coverage[markchar] -- always 1 step - if markanchors then - local base = getprev(start) -- [glyph] [basemark] [start=mark] - local slc = getligaindex(start) - if slc then -- a rather messy loop ... needs checking with husayni - while base do - local blc = getligaindex(base) - if blc and blc ~= slc then - base = getprev(base) - else - break + if mapping then + local markchar = getchar(start) + if marks[markchar] then + local markanchors = mapping[markchar] -- always 1 step + if markanchors then + local base = getprev(start) -- [glyph] [basemark] [start=mark] + local slc = getligaindex(start) + if slc then -- a rather messy loop ... needs checking with husayni + while base do + local blc = getligaindex(base) + if blc and blc ~= slc then + base = getprev(base) + else + break + end end end - end - if base then -- subtype test can go - local basechar = ischar(base,currentfont) - if basechar then - local ba = markanchors[1][basechar] - if ba then - local ma = markanchors[2] - if ma then - local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],true,checkmarks) - if trace_marks then - logprocess("%s, anchor %s, bound %s: anchoring mark %s to basemark %s => (%p,%p)", - cref(dataset,sequence),anchor,bound,gref(markchar),gref(basechar),dx,dy) + if base then -- subtype test can go + local basechar = ischar(base,currentfont) + if basechar then + local ba = markanchors[1][basechar] + if ba then + local ma = markanchors[2] + if ma then + local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],true,checkmarks) + if trace_marks then + logprocess("%s, bound %s, anchoring mark %s to basemark %s => (%p,%p)", + cref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy) + end + return head, start, true end - return head, start, true end + elseif trace_bugs then + logwarning("%s: prev node is no mark, case %i",cref(dataset,sequence),1) end elseif trace_bugs then - logwarning("%s: prev node is no mark, case %i",cref(dataset,sequence),1) + logwarning("%s: prev node is no mark, case %i",cref(dataset,sequence),2) end elseif trace_bugs then - logwarning("%s: prev node is no mark, case %i",cref(dataset,sequence),2) + logwarning("%s: mark %s has no anchors",cref(dataset,sequence),gref(markchar)) end elseif trace_bugs then - logwarning("%s: mark %s has no anchors",cref(dataset,sequence),gref(markchar)) + logwarning("%s: mark %s is no mark",cref(dataset,sequence),gref(markchar)) end - elseif trace_bugs then - logwarning("%s: mark %s is no mark",cref(dataset,sequence),gref(markchar)) end return head, start, false end -function chainprocs.gpos_cursive(head,start,stop,dataset,sequence,currentlookup,rlmode) - local steps = currentlookup.steps - local nofsteps = currentlookup.nofsteps - if nofsteps > 1 then - reportmoresteps(dataset,sequence) +function chainprocs.gpos_cursive(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex) + local mapping = currentlookup.mapping + if mapping == nil then + mapping = getmapping(dataset,sequence,currentlookup) end - local startchar = getchar(start) - local exitanchors = steps[1].coverage[startchar] -- always 1 step - if exitanchors then - local done = false - if marks[startchar] then - if trace_cursive then - logprocess("%s: ignoring cursive for mark %s",pref(dataset,sequence),gref(startchar)) - end - else - local nxt = getnext(start) - while not done and nxt do - local nextchar = ischar(nxt,currentfont) - if not nextchar then - break - elseif marks[nextchar] then - -- should not happen (maybe warning) - nxt = getnext(nxt) - else - local exit = exitanchors[3] - if exit then - local entry = exitanchors[1][nextchar] - if entry then - entry = entry[2] + if mapping then + local startchar = getchar(start) + local exitanchors = mapping[startchar] -- always 1 step + if exitanchors then + if marks[startchar] then + if trace_cursive then + logprocess("%s: ignoring cursive for mark %s",pref(dataset,sequence),gref(startchar)) + end + else + local nxt = getnext(start) + while nxt do + local nextchar = ischar(nxt,currentfont) + if not nextchar then + break + elseif marks[nextchar] then + -- should not happen (maybe warning) + nxt = getnext(nxt) + else + local exit = exitanchors[3] + if exit then + local entry = exitanchors[1][nextchar] if entry then - local dx, dy, bound = setcursive(start,nxt,factor,rlmode,exit,entry,characters[startchar],characters[nextchar]) - if trace_cursive then - logprocess("%s: moving %s to %s cursive (%p,%p) using anchor %s and bound %s in %s mode",pref(dataset,sequence),gref(startchar),gref(nextchar),dx,dy,anchor,bound,mref(rlmode)) + entry = entry[2] + if entry then + local r2lflag = sequence.flags[4] -- mentioned in the standard + local dx, dy, bound = setcursive(start,nxt,factor,rlmode,exit,entry,characters[startchar],characters[nextchar],r2lflag) + if trace_cursive then + logprocess("%s: moving %s to %s cursive (%p,%p) using bound %s in %s mode",pref(dataset,sequence),gref(startchar),gref(nextchar),dx,dy,bound,mref(rlmode)) + end + return head, start, true end - done = true - break end + elseif trace_bugs then + onetimemessage(currentfont,startchar,"no entry anchors",report_fonts) end - elseif trace_bugs then - onetimemessage(currentfont,startchar,"no entry anchors",report_fonts) + break end - break end end - end - return head, start, done - else - if trace_cursive and trace_details then + elseif trace_cursive and trace_details then logprocess("%s, cursive %s is already done",pref(dataset,sequence),gref(getchar(start)),alreadydone) end - return head, start, false end + return head, start, false end -- what pointer to return, spec says stop @@ -1745,36 +1830,50 @@ end -- A previous version had disc collapsing code in the (single sub) handler plus some -- checking in the main loop, but that left the pre/post sequences undone. The best -- solution is to add some checking there and backtrack when a replace/post matches --- but it takes a bit of work to figure out an efficient way (this is what the sweep* --- names refer to). I might look into that variant one day again as it can replace --- some other code too. In that approach we can have a special version for gub and pos --- which gains some speed. This method does the test and passes info to the handlers --- (sweepnode, sweepmode, sweepprev, sweepnext, etc). Here collapsing is handled in the --- main loop which also makes code elsewhere simpler (i.e. no need for the other special --- runners and disc code in ligature building). I also experimented with pushing preceding --- glyphs sequences in the replace/pre fields beforehand which saves checking afterwards --- but at the cost of duplicate glyphs (memory) but it's too much overhead (runtime). +-- but it takes a bit of work to figure out an efficient way (this is what the +-- sweep* names refer to). I might look into that variant one day again as it can +-- replace some other code too. In that approach we can have a special version for +-- gub and pos which gains some speed. This method does the test and passes info to +-- the handlers. Here collapsing is handled in the main loop which also makes code +-- elsewhere simpler (i.e. no need for the other special runners and disc code in +-- ligature building). I also experimented with pushing preceding glyphs sequences +-- in the replace/pre fields beforehand which saves checking afterwards but at the +-- cost of duplicate glyphs (memory) but it's too much overhead (runtime). -- --- In the meantime Kai had moved the code from the single chain into a more general handler --- and this one (renamed to chaindisk) is used now. I optimized the code a bit and brought --- it in sycn with the other code. Hopefully I didn't introduce errors. Note: this somewhat --- complex approach is meant for fonts that implement (for instance) ligatures by character --- replacement which to some extend is not that suitable for hyphenation. I also use some --- helpers. This method passes some states but reparses the list. There is room for a bit of --- speed up but that will be done in the context version. (In fact a partial rewrite of all --- code can bring some more efficientry.) +-- In the meantime Kai had moved the code from the single chain into a more general +-- handler and this one (renamed to chaindisk) is used now. I optimized the code a +-- bit and brought it in sycn with the other code. Hopefully I didn't introduce +-- errors. Note: this somewhat complex approach is meant for fonts that implement +-- (for instance) ligatures by character replacement which to some extend is not +-- that suitable for hyphenation. I also use some helpers. This method passes some +-- states but reparses the list. There is room for a bit of speed up but that will +-- be done in the context version. (In fact a partial rewrite of all code can bring +-- some more efficiency.) -- --- I didn't test it with extremes but successive disc nodes still can give issues but in --- order to handle that we need more complex code which also slows down even more. The main --- loop variant could deal with that: test, collapse, backtrack. +-- I didn't test it with extremes but successive disc nodes still can give issues +-- but in order to handle that we need more complex code which also slows down even +-- more. The main loop variant could deal with that: test, collapse, backtrack. + +local userkern = nuts.pool and nuts.pool.newkern -- context + +do if not userkern then -- generic + + local thekern = nuts.new("kern",1) -- userkern + local setkern = nuts.setkern -- not injections.setkern + + userkern = function(k) + local n = copy_node(thekern) + setkern(n,k) + return n + end -local new_kern = nuts.pool.kern +end end local function checked(head) local current = head while current do if getid(current) == glue_code then - local kern = new_kern(getfield(current,"width")) + local kern = userkern(getwidth(current)) if head == current then local next = getnext(current) if next then @@ -1785,8 +1884,7 @@ local function checked(head) current = next else local prev, next = getboth(current) - setlink(prev,kern) - setlink(kern,next) + setlink(prev,kern,next) flush_node(current) current = next end @@ -1804,7 +1902,131 @@ local function setdiscchecked(d,pre,post,replace) setdisc(d,pre,post,replace) end -local function chaindisk(head,start,last,dataset,sequence,chainlookup,rlmode,k,ck,chainproc) +local noflags = { false, false, false, false } + +local function chainrun(head,start,last,dataset,sequence,rlmode,skiphash,ck) + + local size = ck[5] - ck[4] + 1 + local chainlookups = ck[6] + local done = false + + -- current match + if chainlookups then + -- Lookups can be like { 1, false, 3 } or { false, 2 } or basically anything and + -- #lookups can be less than #current + + if size == 1 then + + -- if nofchainlookups > size then + -- -- bad rules + -- end + + local chainlookup = chainlookups[1] + for j=1,#chainlookup do + local chainstep = chainlookup[j] + local chainkind = chainstep.type + local chainproc = chainprocs[chainkind] + if chainproc then + local ok + head, start, ok = chainproc(head,start,last,dataset,sequence,chainstep,rlmode,skiphash) + if ok then + done = true + end + else + logprocess("%s: %s is not yet supported (1)",cref(dataset,sequence),chainkind) + end + end + + else + + -- See LookupType 5: Contextual Substitution Subtable. Now it becomes messy. The + -- easiest case is where #current maps on #lookups i.e. one-to-one. But what if + -- we have a ligature. Cf the spec we then need to advance one character but we + -- really need to test it as there are fonts out there that are fuzzy and have + -- too many lookups: + -- + -- U+1105 U+119E U+1105 U+119E : sourcehansansklight: script=hang ccmp=yes + -- + -- Even worse are these family emoji shapes as they can have multiple lookups + -- per slot (probably only for gpos). + + -- It's very unlikely that we will have skip classes here but still ... we seldom + -- enter this branch anyway. + + local i = 1 + local laststart = start + local nofchainlookups = #chainlookups -- useful? + while start do + if skiphash then -- hm, so we know we skip some + while start do + local char = ischar(start,currentfont) + if char then + if skiphash and skiphash[char] then + start = getnext(start) + else + break + end + else + break + end + end + end + local chainlookup = chainlookups[i] + if chainlookup then + for j=1,#chainlookup do + local chainstep = chainlookup[j] + local chainkind = chainstep.type + local chainproc = chainprocs[chainkind] + if chainproc then + local ok, n + head, start, ok, n = chainproc(head,start,last,dataset,sequence,chainstep,rlmode,skiphash,i) + -- messy since last can be changed ! + if ok then + done = true + if n and n > 1 and i + n > nofchainlookups then + -- this is a safeguard, we just ignore the rest of the lookups + i = size -- prevents an advance + break + end + end + else + -- actually an error + logprocess("%s: %s is not yet supported (2)",cref(dataset,sequence),chainkind) + end + end + else + -- we skip but we could also delete as option .. what does an empty lookup actually mean + -- in opentype ... anyway, we could map it onto gsub_remove if needed + end + i = i + 1 + if i > size or not start then + break + elseif start then + laststart = start + start = getnext(start) + end + end + if not start then + start = laststart + end + + end + else + -- todo: needs checking for holes in the replacements + local replacements = ck[7] + if replacements then + head, start, done = reversesub(head,start,last,dataset,sequence,replacements,rlmode,skiphash) + else + done = true + if trace_contexts then + logprocess("%s: skipping match",cref(dataset,sequence)) + end + end + end + return head, start, done +end + +local function chaindisk(head,start,dataset,sequence,rlmode,skiphash,ck) if not start then return head, start, false @@ -1819,7 +2041,6 @@ local function chaindisk(head,start,last,dataset,sequence,chainlookup,rlmode,k,c local sweepnode = sweepnode local sweeptype = sweeptype local sweepoverflow = false - local checkdisc = getprev(head) -- hm bad name head local keepdisc = not sweepnode local lookaheaddisc = nil local backtrackdisc = nil @@ -1845,19 +2066,23 @@ local function chaindisk(head,start,last,dataset,sequence,chainlookup,rlmode,k,c hasglue = true elseif id == disc_code then if keepdisc then - keepdisc = false - if notmatchpre[current] ~= notmatchreplace[current] then - lookaheaddisc = current - end + keepdisc = false + lookaheaddisc = current local replace = getfield(current,"replace") - while replace and i <= l do - if getid(replace) == glyph_code then - i = i + 1 + if not replace then + sweepoverflow = true + sweepnode = current + current = getnext(current) + else + while replace and i <= l do + if getid(replace) == glyph_code then + i = i + 1 + end + replace = getnext(replace) end - replace = getnext(replace) + current = getnext(replace) end - last = current - current = getnext(c) + last = current else head, current = flattendisk(head,current) end @@ -1868,7 +2093,7 @@ local function chaindisk(head,start,last,dataset,sequence,chainlookup,rlmode,k,c if current then -- go on elseif sweepoverflow then - -- we already are folling up on sweepnode + -- we already are following up on sweepnode break elseif sweeptype == "post" or sweeptype == "replace" then current = getnext(sweepnode) @@ -1919,7 +2144,8 @@ local function chaindisk(head,start,last,dataset,sequence,chainlookup,rlmode,k,c if notmatchpre[current] ~= notmatchreplace[current] then lookaheaddisc = current end - local replace = getfield(c,"replace") + -- we assume a simple text only replace (we could use nuts.count) + local replace = getfield(current,"replace") while replace and i < s do if getid(replace) == glyph_code then i = i + 1 @@ -1964,6 +2190,7 @@ local function chaindisk(head,start,last,dataset,sequence,chainlookup,rlmode,k,c if notmatchpost[current] ~= notmatchreplace[current] then backtrackdisc = current end + -- we assume a simple text only replace (we could use nuts.count) local replace = getfield(current,"replace") while replace and i > 1 do if getid(replace) == glyph_code then @@ -1981,8 +2208,8 @@ local function chaindisk(head,start,last,dataset,sequence,chainlookup,rlmode,k,c end end end + local done = false - local ok = false if lookaheaddisc then local cf = start @@ -2001,38 +2228,44 @@ local function chaindisk(head,start,last,dataset,sequence,chainlookup,rlmode,k,c break end end - - setprev(lookaheaddisc,cprev) - if cprev then - setnext(cprev,lookaheaddisc) - end + setlink(cprev,lookaheaddisc) setprev(cf) setnext(cl) if startishead then head = lookaheaddisc end local pre, post, replace = getdisc(lookaheaddisc) - local new = copy_node_list(cf) + local new = copy_node_list(cf) -- br, how often does that happen local cnew = new + if pre then + setlink(find_node_tail(cf),pre) + end + if replace then + local tail = find_node_tail(new) + setlink(tail,replace) + end for i=1,insertedmarks do cnew = getnext(cnew) end + cl = start local clast = cnew for i=f,l do + cl = getnext(cl) clast = getnext(clast) end if not notmatchpre[lookaheaddisc] then - cf, start, ok = chainproc(cf,start,last,dataset,sequence,chainlookup,rlmode,k) + local ok = false + cf, start, ok = chainrun(cf,start,cl,dataset,sequence,rlmode,skiphash,ck) + if ok then + done = true + end end if not notmatchreplace[lookaheaddisc] then - new, cnew, ok = chainproc(new,cnew,clast,dataset,sequence,chainlookup,rlmode,k) - end - if pre then - setlink(cl,pre) - end - if replace then - local tail = find_node_tail(new) - setlink(tail,replace) + local ok = false + new, cnew, ok = chainrun(new,cnew,clast,dataset,sequence,rlmode,skiphash,ck) + if ok then + done = true + end end if hasglue then setdiscchecked(lookaheaddisc,cf,post,new) @@ -2040,8 +2273,8 @@ local function chaindisk(head,start,last,dataset,sequence,chainlookup,rlmode,k,c setdisc(lookaheaddisc,cf,post,new) end start = getprev(lookaheaddisc) - sweephead[cf] = getnext(clast) - sweephead[new] = getnext(last) + sweephead[cf] = getnext(clast) or false + sweephead[new] = getnext(cl) or false elseif backtrackdisc then @@ -2060,10 +2293,7 @@ local function chaindisk(head,start,last,dataset,sequence,chainlookup,rlmode,k,c break end end - if cnext then - setprev(cnext,backtrackdisc) - end - setnext(backtrackdisc,cnext) + setlink(backtrackdisc,cnext) setprev(cf) setnext(cl) local pre, post, replace, pretail, posttail, replacetail = getdisc(backtrackdisc,true) @@ -2077,10 +2307,18 @@ local function chaindisk(head,start,last,dataset,sequence,chainlookup,rlmode,k,c clast = getnext(clast) end if not notmatchpost[backtrackdisc] then - cf, start, ok = chainproc(cf,start,last,dataset,sequence,chainlookup,rlmode,k) + local ok = false + cf, start, ok = chainrun(cf,start,last,dataset,sequence,rlmode,skiphash,ck) + if ok then + done = true + end end if not notmatchreplace[backtrackdisc] then - new, cnew, ok = chainproc(new,cnew,clast,dataset,sequence,chainlookup,rlmode,k) + local ok = false + new, cnew, ok = chainrun(new,cnew,clast,dataset,sequence,rlmode,skiphash,ck) + if ok then + done = true + end end if post then setlink(posttail,cf) @@ -2098,539 +2336,486 @@ local function chaindisk(head,start,last,dataset,sequence,chainlookup,rlmode,k,c setdisc(backtrackdisc,pre,post,replace) end start = getprev(backtrackdisc) - sweephead[post] = getnext(clast) - sweephead[replace] = getnext(last) + sweephead[post] = getnext(clast) or false + sweephead[replace] = getnext(last) or false else - head, start, ok = chainproc(head,start,last,dataset,sequence,chainlookup,rlmode,k) + local ok = false + head, start, ok = chainrun(head,start,last,dataset,sequence,rlmode,skiphash,ck) + if ok then + done = true + end end - return head, start, ok + return head, start, done end -local noflags = { false, false, false, false } +local function chaintrac(head,start,dataset,sequence,rlmode,skiphash,ck,match,discseen,sweepnode) + local rule = ck[1] + local lookuptype = ck[8] or ck[2] + local nofseq = #ck[3] + local first = ck[4] + local last = ck[5] + local char = getchar(start) + logwarning("%s: rule %s %s at char %s for (%s,%s,%s) chars, lookuptype %a, %sdisc seen, %ssweeping", + cref(dataset,sequence),rule,match and "matches" or "nomatch", + gref(char),first-1,last-first+1,nofseq-last,lookuptype, + discseen and "" or "no ", sweepnode and "" or "not ") +end + +-- The next one is quite optimized but still somewhat slow, fonts like ebgaramond +-- are real torture tests because they have many steps with one context (having +-- multiple contexts makes more sense) also because we (can) reduce them. Instead of +-- a match boolean variable and check for that I decided to use a goto with labels +-- instead. This is one of the cases where it makes the code more readable and we +-- might even gain a bit performance. -local function handle_contextchain(head,start,dataset,sequence,contexts,rlmode) +-- when we have less replacements (lookups) then current matches we can push too much into +-- the previous disc .. such be it (<before><disc><current=fl><after> with only f done) + +local function handle_contextchain(head,start,dataset,sequence,contexts,rlmode,skiphash) + -- optimizing for rlmode gains nothing local sweepnode = sweepnode local sweeptype = sweeptype + local postreplace + local prereplace + local checkdisc + local discseen -- = false + if sweeptype then + if sweeptype == "replace" then + postreplace = true + prereplace = true + else + postreplace = sweeptype == "post" + prereplace = sweeptype == "pre" + end + checkdisc = getprev(head) + end local currentfont = currentfont - local diskseen = false - local checkdisc = getprev(head) - local flags = sequence.flags or noflags - local done = false - local skipmark = flags[1] - local skipligature = flags[2] - local skipbase = flags[3] - local markclass = sequence.markclass - local skipped = false - for k=1,#contexts do -- i've only seen ccmp having > 1 (e.g. dejavu) - local match = true + + local skipped -- = false + + local startprev, + startnext = getboth(start) + local done -- = false + + -- we can have multiple hits and as we scan (currently) all we need to check + -- if we have a match ... contextchains have no real coverage table (with + -- unique entries) + + -- fonts can have many steps (each doing one check) or many contexts + + -- todo: make a per-char cache so that we have small contexts (when we have a context + -- n == 1 and otherwise it can be more so we can even distingish n == 1 or more) + + local nofcontexts = contexts.n -- #contexts + + local startchar = nofcontext == 1 or ischar(start,currentfont) -- only needed in a chain + + for k=1,nofcontexts do -- does this disc mess work well with n > 1 + + local ck = contexts[k] + local seq = ck[3] + local f = ck[4] -- first current + if not startchar or not seq[f][startchar] then + -- report("no hit in %a at %i of %i contexts",sequence.type,k,nofcontexts) + goto next + end + local s = seq.n -- or #seq + local l = ck[5] -- last current local current = start local last = start - local ck = contexts[k] - local seq = ck[3] - local s = #seq - local size = 1 - -- f..l = mid string - if s == 1 then - -- this seldom happens as it makes no sense (bril, ebgaramond, husayni, minion) - local char = ischar(current,currentfont) - if char then - match = seq[1][char] - end - else - -- maybe we need a better space check (maybe check for glue or category or combination) - -- we cannot optimize for n=2 because there can be disc nodes - local f = ck[4] - local l = ck[5] - -- current match - size = l - f + 1 - if size > 1 then - -- before/current/after | before/current | current/after - local discfound = nil - local n = f + 1 - last = getnext(last) -- the second in current (first already matched) - while n <= l do - if not last and (sweeptype == "post" or sweeptype == "replace") then - last = getnext(sweepnode) - sweeptype = nil - end - if last then - local char, id = ischar(last,currentfont) - if char then - local ccd = descriptions[char] - if ccd then - local class = ccd.class or "base" - if class == skipmark or class == skipligature or class == skipbase or (markclass and class == "mark" and not markclass[char]) then - skipped = true - if trace_skips then - show_skip(dataset,sequence,char,ck,class) - end - last = getnext(last) - elseif seq[n][char] then - if n < l then - last = getnext(last) - end - n = n + 1 - else - if discfound then - notmatchreplace[discfound] = true - match = not notmatchpre[discfound] - else - match = false - end - break - end + + -- current match + + if l > f then + -- before/current/after | before/current | current/after + local discfound -- = nil + local n = f + 1 + last = startnext -- the second in current (first already matched) + while n <= l do + if postreplace and not last then + last = getnext(sweepnode) + sweeptype = nil + end + if last then + local char, id = ischar(last,currentfont) + if char then + if skiphash and skiphash[char] then + skipped = true + if trace_skips then + show_skip(dataset,sequence,char,ck,classes[char]) + end + last = getnext(last) + elseif seq[n][char] then + if n < l then + last = getnext(last) + end + n = n + 1 + elseif discfound then + notmatchreplace[discfound] = true + if notmatchpre[discfound] then + goto next else - if discfound then - notmatchreplace[discfound] = true - match = not notmatchpre[discfound] - else - match = false - end break end - elseif char == false then - if discfound then - notmatchreplace[discfound] = true - match = not notmatchpre[discfound] + else + goto next + end + elseif char == false then + if discfound then + notmatchreplace[discfound] = true + if notmatchpre[discfound] then + goto next else - match = false + break end - break - elseif id == disc_code then - diskseen = true - discfound = last - notmatchpre[last] = nil - notmatchpost[last] = true - notmatchreplace[last] = nil - local pre, post, replace = getdisc(last) - if pre then - local n = n - while pre do - if seq[n][getchar(pre)] then - n = n + 1 - pre = getnext(pre) - if n > l then - break - end - else - notmatchpre[last] = true + else + goto next + end + elseif id == disc_code then + -- elseif id == disc_code and (not discs or discs[last]) then + discseen = true + discfound = last + notmatchpre[last] = nil + notmatchpost[last] = true + notmatchreplace[last] = nil + local pre, post, replace = getdisc(last) + if pre then + local n = n + while pre do + if seq[n][getchar(pre)] then + n = n + 1 + if n > l then break end - end - if n <= l then + pre = getnext(pre) + else notmatchpre[last] = true + break end - else - notmatchpre[last] = true end - if replace then - -- so far we never entered this branch - while replace do - if seq[n][getchar(replace)] then - n = n + 1 - replace = getnext(replace) - if n > l then - break - end + -- commented, for Kai to check + -- if n <= l then + -- notmatchpre[last] = true + -- end + else + notmatchpre[last] = true + end + if replace then + -- so far we never entered this branch + while replace do + if seq[n][getchar(replace)] then + n = n + 1 + if n > l then + break + end + replace = getnext(replace) + else + notmatchreplace[last] = true + if notmatchpre[last] then + goto next else - notmatchreplace[last] = true - match = not notmatchpre[last] break end end - match = not notmatchpre[last] end - -- maybe only if match - last = getnext(last) - else - match = false - break + -- why here again + if notmatchpre[last] then + goto next + end end + -- maybe only if match + last = getnext(last) else - match = false - break + goto next end + else + goto next end end - -- before - if match and f > 1 then - local prev = getprev(start) + end + + -- before + + if f > 1 then + if startprev then + local prev = startprev + if prereplace and prev == checkdisc then + prev = getprev(sweepnode) + end if prev then - if prev == checkdisc and (sweeptype == "pre" or sweeptype == "replace") then - prev = getprev(sweepnode) - -- sweeptype = nil - end - if prev then - local discfound = nil - local n = f - 1 - while n >= 1 do - if prev then - local char, id = ischar(prev,currentfont) - if char then - local ccd = descriptions[char] - if ccd then - local class = ccd.class - if class == skipmark or class == skipligature or class == skipbase or (markclass and class == "mark" and not markclass[char]) then - skipped = true - if trace_skips then - show_skip(dataset,sequence,char,ck,class) - end - prev = getprev(prev) -- moved here - elseif seq[n][char] then - if n > 1 then -- new test - prev = getprev(prev) -- moved here - end - n = n - 1 - else - if discfound then - notmatchreplace[discfound] = true - match = not notmatchpost[discfound] - else - match = false - end - break - end + local discfound -- = nil + local n = f - 1 + while n >= 1 do + if prev then + local char, id = ischar(prev,currentfont) + if char then + if skiphash and skiphash[char] then + skipped = true + if trace_skips then + show_skip(dataset,sequence,char,ck,classes[char]) + end + prev = getprev(prev) + elseif seq[n][char] then + if n > 1 then + prev = getprev(prev) + end + n = n - 1 + elseif discfound then + notmatchreplace[discfound] = true + if notmatchpost[discfound] then + goto next else - if discfound then - notmatchreplace[discfound] = true - match = not notmatchpost[discfound] - else - match = false - end break end - -- prev = getprev(prev) -- moved up - elseif char == false then - if discfound then - notmatchreplace[discfound] = true - match = not notmatchpost[discfound] - else - match = false + else + goto next + end + elseif char == false then + if discfound then + notmatchreplace[discfound] = true + if notmatchpost[discfound] then + goto next end - break - elseif id == disc_code then - -- the special case: f i where i becomes dottless i .. - diskseen = true - discfound = prev - notmatchpre[prev] = true - notmatchpost[prev] = nil - notmatchreplace[prev] = nil - local pre, post, replace, pretail, posttail, replacetail = getdisc(prev,true) - if pre ~= start and post ~= start and replace ~= start then - if post then - local n = n - while posttail do - if seq[n][getchar(posttail)] then - n = n - 1 - if posttail == post then - break - else - posttail = getprev(posttail) - if n < 1 then - break - end - end - else - notmatchpost[prev] = true + else + goto next + end + break + elseif id == disc_code then + -- elseif id == disc_code and (not discs or discs[prev]) then + -- the special case: f i where i becomes dottless i .. + discseen = true + discfound = prev + notmatchpre[prev] = true + notmatchpost[prev] = nil + notmatchreplace[prev] = nil + local pre, post, replace, pretail, posttail, replacetail = getdisc(prev,true) + -- weird test: needs checking + if pre ~= start and post ~= start and replace ~= start then + if post then + local n = n + while posttail do + if seq[n][getchar(posttail)] then + n = n - 1 + if posttail == post or n < 1 then break + else + posttail = getprev(posttail) end - end - if n >= 1 then + else notmatchpost[prev] = true + break end - else + end + if n >= 1 then notmatchpost[prev] = true end - if replace then - -- we seldom enter this branch (e.g. on brill efficient) - while replacetail do - if seq[n][getchar(replacetail)] then - n = n - 1 - if replacetail == replace then - break - else - replacetail = getprev(replacetail) - if n < 1 then - break - end - end + else + notmatchpost[prev] = true + end + if replace then + -- we seldom enter this branch (e.g. on brill efficient) + while replacetail do + if seq[n][getchar(replacetail)] then + n = n - 1 + if replacetail == replace or n < 1 then + break + else + replacetail = getprev(replacetail) + end + else + notmatchreplace[prev] = true + if notmatchpost[prev] then + goto next else - notmatchreplace[prev] = true - match = not notmatchpost[prev] break end end - if not match then - break - end end + else + notmatchreplace[prev] = true -- new, for Kai to check end - -- maybe only if match - prev = getprev(prev) - elseif seq[n][32] and isspace(prev,threshold) then + end + prev = getprev(prev) + -- elseif id == glue_code and seq[n][32] and isspace(prev,threshold,id) then + -- elseif seq[n][32] and spaces[prev] then + -- n = n - 1 + -- prev = getprev(prev) + elseif id == glue_code then + local sn = seq[n] + if (sn[32] and spaces[prev]) or sn[0xFFFC] then n = n - 1 prev = getprev(prev) else - match = false - break + goto next end + elseif seq[n][0xFFFC] then + n = n - 1 + prev = getprev(prev) else - match = false - break + goto next end + else + goto next end - else - match = false end else - match = false + goto next end + else + goto next end - -- after - if match and s > l then - local current = last and getnext(last) - if not current then - if sweeptype == "post" or sweeptype == "replace" then - current = getnext(sweepnode) - -- sweeptype = nil - end - end - if current then - local discfound = nil - -- removed optimization for s-l == 1, we have to deal with marks anyway - local n = l + 1 - while n <= s do - if current then - local char, id = ischar(current,currentfont) - if char then - local ccd = descriptions[char] - if ccd then - local class = ccd.class - if class == skipmark or class == skipligature or class == skipbase or (markclass and class == "mark" and not markclass[char]) then - skipped = true - if trace_skips then - show_skip(dataset,sequence,char,ck,class) - end - current = getnext(current) -- was absent - elseif seq[n][char] then - if n < s then -- new test - current = getnext(current) -- was absent - end - n = n + 1 - else - if discfound then - notmatchreplace[discfound] = true - match = not notmatchpre[discfound] - else - match = false - end - break - end + end + + -- after + + if s > l then + local current = last and getnext(last) + if not current and postreplace then + current = getnext(sweepnode) + end + if current then + local discfound -- = nil + local n = l + 1 + while n <= s do + if current then + local char, id = ischar(current,currentfont) + if char then + if skiphash and skiphash[char] then + skipped = true + if trace_skips then + show_skip(dataset,sequence,char,ck,classes[char]) + end + current = getnext(current) -- was absent + elseif seq[n][char] then + if n < s then -- new test + current = getnext(current) -- was absent + end + n = n + 1 + elseif discfound then + notmatchreplace[discfound] = true + if notmatchpre[discfound] then + goto next else - if discfound then - notmatchreplace[discfound] = true - match = not notmatchpre[discfound] - else - match = false - end break end - elseif char == false then - if discfound then - notmatchreplace[discfound] = true - match = not notmatchpre[discfound] + else + goto next + end + elseif char == false then + if discfound then + notmatchreplace[discfound] = true + if notmatchpre[discfound] then + goto next else - match = false + break end - break - elseif id == disc_code then - diskseen = true - discfound = current - notmatchpre[current] = nil - notmatchpost[current] = true - notmatchreplace[current] = nil - local pre, post, replace = getdisc(current) - if pre then - local n = n - while pre do - if seq[n][getchar(pre)] then - n = n + 1 - pre = getnext(pre) - if n > s then - break - end - else - notmatchpre[current] = true + else + goto next + end + elseif id == disc_code then + -- elseif id == disc_code and (not discs or discs[current]) then + discseen = true + discfound = current + notmatchpre[current] = nil + notmatchpost[current] = true + notmatchreplace[current] = nil + local pre, post, replace = getdisc(current) + if pre then + local n = n + while pre do + if seq[n][getchar(pre)] then + n = n + 1 + if n > s then break + else + pre = getnext(pre) end - end - if n <= s then + else notmatchpre[current] = true + break end - else + end + if n <= s then notmatchpre[current] = true end - if replace then - -- so far we never entered this branch - while replace do - if seq[n][getchar(replace)] then - n = n + 1 + else + notmatchpre[current] = true + end + if replace then + -- so far we never entered this branch + while replace do + if seq[n][getchar(replace)] then + n = n + 1 + if n > s then + break + else replace = getnext(replace) - if n > s then - break - end + end + else + notmatchreplace[current] = true + if notmatchpre[current] then + goto next else - notmatchreplace[current] = true - match = notmatchpre[current] break end end - if not match then - break - end - else - -- skip 'm end - -- maybe only if match - current = getnext(current) - elseif seq[n][32] and isspace(current,threshold) then + else + notmatchreplace[current] = true -- new, for Kai to check + end + current = getnext(current) + elseif id == glue_code then + local sn = seq[n] + if (sn[32] and spaces[current]) or sn[0xFFFC] then n = n + 1 current = getnext(current) else - match = false - break + goto next end + elseif seq[n][0xFFFC] then + n = n + 1 + current = getnext(current) else - match = false - break - end - end - else - match = false - end - end - end - if match then - -- Can lookups be of a different type? - local diskchain = diskseen or sweepnode - if trace_contexts then - local rule = ck[1] - local lookuptype = ck[8] or ck[2] - local first = ck[4] - local last = ck[5] - local char = getchar(start) - logwarning("%s: rule %s matches at char %s for (%s,%s,%s) chars, lookuptype %a", - cref(dataset,sequence),rule,gref(char),first-1,last-first+1,s-last,lookuptype) - end - local chainlookups = ck[6] - if chainlookups then - local nofchainlookups = #chainlookups - -- Lookups can be like { 1, false, 3 } or { false, 2 } or basically anything and - -- #lookups can be less than #current - if size == 1 then - -- if nofchainlookups > size then - -- -- bad rules - -- end - local chainlookup = chainlookups[1] - local chainkind = chainlookup.type - local chainproc = chainprocs[chainkind] - if chainproc then - local ok - if diskchain then - head, start, ok = chaindisk(head,start,last,dataset,sequence,chainlookup,rlmode,1,ck,chainproc) - else - head, start, ok = chainproc(head,start,last,dataset,sequence,chainlookup,rlmode,1) - end - if ok then - done = true + goto next end else - logprocess("%s: %s is not yet supported (1)",cref(dataset,sequence),chainkind) - end - else - -- See LookupType 5: Contextual Substitution Subtable. Now it becomes messy. The - -- easiest case is where #current maps on #lookups i.e. one-to-one. But what if - -- we have a ligature. Cf the spec we then need to advance one character but we - -- really need to test it as there are fonts out there that are fuzzy and have - -- too many lookups: - -- - -- U+1105 U+119E U+1105 U+119E : sourcehansansklight: script=hang ccmp=yes - -- - local i = 1 - while start do - if skipped then - while start do -- todo: use properties - local char = getchar(start) - local ccd = descriptions[char] - if ccd then - local class = ccd.class or "base" - if class == skipmark or class == skipligature or class == skipbase or (markclass and class == "mark" and not markclass[char]) then - start = getnext(start) - else - break - end - else - break - end - end - end - local chainlookup = chainlookups[i] - if chainlookup then - local chainkind = chainlookup.type - local chainproc = chainprocs[chainkind] - if chainproc then - local ok, n - if diskchain then - head, start, ok = chaindisk(head,start,last,dataset,sequence,chainlookup,rlmode,i,ck,chainproc) - else - head, start, ok, n = chainproc(head,start,last,dataset,sequence,chainlookup,rlmode,i) - end - -- messy since last can be changed ! - if ok then - done = true - if n and n > 1 and i + n > nofchainlookups then - -- this is a safeguard, we just ignore the rest of the lookups - break - end - end - else - -- actually an error - logprocess("%s: %s is not yet supported (2)",cref(dataset,sequence),chainkind) - end - end - i = i + 1 - if i > size or not start then - break - elseif start then - start = getnext(start) - end + goto next end end else - -- todo: needs checking for holes in the replacements - local replacements = ck[7] - if replacements then - head, start, done = reversesub(head,start,last,dataset,sequence,replacements,rlmode) - else - done = true - if trace_contexts then - logprocess("%s: skipping match",cref(dataset,sequence)) - end - end - end - if done then - break -- out of contexts (new, needs checking) + goto next end end + + if trace_contexts then + chaintrac(head,start,dataset,sequence,rlmode,skipped and skiphash,ck,true,discseen,sweepnode) + end + if discseen or sweepnode then + head, start, done = chaindisk(head,start,dataset,sequence,rlmode,skipped and skiphash,ck) + else + head, start, done = chainrun(head,start,last,dataset,sequence,rlmode,skipped and skiphash,ck) + end + if done then + break + -- else + -- next context + end + ::next:: + -- if trace_chains then + -- chaintrac(head,start,dataset,sequence,rlmode,skipped and skiphash,ck,false,discseen,sweepnode) + -- end end - if diskseen then + if discseen then notmatchpre = { } notmatchpost = { } notmatchreplace = { } + -- notmatchpre = { a = 1, b = 1 } notmatchpre .a = nil notmatchpre .b = nil + -- notmatchpost = { a = 1, b = 1 } notmatchpost .a = nil notmatchpost .b = nil + -- notmatchreplace = { a = 1, b = 1 } notmatchreplace.a = nil notmatchreplace.b = nil end return head, start, done end @@ -2643,13 +2828,19 @@ handlers.gpos_context = handle_contextchain -- this needs testing -local function chained_contextchain(head,start,stop,dataset,sequence,currentlookup,rlmode) +local function chained_contextchain(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash) local steps = currentlookup.steps local nofsteps = currentlookup.nofsteps if nofsteps > 1 then reportmoresteps(dataset,sequence) end - return handle_contextchain(head,start,dataset,sequence,currentlookup,rlmode) + -- probably wrong + local l = steps[1].coverage[getchar(start)] + if l then + return handle_contextchain(head,start,dataset,sequence,l,rlmode,skiphash) + else + return head, start, false + end end chainprocs.gsub_context = chained_contextchain @@ -2658,28 +2849,40 @@ chainprocs.gsub_reversecontextchain = chained_contextchain chainprocs.gpos_contextchain = chained_contextchain chainprocs.gpos_context = chained_contextchain -local missing = setmetatableindex("table") +------------------------------ + +-- experiment (needs no handler in font-otc so not now): +-- +-- function otf.registerchainproc(name,f) +-- -- chainprocs[name] = f +-- chainprocs[name] = function(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash) +-- local done = currentlookup.nofsteps > 0 +-- if not done then +-- reportzerosteps(dataset,sequence) +-- else +-- head, start, done = f(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash) +-- if not head or not start then +-- reportbadsteps(dataset,sequence) +-- end +-- end +-- return head, start, done +-- end +-- end + +local missing = setmetatableindex("table") +local logwarning = report_process +local resolved = { } -- we only resolve a font,script,language pair once local function logprocess(...) if trace_steps then registermessage(...) + if trace_steps == "silent" then + return + end end report_process(...) end -local logwarning = report_process - -local function report_missing_coverage(dataset,sequence) - local t = missing[currentfont] - if not t[sequence] then - t[sequence] = true - logwarning("missing coverage for feature %a, lookup %a, type %a, font %a, name %a", - dataset[4],sequence.name,sequence.type,currentfont,tfmdata.properties.fullname) - end -end - -local resolved = { } -- we only resolve a font,script,language pair once - -- todo: pass all these 'locals' in a table local sequencelists = setmetatableindex(function(t,font) @@ -2693,81 +2896,96 @@ end) -- fonts.hashes.sequences = sequencelists -local autofeatures = fonts.analyzers.features -local featuretypes = otf.tables.featuretypes -local defaultscript = otf.features.checkeddefaultscript -local defaultlanguage = otf.features.checkeddefaultlanguage - -local function initialize(sequence,script,language,enabled,autoscript,autolanguage) - local features = sequence.features - if features then - local order = sequence.order - if order then - local featuretype = featuretypes[sequence.type or "unknown"] - for i=1,#order do - local kind = order[i] - local valid = enabled[kind] - if valid then - local scripts = features[kind] - local languages = scripts and ( - scripts[script] or - scripts[wildcard] or - (autoscript and defaultscript(featuretype,autoscript,scripts)) - ) - local enabled = languages and ( - languages[language] or - languages[wildcard] or - (autolanguage and defaultlanguage(featuretype,autolanguage,languages)) - ) - if enabled then - return { valid, autofeatures[kind] or false, sequence, kind } +do -- overcome local limit + + local autofeatures = fonts.analyzers.features + local featuretypes = otf.tables.featuretypes + local defaultscript = otf.features.checkeddefaultscript + local defaultlanguage = otf.features.checkeddefaultlanguage + + local wildcard = "*" + local default = "dflt" + + local function initialize(sequence,script,language,enabled,autoscript,autolanguage) + local features = sequence.features + if features then + local order = sequence.order + if order then + local featuretype = featuretypes[sequence.type or "unknown"] + for i=1,#order do + local kind = order[i] + local valid = enabled[kind] + if valid then + local scripts = features[kind] + local languages = scripts and ( + scripts[script] or + scripts[wildcard] or + (autoscript and defaultscript(featuretype,autoscript,scripts)) + ) + local enabled = languages and ( + languages[language] or + languages[wildcard] or + (autolanguage and defaultlanguage(featuretype,autolanguage,languages)) + ) + if enabled then + return { valid, autofeatures[kind] or false, sequence, kind } + end end end + else + -- can't happen end - else - -- can't happen end + return false end - return false -end -function otf.dataset(tfmdata,font) -- generic variant, overloaded in context - local shared = tfmdata.shared - local properties = tfmdata.properties - local language = properties.language or "dflt" - local script = properties.script or "dflt" - local enabled = shared.features - local autoscript = enabled and enabled.autoscript - local autolanguage = enabled and enabled.autolanguage - local res = resolved[font] - if not res then - res = { } - resolved[font] = res - end - local rs = res[script] - if not rs then - rs = { } - res[script] = rs - end - local rl = rs[language] - if not rl then - rl = { - -- indexed but we can also add specific data by key - } - rs[language] = rl - local sequences = tfmdata.resources.sequences - if sequences then - for s=1,#sequences do - local v = enabled and initialize(sequences[s],script,language,enabled,autoscript,autolanguage) - if v then - rl[#rl+1] = v + function otf.dataset(tfmdata,font) -- generic variant, overloaded in context + local shared = tfmdata.shared + local properties = tfmdata.properties + local language = properties.language or "dflt" + local script = properties.script or "dflt" + local enabled = shared.features + local autoscript = enabled and enabled.autoscript + local autolanguage = enabled and enabled.autolanguage + local res = resolved[font] + if not res then + res = { } + resolved[font] = res + end + local rs = res[script] + if not rs then + rs = { } + res[script] = rs + end + local rl = rs[language] + if not rl then + rl = { + -- indexed but we can also add specific data by key + } + rs[language] = rl + local sequences = tfmdata.resources.sequences + if sequences then + for s=1,#sequences do + local v = enabled and initialize(sequences[s],script,language,enabled,autoscript,autolanguage) + if v then + rl[#rl+1] = v + end end end end + return rl end - return rl + end +-- Functions like kernrun, comprun etc evolved over time and in the end look rather +-- complex. It's a bit of a compromis between extensive copying and creating subruns. +-- The logic has been improved a lot by Kai and Ivo who use complex fonts which +-- really helped to identify border cases on the one hand and get insight in the diverse +-- ways fonts implement features (not always that consistent and efficient). At the same +-- time I tried to keep the code relatively efficient so that the overhead in runtime +-- stays acceptable. + local function report_disc(what,n) report_run("%s: %s > %s",what,n,languages.serializediscretionary(n)) end @@ -2815,14 +3033,12 @@ local function kernrun(disc,k_run,font,attr,...) done = true end if prev then - local nest = getprev(pre) setlink(prev,pre) if k_run(prevmarks,"preinjections",pre,font,attr,...) then -- or prev? done = true end - setprev(pre,nest) --- setprev(pre) - setnext(prev,disc) + setprev(pre) + setlink(prev,disc) end end -- @@ -2836,7 +3052,7 @@ local function kernrun(disc,k_run,font,attr,...) done = true end setnext(posttail) - setprev(next,disc) + setlink(disc,next) end end -- @@ -2845,14 +3061,12 @@ local function kernrun(disc,k_run,font,attr,...) done = true end if prev then - local nest = getprev(replace) setlink(prev,replace) if k_run(prevmarks,"replaceinjections",replace,font,attr,...) then -- getnext(replace)) done = true end - setprev(replace,nest) --- setprev(replace) - setnext(prev,disc) + setprev(replace) + setlink(prev,disc) end if next then setlink(replacetail,next) @@ -2860,20 +3074,25 @@ local function kernrun(disc,k_run,font,attr,...) done = true end setnext(replacetail) - setprev(next,disc) + setlink(disc,next) end elseif prev and next then setlink(prev,next) if k_run(prevmarks,"emptyinjections",next,font,attr,...) then done = true end - setlink(prev,disc) - setlink(disc,next) + setlink(prev,disc,next) + end + if done and trace_testruns then + report_disc("done",disc) end return nextstart, done end -local function comprun(disc,c_run,...) +-- fonts like ebgaramond do ligatures this way (less efficient than e.g. dejavu which +-- will do the testrun variant) + +local function comprun(disc,c_run,...) -- vararg faster than the whole list if trace_compruns then report_disc("comp",disc) end @@ -2883,7 +3102,7 @@ local function comprun(disc,c_run,...) -- if pre then sweepnode = disc - sweeptype = "pre" -- in alternative code preinjections is uc_c_sed (also used then for proeprties, saves a variable) + sweeptype = "pre" -- in alternative code preinjections is used (also used then for properties, saves a variable) local new, done = c_run(pre,...) if done then pre = new @@ -2914,12 +3133,18 @@ local function comprun(disc,c_run,...) sweepnode = nil sweeptype = nil if renewed then + if trace_testruns then + report_disc("done",disc) + end setdisc(disc,pre,post,replace) end -- return getnext(disc), renewed end +-- if we can hyphenate in a lig then unlikely a lig so we +-- could have a option here to ignore lig + local function testrun(disc,t_run,c_run,...) if trace_testruns then report_disc("test",disc) @@ -2930,32 +3155,72 @@ local function testrun(disc,t_run,c_run,...) return end local pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true) - local done = false - if replace and prev then - -- this is a bit strange as we only do replace here and not post - -- anyway, we only look ahead ... the idea is that we discard a - -- disc when there is a ligature crossing the replace boundary - setlink(replacetail,next) - local ok, overflow = t_run(replace,next,...) - if ok and overflow then - -- so, we can have crossed the boundary - setfield(disc,"replace") - setlink(prev,replace) - -- setlink(replacetail,next) - setboth(disc) - flush_node_list(disc) - return replace, true -- restart .. tricky ! + local renewed = false + if (post or replace) and prev then + if post then + setlink(posttail,next) + else + post = next + end + if replace then + setlink(replacetail,next) + else + replace = next + end + local d_post = t_run(post,next,...) + local d_replace = t_run(replace,next,...) + if d_post > 0 or d_replace > 0 then + local d = d_replace > d_post and d_replace or d_post + local head = getnext(disc) -- is: next + local tail = head + for i=1,d do + local nx = getnext(tail) + local id = getid(nx) + if id == disc_code then + head, tail = flattendisk(head,nx) + elseif id == glyph_code then + tail = nx + else + -- we can have overrun into a glue + break + end + end + next = getnext(tail) + setnext(tail) + setprev(head) + local new = copy_node_list(head) + if posttail then + setlink(posttail,head) + else + post = head + end + if replacetail then + setlink(replacetail,new) + else + replace = new + end else -- we stay inside the disc - setnext(replacetail) - setprev(next,disc) + if posttail then + setnext(posttail) + else + post = nil + end + if replacetail then + setnext(replacetail) + else + replace = nil + end end + setlink(disc,next) -- pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true) end -- -- like comprun -- - local renewed = false + if trace_testruns then + report_disc("more",disc) + end -- if pre then sweepnode = disc @@ -2991,47 +3256,14 @@ local function testrun(disc,t_run,c_run,...) sweeptype = nil if renewed then setdisc(disc,pre,post,replace) - return next, true - else - return next, done + if trace_testruns then + report_disc("done",disc) + end end + -- next can have changed (copied list) + return getnext(disc), renewed end --- A discrun happens when we have a zwnj. We're gpossing so it is unlikely that --- there has been a match changing the character. Now, as we check again here --- the question is: why do we do this ... needs checking as drun seems useless --- ... maybe that code can go away - --- local function discrun(disc,drun,krun) --- local prev, next = getboth(disc) --- if trace_discruns then --- report_disc("disc",disc) --- end --- if next and prev then --- setnext(prev,next) --- -- setprev(next,prev) --- drun(prev) --- setnext(prev,disc) --- -- setprev(next,disc) --- end --- -- --- if krun then -- currently always false --- local pre = getfield(disc,"pre") --- if not pre then --- -- go on --- elseif prev then --- local nest = getprev(pre) --- setlink(prev,pre) --- krun(prev,"preinjections") --- setprev(pre,nest) --- setnext(prev,disc) --- else --- krun(pre,"preinjections") --- end --- end --- return next --- end - -- We can make some assumptions with respect to discretionaries. First of all it is very -- unlikely that some of the analysis related attributes applies. Then we can also assume -- that the ConTeXt specific dynamic attribute is different, although we do use explicit @@ -3059,24 +3291,28 @@ end local nesting = 0 -local function c_run_single(head,font,attr,lookupcache,step,dataset,sequence,rlmode,handler) +local function c_run_single(head,font,attr,lookupcache,step,dataset,sequence,rlmode,skiphash,handler) local done = false local sweep = sweephead[head] if sweep then start = sweep - sweephead[head] = nil + -- sweephead[head] = nil + sweephead[head] = false else start = head end while start do - local char = ischar(start,font) + local char, id = ischar(start,font) if char then - local a = attr and getattr(start,0) + local a -- happens often so no assignment is faster + if attr then + a = getattr(start,0) + end if not a or (a == attr) then local lookupmatch = lookupcache[char] if lookupmatch then local ok - head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,step,1) + head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step) if ok then done = true end @@ -3101,67 +3337,103 @@ local function c_run_single(head,font,attr,lookupcache,step,dataset,sequence,rlm return head, done end +-- only replace? + local function t_run_single(start,stop,font,attr,lookupcache) + local lastd = nil while start ~= stop do local char = ischar(start,font) if char then - local a = attr and getattr(start,0) + local a -- happens often so no assignment is faster + if attr then + a = getattr(start,0) + end + local startnext = getnext(start) if not a or (a == attr) then local lookupmatch = lookupcache[char] if lookupmatch then -- hm, hyphens can match (tlig) so we need to really check -- if we need more than ligatures we can outline the code and use functions - local s = getnext(start) + local s = startnext + local ss = nil + local sstop = s == stop + if not s then + s = ss + ss = nil + end + -- a bit weird: why multiple ... anyway we can't have a disc in a disc + -- how about post ... we can probably merge this into the while + while getid(s) == disc_code do + ss = getnext(s) + s = getfield(s,"replace") + if not s then + s = ss + ss = nil + end + end local l = nil local d = 0 while s do - if s == stop then - d = 1 - elseif d > 0 then - d = d + 1 - end - local lg = lookupmatch[getchar(s)] - if lg then - l = lg - s = getnext(s) + local char = ischar(s,font) + if char then + local lg = lookupmatch[char] + if lg then + if sstop then + d = 1 + elseif d > 0 then + d = d + 1 + end + l = lg + s = getnext(s) + sstop = s == stop + if not s then + s = ss + ss = nil + end + while getid(s) == disc_code do + ss = getnext(s) + s = getfield(s,"replace") + if not s then + s = ss + ss = nil + end + end + else + break + end else break end end - if l and l.ligature then - return true, d > 1 + if l and l.ligature then -- so we test for ligature + lastd = d end +-- why not: if not l then break elseif l.ligature then return d end + else +-- why not: break + -- no match (yet) end else -- go on can be a mixed one +-- why not: break end - start = getnext(start) + if lastd then + return lastd + end + start = startnext else break end end + return 0 end --- local function d_run_single(prev,font,attr,lookupcache,step,dataset,sequence,rlmode,handler) --- local a = attr and getattr(prev,0) --- if not a or (a == attr) then --- local char = ischar(prev) -- can be disc --- if char then --- local lookupmatch = lookupcache[char] --- if lookupmatch then --- local h, d, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,step,1) --- if ok then --- done = true --- success = true --- end --- end --- end --- end --- end - -local function k_run_single(sub,injection,last,font,attr,lookupcache,step,dataset,sequence,rlmode,handler) - local a = attr and getattr(sub,0) +local function k_run_single(sub,injection,last,font,attr,lookupcache,step,dataset,sequence,rlmode,skiphash,handler) + local a -- happens often so no assignment is faster + if attr then + a = getattr(sub,0) + end if not a or (a == attr) then - for n in traverse_nodes(sub) do -- only gpos + for n in nextnode, sub do -- only gpos if n == last then break end @@ -3169,7 +3441,7 @@ local function k_run_single(sub,injection,last,font,attr,lookupcache,step,datase if char then local lookupmatch = lookupcache[char] if lookupmatch then - local h, d, ok = handler(sub,n,dataset,sequence,lookupmatch,rlmode,step,1,injection) + local h, d, ok = handler(sub,n,dataset,sequence,lookupmatch,rlmode,skiphash,step,injection) if ok then return true end @@ -3179,39 +3451,39 @@ local function k_run_single(sub,injection,last,font,attr,lookupcache,step,datase end end -local function c_run_multiple(head,font,attr,steps,nofsteps,dataset,sequence,rlmode,handler) +local function c_run_multiple(head,font,attr,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler) local done = false local sweep = sweephead[head] if sweep then start = sweep - sweephead[head] = nil + -- sweephead[head] = nil + sweephead[head] = false else start = head end while start do local char = ischar(start,font) if char then - local a = attr and getattr(start,0) + local a -- happens often so no assignment is faster + if attr then + a = getattr(start,0) + end if not a or (a == attr) then for i=1,nofsteps do local step = steps[i] local lookupcache = step.coverage - if lookupcache then - local lookupmatch = lookupcache[char] - if lookupmatch then - -- we could move all code inline but that makes things even more unreadable - local ok - head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,step,i) - if ok then - done = true - break - elseif not start then - -- don't ask why ... shouldn't happen - break - end + local lookupmatch = lookupcache[char] + if lookupmatch then + -- we could move all code inline but that makes things even more unreadable + local ok + head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step) + if ok then + done = true + break + elseif not start then + -- don't ask why ... shouldn't happen + break end - else - report_missing_coverage(dataset,sequence) end end if start then @@ -3236,83 +3508,97 @@ local function c_run_multiple(head,font,attr,steps,nofsteps,dataset,sequence,rlm end local function t_run_multiple(start,stop,font,attr,steps,nofsteps) + local lastd = nil while start ~= stop do local char = ischar(start,font) if char then - local a = attr and getattr(start,0) + local a -- happens often so no assignment is faster + if attr then + a = getattr(start,0) + end + local startnext = getnext(start) if not a or (a == attr) then for i=1,nofsteps do local step = steps[i] local lookupcache = step.coverage - if lookupcache then - local lookupmatch = lookupcache[char] - if lookupmatch then - -- if we need more than ligatures we can outline the code and use functions - local s = getnext(start) - local l = nil - local d = 0 - while s do - if s == stop then - d = 1 - elseif d > 0 then - d = d + 1 - end - local lg = lookupmatch[getchar(s)] + local lookupmatch = lookupcache[char] + if lookupmatch then + -- if we need more than ligatures we can outline the code and use functions + local s = startnext + local ss = nil + local sstop = s == stop + if not s then + s = ss + ss = nil + end + while getid(s) == disc_code do + ss = getnext(s) + s = getfield(s,"replace") + if not s then + s = ss + ss = nil + end + end + local l = nil + local d = 0 + while s do + local char = ischar(s) + if char then + local lg = lookupmatch[char] if lg then + if sstop then + d = 1 + elseif d > 0 then + d = d + 1 + end l = lg s = getnext(s) + sstop = s == stop + if not s then + s = ss + ss = nil + end + while getid(s) == disc_code do + ss = getnext(s) + s = getfield(s,"replace") + if not s then + s = ss + ss = nil + end + end else break end - end - if l and l.ligature then - return true, d > 1 + else + break end end - else - report_missing_coverage(dataset,sequence) + if l and l.ligature then + lastd = d + end end end else -- go on can be a mixed one end - start = getnext(start) + if lastd then + return lastd + end + start = startnext else break end end + return 0 end --- local function d_run_multiple(prev,attr,steps,nofsteps,dataset,sequence,rlmode,handler) --- local a = attr and getattr(prev,0) --- if not a or (a == attr) then --- local char = ischar(prev) -- can be disc --- if char then --- for i=1,nofsteps do --- local step = steps[i] --- local lookupcache = step.coverage --- if lookupcache then --- local lookupmatch = lookupcache[char] --- if lookupmatch then --- -- we could move all code inline but that makes things even more unreadable --- local h, d, ok = handler(head,prev,dataset,sequence,lookupmatch,rlmode,step,i) --- if ok then --- done = true --- break --- end --- end --- else --- report_missing_coverage(dataset,sequence) --- end --- end --- end --- end --- end - -local function k_run_multiple(sub,injection,last,font,attr,steps,nofsteps,dataset,sequence,rlmode,handler) - local a = attr and getattr(sub,0) +local function k_run_multiple(sub,injection,last,font,attr,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler) + local a -- happens often so no assignment is faster + if attr then + a = getattr(sub,0) + end if not a or (a == attr) then - for n in traverse_nodes(sub) do -- only gpos + for n in nextnode, sub do -- only gpos if n == last then break end @@ -3321,16 +3607,12 @@ local function k_run_multiple(sub,injection,last,font,attr,steps,nofsteps,datase for i=1,nofsteps do local step = steps[i] local lookupcache = step.coverage - if lookupcache then - local lookupmatch = lookupcache[char] - if lookupmatch then - local h, d, ok = handler(head,n,dataset,sequence,lookupmatch,step,rlmode,i,injection) - if ok then - return true - end + local lookupmatch = lookupcache[char] + if lookupmatch then + local h, d, ok = handler(sub,n,dataset,sequence,lookupmatch,rlmode,skiphash,step,injection) -- sub was head + if ok then + return true end - else - report_missing_coverage(dataset,sequence) end end end @@ -3341,230 +3623,374 @@ end -- to be checked, nowadays we probably can assume properly matched directions -- so maybe we no longer need a stack +-- local function txtdirstate(start,stack,top,rlparmode) +-- local dir = getdir(start) +-- local new = 1 +-- if dir == "+TRT" then +-- top = top + 1 +-- stack[top] = dir +-- new = -1 +-- elseif dir == "+TLT" then +-- top = top + 1 +-- stack[top] = dir +-- elseif dir == "-TRT" or dir == "-TLT" then +-- if top == 1 then +-- top = 0 +-- new = rlparmode +-- else +-- top = top - 1 +-- if stack[top] == "+TRT" then +-- new = -1 +-- end +-- end +-- else +-- new = rlparmode +-- end +-- return getnext(start), top, new +-- end +-- +-- local function pardirstate(start) +-- local dir = getdir(start) +-- local new = 0 +-- if dir == "TLT" then +-- new = 1 +-- elseif dir == "TRT" then +-- new = -1 +-- end +-- return getnext(start), new, new +-- end + local function txtdirstate(start,stack,top,rlparmode) - local dir = getfield(start,"dir") - local new = 1 + local nxt = getnext(start) + local dir = getdir(start) if dir == "+TRT" then top = top + 1 stack[top] = dir - new = -1 + return nxt, top, -1 elseif dir == "+TLT" then top = top + 1 stack[top] = dir + return nxt, top, 1 elseif dir == "-TRT" or dir == "-TLT" then - top = top - 1 - if stack[top] == "+TRT" then - new = -1 + if top == 1 then + return nxt, 0, rlparmode + else + top = top - 1 + if stack[top] == "+TRT" then + return nxt, top, -1 + else + return nxt, top, 1 + end end else - new = rlparmode + return nxt, top, rlparmode end - if trace_directions then - report_process("directions after txtdir %a: parmode %a, txtmode %a, level %a",dir,mref(rlparmode),mref(new),top) - end - return getnext(start), top, new end local function pardirstate(start) - local dir = getfield(start,"dir") - local new = 0 + local nxt = getnext(start) + local dir = getdir(start) if dir == "TLT" then - new = 1 + return nxt, 1, 1 elseif dir == "TRT" then - new = -1 - end - if trace_directions then - report_process("directions after pardir %a: parmode %a",dir,mref(new)) + return nxt, -1, -1 + else + return nxt, 0, 0 end - return getnext(start), new, new end +-- -- this will become: +-- +-- local getdirection = nuts.getdirection +-- +-- local function txtdirstate1(start,stack,top,rlparmode) +-- local nxt = getnext(start) +-- local dir, sub = getdirection(start) +-- if sub then +-- if top == 1 then +-- return nxt, 0, rlparmode +-- elseif dir < 2 then +-- top = top - 1 +-- if stack[top] == 1 then +-- return nxt, top, -1 +-- else +-- return nxt, top, 1 +-- end +-- else +-- return nxt, top, rlparmode +-- end +-- elseif dir == 1 then +-- top = top + 1 +-- stack[top] = 1 +-- return nxt, top, -1 +-- elseif dir == 0 then +-- top = top + 1 +-- stack[top] = 0 +-- return nxt, top, 1 +-- else +-- return nxt, top, rlparmode +-- end +-- end +-- +-- local function pardirstate1(start) +-- local nxt = getnext(start) +-- local dir = getdirection(start) +-- if dir == 0 then +-- return nxt, 1, 1 +-- elseif dir == 1 then +-- return nxt, -1, -1 +-- else +-- return nxt, 0, 0 +-- end +-- end + otf.helpers = otf.helpers or { } otf.helpers.txtdirstate = txtdirstate otf.helpers.pardirstate = pardirstate -local function featuresprocessor(head,font,attr) +-- This is the main loop. We run over the node list dealing with a specific font. The +-- attribute is a context specific thing. We could work on sub start-stop ranges instead +-- but I wonder if there is that much speed gain (experiments showed that it made not +-- much sense) and we need to keep track of directions anyway. Also at some point I +-- want to play with font interactions and then we do need the full sweeps. Apart from +-- optimizations the principles of processing the features hasn't changed much since +-- the beginning. - local sequences = sequencelists[font] -- temp hack +do - if not sequencelists then - return head, false - end - - nesting = nesting + 1 - - if nesting == 1 then - - currentfont = font - tfmdata = fontdata[font] - descriptions = tfmdata.descriptions - characters = tfmdata.characters - marks = tfmdata.resources.marks - threshold, - factor = getthreshold(font) - checkmarks = tfmdata.properties.checkmarks - - elseif currentfont ~= font then + -- reference: + -- + -- local a = attr and getattr(start,0) + -- if a then + -- a = (a == attr) and (not attribute or getprop(start,a_state) == attribute) + -- else + -- a = not attribute or getprop(start,a_state) == attribute + -- end + -- + -- used: + -- + -- local a -- happens often so no assignment is faster + -- if attr then + -- if getattr(start,0) == attr and (not attribute or getprop(start,a_state) == attribute) then + -- a = true + -- end + -- elseif not attribute or getprop(start,a_state) == attribute then + -- a = true + -- end + + -- This is a measurable experimental speedup (only with hyphenated text and multiple + -- fonts per processor call), especially for fonts with lots of contextual lookups. + + local fastdisc = true + local testdics = false + + directives.register("otf.fastdisc",function(v) fastdisc = v end) -- normally enabled + + -- using a merged combined hash as first test saves some 30% on ebgaramond and + -- about 15% on arabtype .. then moving the a test also saves a bit (even when + -- often a is not set at all so that one is a bit debatable + + local otfdataset = nil -- todo: make an installer + + local getfastdisc = { __index = function(t,k) + local v = usesfont(k,currentfont) + t[k] = v + return v + end } + + local getfastspace = { __index = function(t,k) + -- we don't pass the id so that one can overload isspace + local v = isspace(k,threshold) or false + t[k] = v + return v + end } + + function otf.featuresprocessor(head,font,attr,direction,n) + + local sequences = sequencelists[font] -- temp hack + + nesting = nesting + 1 + + if nesting == 1 then + currentfont = font + tfmdata = fontdata[font] + descriptions = tfmdata.descriptions -- only needed in gref so we could pass node there instead + characters = tfmdata.characters -- but this branch is not entered that often anyway + local resources = tfmdata.resources + marks = resources.marks + classes = resources.classes + threshold, + factor = getthreshold(font) + checkmarks = tfmdata.properties.checkmarks + + if not otfdataset then + otfdataset = otf.dataset + end - report_warning("nested call with a different font, level %s, quitting",nesting) - nesting = nesting - 1 - return head, false + discs = fastdisc and n and n > 1 and setmetatable({},getfastdisc) -- maybe inline + spaces = setmetatable({},getfastspace) - end + elseif currentfont ~= font then - -- some 10% faster when no dynamics but hardly measureable on real runs .. but: it only - -- works when we have no other dynamics as otherwise the zero run will be applied to the - -- whole stream for which we then need to pass another variable which we won't + report_warning("nested call with a different font, level %s, quitting",nesting) + nesting = nesting - 1 + return head, false - -- if attr == 0 then - -- attr = false - -- end + end - head = tonut(head) + -- some 10% faster when no dynamics but hardly measureable on real runs .. but: it only + -- works when we have no other dynamics as otherwise the zero run will be applied to the + -- whole stream for which we then need to pass another variable which we won't - if trace_steps then - checkstep(head) - end + -- if attr == 0 then + -- attr = false + -- end - local rlmode = 0 + if trace_steps then + checkstep(head) + end - local done = false - local datasets = otf.dataset(tfmdata,font,attr) - - local forcedisc = alwaysdisc or not attr - - local dirstack = { } -- could move outside function but we can have local runs - - sweephead = { } - - -- We could work on sub start-stop ranges instead but I wonder if there is that - -- much speed gain (experiments showed that it made not much sense) and we need - -- to keep track of directions anyway. Also at some point I want to play with - -- font interactions and then we do need the full sweeps. - - -- Keeping track of the headnode is needed for devanagari (I generalized it a bit - -- so that multiple cases are also covered.) - - -- We don't goto the next node when a disc node is created so that we can then treat - -- the pre, post and replace. It's a bit of a hack but works out ok for most cases. - - for s=1,#datasets do - local dataset = datasets[s] - ----- featurevalue = dataset[1] -- todo: pass to function instead of using a global - local attribute = dataset[2] - local sequence = dataset[3] -- sequences[s] -- also dataset[5] - local rlparmode = 0 - local topstack = 0 - local success = false - local typ = sequence.type - local gpossing = typ == "gpos_single" or typ == "gpos_pair" -- store in dataset - local handler = handlers[typ] - local steps = sequence.steps - local nofsteps = sequence.nofsteps - if not steps then - -- this permits injection, watch the different arguments - local h, d, ok = handler(head,head,dataset,sequence,nil,nil,nil,0,font,attr) - if ok then - success = true - if h then + local initialrl = direction == "TRT" and -1 or 0 + -- local initialrl = (direction == 1 or direction == "TRT") and -1 or 0 + + -- local done = false + local datasets = otfdataset(tfmdata,font,attr) + local dirstack = { nil } -- could move outside function but we can have local runs + sweephead = { } + -- sweephead = { a = 1, b = 1 } sweephead.a = nil sweephead.b = nil + + -- Keeping track of the headnode is needed for devanagari. (I generalized it a bit + -- so that multiple cases are also covered.) We could prepend a temp node. + + -- We don't goto the next node when a disc node is created so that we can then treat + -- the pre, post and replace. It's a bit of a hack but works out ok for most cases. + + for s=1,#datasets do + local dataset = datasets[s] + local attribute = dataset[2] + local sequence = dataset[3] -- sequences[s] -- also dataset[5] + local rlparmode = initialrl + local topstack = 0 + local typ = sequence.type + local gpossing = typ == "gpos_single" or typ == "gpos_pair" -- store in dataset + local forcetestrun = typ == "gsub_ligature" -- testrun is only for ligatures + local handler = handlers[typ] -- store in dataset + local steps = sequence.steps + local nofsteps = sequence.nofsteps + local skiphash = sequence.skiphash + + if not steps then + -- This permits injection, watch the different arguments. Watch out, the arguments passed + -- are not frozen as we might extend or change this. Is this used at all apart from some + -- experiments? + local h, ok = handler(head,dataset,sequence,initialrl,font,attr) -- less arguments now + -- if ok then + -- done = true + -- end + if h and h ~= head then head = h end - end - elseif typ == "gsub_reversecontextchain" then - -- this is a limited case, no special treatments like 'init' etc - local start = find_node_tail(head) - while start do - local char = ischar(start,font) - if char then - local a = attr and getattr(start,0) - if not a or (a == attr) then - for i=1,nofsteps do - local step = steps[i] - local lookupcache = step.coverage - if lookupcache then - local lookupmatch = lookupcache[char] - if lookupmatch then - -- todo: disc? - local ok - head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,step,i) - if ok then - success = true - break + elseif typ == "gsub_reversecontextchain" then + -- + -- This might need a check: if we have #before or #after > 0 then we might need to reverse + -- the before and after lists in the loader. But first I need to see a font that uses multiple + -- matches. + -- + local start = find_node_tail(head) + local rlmode = 0 -- how important is this .. do we need to check for dir? + local merged = steps.merged + while start do + local char = ischar(start,font) + if char then + local m = merged[char] + if m then + local a -- happens often so no assignment is faster + if attr then + a = getattr(start,0) + end + if not a or (a == attr) then + for i=m[1],m[2] do + local step = steps[i] + -- for i=1,#m do + -- local step = m[i] + local lookupcache = step.coverage + local lookupmatch = lookupcache[char] + if lookupmatch then + local ok + head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step) + if ok then + -- done = true + break + end end end + if start then + start = getprev(start) + end else - report_missing_coverage(dataset,sequence) + start = getprev(start) end - end - if start then + else start = getprev(start) end else start = getprev(start) end - else - start = getprev(start) end - end - else - local start = head -- local ? - rlmode = 0 -- to be checked ? - if nofsteps == 1 then -- happens often - local step = steps[1] - local lookupcache = step.coverage - if not lookupcache then - report_missing_coverage(dataset,sequence) - else + else + local start = head + local rlmode = initialrl + if nofsteps == 1 then -- happens often + local step = steps[1] + local lookupcache = step.coverage while start do local char, id = ischar(start,font) if char then - local a = attr and getattr(start,0) - if a then - a = (a == attr) and (not attribute or getprop(start,a_state) == attribute) + if skiphash and skiphash[char] then -- we never needed it here but let's try + start = getnext(start) else - a = not attribute or getprop(start,a_state) == attribute - end - if a then local lookupmatch = lookupcache[char] if lookupmatch then - local ok - head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,step,1) - if ok then - success = true - -- elseif gpossing and zwnjruns and char == zwnj then - -- discrun(start,d_run,font,attr,lookupcache) + local a -- happens often so no assignment is faster + if attr then + if getattr(start,0) == attr and (not attribute or getprop(start,a_state) == attribute) then + a = true + end + elseif not attribute or getprop(start,a_state) == attribute then + a = true end - -- elseif gpossing and zwnjruns and char == zwnj then - -- discrun(start,d_run,font,attr,lookupcache) - end - if start then - start = getnext(start) + if a then + local ok + head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step) + -- if ok then + -- done = true + -- end + if start then + start = getnext(start) + end + else + start = getnext(start) + end + else + start = getnext(start) end - else - start = getnext(start) end - elseif char == false then - -- whatever glyph - start = getnext(start) + elseif char == false or id == glue_code then + -- a different font|state or glue (happens often) + start = getnext(start) elseif id == disc_code then - -- ctx: we could assume the same attr as the surrounding glyphs but then we loose - -- the option to have interesting disc nodes (we gain upto 10% on extreme tests); - -- if a disc would have a font field we could even be more strict (see oldstyle test - -- case) - local a = forcedisc or getsubtype(start) == discretionary_code or getattr(start,0) == attr - if a then - -- local attr = false + if not discs or discs[start] == true then local ok if gpossing then - start, ok = kernrun(start,k_run_single, font,attr,lookupcache,step,dataset,sequence,rlmode,handler) - elseif typ == "gsub_ligature" then - start, ok = testrun(start,t_run_single,c_run_single,font,attr,lookupcache,step,dataset,sequence,rlmode,handler) + start, ok = kernrun(start,k_run_single, font,attr,lookupcache,step,dataset,sequence,rlmode,skiphash,handler) + elseif forcetestrun then + start, ok = testrun(start,t_run_single,c_run_single,font,attr,lookupcache,step,dataset,sequence,rlmode,skiphash,handler) else - start, ok = comprun(start,c_run_single, font,attr,lookupcache,step,dataset,sequence,rlmode,handler) - end - if ok then - success = true + start, ok = comprun(start,c_run_single, font,attr,lookupcache,step,dataset,sequence,rlmode,skiphash,handler) end + -- if ok then + -- done = true + -- end else start = getnext(start) end @@ -3578,97 +4004,199 @@ local function featuresprocessor(head,font,attr) start = getnext(start) end end - end - - else - - while start do - local char, id = ischar(start,font) - if char then - local a = attr and getattr(start,0) - if a then - a = (a == attr) and (not attribute or getprop(start,a_state) == attribute) - else - a = not attribute or getprop(start,a_state) == attribute - end - if a then - for i=1,nofsteps do - local step = steps[i] - local lookupcache = step.coverage - if lookupcache then - local lookupmatch = lookupcache[char] - if lookupmatch then - -- we could move all code inline but that makes things even more unreadable - local ok - head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,step,i) - if ok then - success = true - break - elseif not start then - -- don't ask why ... shouldn't happen - break - -- elseif gpossing and zwnjruns and char == zwnj then - -- discrun(start,d_run,font,attr,steps,nofsteps) + else + local merged = steps.merged + while start do + local char, id = ischar(start,font) + if char then + if skiphash and skiphash[char] then -- we never needed it here but let's try + start = getnext(start) + else + local m = merged[char] + if m then + local a -- happens often so no assignment is faster + if attr then + if getattr(start,0) == attr and (not attribute or getprop(start,a_state) == attribute) then + a = true end - -- elseif gpossing and zwnjruns and char == zwnj then - -- discrun(start,d_run,font,attr,steps,nofsteps) + elseif not attribute or getprop(start,a_state) == attribute then + a = true + end + if a then + for i=m[1],m[2] do + local step = steps[i] + -- for i=1,#m do + -- local step = m[i] + local lookupcache = step.coverage + local lookupmatch = lookupcache[char] + if lookupmatch then + -- we could move all code inline but that makes things even more unreadable + local ok + head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step) + if ok then + -- done = true + break + elseif not start then + -- don't ask why ... shouldn't happen + break + end + end + end + if start then + start = getnext(start) + end + else + start = getnext(start) end else - report_missing_coverage(dataset,sequence) + start = getnext(start) end end - if start then - start = getnext(start) - end - else + elseif char == false or id == glue_code then + -- a different font|state or glue (happens often) start = getnext(start) - end - elseif char == false then - start = getnext(start) - elseif id == disc_code then - -- see comment above - local a = forcedisc or getsubtype(start) == discretionary_code or getattr(start,0) == attr - if a then - local ok - if gpossing then - start, ok = kernrun(start,k_run_multiple, font,attr,steps,nofsteps,dataset,sequence,rlmode,handler) - elseif typ == "gsub_ligature" then - start, ok = testrun(start,t_run_multiple,c_run_multiple,font,attr,steps,nofsteps,dataset,sequence,rlmode,handler) + elseif id == disc_code then + if not discs or discs[start] == true then + local ok + if gpossing then + start, ok = kernrun(start,k_run_multiple, font,attr,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler) + elseif forcetestrun then + start, ok = testrun(start,t_run_multiple,c_run_multiple,font,attr,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler) + else + start, ok = comprun(start,c_run_multiple, font,attr,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler) + end + -- if ok then + -- done = true + -- end else - start, ok = comprun(start,c_run_multiple, font,attr,steps,nofsteps,dataset,sequence,rlmode,handler) - end - if ok then - success = true + start = getnext(start) end + elseif id == math_code then + start = getnext(end_of_math(start)) + elseif id == dir_code then + start, topstack, rlmode = txtdirstate(start,dirstack,topstack,rlparmode) + elseif id == localpar_code then + start, rlparmode, rlmode = pardirstate(start) else start = getnext(start) end - elseif id == math_code then - start = getnext(end_of_math(start)) - elseif id == dir_code then - start, topstack, rlmode = txtdirstate(start,dirstack,topstack,rlparmode) - elseif id == localpar_code then - start, rlparmode, rlmode = pardirstate(start) - else - start = getnext(start) end end end + + if trace_steps then -- ? + registerstep(head) + end + end - if success then - done = true + nesting = nesting - 1 + + -- return head, done + return head + end + + -- This is not an official helper and used for tracing experiments. It can be changed as I like + -- at any moment. At some point it might be used in a module that can help font development. + + function otf.datasetpositionprocessor(head,font,direction,dataset) + + currentfont = font + tfmdata = fontdata[font] + descriptions = tfmdata.descriptions -- only needed in gref so we could pass node there instead + characters = tfmdata.characters -- but this branch is not entered that often anyway + local resources = tfmdata.resources + marks = resources.marks + classes = resources.classes + threshold, + factor = getthreshold(font) + checkmarks = tfmdata.properties.checkmarks + + if type(dataset) == "number" then + dataset = otfdataset(tfmdata,font,0)[dataset] end - if trace_steps then -- ? - registerstep(head) + + local sequence = dataset[3] -- sequences[s] -- also dataset[5] + local typ = sequence.type + -- local gpossing = typ == "gpos_single" or typ == "gpos_pair" -- store in dataset + + -- gpos_contextchain gpos_context + + -- if not gpossing then + -- return head, false + -- end + + local handler = handlers[typ] -- store in dataset + local steps = sequence.steps + local nofsteps = sequence.nofsteps + + local done = false + local dirstack = { nil } -- could move outside function but we can have local runs (maybe a few more nils) + local start = head + local initialrl = direction == "TRT" and -1 or 0 + -- local initialrl = (direction == 1 or direction == "TRT") and -1 or 0 + local rlmode = initialrl + local rlparmode = initialrl + local topstack = 0 + local merged = steps.merged + + -- local matches = false + local position = 0 + + while start do + local char, id = ischar(start,font) + if char then + position = position + 1 + local m = merged[char] + if m then + if skiphash and skiphash[char] then -- we never needed it here but let's try + start = getnext(start) + else + for i=m[1],m[2] do + local step = steps[i] + local lookupcache = step.coverage + local lookupmatch = lookupcache[char] + if lookupmatch then + local ok + head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step) + if ok then + -- if matches then + -- matches[position] = i + -- else + -- matches = { [position] = i } + -- end + break + elseif not start then + break + end + end + end + if start then + start = getnext(start) + end + end + else + start = getnext(start) + end + elseif char == false or id == glue_code then + -- a different font|state or glue (happens often) + start = getnext(start) + elseif id == math_code then + start = getnext(end_of_math(start)) + elseif id == dir_code then + start, topstack, rlmode = txtdirstate(start,dirstack,topstack,rlparmode) + elseif id == localpar_code then + start, rlparmode, rlmode = pardirstate(start) + else + start = getnext(start) + end end + return head end - nesting = nesting - 1 - head = tonode(head) + -- end of experiment - return head, done end -- so far @@ -3676,32 +4204,49 @@ end local plugins = { } otf.plugins = plugins +local report = logs.reporter("fonts") + function otf.registerplugin(name,f) if type(name) == "string" and type(f) == "function" then plugins[name] = { name, f } + report() + report("plugin %a has been loaded, please be aware of possible side effects",name) + report() + if logs.pushtarget then + logs.pushtarget("log") + end + report("Plugins are not officially supported unless stated otherwise. This is because") + report("they bypass the regular font handling and therefore some features in ConTeXt") + report("(especially those related to fonts) might not work as expected or might not work") + report("at all. Some plugins are for testing and development only and might change") + report("whenever we feel the need for it.") + report() + if logs.poptarget then + logs.poptarget() + end end end -local function plugininitializer(tfmdata,value) +function otf.plugininitializer(tfmdata,value) if type(value) == "string" then tfmdata.shared.plugin = plugins[value] end end -local function pluginprocessor(head,font) +function otf.pluginprocessor(head,font,attr,direction) -- n local s = fontdata[font].shared local p = s and s.plugin if p then if trace_plugins then report_process("applying plugin %a",p[1]) end - return p[2](head,font) + return p[2](head,font,attr,direction) else return head, false end end -local function featuresinitializer(tfmdata,value) +function otf.featuresinitializer(tfmdata,value) -- nothing done here any more end @@ -3711,67 +4256,113 @@ registerotffeature { default = true, initializers = { position = 1, - node = featuresinitializer, - plug = plugininitializer, + node = otf.featuresinitializer, + plug = otf.plugininitializer, }, processors = { - node = featuresprocessor, - plug = pluginprocessor, + node = otf.featuresprocessor, + plug = otf.pluginprocessor, } } -otf.nodemodeinitializer = featuresinitializer -otf.featuresprocessor = featuresprocessor - --- This can be used for extra handlers, but should be used with care! - -otf.handlers = handlers -- used in devanagari +-- This can be used for extra handlers, but should be used with care! We implement one +-- here but some more can be found in the osd (script devanagary) file. Now watch out: +-- when a handlers has steps, it is called as the other ones, but when we have no steps, +-- we use a different call: +-- +-- function(head,dataset,sequence,initialrl,font,attr) +-- return head, done +-- end +-- +-- Also see (!!). --- We implement one here: +otf.handlers = handlers local setspacekerns = nodes.injections.setspacekerns if not setspacekerns then os.exit() end +local tag = "kern" -- maybe some day a merge + if fontfeatures then - function otf.handlers.trigger_space_kerns(head,start,dataset,sequence,_,_,_,_,font,attr) + function handlers.trigger_space_kerns(head,dataset,sequence,initialrl,font,attr) local features = fontfeatures[font] - local enabled = features and features.spacekern and features.kern + local enabled = features and features.spacekern and features[tag] if enabled then setspacekerns(font,sequence) end - return head, start, enabled + return head, enabled end else -- generic (no hashes) - function otf.handlers.trigger_space_kerns(head,start,dataset,sequence,_,_,_,_,font,attr) + function handlers.trigger_space_kerns(head,dataset,sequence,initialrl,font,attr) local shared = fontdata[font].shared local features = shared and shared.features - local enabled = features and features.spacekern and features.kern + local enabled = features and features.spacekern and features[tag] if enabled then setspacekerns(font,sequence) end - return head, start, enabled + return head, enabled end end +-- There are fonts out there that change the space but we don't do that kind of +-- things in TeX. + local function hasspacekerns(data) - local sequences = data.resources.sequences - for i=1,#sequences do - local sequence = sequences[i] - local steps = sequence.steps - if steps and sequence.features.kern then - for i=1,#steps do - local coverage = steps[i].coverage - if not coverage then - -- maybe an issue - elseif coverage[32] then - return true - else - for k, v in next, coverage do - if v[32] then - return true + local resources = data.resources + local sequences = resources.sequences + local validgpos = resources.features.gpos + if validgpos and sequences then + for i=1,#sequences do + local sequence = sequences[i] + local steps = sequence.steps + if steps and sequence.features[tag] then + local kind = sequence.type + if kind == "gpos_pair" or kind == "gpos_single" then + for i=1,#steps do + local step = steps[i] + local coverage = step.coverage + local rules = step.rules + if rules then + -- not now: analyze (simple) rules + elseif not coverage then + -- nothing to do + elseif kind == "gpos_single" then + -- maybe a message that we ignore + elseif kind == "gpos_pair" then + local format = step.format + if format == "move" or format == "kern" then + local kerns = coverage[32] + if kerns then + return true + end + for k, v in next, coverage do + if v[32] then + return true + end + end + elseif format == "pair" then + local kerns = coverage[32] + if kerns then + for k, v in next, kerns do + local one = v[1] + if one and one ~= true then + return true + end + end + end + for k, v in next, coverage do + local kern = v[32] + if kern then + local one = kern[1] + if one and one ~= true then + return true + end + end + end + end end end end @@ -3788,112 +4379,120 @@ otf.readers.registerextender { end } --- we merge the lookups but we still honor the language / script - local function spaceinitializer(tfmdata,value) -- attr local resources = tfmdata.resources local spacekerns = resources and resources.spacekerns - local properties = tfmdata.properties if value and spacekerns == nil then + local rawdata = tfmdata.shared and tfmdata.shared.rawdata + local properties = rawdata.properties if properties and properties.hasspacekerns then local sequences = resources.sequences - local left = { } - local right = { } - local last = 0 - local feat = nil - for i=1,#sequences do - local sequence = sequences[i] - local steps = sequence.steps - if steps then - local kern = sequence.features.kern - if kern then - if feat then - for script, languages in next, kern do - local f = feat[script] - if f then - for l in next, languages do - f[l] = true - end - else - feat[script] = languages - end - end - else - feat = kern - end - for i=1,#steps do - local step = steps[i] - local coverage = step.coverage - local rules = step.rules - local format = step.format - if rules then - -- not now: analyze (simple) rules - elseif coverage then - -- what to do if we have no [1] but only [2] - local single = format == gpos_single - local kerns = coverage[32] - if kerns then - for k, v in next, kerns do - if type(v) ~= "table" then - right[k] = v - elseif single then - right[k] = v[3] - else - local one = v[1] - if one then - right[k] = one[3] + local validgpos = resources.features.gpos + if validgpos and sequences then + local left = { } + local right = { } + local last = 0 + local feat = nil + for i=1,#sequences do + local sequence = sequences[i] + local steps = sequence.steps + if steps then + -- we don't support space kerns in other features + local kern = sequence.features[tag] + if kern then + local kind = sequence.type + if kind == "gpos_pair" or kind == "gpos_single" then + if feat then + for script, languages in next, kern do + local f = feat[script] + if f then + for l in next, languages do + f[l] = true end + else + feat[script] = languages end end + else + feat = kern end - for k, v in next, coverage do - local kern = v[32] - if kern then - if type(kern) ~= "table" then - left[k] = kern - elseif single then - left[k] = v[3] - else - local one = v[1] - if one then - left[k] = one[3] + for i=1,#steps do + local step = steps[i] + local coverage = step.coverage + local rules = step.rules + if rules then + -- not now: analyze (simple) rules + elseif not coverage then + -- nothng to do + elseif kind == "gpos_single" then + -- makes no sense in TeX + elseif kind == "gpos_pair" then + local format = step.format + if format == "move" or format == "kern" then + local kerns = coverage[32] + if kerns then + for k, v in next, kerns do + right[k] = v + end + end + for k, v in next, coverage do + local kern = v[32] + if kern then + left[k] = kern + end + end + elseif format == "pair" then + local kerns = coverage[32] + if kerns then + for k, v in next, kerns do + local one = v[1] + if one and one ~= true then + right[k] = one[3] + end + end + end + for k, v in next, coverage do + local kern = v[32] + if kern then + local one = kern[1] + if one and one ~= true then + left[k] = one[3] + end + end end end end end + last = i end + else + -- no steps ... needed for old one ... we could use the basekerns + -- instead end - last = i end - else - -- no steps ... needed for old one ... we could use the basekerns - -- instead - end - end - left = next(left) and left or false - right = next(right) and right or false - if left or right then - spacekerns = { - left = left, - right = right, - } - if last > 0 then - local triggersequence = { - features = { kern = feat or { dflt = { dflt = true, } } }, - flags = noflags, - name = "trigger_space_kerns", - order = { "kern" }, - type = "trigger_space_kerns", - left = left, - right = right, + end + left = next(left) and left or false + right = next(right) and right or false + if left or right then + spacekerns = { + left = left, + right = right, } - insert(sequences,last,triggersequence) + if last > 0 then + local triggersequence = { + -- no steps, see (!!) + features = { [tag] = feat or { dflt = { dflt = true, } } }, + flags = noflags, + name = "trigger_space_kerns", + order = { tag }, + type = "trigger_space_kerns", + left = left, + right = right, + } + insert(sequences,last,triggersequence) + end end - else - spacekerns = false end - else - spacekerns = false end resources.spacekerns = spacekerns end |