cortav  Check-in [330e1ecfdb]

Overview
Comment:add extension mechanism, move toc to extensions, update docs, fix bugs
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: 330e1ecfdb17218ca9fc8468b0f99ab7c74a33b440b0ea0623462c1fa39369fd
User & Date: lexi on 2021-12-22 10:19:13
Other Links: manifest | tags
Context
2021-12-22
10:19
disable backtraces check-in: 2af5c4085a user: lexi tags: trunk
10:19
add extension mechanism, move toc to extensions, update docs, fix bugs check-in: 330e1ecfdb user: lexi tags: trunk
2021-12-21
05:04
add --version -V flag check-in: 0c6a784678 user: lexi tags: trunk
Changes

Modified cli.lua from [bdd85f20a4] to [497f5957f5].

     7      7   }
     8      8   
     9      9   local function
    10     10   main(input, output, log, mode, suggestions, vars)
    11     11   	local doc = ct.parse(input.stream, input.src, mode)
    12     12   	input.stream:close()
    13     13   	if mode['parse:show-tree'] then
    14         -		log:write(dump(doc))
           14  +		log:write(ss.dump(doc))
    15     15   	end
    16     16   
    17     17   	-- the document has now had a chance to give its say; if it hasn't specified
    18     18   	-- any modes of its own, we now merge in the 'weak modes' (suggestions)
    19     19   	for k,v in pairs(suggestions) do
    20     20   		if not mode[k] then mode[k] = v end
    21     21   	end
................................................................................
   159    159   				if i + nargs > #arg then
   160    160   					ct.exns.cli('not enough arguments for switch --%s (%s expected)', longopt, nargs):throw()
   161    161   				end
   162    162   				local nt = {}
   163    163   				for j = i+1, i+nargs do
   164    164   					table.insert(nt, arg[j])
   165    165   				end
   166         -				print('onsw')
          166  +				onswitch[longopt](table.unpack(nt))
   167    167   			elseif nargs == 1 then
   168    168   				onswitch[longopt](arg[i+1])
   169    169   			else
   170    170   				onswitch[longopt]()
   171    171   			end
   172    172   			i = i + nargs
   173    173   		end
................................................................................
   200    200   		input.stream = file
   201    201   		input.src.file = args[1]
   202    202   	end
   203    203   
   204    204   	return main(input, outp, log, mode, suggestions, vars)
   205    205   end
   206    206   
   207         -local ok, e = pcall(entry_cli)
   208         --- local ok, e = true, entry_cli()
          207  +-- local ok, e = pcall(entry_cli)
          208  +local ok, e = true, entry_cli()
   209    209   if not ok then
   210    210   	local str = 'translation failure'
   211    211   	if ss.exn.is(e) then
   212    212   		str = e.kind.desc
   213    213   	end
   214    214   	local color = false
   215    215   	if log:seek() == nil then

Modified cortav.ct from [e9d2ad32df] to [c71fe3a9e8].

    84     84   
    85     85   ## styled text
    86     86   most blocks contain a sequence of spans. these spans are produced by interpreting a stream of [*styled-text] following the control sequence. styled-text is a sequence of codepoints potentially interspersed with escapes. an escape is formed by an open square bracket [$\[] followed by a [*span control sequence], and arguments for that sequence like more styled-text. escapes can be nested.
    87     87   
    88     88   * strong \[*[!styled-text]\]: causes its text to stand out from the narrative, generally rendered as bold or a brighter color.
    89     89   * emphatic \[![!styled-text]\]: indicates that its text should be spoken with emphasis, generally rendered as italics
    90     90   * literal \[$[!styled-text]\]: indicates that its text is a reference to a literal sequence of characters, variable name, or other discrete token. generally rendered in monospace
    91         -* strikeout \[$[~styled-text]\]: indicates that its text should be struck through or otherwise indicated for deletion
    92         -* insertion \[$[+styled-text]\]: indicates that its text should be indicated as a new addition to the text body. 
           91  +* strikeout \[~[!styled-text]\]: indicates that its text should be struck through or otherwise indicated for deletion
           92  +* insertion \[+[!styled-text]\]: indicates that its text should be indicated as a new addition to the text body. 
    93     93   ** consider using a macro definition [$\edit: [~[#1]][+[#2]]] to save typing if you are doing editing work
    94     94   * link \[>[!ref] [!styled-text]\]: produces a hyperlink or cross-reference denoted by [$ref], which may be either a URL specified with a reference or the name of an object like an image or section elsewhere in the document. the unicode characters [$→] and [$🔗] can also be used instead of [$>] to denote a link.
    95     95   * footnote \[^[!ref] [!styled-text]\]: annotates the text with a defined footnote
    96     96   * raw \[\\[!raw-text]\]: causes all characters within to be interpreted literally, without expansion. the only special characters are square brackets, which must have a matching closing bracket
    97     97   * raw literal \[$\\[!raw-text]\]: shorthand for [\[$[\…]]]
    98     98   * macro \{[!name] [!arguments]\}: invokes a [>ex.mac macro], specified with a reference
    99     99   * argument \[#[!var]\]: in macros only, inserts the [$var]-th argument. otherwise, inserts a context variable provided by the renderer.
................................................................................
   199    199   where possible, instead of [$needs x y z], the directive [$when has-ext x y z] should be used instead. this causes the next section to be rendered only if the named extensions are available. [$unless has-ext x y z] can be used to provide an alternative format.
   200    200   
   201    201   extensions are mainly interacted with through directives. all extension directives must be prefixed with the name of the extension.
   202    202   
   203    203   ### toc
   204    204   sections that have a title will be included in the table of contents. the table of contents is by default inserted at the break between the first level-1 section and the section immediately following it. you may instead place the directive [$toc] where you wish the TOC to be inserted, or suppress it entirely with [$inhibits toc]. note that some renderers may not display the TOC as part of the document itself.
   205    205   
          206  +toc provides the directives:
          207  +
          208  +* [$%[*toc]]: insert a table of contents in the specified position. this can be used more than once, but doing so may have confusing, incorrect, or nonsensical results under some renderers, and some may just ignore the directive entirely
          209  +* [$%[*toc] mark [!styled-text]]: inserts a TOC entry with the label [!styled-text]  pointing to the current location. this can be used to e.g. mark noteworthy images, instances of long quotes or literal blocks, or functions inside an expanded code block.
          210  +* [$%[*toc] name [!id styled-text]]: like [$%[*toc] mark] but allows an additional [!id] parameter which specifies the ID the renderer will assign to an anchor element. this is not meaningful for all renderers and when it is, it is up to the renderer to decide what it means.
          211  +** the [*html] render backend interprets [!id] as the [$id] element for the anchor tag
          212  +** the [*groff] render backend ignores [!id]
          213  +
   206    214   ### smart-quotes
   207    215   a cortav renderer may automatically translate punctuation marks to other punctuation marks depending on their context. 
   208    216   
   209    217   ### hilite
   210    218   code can be highlighted according to the formal language it is written in.
   211    219   
   212    220   ### lua
................................................................................
   378    386   	-m html:hue-spread 35 \
   379    387   	-y html:dark-on-light # could also be written as:
   380    388   $ cortav readme.ct -ommmmy readme.html render:format html html:width 40em html:accent 80 html:hue-spread 35 html:dark-on-light
   381    389   ~~~
   382    390   
   383    391   ## further directions
   384    392   
          393  +### additional backends
          394  +it is eventually intended to support to following backends, if reasonably practicable.
          395  +* [*html]: emit HTML and CSS code to typeset the document. [!in progress]
          396  +* [*svg]: emit SVG, taking advantage of its precise layout features to produce a nicely formatted and paginated document. pagination can be accomplished through emitting multiple files or by assigning one layer to each page. [!long term]
          397  +* [*groff]: the most important output backend, rivalling [*html]. will allow the document to be typeset in a wide variety of formats, including PDF and manpage. [!short term]
          398  +* [*gemtext]: essentially a downrezzing of cortav to make it readable to Gemini clients
          399  +
          400  +some formats may eventually warrant their own renderer, but are not a priority:
          401  +* [*text]: cortav source files are already plain text, but a certain amount of layout could be done using ascii art.
          402  +* [*ansi]: emit sequences of ANSI escape codes to lay out a document in a terminal-friendly way
          403  +* [*tex]: TeX is an unholy abomination and i neither like nor use it, but lots of people do and if cortav ever catches on, a TeX backend should probably be written eventually.
          404  +
          405  +PDF is not on either list because it's a nightmarish mess of a format and groff, which is installed on most linux systems already, can easily generate PDFs
          406  +
   385    407   ### LCH support
   386    408   right now, the use of color in the HTML renderer is very unsatisfactory. the accent mechanism operates on the basis of the CSS HSL function, which is not perceptually uniform; different hues will present different mixes of brightness and some (yellows?) may be ugly or unreadable.
   387    409   
   388    410   the ideal solution would be to simply switch to using LCH based colors. unfortunately, only Safari actually supports the LCH color function right now, and it's unlikely (unless Lea Verou and her husband manage to work a miracle) that Colors Level 4 is going to be implemented very widely any time soon.
   389    411   
   390    412   this leaves us in an awkward position. we can of course do the math ourselves, working in LCH to implement the internal [$@tone] macro, and then "converting" these colors to HSL. unfortunately, you can't actually convert from LCH to HSL; it's like converting from pounds to kilograms. LCH can represent any color the human visual system can perceive; sRGB can't, and CSS HSL is implemented in sRGB. however, we could at least approximate something that would allow for perceptually uniform brightness, which would be an improvement, and this is probably the direction to go in, unless a miracle occurs and [$lch()] or [$color()] pop up in Blink.
   391    413   
   392    414   it may be possible to do a more reasonable job of handling colors in the postscript and TeX outputs. unsure about SVG but i assume it suffers the same problems HTML/CSS do. does groff even support color??
   393    415   
   394    416   ### intent files
   395    417   there's currently no standard way to describe the intent and desired formatting of a document besides placing pragmas in the source file itself. this is extremely suboptimal, as when generating collections of documents, it's ideal to be able to keep all formatting information in one place. users should also be able to specify their own styling overrides that describe the way they prefer to read [$cortav] files, especially for uses like gemini or gopher integration.
   396    418   
   397    419   at some point soon [$cortav] needs to address this by adding intent files that can be activated from outside the source file, such as with a command line flag or a configuration file setting. these will probably consist of lines that are interpreted as pragmata. in addition to the standard intent format however, individual implementations should feel free to provide their own ways to provide intent metadata; e.g. the reference implementation, which has a lua interpreter available, should be able to take a lua script that runs after the parse stage and generates . this will be particularly useful for the end-user who wishes to specify a particular format she likes reading her files in without forcing that format on everyone she sends the compiled document to, as it will be able to interrogate the document and make intelligent decisions about what pragmata to apply.

Modified cortav.lua from [7d896a453b] to [e7bf814ce9].

   106    106   	fns = {
   107    107   		fail = function(self, msg, ...)
   108    108   			ct.exns.tx(msg, self.src.file, self.line or 0, ...):throw()
   109    109   		end;
   110    110   		insert = function(self, block)
   111    111   			block.origin = self:clone()
   112    112   			table.insert(self.sec.blocks,block)
          113  +			return block
   113    114   		end;
   114    115   		ref = function(self,id)
   115    116   			if not id:find'%.' then
   116    117   				local rid = self.sec.refs[id]
   117    118   				if self.sec.refs[id] then
   118    119   					return self.sec.refs[id]
   119    120   				else self:fail("no such ref %s in current section", id or '') end
................................................................................
   149    150   	fns = {
   150    151   		mksec = function(self, id, depth)
   151    152   			local o = ct.sec(id, depth)
   152    153   			if id then self.sections[id] = o end
   153    154   			table.insert(self.secorder, o)
   154    155   			return o
   155    156   		end;
          157  +		allow_ext = function(self,name)
          158  +			if not ct.ext.loaded[name] then return false end
          159  +			if self.ext.inhibit[name] then return false end
          160  +			if self.ext.need[name] or self.ext.use[name] then
          161  +				return true
          162  +			end
          163  +			return ct.ext.loaded[name].default
          164  +		end;
   156    165   		context_var = function(self, var, ctx, test)
   157    166   			local fail = function(...)
   158    167   				if test then return false end
   159    168   				ctx:fail(...)
   160    169   			end
   161    170   			if startswith(var, 'cortav.') then
   162    171   				local v = var:sub(8)
................................................................................
   193    202   			elseif self.vars[var] then
   194    203   				return self.vars[var]
   195    204   			else
   196    205   				if test then return false end
   197    206   				return '' -- is this desirable behavior?
   198    207   			end
   199    208   		end;
          209  +		job = function(self, name, pred, ...) -- convenience func
          210  +			return self.docjob:fork(name, pred, ...)
          211  +		end
   200    212   	};
   201    213   	mk = function() return {
   202    214   		sections = {};
   203    215   		secorder = {};
   204    216   		embed = {};
   205    217   		meta = {};
   206    218   		vars = {};
          219  +		ext = {
          220  +			inhibit = {};
          221  +			need = {};
          222  +			use = {};
          223  +		};
   207    224   	} end;
          225  +	construct = function(me)
          226  +		me.docjob = ct.ext.job('doc', me, nil)
          227  +	end;
   208    228   }
   209    229   
   210    230   -- FP helper functions
   211    231   
   212    232   local function fmtfn(str)
   213    233   	return function(...)
   214    234   		return string.format(str, ...)
................................................................................
   221    241   		ct.exns.ext 'extension missing “id” field':throw()
   222    242   	end
   223    243   	if ct.ext.loaded[ext.id] then
   224    244   		ct.exns.ext('there is already an extension with ID “%s” loaded', ext.id):throw()
   225    245   	end
   226    246   	ct.ext.loaded[ext.id] = ext
   227    247   end
          248  +
          249  +function ct.ext.bind(doc)
          250  +	local fns = {}
          251  +	function fns.each(...)
          252  +		local cext
          253  +		local args = {...}
          254  +		return function()
          255  +			while true do
          256  +				cext = next(ct.ext.loaded, cext)
          257  +				if cext == nil then return nil end
          258  +				if doc == nil or doc:allow_ext(cext.id) then
          259  +					local v = ss.walk(ct.ext.loaded[cext.id], table.unpack(args))
          260  +					if v ~= nil then
          261  +						return v, cext
          262  +					end
          263  +				end
          264  +			end
          265  +		end
          266  +	end
          267  +
          268  +	function fns.hook(h, ...)
          269  +		-- this is the raw hook invocation function, used when hooks won't need
          270  +		-- private state to hold onto between invocation. if private state is
          271  +		-- necessary, construct a job instead
          272  +		local ret = {} -- for hooks that compile lists of responses from extensions
          273  +		for hook in fns.each('hook', h) do table.insert(ret,(hook(...))) end
          274  +		return ret
          275  +	end
          276  +	
          277  +	return fns
          278  +end
          279  +
          280  +do local globalfns = ct.ext.bind()
          281  +	-- use these functions when document restrictions don't matter
          282  +	ct.ext.each, ct.ext.hook = globalfns.each, globalfns.hook
          283  +end
          284  +
          285  +ct.ext.job = declare {
          286  +	ident = 'ext-job';
          287  +	init = {
          288  +		states = {};
          289  +	};
          290  +	construct = function(me,name,doc,pred,...)
          291  +		print('constructing job',name,'for',doc)
          292  +		-- prepare contexts for relevant extensions
          293  +		me.name = name
          294  +		me.doc = doc -- for reqs + limiting
          295  +		for _, ext in pairs(ct.ext.loaded) do
          296  +			if pred == nil or pred(ext) then
          297  +				me.states[ext] = {}
          298  +			end
          299  +		end
          300  +		me:hook('init', ...)
          301  +	end;
          302  +	fns = {
          303  +		fork = function(me, name, pred, ...)
          304  +			-- generate a branch job linked to this job
          305  +			local branch = getmetatable(me)(name, me.doc, pred, ...)
          306  +			branch.parent = me
          307  +			return branch
          308  +		end;
          309  +		delegate = function(me, ext) -- creates a delegate for state access
          310  +			local submethods = {
          311  +				unwind = function(self, n)
          312  +					local function
          313  +					climb(dlg, job, n)
          314  +						if n == 0 then
          315  +							return job:delegate(dlg.extension)
          316  +						else
          317  +							return climb(dlg, job.parent, n-1)
          318  +						end
          319  +					end
          320  +
          321  +					return climb(self._delegate_state, self._delegate_state.target, n)
          322  +				end;
          323  +			}
          324  +			local d = setmetatable({
          325  +				_delegate_state = {
          326  +					target = (me._delegate_state and me._delegate_state.target) or me;
          327  +					extension = ext;
          328  +				};
          329  +			}, {
          330  +				__name = 'job:delegate';
          331  +				__index = function(self, key)
          332  +					local D = self._delegate_state
          333  +					if key == 'state' then
          334  +						return D.target.states[self._delegate_state.extension]
          335  +					elseif submethods[key] then
          336  +						return submethods[key]
          337  +					end
          338  +					return D.target[key]
          339  +				end;
          340  +				__newindex = function(self, key, value)
          341  +					local D = self._delegate_state
          342  +					if key == 'state' then
          343  +						D.target.states[D.extension] = value
          344  +					else
          345  +						D.target[D.extension] = value
          346  +					end
          347  +				end;
          348  +			});
          349  +			return d;
          350  +		end;
          351  +		each = function(me, ...)
          352  +			local ek
          353  +			local path = {...}
          354  +			return function()
          355  +				while true do
          356  +					ek = next(me.states, ek)
          357  +					if not ek then return nil end
          358  +					if me.doc:allow_ext(ek.id) then
          359  +						local v = ss.walk(ek, table.unpack(path))
          360  +						if v then
          361  +							return v, ek, me.states[ek]
          362  +						end
          363  +					end
          364  +				end
          365  +			end
          366  +		end;
          367  +		proc = function(me, ...)
          368  +			local p
          369  +			local owner
          370  +			local state
          371  +			for func, ext, s in me:each(...) do
          372  +				if p == nil then
          373  +					p = func
          374  +					owner = ext
          375  +					state = s
          376  +				else
          377  +					ct.exn.ext('extensions %s and %s define conflicting procedures for %s', owner.id, ext.id, table.concat({...},'.')):throw()
          378  +				end
          379  +			end
          380  +			if p == nil then return nil end
          381  +			if type(p) ~= 'function' then return p end
          382  +			return function(...)
          383  +				return p(me:delegate(owner), ...)
          384  +			end, owner, state
          385  +		end;
          386  +		hook = function(me, hook, ...)
          387  +			-- used when extensions may need to persist state across
          388  +			-- multiple functions or invocations
          389  +			local ret = {}
          390  +			local hook_id = me.name ..'_'.. hook
          391  +			for hookfn, ext, state in me:each('hook', hook_id) do
          392  +				print(' - running hook for ext',ext.id)
          393  +				table.insert(ret, (hookfn(me:delegate(ext),...)))
          394  +			end
          395  +			return ret
          396  +		end;
          397  +	};
          398  +}
   228    399   
   229    400   -- renderer engines
   230    401   function ct.render.html(doc, opts)
   231    402   	local doctitle = opts['title']
   232    403   	local f = string.format
   233    404   	local ids = {}
   234    405   	local canonicalID = {}
................................................................................
   419    590   			}
   420    591   			section > figure.listing > hr {
   421    592   				border: none;
   422    593   				margin: 0;
   423    594   				height: 0.7em;
   424    595   				counter-increment: line-number;
   425    596   			}
   426         -		]];
   427         -		toc = [[
   428         -
   429         -		]];
   430         -		tocFixed = [[
   431         -			@media (min-width: calc(@[width]:[100vw] + 20em)) {
   432         -				ol.toc {
   433         -					position: fixed;
   434         -					padding-top: 1em; padding-bottom: 1em;
   435         -					padding-right: 1em;
   436         -					margin-top: 0; margin-bottom: 0;
   437         -					right: 0; top: 0; bottom: 0;
   438         -					max-width: calc(50vw - ((@[width]:[0]) / 2) - 3.5em);
   439         -					overflow-y: auto;
   440         -				}
   441         -				@media (max-width: calc(@[width]:[100vw] + 30em)) {
   442         -					ol.toc {
   443         -						max-width: calc(100vw - ((@[width]:[0])) - 9.5em);
   444         -					}
   445         -					body {
   446         -						margin-left: 5em;
   447         -					}
   448         -				}
   449         -			}
   450    597   		]];
   451    598   	}
   452    599   
   453    600   	local stylesNeeded = {}
   454    601   
   455         -	local function getSpanRenderers(tag,elt)
          602  +	local render_state_handle = {
          603  +		doc = doc;
          604  +		opts = opts;
          605  +		style_rules = styles; -- use stylesneeded if at all possible
          606  +		stylesets = stylesets;
          607  +		stylesets_active = stylesNeeded;
          608  +		obj_htmlid = getSafeID;
          609  +		-- remaining fields added later
          610  +	}
          611  +
          612  +	local renderJob = doc:job('render_html', nil, render_state_handle)
          613  +
          614  +	local runhook = function(h, ...)
          615  +		return renderJob:hook(h, render_state_handle, ...)
          616  +	end
          617  +
          618  +	local function getSpanRenderers(procs)
          619  +		local tag, elt, catenate = procs.tag, procs.elt, procs.catenate
   456    620   		local htmlDoc = function(title, head, body)
   457    621   			return [[<!doctype html>]] .. tag('html',nil,
   458    622   				tag('head', nil,
   459    623   					elt('meta',{charset = 'utf-8'}) ..
   460    624   					(title and tag('title', nil, title) or '') ..
   461    625   					(head or '')) ..
   462    626   				tag('body', nil, body or ''))
................................................................................
   512    676   
   513    677   		function span_renderers.var(v,b,s)
   514    678   			local val
   515    679   			if v.pos then
   516    680   				if not v.origin.invocation then
   517    681   					v.origin:fail 'positional arguments can only be used in a macro invocation'
   518    682   				elseif not v.origin.invocation.args[v.pos] then
   519         -					v.origin:fail('macro invocation %s missing positional argument #%u', v.origin.invocation.macro, v.pos)
          683  +					v.origin.invocation.origin:fail('macro invocation %s missing positional argument #%u', v.origin.invocation.macro, v.pos)
   520    684   				end
   521    685   				val = v.origin.invocation.args[v.pos]
   522    686   			else
   523    687   				val = v.origin.doc:context_var(v.var, v.origin)
   524    688   			end
   525    689   			if v.raw then
   526    690   				return val
................................................................................
   547    711   			span_renderers = span_renderers;
   548    712   			htmlSpan = htmlSpan;
   549    713   			htmlDoc = htmlDoc;
   550    714   		}
   551    715   	end
   552    716   
   553    717   
   554         -	local function getBlockRenderers(tag,elt,sr,catenate)
   555         -		local function insert_toc(b,s)
   556         -			local lst = {tag = 'ol', attrs={class='toc'}, nodes={}}
   557         -			stylesNeeded.toc = true
   558         -			if opts['width'] then
   559         -				stylesNeeded.tocFixed = true
   560         -			end
   561         -			local stack = {lst}
   562         -			local top = function() return stack[#stack] end
   563         -			local all = s.origin.doc.secorder
   564         -			for i, sec in ipairs(all) do
   565         -				if sec.heading_node then
   566         -					local ent = tag('li',nil,
   567         -						 catenate{tag('a', {href='#'..getSafeID(sec)},
   568         -							sr.htmlSpan(sec.heading_node.spans))})
   569         -					if sec.depth > #stack then
   570         -						local n = {tag = 'ol', attrs={}, nodes={ent}}
   571         -						table.insert(top().nodes[#top().nodes].nodes, n)
   572         -						table.insert(stack, n)
   573         -					else
   574         -						if sec.depth < #stack then
   575         -							for j=#stack,sec.depth+1,-1 do stack[j] = nil end
   576         -						end
   577         -						table.insert(top().nodes, ent)
   578         -					end
   579         -				end
   580         -			end
   581         -			return lst
   582         -		end
          718  +	local function getBlockRenderers(procs, sr)
          719  +		local tag, elt, catenate = procs.tag, procs.elt, procs.catenate
          720  +		local null = function() return catenate{} end
   583    721   
   584    722   		local block_renderers = {
          723  +			anchor = function(b,s)
          724  +				return tag('a',{id = getSafeID(b)},null())
          725  +			end;
   585    726   			paragraph = function(b,s)
   586    727   				stylesNeeded.paragraph = true;
   587    728   				return tag('p', nil, sr.htmlSpan(b.spans, b, s), b)
   588    729   			end;
   589    730   			directive = function(b,s)
   590    731   				-- deal with renderer directives
   591    732   				local _, cmd, args = b.words(2)
................................................................................
   605    746   				else
   606    747   					-- handle other uses of labels here
   607    748   				end
   608    749   			end;
   609    750   			['list-item'] = function(b,s)
   610    751   				return tag('li', nil, sr.htmlSpan(b.spans, b, s), b)
   611    752   			end;
   612         -			toc = insert_toc;
   613    753   			table = function(b,s)
   614    754   				local tb = {}
   615    755   				for i, r in ipairs(b.rows) do
   616    756   					local row = {}
   617    757   					for i, c in ipairs(r) do
   618    758   						table.insert(row, tag(c.header and 'th' or 'td',
   619    759   						{align=c.align}, sr.htmlSpan(c.spans, b)))
................................................................................
   633    773   				end, b.lines)
   634    774   				if b.title then
   635    775   					table.insert(nodes,1,tag('figcaption',nil,sr.htmlSpan(b.title)))
   636    776   				end
   637    777   				if b.lang then langsused[b.lang] = true end
   638    778   				return tag('figure', {class='listing', lang=b.lang, id=b.id and getSafeID(b)}, catenate(nodes))
   639    779   			end;
          780  +			aside = function(b,s)
          781  +				local bn = {}
          782  +				for _,v in pairs(b.lines) do
          783  +					table.insert(bn, tag('p', {}, sr.htmlSpan(v, b, s)))
          784  +				end
          785  +				return tag('aside', {}, bn)
          786  +			end;
   640    787   			['break'] = function() --[[nop]] end;
   641    788   		}
   642    789   		return block_renderers;
   643    790   	end
   644    791   
   645         -	local pspan = getSpanRenderers(function(t,a,v) return v  end,
   646         -	                               function(t,a)   return '' end)
   647         -	 
   648         -	local function getRenderers(tag,elt,catenate)
   649         -		local r = getSpanRenderers(tag,elt)
   650         -		r.block_renderers = getBlockRenderers(tag,elt,r,catenate)
          792  +	local function getRenderers(procs)
          793  +		local r = getSpanRenderers(procs)
          794  +		r.block_renderers = getBlockRenderers(procs, r)
   651    795   		return r
   652    796   	end
   653    797   
   654         -	local elt = function(t,attrs)
   655         -		return f('<%s%s>', t,
   656         -			attrs and ss.reduce(function(a,b) return a..b end, '', 
   657         -				ss.map(function(v,k)
   658         -					if v == true
   659         -						then          return ' '..k
   660         -						elseif v then return f(' %s="%s"', k, v)
   661         -					end
   662         -				end, attrs)) or '')
   663         -	end
   664         -	local tag = function(t,attrs,body)
   665         -		return f('%s%s</%s>', elt(t,attrs), body, t)
   666         -	end
   667         -
          798  +	local tagproc do
          799  +		local elt = function(t,attrs)
          800  +			return f('<%s%s>', t,
          801  +				attrs and ss.reduce(function(a,b) return a..b end, '', 
          802  +					ss.map(function(v,k)
          803  +						if v == true
          804  +							then          return ' '..k
          805  +							elseif v then return f(' %s="%s"', k, v)
          806  +						end
          807  +					end, attrs)) or '')
          808  +		end
          809  +
          810  +		tagproc = {
          811  +			toTXT = {
          812  +				tag = function(t,a,v) return v  end;
          813  +				elt = function(t,a)   return '' end;
          814  +				catenate = table.concat;
          815  +			};
          816  +			toIR = {
          817  +				tag = function(t,a,v,o) return {
          818  +					tag = t, attrs = a;
          819  +					nodes = type(v) == 'string' and {v} or v, src = o
          820  +				} end;
          821  +				
          822  +				elt = function(t,a,o) return {
          823  +					tag = t, attrs = a, src = o
          824  +				} end;
          825  +
          826  +				catenate = function(...) return ... end;
          827  +			};
          828  +			toHTML = {
          829  +				elt = elt;
          830  +				tag = function(t,attrs,body)
          831  +					return f('%s%s</%s>', elt(t,attrs), body, t)
          832  +				end;
          833  +				catenate = table.concat;
          834  +			};
          835  +		}
          836  +	end
          837  +
          838  +	local astproc = {
          839  +		toHTML = getRenderers(tagproc.toHTML);
          840  +		toTXT  = getRenderers(tagproc.toTXT);
          841  +		toIR   = { };
          842  +	}
          843  +	astproc.toIR.span_renderers = ss.clone(astproc.toHTML);
          844  +	astproc.toIR.block_renderers = getBlockRenderers(tagproc.toIR,astproc.toHTML);
          845  +		-- note we use HTML here instead of IR span renderers, because as things
          846  +		-- currently stand we don't need that level of resolution. if we ever
          847  +		-- get to the point where we want to be able to twiddle spans around
          848  +		-- we'll need to introduce an IR span renderer
          849  +
          850  +	render_state_handle.astproc = astproc;
          851  +	render_state_handle.tagproc = tagproc;
          852  +
          853  +	-- bind to legacy names
          854  +	-- yikes this needs to be cleaned up so badly
   668    855   	local ir = {}
   669         -	local toc
   670         -	local dr = getRenderers(tag,elt,table.concat) -- default renderers
   671         -	local plainr = getRenderers(function(t,a,v) return v  end,
   672         -	                            function(t,a)   return '' end, table.concat)
   673         -	local irBlockRdrs = getBlockRenderers(
   674         -		function(t,a,v,o) return {tag = t, attrs = a, nodes = type(v) == 'string' and {v} or v, src = o} end,
   675         -		function(t,a,o) return {tag = t, attrs = a, src = o} end,
   676         -		dr, function(...) return ... end)
          856  +	local dr = astproc.toHTML -- default renderers
          857  +	local plainr = astproc.toTXT
          858  +	local irBlockRdrs = astproc.toIR.block_renderers;
   677    859   
          860  +	render_state_handle.ir = ir;
          861  +
          862  +	runhook('ir_assemble', ir)
   678    863   	for i, sec in ipairs(doc.secorder) do
   679    864   		if doctitle == nil and sec.depth == 1 and sec.heading_node then
   680         -			doctitle = plainr.htmlSpan(sec.heading_node.spans, sec.heading_node, sec)
          865  +			doctitle = astproc.toTXT.htmlSpan(sec.heading_node.spans, sec.heading_node, sec)
   681    866   		end
   682    867   		local irs
   683    868   		if sec.kind == 'ordinary' then
   684    869   			if #(sec.blocks) > 0 then
   685    870   				irs = {tag='section',attrs={id = getSafeID(sec)},nodes={}}
   686    871   
          872  +				runhook('ir_section_build', irs, sec)
          873  +				
   687    874   				for i, block in ipairs(sec.blocks) do
   688         -					local rd = irBlockRdrs[block.kind](block,sec)
          875  +					local rd
          876  +					if irBlockRdrs[block.kind] then
          877  +						rd = irBlockRdrs[block.kind](block,sec)
          878  +					else
          879  +						local rdr = renderJob:proc('render',block.kind,'html')
          880  +						if rdr then
          881  +							rd = rdr({
          882  +								state = render_state_handle;
          883  +								tagproc = tagproc.toIR;
          884  +								astproc = astproc.toIR;
          885  +							}, block, sec)
          886  +						end
          887  +					end
   689    888   					if rd then
   690    889   						if opts['heading-anchors'] and block == sec.heading_node then
   691    890   							stylesNeeded.headingAnchors = true
   692    891   							table.insert(rd.nodes, ' ')
   693    892   							table.insert(rd.nodes, {
   694    893   								tag = 'a';
   695    894   								attrs = {href = '#' .. irs.attrs.id, class='anchor'};
   696    895   								nodes = {type(opts['heading-anchors'])=='string' and opts['heading-anchors'] or '&sect;'};
   697    896   							})
   698    897   						end
   699    898   						table.insert(irs.nodes, rd)
          899  +						runhook('ir_section_node_insert', rd, irs, sec)
   700    900   					end
   701    901   				end
   702    902   			end
   703    903   		elseif sec.kind == 'blockquote' then
   704    904   		elseif sec.kind == 'listing' then
   705    905   		elseif sec.kind == 'embed' then
   706    906   		end
   707    907   		if irs then table.insert(ir, irs) end
   708    908   	end
   709    909   
   710    910   	-- restructure passes
          911  +	runhook('ir_restructure_pre', ir)
   711    912   	
   712    913   	---- list insertion pass
   713    914   	local lists = {}
   714    915   	for _, sec in pairs(ir) do
   715    916   		if sec.tag == 'section' then
   716    917   			local i = 1 while i <= #sec.nodes do local v = sec.nodes[i]
   717    918   				if v.tag == 'li' then
................................................................................
   773    974   
   774    975   					sec.nodes[i] = struc
   775    976   				end
   776    977   			end
   777    978   		end
   778    979   	end
   779    980   	
          981  +	runhook('ir_restructure_post', ir)
   780    982   
   781    983   	-- collection pass
   782    984   	local function collect_nodes(t)
   783    985   		local ts = ''
   784    986   		for i,v in ipairs(t) do
   785    987   			if type(v) == 'string' then
   786    988   				ts = ts .. v
   787    989   			elseif v.nodes then
   788         -				ts = ts .. tag(v.tag, v.attrs, collect_nodes(v.nodes))
          990  +				ts = ts .. tagproc.toHTML.tag(v.tag, v.attrs, collect_nodes(v.nodes))
   789    991   			elseif v.text then
   790         -				ts = ts .. tag(v.tag,v.attrs,v.text)
          992  +				ts = ts .. tagproc.toHTML.tag(v.tag,v.attrs,v.text)
   791    993   			else
   792         -				ts = ts .. elt(v.tag,v.attrs)
          994  +				ts = ts .. tagproc.toHTML.elt(v.tag,v.attrs)
   793    995   			end
   794    996   		end
   795    997   		return ts
   796    998   	end
   797    999   	local body = collect_nodes(ir)
   798   1000   
   799   1001   	for k in pairs(langsused) do
................................................................................
   873   1075   	end
   874   1076   
   875   1077   	local head = {}
   876   1078   	local styletag = ''
   877   1079   	if opts['link-css'] then
   878   1080   		local css = opts['link-css']
   879   1081   		if type(css) ~= 'string' then ct.exns.mode('must be a string', 'html:link-css'):throw() end
   880         -		styletag = styletag .. elt('link',{rel='stylesheet',type='text/css',href=opts['link-css']})
         1082  +		styletag = styletag .. tagproc.toHTML.elt('link',{rel='stylesheet',type='text/css',href=opts['link-css']})
   881   1083   	end
   882   1084   	if next(styles) then
   883   1085   		if opts['gen-styles'] then
   884         -			styletag = styletag .. tag('style',{type='text/css'},table.concat(styles))
         1086  +			styletag = styletag .. tagproc.toHTML.tag('style',{type='text/css'},table.concat(styles))
   885   1087   		end
   886   1088   		table.insert(head, styletag)
   887   1089   	end
   888   1090   
   889   1091   	if opts['fossil-uv'] then
   890         -		return tag('div',{class='fossil-doc',['data-title']=doctitle},styletag .. body)
         1092  +		return tagproc.toHTML.tag('div',{class='fossil-doc',['data-title']=doctitle},styletag .. body)
   891   1093   	elseif opts.snippet then
   892   1094   		return styletag .. body
   893   1095   	else
   894   1096   		return dr.htmlDoc(doctitle, next(head) and table.concat(head), body)
   895   1097   	end
   896   1098   end
   897   1099   
................................................................................
  1046   1248   	end
  1047   1249   	flush()
  1048   1250   	return spans
  1049   1251   end
  1050   1252   
  1051   1253   local function
  1052   1254   blockwrap(fn)
  1053         -	return function(l,c)
  1054         -		local block = fn(l,c)
         1255  +	return function(l,c,j)
         1256  +		local block = fn(l,c,j)
  1055   1257   		block.origin = c:clone();
  1056   1258   		table.insert(c.sec.blocks, block);
         1259  +		j:hook('block_insert', c, block, l)
  1057   1260   	end
  1058   1261   end
  1059   1262   
  1060   1263   local insert_paragraph = blockwrap(function(l,c)
  1061   1264   	if l:sub(1,1) == '.' then l = l:sub(2) end
  1062   1265   	return {
  1063   1266   		kind = "paragraph";
  1064   1267   		spans = ct.parse_span(l, c);
  1065   1268   	}
  1066   1269   end)
  1067   1270   
  1068         -local insert_section = function(l,c)
         1271  +local insert_section = function(l,c,j)
  1069   1272   	local depth, id, t = l:match '^([#§]+)([^%s]*)%s*(.-)$'
  1070   1273   	if id and id ~= "" then
  1071   1274   		if c.doc.sections[id] then
  1072   1275   			c:fail('duplicate section name “%s”', id)
  1073   1276   		end
  1074   1277   	else id = nil end
  1075   1278   
................................................................................
  1085   1288   			origin = s.origin;
  1086   1289   			captions = s;
  1087   1290   		}
  1088   1291   		table.insert(s.blocks, heading)
  1089   1292   		s.heading_node = heading
  1090   1293   	end
  1091   1294   	c.sec = s
         1295  +
         1296  +	j:hook('section_attach', c, s)
  1092   1297   end
  1093   1298   
  1094         -local dsetmeta = function(w,c)
         1299  +local dsetmeta = function(w,c,j)
  1095   1300   	local key, val = w(1)
  1096   1301   	c.doc.meta[key] = val
         1302  +	j:hook('metadata_set', key, val)
  1097   1303   end
  1098   1304   local dextctl = function(w,c)
  1099   1305   	local mode, exts = w(1)
  1100   1306   	for e in exts:gmatch '([^%s]+)' do
  1101   1307   		if mode == 'uses' then
  1102   1308   		elseif mode == 'needs' then
  1103   1309   		elseif mode == 'inhibits' then
................................................................................
  1109   1315   	c.hide_next = mode == 'unless'
  1110   1316   end;
  1111   1317   ct.directives = {
  1112   1318   	author = dsetmeta;
  1113   1319   	license = dsetmeta;
  1114   1320   	keywords = dsetmeta;
  1115   1321   	desc = dsetmeta;
  1116         -	toc = function(w,c)
  1117         -		local toc, op, val = w(2)
  1118         -		if op == nil then
  1119         -			table.insert(c.sec.blocks, {kind='toc'})
  1120         -		end
  1121         -	end;
  1122   1322   	when = dcond;
  1123   1323   	unless = dcond;
  1124   1324   	expand = function(w,c)
  1125   1325   		local _, m = w(1)
  1126   1326   		if m ~= 'off' then
  1127   1327   			c.expand_next = 1
  1128   1328   		else
  1129   1329   			c.expand_next = 0
  1130   1330   		end
  1131   1331   	end;
  1132   1332   }
  1133   1333   
  1134         -local function insert_table_row(l,c)
         1334  +local function insert_table_row(l,c,j)
  1135   1335   	local row = {}
  1136   1336   	local buf
  1137   1337   	local flush = function()
  1138   1338   		if buf then
  1139   1339   			buf.str = buf.str:gsub('%s+$','')
  1140   1340   			table.insert(row, buf)
  1141   1341   		end
................................................................................
  1177   1377   	if buf.str ~= '' then flush() end 
  1178   1378   	for _,v in pairs(row) do
  1179   1379   		v.spans = ct.parse_span(v.str, c)
  1180   1380   	end
  1181   1381   	if #c.sec.blocks > 1 and c.sec.blocks[#c.sec.blocks].kind == 'table' then
  1182   1382   		local tbl = c.sec.blocks[#c.sec.blocks]
  1183   1383   		table.insert(tbl.rows, row)
         1384  +		j:hook('block_table_attach', c, tbl, row, l)
         1385  +		j:hook('block_table_row_insert', c, tbl, row, l)
  1184   1386   	else
  1185         -		table.insert(c.sec.blocks, {
         1387  +		local tbl = {
  1186   1388   			kind = 'table';
  1187   1389   			rows = {row};
  1188   1390   			origin = c:clone();
  1189         -		})
         1391  +		}
         1392  +		table.insert(c.sec.blocks, tbl)
         1393  +		j:hook('block_table_insert', c, tbl, l)
         1394  +		j:hook('block_table_row_insert', c, tbl, tbl.rows[1], l)
  1190   1395   	end
  1191   1396   end
  1192   1397   
  1193   1398   ct.ctlseqs = {
  1194   1399   	{seq = '.', fn = insert_paragraph};
  1195   1400   	{seq = '¶', fn = insert_paragraph};
  1196   1401   	{seq = '❡', fn = insert_paragraph};
  1197   1402   	{seq = '#', fn = insert_section};
  1198   1403   	{seq = '§', fn = insert_section};
  1199   1404   	{seq = '+', fn = insert_table_row};
  1200   1405   	{seq = '|', fn = insert_table_row};
  1201   1406   	{seq = '│', fn = insert_table_row};
         1407  +	{seq = '!', fn = function(l,c,j) 
         1408  +		local last = c.sec.blocks[#c.sec.blocks]
         1409  +		local txt = l:match '^%s*!%s*(.-)$'
         1410  +		if (not last) or last.kind ~= 'aside' then
         1411  +			local aside = {
         1412  +				kind = 'aside';
         1413  +				lines = { ct.parse_span(txt, c) }
         1414  +			}
         1415  +			c:insert(aside)
         1416  +			j:hook('block_aside_insert', c, aside, l)
         1417  +			j:hook('block_aside_line_insert', c, aside, aside.lines[1], l)
         1418  +			j:hook('block_insert', c, aside, l)
         1419  +		else
         1420  +			local sp = ct.parse_span(txt, c)
         1421  +			table.insert(last.lines, sp)
         1422  +			j:hook('block_aside_attach', c, last, sp, l)
         1423  +			j:hook('block_aside_line_insert', c, last, sp, l)
         1424  +		end
         1425  +	end};
  1202   1426   	{pred = function(s,c) return s:match'^[*:]' end, fn = blockwrap(function(l,c) -- list
  1203   1427   		local stars = l:match '^([*:]+)'
  1204   1428   		local depth = utf8.len(stars)
  1205   1429   		local id, txt = l:sub(#stars+1):match '^(.-)%s*(.-)$'
  1206   1430   		local ordered = stars:sub(#stars) == ':'
  1207   1431   		if id == '' then id = nil end
  1208   1432   		return {
  1209   1433   			kind = 'list-item';
  1210   1434   			depth = depth;
  1211   1435   			ordered = ordered;
  1212   1436   			spans = ct.parse_span(txt, c);
  1213   1437   		}
  1214   1438   	end)};
  1215         -	{seq = '\t', fn = function(l,c)
         1439  +	{seq = '\t', fn = function(l,c,j)
  1216   1440   		local ref, val = l:match '\t+([^:]+):%s*(.*)$'
  1217   1441   		c.sec.refs[ref] = val
         1442  +		j:hook('section_ref_attach', c, ref, val, l)
  1218   1443   	end};
  1219         -	{seq = '%', fn = function(l,c) -- directive
         1444  +	{seq = '%', fn = function(l,c,j) -- directive
  1220   1445   		local crit, cmdline = l:match '^%%([!%%]?)%s*(.*)$'
  1221   1446   		local words = function(i)
  1222   1447   			local wds = {}
  1223   1448   			if i == 0 then return cmdline end
  1224   1449   			for w,pos in cmdline:gmatch '([^%s]+)()' do
  1225   1450   				table.insert(wds, w)
  1226   1451   				i = i - 1
................................................................................
  1229   1454   					return table.unpack(wds)
  1230   1455   				end
  1231   1456   			end
  1232   1457   		end
  1233   1458   
  1234   1459   		local cmd, rest = words(1)
  1235   1460   		if ct.directives[cmd] then
  1236         -			ct.directives[cmd](words,c)
         1461  +			ct.directives[cmd](words,c,j)
  1237   1462   		elseif cmd == c.doc.stage.mode['render:format'] then
  1238   1463   			-- this is a directive for the renderer; insert it into the tree as is
  1239         -			c:insert {
         1464  +			local dir = {
  1240   1465   				kind = 'directive';
  1241   1466   				critical = crit == '!';
  1242   1467   				words = words;
  1243   1468   			}
         1469  +			c:insert(dir)
         1470  +			j:hook('block_directive_render', j, c, dir)
         1471  +		elseif c.doc:allow_ext(cmd) then -- extension directives begin with their id
         1472  +			local ext = ct.ext.loaded[cmd]
         1473  +			if ext.directives then
         1474  +				local _, topcmd = words(2)
         1475  +				if ext.directives[topcmd] then
         1476  +					ext.directives[topcmd](j:delegate(ext), c, words)
         1477  +				elseif ext.directives[true] then -- catch-all
         1478  +					ext.directives[true](j:delegate(ext), c, words)
         1479  +				elseif crit == '!' then
         1480  +					c:fail('extension %s does not support critical directive %s', cmd, topcmd)
         1481  +				end
         1482  +			end
  1244   1483   		elseif crit == '!' then
  1245   1484   			c:fail('critical directive %s not supported',cmd)
  1246   1485   		end
  1247   1486   	end;};
  1248         -	{seq = '~~~', fn = blockwrap(function(l,c)
         1487  +	{seq = '~~~', fn = blockwrap(function(l,c,j)
  1249   1488   		local extract = function(ptn, str)
  1250   1489   			local start, stop = str:find(ptn)
  1251   1490   			if not start then return nil, str end
  1252   1491   			local ex = str:sub(start,stop)
  1253   1492   			local n = str:sub(1,start-1) .. str:sub(stop+1)
  1254   1493   			return ex, n
  1255   1494   		end
................................................................................
  1260   1499   			lang, s = extract('%b[]', s)
  1261   1500   			if lang then lang = lang:sub(2,-2) end
  1262   1501   			id, title = extract('#[^%s]+', s)
  1263   1502   			if id then id = id:sub(2) end
  1264   1503   		elseif l:match '^~~~' then -- MD shorthand style
  1265   1504   			lang = l:match '^~~~%s*(.-)%s*$'
  1266   1505   		end
  1267         -		c.mode = {
         1506  +		local mode = {
  1268   1507   			kind = 'code';
  1269   1508   			listing = {
  1270   1509   				kind = 'listing';
  1271   1510   				lang = lang, id = id, title = title and ct.parse_span(title,c);
  1272   1511   				lines = {};
  1273   1512   			}
  1274   1513   		}
         1514  +		j:hook('mode_switch', c, mode)
         1515  +		c.mode = mode
  1275   1516   		if id then
  1276   1517   			if c.sec.refs[id] then c:fail('duplicate ID %s', id) end
  1277   1518   			c.sec.refs[id] = c.mode.listing
  1278   1519   		end
         1520  +		j:hook('block_insert', c, mode.listing, l)
  1279   1521   		return c.mode.listing;
  1280   1522   	end)};
  1281   1523   	{pred = function(s,c)
  1282   1524   		if s:match '^[%-_][*_%-%s]+' then return true end
  1283   1525   		if startswith(s, '—') then
  1284   1526   			for c, p in eachcode(s) do
  1285   1527   				if ({
................................................................................
  1294   1536   	end)};
  1295   1537   	{fn = insert_paragraph};
  1296   1538   }
  1297   1539   
  1298   1540   function ct.parse(file, src, mode)
  1299   1541   	local function
  1300   1542   	is_whitespace(cp)
  1301         -		return cp == 0x20
         1543  +		return cp == 0x20 or cp == 0xe390
  1302   1544   	end
  1303   1545   
  1304   1546   	local ctx = ct.ctx.mk(src)
  1305   1547   	ctx.line = 0
  1306   1548   	ctx.doc = ct.doc.mk()
  1307   1549   	ctx.doc.src = src
  1308   1550   	ctx.doc.stage = {
  1309   1551   		kind = 'parse';
  1310   1552   		mode = mode;
  1311   1553   	}
  1312   1554   	ctx.sec = ctx.doc:mksec() -- toplevel section
  1313   1555   	ctx.sec.origin = ctx:clone()
  1314   1556   
         1557  +	-- create states for extension hooks
         1558  +	local job = ctx.doc:job('parse',nil,ctx)
         1559  +
  1315   1560   	for full_line in file:lines() do ctx.line = ctx.line + 1
  1316   1561   		local l
  1317   1562   		for p, c in utf8.codes(full_line) do
  1318   1563   			if not is_whitespace(c) then
  1319   1564   				l = full_line:sub(p)
  1320   1565   				break
  1321   1566   			end
  1322   1567   		end
         1568  +		job:hook('line_read',ctx,l)
         1569  +
  1323   1570   		if ctx.mode then
  1324   1571   			if ctx.mode.kind == 'code' then
  1325   1572   				if l and l:match '^~~~%s*$' then
         1573  +					job:hook('block_listing_end',ctx,ctx.mode.listing)
         1574  +					job:hook('mode_switch', c, nil)
  1326   1575   					ctx.mode = nil
  1327   1576   				else
  1328   1577   					-- TODO handle formatted code
  1329         -					table.insert(ctx.mode.listing.lines, {l})
         1578  +					local newline = {l}
         1579  +					table.insert(ctx.mode.listing.lines, newline)
         1580  +					job:hook('block_listing_newline',ctx,ctx.mode.listing,newline)
  1330   1581   				end
  1331   1582   			else
  1332   1583   				ctx:fail('unimplemented syntax mode %s', ctx.mode.kind)
  1333   1584   			end
  1334   1585   		else
  1335   1586   			if l then
  1336         -				local found = false
  1337         -				for _, i in pairs(ct.ctlseqs) do
  1338         -					if  ((not i.seq ) or startswith(l, i.seq)) and
  1339         -						((not i.pred) or i.pred    (l, ctx  )) then
  1340         -						found = true
  1341         -						i.fn(l, ctx)
  1342         -						break
         1587  +				local function tryseqs(seqs, ...)
         1588  +					for _, i in pairs(seqs) do
         1589  +						if  ((not i.seq ) or startswith(l, i.seq)) and
         1590  +							((not i.pred) or i.pred    (l, ctx  )) then
         1591  +							i.fn(l, ctx, job, ...)
         1592  +							return true
         1593  +						end
  1343   1594   					end
         1595  +					return false
  1344   1596   				end
  1345         -				if not found then
  1346         -					ctx:fail 'incomprehensible input line'
         1597  +
         1598  +				if not tryseqs(ct.ctlseqs) then
         1599  +					local found = false
         1600  +					
         1601  +					for eb, ext, state in job:each('blocks') do
         1602  +						if tryseqs(eb, state) then found = true break end
         1603  +					end
         1604  +
         1605  +					if not found then
         1606  +						ctx:fail 'incomprehensible input line'
         1607  +					end
  1347   1608   				end
  1348   1609   			else
  1349   1610   				if next(ctx.sec.blocks) and ctx.sec.blocks[#ctx.sec.blocks].kind ~= 'break' then
  1350         -					table.insert(ctx.sec.blocks, {kind='break'})
         1611  +					local brk = {kind='break'}
         1612  +					job:hook('block_break', ctx, brk, l)
         1613  +					table.insert(ctx.sec.blocks, brk)
  1351   1614   				end
  1352   1615   			end
  1353   1616   		end
         1617  +		job:hook('line_end',ctx,l)
  1354   1618   	end
  1355   1619   
  1356   1620   	return ctx.doc
  1357   1621   end

Modified ext/toc.lua from [ed743c91e7] to [706a61f3d9].

     1      1   local ct = require 'cortav'
     2      2   local ss = require 'sirsem'
            3  +
            4  +local css_toc = [[
            5  +
            6  +]]
            7  +
            8  +local css_toc_fixed = [[
            9  +	@media (min-width: calc(@[width]:[100vw] + 20em)) {
           10  +		ol.toc {
           11  +			position: fixed;
           12  +			padding-top: 1em; padding-bottom: 1em;
           13  +			padding-right: 1em;
           14  +			margin-top: 0; margin-bottom: 0;
           15  +			right: 0; top: 0; bottom: 0;
           16  +			max-width: calc(50vw - ((@[width]:[0]) / 2) - 3.5em);
           17  +			overflow-y: auto;
           18  +		}
           19  +		@media (max-width: calc(@[width]:[100vw] + 30em)) {
           20  +			ol.toc {
           21  +				max-width: calc(100vw - ((@[width]:[0])) - 9.5em);
           22  +			}
           23  +			body {
           24  +				margin-left: 5em;
           25  +			}
           26  +		}
           27  +	}
           28  +]]
     3     29   
     4     30   ct.ext.install {
     5     31   	id = 'toc';
     6     32   	desc = 'provides a table of contents for HTML renderer plus generic fallback';
     7     33   	version = ss.version {0,1; 'devel'};
     8     34   	contributors = {{name='lexi hale', handle='velartrill', mail='lexi@hale.su', homepage='https://hale.su'}};
     9         -	directive = function(words)
           35  +	default = true; -- on unless inhibited
           36  +	hook = {
           37  +		doc_init = function(job)
           38  +			print('initing doc:toc',job.doc)
           39  +			job.state.toc_custom_position = false
           40  +		end;
           41  +
           42  +		render_html_init = function(job, render)
           43  +			render.stylesets.toc = css_toc
           44  +			render.stylesets.tocFixed = css_toc_fixed
           45  +		end;
           46  +
           47  +		render_html_ir_assemble = function(job, render, ir)
           48  +			-- the custom position state is part of the document job,
           49  +			-- but rendering is a separate job, so we need to get the
           50  +			-- state of this extension in the parent job, which is
           51  +			-- done with the job:unwind(depth) call. unwind is a method
           52  +			-- of the delegate we access the job through which gives us
           53  +			-- direct access to the job state of this extension; unwind
           54  +			-- climbs the jobtree and constructs a similar delegate for
           55  +			-- the nth parent. note that this will only work if the
           56  +			-- current extension hasn't been excluded by predicate from
           57  +			-- the nth parent!
           58  +			if not job:unwind(1).state.toc_custom_position then
           59  +				-- TODO insert %toc end of first section
           60  +			end
           61  +		end;
           62  +	};
           63  +	directives = {
           64  +		mark = function (job, ctx, words) 
           65  +			local _, _, text = words(2)
           66  +			ctx:insert {kind = 'anchor', _toc_label = ct.parse_span(text,ctx)}
           67  +		end;
           68  +		name = function (job, ctx, words) 
           69  +			local _, _, id, text = words(3)
           70  +			ctx:insert {kind = 'anchor', id=id, _toc_label = ct.parse_span(text,ctx)}
           71  +		end;
           72  +		[true] = function (job, ctx, words) 
           73  +			local _, op, val = words(2)
           74  +			if op == nil then
           75  +				local toc = {kind='toc'}
           76  +				ctx:insert(toc)
           77  +				-- same deal here -- directives are processed as part of
           78  +				-- the parse job, which is forked off the document job,
           79  +				-- so we need to climb the jobstack
           80  +				job:unwind(1).state.toc_custom_position = true
           81  +				job:hook('ext_toc_position', ctx, toc)
           82  +			else
           83  +				ctx:fail 'bad %toc directive'
           84  +			end
           85  +		end;
           86  +	};
           87  +	render = {
           88  +		toc = {
           89  +			html = function(job, renderer, block, section)
           90  +				-- “tagproc” contains the functions that determine what kind
           91  +				-- of data our abstract tags will be transformed into. this
           92  +				-- is needed to that plain text, HTML, and HTML IR can be
           93  +				-- produced from the same functions just by varying the
           94  +				-- proc set.
           95  +				-- 
           96  +				-- “astproc” contains the functions that determine what form
           97  +				-- our span arrays (and blocks, but not relevant here) will
           98  +				-- be transformed into, and is analogous to “tagproc”
           99  +				local tag = renderer.tagproc.tag;
          100  +				local elt = renderer.tagproc.elt;
          101  +				local catenate = renderer.tagproc.catenate;
          102  +				local sr = renderer.astproc.span_renderers;
          103  +				local getSafeID = renderer.state.obj_htmlid;
          104  +				
          105  +				-- toplevel HTML IR
          106  +				local lst = {tag = 'ol', attrs={class='toc'}, nodes={}}
          107  +
          108  +				-- "renderer.state" contains the stateglob of the renderer
          109  +				-- itself, not to be confused with the "state" parameter
          110  +				-- which contains this extension's share of the job state
          111  +				-- we use it to activate the stylesets we injected earlier
          112  +				renderer.state.stylesets_active.toc = true
          113  +				if renderer.state.opts['width'] then
          114  +					renderer.state.stylesets_active.tocFixed = true
          115  +				end
          116  +
          117  +				-- assemble a tree of links from the document section
          118  +				-- structure. this is tricky, because we need a tree, but
          119  +				-- all we have is a flat list with depth values attached to
          120  +				-- each node.
          121  +				local stack = {lst}
          122  +				local top = function() return stack[#stack] end
          123  +				-- job.doc is the document the render job is bound to, and
          124  +				-- its secorder field is a list of all the doc's sections in
          125  +				-- the order they occur ("doc.sections" is a hashmap from name
          126  +				-- to section object)
          127  +				local all = job.doc.secorder
          128  +
          129  +				for i, sec in ipairs(all) do
          130  +					if sec.heading_node then -- does this section have a label?
          131  +						local ent = tag('li',nil,
          132  +							 catenate{tag('a', {href='#'..getSafeID(sec)},
          133  +								sr.htmlSpan(sec.heading_node.spans, sec.heading_node, sec))})
          134  +						if sec.depth > #stack then
          135  +							local n = {tag = 'ol', attrs={}, nodes={ent}}
          136  +							table.insert(top().nodes[#top().nodes].nodes, n)
          137  +							table.insert(stack, n)
          138  +						else
          139  +							if sec.depth < #stack then
          140  +								for j=#stack,sec.depth+1,-1 do stack[j] = nil end
          141  +							end
          142  +							table.insert(top().nodes, ent)
          143  +						end
          144  +
          145  +						-- now we need to assemble a list of items within the
          146  +						-- section worthy of an entry on their own. currently
          147  +						-- this is only anchors created with %toc mark|name
          148  +						local innerlinks = {}
          149  +						local noteworthy = { anchor = true }
          150  +						for j, block in pairs(sec.blocks) do
          151  +							if noteworthy[block.kind] then
          152  +								local label = ss.coalesce(block._toc_label, block.label, block.spans)
          153  +								if label then
          154  +									table.insert(innerlinks, {
          155  +										id = renderer.state.obj_htmlid(block);
          156  +										label = label;
          157  +										block = block;
          158  +									})
          159  +								end
          160  +							end
          161  +						end
          162  +
          163  +						if next(innerlinks) then
          164  +							local n = {tag = 'ol', attrs = {}, nodes = {}}
          165  +							for i, l in ipairs(innerlinks) do
          166  +								local nn = {
          167  +									tag = 'a';
          168  +									attrs = {href = '#' .. l.id};
          169  +									nodes = {sr.htmlSpan(l.label, l.block, sec)};
          170  +								}
          171  +								table.insert(n.nodes, {tag = 'li', attrs = {}, nodes={nn}})
          172  +							end
          173  +							table.insert(ent.nodes, n)
          174  +						end
          175  +            print(ss.dump(ent))
          176  +					end
          177  +				end
          178  +				return lst
          179  +			end;
    10    180   
    11         -	end;
          181  +			[true] = function() end; -- fallback // convert to different node types
          182  +		};
          183  +	};
    12    184   }

Modified sirsem.lua from [8f6ee343ec] to [1f16b393f5].

   156    156   					exp = '<' .. state.tbls[p] ..'>'
   157    157   					done = true
   158    158   				else
   159    159   					state.tbls[p] = path and string.format('%s.%s', path, k) or k
   160    160   				end
   161    161   			end
   162    162   			if not done then
   163         -				local function dodump() return dump(
          163  +				local function dodump() return ss.dump(
   164    164   					p, state,
   165    165   					path and string.format("%s.%s", path, k) or k,
   166    166   					depth + 1
   167    167   				) end
   168    168   				-- boy this is ugly
   169    169   				if type(p) ~= 'table' or
   170    170   					getmetatable(p) == nil or
................................................................................
   364    364   	if mm.__name == 'class' then
   365    365   		return g
   366    366   	else
   367    367   		return nil
   368    368   	end
   369    369   end
   370    370   
          371  +function ss.walk(o, key, ...)
          372  +	if o[key] then
          373  +		if select('#', ...) == 0 then
          374  +			return o[key]
          375  +		else
          376  +			return ss.walk(o[key], ...)
          377  +		end
          378  +	end
          379  +	return nil
          380  +end
          381  +
          382  +function ss.coalesce(x, ...)
          383  +	if x ~= nil then
          384  +		return x
          385  +	elseif select('#', ...) == 0 then
          386  +		return nil
          387  +	else
          388  +		return ss.coalesce(...)
          389  +	end
          390  +end