cortav  Check-in [5e07b52c57]

# cortav
[*cortav] is a markup language designed to be a simpler, but more capable alternative to markdown. its name derives from the [>dict Ranuir words] [!cor] "writing" and [!tav] "document", translating to something like "(plain) text document".

	dict: http://ʞ.cc/fic/spirals/glossary

%toc

## cortav vs. markdown
the most important difference between cortav and markdown is that cortav is strictly line-oriented. this choice was made to ensure that cortav was relatively easy to parse. so while a simple [$.ct] file may look a bit like a [$.md] file, in reality it's a lot closer to Gemini structured text than any flavor of markdown.

## encoding
a cortav document is made up of a sequence of codepoints. UTF-8 must be supported, but other encodings (such as UTF-32 or C6B) may be supported as well. lines will be derived by splitting the codepoints at the linefeed character or equivalent. note that unearthly encodings like C6B or EBCDIC will need to select their own control sequences.

## structure
cortav is based on an HTML-like block model, where a document consists of sections, which are made up of blocks, which may contain a sequence of spans. flows of text are automatically conjoined into spans, and blocks are separated by one or more newlines. this means that, unlike in markdown, a single logical paragraph [*cannot] span multiple ASCII lines. the primary purpose of this was to ensure ease of parsing, but also, both markdown and cortav are supposed to be readable from within a plain text editor. this is the 21st century. every reasonable text editor supports soft word wrap, and if yours doesn't, that's entirely your own damn fault.

the first character(s) of every line (the "control sequence") indicates the role of that line. if no control sequence is recognized, the sequence [$.] is implied instead. the standard line classes and their associated control sequences are listed below. some control sequences have alternate forms, in order to support modern, readable unicode characters as well as plain ascii text.

* paragraphs (. ¶ ❡): a paragraph is a simple block of text. the backslash control sequence is only necessary if the paragraph text begins with something that would otherwise be interpreted as a control sequence.
* newlines (\): inserts a line break into previous paragraph and attaches the following text. mostly useful for poetry or lyrics.
* section starts (# §): starts a new section. all sections have an associated depth, determined by the number of sequence repetitions (e.g. "###" indicates depth-three"). sections may have headers and IDs; both are optional. IDs, if present, are a sequence of raw-text immediately following the hash marks. if the line has one or more space character followed by styled-text, a header will be attached. the character immediately following the hashes can specify a particular type of section. e.g.:
** [$#] is a simple section break.
** [$#anchor] opens a new section with the ID [$anchor].
** [$# header] opens a new section with the title "header".
** [$#anchor header] opens a new section with both the ID [$anchor] and the title "header".
** [$#>conversation] opens a blockquote section named [$conversation] without a header.
** [$#^id] opens a footnote section for the multiline footnote [$id]. the ID must be specified.
** [$#$id] opens the multiline macro [$id]. the ID must be specified.
** [$#&id mime] opens a new inline object [$id] of type [$mime]. useful for embedding SVGs. the ID and mime type must be specified.
* lists (* :): these are like paragraph nodes, but list nodes that occur next to each other will be arranged so as to show they compose a sequence. depth is determined by the number of stars/colons. like headers, a list entry may have an ID that can be used to refer back to it; it is indicated in the same way. if colons are used, this indicates that the order of the items is signifiant. :-lists and *-lists may be intermixed; however, note than only the last character in the sequence actually controls the depth type.
* directives (%): a directive issues a hint to the renderer in the form of an arbitrary string. directives are normally ignored if they are not supported, but you may cause a warning to be emitted where the directive is not supported with [$%!] or mark a directive critical with [$%!!] so that rendering will entirely fail if it cannot be parsed.
* comments (%%): a comment is a line of text that is simply ignored by the renderer. 
* asides (!): indicates text that diverges from the narrative, and can be skipped without interrupting it. think of it like block-level parentheses. asides which follow one another are merged as paragraphs of the same aside, usually represented as a sort of box. if the first line of an aside contains a colon, the stretch of styled-text from the beginning to the aside to the colon will be treated as a "type heading," e.g. "Warning:"
* code (~~~): a line beginning with ~~~ begins or terminates a block of code. the opening line should look like one of the below
** [$~~~]
** [$~~~ language] (markdown-style shorthand syntax)
** [$~~~ \[language\] ~~~] (cortav syntax)
** [$~~~ \[language\] #id ~~~]
** [$~~~ title ~~~]
** [$~~~ title \[language\] ~~~]
** [$~~~ \[language\] title ~~~]
** [$~~~ title \[language\] #id ~~~]
* reference (tab): a line beginning with a tab is treated as a "reference." references hold out-of-line metadata for preceding text like links and footnotes. a reference consists of an identifier followed by a colon and an arbitrary number of spaces or tabs, followed by text. whether this text is interpreted as raw-text or styled-text depends on the context in which the reference is used.
* quotation (<): a line of the form [$<[!name]> [!quote]] denotes an utterance by [$name].
* blockquote (>): alternate blockquote syntax. can be nested by repeating the 
* subtitle (--): attaches a subtitle to the previous header
* embed (&): embeds a referenced object. can be used to show images or repeat previously defined objects like lists or tables, optionally with a caption.
** &myimg All that remained of the unfortunate blood magic pageant contestants and audience (police photo)
** &$mymacro arg 1|arg 2|arg 3
* break (---): inserts a horizontal rule or other context break; does not end the section. must be followed by newline.
* table cells (+ |): see [>ex.tab table examples].

## styled text
most blocks contain a sequence of spans. these spans are produced by interpreting a stream of [*styled-text] following the control sequence. styled-text is a sequence of codepoints potentially interspersed with escapes. an escape is formed by an open square bracket [$\[] followed by a [*span control sequence], and arguments for that sequence like more styled-text. escapes can be nested.

* strong \[*[!styled-text]\]: causes its text to stand out from the narrative, generally rendered as bold or a brighter color.
* emphatic \[![!styled-text]\]: indicates that its text should be spoken with emphasis, generally rendered as italics
* literal \[$[!styled-text]\]: indicates that its text is a reference to a literal sequence of characters, variable name, or other discrete token. generally rendered in monospace
* link \[>[!ref] [!styled-text]\]: produces a hyperlink or cross-reference denoted by [$ref], which may be either a URL specified with a reference or the name of an object like an image or section elsewhere in the document.
* footnote \[^[!ref] [!styled-text]\]: annotates the text with a defined footnote
* raw \[\\[!raw-text]\]: causes all characters within to be interpreted literally, without expansion. the only special characters are square brackets, which must have a matching closing bracket
* raw literal \[$\\[!raw-text]\]: shorthand for [\[$[\…]]]
* macro \{[!name] [!arguments]}: invokes a [>ex.mac macro], specified with a reference
* argument \[#[!var]\]: in macros only, inserts the [$var]-th argument. otherwise, inserts a context variable provided by the renderer.
* raw argument \[##[!var]\]: like above, but does not evaluate [$var].
* term \[&[!name] ([!label])\]: quotes a defined term with a link to its definition
* inline image \[&@[!name]\]: shows a small image or other object inline

## identifiers
any identifier (including a reference) that is defined within a named section must be referred to from outside that section as [$[!sec].[!obj]], where [$sec] is the ID of the containing section and [$obj] is the ID of the object one wishes to reference.

## context variables
context variables are provided so that cortav renderers can process templates. certain context variables are provided for by the standard. you can test for the presence of a context variable with the directive [$when ctx [!var]].

* cortav.file: the name of the file currently being rendered
* cortav.path: the absolute path of the file currently being rendered
* cortav.time: the current system time
* cortav.date: the current system date
* cortav.page: the number of the page currently being rendered
* cortav.id: the identifier of the renderer
* cortav.hash: the SHA3 hash of the source file being rendered

on systems with environment variables, these may be accessed as context variables by prefixing their name with [$env.].

different renderers may provide context in different ways, such as from command line options or a context file. any predefined variables should carry an appropriate prefix to prevent conflation. 

## directives
* format: gives a hint on how the document should be formatted. the first hint that is understood will be applied; all others will be discarded. standard hints include
** essay
** narrative
** screenplay: uses asides to denote actions, quotes for dialogue
** stageplay: uses asides to denote actions, quotes for dialogue
** manual
** glossary
** news
* author: encodes document authorship
* cols: specifies the number of columns the next object should be rendered with
* include: transcludes another file
* quote: transcludes another file, without expanding the text except for paragraphs 
* embed: where possible, embeds another file as an object within the current one. in HTML this could be accomplished with e.g. an iframe.
* expand: causes the next object (usually a code block) to be fully expanded when it would otherwise not be

##ex examples

~~~ blockquotes #bq [cortav] ~~~
the following excerpts of text were recovered from a partially erased hard drive found in the Hawthorne manor in the weeks after the Incident. context is unknown.

#>
—spoke to the man under the bridge again, the one who likes to bite the heads off the fish, and he suggested i take a brief sabbatical and journey to the Wandering Oak (where all paths meet) in search of inspiration and the forsaken sword of Pirate Queen Granuaile. a capital idea! i shall depart upon the morrow, having honored the Lord Odin and poisoned my accursed minstrels as is tradition—
—can't smell my soul anymore, but that's beside the point entirely—
—that second moon (always have wondered why nobody else seems to notice the damn fool thing except on Michaelmas day). alas, my luck did not endure, and i was soon to find myself knee-deep in—
—just have to see about that, won't we!—
#

the nearest surviving relative of Lord Hawthorne is believed to be a wandering beggar with a small pet meerkat who sells cursed wooden trinkets to unwary children. she will not be contacted, as the officers of the Yard fear her.
~~~

~~~links & notes #lnr [cortav] ~~~
this sentence contains a [>zombo link] to zombo com. you can do anything[^any] at zombo com.
	zombo: https://zombo.com
	any: anything you want
~~~

~~~ macros #mac [cortav] ~~~
the ranuir word {gloss cor|writing}…
	gloss: [*[#1]] “[#2]”
~~~

~~~ tables #tab [cortav] ~~~
here is a glossary table.

+ english :+ ranuir + zia ţai  + thaliste        +
| honor   :| tef    | pang     | mbecheve        |
| rakewym :| hirvag | hi phang | nache umwelinde |
| eat     :| fese   | dzia     | rotechqa        |

and now the other way around!

+:english  :| honor |
+:ranuir   :| tef   |
+:zia ţai  :| pang  |
+:thalishte:| mbecheve |
~~~

## extensions
the cortav specification also specifies a number of extensions that do not have to be supported for a renderer to be compliant. the extension mechanism supports the following directives.

* inhibits: prevents an extension from being used even where available
* uses: turns on an extension that is not specified by the user operating the renderer (e.g. on the command line)
* needs: causes rendering to fail with an error if the extensions are not available

where possible, instead of [$needs x y z], the directive [$when has-ext x y z] should be used instead. this causes the next section to be rendered only if the named extensions are available. [$unless has-ext x y z] can be used to provide an alternative format.

extensions are mainly interacted with through directives. all extension directives must be prefixed with the name of the extension.

### toc
sections that have a title will be included in the table of contents. the table of contents is by default inserted at the break between the first level-1 section and the section immediately following it. you may instead place the directive [$toc] where you wish the TOC to be inserted, or suppress it entirely with [$inhibits toc]. note that some renderers may not display the TOC as part of the document itself.

### smart-quotes
a cortav renderer may automatically translate punctuation marks to other punctuation marks depending on their context. 

### hilite
code can be highlighted according to the formal language it is written in.

### lua
renderers with a lua interpreter available can evaluate lua code:
* [$%lua use [!file]]: evaluates [$file] and makes its definitions available
* [$\[%lua raw [!script]\]]: evaluates [$script] and emits the string it returns (if any) in raw span context.
* [$\[%lua exp [!script]\]]: evaluates [$script] and emits the string it returns (if any) in expanded span context.
* [$%lua raw [!script]]: evaluates [$script] and emits the string array it returns (if any) in raw block context.
* [$%lua exp [!script]]: evaluates [$script] and emits the string array it returns (if any) in expanded block context.

the interpreter should provide a [$cortav] table with the objects:
* ctx: contains context variables

used files should return a table with the following members
* macros: an array of functions that return strings or arrays of strings when invoked. these will be injected into the global macro namespace.

### ts
the [*ts] extension allows documents to be marked up for basic classification constraints and automatically redacted. if you are seriously relying on ts for confidentiality, make damn sure you start the file with [$\[requires ts\]], so that rendering will fail with an error if the extension isn't supported.

ts enables the directives:
* [$ts class [!scope] [!level] (styled-text)]: indicates a classification level for either the while document (scope [!doc]) or the next section (scope [!sec]). if the ts level is below [$level], the section will be redacted or rendering will fail with an error, as appropriate. if styled-text is included, this will be treated as the name of the classification level.
* [$ts word [!scope] [!word] (styled-text)]: indicates a codeword clearance that must be present for the text to render. if styled-text is present, this will be used to render the name of the codeword instead of [$word].
* [$when ts level [!level]]
* [$when ts word [!word]]

ts enables the spans:
* [$\[🔒#[!level] [!styled-text]\]]: redacts the span if the security level is below that specified.
* [$\[🔒.[!word] [!styled-text]\]]: redacts the span if the specified codeword clearance is not enabled.
(the padlock emoji is shorthand for [$%ts].)

ts redacts spans securely; that is, they are simply replaced with an indicator that they have been redacted, without visually leaking the length of the redacted text.

~~~ts-example example ~~~ cortav
%ts word doc sorrowful-pines SORROWFUL PINES

# intercept R1440 TCT S3
this communication between the ambassador of [*POLITY DOORMAT CRIMSON] "Socialist League world Glory" and an unknown noble of [*POLITY ROSE] "the Empire of a Thousand Suns" was intercepted by [*SYSTEM SUPINE WARBLE].

## involved individuals
* (A) [*DOORMAT CRIMSON] Ambassador [🔒.morose-frenzy Hyacinth Autumn-Lotus] (confidence 1.0)
* (B) [*ROSE] Duchess [!UNKNOWN] (confidence 0.4)

## provenance
this communication was retrieved by [🔒#3 automated buoy downlink] from [*SYSTEM SUPINE WARBLE].

%ts level sec 9 ULTRAVIOLET
##> transcript
<A> we may have a problem
<B> Hyacinth, I told you not to contact me without—
<A, shouting> god DAMMIT woman I am trying to SAVE your worthless skin
<B> Hyacinth! your Godforsaken scrambler!
<A> …oh, [!fuck].
(signal lost)
~~~

-- [ʞ] cortav.lua
--  ~ lexi hale <lexi@hale.su>
--  © AGPLv3
--  ? renderer

local ct = { render = {} }

local function hexdump(s)
	local hexlines, charlines = {},{}
	for i=1,#s do
		local line = math.floor((i-1)/16) + 1
		hexlines[line] = (hexlines[line] or '') .. string.format("%02x ",string.byte(s, i))
		charlines[line] = (charlines[line] or '') .. ' ' .. string.gsub(string.sub(s, i, i), '[^%g ]', '\x1b[;35m·\x1b[36;1m') .. ' '
	end
	local str = ''
	for i=1,#hexlines do
		str = str .. '\x1b[1;36m' .. charlines[i] .. '\x1b[m\n' .. hexlines[i] .. '\n'
	end
	return str
end

local function dump(o, state, path, depth)
	state = state or {tbls = {}}
	depth = depth or 0
	local pfx = string.rep('   ', depth)
	if type(o) == "table" then
		local str = ''
		for k,p in pairs(o) do
			local done = false
			local exp
			if type(p) == 'table' then
				if state.tbls[p] then
					exp = '<' .. state.tbls[p] ..'>'
					done = true
				else
					state.tbls[p] = path and string.format('%s.%s', path, k) or k
				end
			end
			if not done then
				local function dodump() return dump(
					p, state,
					path and string.format("%s.%s", path, k) or k,
					depth + 1
				) end
				-- boy this is ugly
				if type(p) ~= 'table' or
					getmetatable(p) == nil or
					getmetatable(p).__tostring == nil then
					exp = dodump()
				end
				if type(p) == 'table' then
					exp = string.format('{\n%s%s}', exp, pfx)
					local meta = getmetatable(p)
					if meta then
						if meta.__tostring then
							exp = tostring(p)
						end
						if meta.__name then
							exp = meta.__name .. ' ' .. exp
						end
					end
				end
			end
			str = str .. pfx .. string.format("%s = %s\n", k, exp)
		end
		return str
	elseif type(o) == "string" then
		return string.format('“%s”', o)
	else
		return tostring(o)
	end
end

local function declare(c)
	local cls = setmetatable({
		__name = c.ident;
	}, {
		__name = 'class';
		__tostring = function() return c.ident or '(class)' end;
	})

	cls.__call = c.call
	cls.__index = function(self, k)
		if c.default and c.default[k] then
			return c.default[k]
		end
		if k == 'clone' then
			return function(self)
				local new = cls.mk()
				for k,v in pairs(self) do
					new[k] = v
				end
				if c.clonesetup then
					c.clonesetup(new, self)
				end
				return new
			end
		elseif k == 'to' then
			return function(self, to, ...)
				if to == 'string' then return tostring(self)
				elseif to == 'number' then return tonumber(self)
				elseif to == 'int' then return math.floor(tonumber(self))
				elseif c.cast and c.cast[to] then
					return c.cast[to](self, ...)
				elseif type(to) == 'table' and getmetatable(to) and getmetatable(to).cvt and getmetatable(to).cvt[cls] then
				else error((c.ident or 'class') .. ' is not convertible to ' .. (type(to) == 'string' and to or tostring(to))) end
			end
		end
		if c.fns then return c.fns[k] end
	end

	if c.cast then
		if c.cast.string then
			cls.__tostring = c.cast.string
		end
		if c.cast.number then
			cls.__tonumber = c.cast.number
		end
	end

	cls.mk = function(...)
		local val = setmetatable(c.mk and c.mk(...) or {}, cls)
		if c.init then
			for k,v in pairs(c.init) do
				val[k] = v
			end
		end
		if c.construct then
			c.construct(val, ...)
		end
		return val
	end
	getmetatable(cls).__call = function(_, ...) return cls.mk(...) end
	cls.is = function(o) return getmetatable(o) == cls end
	return cls
end
ct.exn = declare {
	ident = 'exn';
	mk = function(kind, ...)
		return {
			vars = {...};
			kind = kind;
		}
	end;
	cast = {
		string = function(me)
			return me.kind.report(table.unpack(me.vars))
		end;
	};
	fns = {
		throw = function(me) error(me) end;
	}
}
ct.exnkind = declare {
	ident = 'exn-kind';
	mk = function(desc, report)
		return { desc = desc, report = report }
	end;
	call = function(me, ...)
		return ct.exn(me, ...)
	end;
}

ct.exns = {
	tx = ct.exnkind('translation error', function(msg,...)
		return string.format("(%s:%u) "..msg, ...)
	end)
}

ct.ctx = declare {
	mk = function(src) return {src = src} end;
	ident = 'context';
	cast = {
		string = function(me)
			return string.format("%s:%s [%u]", me.src.file, me.line, me.generation or 0)
		end;
	};
	clonesetup = function(new, old)
		for k,v in pairs(old) do new[k] = v end
		if old.generation then
			new.generation = old.generation + 1
		else
			new.generation = 1
		end
	end;
	fns = {
		fail = function(self, msg, ...)
			ct.exns.tx(msg, self.src.file, self.line or 0, ...):throw()
		end;
		ref = function(self,id)
			if not id:find'%.' then
				local rid = self.sec.refs[id]
				if self.sec.refs[id] then
					return self.sec.refs[id]
				else self:fail("no such ref %s in current section", id or '') end
			else
				local sec, ref = string.match(id, "(.-)%.(.+)")
				local s = self.doc.sections[sec]
				if s then
					if s.refs[ref] then
						return s.refs[ref]
					else self:fail("no such ref %s in section %s", ref, sec) end
				else self:fail("no such section %s", sec) end
			end
		end
	};
}

ct.sec = declare {
	ident = 'section';
	mk = function() return {
		blocks = {};
		refs = {};
		depth = 0;
		kind = 'ordinary';
	} end;
	construct = function(self, id, depth)
		self.id = id
		self.depth = depth
	end;
}

ct.doc = declare {
	ident = 'doc';
	fns = {
		mksec = function(self, id, depth)
			local o = ct.sec(id, depth)
			if id then self.sections[id] = o end
			table.insert(self.secorder, o)
			return o
		end;
	};
	mk = function() return {
		sections = {};
		secorder = {};
		embed = {};
		meta = {};
	} end;
}

local function map(fn, lst)
	local new = {}
	for k,v in pairs(lst) do
		table.insert(new, fn(v,k))
	end
	return new
end
local function reduce(fn, acc, lst)
	for i,v in ipairs(lst) do
		acc = fn(acc, v, i)
	end
	return acc
end
local function fmtfn(str)
	return function(...)
		return string.format(str, ...)
	end
end

function ct.render.html(doc, opts)
	local f = string.format
	local ids = {}
	local canonicalID = {}
	local function getSafeID(obj)
		if canonicalID[obj] then
			return canonicalID[obj]
		elseif obj.id and ids[obj.id] then
			local newid
			local i = 1
			repeat newid = obj.id .. string.format('-%x', i)
				i = i + 1 until not ids[newid]
			ids[newid] = obj
			canonicalID[obj] = newid
			return newid
		else
			local cid = obj.id
			if not cid then
				local i = 1
				repeat cid = string.format('x-%x', i)
					i = i + 1 until not ids[cid]
			end
			ids[cid] = obj
			canonicalID[obj] = cid
			return cid
		end
	end

	local langsused = {}
	local langpairs = {
		lua = { color = 0x9377ff };
		terra = { color = 0xff77c8 };
		c = { name = 'C', color = 0x77ffe8 };
		html = { color = 0xfff877 };
		scheme = { color = 0x77ff88 };
		lisp = { color = 0x77ff88 };
		fortran = { color = 0xff779a };
		python = { color = 0xffd277 };
		python = { color = 0xcdd6ff };
	}

	local stylesets = {
		code = [[
			code {
				background: #000;
				color: #fff;
				font-family: monospace;
				font-size: 90%;
				padding: 3px 5px;
			}
		]];
		abbr = [[
			abbr[title] { cursor: help; }
		]];
		editors_markup = [[]];
		block_code_listing = [[
			section > figure.listing {
				font-family: monospace;
				background: #000;
				color: #fff;
				padding: 0;
				margin: 0.3em 0;
				counter-reset: line-number;
				position: relative;
			}
			section > figure.listing>div {
				white-space: pre-wrap;
				counter-increment: line-number;
				text-indent: -2.3em;
				margin-left: 2.3em;
			}
			section > figure.listing>:is(div,hr)::before {
				width: 1.0em;
				padding: 0.2em 0.4em;
				text-align: right;
				display: inline-block;
				background-color: #333;
				border-right: 1px solid #fff;
				content: counter(line-number);
				margin-right: 0.3em;
			}
			section > figure.listing>hr::before {
				color: #333;
				padding-top: 0;
				padding-bottom: 0;
			}
			section > figure.listing>div::before {
				color: #fff;
			}
			section > figure.listing>div:last-child::before {
				padding-bottom: 0.5em;
			}
			section > figure.listing>figcaption:first-child {
				border: none;
				border-bottom: 1px solid #fff;
			}
			section > figure.listing>figcaption::after {
				display: block;
				float: right;
				font-weight: normal;
				font-style: italic;
				font-size: 70%;
				padding-top: 0.3em;
			}
			section > figure.listing>figcaption {
				font-family: sans-serif;
				font-weight: bold;
				font-size: 130%;
				padding: 0.2em 0.4em;
				border: none;
			}
			section > figure.listing > hr {
				border: none;
				margin: 0;
				height: 0.7em;
				counter-increment: line-number;
			}
		]];
	}

	local stylesNeeded = {}

	local function getSpanRenderers(tag,elt)
		local htmlDoc = function(title, head, body)
			return [[<!doctype html>]] .. tag('html',nil,
				tag('head', nil,
					elt('meta',{charset = 'utf-8'}) ..
					(title and tag('title', nil, title) or '') ..
					(head or '')) ..
				tag('body', nil, body or ''))
		end

		local span_renderers = {}
		local function htmlSpan(spans, block, sec)
			local text = {}
			for k,v in pairs(spans) do
				if type(v) == 'string' then
					table.insert(text,(v:gsub('[<>&]',
						function(x)
							return string.format('&#%02u', string.byte(x))
						end)))
				else
					table.insert(text, span_renderers[v.kind](v, block, sec))
				end
			end
			return table.concat(text)
		end

		function span_renderers.format(sp)
			local tags = { strong = 'strong', emph = 'em', strike = 'del', insert = 'ins', literal = 'code' }
			if sp.style == 'literal' then
				stylesNeeded.code = true
			end
			if sp.style == 'del' or sp.style == 'ins' then
				stylesNeeded.editors_markup = true
			end
			return tag(tags[sp.style],nil,htmlSpan(sp.spans))
		end

		function span_renderers.term(t,b)
			local r = b.origin:ref(t.ref)
			local name = t.ref
			if name:find'%.' then name = name:match '^[^.]*%.(.+)$' end
			if type(r) ~= 'string' then
				b.origin:fail('%s is an object, not a reference', t.ref)
			end
			stylesNeeded.abbr = true
			return tag('abbr',{title=r},next(t.spans) and htmlSpan(t.spans) or name)
		end

		function span_renderers.link(sp,b)
			local href
			if b.origin.doc.sections[sp.ref] then
				href = '#' .. sp.ref
			else
				if sp.addr then href = sp.addr else
					local r = b.origin:ref(sp.ref)
					if type(r) == 'table' then
						href = '#' .. getSafeID(r)
					else href = r end
				end
			end
			return tag('a',{href=href},next(sp.spans) and htmlSpan(sp.spans) or href)
		end
		return {
			span_renderers = span_renderers;
			htmlSpan = htmlSpan;
			htmlDoc = htmlDoc;
		}
	end


	local function getBlockRenderers(tag,elt,sr,catenate)
		local function insert_toc(b,s)
			local lst = {tag = 'ol', attrs={}, nodes={}}
			local stack = {lst}
			local top = function() return stack[#stack] end
			local all = s.origin.doc.secorder
			for i, sec in ipairs(all) do
				if sec.heading_node then
					local ent = tag('li',nil,
						 catenate{tag('a', {href='#'..getSafeID(sec)},
							sr.htmlSpan(sec.heading_node.spans))})
					if sec.depth > #stack then
						local n = {tag = 'ol', attrs={}, nodes={ent}}
						table.insert(top().nodes[#top().nodes].nodes, n)
						table.insert(stack, n)
					else
						if sec.depth < #stack then
							for j=#stack,sec.depth+1,-1 do stack[j] = nil end
						end
						table.insert(top().nodes, ent)
					end
				end
			end
			return lst
		end

		local block_renderers = {
			paragraph = function(b,s)
				return tag('p', nil, sr.htmlSpan(b.spans, b, s), b)
			end;
			label = function(b,s)
				if ct.sec.is(b.captions) then
					local h = math.min(6,math.max(1,b.captions.depth))
					return tag(f('h%u',h), nil, sr.htmlSpan(b.spans, b, s), b)
				else
					-- handle other uses of labels here
				end
			end;
			['list-item'] = function(b,s)
				return tag('li', nil, sr.htmlSpan(b.spans, b, s), b)
			end;
			toc = insert_toc;
			table = function(b,s)
				local tb = {}
				for i, r in ipairs(b.rows) do
					local row = {}
					for i, c in ipairs(r) do
						table.insert(row, tag(c.header and 'th' or 'td',
						{align=c.align}, sr.htmlSpan(c.spans, b)))
					end
					table.insert(tb, tag('tr',nil,catenate(row)))
				end
				return tag('table',nil,catenate(tb))
			end;
			listing = function(b,s)
				stylesNeeded.block_code_listing = true
				local nodes = map(function(l)
					if #l > 0 then
						return tag('div',nil,sr.htmlSpan(l, b, s))
					else
						return elt('hr')
					end
				end, b.lines)
				if b.title then
					table.insert(nodes,1,tag('figcaption',nil,sr.htmlSpan(b.title)))
				end
				langsused[b.lang] = true
				return tag('figure', {class='listing', lang=b.lang, id=b.id and getSafeID(b)}, catenate(nodes))
			end;
			['break'] = function() --[[nop]] end;
		}
		return block_renderers;
	end

	local pspan = getSpanRenderers(function(t,a,v) return v  end,
	                               function(t,a)   return '' end)
	 
	local function getRenderers(tag,elt,catenate)
		local r = getSpanRenderers(tag,elt)
		r.block_renderers = getBlockRenderers(tag,elt,r,catenate)
		return r
	end

	local elt = function(t,attrs)
		return f('<%s%s>', t,
			attrs and reduce(function(a,b) return a..b end, '', 
				map(function(v,k)
					if v == true
						then          return ' '..k
						elseif v then return f(' %s="%s"', k, v)
					end
				end, attrs)) or '')
	end
	local tag = function(t,attrs,body)
		return f('%s%s</%s>', elt(t,attrs), body, t)
	end

	local doctitle
	local ir = {}
	local toc
	local dr = getRenderers(tag,elt,table.concat) -- default renderers
	local plainr = getRenderers(function(t,a,v) return v  end,
	                            function(t,a)   return '' end, table.concat)
	local irBlockRdrs = getBlockRenderers(
		function(t,a,v,o) return {tag = t, attrs = a, nodes = type(v) == 'string' and {v} or v, src = o} end,
		function(t,a,o) return {tag = t, attrs = a, src = o} end,
		dr, function(...) return ... end)

	for i, sec in ipairs(doc.secorder) do
		if doctitle == nil and sec.depth == 1 and sec.heading_node then
			doctitle = plainr.htmlSpan(sec.heading_node.spans, sec.heading_node, sec)
		end
		local irs
		if sec.kind == 'ordinary' then
			if #(sec.blocks) > 0 then
				irs = {tag='section',attrs={id = getSafeID(sec)},nodes={}}

				for i, block in ipairs(sec.blocks) do
					local rd = irBlockRdrs[block.kind](block,sec)
					if rd then table.insert(irs.nodes, rd) end
				end
			end
		elseif sec.kind == 'blockquote' then
		elseif sec.kind == 'listing' then
		elseif sec.kind == 'embed' then
		end
		if irs then table.insert(ir, irs) end
	end

	-- restructure passes
	
	---- list insertion pass
	local lists = {}
	for _, sec in pairs(ir) do
		if sec.tag == 'section' then
			local i = 1 while i <= #sec.nodes do local v = sec.nodes[i]
				if v.tag == 'li' then
					local ltag
					if v.src.ordered
						then ltag = 'ol'
						else ltag = 'ul'
					end
					local last = i>1 and sec.nodes[i-1]
					if last and last.embed == 'list' and not (
						last.ref[#last.ref].src.depth   == v.src.depth and
						last.ref[#last.ref].src.ordered ~= v.src.ordered
					) then
						-- add to existing list
						table.insert(last.ref, v)
						table.remove(sec.nodes, i) i = i - 1
					else
						-- wrap in list
						local newls = {v}
						sec.nodes[i] = {embed = 'list', ref = newls}
						table.insert(lists,newls)
					end
				end
			i = i + 1 end
		end
	end

	for _, sec in pairs(ir) do
		if sec.tag == 'section' then
			for i, elt in pairs(sec.nodes) do
				if elt.embed == 'list' then
					local function fail_nest()
						elt.ref[1].src.origin:fail('improper list nesting')
					end
					local struc = {attrs={}, nodes={}}
					if elt.ref[1].src.ordered then struc.tag = 'ol' else struc.tag = 'ul' end
					if elt.ref[1].src.depth ~= 1 then fail_nest() end

					local stack = {struc}
					local copyNodes = function(old,new)
						for i,v in ipairs(old) do new[#new + i] = v end
					end
					for i,e in ipairs(elt.ref) do
						if e.src.depth > #stack then
							if e.src.depth - #stack > 1 then fail_nest() end
							local newls = {attrs={}, nodes={e}}
							copyNodes(e.nodes,newls)
							if e.src.ordered then newls.tag = 'ol' else newls.tag='ul' end
							table.insert(stack[#stack].nodes[#stack[#stack].nodes].nodes, newls)
							table.insert(stack, newls)
						else
							if e.src.depth < #stack then
								-- pop entries off the stack
								for i=#stack, e.src.depth+1, -1 do stack[i] = nil end
							end
							table.insert(stack[#stack].nodes, e)
						end
					end

					sec.nodes[i] = struc
				end
			end
		end
	end
	

	-- collection pass
	local function collect_nodes(t)
		local ts = ''
		for i,v in ipairs(t) do
			if type(v) == 'string' then
				ts = ts .. v
			elseif v.nodes then
				ts = ts .. tag(v.tag, v.attrs, collect_nodes(v.nodes))
			elseif v.text then
				ts = ts .. tag(v.tag,v.attrs,v.text)
			else
				ts = ts .. elt(v.tag,v.attrs)
			end
		end
		return ts
	end
	local body = collect_nodes(ir)

	for k in pairs(langsused) do
		local spec = langpairs[k] or {color=0xaaaaaa}
		stylesets.block_code_listing = stylesets.block_code_listing .. string.format(
			[[section > figure.listing[lang="%s"]>figcaption::after
				{ content: '%s'; color: #%06x }]],
			k, spec.name or k, spec.color)
	end

	local styles = {}
	for k in pairs(stylesNeeded) do
		table.insert(styles, (stylesets[k]:gsub('%s+',' ')))
	end

	local head = {}
	if next(styles) then
		table.insert(head, tag('style',{type='text/css'},table.concat(styles)))
	end

	if opts.snippet then
		return body
	else
		return dr.htmlDoc(doctitle, next(head) and table.concat(head), body)
	end
end

local function
startswith(str, pfx)
	return string.sub(str, 1, #pfx) == pfx
end

local function eachcode(str, ascode)
	local pos = {
		code = 1;
		byte = 1;
	}
	return function()
		if pos.byte > #str then return nil end
		local thischar = utf8.codepoint(str, pos.byte)
		local lastpos = {
			code = pos.code;
			byte = pos.byte;
			next = pos;
		}
		if not ascode then
			thischar = utf8.char(thischar)
			pos.byte = pos.byte + #thischar
		else
			pos.byte = pos.byte + #utf8.char(thischar)
		end
		pos.code = pos.code + 1
		return thischar, lastpos
	end
end

local function formatter(sty)
	return function(s,c)
		return {
			kind = 'format';
			style = sty;
			spans = ct.parse_span(s, c);
			origin = c:clone();
		}
	end
end
ct.spanctls = {
	{seq = '$', parse = formatter 'literal'};
	{seq = '!', parse = formatter 'emph'};
	{seq = '*', parse = formatter 'strong'};
	{seq = '\\', parse = function(s, c) -- raw
		return s
	end};
	{seq = '$\\', parse = function(s, c) -- raw
		return {
			kind = 'format';
			style = 'literal';
			spans = {s};
			origin = c:clone();
		}
	end};
	{seq = '&', parse = function(s, c)
		local r, t = s:match '^([^%s]+)%s*(.-)$'
		return {
			kind = 'term';
			spans = (t and t ~= "") and ct.parse_span(t, c) or {};
			ref = r; 
			origin = c:clone();
		}
	end};
	{seq = '^', parse = function(s, c)
		local fn, t = s:match '^([^%s]+)%s*(.-)$'
		return {
			kind = 'footnote';
			spans = (t and t~='') and ct.parse_span(t, c) or {};
			ref = fn;
			origin = c:clone();
		}
	end};
	{seq = '>', parse = function(s, c)
		local to, t = s:match '^([^%s]+)%s*(.-)$'
		if not to then c:fail('invalid link syntax >%s', s) end
		if t == "" then t = nil end
		return {
			kind = 'link';
			spans = (t and t~='') and ct.parse_span(t, c) or {};
			ref = to;
			origin = c:clone();
		}
	end};
}

function ct.parse_span(str,ctx)
	local function delimited(start, stop, s)
		local depth = 0
		if not startswith(s, start) then return nil end
		for c,p in eachcode(s) do
			if c == '\\' then
				p.next.byte = p.next.byte + #utf8.char(utf8.codepoint(s, p.next.byte))
				p.next.code = p.next.code + 1
			elseif c == start then
				depth = depth + 1
			elseif c == stop then
				depth = depth - 1
				if depth == 0 then
					return s:sub(1+#start, p.byte - #stop), p.byte -- FIXME
				elseif depth < 0 then
					ctx:fail('out of place %s', stop)
				end
			end
		end

		ctx:fail('%s expected before end of line', stop)
	end
	local buf = ""
	local spans = {}
	local function flush()
		if buf ~= "" then
			table.insert(spans, buf)
			buf = ""
		end
	end
	local skip = false
	for c,p in eachcode(str) do
		if skip == true then
			skip = false
			buf = buf .. c
		elseif c == '\\' then
			skip = true
		elseif c == '[' then
			flush()
			local substr, following = delimited('[',']',str:sub(p.byte))
			p.next.byte = following + p.byte
			local found = false
			for _,i in pairs(ct.spanctls) do
				if startswith(substr, i.seq) then
					found = true
					table.insert(spans, i.parse(substr:sub(1+#i.seq), ctx))
					break
				end
			end
			if not found then
				ctx:fail('no recognized control sequence in [%s]', substr)
			end
		else
			buf = buf .. c
		end
	end
	flush()
	return spans
end


local function
blockwrap(fn)
	return function(l,c)
		local block = fn(l,c)
		block.origin = c:clone();
		table.insert(c.sec.blocks, block);
	end
end

local insert_paragraph = blockwrap(function(l,c)
	if l:sub(1,1) == '.' then l = l:sub(2) end
	return {
		kind = "paragraph";
		spans = ct.parse_span(l, c);
	}
end)

local insert_section = function(l,c)
	local depth, id, t = l:match '^([#§]+)([^%s]*)%s*(.-)$'
	if id and id ~= "" then
		if c.doc.sections[id] then
			c:fail('duplicate section name “%s”', id)
		end
	else id = nil end

	local s = c.doc:mksec(id, utf8.len(depth))
	s.depth = utf8.len(depth)
	s.origin = c:clone()
	s.blocks={}

	if t and t ~= "" then
		local heading = {
			kind = "label";
			spans = ct.parse_span(t,c);
			origin = s.origin;
			captions = s;
		}
		table.insert(s.blocks, heading)
		s.heading_node = heading
	end
	c.sec = s
end

local dsetmeta = function(w,c)
	local key, val = w(1)
	c.doc.meta[key] = val
end
local dextctl = function(w,c)
	local mode, exts = w(1)
	for e in exts:gmatch '([^%s]+)' do
		if mode == 'uses' then
		elseif mode == 'needs' then
		elseif mode == 'inhibits' then
		end
	end
end
local dcond = function(w,c)
	local mode, cond, exp = w(2)
	c.hide_next = mode == 'unless'
end;
ct.directives = {
	author = dsetmeta;
	license = dsetmeta;
	keywords = dsetmeta;
	desc = dsetmeta;
	toc = function(w,c)
		local toc, op, val = w(2)
		if op == nil then
			table.insert(c.sec.blocks, {kind='toc'})
		end
	end;
	when = dcond;
	unless = dcond;
	expand = function(w,c)
		local _, m = w(1)
		if m ~= 'off' then
			c.expand_next = 1
		else
			c.expand_next = 0
		end
	end;
}

local function insert_table_row(l,c)
	local row = {}
	for kind, a1, text, a2 in l:gmatch '([+|])(:?)%s*([^:+|]*)%s*(:?)' do
		local header = kind == '+'
		local align
		if     a1 == ':' and a2 ~= ':' then
			align = 'left'
		elseif a1 == ':' and a2 == ':' then
			align = 'center'
		elseif a1 ~= ':' and a2 == ':' then
			align = 'right'
		end
		table.insert(row, {
			spans = ct.parse_span(text, c);
			align = align;
			header = header;
		})
	end
	if #c.sec.blocks > 1 and c.sec.blocks[#c.sec.blocks].kind == 'table' then
		local tbl = c.sec.blocks[#c.sec.blocks]
		table.insert(tbl.rows, row)
	else
		table.insert(c.sec.blocks, {
			kind = 'table';
			rows = {row};
			origin = c:clone();
		})
	end
end

ct.ctlseqs = {
	{seq = '.', fn = insert_paragraph};
	{seq = '¶', fn = insert_paragraph};
	{seq = '#', fn = insert_section};
	{seq = '§', fn = insert_section};
	{seq = '+', fn = insert_table_row};
	{seq = '|', fn = insert_table_row};
	{seq = '│', fn = insert_table_row};
	{pred = function(s,c) return s:match'^[*:]' end, fn = blockwrap(function(l,c) -- list
		local stars = l:match '^([*:]+)'
		local depth = utf8.len(stars)
		local id, txt = l:sub(#stars+1):match '^(.-)%s*(.-)$'
		local ordered = stars:sub(#stars) == ':'
		if id == '' then id = nil end
		return {
			kind = 'list-item';
			depth = depth;
			ordered = ordered;
			spans = ct.parse_span(txt, c);
		}
	end)};
	{seq = '\t', fn = function(l,c)
		local ref, val = l:match '\t+([^:]+):%s*(.*)$'
		c.sec.refs[ref] = val
	end};
	{seq = '%', fn = function(l,c) -- directive
		local crit, cmdline = l:match '^%%([!%%]?)%s*(.*)$'
		local words = function(i)
			local wds = {}
			if i == 0 then return cmdline end
			for w,pos in cmdline:gmatch '([^%s]+)()' do
				table.insert(wds, w)
				i = i - 1
				if i == 0 then
					return table.unpack(wds), cmdline:sub(pos)
				end
			end
		end

		local cmd, rest = words(1)
		if ct.directives[cmd] then
			ct.directives[cmd](words,c)
		elseif crit == '!' then
			c:fail('critical directive %s not supported',cmd)
		end
	end;};
	{seq = '~~~', fn = blockwrap(function(l,c)
		local extract = function(ptn, str)
			local start, stop = str:find(ptn)
			if not start then return nil, str end
			local ex = str:sub(start,stop)
			local n = str:sub(1,start-1) .. str:sub(stop+1)
			return ex, n
		end
		local lang, id, title
		if l:match '^~~~%s*$' then -- no args
		elseif l:match '^~~~.*~~~%s*$' then -- CT style
			local s = l:match '^~~~%s*(.-)%s*~~~%s*$'
			lang, s = extract('%b[]', s)
			lang = lang:sub(2,-2)
			id, title = extract('#[^%s]+', s)
			if id then id = id:sub(2) end
		elseif l:match '^~~~' then -- MD shorthand style
			lang = l:match '^~~~%s*(.-)%s*$'
		end
		c.mode = {
			kind = 'code';
			listing = {
				kind = 'listing';
				lang = lang, id = id, title = title and ct.parse_span(title,c);
				lines = {};
			}
		}
		if id then
			if c.sec.refs[id] then c:fail('duplicate ID %s', id) end
			c.sec.refs[id] = c.mode.listing
		end
		return c.mode.listing;
	end)};
	{pred = function(s,c)
		if s:match '^[%-_][*_%-%s]+' then return true end
		if startswith(s, '—') then
			for c, p in eachcode(s) do
				if ({
					['—'] = true, ['-'] = true, [' '] = true;
					['*'] = true, ['_'] = true, ['\t'] = true;
				})[c] ~= true then return false end
			end
			return true
		end
	end; fn = blockwrap(function()
		return { kind = 'horiz-rule' }
	end)};
	{fn = insert_paragraph};
}

function ct.parse(file, src)
	local function
	is_whitespace(cp)
		return cp == 0x20
	end

	local ctx = ct.ctx.mk(src)
	ctx.line = 0
	ctx.doc = ct.doc.mk()
	ctx.sec = ctx.doc:mksec() -- toplevel section
	ctx.sec.origin = ctx:clone()

	for full_line in file:lines() do ctx.line = ctx.line + 1
		local l
		for p, c in utf8.codes(full_line) do
			if not is_whitespace(c) then
				l = full_line:sub(p)
				break
			end
		end
		if ctx.mode then
			if ctx.mode.kind == 'code' then
				if l and l:match '^~~~%s*$' then
					ctx.mode = nil
				else
					-- TODO handle formatted code
					table.insert(ctx.mode.listing.lines, {l})
				end
			else
				ctx:fail('unimplemented syntax mode %s', ctx.mode.kind)
			end
		else
			if l then
				local found = false
				for _, i in pairs(ct.ctlseqs) do
					if  ((not i.seq ) or startswith(l, i.seq)) and
						((not i.pred) or i.pred    (l, ctx  )) then
						found = true
						i.fn(l, ctx)
						break
					end
				end
				if not found then
					ctx:fail 'incomprehensible input line'
				end
			else
				if next(ctx.sec.blocks) and ctx.sec.blocks[#ctx.sec.blocks].kind ~= 'break' then
					table.insert(ctx.sec.blocks, {kind='break'})
				end
			end
		end
	end

	return ctx.doc
end

local default_mode = {
	format = 'html';
}

local function main(input, output, log, mode, vars)
	local doc = ct.parse(input.stream, input.src)
	input.stream:close()
	if mode['show-tree'] then
		log:write(dump(doc))
	end

	if not mode.format then
		error 'what output format should i translate the input to?'
	end
	if not ct.render[mode.format] then
		error(string.format('output format “%s” unsupported', mode.format))
	end
	
	output:write(ct.render[mode.format](doc, {}))
end

local inp,outp,log = io.stdin, io.stdout, io.stderr

local function entry_cli()
	local mode, vars, input = default_mode, {}, {
		stream = inp;
		src = {
			file = '(stdin)';
		}
	}

	if arg[1] and arg[1] ~= '' then
		local file = io.open(arg[1], "rb")
		if not file then error('unable to load file ' .. arg[1]) end
		input.stream = file
		input.src.file = arg[1]
	end

	main(input, outp, log, mode, vars)
end

-- local ok, e = pcall(entry_cli)
local ok, e = true, entry_cli()
if not ok then
	local str = 'translation failure'
	local color = false
	if log:seek() == nil then
		-- this is not a very reliable heuristic for detecting
		-- attachment to a tty but it's better than nothing
		if os.getenv('COLORTERM') then
			color = true
		else
			local term = os.getenv('TERM')
			if term:find 'color' then color = true end
		end
	end
	if color then
		str = string.format('\27[1;31m%s\27[m', str)
	end
	log:write(string.format('[%s] %s\n\t%s\n', os.date(), str, e))
	os.exit(1)
end