cortav  Check-in [e551f71321]

* custom style {span .|id|[$styled-text]}: applies a specially defined font style. for example, if you have defined [`caution] to mean "demibold italic underline", cortav will try to apply the proper weight and styling within the constraints of the current font to the span [$styled-text]. see the [>fonts-sty fonts section] for more information about this mechanism.
* literal {obj `|styled-text}: indicates that its text is a reference to a literal sequence of characters or other discrete token. generally rendered in monospace
* variable {obj $|styled-text}: indicates to the reader that its text is a placeholder, rather than a literal representation. generally rendered in italic monospace, ideally of a different color
* underline {obj _|styled-text}: underlines the text. use sparingly on text intended for webpages -- underlined text  [!is] distinct from links, but underlining non-links is still a violation of convention.
* strikeout {obj ~|styled-text}: indicates that its text should be struck through or otherwise indicated for deletion
* insertion {obj +|styled-text}: indicates that its text should be indicated as a new addition to the text body.
** consider using a macro definition [`\edit: [~[#1]][+[#2]]] to save typing if you are doing editing work
* link \[>[!ref] [!styled-text]\]: produces a hyperlink or cross-reference denoted by [$ref], which may be either a URL specified with a reference or the name of an object like an image or section elsewhere in the document. the unicode characters [`→] and [`🔗] can also be used instead of [`>] to denote a link.
* footnote {span ^|ref|[$styled-text]}: annotates the text with a defined footnote. in interactive output media [`\[^citations.qtheo Quantum Theosophy: A Neophyte's Catechism\]] will insert a link with the text [`Quantum Theosophy: A Neophyte's Catechism] that, when clicked, causes a footnote to pop up on the screen. for static output media, the text will simply have a superscript integer after it denoting where the footnote is to be found.
* superscript {obj '|[$styled-text]}
* subscript {obj ,|[$styled-text]}
* raw {obj \\ |[$raw-text]}: causes all characters within to be interpreted literally, without expansion. the only special characters are square brackets, which must have a matching closing bracket, and backslashes.
* raw literal \[$\\[!raw-text]\]: shorthand for [\[$[\…]]]
* macro [` \{[!name] [!arguments]}]: invokes a [>ex.mac macro], specified with a reference
* argument {obj #|var}: in macros only, inserts the [$var]-th argument. otherwise, inserts a context variable provided by the renderer.
* raw argument {obj ##|var}: like above, but does not evaluate [$var].
* term {obj &|name}, {span &|name|[$expansion]}: quotes a defined term with a link to its definition, optionally with a custom expansion of the term (for instance, to expand the first use of an acronym)
* inline image {obj &@|name}: shows a small image or other object inline. the unicode character [`🖼] can also be used instead of [`&@].
* unicode codepoint {obj U+|hex-integer}: inserts an arbitrary UCS codepoint in the output, specified by [$hex-integer]. lowercase [`u] is also legal.
* math mode {obj =|equation}: activates additional transformations on the span to format it as a mathematical equation; e.g. [`*] becomes [`×] and [`/] --> [`÷].
* extension {span %|ext|…}: invokes extension named in [$ext]. [$ext] will usually be an extension name followed by a symbol (often a period) and then an extension-specific directive, although for some simple extensions it may just be the plain extension name. further syntax and semantics depend on the extension. this syntax can also be used to apply formatting specific to certain renderers, such as assigning a CSS class in the [`html] renderer ([`\[%html.myclass my [!styled] text]]).
* critical extension {span %!|ext|…}: like [!extension], but will trigger an error if the requested extension is not available
* extension text {span %:|ext|styled-text}: like [!extension], but when the requested extension is not present, [$styled-text] wlil be emitted as-is. this is a better way to apply CSS classes, as the text will still be visible when rendered to formats other than HTML.
* inline comment {obj %%|...}: ignored. useful for editorial annotations not intended to be part of the rendered product.

	span: [` \[[*[#1]][$[#2]] [#3]\]]
	obj: [` \[[*[#1]][$[#2]]\]]

##tabs tables
tables are encoded using a very simple notation. any line that begins with a plus [`+] or bar [`|] denotes a table row. each plus or bar separates one column from the other: a plus opens a new header cell, a bar opens a new normal cell.
................................................................................
a resource definition in use looks like this:

~~~cortav
this is a demonstration of resources
@smiley
	src: link image/webp http://cdn.example.net/img/smile.webp
		  link image/png file:img/smile.png
		  embed image/gif file img/smile.gif
	desc: the Smiling Man would like to see you in his office
here is the resource in span context [&smiley]
and here it is in block context:
&smiley
~~~

rendered as HTML, this might produce the following:
................................................................................
%% (except that the last wil require embedding)
~~~

inline resources are defined a bit differently:

~~~cortav
@smiling-man-business-card text/plain {
	THE SMILING MAN  | tel. 0-Ω00-666█
	if you can read this | email: nameless@smiles.gov
	it is already too late | address: right behind you
}
@smiling-man-business-card image/png;base64 {
	%% incomprehensible gibbering redacted
}
~~~

for an inline resource, the identifier is followed by a MIME type and an opening bracket. the opening bracket may be any of the characters [`\{][`\[][`(][`<], and can optionally be followed by additional characters to help disambiguate the closing bracket. the closing bracket is determined by "flipping" the opening bracket, producing bracket pairs like the following:
* [`\{:][`:}]
* [`\<!--] [`\--!>]
* [`(*<][`>*)]
* [`<>][`<>] [!(disables nesting!)]
if the open and closing brackets are distinguishable, they will nest appropriately, meaning that [`\{][`\}] alone is very likely to be a safe choice to escape a syntactically correct C program (that doesn't abuse macros too badly). brackets are searched for during parsing; encoded resources are not decoded until a later stage, so a closing bracket character in a base64-encoded text file cannot break out of its escaping.

as a convenience, if the first line of the resource definition begins with a single tab, one tab will be dropped from every following line in order to allow legible indentation. similarly, if an opening bracket is followed immediately by a newline, this newline is discarded.

text within a resource definition body is not expanded unless the resource definition is preceded with an [`%[*expand]] directive or the resource MIME type is [`text/x.cortav]. if an expand directive is found, the MIME type will be used to try and determine an appropriate type of formatting, potentially invoking a separate renderer. for example, [`text/html] will invoke the [`html] backend, and [`application/x-troff] will invoke the [`groff] backend. if no suitable renderer is available, expansions will generate only plain text.

two suffixes are accepted: [`;base64] and [`;hex]. the former will decode the presented strings using the base64 algorithm to obtain the resources data; the second will ignore all characters but ASCII hexadecimal digits and derive the resource data byte-by-byte by reading in hexadecimal pairs. for instance, the following sections are equivalent:

~~~cortav
@propaganda text/plain {
	WORLDGOV SAYS
	“don't waste time with unproductive thoughts
	 your wages will be docked accordingly”
}
................................................................................
*** [`application/x-troff] can be used to supply sections of text written in raw [`groff] syntax. these are ignored by other renderers.
*** [`text/html] can be used to supply sections of text written in raw HTML. these are ignored by non-HTML outputs.
*** any MIME-type that matches the type of file being generated by the renderer can be used to include a block of data that will be passed directly to the renderer.
** URI types: additional URI types can be added by extensions or different implementations, but every compliant implementation must support these URIs.
*** [`http], [`https]/[`http+tls]: accesses resources over HTTP. add a [`file] fallback if possible for the benefit of renderers/viewers that do not have internet access abilities.
*** [`file]: references local files. (the meaning of "local" varies depending on the translation format.) absolute paths should begin [`file:/]; the slash should be omitted for relative paths. note that this doesn't have quite the same meaning as in HTML -- [`file] can (and usually should be) used with HTML outputs to refer to resources that reside on the same server. a cortav URI of [`file:/etc/passwd] will actually result in the link [`/etc/passwd], not [`file:///etc/passwd] when converted to HTML. generally, you only should use [`http] when you're referring to a resource that exists on a different domain.
*** [`name]: a special URI used generally for referencing resources that are already installed on a target system and do not need to be embedded or linked, the name and type are enough for a renderer on another machine to locate the correct resource. this is useful mostly for [>fonts fonts], where it's more typical to refer to fonts that are installed on your system rather than providing paths to font files.
*** [`gemini]: accesses resources over the gemini protocol. currently you should really only use this for [`local] resources unless you're using the gemtext renderer backend, since nothing but gemini browsers are liable to support this protocol.
*** [`role]: specifies an abstract resource determined by context, e.g. [`role:backdrop], [`role:body-font]. for use by translators to formats which make provisions for viewer control. a [`role] URI is special in that it is never embedded; it always depends on context — user preferences, environment variables, system stylesheets, what have you — at the time the output file is viewed, rather than the time of the input file being rendered.
* [`desc]: supplies a narrative description of the resources, for use as an "alt-text" when the image cannot be loaded and for screenreaders.
* [`detail]: supplies extra narrative commentary that is displayed contextually, e.g. when the user hovers her mouse cursor over the embedded object. also used for [`desc] if [`desc] is not supplied.

note that in certain cases, full MIME types do not need to be used. say you're defining a font with the [`name] URI -- you can't necessary know what file type the system fonts on another computer are going to be. in this case, you can just write [`font] instead of [`font/ttf] or [`font/woff2] or similar. all cortav needs to know in this case is what abstract kind of object you're referencing. [`groff] fonts (referenced with the [`dit] URI) don't have a specific MIME type either.


................................................................................
~~~

~~~ tables #tab [cortav] ~~~
here is a glossary table.

+ english :+ ranuir + zia ţai  + thaliste        +
| honor   :| tef    | pang     | mbecheve        |
| rakewym :| hirvag | hi phang | nache umwelinde |
| eat     :| fese   | dzia     | rotechqa        |

and now the other way around!

+:english  :| honor |
+:ranuir   :| tef   |
+:zia ţai  :| pang  |
................................................................................
** the [*groff] render backend ignores [$id]

###tsmog transmogrify
a cortav renderer may automatically translate punctuation marks or symbol sequences to superior representations depending on their context. to be compliant this extension should implement, at minimum:
* smart quotes (with consideration for the typographical conventions languages like German or Spanish)
** {dir.d transmogrify|language [$lang]} can be used to explicitly set the language; otherwise, it must be determined from the value of {dir.d pragma|lang}. if this is not present, implementations may fall back on their own methods for determining the language in use, such as command-line flags.
* multigraph to glyph conversion, including at least:
** [`\--] --> "—"
** [`\-->] --> "→"
** [`\<--] -->  "←"

an escape character before any of the sequence characters should prevent the sequence from being rendered. raw nodes (that is, [`\[\…\]] and [`\[`\…\]]) should not be scanned for transmogrification, nor should the contents of code blocks unless marked with the [`%[*expand]] directive

transmogrification shall only take place after all other parsing steps are completed.

###hilite hilite
code can be highlighted according to the formal language it is written in. a compliant hilite implementation must implement basic keyword, symbol, comment, pragma, and literal highlighing for the following formal languages.
* C
* [>lua Lua]

* custom style {span .|id|[$styled-text]}: applies a specially defined font style. for example, if you have defined [`caution] to mean "demibold italic underline", cortav will try to apply the proper weight and styling within the constraints of the current font to the span [$styled-text]. see the [>fonts-sty fonts section] for more information about this mechanism.
* literal {obj `|styled-text}: indicates that its text is a reference to a literal sequence of characters or other discrete token. generally rendered in monospace
* variable {obj $|styled-text}: indicates to the reader that its text is a placeholder, rather than a literal representation. generally rendered in italic monospace, ideally of a different color
* underline {obj _|styled-text}: underlines the text. use sparingly on text intended for webpages -- underlined text  [!is] distinct from links, but underlining non-links is still a violation of convention.
* strikeout {obj ~|styled-text}: indicates that its text should be struck through or otherwise indicated for deletion
* insertion {obj +|styled-text}: indicates that its text should be indicated as a new addition to the text body.
** consider using a macro definition [`\edit: [~[#1]][+[#2]]] to save typing if you are doing editing work
* link [` \[>[$ref] [$styled-text]\]]: produces a hyperlink or cross-reference denoted by [$ref], which may be either a URL specified with a reference or the name of an object like an image or section elsewhere in the document. the unicode characters [`→] and [`🔗] can also be used instead of [`>] to denote a link.
* footnote {span ^|ref|[$styled-text]}: annotates the text with a defined footnote. in interactive output media [`\[^citations.qtheo Quantum Theosophy: A Neophyte's Catechism\]] will insert a link with the text [`Quantum Theosophy: A Neophyte's Catechism] that, when clicked, causes a footnote to pop up on the screen. for static output media, the text will simply have a superscript integer after it denoting where the footnote is to be found.
* superscript {obj '|[$styled-text]}
* subscript {obj ,|[$styled-text]}
* raw {obj \\ |[$raw-text]}: causes all characters within to be interpreted literally, without expansion. the only special characters are square brackets, which must have a matching closing bracket, and backslashes.
* raw literal [` \["[$raw-text]\]]: shorthand for a raw inside a literal, that is ["[`[\\…]]]
* macro [` \{[$name] [$arguments]}]: invokes a [>ex.mac macro], specified with a reference
* argument {obj #|var}: in macros only, inserts the [$var]-th argument. otherwise, inserts a context variable provided by the renderer.
* raw argument {obj ##|var}: like above, but does not evaluate [$var].
* term {obj &|name}, {span &|name|[$expansion]}: quotes a defined term with a link to its definition, optionally with a custom expansion of the term (for instance, to expand the first use of an acronym)
* inline image {obj &@|name}: shows a small image or other object inline. the unicode character [`🖼] can also be used instead of [`&@].
* unicode codepoint {obj U+|hex-integer}: inserts an arbitrary UCS codepoint in the output, specified by [$hex-integer]. lowercase [`u] is also legal.
* math mode {obj =|equation}: activates additional transformations on the span to format it as a mathematical equation; e.g. [`*] becomes [`×] and [`/] --> [`÷].
* extension {span %|ext|…}: invokes extension named in [$ext]. [$ext] will usually be an extension name followed by a symbol (often a period) and then an extension-specific directive, although for some simple extensions it may just be the plain extension name. further syntax and semantics depend on the extension. this syntax can also be used to apply formatting specific to certain renderers, such as assigning a CSS class in the [`html] renderer (["[%html.myclass my [!styled] text]]).
* critical extension {span %!|ext|…}: like [!extension], but will trigger an error if the requested extension is not available
* extension text {span %:|ext|[$styled-text]}: like [!extension], but when the requested extension is not present, [$styled-text] wlil be emitted as-is. this is a better way to apply CSS classes, as the text will still be visible when rendered to formats other than HTML.
* inline comment {obj %%|...}: ignored. useful for editorial annotations not intended to be part of the rendered product.

	span: [` \[[*[#1]][$[#2]] [#3]\]]
	obj: [` \[[*[#1]][$[#2]]\]]

##tabs tables
tables are encoded using a very simple notation. any line that begins with a plus [`+] or bar [`|] denotes a table row. each plus or bar separates one column from the other: a plus opens a new header cell, a bar opens a new normal cell.
................................................................................
a resource definition in use looks like this:

~~~cortav
this is a demonstration of resources
@smiley
	src: link image/webp http://cdn.example.net/img/smile.webp
		  link image/png file:img/smile.png
		  embed image/gif file:img/smile.gif
	desc: the Smiling Man would like to see you in his office
here is the resource in span context [&smiley]
and here it is in block context:
&smiley
~~~

rendered as HTML, this might produce the following:
................................................................................
%% (except that the last wil require embedding)
~~~

inline resources are defined a bit differently:

~~~cortav
@smiling-man-business-card text/plain {
	THE SMILING MAN        | tel. 0-Ω00-666█
	if you can read this   | email: nameless@smiles.gov
	it is already too late | address: right behind you
}
@smiling-man-business-card image/png;base64 {
	%% incomprehensible gibbering redacted
}
~~~

for an inline resource, the identifier is followed by a MIME type and an opening bracket. the opening bracket may be any of the characters ["{] ["\[] ["(] ["<], and can optionally be followed by additional characters to help disambiguate the closing bracket. the closing bracket is determined by "flipping" the opening bracket, producing bracket pairs like the following:
* ["{:][`:}]
* ["<!--] ["--!>]
* [`(*<][`>*)]
* [`<>][`<>] [!(disables nesting!)]
if the open and closing brackets are distinguishable, they will nest appropriately, meaning that ["{]["}] alone is very likely to be a safe choice to escape a syntactically correct C program (that doesn't abuse macros too badly). brackets are searched for during parsing; encoded resources are not decoded until a later stage, so a closing bracket character in a base64-encoded text file cannot break out of its escaping.

as a convenience, if the first line of the resource definition begins with a single tab, one tab will be dropped from every following line in order to allow legible indentation. similarly, if an opening bracket is followed immediately by a newline, this newline is discarded.

text within a resource definition body is not expanded unless the resource definition is preceded with an [`%[*expand]] directive or the resource MIME type is [`text/x.cortav]. if an expand directive is found, the MIME type will be used to try and determine an appropriate type of formatting, potentially invoking a separate renderer. for example, [`text/html] will invoke the [`html] backend, and [`application/x-troff] will invoke the [`groff] backend. if no suitable renderer is available, expansions will generate only plain text.

two suffixes are accepted: [`;base64] and [`;hex]. the former will decode the presented strings using the base64 algorithm to obtain the resource's data; the second will ignore all characters but ASCII hexadecimal digits and derive the resource data byte-by-byte by reading in hexadecimal pairs. for instance, the following sections are equivalent:

~~~cortav
@propaganda text/plain {
	WORLDGOV SAYS
	“don't waste time with unproductive thoughts
	 your wages will be docked accordingly”
}
................................................................................
*** [`application/x-troff] can be used to supply sections of text written in raw [`groff] syntax. these are ignored by other renderers.
*** [`text/html] can be used to supply sections of text written in raw HTML. these are ignored by non-HTML outputs.
*** any MIME-type that matches the type of file being generated by the renderer can be used to include a block of data that will be passed directly to the renderer.
** URI types: additional URI types can be added by extensions or different implementations, but every compliant implementation must support these URIs.
*** [`http], [`https]/[`http+tls]: accesses resources over HTTP. add a [`file] fallback if possible for the benefit of renderers/viewers that do not have internet access abilities.
*** [`file]: references local files. (the meaning of "local" varies depending on the translation format.) absolute paths should begin [`file:/]; the slash should be omitted for relative paths. note that this doesn't have quite the same meaning as in HTML -- [`file] can (and usually should be) used with HTML outputs to refer to resources that reside on the same server. a cortav URI of [`file:/etc/passwd] will actually result in the link [`/etc/passwd], not [`file:///etc/passwd] when converted to HTML. generally, you only should use [`http] when you're referring to a resource that exists on a different domain.
*** [`name]: a special URI used generally for referencing resources that are already installed on a target system and do not need to be embedded or linked, the name and type are enough for a renderer on another machine to locate the correct resource. this is useful mostly for [>fonts fonts], where it's more typical to refer to fonts that are installed on your system rather than providing paths to font files.
*** [`gemini]: accesses resources over the gemini protocol. currently you should really only use this for [`embed] resources unless you're using the gemtext renderer backend, since nothing but gemini browsers are liable to support this protocol.
*** [`role]: specifies an abstract resource determined by context, e.g. [`role:backdrop], [`role:body-font]. for use by translators to formats which make provisions for viewer control. a [`role] URI is special in that it is never embedded; it always depends on context — user preferences, environment variables, system stylesheets, what have you — at the time the output file is viewed, rather than the time of the input file being rendered.
* [`desc]: supplies a narrative description of the resources, for use as an "alt-text" when the image cannot be loaded and for screenreaders.
* [`detail]: supplies extra narrative commentary that is displayed contextually, e.g. when the user hovers her mouse cursor over the embedded object. also used for [`desc] if [`desc] is not supplied.

note that in certain cases, full MIME types do not need to be used. say you're defining a font with the [`name] URI -- you can't necessary know what file type the system fonts on another computer are going to be. in this case, you can just write [`font] instead of [`font/ttf] or [`font/woff2] or similar. all cortav needs to know in this case is what abstract kind of object you're referencing. [`groff] fonts (referenced with the [`dit] URI) don't have a specific MIME type either.


................................................................................
~~~

~~~ tables #tab [cortav] ~~~
here is a glossary table.

+ english :+ ranuir + zia ţai  + thaliste        +
| honor   :| tef    | pang     | mbecheve        |
| rakewyrm:| hirvag | hi phang | nache umwelinde |
| eat     :| fese   | dzia     | rotechqa        |

and now the other way around!

+:english  :| honor |
+:ranuir   :| tef   |
+:zia ţai  :| pang  |
................................................................................
** the [*groff] render backend ignores [$id]

###tsmog transmogrify
a cortav renderer may automatically translate punctuation marks or symbol sequences to superior representations depending on their context. to be compliant this extension should implement, at minimum:
* smart quotes (with consideration for the typographical conventions languages like German or Spanish)
** {dir.d transmogrify|language [$lang]} can be used to explicitly set the language; otherwise, it must be determined from the value of {dir.d pragma|lang}. if this is not present, implementations may fall back on their own methods for determining the language in use, such as command-line flags.
* multigraph to glyph conversion, including at least:
** ["--] --> "—"
** ["-->] --> "→"
** ["<--] -->  "←"

an escape character before any of the sequence characters should prevent the sequence from being rendered. raw nodes (that is, ["[\…]] and ["["…]]) should not be scanned for transmogrification, nor should the contents of code blocks unless marked with the [`%[*expand]] directive

transmogrification shall only take place after all other parsing steps are completed.

###hilite hilite
code can be highlighted according to the formal language it is written in. a compliant hilite implementation must implement basic keyword, symbol, comment, pragma, and literal highlighing for the following formal languages.
* C
* [>lua Lua]

		}
	end
	ct.spanctls = {
		{seq = '!', parse = formatter 'emph'};
		{seq = '*', parse = formatter 'strong'};
		{seq = '~', parse = formatter 'strike'};
		{seq = '+', parse = formatter 'insert'};


		{seq = '`\\', parse = rawcode};
		{seq = '\\\\', parse = rawcode};
		{seq = '\\', parse = function(s, c) -- raw
			return {
				kind = 'raw';
				spans = {s};
				origin = c:clone();
			}
		end};

		}
	end
	ct.spanctls = {
		{seq = '!', parse = formatter 'emph'};
		{seq = '*', parse = formatter 'strong'};
		{seq = '~', parse = formatter 'strike'};
		{seq = '+', parse = formatter 'insert'};
		{seq = '"', parse = rawcode};
		-- deprecated
			{seq = '`\\', parse = rawcode};
			{seq = '\\\\', parse = rawcode};
		{seq = '\\', parse = function(s, c) -- raw
			return {
				kind = 'raw';
				spans = {s};
				origin = c:clone();
			}
		end};