this is a demonstration of resources
+here is the resource in span context:
+and here it is in block context:
+ +~~~ + +note that empty elements with CSS classes are used in the output, to avoid repeating long image definitions (especially base64 inline encoded ones!) + +### supported parameters +* [`src] (all): specifies where to find the file, what it is, and how to embed it. each line of [`src] should consist of three whitespace-separated words: embed method, MIME type, and URI. +** embed methods +*** [`local]: loads the resource at build time and embeds it into the output file. not all implementations may allow loading remote network resources at build time. +*** [`remote]: only embeds a reference to the location of the resource. use this for e.g. live iframes, dynamic images, or images hosted by a CDN. +*** [`auto]: embeds a reference in file formats where that's practical, and use a remote reference otherwise. +** MIME types: which file types are supported depends on the individual implementation and renderer backend; additionally, extensions can add support for extra types. MIME-types that have no available handler will, where possible, result in an attachment that can be extracted by the user, usually by clicking on a link. however, the following should be usable with all compliant implementations +*** [`image/*] (graphical outputs only) +*** [`video/*] (interactive outputs only) +*** [`image/svg+xml] is handled specially for HTML files, and may or may not be compatible with other renderer backends. +*** [`font/*] can be used with the HTML backend to reference a web font +*** [`font/woff2] can be used with the HTML backend to reference a web font +*** [`text/plain] (will be inserted as a preformatted text block) +*** [`text/css] (can be used when producing HTML files to link in an extra stylesheet, either by embedding it or referencing it from the header) +*** [`text/x-cortav] (will be parsed and inserted as a formatted text block; context variables can be passed to the file with [`ctx.[$var]] parameters) +*** any MIME-type that matches the type of file being generated by the renderer can be used to include a block of data that will be passed directly to the renderer. +** URI types: additional URI types can be added by extensions or different implementations, but every compliant implementation must support these URIs. +*** [`http], [`https]: accesses resources over HTTP. add a [`file] fallback if possible for the benefit of renderers/viewers that do not have internet access abilities. +*** [`file]: references local files. absolute paths should begin [`file:/]; the slash should be omitted for relative paths. note that this doesn't have quite the same meaning as in HTML -- [`file] can (and usually should be) used with HTML outputs to refer to resources that reside on the same server. a cortav URI of [`file:/etc/passwd] will actually result in the link [`/etc/passwd], not [`file:///etc/passwd] when converted to HTML. generally, you only should use [`http] when you're referring to a resource that exists on a different domain. +*** [`name]: a special URI used generally for referencing resources that are already installed on a target system and do not need to be embedded or linked, the name and type are enough for a renderer on another machine to locate the correct resource. this is useful mostly for [>fonts fonts], where it's more typical to refer to fonts that are installed on your system rather than providing paths to font files. +*** [`gemini]: accesses resources over the gemini protocol. currently you should really only use this for [`local] resources unless you're using the gemtext renderer backend, since nothing but gemini browsers are liable to support this protocol. +* [`desc]: supplies a narrative description of the resources, for use as an "alt-text" when the image cannot be loaded and for screenreaders. +* [`detail]: supplies extra narrative commentary that is displayed contextually, e.g. when the user hovers her mouse cursor over the embedded object. + +note that in certain cases, full MIME types do not need to be used. say you're defining a font with the [`name] URI -- you can't necessary know what file type the system fonts on another computer are going to be. in this case, you can just write [`font] instead of [`font/ttf] or [`font/woff2] or similar. all cortav needs to know in this case is what abstract kind of object you're referencing. + + +##ctxvar context variables +context variables are provided so that cortav renderers can process templates. certain context variables are provided for by the standard. you can test for the presence of a context variable with the directive [`%[*when] ctx [$var]]. context variables are accessed with the [` \[#[$name]\]] span. * {def cortav.file} the name of the file currently being rendered * {def cortav.path} the absolute path of the file currently being rendered -* {def cortav.time} the current system time in the form [$[#cortav.time]] -* {def cortav.date} the current system date in the form [$[#cortav.date]] -* {def cortav.datetime} the current system date and time represented in the locale or system-standard manner (e.g. [$[#cortav.datetime]]) +* {def cortav.time} the current system time in the form [`[#cortav.time]] +* {def cortav.date} the current system date in the form [`[#cortav.date]] +* {def cortav.datetime} the current system date and time represented in the locale or system-standard manner (e.g. [`[#cortav.datetime]]) * {def cortav.page} the number of the page currently being rendered * {def cortav.id} the identifier of the renderer * {def cortav.hash} the SHA3 hash of the source file being rendered def: [*[#1]]: -on systems with environment variables, these may be accessed as context variables by prefixing their name with [$env.]. +on systems with environment variables, these may be accessed as context variables by prefixing their name with [`env.]. different renderers may provide context in different ways, such as from command line options or a context file. any predefined variables should carry an appropriate prefix to prevent conflation. -## directives - d: [$%[*[##1]]] -* {d author} encodes document authorship +##fonts fonts +for output backends that support font specification, cortav provides a sophisticated font management system by means of the [!font stack]. + +when a document parse begins, the font stack is empty (unless a default font has already been loaded by an intent file). +when the font stack is empty, cortav does not include font specifications in its output, and thus will use whatever the default of the various rendering programs is. + +to use fonts, we first have to define the fonts as [>rsrc resources]. + +~~~cortav +%% first, we create a new section to namespace the fonts +#^fonts +%% we then define each font as a resource +@serif + src: auto font name:Alegreya + embed font/ttf file:project-fonts/alegreya.ttf + link font/woff2 file:/assets/font/alegreya.woff2 + auto font name:Times New Roman +@sans + src: link font name:Alegreya Sans + link font name:Open Sans + link font name:sans-serif +~~~ + +here we have defined two font families, [`fonts.serif] and [`fonts.sans]. each contains a list of references to fonts which will be tried in order. for example, this could be translated into the following CSS: + +~~~css +@font-face { + font-family: "fontdef-serif"; + src: local("Alegreya"), + url("data:font/ttf;base64,…") format("font/ttf"), + url("/assets/font/alegreya.woff2") format("font/woff2"), + local("Times New Roman"); +} +@font-face { + font-family: "fontdef-sans"; + src: local("Alegreya Sans"), + local("Open Sans"), + local("sans-serif"); +} +~~~ + +there are two things that aren't super clear from the CSS, however. notice how we used [`auto] on a couple of those specs? this means it's up to the renderer to decide whether to link or embed the font. for html, a font specified by name can't really be embedded, but for some file formats, it can be. [`auto] lets us produce valid HTML while still taking advantage of font embedding in other formats. + +now that we have our font families defined, we can use their identifiers with the [`%[*font]] directive to control the font stack. the first thing we need to do is push a new font context. there's two ways we can do this: + fnd: [`%[*font] [#1]] +* {fnd dup} will create a copy of the current font context, allowing us to make some changes and then revert later with the {fnd pop} command. this isn't useful in our case, however, because right now the stack is empty; there's nothing to duplicate. +* {fnd new} will create a brand new empty context for us to work with and push it to the stack. this can also be used to temporarily revert to the system default fonts, and then switch back with {fnd pop}. +* {fnd set} changes one or more entries in the current font context. it can take a space-separated list of arguments in the form [`[$entry]=[$font-id]]. the supported entries are: +** [`body]: the fallback font. if only this is set in a given font context, it will be used for everything +** [`paragraph]: the font used for normal paragraphs +** [`header]: the font used in headers +** [`subtitle]: the font used in subtitles +** [`list]: the font used in lists +** [`table]: the font used in tables +** [`caption]: the font used for captions +* {fnd pop} removes the top context from the font stack. + +note that extensions may consult the font context for their entries specific to them. for instance, [>toc toc] checks for [`toc] before falling back to [`body] and then the default font. + +these commands are enough to give us a very flexible setup. consider the following: + +~~~cortav +%% let's pretend we've also defined the fonts 'title', 'cursive', and 'thin' + +%font new +%font set body=sans header=serif +%font dup +%font header=title +# lorem ipsum dolor +%font pop + +%% we've now set up a default font context, created a new context for the title of the +%% document, and then popped it back off after the title was inserted so that our +%% first font context is active again. everything after that last '%font pop' will +%% be printed in sans, except for headers, which will be printed in 'serif' + +lorem ipsum dolor sit amet, sed consectetur apiscing elit… + +%font dup +%font set body=cursive +> sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. +> Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut +%font pop + +%% above we created a blockquote whose text is printed in a cursive font; afterwards, +%% we simply remove this new context— + +and everything is back the way it was at "lorem ipsum" + +%% the font mechanism is at its most powerful when used with multiline macros: + + cursive-quote: %font dup + %font set body=cursive + > [#1] + %font pop + +%% now, whenever we want a block with a cursive body, we can simply invoke + +&$cursive-quote Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident + +%% without affecting the overall font context. in fact, since 'cursive-quote' creates +%% its context using 'dup', it would import all font specifications besides 'body' +%% from the environment it is invoked in +~~~ + +##dir directives + d: [`%[*[##1]]] +* {d author} encodes document authorship. multiple author directives can be issued to add additional coauthors * {d cols} specifies the number of columns the next object should be rendered with * {d include} transcludes another file +* {d import} reads in the contents of another file as an embeddable section * {d quote} transcludes another file, without expanding the text except for paragraphs * {d embed}, where possible, embeds another file as an object within the current one. in HTML this could be accomplished with e.g. an iframe. * {d expand} causes the next object (usually a code block) to be fully expanded when it would otherwise not be -* {d pragma} supplies semantic data about author intent, the kind of information document contains and hints about how it should be displayed to the user. think of them like offhand remarks to the renderer -- there's no guarantee that it'll pay any attention, but if it does, your document will look better. pragmas have no scope; they affect the entire document. the pragma function exists primarily as a means to allow parameters that would normally need to be specified on e.g. the command line to be encoded in the document instead in a way that multiple implementations can understand. a few standard pragmas are defined. +* {d font} controls the font stack, for outputs that support changing fonts. see [>fonts fonts] for more information. +* {d lang} changes the current language, which is used by extensions to e.g. control typographical conventions, and may be encoded into the output by certain renderers (e.g. HTML). note that quotes and blockquotes can be set to a separate language with a simpler syntax. the language should be notated using IETF language tags +** {d lang is x-ranuir-CR8} sets the current language to Ranuir as spoken in the Central Worlds, written in Corran and encoded using UTF-8. this might be used at the top of a document to set its primary language. +** {d lang push gsw-u-sd-chzh} temporarily switches to Zürich German, e.g. to quote a German passage in an otherwise Ranuir document +** {d lang sec en-US} switches to American English for the duration of a section. does not affect the language stack. +** {d lang pop} drops the current language off the language stack, returning to whatever was pushed or set before it. this would be used, for instance, at the end of a passage +* {d pragma} supplies semantic data about author intent, the kind of information the document contains and hints about how it should be displayed to the user. think of them like offhand remarks to the renderer -- there's no guarantee that it'll pay any attention, but if it does, your document will look better. pragmas have no scope; they affect the entire document. the pragma function exists primarily as a means to allow parameters that would normally need to be specified on e.g. the command line to be encoded in the document instead in a way that multiple implementations can understand. a few standard pragmas are defined. ** {d pragma layout} gives a hint on how the document should be layed out. the first hint that is understood will be applied; all others will be discarded. standard hints include: -*** essay -*** narrative -*** screenplay: uses asides to denote actions, quotes for dialogue -*** stageplay: uses asides to denote actions, quotes for dialogue -*** manual -*** glossary -*** news +*** [`essay] +*** [`narrative] +*** [`screenplay]: uses asides to denote actions, quotes for dialogue +*** [`stageplay]: uses asides to denote actions, quotes for dialogue +*** [`manual] +*** [`glossary] +*** [`news] +*** [`book]: section depths 1-3 gain additional semantics +***: [*part]: the section gets a page to itself to announce the beginning of a new part or appendix +***: [*chapter]: the section is preceded by a page break +***: [*heading]: the section can occur on the same page as text and headings from other sections ** {d pragma accent} specifies an accent hue (in degrees around the color wheel) for renderers which support colorized output ** {d pragma accent-spread} is a factor that controls the "spread" of hues used in the document. if 0, only the accent color will be used; if larger, other hues will be used in addition to the primary accent color. -** {d pragma dark-on-light on|off} controls whether the color scheme used should be light-on-dark or dark-on-light +** {d pragma dark-on-light on\|off} controls whether the color scheme used should be light-on-dark or dark-on-light ** {d pragma page-width} indicates how wide the pages should be +** {d pragma title-page} specifies a section to use as a title page, for renderer backends that support pagination -! note on pragmas: particularly when working with collections of documents, you should not keep formatting metadata in the documents themselves! the best thing to do is to have a makefile for compiling the documents using whatever tools you want to support, and encoding the rendering options in this file (for the reference implementation this currently means as command line arguments, but eventually it will support intent files as well) so they can all be changed in one place; pragmas should instead be used for per-document [*overrides] of default settings. +! note on pragmata: particularly when working with collections of documents, you should not keep formatting metadata in the documents themselves! the best thing to do is to have a makefile for compiling the documents using whatever tools you want to support, and encoding the rendering options in this file (for the reference implementation this currently means as command line arguments, but eventually it will support intent files as well) so they can all be changed in one place; pragmas should instead be used for per-document [*overrides] of default settings. ! a workaround for the lack of intent files in the reference implementation is to have a single pseudo-stylesheet that contains only {d pragma} statements, and then import this file from each individual source file using the {d include} directive. this is suboptimal and recommended only when you need to ensure compatibility between different implementations. ! when creating HTML files, an even better alternative may be to turn off style generation entirely and link in an external, hand-written CSS stylesheet. this is generally the way you should compile sources for existing websites if you aren't going to write your own extension. ##ex examples @@ -187,65 +392,91 @@ +:ranuir :| tef | +:zia ţai :| pang | +:thalishte:| mbecheve | ~~~ -## extensions +##extns extensions the cortav specification also specifies a number of extensions that do not have to be supported for a renderer to be compliant. the extension mechanism supports the following directives. * inhibits: prevents an extension from being used even where available * uses: turns on an extension that is not specified by the user operating the renderer (e.g. on the command line) * needs: causes rendering to fail with an error if the extensions are not available -where possible, instead of [$needs x y z], the directive [$when has-ext x y z] should be used instead. this causes the next section to be rendered only if the named extensions are available. [$unless has-ext x y z] can be used to provide an alternative format. +where possible, instead of [`needs [$x y z]], the directive [`when has-ext [$x y z]] should be used instead. this causes the next section to be rendered only if the named extensions are available. [`unless has-ext [$x y z]] can be used to provide an alternative format. extensions are mainly interacted with through directives. all extension directives must be prefixed with the name of the extension. -### toc -sections that have a title will be included in the table of contents. the table of contents is by default inserted at the break between the first level-1 section and the section immediately following it. you may instead place the directive [$toc] where you wish the TOC to be inserted, or suppress it entirely with [$inhibits toc]. note that some renderers may not display the TOC as part of the document itself. +the reference implementation seeks to support all standardized extensions. it's not quite there yet, however. + +###toc toc +sections that have a title will be included in the table of contents. the table of contents is by default inserted at the break between the first level-1 section and the section immediately following it. you may instead place the directive [`toc] where you wish the TOC to be inserted, or suppress it entirely with [`inhibits toc]. note that some renderers may not display the TOC as part of the document itself. toc provides the directives: -* [$%[*toc]]: insert a table of contents in the specified position. this can be used more than once, but doing so may have confusing, incorrect, or nonsensical results under some renderers, and some may just ignore the directive entirely -* [$%[*toc] mark [!styled-text]]: inserts a TOC entry with the label [!styled-text] pointing to the current location. this can be used to e.g. mark noteworthy images, instances of long quotes or literal blocks, or functions inside an expanded code block. -* [$%[*toc] name [!id styled-text]]: like [$%[*toc] mark] but allows an additional [!id] parameter which specifies the ID the renderer will assign to an anchor element. this is not meaningful for all renderers and when it is, it is up to the renderer to decide what it means. -** the [*html] render backend interprets [!id] as the [$id] element for the anchor tag -** the [*groff] render backend ignores [!id] - -### smart-quotes -a cortav renderer may automatically translate punctuation marks to other punctuation marks depending on their context. - -### hilite -code can be highlighted according to the formal language it is written in. - -### lua +* [`%[*toc]]: insert a table of contents in the specified position. this can be used more than once, but doing so may have confusing, incorrect, or nonsensical results under some renderers, and some may just ignore the directive entirely +* [`%[*toc] mark [$styled-text]]: inserts a TOC entry with the label [$styled-text] pointing to the current location. this can be used to e.g. mark noteworthy images, instances of long quotes or literal blocks, or functions inside an expanded code block. +* [`%[*toc] name [$id styled-text]]: like [`%[*toc] mark] but allows an additional [$id] parameter which specifies the ID the renderer will assign to an anchor element. this is not meaningful for all renderers and when it is, it is up to the renderer to decide what it means. +** the [*html] render backend interprets [$id] as the [`id] element for the anchor tag +** the [*groff] render backend ignores [$id] + +###tsmog transmogrify +a cortav renderer may automatically translate punctuation marks or symbol sequences to superior representations depending on their context. to be compliant this extension should implement, at minimum: +* smart quotes (with consideration for the typographical conventions languages like German or Spanish) +** {dir.d transmogrify|language [$lang]} can be used to explicitly set the language; otherwise, it must be determined from the value of {dir.d pragma|lang}. if this is not present, implementations may fall back on their own methods for determining the language in use, such as command-line flags. +* multigraph to glyph conversion, including at least: +** [`\--] --> "—" +** [`\-->] --> "→" +** [`\<--] --> "←" + +an escape character before any of the sequence characters should prevent the sequence from being rendered. raw nodes (that is, [`\[\…\]] and [`\[`\…\]]) should not be scanned for transmogrification, nor should the contents of code blocks unless marked with the [`%[*expand]] directive + +transmogrification shall only take place after all other parsing steps are completed. + +###hilite hilite +code can be highlighted according to the formal language it is written in. a compliant hilite implementation must implement basic keyword, symbol, comment, pragma, and literal highlighing for the following formal languages. +* C +* [>lua Lua] +* [>html HTML] +* [>scheme Scheme] +* [>terra Terra] +* [>libconfig libconfig] + + lua: https://lua.org + scheme: https://call-cc.org + terra: https://terralang.org + html: https://dev.w3.org/html5/spec-LC/ + libconfig: http://hyperrealm.github.io/libconfig/ + +the highlighter should make use of semantic HTML tags like [`] where possible. + +###lua lua renderers with a lua interpreter available can evaluate lua code: -* [$%lua use [!file]]: evaluates [$file] and makes its definitions available -* [$\[%lua raw [!script]\]]: evaluates [$script] and emits the string it returns (if any) in raw span context. -* [$\[%lua exp [!script]\]]: evaluates [$script] and emits the string it returns (if any) in expanded span context. -* [$%lua raw [!script]]: evaluates [$script] and emits the string array it returns (if any) in raw block context. -* [$%lua exp [!script]]: evaluates [$script] and emits the string array it returns (if any) in expanded block context. +* [`%lua use [!file]]: evaluates [$file] and makes its definitions available +* [`\[%lua raw [!script]\]]: evaluates [$script] and emits the string it returns (if any) in raw span context. +* [` \[%lua exp [!script]\]]: evaluates [$script] and emits the string it returns (if any) in expanded span context. +* [`%lua raw [!script]]: evaluates [$script] and emits the string array it returns (if any) in raw block context. +* [`%lua exp [!script]]: evaluates [$script] and emits the string array it returns (if any) in expanded block context. -the interpreter should provide a [$cortav] table with the objects: +the interpreter should provide a [`cortav] table with the objects: * ctx: contains context variables used files should return a table with the following members * macros: an array of functions that return strings or arrays of strings when invoked. these will be injected into the global macro namespace. -### ts +###ts ts the [*ts] extension allows documents to be marked up for basic classification constraints and automatically redacted. if you are seriously relying on ts for confidentiality, make damn sure you start the file with [$%[*requires] ts], so that rendering will fail with an error if the extension isn't supported. ts enables the directives: -* [$ts class [!scope] [!level] (styled-text)]: indicates a classification level for either the while document (scope [!doc]) or the next section (scope [!sec]). if the ts level is below [$level], the section will be redacted or rendering will fail with an error, as appropriate. if styled-text is included, this will be treated as the name of the classification level. -* [$ts word [!scope] [!word] (styled-text)]: indicates a codeword clearance that must be present for the text to render. if styled-text is present, this will be used to render the name of the codeword instead of [$word]. -* [$when ts level [!level]] -* [$when ts word [!word]] +* [`%[*ts] class [$scope level] ([$styled-text])]: indicates a classification level for either the while document (scope [$doc]) or the next section (scope [$sec]). if the ts level is below [$level], the section will be redacted or rendering will fail with an error, as appropriate. if styled-text is included, this will be treated as the name of the classification level. +* [`%[*ts] word [$scope word] ([$styled-text])]: indicates a codeword clearance that must be present for the text to render. if styled-text is present, this will be used to render the name of the codeword instead of [$word]. +* [`%[*when] ts level [$level]] +* [`%[*when] ts word [$word]] ts enables the spans: -* [$\[🔒#[!level] [!styled-text]\]]: redacts the span if the security level is below that specified. -* [$\[🔒.[!word] [!styled-text]\]]: redacts the span if the specified codeword clearance is not enabled. -(the padlock emoji is shorthand for [$%ts].) +* [`\[🔒#[!level] [$styled-text]\]]: redacts the span if the security level is below that specified. +* [`\[🔒.[!word] [$styled-text]\]]: redacts the span if the specified codeword clearance is not enabled. +(the padlock emoji is shorthand for [`%[*ts]].) ts redacts spans securely; that is, they are simply replaced with an indicator that they have been redacted, without visually leaking the length of the redacted text. ~~~#ts-example example [cortav] ~~~ %ts word doc sorrowful-pines SORROWFUL PINES @@ -268,16 +499,16 @@ Hyacinth! your Godforsaken scrambler! …oh, [!fuck]. (signal lost) ~~~ -# reference implementation -the cortav standard is implemented in [$cortav.lua], found in this repository. only the way [$cortav.lua] interprets the cortav language is defined as a reference implementation; other behaviors are simply how [$cortav.lua] implements the specification and may be copied, ignored, tweaked, violently assaulted, or used as inspiration by a compliant parser. +#refimpl reference implementation +the cortav standard is implemented in [`cortav.lua], found in this repository. only the way [`cortav.lua] interprets the cortav language is defined as a reference implementation; other behaviors are simply how [`cortav.lua] implements the specification and may be copied, ignored, tweaked, violently assaulted, or used as inspiration by a compliant parser. -the reference implementation can be used both as a lua library and from the command line. [$cortav.lua] contains the parser and renderers, [$ext/*] contain various extensions, [$sirsem.lua] contains utility functions, and [$cli.lua] contains the CLI driver. +the reference implementation can be used both as a lua library and from the command line. [`cortav.lua] contains the parser and renderers, [`ext/*] contain various extensions, [`sirsem.lua] contains utility functions, and [`cli.lua] contains the CLI driver. -## lua library +##refimpl-lib lua library there are various ways to use cortav from a lua script; the simplest however is probably to precompile your script with luac and link in the necessary components of the implementation. for instance, say we have the following program ~~~ stdin2html.lua [lua] ~~~ local ct = require 'cortav' local mode = {} @@ -294,40 +525,40 @@ ~~~ $ luac -s -o stdin2html.lc $cortav_repo/{sirsem,cortav,ext/toc}.lua stdin2html.lua ~~~ -and can then be operated with the command [$lua stdin2html.lc], with no further need for the cortav repository files. note that the order of the [$luac] command is important! [$sirsem.lua] must come first, followed by [$cortav.lua], followed by any extensions. your driver script (i.e. the script with the entry point into the application) should always come last. +and can then be operated with the command [`lua stdin2html.lc], with no further need for the cortav repository files. note that the order of the [`luac] command is important! [`sirsem.lua] must come first, followed by [`cortav.lua], followed by any extensions. your driver script (i.e. the script with the entry point into the application) should always come last. -### building custom tools +###refimpl-tools building custom tools generally, most existing file-format conversion tools (cmark, pandoc, and so on) have a crucial limitation: they hardcode specific assumptions like document structure. this means that the files they output are generally not suitable as-is for the users' purposes, and require further munging, usually by hateful shell or perl scripts. some tools do provide libraries end users to use as a basis for designing their own tools, but these are often limited, and in any case the user ends up having to write their own (non-standard) driver. it's no surprise that very few people end up doing this. -[$cortav.lua]'s design lends itself to a more elegant solution. one can of course write their own driver using [$cortav] as a library, but most of the time when you're compiling document sources, you just want a binary you can run from the command line or a makefile. with [$cortav.lua], you can extend its capabilities easily while keeping the same driver. +[`cortav.lua]'s design lends itself to a more elegant solution. one can of course write their own driver using [`cortav] as a library, but most of the time when you're compiling document sources, you just want a binary you can run from the command line or a makefile. with [`cortav.lua], you can extend its capabilities easily while keeping the same driver. -in the [$cortav] spec, extensions are mostly intended to give different implementations the ability to offer extra capabilities, but the reference implementation uses an extension architecture that makes it easy to write and add your own. for each type of new behavior you want to implement, just create a new extension and list it on the make command line: +in the [`cortav] spec, extensions are mostly intended to give different implementations the ability to offer extra capabilities, but the reference implementation uses an extension architecture that makes it easy to write and add your own. for each type of new behavior you want to implement, just create a new extension and list it on the make command line: ~~~ $ nvim ~/dev/my-cortav-exts/imperial-edict.lua $ make cortav extens+=$HOME/dev/my-cortav-exts/*.lua ~~~ -the cortav binary this produces will have all the extra capabilities you personally need, without any need to fork [$cortav.lua] itself or even touch the repository. +the cortav binary this produces will have all the extra capabilities you personally need, without any need to fork [`cortav.lua] itself or even touch the repository. -there's no reason [$cortav.lua] shouldn't be able to load extensions at runtime as well; i just haven't implemented this behavior yet. it probably would only take a few extra lines of code tho. +there's no reason [`cortav.lua] shouldn't be able to load extensions at runtime as well; i just haven't implemented this behavior yet. it probably would only take a few extra lines of code tho. -i will eventually document the extension API, but for now, look at [$ext/toc.lua] for a simple example of how to register an extension. +i will eventually document the extension API, but for now, look at [`ext/toc.lua] for a simple example of how to register an extension. -## command line driver -the [$cortav.lua] command line driver can be run from the repository directory with the command [$lua ./cli.lua], or by first compiling it into a bytecode form that links in all its dependencies. this is the preferred method for installation, as it produces a self-contained executable which loads more quickly, but running the driver in script form may be desirable for development or debugging. +##refimpl-cli command line driver +the [$cortav.lua] command line driver can be run from the repository directory with the command [`lua ./cli.lua], or by first compiling it into a bytecode form that links in all its dependencies. this is the preferred method for installation, as it produces a self-contained executable which loads more quickly, but running the driver in script form may be desirable for development or debugging. -the repository contains a GNU makefile to automate compilation of the reference implementation on unix-like OSes. simply run [$$ make cortav] or [$$ gmake cortav] from the repository root to produce a self-contained bytecode executable that can be installed anywhere on your filesystem, with no dependencies other than the lua interpreter. +the repository contains a GNU makefile to automate compilation of the reference implementation on unix-like OSes. simply run [`$ make cortav] or [`$ gmake cortav] from the repository root to produce a self-contained bytecode executable that can be installed anywhere on your filesystem, with no dependencies other than the lua interpreter. -! note that the makefile strips debugging symbols to save space, so running [$cli.lua] directly as a script may be helpful if you encounter errors and need stacktraces or other debugging information. +! note that the makefile strips debugging symbols to save space, so running [`cli.lua] directly as a script may be helpful if you encounter errors and need stacktraces or other debugging information. -henceforth it will be assumed that you have produced the [$cortav] executable and placed it somewhere in your [$$PATH]; if you are instead running [$cortav.lua] directly as an interpreted script, you'll need to replace [$$ cortav] with [$$ lua ./cli.lua] in incantations. +henceforth it will be assumed that you have produced the [`cortav] executable and placed it somewhere in your [$$PATH]; if you are instead running [`cortav.lua] directly as an interpreted script, you'll need to replace [`$ cortav] with [`$ lua ./cli.lua] in incantations. -when run without commands, [$cortav.lua] will read input from standard input and write to standard output. alternately, a source file can be given as an argument. to write to a specific file instead of the standard output stream, use the [$-o [!file]] flag. +when run without commands, [`cortav.lua] will read input from standard input and write to standard output. alternately, a source file can be given as an argument. to write to a specific file instead of the standard output stream, use the [`-o [!file]] flag. ~~~ $ cortav readme.ct -o readme.html # reads from readme.ct, writes to readme.html $ cortav -o readme.html @@ -334,51 +565,66 @@ # reads from standard input, writes to readme.html $ cortav readme.ct # reads from readme.ct, writes to standard output ~~~ -### switches -[$cortav.lua] offers various switches to control its behavior. +###refimpl-build building +the command line driver is built and installed with a GNU [$make] script. this script accepts the variables shown below with their default values: ++ prefix | [`[$$HOME]/.local] | the path under which the package will be installed ++ build | [`build] | the directory where generated objects will be placed; useful for out-of-tree builds ++ bin-prefix | [`[$$prefix]/bin] | directory to install the executables to" ++ default-format-flags | [`-m html:width 35em] | a list of flags that will be passed by the viewer script to [`cortav] when generating a html fille + +the following targets are supplied to automate the build: +* [`install] builds everything, installs the executable and the viewer script to [$$bin_prefix], and registers the viewer script with XDG +* [`excise] deletes everything installed and deregisters the file handlers (note that the same variables must be passed to [`exicse] as were passed to [`install]!) +* [`clean] deletes build artifacts from the [$$build] directory like it was never there +* [`wipe] is equivalent to [`$ make excise && make clean] + +###refimpl-switches switches +[`cortav.lua] offers various switches to control its behavior. + long + short + function + -| [$--out [!file]] :|:[$-o]:| sets the output file (default stdout) | -| [$--log [!file]] :|:[$-l]:| sets the log file (default stderr) | -| [$--define [!var] [!val]] :|:[$-d]:| sets the context variable [$var] to [$val] | -| [$--mode-set [!mode]] :|:[$-y]:| activates the [>refimpl-mode mode] with ID [!mode] -| [$--mode-clear [!mode]] :|:[$-n]:| disables the mode with ID [!mode] | -| [$--mode [!id] [!val]] :|:[$-m]:| configures mode [!id] with the value [!val] | -| [$--mode-set-weak [!mode]] :|:[$-Y]:| activates the [>refimpl-mode mode] with ID [!mode] if the source file does not specify otherwise -| [$--mode-clear-weak [!mode]] :|:[$-N]:| disables the mode with ID [!mode] if the source file does not specify otherwise -| [$--mode-weak [!id] [!val]] :|:[$-M]:| configures mode [!id] with the value [!val] if the source file does not specify otherwise -| [$--help] :|:[$-h]:| display online help | -| [$--version] :|:[$-V]:| display the interpreter version | +| [`--out [!file]] :|:[`-o]:| sets the output file (default stdout) | +| [`--log [!file]] :|:[`-l]:| sets the log file (default stderr) | +| [`--define [!var] [!val]] :|:[`-d]:| sets the context variable [$var] to [$val] | +| [`--mode-set [!mode]] :|:[`-y]:| activates the [>refimpl-mode mode] with ID [!mode] +| [`--mode-clear [!mode]] :|:[`-n]:| disables the mode with ID [!mode] | +| [`--mode [!id] [!val]] :|:[`-m]:| configures mode [!id] with the value [!val] | +| [`--mode-set-weak [!mode]] :|:[`-Y]:| activates the [>refimpl-mode mode] with ID [!mode] if the source file does not specify otherwise +| [`--mode-clear-weak [!mode]] :|:[`-N]:| disables the mode with ID [$mode] if the source file does not specify otherwise +| [`--mode-weak [!id] [!val]] :|:[`-M]:| configures mode [$id] with the value [$val] if the source file does not specify otherwise +| [`--help] :|:[`-h]:| display online help | +| [`--version] :|:[`-V]:| display the interpreter version | ###refimpl-mode modes -most of [$cortav.lua]'s implementation-specific behavior is controlled by use of [!modes]. these are namespaced options which may have a boolean, string, or numeric value. boolean modes are set with the [$-y] [$-n] flags; other modes use the [$-m] flags. +most of [`cortav.lua]'s implementation-specific behavior is controlled by use of [!modes]. these are namespaced options which may have a boolean, string, or numeric value. boolean modes are set with the [`-y] [`-n] flags; other modes use the [`-m] flags. most modes are defined by the renderer backend. the following modes affect the behavior of the frontend: + ID + type + effect -| [$render:format]:| string | selects the [>refimpl-rend renderer] (default [$html]) -| [$parse:show-tree]:| flag | dumps the parse tree to the log after parsing completes +| [`render:format]:| string | selects the [>refimpl-rend renderer] (default [`html]) +| [`parse:show-tree]:| flag | dumps the parse tree to the log after parsing completes ##refimpl-rend renderers -[$cortav.lua] implements a frontend-backend architecture, separating the parsing stage from the rendering stage. this means new renderers can be added to [$cortav.lua] relatively easily. currently, only an [>refimpl-rend-html HTML renderer] is included; however, a [$groff] backend is planned at some point in the future, so that PDFs and manpages can be generated from cortav files. +[`cortav.lua] implements a frontend-backend architecture, separating the parsing stage from the rendering stage. this means new renderers can be added to [`cortav.lua] relatively easily. currently, only an [>refimpl-rend-html HTML renderer] is included; however, a [`groff] backend is planned at some point in the future, so that PDFs and manpages can be generated from cortav files. ###refimpl-rend-html html -the HTML renderer is activated with the incantation [$-m render:format html]. it is currently the default backend. it produces a single HTML file, optionally with CSS styling data, from a [$.ct] input file. +the HTML renderer is activated with the incantation [`-m render:format html]. it is currently the default backend. it produces a single HTML file, optionally with CSS styling data, from a [`.ct] input file. -it supports the following modes: +####refimpl-rend-html-modes modes +[`html] supports the following modes: -* string (css length) [$html:width] sets a maximum width for the body content in order to make the page more readable on large displays -* number [$html:accent] applies an accent hue to the generated webpage. the hue is specified in degrees, e.g. [$-m html:accent 0] applies a red accent. -* flag [$html:dark-on-light] uses dark-on-light styling, instead of the default light-on-dark -* flag [$html:fossil-uv] outputs an HTML snippet suitable for use with the Fossil VCS webserver. this is intended to be used with the unversioned content mechanism to host rendered versions of documentation written in cortav that's stored in a Fossil repository. -* number [$html:hue-spread] generates a color palette based on the supplied accent hue. the larger the value, the more the other colors diverge from the accent hue. -* string [$html:link-css] generates a document linking to the named stylesheet -* flag [$html:gen-styles] embeds appropriate CSS styles in the document (default on) -* flag [$html:snippet] produces a snippet of html instead of an entire web page. note that proper CSS scoping is not yet implemented (and can't be implemented hygienically since [$scoped] was removed 😢) -* string [$html:title] specifies the webpage titlebar contents (normally autodetected from the document based on headings or directives) +* string (css length) [`html:width] sets a maximum width for the body content in order to make the page more readable on large displays +* number [`html:accent] applies an accent hue to the generated webpage. the hue is specified in degrees, e.g. [$-m html:accent 0] applies a red accent. +* flag [`html:dark-on-light] uses dark-on-light styling, instead of the default light-on-dark +* flag [`html:fossil-uv] outputs an HTML snippet suitable for use with the Fossil VCS webserver. this is intended to be used with the unversioned content mechanism to host rendered versions of documentation written in cortav that's stored in a Fossil repository. +* number [`html:hue-spread] generates a color palette based on the supplied accent hue. the larger the value, the more the other colors diverge from the accent hue. +* string [`html:link-css] generates a document linking to the named stylesheet +* flag [`html:gen-styles] embeds appropriate CSS styles in the document (default on) +* flag [`html:snippet] produces a snippet of html instead of an entire web page. note that proper CSS scoping is not yet implemented (and can't be implemented hygienically since [$scoped] was removed 😢) +* string [`html:title] specifies the webpage titlebar contents (normally autodetected from the document based on headings or directives) +* string [`html:font] specifies the default font to use when rendering as a CSS font specification (e.g. [`-m html:font 'Alegreya, Junicode, Georgia, "Times New Roman"]) ~~~ $ cortav readme.ct --out readme.html \ -m render:format html \ -m html:width 40em \ @@ -386,18 +632,118 @@ -m html:hue-spread 35 \ -y html:dark-on-light # could also be written as: $ cortav readme.ct -ommmmy readme.html render:format html html:width 40em html:accent 80 html:hue-spread 35 html:dark-on-light ~~~ +#### directives +[`html] supplies the following render directives. + +* [`%[*html] link [$rel] [$mime] [$href]]: inserts a [`] tag in the header, for example, to link in an alternate stylesheet, or help feed readers find your atom or rss feed. +** [`%[*html] link alternate\\ stylesheet text/css /res/style2.css] +** [`%[*html] link alternate application/atom+xml /feed.atom] +* [`%[*html] style [$id]]: adds the stylesheet referenced by [$id] into the document stylesheet. the stylesheet is specified using a [>rsrc resource]. + +#### stylsheets +the [`html] backend offers some additional directives for external CSS files that are embedded into the document, in order to simplify integration with the accent mechanism. these are: + +* [`@[*fg]]: resolves to a color expression denoting the selected foreground color. equivalent to [`[*tone](1)] +* [`@[*bg]]: resolves to a color expression denoting the selected background color. equivalent to [`[*tone](0)] +* [`@[*tone]\[/[$alpha]\]([$fac] \[[$shift] \[[$saturate]\]\] )]: resolves to a color expression. [$fac] is a floating-point value scaling from the background color to the foreground color. [$shift] is a value in degrees controlling how far the hue will shift relative to the accent. [$saturate] is a floating-point value controlling how satured the color is. + +###refimpl-rend-groff groff +the [`groff] backend produces a text file suitable for supplying to a [`groff] compiler. [`groff] is the GNU implementation of a venerable typesetting system from the early days of UNIX + +as a convenience, the groff backend supports two modes of operation: it can write a [`groff] file directly to disk, or it can automatically launch a [`groff] process with the appropriate command line options and environment variables. this second mode is recommended unless you're rendering very large files to multiple formats, as [`groff] invocation is nontrivial and it's best to let the renderer handle that for you. + +####refimpl-rend-groff-modes modes +[`groff] supports the following modes: + +* string [`groff:annotate] controls how footnotes will be handled. +** [`footnote] places footnotes at the end of the page they are referenced on. if the same footnote is used on multiple pages, it will be duplicated on each. +** [`secnote] places footnotes at the end of each section. footnotes used in multiple sections will be duplicated for each +** [`endnote] places all footnotes at the end of the rendered document. +* string [`groff:dev] names an output device (such as [`dvi] or[`pdf]). if this mode is present, [`groff] will be automatically invoked +* string [`groff:title-page] takes an identifier that names a section. this section will be treated as the title page for the document. + +### directives +* [`%[*pragma] title-page [$id]] sets the title page to section [$id]. this causes it to be specially formatted, with a large, centered title and subtitle. + +### quirks +if the [`toc] extension is active but [`%[*toc]] directive is provided, the table of contents will be given its own section at the start of the document (after the title page, if any). + ## further directions ### additional backends it is eventually intended to support to following backends, if reasonably practicable. * [*html]: emit HTML and CSS code to typeset the document. [!in progress] * [*svg]: emit SVG, taking advantage of its precise layout features to produce a nicely formatted and paginated document. pagination can be accomplished through emitting multiple files or by assigning one layer to each page. [!long term] * [*groff]: the most important output backend, rivalling [*html]. will allow the document to be typeset in a wide variety of formats, including PDF and manpage. [!short term] * [*gemtext]: essentially a downrezzing of cortav to make it readable to Gemini clients +* [*ast]: produces a human- and/or machine-readable dump of the document's syntax tree, to aid in debugging or for interoperation with systems that do not support `cortav` direcly. mode [`ast:repr] wil allow selecting formats for the dump. [`ast:rel] can be [`tree] (the default) to emit a hierarchical representation, or [`flat] to emit an array of nodes that convey hierarchy [^flatdoc by naming one another], rather than being placed inside one another. [`tree] is easier for humans to parse; [`flat] is easier for computers. origin information can be included for each node with the flag [`ast:debug-syms], but be aware this will greatly increase file size. +** [`tabtree] [!(default)]: a hierarchical tree view, with the number of tabs preceding an item showing its depth in the tree +** [`sexp] +** [`binary]: emit a raw binary format that is easier for programs to read. maybe an lmdb or cdb file? +** [`json] + + flatdoc: ~~~flat sexp example output [scheme]~~~ + (nodes + (section (id . "section1") + (anchor "introduction") + (kind . "ordinary") + (label . "section1-heading") + (nodes + "section1-heading" + "para1" + "para2" + "hzrule" + "para3")) + (section (id . "section2") + (kind . "ordinary") + (label . "section2-heading") + (nodes + "para4" + "hzrule" + "para5" + "list1")) + (block list (id . "list1") + (kind . "ordered") + (nodes + "para6" + "list2" + "para7")) + (block list (id . "list2") + (kind . "unordered") + (nodes + "para8" + "para9" + "para10")) + (block para (id . "para1") + (nodes "text1" "format1" "text3" "foonote1" "text4")) + (block label (id . "section1-heading") (nodes "section1-heading-text")) + (text (id . "section1-heading-text") "Contemplating the Anathema") + (text (id . "text1") + "Disquieting information has recently been disclosed to virtual journalists of the Giedi Prime infomatrix by sources close to the Hyperion Entity regarding the catatrophic Year of Schisms and the unidentified agents believed to be responsible for memetically engineering the near-collapse of the Church Galactic.") + (span format (id . "format1") + (style . "emph") + (nodes . "text2")) + (text (id . "text2") "Curiously,") + (text (id . "text3") "his Cyber-Holiness") + (text (id . "footnote1-caption-text") "Pope Chewbacca III") + (span footnote (id . "footnote1") + (note . "footnote1-text") + (ref . "papal-disclaimer") + (nodes + "footnode1-caption-text")) + (text (id . "text4") "has thus far had little to say on the matter, provoking rampant speculation among the faithful.") + (footnote-def (id . "footnote1-def") + (nodes "footnote1-text") + (text (id . "footnote1-text") "Currently recognized as legitimate successor to Peter of Terra by 2,756 sects, rejected by 678 of mostly Neo-Lutheran origin, and decried as an antipope by 73, most notably Pope Peter II of Centaurus Secundus, leader of the ongoing relativistic crusade against star systems owned by Microsoft.") + ;;; snip ;;; + (document + (nodes + "section1" "section2"))) + ~~~ some formats may eventually warrant their own renderer, but are not a priority: * [*text]: cortav source files are already plain text, but a certain amount of layout could be done using ascii art. * [*ansi]: emit sequences of ANSI escape codes to lay out a document in a terminal-friendly way * [*tex]: TeX is an unholy abomination and i neither like nor use it, but lots of people do and if cortav ever catches on, a TeX backend should probably be written eventually. @@ -407,13 +753,15 @@ ### LCH support right now, the use of color in the HTML renderer is very unsatisfactory. the accent mechanism operates on the basis of the CSS HSL function, which is not perceptually uniform; different hues will present different mixes of brightness and some (yellows?) may be ugly or unreadable. the ideal solution would be to simply switch to using LCH based colors. unfortunately, only Safari actually supports the LCH color function right now, and it's unlikely (unless Lea Verou and her husband manage to work a miracle) that Colors Level 4 is going to be implemented very widely any time soon. -this leaves us in an awkward position. we can of course do the math ourselves, working in LCH to implement the internal [$@tone] macro, and then "converting" these colors to HSL. unfortunately, you can't actually convert from LCH to HSL; it's like converting from pounds to kilograms. LCH can represent any color the human visual system can perceive; sRGB can't, and CSS HSL is implemented in sRGB. however, we could at least approximate something that would allow for perceptually uniform brightness, which would be an improvement, and this is probably the direction to go in, unless a miracle occurs and [$lch()] or [$color()] pop up in Blink. +this leaves us in an awkward position. we can of course do the math ourselves, working in LCH to implement the internal [`@tone] macro, and then "converting" these colors to HSL. unfortunately, you can't actually convert from LCH to HSL; it's like converting from pounds to kilograms. LCH can represent any color the human visual system can perceive; sRGB can't, and CSS HSL is implemented in sRGB. however, we could at least approximate something that would allow for perceptually uniform brightness, which would be an improvement, and this is probably the direction to go in, unless a miracle occurs and [`lch()] or [`color()] pop up in Blink. it may be possible to do a more reasonable job of handling colors in the postscript and TeX outputs. unsure about SVG but i assume it suffers the same problems HTML/CSS do. does groff even support color?? ### intent files -there's currently no standard way to describe the intent and desired formatting of a document besides placing pragmas in the source file itself. this is extremely suboptimal, as when generating collections of documents, it's ideal to be able to keep all formatting information in one place. users should also be able to specify their own styling overrides that describe the way they prefer to read [$cortav] files, especially for uses like gemini or gopher integration. +there's currently no standard way to describe the intent and desired formatting of a document besides placing pragmata in the source file itself. this is extremely suboptimal, as when generating collections of documents, it's ideal to be able to keep all formatting information in one place. users should also be able to specify their own styling overrides that describe the way they prefer to read [`cortav] files, especially for uses like gemini or gopher integration. + +at some point soon [`cortav] needs to address this by adding intent files that can be activated from outside the source file, such as with a command line flag or a configuration file setting. these will probably consist of lines that are interpreted as pragmata. in addition to the standard intent format however, individual implementations should feel free to provide their own ways to provide intent metadata; e.g. the reference implementation, which has a lua interpreter available, should be able to take a lua script that runs after the parse stage and generates . this will be particularly useful for the end-user who wishes to specify a particular format she likes reading her files in without forcing that format on everyone she sends the compiled document to, as it will be able to interrogate the document and make intelligent decisions about what pragmata to apply. -at some point soon [$cortav] needs to address this by adding intent files that can be activated from outside the source file, such as with a command line flag or a configuration file setting. these will probably consist of lines that are interpreted as pragmata. in addition to the standard intent format however, individual implementations should feel free to provide their own ways to provide intent metadata; e.g. the reference implementation, which has a lua interpreter available, should be able to take a lua script that runs after the parse stage and generates . this will be particularly useful for the end-user who wishes to specify a particular format she likes reading her files in without forcing that format on everyone she sends the compiled document to, as it will be able to interrogate the document and make intelligent decisions about what pragmata to apply. +intent files should also be able to define [>rsrc resources], [>ctxvar context variables], and macros. Index: cortav.lua ================================================================== --- cortav.lua +++ cortav.lua @@ -4,11 +4,10 @@ -- ? reference implementation of the cortav document language local ss = require 'sirsem' -- aliases for commonly used sirsem funcs local startswith = ss.str.begins -local eachcode = ss.str.enc.utf8.each local dump = ss.dump local declare = ss.declare -- make this module available to require() when linked into a lua bytecode program with luac local ct = ss.namespace 'cortav' @@ -83,10 +82,13 @@ mode = ss.exnkind('bad mode', function(msg, ...) return string.format("mode “%s” "..msg, ...) end); unimpl = ss.exnkind 'feature not implemented'; ext = ss.exnkind 'extension error'; + enc = ss.exnkind('encoding error', function(msg, ...) + return string.format('[%s]' .. msg, ...) + end); } ct.ctx = declare { mk = function(src) return {src = src} end; ident = 'context'; @@ -114,18 +116,18 @@ end; ref = function(self,id) if not id:find'%.' then local rid = self.sec.refs[id] if self.sec.refs[id] then - return self.sec.refs[id] + return self.sec.refs[id], id, self.sec else self:fail("no such ref %s in current section", id or '') end else local sec, ref = string.match(id, "(.-)%.(.+)") local s = self.doc.sections[sec] if s then if s.refs[ref] then - return s.refs[ref] + return s.refs[ref], ref, sec else self:fail("no such ref %s in section %s", ref, sec) end else self:fail("no such section %s", sec) end end end }; @@ -219,10 +221,11 @@ ext = { inhibit = {}; need = {}; use = {}; }; + enc = ss.str.enc.utf8; } end; construct = function(me) me.docjob = ct.ext.job('doc', me, nil) end; } @@ -399,34 +402,39 @@ function ct.render.html(doc, opts) local doctitle = opts['title'] local f = string.format local ids = {} local canonicalID = {} - local function getSafeID(obj) + local function getSafeID(obj,pfx) + pfx = pfx or '' if canonicalID[obj] then return canonicalID[obj] - elseif obj.id and ids[obj.id] then + elseif obj.id and ids[pfx .. obj.id] then + local objid = pfx .. obj.id local newid local i = 1 - repeat newid = obj.id .. string.format('-%x', i) + repeat newid = objid .. string.format('-%x', i) i = i + 1 until not ids[newid] ids[newid] = obj canonicalID[obj] = newid return newid else local cid = obj.id if not cid then local i = 1 - repeat cid = string.format('x-%x', i) + repeat cid = string.format('%sx-%x', pfx, i) i = i + 1 until not ids[cid] end ids[cid] = obj canonicalID[obj] = cid return cid end end + local footnotes = {} + local footnotecount = 0 + local langsused = {} local langpairs = { lua = { color = 0x9377ff }; terra = { color = 0xff77c8 }; c = { name = 'C', color = 0x77ffe8 }; @@ -433,15 +441,89 @@ html = { color = 0xfff877 }; scheme = { color = 0x77ff88 }; lisp = { color = 0x77ff88 }; fortran = { color = 0xff779a }; python = { color = 0xffd277 }; - python = { color = 0xcdd6ff }; + ruby = { color = 0xcdd6ff }; } local stylesets = { + footnote = [[ + div.footnote { + font-family: 90%; + display: none; + grid-template-columns: 1em 1fr min-content; + grid-template-rows: 1fr min-content; + position: fixed; + padding: 1em; + background: @tone(0.05); + border: black; + margin:auto; + } + div.footnote:target { display:grid; } + @media screen { + div.footnote { + left: 10em; + right: 10em; + max-width: calc(@width + 2em); + max-height: 30vw; + bottom: 1em; + } + } + @media print { + div.footnote { + position: relative; + } + div.footnote:first-of-type { + border-top: 1px solid black; + } + } + + div.footnote > a[href="#0"]{ + grid-row: 2/3; + grid-column: 3/4; + display: block; + padding: 0.2em 0.7em; + text-align: center; + text-decoration: none; + background: @tone(0.2); + color: @tone(1); + border: 1px solid black; + margin-top: 0.6em; + -webkit-user-select: none; + -ms-user-select: none; + user-select: none; + -webkit-user-drag: none; + user-drag: none; + } + div.footnote > a[href="#0"]:hover { + background: @tone(0.3); + color: @tone(2); + } + div.footnote > a[href="#0"]:active { + background: @tone(0.05); + color: @tone(0.4); + } + @media print { + div.footnote > a[href="#0"]{ + display:none; + } + } + div.footnote > div.number { + text-align:right; + grid-row: 1/2; + grid-column: 1/2; + } + div.footnote > div.text { + grid-row: 1/2; + grid-column: 2/4; + padding-left: 1em; + overflow-y: scroll; + } + ]]; header = [[ + body { padding: 0 2.5em !important } h1,h2,h3,h4,h5,h6 { border-bottom: 1px solid @tone(0.7); } h1 { font-size: 200%; border-bottom-style: double !important; border-bottom-width: 3px !important; margin: 0em -1em; } h2 { font-size: 130%; margin: 0em -0.7em; } h3 { font-size: 110%; margin: 0em -0.5em; } h4 { font-size: 100%; font-weight: normal; margin: 0em -0.2em; } @@ -492,103 +574,156 @@ } ]]; paragraph = [[ p { margin: 0.7em 0; + text-align: justify; } section { margin: 1.2em 0; } section:first-child { margin-top: 0; } ]]; accent = [[ - body { background: @bg; color: @fg } - a[href] { - color: @tone(0.7 30); - text-decoration-color: @tone/0.4(0.7 30); + @media screen { + body { background: @bg; color: @fg } + a[href] { + color: @tone(0.7 30); + text-decoration-color: @tone/0.4(0.7 30); + } + a[href]:hover { + color: @tone(0.9 30); + text-decoration-color: @tone/0.7(0.7 30); + } + h1 { color: @tone(2); } + h2 { color: @tone(1.5); } + h3 { color: @tone(1.2); } + h4 { color: @tone(1); } + h5,h6 { color: @tone(0.8); } } - a[href]:hover { - color: @tone(0.9 30); - text-decoration-color: @tone/0.7(0.7 30); + @media print { + a[href] { + text-decoration: none; + color: black; + font-weight: bold; + } + h1,h2,h3,h4,h5,h6 { + border-bottom: 1px black; + } } - h1 { color: @tone(2); } - h2 { color: @tone(1.5); } - h3 { color: @tone(1.2); } - h4 { color: @tone(1); } - h5,h6 { color: @tone(0.8); } ]]; + aside = [[ + section > aside { + text-align: justify; + margin: 0 1.5em; + padding: 0.5em 0.8em; + background: @tone(0.05); + font-size: 90%; + border-left: 5px solid @tone(0.2 15); + border-right: 5px solid @tone(0.2 15); + } + section > aside p { + margin: 0; + margin-top: 0.6em; + } + section > aside p:first-child { + margin: 0; + } + ]]; code = [[ code { - background: @fg; + display: inline-block; + background: @tone(0.9); color: @bg; font-family: monospace; font-size: 90%; padding: 3px 5px; } + ]]; + var = [[ + var { + font-style: italic; + font-family: monospace; + color: @tone(0.7); + } + code var { + color: @tone(0.25); + } + ]]; + math = [[ + span.equation { + display: inline-block; + background: @tone(0.08); + color: @tone(2); + padding: 0.1em 0.3em; + border: 1px solid @tone(0.5); + } ]]; abbr = [[ abbr[title] { cursor: help; } ]]; editors_markup = [[]]; block_code_listing = [[ - section > figure.listing { + figure.listing { font-family: monospace; background: @tone(0.05); color: @fg; padding: 0; margin: 0.3em 0; counter-reset: line-number; position: relative; border: 1px solid @fg; } - section > figure.listing>div { + figure.listing>div { white-space: pre-wrap; + tab-size: 3; + -moz-tab-size: 3; counter-increment: line-number; text-indent: -2.3em; margin-left: 2.3em; } - section > figure.listing>:is(div,hr)::before { + figure.listing>:is(div,hr)::before { width: 1.0em; padding: 0.2em 0.4em; text-align: right; display: inline-block; background-color: @tone(0.2); border-right: 1px solid @fg; content: counter(line-number); margin-right: 0.3em; } - section > figure.listing>hr::before { + figure.listing>hr::before { color: transparent; padding-top: 0; padding-bottom: 0; } - section > figure.listing>div::before { + figure.listing>div::before { color: @fg; } - section > figure.listing>div:last-child::before { + figure.listing>div:last-child::before { padding-bottom: 0.5em; } - section > figure.listing>figcaption:first-child { + figure.listing>figcaption:first-child { border: none; border-bottom: 1px solid @fg; } - section > figure.listing>figcaption::after { + figure.listing>figcaption::after { display: block; float: right; font-weight: normal; font-style: italic; font-size: 70%; padding-top: 0.3em; } - section > figure.listing>figcaption { + figure.listing>figcaption { font-family: sans-serif; font-size: 120%; padding: 0.2em 0.4em; border: none; color: @tone(2); } - section > figure.listing > hr { + figure.listing > hr { border: none; margin: 0; height: 0.7em; counter-increment: line-number; } @@ -606,16 +741,57 @@ obj_htmlid = getSafeID; -- remaining fields added later } local renderJob = doc:job('render_html', nil, render_state_handle) + doc.stage.job = renderJob; local runhook = function(h, ...) return renderJob:hook(h, render_state_handle, ...) end - local function getSpanRenderers(procs) + local tagproc do + local elt = function(t,attrs) + return f('<%s%s>', t, + attrs and ss.reduce(function(a,b) return a..b end, '', + ss.map(function(v,k) + if v == true + then return ' '..k + elseif v then return f(' %s="%s"', k, v) + end + end, attrs)) or '') + end + + tagproc = { + toTXT = { + tag = function(t,a,v) return v end; + elt = function(t,a) return '' end; + catenate = table.concat; + }; + toIR = { + tag = function(t,a,v,o) return { + tag = t, attrs = a; + nodes = type(v) == 'string' and {v} or v, src = o + } end; + + elt = function(t,a,o) return { + tag = t, attrs = a, src = o + } end; + + catenate = function(...) return ... end; + }; + toHTML = { + elt = elt; + tag = function(t,attrs,body) + return f('%s%s', elt(t,attrs), body, t) + end; + catenate = table.concat; + }; + } + end + + local function getBaseRenderers(procs, span_renderers) local tag, elt, catenate = procs.tag, procs.elt, procs.catenate local htmlDoc = function(title, head, body) return [[]] .. tag('html',nil, tag('head', nil, elt('meta',{charset = 'utf-8'}) .. @@ -622,56 +798,85 @@ (title and tag('title', nil, title) or '') .. (head or '')) .. tag('body', nil, body or '')) end - local span_renderers = {} local function htmlSpan(spans, block, sec) local text = {} for k,v in pairs(spans) do if type(v) == 'string' then - table.insert(text,(v:gsub('[<>&"]', - function(x) + v=v:gsub('[<>&"]', function(x) return string.format('&#%02u;', string.byte(x)) - end))) + end) + for fn, ext in renderJob:each('hook','render_html_sanitize') do + v = fn(renderJob:delegate(ext), v) + end + table.insert(text,v) else - table.insert(text, span_renderers[v.kind](v, block, sec)) + table.insert(text, (span_renderers[v.kind](v, block, sec))) end end return table.concat(text) end + return {htmlDoc=htmlDoc, htmlSpan=htmlSpan} + end + + local spanparse = function(...) + local s = ct.parse_span(...) + doc.docjob:hook('meddle_span', s) + return s + end + + local cssRulesFor = {} + local function getSpanRenderers(procs) + local tag, elt, catenate = procs.tag, procs.elt, procs.catenate + local span_renderers = {} + local plainrdr = getBaseRenderers(tagproc.toTXT, span_renderers) + local htmlSpan = getBaseRenderers(procs, span_renderers).htmlSpan function span_renderers.format(sp,...) - local tags = { strong = 'strong', emph = 'em', strike = 'del', insert = 'ins', literal = 'code' } + local tags = { strong = 'strong', emph = 'em', strike = 'del', insert = 'ins', literal = 'code', variable = 'var'} if sp.style == 'literal' and not opts['fossil-uv'] then stylesNeeded.code = true - end - if sp.style == 'del' or sp.style == 'ins' then + elseif sp.style == 'strike' or sp.style == 'insert' then stylesNeeded.editors_markup = true + elseif sp.style == 'variable' then + stylesNeeded.var = true end return tag(tags[sp.style],nil,htmlSpan(sp.spans,...)) end - function span_renderers.term(t,b,s) + function span_renderers.deref(t,b,s) local r = b.origin:ref(t.ref) local name = t.ref if name:find'%.' then name = name:match '^[^.]*%.(.+)$' end - if type(r) ~= 'string' then - b.origin:fail('%s is an object, not a reference', t.ref) + if type(r) == 'string' then + stylesNeeded.abbr = true + return tag('abbr',{title=r},next(t.spans) and htmlSpan(t.spans,b,s) or name) end - stylesNeeded.abbr = true - return tag('abbr',{title=r},next(t.spans) and htmlSpan(t.spans,b,s) or name) - end - - function span_renderers.macro(m,b,s) - local r = b.origin:ref(m.macro) - if type(r) ~= 'string' then - b.origin:fail('%s is an object, not a reference', t.ref) + if r.kind == 'resource' then + local rid = getSafeID(r, 'res-') + if r.class == 'image' then + if not cssRulesFor[r] then + local css = prepcss(string.format([[ + section p > .%s { + } + ]], rid)) + stylesets[r] = css + cssRulesFor[r] = css + stylesNeeded[r] = true + end + return tag('div',{class=rid},catenate{'blaah'}) + elseif r.class == 'video' then + local vid = {} + return tag('video',nil,vid) + elseif r.class == 'font' then + b.origin:fail('fonts cannot be instantiated, use %font directive instead') + end + else + b.origin:fail('%s is not an object that can be embedded', t.ref) end - local mctx = b.origin:clone() - mctx.invocation = m - return htmlSpan(ct.parse_span(r, mctx),b,s) end function span_renderers.var(v,b,s) local val if v.pos then @@ -688,10 +893,14 @@ return val else return htmlSpan(ct.parse_span(val, v.origin), b, s) end end + + function span_renderers.raw(v,b,s) + return htmlSpan(v.spans, b, s) + end function span_renderers.link(sp,b,s) local href if b.origin.doc.sections[sp.ref] then href = '#' .. sp.ref @@ -703,17 +912,58 @@ else href = r end end end return tag('a',{href=href},next(sp.spans) and htmlSpan(sp.spans,b,s) or href) end - return { - span_renderers = span_renderers; - htmlSpan = htmlSpan; - htmlDoc = htmlDoc; - } + + span_renderers['line-break'] = function(sp,b,s) + return elt('br') + end + + function span_renderers.macro(m,b,s) + local macroname = plainrdr.htmlSpan( + ct.parse_span(m.macro, b.origin), b,s) + local r = b.origin:ref(macroname) + if type(r) ~= 'string' then + b.origin:fail('%s is an object, not a reference', t.ref) + end + local mctx = b.origin:clone() + mctx.invocation = m + return htmlSpan(ct.parse_span(r, mctx),b,s) + end + function span_renderers.math(m,b,s) + stylesNeeded.math = true + return tag('span',{class='equation'},htmlSpan(m.spans, b, s)) + end; + function span_renderers.directive(d,b,s) + if d.ext == 'html' then + elseif b.origin.doc:allow_ext(d.ext) then + elseif d.crit then + b.origin:fail('critical extension %s unavailable', d.ext) + elseif d.failthru then + return htmlSpan(d.spans, b, s) + end + end + function span_renderers.footnote(f,b,s) + stylesNeeded.footnote = true + local source, sid, ssec = b.origin:ref(f.ref) + local cnc = getSafeID(ssec) .. ' ' .. sid + local fn + if footnotes[cnc] then + fn = footnotes[cnc] + else + footnotecount = footnotecount + 1 + fn = {num = footnotecount, origin = b.origin, fnid=cnc, source = source} + fn.id = getSafeID(fn) + footnotes[cnc] = fn + end + return tag('a', {href='#'..fn.id}, htmlSpan(f.spans) .. + tag('sup',nil, fn.num)) + end + + return span_renderers end - local function getBlockRenderers(procs, sr) local tag, elt, catenate = procs.tag, procs.elt, procs.catenate local null = function() return catenate{} end @@ -768,71 +1018,40 @@ else return elt('hr') end end, b.lines) if b.title then - table.insert(nodes,1,tag('figcaption',nil,sr.htmlSpan(b.title))) + table.insert(nodes,1, tag('figcaption',nil,sr.htmlSpan(b.title))) end if b.lang then langsused[b.lang] = true end return tag('figure', {class='listing', lang=b.lang, id=b.id and getSafeID(b)}, catenate(nodes)) end; aside = function(b,s) local bn = {} - for _,v in pairs(b.lines) do - table.insert(bn, tag('p', {}, sr.htmlSpan(v, b, s))) + stylesNeeded.aside = true + if #b.lines == 1 then + bn[1] = sr.htmlSpan(b.lines[1], b, s) + else + for _,v in pairs(b.lines) do + table.insert(bn, tag('p', {}, sr.htmlSpan(v, b, s))) + end end return tag('aside', {}, bn) end; - ['break'] = function() --[[nop]] end; + ['break'] = function() -- HACK + -- lists need to be rewritten to work like asides + return ''; + end; } return block_renderers; end local function getRenderers(procs) - local r = getSpanRenderers(procs) + local span_renderers = getSpanRenderers(procs) + local r = getBaseRenderers(procs,span_renderers) r.block_renderers = getBlockRenderers(procs, r) return r - end - - local tagproc do - local elt = function(t,attrs) - return f('<%s%s>', t, - attrs and ss.reduce(function(a,b) return a..b end, '', - ss.map(function(v,k) - if v == true - then return ' '..k - elseif v then return f(' %s="%s"', k, v) - end - end, attrs)) or '') - end - - tagproc = { - toTXT = { - tag = function(t,a,v) return v end; - elt = function(t,a) return '' end; - catenate = table.concat; - }; - toIR = { - tag = function(t,a,v,o) return { - tag = t, attrs = a; - nodes = type(v) == 'string' and {v} or v, src = o - } end; - - elt = function(t,a,o) return { - tag = t, attrs = a, src = o - } end; - - catenate = function(...) return ... end; - }; - toHTML = { - elt = elt; - tag = function(t,attrs,body) - return f('%s%s', elt(t,attrs), body, t) - end; - catenate = table.concat; - }; - } end local astproc = { toHTML = getRenderers(tagproc.toHTML); toTXT = getRenderers(tagproc.toTXT); @@ -855,57 +1074,78 @@ local plainr = astproc.toTXT local irBlockRdrs = astproc.toIR.block_renderers; render_state_handle.ir = ir; + local function renderBlocks(blocks, irs) + for i, block in ipairs(blocks) do + local rd + if irBlockRdrs[block.kind] then + rd = irBlockRdrs[block.kind](block,sec) + else + local rdr = renderJob:proc('render',block.kind,'html') + if rdr then + rd = rdr({ + state = render_state_handle; + tagproc = tagproc.toIR; + astproc = astproc.toIR; + }, block, sec) + end + end + if rd then + if opts['heading-anchors'] and block == sec.heading_node then + stylesNeeded.headingAnchors = true + table.insert(rd.nodes, ' ') + table.insert(rd.nodes, { + tag = 'a'; + attrs = {href = '#' .. irs.attrs.id, class='anchor'}; + nodes = {type(opts['heading-anchors'])=='string' and opts['heading-anchors'] or '§'}; + }) + end + if rd.src and rd.src.origin.lang then + if not rd.attrs then rd.attrs = {} end + rd.attrs.lang = rd.src.origin.lang + end + table.insert(irs.nodes, rd) + runhook('ir_section_node_insert', rd, irs, sec) + end + end + end runhook('ir_assemble', ir) for i, sec in ipairs(doc.secorder) do if doctitle == nil and sec.depth == 1 and sec.heading_node then doctitle = astproc.toTXT.htmlSpan(sec.heading_node.spans, sec.heading_node, sec) end local irs if sec.kind == 'ordinary' then if #(sec.blocks) > 0 then irs = {tag='section',attrs={id = getSafeID(sec)},nodes={}} - runhook('ir_section_build', irs, sec) - - for i, block in ipairs(sec.blocks) do - local rd - if irBlockRdrs[block.kind] then - rd = irBlockRdrs[block.kind](block,sec) - else - local rdr = renderJob:proc('render',block.kind,'html') - if rdr then - rd = rdr({ - state = render_state_handle; - tagproc = tagproc.toIR; - astproc = astproc.toIR; - }, block, sec) - end - end - if rd then - if opts['heading-anchors'] and block == sec.heading_node then - stylesNeeded.headingAnchors = true - table.insert(rd.nodes, ' ') - table.insert(rd.nodes, { - tag = 'a'; - attrs = {href = '#' .. irs.attrs.id, class='anchor'}; - nodes = {type(opts['heading-anchors'])=='string' and opts['heading-anchors'] or '§'}; - }) - end - table.insert(irs.nodes, rd) - runhook('ir_section_node_insert', rd, irs, sec) - end - end + renderBlocks(sec.blocks, irs) end elseif sec.kind == 'blockquote' then elseif sec.kind == 'listing' then elseif sec.kind == 'embed' then end if irs then table.insert(ir, irs) end end + + for _, fn in pairs(footnotes) do + local tag = tagproc.toIR.tag + local body = {nodes={}} + local ftir = {} + for l in fn.source:gmatch('([^\n]*)') do + ct.parse_line(l, fn.origin, ftir) + end + renderBlocks(ftir,body) + local note = tag('div',{class='footnote',id=fn.id}, { + tag('div',{class='number'}, tostring(fn.num)), + tag('div',{class='text'}, body.nodes), + tag('a',{href='#0'},'close') + }) + table.insert(ir, note) + end -- restructure passes runhook('ir_restructure_pre', ir) ---- list insertion pass @@ -1035,10 +1275,12 @@ local tfg = opts['dark-on-light'] and tonespan or 1.0 - tonespan if var == 'bg' then return tone(tbg,nil,nil,tonumber(alpha)) elseif var == 'fg' then return tone(tfg,nil,nil,tonumber(alpha)) + elseif var == 'width' then + return opts['width'] or '100vw' elseif var == 'tone' then local l, sep, sat for i=1,3 do -- 🙄 l,sep,sat = param:match('^%('..string.rep('([^%s]*)%s*',i)..'%)$') if l then break end @@ -1126,32 +1368,126 @@ raw = raw; var = not pos and s or nil; origin = c:clone(); } end + end + local function insert_span_directive(crit, failthru) + return function(s,c) + local args = ss.str.breakwords(d.doc.enc, s, 1) + local brksyms = map(enc.encodeUCS, { + '.', ',', ':', ';', '!', '$', '&', '^', + '/', '?', '@', '=' + }) + local brkhash = {} for _,s in pairs(brksyms) do + brkhash[s] = true + end + + local extname = '' + local sym + local cmd = '' + for ch,p in ss.str.each(c.doc.enc, args[1]) do + if sym == nil then + if brkhash[ch] then + sym = ch + else + extname = extname .. ch + end + elseif brkhash[ch] then + sym = sym + ch + else + cmd = cmd + ch + end + end + if cmd == '' then cmd = nil end + local spans if failthru then + spans = ct.parse_span(args[2], c) + end + return { + kind = 'directive'; + ext = extname; + cmd = cmd; + args = args; + crit = crit; + failthru = failthru; + spans = spans; + } + end end ct.spanctls = { {seq = '!', parse = formatter 'emph'}; {seq = '*', parse = formatter 'strong'}; {seq = '~', parse = formatter 'strike'}; - {seq = '+', parse = formatter 'inser'}; + {seq = '+', parse = formatter 'insert'}; {seq = '\\', parse = function(s, c) -- raw - return s - end}; - {seq = '$\\', parse = function(s, c) -- raw return { - kind = 'format'; - style = 'literal'; + kind = 'raw'; spans = {s}; origin = c:clone(); } end}; - {seq = '$', parse = formatter 'literal'}; + {seq = '`\\', parse = function(s, c) -- raw + local o = c:clone(); + local str = '' + for c, p in ss.str.each(c.doc.enc, s) do + local q = p:esc() + if q then + str = str .. q + p.next.byte = p.next.byte + #q + else + str = str .. c + end + end + return { + kind = 'format'; + style = 'literal'; + spans = {{ + kind = 'raw'; + spans = {str}; + origin = o; + }}; + origin = o; + } + end}; + {seq = '`', parse = formatter 'literal'}; + {seq = '$', parse = formatter 'variable'}; + {seq = '^', parse = function(s,c) --footnotes + local r, t = s:match '^([^%s]+)%s*(.-)$' + return { + kind = 'footnote'; + ref = r; + spans = ct.parse_span(t, c); + origin = c:clone(); + } + -- TODO support for footnote sections + end}; + {seq = '=', parse = function(s,c) --math mode + local tx = { + ['%*'] = '×'; + ['/'] = '÷'; + } + for k,v in pairs(tx) do s = s:gsub(k,v) end + s=s:gsub('%^([0-9]+)', function(num) + local sup = {'⁰','¹','²','³','⁴','⁵','⁶','⁷','⁸','⁹'}; + local r = '' + for i=1,#num do + r = r .. sup[1 + (num:byte(i) - 0x30)] + end + return r + end) + local m = {s} --TODO + return { + kind = 'math'; + original = s; + spans = m; + origin = c:clone(); + }; + end}; {seq = '&', parse = function(s, c) local r, t = s:match '^([^%s]+)%s*(.-)$' return { - kind = 'term'; + kind = 'deref'; spans = (t and t ~= "") and ct.parse_span(t, c) or {}; ref = r; origin = c:clone(); } end}; @@ -1167,10 +1503,14 @@ {seq = '>', parse = insert_link}; {seq = '→', parse = insert_link}; {seq = '🔗', parse = insert_link}; {seq = '##', parse = insert_var_ref(true)}; {seq = '#', parse = insert_var_ref(false)}; + {seq = '%%', parse = function() --[[NOP]] end}; + {seq = '%!', parse = insert_span_directive(true,false)}; + {seq = '%:', parse = insert_span_directive(false,true)}; + {seq = '%', parse = insert_span_directive(false,false)}; } end function ct.parse_span(str,ctx) local function delimited(start, stop, s) @@ -1180,21 +1520,29 @@ end local buf = "" local spans = {} local function flush() if buf ~= "" then + -- for fn, ext in ctx.doc.docjob:each('hook','doc_meddle_string') do + -- buf = fn(ctx.doc.docjob:delegate(ext), ctx, buf) + -- end table.insert(spans, buf) buf = "" end end local skip = false - for c,p in eachcode(str) do - if skip == true then - skip = false - buf = buf .. c - elseif c == '\\' then - skip = true + for c,p in ss.str.each(ctx.doc.enc,str) do + local ba, ca, es = ctx.doc.enc.parse_escape(str:sub(p.byte)) + if es then + flush() + table.insert(spans, { + kind = 'raw'; + spans = {es}; + origin = ctx:clone() + }) + p.next.byte = p.next.byte + ba; + p.next.code = p.next.code + ca; elseif c == '{' then flush() local substr, following = delimited('{','}',str:sub(p.byte)) local splitstart, splitstop = substr:find'%s+' local id, argstr @@ -1216,10 +1564,11 @@ while i<=#argstr and (argstr:sub(i,i) ~= '|' or argstr:sub(i-1,i) == '\\|') do i = i + 1 end local arg = argstr:sub(start, i == #argstr and i or i-1) start = i+1 + arg=arg:gsub('\\|','|') table.insert(o.args, arg) i = i + 1 end end @@ -1238,10 +1587,13 @@ end end if not found then ctx:fail('no recognized control sequence in [%s]', substr) end + elseif c == '\n' then + flush() + table.insert(spans,{kind='line-break',origin=ctx:clone()}) else buf = buf .. c end end flush() @@ -1248,15 +1600,18 @@ return spans end local function blockwrap(fn) - return function(l,c,j) - local block = fn(l,c,j) + return function(l,c,j,d) + local block = fn(l,c,j,d) block.origin = c:clone(); - table.insert(c.sec.blocks, block); + table.insert(d, block); j:hook('block_insert', c, block, l) + if block.spans then + c.doc.docjob:hook('meddle_span', block.spans, block) + end end end local insert_paragraph = blockwrap(function(l,c) if l:sub(1,1) == '.' then l = l:sub(2) end @@ -1284,10 +1639,11 @@ kind = "label"; spans = ct.parse_span(t,c); origin = s.origin; captions = s; } + c.doc.docjob:hook('meddle_span', heading.spans, heading) table.insert(s.blocks, heading) s.heading_node = heading end c.sec = s @@ -1301,12 +1657,15 @@ end local dextctl = function(w,c) local mode, exts = w(1) for e in exts:gmatch '([^%s]+)' do if mode == 'uses' then + c.doc.ext.use[e] = true elseif mode == 'needs' then + c.doc.ext.need[e] = true elseif mode == 'inhibits' then + c.doc.ext.inhibit[e] = true end end end local dcond = function(w,c) local mode, cond, exp = w(2) @@ -1317,16 +1676,34 @@ license = dsetmeta; keywords = dsetmeta; desc = dsetmeta; when = dcond; unless = dcond; + pragma = function(w,c) + end; + lang = function(w,c) + local _, op, l = w(2) + local langstack = c.doc.stage.langstack + if op == 'is' then + langstack[math.max(1, #langstack)] = l + elseif op == 'push' then + table.insert(langstack, l) + elseif op == 'pop' then + if next(langstack) then + langstack[#langstack] = nil + end + elseif op == 'sec' then + c.sec.lang = l + else c:fail('bad language directive “%s”', op) end + c.lang = langstack[#langstack] + end; expand = function(w,c) local _, m = w(1) if m ~= 'off' then - c.expand_next = 1 + c.doc.stage.expand_next = 1 else - c.expand_next = 0 + c.doc.stage.expand_next = 0 end end; } local function insert_table_row(l,c,j) @@ -1337,11 +1714,11 @@ buf.str = buf.str:gsub('%s+$','') table.insert(row, buf) end buf = { str = '' } end - for c,p in eachcode(l) do + for c,p in ss.str.each(c.doc.enc,l) do if c == '|' or c == '+' and (p.code == 1 or l:sub(p.byte-1,p.byte-1)~='\\') then flush() buf.header = c == '+' elseif c == ':' then local lst = l:sub(p.byte-#c,p.byte-#c) @@ -1373,10 +1750,11 @@ end end if buf.str ~= '' then flush() end for _,v in pairs(row) do v.spans = ct.parse_span(v.str, c) + c.doc.docjob:hook('meddle_span', v.spans, v) end if #c.sec.blocks > 1 and c.sec.blocks[#c.sec.blocks].kind == 'table' then local tbl = c.sec.blocks[#c.sec.blocks] table.insert(tbl.rows, row) j:hook('block_table_attach', c, tbl, row, l) @@ -1400,24 +1778,27 @@ {seq = '#', fn = insert_section}; {seq = '§', fn = insert_section}; {seq = '+', fn = insert_table_row}; {seq = '|', fn = insert_table_row}; {seq = '│', fn = insert_table_row}; - {seq = '!', fn = function(l,c,j) - local last = c.sec.blocks[#c.sec.blocks] + {seq = '!', fn = function(l,c,j,d) + local last = d[#d] local txt = l:match '^%s*!%s*(.-)$' if (not last) or last.kind ~= 'aside' then local aside = { kind = 'aside'; - lines = { ct.parse_span(txt, c) } + lines = { ct.parse_span(txt, c) }; + origin = c:clone(); } - c:insert(aside) + c.doc.docjob:hook('meddle_span', aside.lines[1], aside) + table.insert(d,aside) j:hook('block_aside_insert', c, aside, l) j:hook('block_aside_line_insert', c, aside, aside.lines[1], l) j:hook('block_insert', c, aside, l) else local sp = ct.parse_span(txt, c) + c.doc.docjob:hook('meddle_span', sp, last) table.insert(last.lines, sp) j:hook('block_aside_attach', c, last, sp, l) j:hook('block_aside_line_insert', c, last, sp, l) end end}; @@ -1432,25 +1813,50 @@ depth = depth; ordered = ordered; spans = ct.parse_span(txt, c); } end)}; - {seq = '\t', fn = function(l,c,j) - local ref, val = l:match '\t+([^:]+):%s*(.*)$' - c.sec.refs[ref] = val - j:hook('section_ref_attach', c, ref, val, l) + {seq = '\t\t', fn = function(l,c,j,d) + local last = d[#d] + if (not last) or (last.kind ~= 'reference') then + c:fail('reference continuations must immediately follow a reference') + end + local str = l:match '^\t\t(.-)%s*$' + last.val = last.val .. '\n' .. str + c.sec.refs[last.key] = last.val end}; - {seq = '%', fn = function(l,c,j) -- directive + {seq = '\t', fn = blockwrap(function(l,c,j,d) + local ref, val = l:match '\t+([^:]+):%s*(.*)$' + local last = d[#d] + local rsrc + if last and last.kind == 'resource' then + last.props[ref] = val + rsrc = last + elseif last and last.kind == 'reference' and last.rsrc then + last.rsrc.props[ref] = val + rsrc = last.rsrc + else + c.sec.refs[ref] = val + end + j:hook('section_ref_attach', c, ref, val, l) + return { + kind = 'reference'; + rsrc = rsrc; + key = ref; + val = val; + } + end)}; + {seq = '%', fn = function(l,c,j,d) -- directive local crit, cmdline = l:match '^%%([!%%]?)%s*(.*)$' local words = function(i) local wds = {} if i == 0 then return cmdline end for w,pos in cmdline:gmatch '([^%s]+)()' do table.insert(wds, w) i = i - 1 if i == 0 then - table.insert(wds,cmdline:sub(pos)) + table.insert(wds,(cmdline:sub(pos):match('^%s*(.-)%s*$'))) return table.unpack(wds) end end end @@ -1461,12 +1867,13 @@ -- this is a directive for the renderer; insert it into the tree as is local dir = { kind = 'directive'; critical = crit == '!'; words = words; + origin = c; } - c:insert(dir) + table.insert(d, dir) j:hook('block_directive_render', j, c, dir) elseif c.doc:allow_ext(cmd) then -- extension directives begin with their id local ext = ct.ext.loaded[cmd] if ext.directives then local _, topcmd = words(2) @@ -1507,10 +1914,14 @@ kind = 'listing'; lang = lang, id = id, title = title and ct.parse_span(title,c); lines = {}; } } + if c.doc.stage.expand_next and c.doc.stage.expand_next > 0 then + c.doc.stage.expand_next = c.doc.stage.expand_next - 1 + mode.expand = true + end j:hook('mode_switch', c, mode) c.mode = mode if id then if c.sec.refs[id] then c:fail('duplicate ID %s', id) end c.sec.refs[id] = c.mode.listing @@ -1519,101 +1930,178 @@ return c.mode.listing; end)}; {pred = function(s,c) if s:match '^[%-_][*_%-%s]+' then return true end if startswith(s, '—') then - for c, p in eachcode(s) do + for c, p in ss.str.each(c.doc.enc,s) do if ({ ['—'] = true, ['-'] = true, [' '] = true; ['*'] = true, ['_'] = true, ['\t'] = true; })[c] ~= true then return false end end return true end end; fn = blockwrap(function() return { kind = 'horiz-rule' } + end)}; + {seq='@', fn=blockwrap(function(s,c) + local id = s:match '^@%s*(.-)%s*$' + local rsrc = { + kind = 'resource'; + props = {}; + id = id; + } + if c.sec.refs[id] then + c:fail('an object with id “%s” already exists in that section',id) + else + c.sec.refs[id] = rsrc + end + return rsrc end)}; {fn = insert_paragraph}; } -function ct.parse(file, src, mode) - local function - is_whitespace(cp) - return cp == 0x20 or cp == 0xe390 +function ct.parse_line(l, ctx, dest) + local newspan + local job = ctx.doc.stage.job + job:hook('line_read',ctx,l) + if ctx.mode then + if ctx.mode.kind == 'code' then + if l and l:match '^~~~%s*$' then + job:hook('block_listing_end',ctx,ctx.mode.listing) + job:hook('mode_switch', c, nil) + ctx.mode = nil + else + -- TODO handle formatted code + local newline + if ctx.mode.expand + then newline = ct.parse_span(l, ctx) + else newline = {l} + end + table.insert(ctx.mode.listing.lines, newline) + job:hook('block_listing_newline',ctx,ctx.mode.listing,newline) + end + else + local mf = job:proc('modes', ctx.mode.kind) + if not mf then + ctx:fail('unimplemented syntax mode %s', ctx.mode.kind) + end + mf(job, ctx, l, dest) --NOTE: you are responsible for triggering the appropriate hooks if you insert anything! + end + else + if l then + local function tryseqs(seqs, ...) + for _, i in pairs(seqs) do + if ((not i.seq ) or startswith(l, i.seq)) and + ((not i.pred) or i.pred (l, ctx )) then + i.fn(l, ctx, job, dest, ...) + return true + end + end + return false + end + + if not tryseqs(ct.ctlseqs) then + local found = false + + for eb, ext, state in job:each('blocks') do + if tryseqs(eb, state) then found = true break end + end + + if not found then + ctx:fail 'incomprehensible input line' + end + end + else + if next(dest) and dest[#dest].kind ~= 'break' then + local brk = {kind='break', origin = ctx:clone()} + job:hook('block_break', ctx, brk, l) + table.insert(dest, brk) + end + end end + job:hook('line_end',ctx,l) +end + +function ct.parse(file, src, mode, setup) local ctx = ct.ctx.mk(src) ctx.line = 0 ctx.doc = ct.doc.mk() ctx.doc.src = src - ctx.doc.stage = { - kind = 'parse'; - mode = mode; - } ctx.sec = ctx.doc:mksec() -- toplevel section ctx.sec.origin = ctx:clone() + ctx.lang = mode['meta:lang'] + if mode['parse:enc'] then + local e = ss.str.enc[mode['parse:enc']] + if not e then + ct.exns.enc('requested encoding not supported',mode['parse:enc']):throw() + end + ctx.doc.enc = e + end -- create states for extension hooks local job = ctx.doc:job('parse',nil,ctx) + ctx.doc.stage = { + kind = 'parse'; + mode = mode; + job = job; + langstack = {ctx.lang}; + fontstack = {}; + } + + local function + is_whitespace(cp) + return ctx.doc.enc.iswhitespace(cp) + end + + if setup then setup(ctx) end + for full_line in file:lines() do ctx.line = ctx.line + 1 local l for p, c in utf8.codes(full_line) do if not is_whitespace(c) then l = full_line:sub(p) break end end - job:hook('line_read',ctx,l) + ct.parse_line(l, ctx, ctx.sec.blocks) + end - if ctx.mode then - if ctx.mode.kind == 'code' then - if l and l:match '^~~~%s*$' then - job:hook('block_listing_end',ctx,ctx.mode.listing) - job:hook('mode_switch', c, nil) - ctx.mode = nil - else - -- TODO handle formatted code - local newline = {l} - table.insert(ctx.mode.listing.lines, newline) - job:hook('block_listing_newline',ctx,ctx.mode.listing,newline) - end - else - ctx:fail('unimplemented syntax mode %s', ctx.mode.kind) - end - else - if l then - local function tryseqs(seqs, ...) - for _, i in pairs(seqs) do - if ((not i.seq ) or startswith(l, i.seq)) and - ((not i.pred) or i.pred (l, ctx )) then - i.fn(l, ctx, job, ...) - return true - end + for i, sec in ipairs(ctx.doc.secorder) do + for refid, r in ipairs(sec.refs) do + if type(r) == 'table' and r.kind == 'resource' and r.props.src then + local lines = ss.str.breaklines(ctx.doc.enc, r.props.src) + local srcs = {} + for i,l in ipairs(lines) do + local args = ss.str.breakwords(ctx.doc.enc, l, 2, {escape=true}) + if #args < 3 then + r.origin:fail('invalid syntax for resource %s', t.ref) + end + local mimebreak = function(s) + local wds = ss.str.split(ctx.doc.enc, s, '/', 1, {escape=true}) + return wds end - return false + local mime = mimebreak(args[2]); + local mimeclasses = { + ['application/svg+xml'] = 'image'; + } + local class = mimeclasses[mime] + table.insert(srcs, { + mode = args[1]; + mime = mime; + uri = args[3]; + class = class or mime[1]; + }) end - - if not tryseqs(ct.ctlseqs) then - local found = false - - for eb, ext, state in job:each('blocks') do - if tryseqs(eb, state) then found = true break end - end - - if not found then - ctx:fail 'incomprehensible input line' - end - end - else - if next(ctx.sec.blocks) and ctx.sec.blocks[#ctx.sec.blocks].kind ~= 'break' then - local brk = {kind='break'} - job:hook('block_break', ctx, brk, l) - table.insert(ctx.sec.blocks, brk) - end + --ideally move this into its own mimetype lib + local kind = r.props.as or srcs[1].class + r.class = kind + r.srcs = srcs end end - job:hook('line_end',ctx,l) end - + ctx.doc.stage = nil + ctx.doc.docjob:hook('meddle_ast') return ctx.doc end Index: desk/cortav.xml ================================================================== --- desk/cortav.xml +++ desk/cortav.xml @@ -10,10 +10,23 @@