On beyond Invisible XML: parsing with implicit (and explicit) next-generation markup Balisage: The Markup Conference 2026 August 3-7, 2026 In this paper we explore what Invisible-XML-like (ixml-like) processing might look like when applied to what we refer to as next-generation features of markup, that is, structural properties of textual documents like overlap, discontinuity, and unordered content, which XML was not designed to represent at the level of the model. Like actual ixml, our ixml-like application is intended to recognize implicit, untagged markup and express it with explicit markup, but it differs from ixml by also supporting next-generation features that are not part of XML modeling and syntax. We implement this ixml-like insertion of markup as a thin wrapper around the same data model and code base used for the Creole-like validation that we described in . Our exploration of ixml-like processing with next-generation features does not depend on that shared code base, but the appeal of coordinating the implementation of a validation tool with the implementation of a tool for inserting explicit markup foregrounds what the two operations have in common and suggests that each might contribute insights into efficient ways of implementing the other. Ronald Haentjens Dekker Ronald Haentjens Dekker is a researcher and software architect at DHLab and the Huygens Institute for the History of the Netherlands, part of the Royal Netherlands Academy of Arts and Sciences. As a software architect, he is responsible for translating research questions into technology or algorithms and explaining to researchers and management how specific technologies will influence their research. He has worked on transcription and annotation software, collation software, and repository software, and he is the lead developer of the CollateX collation tool. He also conducts workshops to teach researchers how to use scripting languages in combination with digital editions to enhance their research. Researcher DH Lab, Huygens Institute for the History of the Netherlands ronald.dekker@di.huc.knaw.nl https://orcid.org/0000-0001-6737-7986 David J. Birnbaum David J. Birnbaum is Professor Emeritus of Slavic Languages and Literatures at the University of Pittsburgh. He has been involved in the study of electronic text technology since the mid-1980s, has delivered presentations at a variety of electronic text technology conferences, and has served on the board of the Association for Computers and the Humanities, the editorial board of Markup languages: theory and practice, and the Text Encoding Initiative Technical Council. Much of his electronic text work intersects with his research in medieval Slavic manuscript studies, but he also often writes about issues in the philosophy of markup. Professor Emeritus Department of Slavic Languages and Literatures, University of Pittsburgh djbpitt@gmail.com https://orcid.org/0000-0002-2867-7473 Bram Buitendijk Bram Buitendijk is a software engineer in the Research and Development team at the Humanities Cluster, part of the Royal Netherlands Academy of Arts and Sciences. He has worked on transcription and annotation software, collation software, and repository software. Software Engineer Digital Infrastructure, KNAW Humanities Cluster bram.buitendijk@di.huc.knaw.nl https://pure.knaw.nl/portal/en/persons/bram-buitendijk https://orcid.org/0000-0002-3755-5929 Copyright © 2026 by the authors. All rights reserved.
Overview A well-known limitation of XML markup is that it is not designed to represent at the model level structural properties of textual documents like overlap, discontinuity, unordered content, and others, which we refer to as next-generation features. See “Challenges for text modeling” in . In an XML environment these features can be represented with workarounds that must be managed during processing, and in previous Balisage presentations we explained why if might be better if inherent structural features of documents could, for reasons both philosophical and practical, be represented instead at the level of the model. See “Semantics versus application level” in . The current presentation explores what Invisible-XML-like (ixml-like) processing might look like when applied to next-generation markup languages, such as LMNL (), TexMECS (), or TAGML (). We employ the spelling ixml (instead of IXML or iXML). (the official specification) does not recommend a spelling and inconsistently uses both ixml and IXML. The ixml portal at https://invisiblexml.org/, managed by the Community Group that oversees the maintenance and development of the official specification, also uses iXML.
ixml and ixml-like processing
Uses of ixml-like processing We identify the following three types of situations for ixml-like processing: An ixml-like grammar can be used to parse input without explicit next-generation markup and add that markup, much as ixml can be used to add explicit XML markup to character-stream input. The input in question may be plain text or it may include characters that function as markup in an alternative structured markup language, such as JSON. Cf. the examples in of input that may be not only plain text, but also formats such as JSON, CSS, URLs, or structured mathematical notation. An ixml-like grammar can be used to parse input that has already been tagged according to a next-generation markup language without inserting additional markup or otherwise modifying the input. This operation uses ixml-like processing only as a recognizer, so that the only result that matters is whether the input document can or cannot be parsed successfully against a grammar. The terms recognizer and validator can be used similarly, but we employ the former to emphasize a Boolean outcome (successful or unsuccessful interpretation of the input according to a grammar) and the latter to point to the possibility of more detailed messaging, in case of failure, that can help a user locate and redress mistakes in either the input or the grammar. Much as ixml parsing has no built-in knowledge about XML markup that might already be present in an input document, and must therefore use grammar rules to distinguish XML markup from content, ixml-like parsing of documents that already contain next-generation markup must read the input as a character stream and use grammar rules to recognize existing next-generation markup. A processor might ingest an input document that is already (partially) tagged according to a next-generation markup language and insert additional markup. Alternatively, a processor might ingest and parse a document that contains next-generation markup and emit a document that is marked up in other ways. For example, the system described in reads and parses a document with LMNL input and emits a document with XML markup. These variants are similar in combining the two previous scenarios: ingesting and parsing a document that contains next-generation markup (scenario #2) and adding markup to it (scenario #1). Adding markup to an input document (the first and third examples above) assumes that the input can be parsed successfully against a supplied grammar. The second example above makes no such assumption and, instead, seeks to learn whether a parse can or cannot succeed. Our perspective on assumed correct vs check whether correct is informed by , which distinguishes: Permissive grammars: we assume the input is correct Strict grammars: we check that the input is correct.
Failure vs error in ixml processing There are four possible types of outcome of an ixml operation, with the following behaviors described in : Success in parsing the grammar, parsing the input against the grammar, and serializing a parse result returns well-formed XML according to the rules of the grammar. Failure to parse the grammar is a static error, with the error number reflecting the reason for the problem. Except for the error number, no specific returned result, messaging, or exit code is described. Success in parsing the grammar but failure to parse the input against it is not described as an error condition. It returns XML with an @ixml:state attribute that includes the string failed. The root element and content of the XML result are not described, except that the specification suggests optional helpful content. Because this case is not regarded as an error, no error number is described. Success in parsing the grammar, success in parsing the input against it, but failure to serialize the parse result as well-formed XML is a dynamic error, with the error number reflecting the reason for the problem. Except for the error number, no returned result, messaging, or exit code is described. Because describes expected behavior in these four cases only partially, it is not surprising that although the three command-line tools that we had available (Coffeepot, Markup Blitz, xmq) conform to what the spec requires, they not agree fully in what they report. We tested all four conditions, as follows, running the operations at the zsh command line and checking the exit code with echo $?:
Grammar correct? Input matches grammar Output can be serialized Result category Coffeepot 3.3.13 Markup Blitz 1.12-SNAPSHOT xmq 4.1.0-modified
Yes Yes Yes stderr N/A N/A N/A
stdout Valid XML Valid XML Valid XML
Exit code 0 0 0
No N/A N/A stderr Prose, no error code Prose, error code, stack trace None
stdout None None <ixml ixml:state="failed">, no error code, includes "syntax error" and input location
Exit code 0 1 1
Yes No N/A stderr None None None
stdout <fail ixml:state="failed"> with xml description <ixml ixml:state="failed"> with prose description <ixml ixml:state="failed"> with xml description
Exit code 1 0 1
Yes Yes No stderr Prose, no error code None None
stdout None <ixml ixml:status="failed" ixml:error-code="[value]"> with prose in body Malformed XML, no mention of error
Exit code 3 0 0
(xmq serialization of malformed XML has been reported as an xmq bug. Watch this space for an update.) To summarize, all three processors conform to the requirements of the spec in all cases, but in divergent ways: Correct grammar, input matches: all produce the same result Incorrect grammar: All report an error, but in divergent ways, on divergent file handles, and with divergent exit codes Correct grammar but input does not match: All emit XML (with divergent root element names) on stdout with the required @ixml:state attribute value of failed. Divergent error codes. Correct grammar, input matches, but output cannot be serialized: Divergent result, file handles, and exit codes. We identify three places in which ixml processing (and our next-generation extension thereof) might fail to emit tagged output: The grammar cannot be parsed. The requirement that ixml grammars be valid according to the rules described in is similar to the requirement that RELAX NG schemas be valid according to the rules described in the RELAX NG specification. The grammar itself can be parsed but the input cannot be parsed against it. This is similar to the use of traditional XML schema validation to determine whether documents do or do not conform to structures represented in a schema, but see also the more detailed exploration of failure vs error in ixml below. Because ixml is defined according to an ixml grammar, case 1 and case 2 both involve failure to parse an instance against a grammar. The two situations are nonetheless meaningfully different for an end-user because if phase 1 does not succeed, the user must adjust the user’s own grammar, which is effectively the input being parsed against the ixml grammar for ixml, while if phase 2 does not succeed, the user may adjust either the user’s own grammar or the input document. The grammar itself can be parsed and the input can be parsed against it, but the result cannot be serialized as well-formed XML. An ambiguous ixml grammar may produce both parses that can be serialized as well-formed XML and those that cannot, and an ixml processor choose any parse to (attempt to) serialize for output. This means that if an ixml tool chooses a parse arbitrarily (as it is permitted to do), whether the operation reports success or failure may depend on which parse is selected for serialization. This indeterminacy could be mitigated by requiring the tool to choose only among successful parses, if any are available, for serialization; such behavior is not required by the ixml specification, but it is not prohibited because it is consistent with the requirement that the process emit one or more parses. The exact language of about this issue is that:
Processors must accept and parse any conforming grammar, and produce at least one parse of supplied input that matches the grammar starting at the root symbol. If more than one parse results, one is chosen; it is not defined how this choice is made, but the resulting serialization should include the attribute ixml:state on the document element with a value that includes the word ambiguous. Different processors may vary in whether input is detected as ambiguous or not.
The first and third of the preceding types of failure are regarded as ixml errors, but the second, perhaps surprisingly, is not. An ixml processor must report the failure, but there is no associated numbered error message. says:
If the parse fails, some XML document must be produced with ixml:state on the document element with a value that includes the word failed. The document should provide helpful information about where and why it failed; it may be a partial parse tree that includes parts of the parse that succeeded.
The failure to parse an XML document against a RELAX NG schema that is, itself, valid is regarded as an error in RELAX NG; see, on this topic, the section entitled Error handling in . We find this discrepancy between what ixml and RELAX NG consider to be an error surprising and confusing for the following reason. The primary use of ixml processing, as described in , is:
Invisible XML (ixml) is a method for treating non-XML documents as if they were XML, enabling authors to write documents and data in a format they prefer while providing XML for processes that are more effective with XML content.
The computational pipeline that implements this functionality has three parts: Read and parse an ixml grammar. Read and parse an input document against that grammar. Serialize the result of parsing the input document against the grammar as well-formed XML by writing explicit markup into the parse results according to rules specified in the grammar. ixml treats failure in the first of these three stages (cannot parse and interpret the ixml grammar) as a static error, it treats failure in the third stage (cannot emit well-formed XML) as a dynamic error, but it treats failure in the second stage as Not An Error—that is, does not refer to failure as an error and it does not associate a numbered error message with this failure. We find this behavior unintuitive because: When ixml is used for its typical purpose of interpreting and processing non-XML input as XML, a lack of success in any of the three stages equivalently prevents the process from satisfying that purpose. A user might reasonably focus on the common frustrated outcome and not expect some lack-of-success situations to be modeled as errors and others as non-error failures. When ixml processing is used only as a recognizer (a use that is not, by the way, described in as a goal of the technology), users might expect recognizer-like (that is, RELAX-NG-like) behavior, and therefore error reporting (not just failure reporting) if the input document cannot be recognized as matching the grammar that is used to (attempt to) interpret it. For these reasons, our ixml-like application regards failure to parse an input document against a supplied grammar as a dynamic error.
Grammar fundamentals
Grammar terminology A grammar is expressed as a set of rules (called productions) and map from a left-hand-side (LHS) to a right-hand-side (RHS). For example, ixml article: metadata, content. says that an article can be matched by a sequence, in the input, of whatever matches a metadata followed by whatever matches a content. Syntactic operators that represent repetition, optionality, and alternation are common features of higher-level, human-facing grammars, such as XML schema languages and ixml. A lower-level grammar can be compiled for more convenient machine processing by rewriting the productions to remove those operators and introduce epsilon (ε, which signifies that that portion of a rule can be matched against no input). We discuss what we mean by higher and lower level in the Schemas and grammars section of . A context-free grammar (CFG) has a single symbol on the LHS and a single sequence of terminal and non-terminal symbols on the RHS. A mildly context-sensitive grammar (MCSG) also has a single symbol on the LHS, but it may have two sequences on the RHS. The relationship between the two sequences on the right is similar (although not identical) to RELAX NG interleave. RELAX NG interleave is often understood as all of these items in any order, but in addition to that type of unordered content, interleave is also designed to model partially-ordered content. See the section titled Partially ordered content: how Interleave works in for details. A MCSG rule can be matched only if all parts of both sequences on the RHS are matched in a way that conforms to the ordering of each of the individual sequences, and the rule may incorporate an ordering constraint that describes how the two sequences can be interleaved. Creole is a proposed schema language for next-generation markup languages, originally designed within the framework of the LMNL project. It is described in . In the Schemas and grammars section of we distinguished the term schema, for productions authored by users (as in a RELAX NG schema), and grammar, for a constrained set of productions, optimized for machine interpretation and processing, that is generated programmatically by compiling the schema into a grammar. While schema design prioritizes human legibility, grammar design prioritizes efficient computational implementation. That distinction is comparable to the one in between Extended Backus-Naur forms (EBNF) and Backus-Naur forms (BNF), which explains that BNF grammars are simpler to reason about and simpler for implementations to process. EBNF grammars are easier to author. The How it works section of similarly distinguishes an Invisible XML grammar, authored by a human with EBNF-like syntax and comparable to what we called a schema, and the Earley grammar into which it is compiled, expressed with BNF-like syntax and comparable to what we called a grammar. Because the established term in for the set of productions authored by a human is not schema, but grammar, for the present report we modify the terminology from and, in this discussion of ixml-like processing, we use the terms ixml-like grammar (instead of schema) and compiled grammar (instead of just grammar). We avoid the terms EBNF and BNF because different applications may use them to refer to slightly different sets of syntactic rules, about which see .
Grammars for next-generation features Next-generation processing requires two additional operators, one for concurrency (overlap) and one for sets (unordered content). In this report and in our implementation we use the following operators: For concurrency the Creole-like implementation in used the tilde character (~), adopted from . Because tilde in is defined as a not operator (e.g., ~[",."] matches any single character that is not a comma or a dot), in our ixml-like grammar notation we use a double tilde (~~) for concurrency. For unordered (set) content we use parentheses preceded by an ampersand, e.g., &(x0, x1, x2) matches exactly one instance each of the three nonterminals in any order. In our Creole-like application we followed Creole and RELAX NG and used an ampersand between terms for that purpose. RELAX NG does not permit interleave that shares operands. For example, the following RELAX NG schema: start = doc doc = element doc { a & b } a = x, y b = x, z x = element x { text } y = element y { text } z = element z { text } is not valid because A and b both include x. In keeping with the spirit of ixml, which permits ambiguity, we allow this pattern, which can be matched by x, x, y, z and others. In the case of x, x, y, z there is structural ambiguity concerning which x matches which of a and b with no way to decide, that is, with the interpretations converging on the same result. The principal container unit in is an element, and elements are subject to possible concurrency (overlap). We followed in recognizing also a need for a container unit that would suspend overlap, until the end of the group, in an otherwise overlapping context, which Creole called an element. The term element was not available for this purpose for us because it was already used for our primary container unit, so we represented a container unit that is not subject to concurrency with the keyword PARTITION, with no operator. In our ixml-like grammar a nonterminal that represents a partition is preceded by a double caret (^^). Our Creole-like schema language also models self-overlap, that is, overlaps of multiple elements of the same type. This sort of phenomenon is well known from annotation applications (such as https://web.hypothes.is/), where users might highlight overlapping strings of text and associate annotations with them. In our Creole-like schema language we adopted the Creole keywords concurOneOrMore and concurZeroOrMore for that purpose, with no assocated operators. We do not advocate for the specific notations we adopt here for these operators. Our options are constrained by established and sometimes contradictory interpretations of operator characters in RELAX NG, Creole, and ixml, which makes it difficult to rely on user familiarity with the notational conventions of those other specifications. We leave open the question of whether alternative operator characters might ultimately prove more useful than those that we employ here. For what it’s worth, the operators in question appear only in the human-facing Creole-like schema or ixml-like grammar, and not in the compiled grammar, where, as we write in , they are replaced by words like INTERLEAVE and CONCUR. It is unfortunate that allows alternative operators for the same concept, specifically : and = to separate the LHS and RHS of a production and | and ; for an or-group. The provision of alternatives does not contribute to the expressivity of the language and it reduces the inventory of operators that might be available for other purposes.
Types of grammars for ixml and ixml-like processing ixml processors generally use context-free grammars (CFG), but, in our implementation, both Creole-like schema validation and ixml-like recognition or transformation rely instead on mildly context-sensitive grammars (MCSG). We are grateful for Bethan Tovey-Walsh for her generous responses to some of the ideas we discuss here, and for providing some of the ixml grammars that we used as we developed those ideas. In particular, our support for partial order (interleave) requires functionality that is not available directly in a CFG. The RELAX NG validator uses parsing with derivatives for this purpose; we use a MCSG. We illustrate this functionality below.
Invisible markup, ambiguity, and concurrency
Thinking about invisible next-generation markup In this paper we begin to explore the possibility of implementing ixml-like processing by using the same types of mildly context-sensitive grammars (MCSG) that we employed in for validation against a schema that used Creole-like syntax. We now ask (and answer) three related questions: Is it realistic to expect to use a MCSG to parse implicit markup that employs next-generation features and to validate those features during parsing? As we illustrate below, we are able to parse implicit markup that employs next-generation features, such as overlap, and add markup explicitly, as ixml does when it adds explicit XML markup to input that it ingests as a stream of characters. Our model is also able to recognize unordered (set) models, and although RELAX NG interleave is capable of modeling set content during validation, ixml can recognize sets only by pregenerating all possible combinations. This may be viable for attributes on an element, which typically are not unmanageably numerous, and where ixml does not require it because it already recognizes the nonrepeatability attribute nodes, but it would fail with the potentially much larger models that correspond to unordered RDF statements or JSON dictionaries. The extent to which we are able to recognize implicit next-generation features requires further exploration and testing, but the examples below confirm that the goal is attainable at least for some types of documents and some types of implicit markup. Is the incorporation of a MCSG into an ixml-like parsing and validation process likely to be a useful feature for end-users? More specifically, what useful next-generation structural features of documents can be supported once we employ a MCSG? Our experiment below demonstrates that at least some types of implicit next-generation structuring can be identified and tagged with an ixml-like process. Insofar as adding explicit markup to documents motivated the development of ixml, we see no reason why the ability to add at least some types of next-generation markup should not prove equally useful. We are mindful here of what Norm Tovey-Walsh writes about the utility of ixml:
Bear in mind as you go that the goal of Invisible XML is to produce visible XML. That’s not necessarily the final output form that you want. It’s text to XML, not text to DocBook 5.2 with embedded MathML and SVG. There are other, better tools to transform one flavor of XML into another, Invisible XML doesn’t have to do all the lifting. ()
Can the incorporation of a MCSG into an ixml-like parsing and validation process reduce undesired grammatical ambiguity, where by undesired we mean ambiguity that is not included deliberately with the goal of capturing an ambiguity in the input that is intended to be modeled as ambiguity in the interpretation? For a discussion of when ambiguity might be a desirable feature of an ixml grammar see . Our experiment below does not explore ambiguity in ixml-like grammars extensively. With that said, the Hamlet excerpt has to navigate whitespace-handling issues comparable to those that frequently arise with ixml, and the same strategies that help manage that type of ambiguity in ixml are available in our ixml-like application. Furthermore, as we show below, the new model successfully resolves the type of ambiguity in ixml that corresponds to what we regard as concurrency. We also show, however, that ambiguity is possible even in the presence of concurrency. A thorough anaylsis of ambiguity and the mitigation thereof in ixml-like processing requires more attention that we are able to allocate in this initial exploration. The remainder of this section explores the experimentation with invisible next-generation markup that led us to answer the preceding questions, at least provisionally, as we did.
Invisible markup and next-generation features ixml is able to parse certain types of overlap, which it models as ambiguity. For example, assume we have the following excerpt from Shakespeare’s Hamlet: GERTRUDE Good Hamlet, cast thy nighted colour off, And let thine eye look like a friend on Denmark. Do not for ever with thy vailed lids Seek for thy noble father in the dust: Thou know'st 'tis common; all that lives must die, Passing through nature to eternity. HAMLET Ay, madam, it is common. GERTRUDE If it be, Why seems it so particular with thee? These three speeches contain eight lines of iambic pentameter, and the detail that interests us is that the seventh metrical line is split over two speakers, where Hamlet speaks seven syllables and Gertrude speaks three. We base this example on the one from Peer Gynt in . We can visualize these overlapping structures with the following graph:
The left-most (Speeches) and rightmost (Metrical lines) nodes both represent the same root element and the gray rectangles in the middle represent the plain text, all of which is part of both hierarchies. We have chosen to model speaker names as part of the first metrical line of the speech, but we connect them to the Speech and Line nodes with dotted edges to distinguish them from spoken text. The last five gray textual rectangles are modeled, to the left, as two speeches, one by Hamlet and one by Gertrude, and, to the right, as two lines, one split between Hamlet and Gertrude and the other by Gertrude alone.That is, at this location speeches and metrical lines overlap, and neither type wholly and consistently contains the other.
As we ingest and parse the plain text input, we identify the transition between speeches by recognizing that each speech begins with a speaker name and each speech ends before the next speaker name or at the end of the document. We identify the transition between metrical lines by recognizing that each metrical line begins at the beginning of the document or after another metrical line and ends as soon as we have accumulated ten or eleven syllables with a (whitespace-separated) word before a newline character. There are, to be sure, lines in the Shakespeare drama corpus that are not iambic pentameter. Some are in other verse meters, such as the famous trochaic tetrameter witches’ chant (Double, double, toil and trouble) in Macbeth; others cannot be parsed as any regular meter, such as Dogberry’s speeches in Much ado about nothing. The robust recognition of all types of structural lines, which would be a requirement for an application that could recognize and tag such lines, is not a natural task for ixml alone. Because English syllabification is not easily deduced from English orthography, we assume a pre-processing NLP operation that prepends a running syllable count to each word token (speech only, not counting speaker names), and resets when the total reaches 10 or 11. For example, English react has two syllables but reach has one. The output of that preprocessing step looks like: GERTRUDE 1Good 3Hamlet, 4cast 5thy 7nighted 9colour 10off, 1And 2let 3thine 4eye 5look 6like 7a 8friend 9on 11Denmark. 1Do 2not 3for 5ever 6with 7thy 9vailed 10lids 1Seek 2for 3thy 5noble 7father 8in 9the 10dust: 1Thou 2know'st 3'tis 5common; 6all 7that 8lives 9must 10die, 2Passing 3through 5nature 6to 10eternity. HAMLET 1Ay, 3madam, 4it 5is 7common. GERTRUDE 8If 9it 10be, 1Why 2seems 3it 4so 8particular 9with 10thee? Our goal is to tag both speeches (there are three: by Gertrude, then Hamlet, and then Gertrude again) and metrical lines (there are eight: six by Gertrude, then one shared by Hamlet and Gertrude, and then one more by Gertrude). A meaningful feature of this task is that the overlapping hierarchies are fully tesselated, that is, that the document could be tagged in XML as an uninterrupted (except by insignificant whitespace) sequence of metrical lines, which, in our modeling, may have speaker-name children, or as a sequence of of uninterrupted (except by insignificant whitespace) speeches, which must have, as a first child, a speaker-name element. These are easily modeled with ixml, as follows: For the sequence of metrical lines, the ixml grammar: hamlet: (m_line)++eol. m_line: (token; speaker)+, f_token. -f_token: ddigit, [L; P]+. -token: digit, [L; P]+, (sp; eol). speaker: ["A"-"Z"]+, eol. -digit: -["0"-"9"]. -ddigit: digit, digit. -eol: -#a. -sp: " ". produces the following unambiguous output: <hamlet xmlns:ixml="http://invisiblexml.org/NS" ixml:version="1.1+"> <m_line> <speaker>GERTRUDE</speaker>Good Hamlet, cast thy nighted colour off,</m_line> <m_line>And let thine eye look like a friend on Denmark.</m_line> <m_line>Do not for ever with thy vailed lids</m_line> <m_line>Seek for thy noble father in the dust:</m_line> <m_line>Thou know'st 'tis common; all that lives must die,</m_line> <m_line>Passing through nature to eternity.</m_line> <m_line> <speaker>HAMLET</speaker>Ay, madam, it is common.<speaker>GERTRUDE</speaker>If it be,</m_line> <m_line>Why seems it so particular with thee?</m_line> </hamlet> For the sequence of speeches, the ixml grammar: hamlet: speech++eol. speech: speaker, eol, line++eol. speaker: ["A"-"Z"]+. line: word++sp. -word: digits, [L; P]+. -digits: -["0"-"9"]+. -eol: -#a. -sp: " ". produces the following unambiguous output: <hamlet> <speech> <speaker>GERTRUDE</speaker> <line>Good Hamlet, cast thy nighted colour off,</line> <line>And let thine eye look like a friend on Denmark.</line> <line>Do not for ever with thy vailed lids</line> <line>Seek for thy noble father in the dust:</line> <line>Thou know'st 'tis common; all that lives must die,</line> <line>Passing through nature to eternity.</line> </speech> <speech> <speaker>HAMLET</speaker> <line>Ay, madam, it is common.</line> </speech> <speech> <speaker>GERTRUDE</speaker> <line>If it be,</line> <line>Why seems it so particular with thee?</line> </speech> </hamlet> We can merge the two ixml grammars by changing the first rule (for hamlet) to allow either one or the other, that is, to separate the two with an or connector (;). No other structural modifications to the two single-hierarchy grammars are required, except ensuring that no nonterminal is defined more than once. In our example the only nonterminals defined in both single-hierarchy grammars are defined identically, which means that we can satisfy the requirement in question by removing the duplicates. In this example no non-terminal is defined differently in the two single-hierarchy grammars, so there are no inconsistencies that need to be reconciled. Below is an ixml grammar that does that, which incorporates the following changes: A shared section creates a top-level rule that matches either of the two hierarchies and unifies shared rules for end-of-line and space. The models for lines were different in the two independent grammars, where the one for metrical lines is m_line: (token; m_speaker)+,f_token. and the one for speeches is line: word++sp. We use renaming to output <line> for both. The models for speaker names were different in the two independent grammars, where the one for metrical lines is m_speaker: ["A"-"Z"]+,eol. and the one for speeches is speaker: ["A"-"Z"]+. We use renaming to output <speaker> for both ixml version "1.1". { shared } hamlet: (m_line)++eol; speech++eol. -eol: -#a. -sp: " ". { metrical only } m_line>line: (token; m_speaker)+, f_token. -f_token: ddigit, [L; P]+. -token: digit, [L; P]+, (sp; eol). m_speaker>speaker: ["A"-"Z"]+, eol. -digit: -["0"-"9"]. -ddigit: digit, digit. { speech only } speech: speaker, eol, line++eol. speaker: ["A"-"Z"]+. line: word++sp. -word: digits, [L; P]+. -digits: -["0"-"9"]+. This grammar is ambiguous, with two possible parses, which correspond to the outputs of the two individual grammars: <ixml parses="2" totalParses="2"> <hamlet xmlns:ixml='http://invisiblexml.org/NS' ixml:state='ambiguous'> <line> <speaker>GERTRUDE</speaker>Good Hamlet, cast thy nighted colour off,</line> <line>And let thine eye look like a friend on Denmark.</line> <line>Do not for ever with thy vailed lids</line> <line>Seek for thy noble father in the dust:</line> <line>Thou know'st 'tis common; all that lives must die,</line> <line>Passing through nature to eternity.</line> <line> <speaker>HAMLET</speaker>Ay, madam, it is common. <speaker>GERTRUDE</speaker>If it be,</line> <line>Why seems it so particular with thee?</line> </hamlet> <hamlet xmlns:ixml='http://invisiblexml.org/NS' ixml:state='ambiguous'> <speech> <speaker>GERTRUDE</speaker> <line>Good Hamlet, cast thy nighted colour off,</line> <line>And let thine eye look like a friend on Denmark.</line> <line>Do not for ever with thy vailed lids</line> <line>Seek for thy noble father in the dust:</line> <line>Thou know'st 'tis common; all that lives must die,</line> <line>Passing through nature to eternity.</line> </speech> <speech> <speaker>HAMLET</speaker> <line>Ay, madam, it is common.</line> </speech> <speech> <speaker>GERTRUDE</speaker> <line>If it be,</line> <line>Why seems it so particular with thee?</line> </speech> </hamlet> </ixml> This application of renaming is seductive because it is easy to implement and because a human intuitively understands that the concept line has different meanings with respect to meter and orthographic lines in a printed script. At the same time, applying this type of renaming in an XML context associates the same element type with two different meanings, which impinges on the ability of the markup to make sameness and difference accessible in an intuitive way to automated processing. It is, to be sure, easy to preserve different names for the two types of lines, should that prove more useful down stream. Our next task was to see whether ixml could create LMNL output that could contain overlapping markup in a single serialization. LMNL is, of course, able to represent overlap, but ixml is able to serialize only as XML. We work around that discrepancy in the following ways: We create a root XML element around the LMNL markup so that the LMNL content will satisfy the requirements for well-formed XML. Any subsequent LMNL processing would need to ignore that XML wrapper element. We suppress all XML tags and create LMNL tags using ixml insertions. Here is an unambiguous ixml grammar that creates LMNL output: hamlet: +#a, +"[hamlet}", +#a, first_speaker, body, +"{line]", +#a, +"{hamlet]", -#a?, +#a. { First line precedes line traversal, open first line and first speech and process speaker Transitions below depend on newline, and first line has no preceding newline } -first_speaker: +"[line}[speech}[speaker}", speaker, +"{speaker] ", no_eol. -body: soft_line; hard_line. { STATE MACHINE: each line transition signals one of four possible states based on end of line } { STATE 1: Line ending in a single-digit token continues metrical line with new speaker } -soft_line: token_sp*, token_end, -#a, soft_speaker_change. { STATES 2, 3, 4: Line ending in a double-digit f_token STATE 2: new metrical line continues with same speaker STATE 3: new metrical line changes speaker STATE 4: no new metrical line means end of file } -hard_line: token_sp*, f_token_end, (hard_continuation; hard_speaker_change; final_line). { Inject a space explicitly between mid-line speech boundaries and continue } -soft_speaker_change: +"{speech] [speech}[speaker}", speaker, +"{speaker] ", no_eol, body. { After a hard line ending insert line transition tags with newline between and continue } -hard_continuation: +"{line]", #a, +"[line}", body. { Remove space character between closing speech and line tags } -hard_speaker_change: +"{speech]{line]", #a, +"[line}[speech}[speaker}", speaker, +"{speaker] ", no_eol, body. -final_line: +"{speech]". { THREE TYPES OF METRICAL-LINE TOKENS, PLUS SPEAKER: token_sp: Mid-line tokens must be followed by a space token_end: Line-final token that is not end of metrical line f_token_end: Line-final token that signals end of metrical line speaker: all upper case } -token_sp: digit, [L; P]+, sp. -token_end: digit, [L; P]+. -f_token_end: ddigit, [L; P]+. -speaker: ["A"-"Z"]+. { Misc } -digit: -["0"-"9"]. -ddigit: digit, digit. -no_eol: -#a. -sp: " ". The output is: <hamlet> [hamlet} [line}[speech}[speaker}GERTRUDE{speaker] Good Hamlet, cast thy nighted colour off,{line] [line}And let thine eye look like a friend on Denmark.{line] [line}Do not for ever with thy vailed lids{line] [line}Seek for thy noble father in the dust:{line] [line}Thou know'st 'tis common; all that lives must die,{line] [line}Passing through nature to eternity.{speech]{line] [line}[speech}[speaker}HAMLET{speaker] Ay, madam, it is common.{speech] [speech}[speaker}GERTRUDE{speaker] If it be,{line] [line}Why seems it so particular with thee?{speech]{line] {hamlet] </hamlet> We note that this is not a general-purpose invisible LMNL processing model for at least the following reasons: It works with MNML, a restricted subset of LMNL, and not with the full range of markup permitted by the original LMNL model. Concerning MNML see and . It takes advantage of the simple and regular structure of the input, which has only five types of event (other than start and end): speaker, metrical-line-non-final token, metrical-line-final token, space, eol. It is theoretically possible for a line of iambic pentameter to end with a monosyllable as the eleventh syllable, which means that we would have 10word 11word at the end of the input line, which our grammar cannot match. As a simplification in the interest of clarity we omit that possibility here. There is no self-overlap. We have two independent hierarchies, which requires us to map out all possible interactions of the two: 1) same metrical line continues with new speaker, 2) new metrical line with same speaker, 3) new metrical line with different speaker, and 4) end of input. These combinations would increase rapidly if there were additional overlapping hierarchies. A concurrency operator would allow us to describe the hierarchies separately and assert that they concur (that is, that they overlap), without our having to manage, in the grammar that we write, all possible states that result from the interactions of speech-related state transitions and metrical-line-related state transisions. Our ixml grammar with ambiguity, above, comes close to this model; the principal difference is that because ixml lacks a concurrency operator, we fall back on describing concurrency as ambiguity. An ixml processor cannot serialize concurrency because it is restricted to serializing XML. That is a serialization concern and independent of the capabilities of the grammar. As we discuss below, however, not all ambiguity can be modeled as concurrency, which means that when we build on ixml capabailities, below, by adding both a concurrency operator (and the ability to process it without generating ambiguity) and a non-hacky serialization capability, those enhancements do not preclude the construction of other types of ambiguous grammars that yield ambiguous parses. Because ixml can emit only well-formed XML, our LMNL output is wrapped in an XML root element, so that from an XML perspective the root element has just a single child, which is a text node. This means that the output is not actually LMNL, and should we wish to embed this ixml operation within a pipeline, we would have to handle the XML root element. The most important detail is that the ixml that produces our LMNL output is complex because it needs to recognize and act on the interaction of two hierarchies, and not on each of them separately from the other. Non-ambiguous grammars that recognize transitions in these two hierarchies independently of each other are not supported directly in ixml, and we work around that limitation by consuming tokens from both the metrical-line and the speech hierarchies before recognizing and responding to a specific state that depends on sequences of those two types of tokens. Our ixml grammar that emits LMNL would be simpler if we allowed it to recognize and process transitions in the two hierarchies one by one, so that each could be managed independently of the other. LMNL can regard adjacent tags as simultaneous, but for human legibility we wanted to avoid transitions like: [line}Passing through nature to eternity.{line] [line}{speech][speech}[speaker}HAMLET{speaker] Ay, madam, … This requirement frustrated our ability to treat both line and speech transitions simply as milestones. For example, if we wanted always to rewrite an incoming metrical line transition (newline) as {line]\n[line} and also always to rewrite an incoming speech transition as {speech] [speech}, it would not be possible to emit {speech]{line]\n[line}[speech}. Our grammar works around this limitation successfully, but only by complicating the set of productions, and doing so in a way that does not scale well with the incorporation of additional overlapping hierarchies.
Input issues
Parsing input that includes explicit next-generation markup The ixml approach to LMNL processing illustrated in starts with input that contains explicit LMNL markup delimiters, and the pipeline uses ixml to parse that input into xMNML (XML representation of Minimally Annotated Markup in LMNL, a LMNL subset). xMNML uses empty elements, attribute pointers, and other standard XML resources to work around the inability of XML to represent next-generation features (overlapping ranges in the case of LMNL) directly in the model, that is, in ways that do not rely on domain-specific interpretation and processing conventions. This parsing, using a library that Piez calls the Laminator, replaces Piez’s earlier XSLT and XPath Luminescent library, recognizing that ixml, which was not available when Luminescent was first developed, is well-designed for this type of task. Subsequent processing of the tagged LMNL input, after conversion via ixml to xMNML, can then use the standard XML toolkit. For example, as Piez has demonstrated, subsequent processing can use XSLT to transform an xMNML representation into HTML with SVG to render visualizations of overlapping structures using standard features of existing web browsers. Because currently there are no robust toolkits for markup languages that support next-generation features, parsing such markup into an XML representation of the data model and then employing XML tools is a natural work-around for processing next-generation markup today. If, though, we imagine a future with processing toolkits (transformation, query, path expressions, etc.) for at least some popular next-generation markup languages, the construction of an XML representation of the data would not be necessary; a pipeline could parse tagged input into an internal, markup-syntax-agnostic representation of a data model that could then be processed in ways that would not depend on XML syntax and XML tools. We bracket for now the question of what this internal data model might look like. Next-generation languages like TAGML (representing a TAG model; and ), TexMECS (representing a GODDAG model; ), and LMNL (representing a LMNL model; ) model relationships among markup units in somewhat different ways, even where the underlying conceptual features (such as overlap) are the same or similar. We see no reason that these differences cannot converge on a model that reconciles and accommodates them, but we regard the elaboration of such a model as a separate, although related topic, and one that we intend to address elsewhere. This type of future system could be preconfigured to recognize a variety of markup languages, but because, like RELAX NG and Creole as implemented in , it operates with prior information about and knowledge of language-specific markup tokens, it would nonetheless be constrained to handle only tokenizers that are supported explicitly. An ixml-like approach to recognition and validation, on the other hand, operates at the character level without applying any built-in, markup-language-specific distinction between markup and content during input parsing, which means that if the input already contains markup, a user-specified grammar could assume responsibility for recognizing and managing it. Insofar as ixml-like rules could be written for any markup language, such a system could be truly agnostic about input markup, and therefore equally applicable to any next-generation markup language (and, for that matter, also to XML). Ongoing discussion of modular ixml, which is not currently part of , invites the possibility of implementing ixml-like markup tokenizers for specific markup languages as ixml-like modules that might be standardized, distributed, and then imported as needed. Markup tokenizers for languages not covered by such imports could be implemented by end-users, much as, as has shown, it is both possible and useful to recognize XML markup tokens with ixml-like processing even though ixml input tokenization has no built-in knowledge of XML. There is, of course, no free lunch. The Creole-like approach is limited to processing the specific markup and schema languages it knows about, but although the ixml-like approach is agnostic about markup and does not require a schema language beyond the rules implemented through an ixml-like grammar, someone nonetheless needs to write that grammar. End-users might, then, anticipate a tooling situation like the following: Where resources might come to exist for relatively popular next-generation languages, such as LMNL or TexMECS, those resources, implemented as Creole-like validators or ixml-like grammar modules, might reasonably meet the validation needs of end-users. Where projects employ domain-specific markup languages for which schema validation and processing toolkits do not exist, the flexibility of the ixml-like methods, which expose parsing rules for recognizing markup more directly to users than Creole-like methods, could meet validation needs, albeit at the expense of requiring users to write their own ixml-like grammars. Insofar as the Creole-like approach is only about validation, and not about inserting new markup into tagged (or not-yet-tagged) input, only an ixml-like approach would enable users both to recognize implicit structures and to insert additional explicit markup into documents and emit enriched output. A key feature of our perspective and approach is that both Creole-like validation and ixml-like modification can use exactly the same data structures and code base, which allows us to implement both Creole-like and ixml-like tools as thin wrappers around that shared data model and code base. To the best of our knowledge this is not currently the case with existing XML tools, where the only RELAX NG validator in wide use () is implemented through parsing with derivatives (, ) and the ixml tools listed at https://invisiblexml.org/ appear to be implemented in other ways (e.g., with Earley or GLL parsers). We are not able to explore parsing input that already contains next-generation markup in this paper, but we intend to return to it in the future.
Managing unordered content with validation while parsing Many types of documents model information that is inherently unordered. These include: RDF statements RDBMS records and fields JSON dictionary items ixml can parse and validate inherently unordered structural components, and it naturally recognizes that XML attributes associated with the same parent element are not ordered. XML schema languages similarly recognize that attributes are not ordered, as does XML processing (so that, for example, two XML elements are deep-equal regardless of the order of their serialized attributes). At the same time, the XML data model does not distinguish ordered from unordered element content. Whether sibling elements are ordered can be specified in a RELAX NG schema through the use of interleave, but a document instance does not represent that difference directly. XML processing, such as with XSLT, can treat content as ordered or not, but that becomes a feature of the application, and not of the document. Insofar as unordered content is a feature of some documents, the ability to represent it at the document level is a next-generation feature, that is, one that is not part of the design of XML. CFGs on their own cannot represent unordered content except in a brute-force way. For example, if an XML element must contain exactly one instance each of elements A, B, and C, in any order, ixml can represent that requirement only by fully listing all permutations. RELAX NG can represent this structure using interleave, but interleave is not context free. Spelling out all possible combinations quickly becomes unrealistic, at least in a human-authored schema or grammar, as the number of possibilities grows. A common workaround for this limitation is the use of a repeatable or-group, although such a statement would, incorrectly, allow an individual element to repeat. Furthermore, ixml does not support numerical quantification, so although it is easy to express unlimited repetition (using a Kleene star or plus), the only way to model a specific number of repetitions is by repeating the item literally the required number of times. This, too, is easy to manage with a small number of repetitions but increasingly difficult as that number grows. In the context of XML schema languages, in addition to RELAX NG interleave, grammatical rules based on structures like repeatable or-groups can be supplemented by Schematron constraints. This is a popular and effective approach to working around the inherent limitations of CFGs. This raises the question of whether an ixml-like strategy built on top of a MCSG could overcome that limitation and make it possible to validate unordered content during parsing without invoking extra-grammatical operations. That a MCSG grammar can support unordered content is clear because, among other things, we use it in for validation. We are not able to explore the possibility of employing this feature for markup generation in this paper, but we intend to return to it in the future.
Ambiguity not resolved by concurrency Standard ixml parses the input into a forest, from which it then selects and serializes (or, rather, attempts to serialize) one tree. A MCSG, though, allows us to regard multiple parses of the same input not as ambiguity, but as concurrency. If we think of ambiguity as this or that and concurrency as both this and that simultaneously, we might model ambiguity as multiple trees, from which we then select exactly one for serialization, and concurrency as a graph, that is, a forest within which all trees are simultaneously available and selection among them is not part of the model. The preceding suggests two points of comparison, input parsing and output serialization, as follows: With respect to input parsing, as we write above, a production in a MCSG can have two patterns in the RHS, both of which must be satisfied for the input to match the rule. Standard ixml regards two patterns that match the same input as competing, so that both cannot be serialized in the same output, which is the terminology of ambiguity. Parsing with a MCSG regards this matching as simultaneity, so that both can be serialized in the same output, which is the terminology of concurrency. Concurrency is a standard feature of next-generation modeling. A standard ixml forest and the graph that we construct when parsing with a MCSG are structurally similar in that in both cases a node in the representation of the parsing operation may have multiple in-edges and multiple out-edges. What is different is the semantics associated with multiple paths (alternative in ixml vs simultaneous with next-generation modeling) and the implications of those semantics for serialization (standard ixml cannot serialize multiple paths simultaneously because it must serialize a tree, a limitation that does not apply to our next-generation modeling). The preceding addresses only the type of ambiguity that arises in ixml when the same input matches multiple patterns. It does not address some other types of ambiguity. Here are two examples: Example #1: The following ixml grammar is ambiguous: doc: CAT; ANIMAL. CAT: "cat". ANIMAL: "cat". When used to parse the input cat, it would report ambiguity, with the following two parses: <doc ixml:state="ambiguous"> <CAT>cat</CAT> </doc> <doc ixml:state="ambiguous"> <ANIMAL>cat</ANIMAL> </doc> The same grammar would also be ambiguous under our proposed enhancements for next-generation features. The difference is that our proposal would also support the following alternative grammar (where ~~ means CONCUR), where: doc: CAT ~~ ANIMAL. CAT: "cat". ANIMAL: "cat". The single unambiguous LMNL output of parsing the input cat against this grammar is [ANIMAL}[CAT}cat{ANIMAL]{CAT] If we regard the context as overlap, that is, if we want to tag the text as both <CAT> and <ANIMAL> simultaneously, we could select the concur operator, and if we want to regard the document as having two competing parses, we could select the alternation operator. Our point with this example is that if we want to describe the situation as an alternative, the availability of a concur operator does not help avoid ambiguity because the concept of alternative, in this example, is inherently structurally ambiguous. Example #2: Given the Creole-like grammar like the following: doc: A+ ~~ B+. A: [“x”]+. B: [“x”]+. This says that a doc contains one or more instances of A that run concurrently with one or more instances of B. Since the only content of both A and B is x, it means that every instances of the character x must be within both an A and a B. This, in turn, means that: The input x is unambiguous and resolve to LMNL [A}[B}x{A]{B]. (Recall that the order of concurrent tags in serialized LMNL is not significant.) The input xx (and input with more instances of x) is ambiguous. The input xx can be tagged with the preceding grammar in four ways: [A}[B}xx{A]{B] (one instance of A and one of B) [A}[B}x{B][B}x{A]{B] (one instance of A and two of B) [A}[B}x{A][A}x{A]{B] (two instances of A and one of B) [A}[B}x{A]{B][A}[B}x{A]{B] (two instances of A and two of B) The examples above correspond to what calls vertical (our example #1) and horizontal (our example #2) ambiguity. The point of this example is that although concurrency is a way of understanding some instances of ambiguity, here we have both vertical and horizontal ambiguity even within a concurrent model.
Serialization considerations Standard ixml parses the input into a forest, from which it then selects and serializes (or, rather, attempts to serialize) one tree. A grammar that supports concurrency, though, allows us, at least in some situations, to regard multiple parses of the same input not as ambiguity, but as concurrency. If we think of ambiguity as this or that and concurrency as both this and that simultaneously, we might model ambiguity as multiple trees, from which we can then select exactly one at a time for serialization, and concurrency as a graph, that is, a forest within which all trees are simultaneously available and selection among them is not part of the model. A forest with concurrent hierarchies could then be serialized, in its entirety, with a next-generation markup language like LMNL or TexMECS or TAGML. The preceding suggests two points of comparison, input parsing and output serialization, as follows: With respect to input parsing, as we write above, standard ixml regards two patterns that match the same input as competing alternatives, so that both cannot be serialized in the same output, which is the terminology of ambiguity. Parsing with a concurrency operator regards this matching as simultaneity, so that both can be serialized in the same output, which is the terminology of concurrency. Concurrency is a standard feature of next-generation modeling. A standard ixml forest and the graph that we construct when parsing with a MCSG are structurally similar in that in both cases a node in the representation of the parsing operation may have multiple in-edges and multiple out-edges. What is different is the semantics associated with multiple paths (alternative in ixml vs simultaneous with next-generation modeling) and the implications of those semantics for serialization (standard ixml cannot serialize multiple paths simultaneously because it must serialize a tree, a limitation that does not apply to our next-generation modeling). The preceding suggests that where users need to model concurrency, parsing with a concurrency operator and modeling as a graph that can be serialized in its entirety offers a formal model that matches the mental model. Furthermore, it does so without structural ambiguity because instead of having to choose between coexisting paths it regards all paths as simultaneously available.
Comparing processing with schema languages and processing with ixml-like grammars
How validation and generation are similar and different The Creole-like and ixml-like technologies described above are similar on both the syntactic and modeling levels, which we elaborate below.
RELAX NG schema validation and ixml processing Within XML technologies the roles of RELAX NG and ixml do not normally overlap. The normal role of RELAX NG is schema validation, that is, verifying whether an XML document with explicit markup conforms to a RELAX NG schema. The normal role of ixml is to add explicit markup to character-stream input, which is typically either plain text or non-XML structured text, such as JSON. ixml can read XML input as a character stream and it would be possible to use ixml processing to perform most of RELAX NG schema validation, that is, to construct an ixml grammar that implements most of the same modeling as a RELAX NG schema, but it would peculiar to do so when the task is easier with RELAX NG, which, after all, was designed specifically for that purpose. Although ixml always recognizes whether a specific document can be parsed against a specific grammar, complications arise with trying to use it to mimic schema validation because it was not developed specifically for that purpose. For example: Unlike RELAX NG, ixml does not come with a knowledge of XML markup conventions, which means that parsing XML input with RELAX NG means that the ixml grammar, which reads the input as a character stream, would be responsible for distinguishing markup from content. ixml is not designed to represent RELAX NG interleave, which would have to be simulated and approximated if expressed through an ixml grammar. Although ixml is required to return only one path through the forest constructed during parsing, it needs to know whether there is more than one because it is expected to report ambiguity. says that a ixml processor should report ambiguity, and that processors may vary in whether input is detected as ambiguous or not. This technically means that ixml parsers are not required to detect or report ambiguity when it is present, but, as far as we know, all current ixml applications detect and report ambiguity. Ambiguity is a property of grammars, and not of input documents, but parsing with an ambiguous grammar may produce multiple paths through the forest with some input and not with other input. If parsing a specific input document aginst an ambiguous grammar produces more than one path, strongly suggests (should, but not must) that a conformant processor report the presence of ambiguity. Because RELAX NG is about validation, its only responsibility is to determine whether the instance is a document in the language described by the schema. Explicit XML markup describes a single tree, which means that ambiguous paths through a forest is not part of the RELAX NG processing model. The output tasks of RELAX NG validation and ixml processing diverge substantially, so although ixml processing includes determining whether an input document does or does not conform to an ixml grammar, and whether a well-formed output document can be emitted, parsing failures are not reported in the same way as they are with RELAX NG validation.
Purpose Much as RELAX NG is intended to be used as a recognizer and ixml is intended to be used to add explicit markup, our Creole-like validation is intended to be used as a next-genreation markup recognizer and our ixml-like processing is intended to be used to add explicit next-generation markup. The considerations that distinguish RELAX NG processing from ixml do not intefere with our using a shared code base for parsing. Because the goals and behaviors of the two types of processing diverge, some other types of functionality cannot be shared, but those differences do not impinge on the shared parsing. Specifically: The code base currently contains tokenizers for LMNL and TexMECs, which are used by our Creole-like validation. Our ixml-like processing does not currently use the built-in tokenizers, so, as with ixml processing, a user-supplied ixml-like grammar would be responsible for recognizing markup tokens already in the input. We have not yet explored the extent to which the built-in tokenizers might be useful for ixml-like processing of input already tagged in LMNL or TexMECS. Support for interleave is built into our code base, and therefore equally available to our Creole-like validation and our ixml-like processing. Our shared code base means that our ixml-like processing recognizes ambiguity. That recognition is not needed by our Creole-like validation, but its presence does no harm. Creole-like validation and ixml-like processing requires different reporting modules for the results of the two types of operations. Those differences do not interfere with our using shared parsing technology for both.
Domain RELAX NG and ixml are designed to model exactly the same domain: a universe that includes any possible XML document. Both can describe any type of XML node and node content or value. RELAX NG can be used to validate any XML document and ixml can be used to create any XML document in the XML document space. The only difference in what the two technologies can and cannot model is that RELAX NG can describe interleave idiomatically, while ixml, which is context-free, would have to simulate it. There are some differences in syntax between the two technolgoies, to be sure, and we discuss those below, but the domain of any XML document is the same. Similarly, validation and the insertion of explicit markup in the domain of a next-generaion markup like LMNL or TexMECS also use a shared model, and our use of a shared code base builds on that common model.
Input and output RELAX NG incorporates a tokenizer that recognizes XML markup tokens during input, while ixml input tokenization is character based, and the same is true of our Creole-based validation and our ixml-like markup generator. A tokenizer, or lexer, reads character-stream input and groups the characters to recognize higher-level predefined tokens. write that:
The lexer decides what parts of the input are the actual tokens. It might remove insignificant whitespace, strip out comments, normalize strings, etc. The parser then operates on those tokens. […] If Invisible XML was going to work this way, it would have to provide two sets of rules, one for lexing and one for parsing. That would add all kinds of complexity to the language. Instead, Invisible XML has the simplest possible lexer: your input string becomes a sequence of characters. What you’re matching in rules are the individual characters of the input: every single one of them.
A single character might technically be regarded as an input token, but in the absence in ixml of grouping characters into higher-level predefined tokens and the absence of separate lexing and parsing stages in the processing, it might be reasonable to say not just that an ixml processor uses a different type of tokenizer, but that it has no tokenization step that is separable from parsing at all. Similarly, ixml incorporates code to create XML markup tokens during output and our code base contains code that knows how to emit LMNL and TexMECS markup tokens. Our application also includes code for emitting XML markup, originally developed to explore the creation of XML with traditional workarounds to represent next-generation features. We have not prioritized the further development of this feature because it requires decision-making that we have not yet explored adequately, such as, for example, how to determine which hierarchies to flatten in the case of overlap. Currently our code base uses different code for recognizing markup tokens during input and creating them during output; we have not yet explored whether our use of a common code base offers an opportunity for sharing code between input markup recognition and output markup generation.
Grammar and syntax Both RELAX NG and our Creole-like system, on the one hand, and ixml and our ixml-like processing, on the other, rely on grammar productions that describe non-terminals (LHS) in terms of terminals, other non-terminals (RHS), and operators. Both support rules for elements (or something similar, such as LMNL ranges), attributes (or LMNL annotations), and (sub)patterns (not serialized as markup). There are, at the same time, conspicuous differences in the syntax defined by the two specifications:: RELAX NG and our Creole-like syntax have element and attribute operators, while ixml and our ixml-like syntax define nodes in the parse forest on the RHS without those operators. One consequence of this difference is that ixml and our ixml-like system do not distinguish types of nodes (those serialized as elements, as attributes, or without markup) during parsing (beyond, perhaps, decorating the node in the parse forest with a type property that can be consulted during serialization). This is why, for example, an ixml grammar can define an attribute node that is the root of a parse tree, and the parsing itself will succeed, although the result cannot be serialized, and will fail with D05. For example, the following grammar attempts to create a root node that is an attribute, a structure that is not permitted in XML and that is recognized as dynamic error D05 in : @doc: A, B. @A: "a". B: "b" The Early grammar compiled for this by Coffeepot is: 1. $$ ::= doc 2. doc ::= A, B 3. A ::= 'a' 4. B ::= 'b' (note that the grammar does not mention elements or attributes), but when Coffeepot attempts to serialize the result of applying this grammar to matching input, it correctly reports an error: The root node (‘doc’) must not be marked ‘@’. The element and attribute operators in RELAX NG allow recursive structures on the RHS, so that, for example, an entire RELAX NG schema can be expressed as a single recursive rule, an organizational pattern known as Russian doll. The term Russian doll is based on the matryoshka, a set of nested dolls that is famous in Russian folk art. For a description of the types of patterns that can be used (alone or combined) to organize a schema (written for XML Schema but largely applicable also to RELAX NG) see . For example: start = element doc { element section { element p { text }+ }+ } says that a <doc> contains one or more <section> children, each of which contains one or more <p> children, each of which contains plain text. The keyword start is defined as a production for a <doc> element, which contains, as its RHS, a production for a <section>, which contains, as its RHS, a production for a <p> element. There are no labels for the elements; labeling is used only when a label (not full production rule) used in a content model is, in turn, described in its own separate production. The following schema, which uses labels instead of nested productions, is equivalent to the one above: start = doc doc = element doc { section+ } section = element section { p+ } p = element p { text } Because ixml and our ixml-like processing lack the element and attribute operators, the RHS of productions in those language cannot embed other complete (both LHS and RHS) productions. This means that ixml and ixml-like grammars are necessarily flat, while the degree of nesting in a RELAX NG or Creole-like schema is under the control of the developer. imxl supports the ++ and ** separator operators, which are useful for resolving modeling that might otherwise be challenging to express unambiguously. These operators are not present in RELAX NG, but our code base needs to recognize them to support our ixml-like processing. Separator notation is not strictly necessary even for ixml, since any expression that uses it can be rewritten without it (and is rewritten during compilation), but its use in the grammars authored by humans greatly improves the legibility of the productions. Although both RELAX NG (and Creole-like schemas) and ixml (and ixml-like processing) appear to support renaming elements and attributes, the similarity is superficial in the following way: Parsing RELAX NG builds a tree of expectations. Labels used on the LHS of compact-syntax productions (e.g., the para in RELAX NG para = element p { text }) are not the name properties of nodes in that tree. While to a user it might appear that a para in the content model is being rewritten as a p, a LHS label is a feature only of the compact syntax, and not of the tree. The LHS labels might be retained when the schema is processed, since they could be useful for error reporting, but the name property of the node in the tree is based on the value after the element or attribute keyword, and not on the value in the label on a possible LHS. When parsing ixml (and our ixml-like grammar), on the other hand, the name associated with the node in the resulting parse forest is the value on the LHS of the grammar production. In ixml 1.0 that name value is part of the serialization of the node. ixml 1.1 permits renaming (e.g., A>B: "x"), but the name associated with the node in the parse forest is the name before renaming, and the name after renaming is associated with the node during serialization. We can see this when we look at the compilation of an ixml grammar that uses renaming. For example, the ixml grammar: doc: A. A>B: "x". compiles to: 1. $$ ::= doc 2. doc ::= A 3. A ::= 'x' When we parse a document with the value x against this grammar, it produces: <doc> <B>x</B> </doc> Note that although the element name B is used during serialization (and is therefore associated with A in some way), the name of the node in the grammar after compilation is A. The difference between the RELAX NG (and Creole-like) vs ixml (and ixml-like) behavior may appear superficial to an end-user, who might reasonably understand both as sharing the meaning of call it A in a RHS but spell it B in the output. The difference nonetheless matters for application development because an ixml (and ixml-like) renamed node has one name in the parse forest and different name in serialization, while an apparently renamed node in a RELAX NG (and Creole-like) schemas has the same name after parsing as it will use during serialization. The following are apparent or historical differences, and in these cases RELAX NG and ixml have similar capabilities, even though those may be (or have previously been) expressed differently syntactically: RELAX NG and our Creole-like system allow multiple start values, e.g., start = element start { A | B} ixml and our ixml-like system support this same feature through suppressing the serialization of an element node for the root of the grammar, as in: -start: A; B. A: "a". B: "b". Repetition indicators are greedy in RELAX NG and not in ixml, which is a common source of horizontal ambiguity. An ixml grammar like: doc: A, B. A: "x"*. B: "x"*. can parse xxx in three ways: <ixml parses="3" totalParses="3"> <doc xmlns:ixml='http://invisiblexml.org/NS' ixml:state='ambiguous'> <A>x</A> <B>x</B> </doc> <doc xmlns:ixml='http://invisiblexml.org/NS' ixml:state='ambiguous'> <A/> <B>xx</B> </doc> <doc xmlns:ixml='http://invisiblexml.org/NS' ixml:state='ambiguous'> <A>xx</A> <B/> </doc> </ixml> The addition of support for greedy matching of repeated patterns is under active discussion within the ixml Community Group.
What those similarities and differences mean for tool development Our approaches to schema validation (Creole-like) and to adding markup (ixml-like) are sufficiently similar that we found it useful to explore the development of a shared data model and a shared code base. Below we discuss where our Creole-like processing and our ixml-like processing are similar and where they are different.
Parsing user-supplied schema or grammar (shared) The first processing step reads a Creole-like schema or ixml-like grammar and parses it into a high-level schema model that represents the features of both types of input in a unified way. Both a Creole-like schema and an ixml-like grammar describe expectations for nodes of different types, character classes and ranges, and high-level operators (and (group), or (alternatives), oneOrMore (optional), zeroOrOne (optional) zeroOrMore (optional), interleave, concur, partition, etc.). ixml-like separators ** and ++ are rewritten directly as zeroOrMoreWithSeparator and oneOrMoreWithSeparator. The writing of operators at this stage is high-level, with the goal of translating reserved-character notation, which would be challenging for the eventual low-level parser, into keyword operators. The intelligent interpretation of those keyword operators happens later, as we describe below.
High-level schema or grammar model (shared) The output of the preceding stage is a high-level description of the types of documents that are part of the language, whether for recognition and validation (Creole-like) or for generation (ixml-like). This step abstracts away syntactic differences between Creole-like schemas and ixml-like grammars. The operators in the input schema or grammar become nodes in the tree of expectations (expression trees), alongside elements and attributes.
Schema or grammar compiler (shared) The preceding step is compiled down into a MCSG with a set of linear production rules (no operators and no node definitions). The model for this low-level grammar is implemented with optional features, which is to say that it’s a shared model, but which optional features are used depends on whether we are performing validation or adding markup. Specifically: For Creole-like validation we know to expect markup tokens (e.g., start- and end-tokens), while for ixml-like processing there is no incoming markup. For ixml-like prodessing rules have properties about whether they should result in serialization as elements, attributes, or patterns. Patterns (a type of node in the tree) can include nonterminals (other productions), string literals, character classes, character ranges, or numeric character references (e.g., #a), and epsilon. These all express expectations for the input. When input matches a pattern, the pattern is added to the parse forest and typed as a named pattern (which tells the eventual serializer to ignore it) and the input is added to parse forest as children of the named-pattern node.. For Creole-like processing there is no output serialization.
Tokenizer for tagged input (schema-validation only) The tokenizer divides the input stream into markup tokens and text tokens, based on built-in tokenizers for (currently) LMNL (MNML version) and TexMECS. for schema validation we have prioritized next-generation markup, rather than XML, but plan also eventually to include a tokenizer for XML input. Input for ixml-like processing does not have markup tokens, and therefore does not use the tokenizer module. As we note above, text with explicit markup can serve as input to ixml-like processing, but the input is ingested as a character stream, and any recognition of existing markup is part of the user-supplied grammar, rather than a tokenizer supplied by the tool.
Parser to parse input file against compiled schema or grammar (shared) The input is a character stream for ixml-like processing and the result of the tokenization step (above) for schema validation. Both use a MCSG and produce: A GSS (graph-structured stack, also called a multistack) of expectations that stores information about concurrency, interleave, and, just for the generator, ambiguity. A parse forest. Currently this is used only for ixml-like processing, but it might be used for Creole-like schema validation in the future to enhance error messaging based on information available when invalidity or other error is detected. The validator ends here.
Serializer (ixml-like processing only) The serializer serializes and emits one or more paths through the parse forest, with next-generation features. We currently have serializers for LMNL, TexMECS, and XML. Currently our XML serialization works only in the absence of concurrency. While XML does not support overlapping markup, an XML serializer might be able to create well-formed XML markup according to one of the work-around strategies currently available, such as creating flat output, where the children of the root node are a single sequence of text nodes and empty elements, with attributes on those elements to indicate how they mark the starts and ende of possibly overlapping elements.
Discussion As this section has shown, despite the patent differences in the XML ecosystem between the roles of schema validation, one the one hand, and the addition of explicit markup, on the other, the tasks have much in common. Those similarities create an opportunity to develop tools that rely on a shared data model and a largely shared code base. There is, at the same time, good reason that the XML toolkit does not include validation and markup-generation tools that employ a shared data model and code base. These include: Validation and markup generation are very different tasks. Those genuine and profound differences may have made it difficult to ask what the two have in common and whether they might be approached in a coordinated way. The emergence of RELAX NG and the robust and highly performant Jing implementation thereof makes XML schema validation largely a solved problem. ixml is a much younger technology, but it has already acquired several robust and highly performant implementations, and while the language continues to evolve, tooling to support it can also reasonably be regarded as a settled problem—or, at least, a problem that is well on course toward being settled. As a result, there is little reason to deviate from established approaches to RELAX NG and to ixml; both are well served by the available approaches and the available implementations of those approaches. The same cannot be said for next-generation markup processing, which is very much an emerging and not-yet-settled field. Validation and markup generation for next-generation languages are necessarily new, which encourages new approaches in a way that does not obtain, at least to the same extent, in the case of the far more mature XML technologies. In the XML ecosystem, ixml is context free but RELAX NG is not, which creates different, not entirely compatible contexts for the modeling and the coding to support it. Our decision to approach both tasks with a MCSG removed this fundamental difference and made it easier to see what the specific MCSG implementations might have in common.
Conclusion The impetus for this paper was our interest in exploring what ixml-like processing might look like for next-generation markup languages. We approached this question by starting from the MCSG concept that informed and applying it to next-generation ixml-like processing. Our observations during the course of these explorations included the following: Our work with Hamlet took us through identifying overlap, creating independent ixml grammars for the two hierarchies, exploring the consequences of attempting to merge those hierarchies, and creating an ixml grammar that was capable of recognizing and serializing LMNL markup of the overlap. These activities encouraged us to recognize the value of modeling overlap as concurrency, rather than as ambiguity, and of the ability of processing with a MCSG to represent concurrency in a more robust and scalable way than ixml. While we understood from the beginning that there was likely to be reusable code as we moved from validation to markup generation, the extent of the opportunities for reuse was unexpected, and led us to regard the potential for shared modeling and development as an additional goal of our explorations Potential goals for further work include: Although we were not able to complete experiments with ixml-like parsing that already contains next-generation markup, or with managing unordered content, we began theorizing about how our ongoing work might address those types of tasks. We anticipate undertaking experiments with these activities in a future report. We do not mention discontinuity in the body of our paper although it is a highly important next-generation document feature, and it is supported directly by markup in TexMECS. Managing discontinuity already present in markup during validation would be straightforward, but it is unclear to us, at this time, how we might usefully and reliably recognize implicit discontinuity for markup generation. The shared metrical line in our Hamlet experiment might be regarded as discontinuously separated by the intervening speaker, and this particular discontinuity could be recognized algorithmically, but further theorization is required before we will be prepared to suggest a more general and generalizable implementation strategy.
Works cited Tovey-Walsh, Bethan. Document on ambiguity. 2026-02-04. https://lists.w3.org/Archives/Public/public-ixml/2026Feb/0040.html Brüggemann-Klein, Anne. Four basic building principles (patterns) for XML schemas. Presented at Balisage: The Markup Conference 2020, Washington, DC, July 27 - 31, 2020. In Proceedings of Balisage: The Markup Conference 2020. Balisage Series on Markup Technologies, vol. 25 (2020). doi:10.4242/BalisageVol25.Bruggemann-Klein01 Clark, James. An algorithm for RELAX NG validation. 2002. https://relaxng.org/jclark/derivative.html CoffeePot An Invisible XML processor. Version 3.3.11 https://docs.nineml.org/current/coffeepot/ Accessed 2026-07-04. Haentjens Dekker, Ronald, and David J. Birnbaum. It's more than just overlap: Text As Graph. Presented at Balisage: The Markup Conference 2017, Washington, DC, August 1 - 4, 2017. In Proceedings of Balisage: The Markup Conference 2017. Balisage Series on Markup Technologies, vol. 19 (2017). doi:10.4242/BalisageVol19.Dekker01 Haentjens Dekker, Ronald, Elli Bleeker, Bram Buitendijk, Astrid Kulsdom and David J. Birnbaum. TAGML: A markup language of many dimensions. Presented at Balisage: The Markup Conference 2018, Washington, DC, July 31 - August 3, 2018. In Proceedings of Balisage: The Markup Conference 2018. Balisage Series on Markup Technologies, vol. 21 (2018). doi:10.4242/BalisageVol21.HaentjensDekker01 Haentjens Dekker, Ronald, David J. Birnbaum, Bram Buitendijk and Joris J. van Zundert. A schema language and parser for next-generation markup languages. Presented at Balisage: The Markup Conference 2025, Washington, DC, August 4 - 8, 2025. In Proceedings of Balisage: The Markup Conference 2025. Balisage Series on Markup Technologies, vol. 30 (2025). doi:10.4242/BalisageVol30.HaentjensDekker01 Huitfeldt, Claus, and C. M. Sperberg-McQueen. TexMECS: An experimental markup meta-language for complex documents. Working paper of the project Markup Languages for Complex Documents (MLCD), University of Bergen. January 2001, rev. October 2003. http://mlcd.blackmesatech.com/mlcd/2003/Papers/texmecs.html Invisible XML Specification. Editor: Steven Pemberton, CWI, Amsterdam Version: 2022-06-20 Jinks, Pete. Notations for context-free grammars. https://www.cs.man.ac.uk/~pjj/bnf/bnf.html Clark, James. Jing. A RELAX NG validator in Java. 2001–08. https://relaxng.org/jclark/jing.html Tovey-Walsh, Norm. Ambiguity in iXML: and how to control it. Presented at Balisage: The Markup Conference 2023, Washington, DC, July 31–August 4, 2023. In Proceedings of Balisage: The Markup Conference 2023. Balisage Series on Markup Technologies, vol. 28 (2023). doi:10.4242/BalisageVol28.Tovey-Walsh01 Tovey-Walsh, Norm. Writing Invisible XML Grammars. December 16, 2025. https://www.xml.com/articles/2025/12/16/invisible-xml-grammars-update/ Pemberton, Steven. Invisible XML case studies. Presented at XML Prague, 2026-06-06. https://homepages.cwi.nl/~steven/Talks/2026/06-06-prague-tutorial/ Piez, Wendell. Scholia 2026: DIY study aids with TEI, XProc, browser and printer. Presented at Balisage: The Markup Conference 2026, Washington, DC, August 3–7, 2026. In Proceedings of Balisage: The Markup Conference 2026. Balisage Series on Markup Technologies, vol. 31 (2026). doi:10.4242/BalisageVol31.Piez01 Piez, Wendell. Overlap meets Invisible XML. Presented at The First International Symposium on Invisible XML, 26/27 February 2026. https://github.com/wendellpiez/Laminator/tree/main/papers/iXMLSymposium2026 Sperberg-McQueen, C. M., and Claus Huitfeldt. 2000. GODDAG: a data structure for overlapping hierarchies. Paper given at Digital Documents: Systems and Principles. 8th International Conference on Digital Documents and Electronic Publishing, DDEP 2000, 5th International Workshop on the Principles of Digital Document Processing, PODDP 2000, Munich, Germany, September 13–15, 2000. Published in DDEP-PODDP 2000, ed. P. King and E.V. Munson. Lecture Notes in Computer Science 2023. Berlin: Springer, 2004, pp. 139–60. doi:10.1007/978-3-540-39916-2_12. Available on the Web at http://www.w3.org/People/cmsmcq/2000/poddp2000.html Sperberg-McQueen, C. M., and Claus Huitfeldt. Containment and dominance in Goddag structures. Processing Text-Technological Resources, Bielefeld, March 2008. https://cmsmcq.com/2008/bielefeld/slides.html Tennison, Jeni. Creole: Validating Overlapping Markup. Presented at XTech 2007. https://www.princexml.com/howcome/2007/xtech/papers/output/0077-30.pdf Tennison, Jeni and Wendell Piez. The Layered Markup and Annotation Language (LMNL). Presented at Extreme Markup Languages® 2002 (Montréal, Canada). 2002. Verwer, Nico. Between invisible XML and visible XML. Presented at The First International Symposium on Invisible XML, 26/27 February 2026. https://invisiblexml.org/events/symposium2026/slides/Between_Invisible_XML_and_Visible_XML_-_Nico_Verwer.pdf