Balisage Paper: Bunkankun: Digital Text as distributed layers

What about TEI?

TEI/XML is of course perfectly capable of representing Chinese text and has been used for this for more than 30 years. Probably the largest, at least the largest publicly available, collection is the CBETA 電子佛典集成 (CBETA Collection of Buddhist Scriptures^[5]), but there might be others. In another project I have spend many years developing a platform for collaborative research on such texts, proving that it is indeed possible.

However, over the years I also encountered problems, not so much with the representation itself, but with the overall system design with the affordances provided by this structure.

One of the aims of the collaborative platform was to provide full support for the research workflow for a text, which starts from establishing the exact characters of the text, including comparison of variant editions or parallel occurrences while referencing a digital facsimile of the text. The next step then is identifying parts of a texts, such as front matter, body or back matter, as well as internal structure, as far as it is visible^[6]. This is followed by adding sections, paragraphs and punctuation to the text. At this point the text is ready to be annotated and translated, which are the main tasks for the project. It might be blamed on the way TEI was applied to this project, but this workflow essentially required to prepare the texts outside the collaborative platform and upload them once they reached a satisfying state, thus resulting in a waterfall workflow rather than a circular one^[7].

So in a way of speaking, the text format introduced here might not so much be seen as replacing TEI, but rather providing a scaffolding for the production of a text that can then be interchanged with others using TEI.

So then, what exactly is a text?

One of the main tasks scheduled for our project this year was the integration of a large additional trove of texts into the existing platform, which would result in a tenfold expansion of the number of covered texts. The main catch was that the new corpus is unstructured and raw, and could thus not easily be added to the existing system. In a number of research meetings, we ended up questioning some of the fundamental assumptions about what a text is, and how it should be treated. Coming from the raw representation of a text in the scrolls of the first millenium, everything that was added later could be seen as an interpretative addition to a text, not an essential part. So we started to see the text not as a fixed publication, but rather as a composite artefact that could be made up from various ingredients at the convenience of the producer (or even of the reader).

Right, but how do we tell the computer?

This encouraged me to look for a solution that would not rely on TEI. It still needed to be fit for markup statements, but rather standoff than inline. One experiment tried to use JSON for text representation, but that is both awful to edit and to manage in a git based environment. YAML then seemed a better fit, and so the current implementation and data format took shape. It is still in a somewhat experimental state, but with help from today's coding agents, it could be quickly turned into a viable implementation and tested on the whole corpus. However, there are still some components completely missing.

Archival format

All text fields, no matter their location or type, are accompanied by a hash field whose value audits the content. This will allow the standoff references to the content by offset to be pinned to discoverable versions and allow detection of drift.

A juan file has a front, a body, and a back. Only the body must be non-empty; the others are optional and need not be present if empty. Additional metadata fields are available. A typical front contains an opening line that locates the juan in a larger collection, the title of the text, the sequential number of the juan, and an attribution naming persons and roles with respect to the body. The back contains a closing line. The placement of prefaces, postfaces, colophons, and similar paratextual material is open: such material may go into the body or be separated out into front or back, at the discretion of the project applying the format.

The body has one text element that holds the canonical character content of the whole juan. Space characters, punctuation, line breaks, and similar content are not present in this stream — they are extracted into the juan's logical marker collection. A marker has at minimum a type and an offset; further fields are optional and typically include id, content, and additional structured information appropriate to the marker's type. The set of marker types is open; a small core vocabulary is defined separately.

Markers may be stored in several physical locations. A bucket may carry an inline markers list for structural markers that are useful when inspecting the juan by itself. Most markers may instead be stored in a per-juan marker asset under the bundle's assets/ directory and pinned from the manifest. Consumers treat inline markers and marker-asset markers as one logical collection, sorted by offset. This keeps juan files compact while preserving the same text-addressing model.

One text can be represented in several editions. Additional editions in a format that parallels the main structure are storied in a subdirectory editions. In addition, they can be made accessible through the master edition of the text through markers that indicate at which offsets the text differs from the surface content.

Canonicalization

The markers link to their associated text files by raw offset. Since this is a potential fragile mechanism, extra care needs to be taken to ensure the integrity of the text fields. As has been mentioned, a hash is used to make changes to the text discoverable. For Chinese text, there are additional steps required to make this process robust. For this purpose, we define a set of canonical characters. These characters are drawn from Unicode, but due to the history of Unicode, there are many codepoints that essentially encode the same character. For this reason, we need to exclude some, while providing a mapping of the excluded characters to corresponding characters in the canonical set. On the other hand, there are rare characters which do not (yet?) have an assigned codepoint in Unicode. For these cases, we designate an area within the large block of PUA (private use area) characters for the purpose of refering to these characters within our texts, while maintaining external information as part of the text format, which provides ways to render the characters, similar to the g element in the TEI. This constitutes a reference character set, which is part of the file format together with other reference assets.

Source material is brought into this canonical form through a deterministic procedure; every step that alters the source is recorded as a marker, so that the source can be reconstructed exactly from the canonical text and its markers.

The procedure is applied in order:

Offsets count Unicode codepoints into the post-substitution text stream. Variation selectors and similar combining characters are not independent offset targets; they remain attached to the preceding base character.

Characters that fall outside the declared set are not forbidden in a source — they are replaced during canonicalization and the replacement is recorded as a substitution marker. Common reasons for exclusion include compatibility characters with a documented preferred form, deprecated codepoints, ad-hoc Private Use Area assignments from third-party encoding schemes, and blocks deliberately not yet supported.

Markers

Markers are the mechanism by which information that has been removed from the text stream during canonicalization, or that annotates the text stream from outside it, is reattached to specific positions in the canonical text.

The logical marker collection for a bucket is assembled from the bucket's inline markers list, if present, plus the matching list in the juan's marker asset, if the manifest declares one. Inline markers are therefore a storage convenience, not a separate semantic class. When both locations contain markers, consumers merge them by offset and preserve source order for markers with the same offset.

Markers come in two shapes. Point markers apply at a single position; they carry an offset alone. Range markers apply to a slice of the text stream; they carry an additional length field, and apply to the slice [offset, offset + length). The shape is fixed by the marker type. The length field mirrors the codepoint-slice form also used in the recipe-selection vocabulary.

Every marker carries the following required fields:

Optional fields are defined per marker type and may include id, content, note, responds-to (an id referring to another marker, used by markers that comment on or extend another marker), and structured fields specific to the type. A marker carrying an id may be referenced from elsewhere; ids are local to the juan unless qualified. Id uniqueness is checked over the assembled logical marker collection, not merely over one physical file.

The set of marker types is open. A small core vocabulary is defined for layout, structural, voicing, and substitution markers: page break, line break, indent, punctuation, paragraph break, register divider, head, comment, and substitution are point-typed; voice (§"Voices") is range-typed. Projects may extend the set for their own purposes; a marker type that is not in the core vocabulary should be namespaced by the project introducing it.

XML-based imports may emit a generic point marker for source elements whose presence should be preserved but whose semantics are not otherwise modeled by the core importer. This marker has type xml-element and carries the required fields offset, name, and role. The name field records the source element name, using the same prefix form used elsewhere in source metadata, and role is normally open or close. Additional fields may be present, for example, id or an attrs mapping of source attributes. Page and line breaks, tls:head, and tls:seg remain represented by their specialized markers and are not emitted as xml-element markers^[8].

By default, structural and navigation-critical markers remain inline: tls:head, tls:div-start, tls:div-end, cbeta:juan-start, cbeta:juan-end, cbeta:mulu, and any marker directly referenced by the table of contents. Layout markers, punctuation, page and line breaks, voice markers, variant markers, substitution markers, comments, and other bulky project markers normally live in the marker asset. This default is a packaging policy rather than a semantic restriction.

Voice markers

A voice marker records that a slice of the canonical text stream belongs to a named textual layer. It is the mechanism by which interleaved threads — a root text and its commentaries, for example — are disentangled so that a consumer can extract any one layer, present several in parallel, or reproduce the interleaved source as published.

A voice marker is range-typed. It carries the required marker fields together with:

The name vocabulary is open. A small recognized set is defined so that cross-project work stays legible:

Projects extend by prefixing — kr:apparatus, tls:swl, and so on — consistent with the rest of the marker vocabulary.

A voice marker may carry optional fields including id, note, and responds-to — the id of another voice marker that this voice answers to. A subcommentary voice typically responds-to a commentary voice; a commentary voice responds-to a root voice. The relationship is many-to-one: a single root span may attract several commentary spans, each of which names that root span as its responds-to target.

Voices coexist with the single text stream of the juan; there is no separate per-voice text element. The juan's canonical text contains every voice's content in source order, and voice markers carve it into named slices. A consumer extracting only the root filters by name=root and reads the slices in offset order; one reproducing the interleaved form walks all voice markers in offset order. Both presentations are derivations of the same canonical text and guarded by the same hash.

The lighter-weight point markers comment and head (which carry out-of-stream content) remain available for sparse interlinear material — short editor's notes, occasional headings — where promoting the inserted text into a first-class slice of the text stream would be overkill.

Example. A short commentary passage alternates root lines and commentary on those lines. The canonical text of the juan, rendered as a single stream, reads:

色不異空謂色相虛幻不離空性空不異色謂空性遍周不離色相

The juan carries four voice markers:

                     
markers:
  - {type: voice, offset: 0,  length: 4, name: root,       id: r1}
  - {type: voice, offset: 4,  length: 9, name: commentary, responds-to: r1}
  - {type: voice, offset: 13, length: 4, name: root,       id: r2}
  - {type: voice, offset: 17, length: 9, name: commentary, responds-to: r2}

A consumer extracting name=root walks the marker list in offset order and concatenates the slices [0,4) and [13,17), yielding 色不異空空不異色. A consumer reproducing the interleaved source walks every voice marker in offset order and emits the slices in sequence, producing the original stream. Both presentations are derivations of the same canonical text and the same hash; neither presentation requires altering the underlying juan.

Overlay bundles

An overlay bundle is a bundle that contributes additional markers to another bundle's text stream without modifying it. Its manifest declares a target — the canonical identifier and hash of a target bundle — and its juan files carry marker collections that address offsets in the corresponding juans of the target. The target is unaffected: its hash does not change, and existing pins to it remain valid. The overlay is independently addressable, hashed, and versioned per the usual bundle pattern.

The motivation is straightforward. Useful markers can be authored after a bundle has been published, by parties who do not own the target, for purposes its original author may not have considered: segmentation introduced for the sake of a translation, modern punctuation added to a classical source, voice markers added to disentangle a layout the source did not mark machine-readably, scholarly annotation distributed independently. A consumer composes the target with whichever overlays they want via a recipe.

These distributed layers are what makes this system adaptable to a variety of needs and use cases. A research paper could, for example, keep the passages it quotes as an overlay and could then use a recipe to pull the source and an accompanying translation from the corpus, in a similar way to the handling of bibliographies. Readers could then use such an overlay to inspect the quotations in their original context, comment or add their own translations.