Introduction

As a Sinologist, my main interest for the past 35 years lay in the digital representation of premodern Chinese texts for use in research environments. Encountering the TEI and SGML at that time was a life changing experience and has shaped my activities for many years; one of the outcomes is a large corpus of Buddhist texts[1]. Among colleagues the uptake was mixed, and looking for more approachable ways to encode text, I embraced an approach of making the markup as implicit as possible with a simple data format derived from Emacs org-mode, called the Kanseki Repository[2]. This went on for a dozen years, then a different project brought me back to TEI – by this time I had given up the idea of having researchers edit the source directly and concentrated on the creation of a collaborative platform, hiding the technical complexity and angled brackets[3]. Ten years into this project, out of an attempt to bring the Kanseki Repository and the TLS on the same page, I happened on the idea to stop trying to contain all features of a text in one container and instead embracing an approach which bears some similarity to Docker images: Composable layers of a text as separate components, drawn together according to a recipe (think Docker file) by a composer that performs the steps outlined in the recipe and produces the fleeting shape of a text according to the wishes of the reader: with or without annotations, translations or facsimile images, and in the format needed at the moment: PDF for printing, HTML for the web, ePub for a ebook reader.

As always, getting the details right is where the rubber hits the road. JSON is the native language of most modern communication, but not really usable as an archival file format; YAML fits this bill better. A few weeks of experimenting resulted in a shape of data that is both fit for the purpose and has interesting properties inherited from thousands of years of writing Chinese: Texts written on scrolls, kept in bundles with minimal metadata attached. So in this presentation, I would like to introduce this text format, which is specialized for this specific use of premodern Chinese texts, and probably not for much else, but might still contain some ideas that inspire imitation. A first realization of an interface is now online[4], and a first version of the texts is published.

A short look at the shape of Chinese texts

Before we dive in, I will give a very short introduction to premodern Chinese texts. The earliest surfaces used for writing in China have been bones of cattle or turtle plastrons, followed by bronze vessels. These could only be used for short inscriptions, so bamboo strips and silk came also to be used. Before woodblock printing started in the 9th century (see Fig 1), texts have been written on paper since the Eastern Han period (25 - 220 AD). To make handling of longer texts convenient, sheets of paper were glued together to form a long surface of several meters (ready to hold several thousand characters), these were rolled for storage and put in boxes. For convenience, the outside usually had the title and sequential number and the beginning would start with some metadata for housekeeping, including a table of contents if there were several shorter texts on a scroll.

Figure 1: Woodblock print of the Diamond Sutra, ca 868

The beginning section of a woodblock print, discovered in Central Asia, now held in the British Library, shelfmark Or. 8210.

Paper made writing much more convenient and marked the beginning of the production of larger amounts of texts; some conventions that arose around this time carried over to the period of woodblock printing, when even larger numbers of texts could be produced conveniently. Sheets of paper would still be glued together, but only after the printing, so now the sheets themselves also needed some metadata, which included in addition to a short title and a scroll number, also the sheet number and in many cases the number of characters on the sheet and the name of the carver – the latter was necessary to calculate the pay.

So a scroll, juan in Chinese, continued to be a basic unit for the production of texts. Since the length could be adjusted to a certain degree, the juan came also to resemble not just a physical, but also a logical unit of a text and it became customary to refer to specific locations in a text by title and scroll number. Within a scroll though, there was very little structure, apart from very few paragraph indications – starting a new line before the space was used up – the text would flow without interruption, hundreds and hundreds of characters.

There are, of course, texts that have more internal structure, for example, with a commentary interlaced with the main text. In this case, the commentary might be in smaller characters occupying a quarter of the square a normal character would take up, thus slicing the text up in (at least) two separate logical streams within a text. And for texts that had been commented on several times over the centuries (the way to make a new statement in Chinese thinking was to make it a comment to a classic, thus making it look like it was not new at all), sometimes these commentaries would be collected, and such a text could have six or more additional voices layered on the source text. As an example for a woodblock print, Fig 2 shows a text with commentary.

Figure 2: History of the later Han, first part of first juan

Beginning of the main text, with the frontmatter marked in yellow – this includes the title and name of the author, as well of the main commentator. The first heading is marked in orange, while the root text is in blue, and the commentary is in black. I preserved some of the text unmarked to show the original shape.

Now this handling is not yet implemented for the new data format, but Fig. 3 shows how it looks like in the upstream source, displayed in the web interface.

Figure 3: The same location as Fig 2, but a different source

This is a screenshot of a display of this same text location in the Kanseki Repository, with the smaller print text rendered as pink to make the distinction more obvious. The right hand side shows the optionally activated digital facsimile of the edition used here.

Digital representation and system design

What about TEI?

TEI/XML is of course perfectly capable of representing Chinese text and has been used for this for more than 30 years. Probably the largest, at least the largest publicly available, collection is the CBETA 電子佛典集成 (CBETA Collection of Buddhist Scriptures[5]), but there might be others. In another project I have spend many years developing a platform for collaborative research on such texts, proving that it is indeed possible.

However, over the years I also encountered problems, not so much with the representation itself, but with the overall system design with the affordances provided by this structure.

One of the aims of the collaborative platform was to provide full support for the research workflow for a text, which starts from establishing the exact characters of the text, including comparison of variant editions or parallel occurrences while referencing a digital facsimile of the text. The next step then is identifying parts of a texts, such as front matter, body or back matter, as well as internal structure, as far as it is visible[6]. This is followed by adding sections, paragraphs and punctuation to the text. At this point the text is ready to be annotated and translated, which are the main tasks for the project. It might be blamed on the way TEI was applied to this project, but this workflow essentially required to prepare the texts outside the collaborative platform and upload them once they reached a satisfying state, thus resulting in a waterfall workflow rather than a circular one[7].

So in a way of speaking, the text format introduced here might not so much be seen as replacing TEI, but rather providing a scaffolding for the production of a text that can then be interchanged with others using TEI.

So then, what exactly is a text?

One of the main tasks scheduled for our project this year was the integration of a large additional trove of texts into the existing platform, which would result in a tenfold expansion of the number of covered texts. The main catch was that the new corpus is unstructured and raw, and could thus not easily be added to the existing system. In a number of research meetings, we ended up questioning some of the fundamental assumptions about what a text is, and how it should be treated. Coming from the raw representation of a text in the scrolls of the first millenium, everything that was added later could be seen as an interpretative addition to a text, not an essential part. So we started to see the text not as a fixed publication, but rather as a composite artefact that could be made up from various ingredients at the convenience of the producer (or even of the reader).

Right, but how do we tell the computer?

This encouraged me to look for a solution that would not rely on TEI. It still needed to be fit for markup statements, but rather standoff than inline. One experiment tried to use JSON for text representation, but that is both awful to edit and to manage in a git based environment. YAML then seemed a better fit, and so the current implementation and data format took shape. It is still in a somewhat experimental state, but with help from today's coding agents, it could be quickly turned into a viable implementation and tested on the whole corpus. However, there are still some components completely missing.

Bunkankun format

The building housing my office in Kyoto is the branch office of our Institute, Bunkan in Japanese, so I nicknamed my coding agent Bunkankun =~ the Bunkan guy. Somehow the name stuck; a domain could be registered, and no clashes with other names were discovered by a few casual searches on the Internet and on GitHub (bkk was already taken, unfortunately), so that became the name for this endeavor.

The texts we are dealing with vary considerably in length, from a few hundred characters to many millions, with the largest text running to 1000 juan in the order of 20000 characters. For convenience, this is used as a handy unit of division for the texts, so each juan will reside in its own YAML file. These are then collected in text bundles, with a manifest file declaring the assets in a bundle and adding some administrative metadata. There will be at least a surface edition – the curated text we establish – and possibly an unlimited number of additional editions, residing in a similar way under an edition subdirectory.

Archival format

All text fields, no matter their location or type, are accompanied by a hash field whose value audits the content. This will allow the standoff references to the content by offset to be pinned to discoverable versions and allow detection of drift.

A juan file has a front, a body, and a back. Only the body must be non-empty; the others are optional and need not be present if empty. Additional metadata fields are available. A typical front contains an opening line that locates the juan in a larger collection, the title of the text, the sequential number of the juan, and an attribution naming persons and roles with respect to the body. The back contains a closing line. The placement of prefaces, postfaces, colophons, and similar paratextual material is open: such material may go into the body or be separated out into front or back, at the discretion of the project applying the format.

The body has one text element that holds the canonical character content of the whole juan. Space characters, punctuation, line breaks, and similar content are not present in this stream — they are extracted into the juan's logical marker collection. A marker has at minimum a type and an offset; further fields are optional and typically include id, content, and additional structured information appropriate to the marker's type. The set of marker types is open; a small core vocabulary is defined separately.

Markers may be stored in several physical locations. A bucket may carry an inline markers list for structural markers that are useful when inspecting the juan by itself. Most markers may instead be stored in a per-juan marker asset under the bundle's assets/ directory and pinned from the manifest. Consumers treat inline markers and marker-asset markers as one logical collection, sorted by offset. This keeps juan files compact while preserving the same text-addressing model.

One text can be represented in several editions. Additional editions in a format that parallels the main structure are storied in a subdirectory editions. In addition, they can be made accessible through the master edition of the text through markers that indicate at which offsets the text differs from the surface content.

Canonicalization

The markers link to their associated text files by raw offset. Since this is a potential fragile mechanism, extra care needs to be taken to ensure the integrity of the text fields. As has been mentioned, a hash is used to make changes to the text discoverable. For Chinese text, there are additional steps required to make this process robust. For this purpose, we define a set of canonical characters. These characters are drawn from Unicode, but due to the history of Unicode, there are many codepoints that essentially encode the same character. For this reason, we need to exclude some, while providing a mapping of the excluded characters to corresponding characters in the canonical set. On the other hand, there are rare characters which do not (yet?) have an assigned codepoint in Unicode. For these cases, we designate an area within the large block of PUA (private use area) characters for the purpose of refering to these characters within our texts, while maintaining external information as part of the text format, which provides ways to render the characters, similar to the g element in the TEI. This constitutes a reference character set, which is part of the file format together with other reference assets.

Source material is brought into this canonical form through a deterministic procedure; every step that alters the source is recorded as a marker, so that the source can be reconstructed exactly from the canonical text and its markers.

The procedure is applied in order:

  1. Source. Input is treated as Unicode encoded in UTF-8.

  2. Entity expansion. Entity references in the source — for example, &KRxxxx; style references in Kanripo material, or TEI <g/> elements — that point to characters not available in current Unicode are expanded to codepoints in the Supplementary Private Use Area (SPUA-A, U+F0000–U+FFFFD) using the bundle's declared entity encoding. The expansion is deterministic and produces no markers: the PUA codepoint, taken together with the declared encoding, is sufficient to recover the entity reference. The PUA codepoints produced this way are first-class members of the canonical text stream and participate in offsets, the text hash, and downstream marker logic like any other character.

  3. Unicode normalization. NFC is applied so that base characters are in precomposed form.

  4. Extraction of layout features. Characters carrying layout, structural, or paratextual information — whitespace, punctuation, indent characters, page and line breaks, register dividers, and similar — are removed from the text element and emitted as markers, each with an offset into the remaining text stream.

  5. Substitution to canonical characters. Any character in the post-extraction stream that is not a member of the canonical character set is replaced by its canonical equivalent. Each such substitution is emitted as a marker recording the offset, the character or sequence replaced, and the reason for the substitution.

  6. Hash. The resulting text stream, encoded as UTF-8, is the input to the hash field that accompanies the text element.

Offsets count Unicode codepoints into the post-substitution text stream. Variation selectors and similar combining characters are not independent offset targets; they remain attached to the preceding base character.

Characters that fall outside the declared set are not forbidden in a source — they are replaced during canonicalization and the replacement is recorded as a substitution marker. Common reasons for exclusion include compatibility characters with a documented preferred form, deprecated codepoints, ad-hoc Private Use Area assignments from third-party encoding schemes, and blocks deliberately not yet supported.

Markers

Markers are the mechanism by which information that has been removed from the text stream during canonicalization, or that annotates the text stream from outside it, is reattached to specific positions in the canonical text.

The logical marker collection for a bucket is assembled from the bucket's inline markers list, if present, plus the matching list in the juan's marker asset, if the manifest declares one. Inline markers are therefore a storage convenience, not a separate semantic class. When both locations contain markers, consumers merge them by offset and preserve source order for markers with the same offset.

Markers come in two shapes. Point markers apply at a single position; they carry an offset alone. Range markers apply to a slice of the text stream; they carry an additional length field, and apply to the slice [offset, offset + length). The shape is fixed by the marker type. The length field mirrors the codepoint-slice form also used in the recipe-selection vocabulary.

Every marker carries the following required fields:

  • type: the kind of marker.

  • offset: the codepoint position in the canonical text stream where the marker begins.

  • length: the codepoint span of the marker. Present on range-typed markers, absent on point-typed.

Optional fields are defined per marker type and may include id, content, note, responds-to (an id referring to another marker, used by markers that comment on or extend another marker), and structured fields specific to the type. A marker carrying an id may be referenced from elsewhere; ids are local to the juan unless qualified. Id uniqueness is checked over the assembled logical marker collection, not merely over one physical file.

The set of marker types is open. A small core vocabulary is defined for layout, structural, voicing, and substitution markers: page break, line break, indent, punctuation, paragraph break, register divider, head, comment, and substitution are point-typed; voice (§"Voices") is range-typed. Projects may extend the set for their own purposes; a marker type that is not in the core vocabulary should be namespaced by the project introducing it.

XML-based imports may emit a generic point marker for source elements whose presence should be preserved but whose semantics are not otherwise modeled by the core importer. This marker has type xml-element and carries the required fields offset, name, and role. The name field records the source element name, using the same prefix form used elsewhere in source metadata, and role is normally open or close. Additional fields may be present, for example, id or an attrs mapping of source attributes. Page and line breaks, tls:head, and tls:seg remain represented by their specialized markers and are not emitted as xml-element markers[8].

By default, structural and navigation-critical markers remain inline: tls:head, tls:div-start, tls:div-end, cbeta:juan-start, cbeta:juan-end, cbeta:mulu, and any marker directly referenced by the table of contents. Layout markers, punctuation, page and line breaks, voice markers, variant markers, substitution markers, comments, and other bulky project markers normally live in the marker asset. This default is a packaging policy rather than a semantic restriction.

Voice markers

A voice marker records that a slice of the canonical text stream belongs to a named textual layer. It is the mechanism by which interleaved threads — a root text and its commentaries, for example — are disentangled so that a consumer can extract any one layer, present several in parallel, or reproduce the interleaved source as published.

A voice marker is range-typed. It carries the required marker fields together with:

  • type: the literal value voice.

  • length: the slice length, in codepoints.

  • name: the textual layer this slice belongs to.

The name vocabulary is open. A small recognized set is defined so that cross-project work stays legible:

  • root — the primary, commented-upon text.

  • commentary — a layer of comment on a root text.

  • subcommentary — a layer of comment on a commentary.

  • gloss — a short interlinear annotation (typical layout: between the lines, in smaller characters).

  • note — a bracketed span whose specific role (commentary, gloss, alternate reading, citation, …) is left unresolved. Used when the only available signal is a punctuation fence such as (…=)= and the deriver can't claim anything stronger.

Projects extend by prefixing — kr:apparatus, tls:swl, and so on — consistent with the rest of the marker vocabulary.

A voice marker may carry optional fields including id, note, and responds-to — the id of another voice marker that this voice answers to. A subcommentary voice typically responds-to a commentary voice; a commentary voice responds-to a root voice. The relationship is many-to-one: a single root span may attract several commentary spans, each of which names that root span as its responds-to target.

Voices coexist with the single text stream of the juan; there is no separate per-voice text element. The juan's canonical text contains every voice's content in source order, and voice markers carve it into named slices. A consumer extracting only the root filters by name=root and reads the slices in offset order; one reproducing the interleaved form walks all voice markers in offset order. Both presentations are derivations of the same canonical text and guarded by the same hash.

The lighter-weight point markers comment and head (which carry out-of-stream content) remain available for sparse interlinear material — short editor's notes, occasional headings — where promoting the inserted text into a first-class slice of the text stream would be overkill.

Example. A short commentary passage alternates root lines and commentary on those lines. The canonical text of the juan, rendered as a single stream, reads: 

色不異空謂色相虛幻不離空性空不異色謂空性遍周不離色相

The juan carries four voice markers: 

                     
markers:
  - {type: voice, offset: 0,  length: 4, name: root,       id: r1}
  - {type: voice, offset: 4,  length: 9, name: commentary, responds-to: r1}
  - {type: voice, offset: 13, length: 4, name: root,       id: r2}
  - {type: voice, offset: 17, length: 9, name: commentary, responds-to: r2}

A consumer extracting name=root walks the marker list in offset order and concatenates the slices [0,4) and [13,17), yielding 色不異空空不異色. A consumer reproducing the interleaved source walks every voice marker in offset order and emits the slices in sequence, producing the original stream. Both presentations are derivations of the same canonical text and the same hash; neither presentation requires altering the underlying juan.

Overlay bundles

An overlay bundle is a bundle that contributes additional markers to another bundle's text stream without modifying it. Its manifest declares a target — the canonical identifier and hash of a target bundle — and its juan files carry marker collections that address offsets in the corresponding juans of the target. The target is unaffected: its hash does not change, and existing pins to it remain valid. The overlay is independently addressable, hashed, and versioned per the usual bundle pattern.

The motivation is straightforward. Useful markers can be authored after a bundle has been published, by parties who do not own the target, for purposes its original author may not have considered: segmentation introduced for the sake of a translation, modern punctuation added to a classical source, voice markers added to disentangle a layout the source did not mark machine-readably, scholarly annotation distributed independently. A consumer composes the target with whichever overlays they want via a recipe.

These distributed layers are what makes this system adaptable to a variety of needs and use cases. A research paper could, for example, keep the passages it quotes as an overlay and could then use a recipe to pull the source and an accompanying translation from the corpus, in a similar way to the handling of bibliographies. Readers could then use such an overlay to inspect the quotations in their original context, comment or add their own translations.

Conclusions

The text format introduced here is just a few weeks old and still in a very early phase of experimentation and implementation. After initial testing with a small subset of texts, a more thorough conversion of texts from three different sources is now live at the GitHub account bkkbooks, with almost 12500 titles. Code for a variety of maintenance tasks, including import, export, a web server and the frontend UI are all available at bunkankun/bkk.

The next step will involve adding proper segmentation to the digital text, this will also be added as markers. Text sections in our sources that start with a separate header line are easily recognized from analyzing the existing markers. Paragraphs in the modern sense are not existing and thus not indicated, but preliminary experiments show that large language models, with the right prompting and harness, can be pressed into service here. The same also holds for punctuation. And here is where this data format shines: Responses from different models can easily be overlaid, compared and further processed, without affecting the base text at all.

At the moment, this looks like a real opportunity to take care of the first few steps of the research workflow, the establishing of a well understood text, with the option of downstream paths to either the Kanseki Repository or the TEI based TLS format. Time will show how these three projects will evolve.


[2] A text repository based on this is live at https://kanripo.org, source at https://github.com/kanripo.

[3] The platform, called Thesaurus Linguae Sericae (TLS), also known as Hanxue Wendian (HXWD), is live at https://hxwd.org; the source is at https://github.com/tls-kr.

[4] To visit at https://bunkankun.org, supporting material and code at https://github.com/bunkankun; the texts are available at https://github.com/bkkbooks.

[5] Accessible in TEI P5 XML source at https://github.com/cbeta-org/xml-p5

[6] Ideally, these steps would be applied as needed and in a repetitive, circular manner, so that only parts of the text that were needed immediately had to be treated, rather than completing every step completely before moving on to the next step.

[7] At some point, user interface elements for most of the text critical work to establish a text were implemented, but they proved to be hard to use and brittle and saw little actual use.

[8] In the reference implementation bkk, a list of elements to be preserved in this way can be supplied.

Christian Wittern

Christian Wittern is Professor at the Center for Innovative Informatics of the Humanities, Institute for Research in Humanities, Kyoto University. He was Chair of the TEI Technical Council from 2003 to 2007 and chaired the TEI Character Encoding Workgroup.