Balisage Paper: Specifying a TEI-XML Based Format for Aligning Text to Image at Character Level

Palaeographic (Allographetic) Transcription

Transcriptions of this kind are also called imitative but we prefer the term of allographetic that corresponds to the linguistic status of letter variants in the framework of the theory of graphemics [Coulmas 1999]. ^[3] This kind of transcription represents very faithfully the aspect of the source document writing. It distinguishes major letter variants, like long and round s (ſ vs. s) or round and straight r ( vs. r), reproduces abbreviation markers, original punctuation and word segmentation. As a rule, allographetic transcriptions reproduce (or represent somehow) the layout of the original document (including line breaks, scribal correction markers, marginalia, etc.).

Allographetic transcriptions are rare in paper editions and can mostly be found in handbooks and albums for palaeographers (e.g. [Koschwitz 1879] and [Careri et al. 2001]). In electronic textual editing pure such transcriptions are also very rare: the original manuscript transcriptions of the Charrette Project are one the few examples that we could find. More often, some allographetic features can be found in multi-layer electronic transcriptions that will be discussed later.

This kind of transcription is the easiest to align with image zones. However, problems may arise in the process of word tokenization, as the white spaces that appear in manuscripts (and reproduced as they are in allographetic transcriptions) do not always correspond to word limits in terms of modern linguistic analysis. Another problem may concern abbreviation markers, some of which (but not all) are superscribed over the baseline characters. This has however more to do with the technique of image segmentation than with the transcription itself.

Diplomatic Transcriptions

Diplomatics is a scholarly discipline that studies the tradition, the form, and the production of written acts (especially historical documents) [VID 1997: 21]. In diplomatic editing, it is important to preserve as much information as possible from the source document without overcharging it with information not relevant for its interpretation (such as, typically, letter variants, or allographs). The practice of diplomatic editions has been far from being stable along its history and different national traditions. Some normalization efforts were undertaken by the Commission internationale de diplomatique (http://cidipl.org) but their recommendations [Bautier ed. 1984] are very general and allow considerable variation in local practices. It is nevertheless possible to point out some typical features of diplomatic transcriptions. For instance, the abbreviations are expanded but the letters supplied are typographically marked (using italics or some kind of braces). The original document layout is usually preserved. The punctuation and word segmentation may be normalized according to modern typographical rules. Some diplomatic editions also use character disambiguation (introducing u / v and i / j phonetic distinction and adding some diacritics that help reading the text) and capitalize the proper names. In some cases the editor may supply a missing word or part of word, or point out to a repeated word or phrase but these emendations are always clearly marked typographically. As an example of modern diplomatic edition we can quote Les plus anciens documents linguistiques de la France published online by Zurich University (http://www.rose.uzh.ch/docling).

Diplomatic transcriptions can in principle be aligned to image zones, provided that the abbreviation expansions, scribal corrections and editorial emendations are properly tagged. The punctuation marks may need to be ignored if the punctuation is modernized in the transcription. The neutralization of allographs, disambiguation of some characters and capitalization of proper names do not affect the alignment but need to be addressed in the markup model.

Normalized (Critical) Transcriptions

Transcriptions of this kind are the most widespread in the field of literary text editing. As far as the Old French texts are concerned, the first normalization recommendations (although the term was not used) were published by a commission of Société des Anciens Textes Français in 1926 [Roques 1926]. These recommendations were precise enough at some points (like using diacritics) but a number of questions (like word separation) were not mentioned at all. In early 2000s, École Nationale des Chartes published a series of recommendations for editing various kinds of medieval texts (Latin and vernacular, literary and documentary) [Vielliard ed. 2001, Guyotjeannin ed. 2001 and Bourgain ed. 2002] but they are not always followed in practice.

Without speaking of reconstructive (or Lachmannian^[4]) critical editions based on the comparison of multiple witnesses, which are by definition not alignable to an image of a single manuscript, even relatively faithful normalized transcriptions are hardly usable for the purposes of alignment. Whereas major editorial emendations to the source document are usually clearly marked (e.g. supplied words or letters are placed into square brackets), some information important for alignment is lost: the expansions of abbreviations are unmarked, as are the scribal corrections, and the manuscript page layout is rarely preserved (except, to a certain extent, the verse lines). However, thanks to digital markup, critical transcriptions may become more alignment-friendly. In this case, they should not be referred to as plain critical but rather as hybrid or multi-layer transcriptions.

Hybrid Digital Transcriptions

The possibility offered by computer technologies to have both abbreviation markers and expansions, corrected or regularized and original erroneous or irregular forms is one of the main arguments in favor of producing digital editions. The TEI provided special elements for these purposes since its very early versions. Initially, the alternative versions were encoded by means of attributes (e.g. the <expan>^[5] element had an abbr attribute), but the P5 version introduced a new <choice> element in order to group alternative encodings of the same segment of a text. This mechanism is much more flexible and powerful than the old attribute-based system but it may require more complex processing. TEI allows using more than two child elements inside <choice>, and <choice>s may nest. Some restrictions had to be added to the content model of <choice> for the alignment project, such as allowing only two child elements. Some more restrictions and requirements for the alignable transcription format will be described in section “Target Format”.

The actual transcriptions used in the project can all be qualified as hybrid but precise encoding practices differed considerably. Characteristic features of three of them are listed below.

<l n='1' id='MS-A-196-r-a-1' key='FU-31'>
  <chr_large size="8" color="" detail="">A</chr_large> un jor 
  dunea<chr_var letter="s" var="long">&#x222B;</chr_var>cen<chr_var 
    letter="s" var="long">&#x222B;</chr_var>i<chr_abbr type="" class="" 
    expan="" cert="no">[apost]</chr_abbr>o
</l>

<w type="NOMpro" xml:id="w106_000286">
  <choice>
    <me:norm>Lancelot</me:norm>
    <me:dipl>lanc<ex>elot</ex></me:dipl>
    <me:facs>lan<bfm:mdvAbbr>c&#x0305;</bfm:mdvAbbr></me:facs>
  </choice>
</w>

<lb n="14"/> Actum est <app>
  <lem>hoc</lem>
  <rdg wit="B" rend="omis"/>
</app> <placeName>Eduę</placeName> in plena ſynodo <date>
  <app>
    <lem>.XV.</lem>
    <rdg wit="B">XV°</rdg>
  </app> kl. julii</date>, laudanti<choice>
  <expan>bus</expan>  <abbr>b;</abbr>
</choice> archidiaconi<choice>
  <expan>bus</expan>
  <abbr>b;</abbr>
</choice>
<persName> Gaufrido</persName>,

Layout Markup and Linearity Conflicts

In order to associate a transcription of a page with an image file, a <pb/> must be provided at the beginning of each transcribed page. It must have an xml:id attribute for stand-off alignment and may have a facs attribute for direct linking to the corresponding image file or to the corresponding zone element in the facsimile section.

A <cb/> must be used in the beginning of each text column if there are more than one of them on a page.

A <lb/> must be used in the beginning of each text line. It must have an n attribute indicating the line number. In most cases this attribute may be generated automatically but sometimes the physical organization of text segments in lines does not correspond to the logical order of the text structure. This actually happens at the border of some text divisions where the title of the new division (often written with red ink, as a rubric) overlaps with the end of the previous division (See Figure 1).

Figure 1: Rubric Overlapping with Text Divisions

Example of rubric overlapping with a frontier of text divisions. Manuscript Paris, BnF, fr. 574, fol. 83r. Image obtained from the Gallica Virtual Library (http://gallica.bnf.fr)

In this case, the linearity of the transcription follows the logical order of the text, and two <lb/> elements with the same value of the n attribute are used in the rubric and in the text body, as in the example below:

<div>
   <p>[...]
    <lb n="4"/> est desus ia ne sera si fo<lb break="no" n="5"/>rt
    tour .</p>
</div>
<div>
  <head>comm<ex>en</ex>t la t<ex>er</ex>re
    <lb type="rubrique" rend="align(right)" n="6"/> crolle et fent.</head>
    <p>
      <lb n="6"/> <hi rend="lettrine">O</hi>re 
      <lb n="7"/> entendez donq<ex>ue</ex>s [...]
    </p>
   </div>

Another case where the linearity of the transcription does not correspond to the physical order of lines in the manuscript is that of interlinear or marginal additions. These must be marked up using the <add> tag and are considered non-alignable at the current stage of the project, as the image analysis software only detects the regular lines of the text.

A yet more complex situation is created in the case of scribal corrections of word order usually indicated by special transposition markers (See Figure 2).

Figure 2: Transposition

Example of transposition. Manuscript Lyon, B.M., P.A. 77, col. 163d. Image provided by Bibliothque Municipale de Lyon. The corrected text reads ueu plus apertement qu'il ne

If the transcription follows the final text order (which is usually the case), special markup is necessary in order to ensure the possibility of alignment. A combination of <ptr> and <seg> elements is proposed for this purpose in the Oriflamms project:

<ptr type="transposition-orig" target="#tps106_163d2_1"/>
<w type="ADVgen" xml:id="w106_006642">plus</w>
<seg type="transposition-target" xml:id="tps106_163d2_1">
  <w type="ADVgen" xml:id="w106_006643">apertement</w>
</seg>

The following XSL templates can be used to rearrange the text in the physical order:

<xsl:template match="tei:ptr[@type='transposition-orig']">
  <xsl:apply-templates select="id(substring-after(@target,'#'))"/>
</xsl:template>

<xsl:template match="tei:seg[@type='transposition-target']"/>

Manuscript pages can contain inscriptions that do not belong to the text being transcribed. These include catchwords and posterior annotations (such as library shelf marks). In some cases these elements may be transcribed (using <fw> and <add> tags) but in most cases they are not, and the alignment software has to guess which lines on the manuscript page are suspect and dropped from further analysis.

Alignable and Non-Alignable Elements

In addition to the transription stricto sensu, XML documents may contain a whole number of text containing elements that do not correspond to any graphical object in the source manuscript. These include the metadata recorded in the <teiHeader>, editorial notes (<note>), text supplied by the editor for missing or illegible passages (<supplied>). These are of course non-alignable. On the contrary, the elements <del> and <surplus> contain parts of text which should not appear in the edition (e.g. repeated or expunctuated words), but should be taken into account by the alignement. The transcription of these text segments had to be added to the source documents where appropriate.

Some transcriptions include critical apparatus providing variants from different manuscripts of the same text tradition (cf. the Fontenay transcription fragment provided above). These are encoded using <app> with <lem> and <rdg> child elements. In this case, the alignable text content is situated in the <lem> or <rdg> where the wit attribute contains a reference to the base manuscript of the transcription.

Whereas <choice> elements are used, only one of their child elements is alignable: <sic> (and not its <corr> sibling), orig (and not its <reg> sibling), <abbr> (and not its <expan> sibling). However, if <reg> and <expan> are used alone, they are alignable at word level, even though some characters in the transcription will not match those on the image. The <corr> element used alone may be alignable at word level but as is contains a segment of text somehow corrected by the editor, there is no guarantee that the characters contained by this element have any correspondence on the image.

The abbreviations may be particularly hard to align, as some of the markers occupy a position on the horizontal text line (as baseline or spacing superscript characters) and others are placed above other characters (as combining diacritics or superscript letters). The former are alignable, the latter are not. When the transcription represents the abbreviation markers directly, the Unicode areas labelled as combining can be used to identify the non alignable characters. When the transcription only contains an expansion with the supplied letters marked up using the <ex> tag, the only way to identify the alignable expansions (which are a minority) is to provide a list of them. Here is the list that has been used in the project so far:

Word and Character Level Tokenization

The <w> and <pc> tags are used on every alignable word and punctuation mark. All of them are equipped with an xml:id attribute which is used for alignment. The same technique is applied at character level using the <c> tag. The tokenization at both levels is performed automatically using pre-processing XSLT stylesheet librrary.

Image Markup and Linking

The results of the alignment software are recorded in separate files using standard TEI digital facsimile encoding mechanism with <zone> elements equipped with attributes indicating coordinates and with xml:id.

The linking is between the transcriptions and image zones is recorded in separate TEI files containing <link> and <linkGrp> elements joining references to the identifiers of words or characters in the transcription and of zones in the facsimile.