Balisage Paper: Beyond Eighteen Wheels

An XML Document

Before we can talk about storing documents for a long time, we must decide what we mean by the term document. This is not sophistry: an XML document generally consists of multiple parts, and, as will be discussed further, we must be careful in determining the boundary of the document for the purpose of preservation. Let the term XML Document, then, in this paper, denote a sequence of characters represented digitally on a computer such that the sequence of characters satisfies the productions and constraints of the XML specification. We shall be neither more not less precise, but rather shall qualify or expand upon this base term as needed. Our definition excludes non-digital representations: if one were to print out an XML document onto paper, and then make a video of the paper, the video would not, by our definition, constitute an XML document.

A Long Time

The overview for the Balisage pre-conference symposium suggests that a long time, in terms of document storage, could be for as long as a thousand years. The author of this document, however, suggests that A Long Time, however long it may actually be, in any case starts now, in the present, and gradually lengthens. What is significant is that the people who created the document are not the people who decode it, and that the context of that decoding is not necessarily the same as the social, technological or political context in which the document was encoded.

Storing

By our definition an XML Document exists only within computer storage. A document will be said to have been stored for a given period of time if, at the end of that time, the same sequence of characters can be retrieved. This definition permits changes in the encoding of the document, for example from UTF-8 to UTF-16, as long as the sequence of encoded characters remains the same. Our definition is also silent on whether the document may be inaccessible at times during the stored period.

One example of a document becoming inaccessible might be a so-called dark archive, created to hold copyrighted information until such time as the copyright expires. That is a political or social inaccessibility: the document might be accessible technically, but perhaps not legally. Documents can easily become technically inaccessible. A promise of XML is that the format is open, and, unlike, say, a proprietary word processing format, will still be readable in the future. The same promise was made of SGML documents, but few people make SGML software today. Fortunately, software exists to convert SGML documents to XML, but doing so correctly requires human expertise in both formats in order to select correct options. In order to retain technical accessibility, then, documents may need to be migrated between formats. If this is not done, however, keeping the format definition along with the documents may facilitate such work after a Very Long Time.

The Physical Substrate

It is not reasonable to expect modern computer storage devices to remain functional for A Very Long Time. Typical values for A Very Long Time for most computer equipment today are measured in thousands of hours, not thousands of years. Magnetic tapes degrade over time, as do optical storage media such as compact discs. Active devices such as rotating hard drives have dependencies on voltage and current levels, on specific versions of software drivers for specific operating systems, and, since they contain firmware, may also fail after a specific date.

It is possible to run a digital archive in such a way that data is periodically migrated to newer media. Such a strategy assumes a continued supply of funding and replacement media. It is also possible for an organization to rely on external archiving services, but this does not solve the question of ensuring that A Very Long Time is sufficiently great.

Suitable physical media for long-term storage of digital data remains an unsolved problem at this time.

The author of this document once unpacked a computer system; inside the box was also a manual in many ring-bound volumes, and it was necessary to open shrink-wrapped stacks of hole-punched paper and insert them into the proper binders. One of these manuals was a chapter explaining how to off-load the box with a fork-lift truck, and how to open the box. Of course, in order to discover these instructions, it was necessary to open the box. When archiving for a Very Long Time, it is important to label the archive in multiple languages, with a pen, on the outside of the box. Who, in a thousand years from now, would guess that an object clearly marked 90 Minutes Audio Cassette actually contained a computer program? If the instructions for unpacking the archive are inside the archive, how will they be used?

The Logical Layers

Computer users are accustomed to metaphors presented by graphical user interfaces. For example, a Folder is used as a metaphor for a group of documents. But the actual implementation of a File System on most operating systems today involves a list of hard disk block numbers or storage extents. It might be that a future data archaeologist will have to inspect those individual disk blocks and piece them together. This process is made considerably easier if files larger than a single block are in plain text wherever possible, rather than (for example) being compressed. In addition, the fewer layers that must be penetrated, the easier the task, so store individual files in folders (directories) rather than in binary formats such as zip or tar archives. The process of reconstructing data from a damaged CD-ROM or hard drive is tedious, but today at least it is a known skill; many skills fall into disuse, and today few people can repair a hole in a saucepan, sharpen a wooden ploughshare, or correctly aim a ballista. If an important archive is to be stored for a long time, the layout of the storage system file systems must be documented on a separate physical medium.

A text file is in actual fact stored digitally in a way that could be thought of as a sequence of integers, with an implied mapping from integers to characters and from character sequences to visible representations known as glyphs. The mapping from integers to characters is known as an encoding; the mapping from characters to glyphs is implemented by fonts.

Avoid Implicit Content

Some XML Schema languages, including XSD and DTDs, have the ability to provide “default” attribute values, or, in some cases, even default element content. The document is augmented by this content after validation against a schema by a software process. Since we cannot assume that software processes will still be runnable A Very Long Time from now, we should make sure that the archived XML Document can be used without schema validation, and, in XML terminology, is a stand-alone document.

Avoid Obscure Features

Any feature of XML, or of any other format, that is not widely implemented, or whose behaviour varies between implementations, or whose semantics are not clearly documented, should be avoided. For XML that might include, for example:

Avoid Cryptic Names

A necessary precondition of usability is comprehensibility. If someone else if to make use of our markup they must understand it. Consider the example in the following listing, in which short element names and a flat strucure have been used for a novel (the actual content has been reduced to numbers to avoid accidentally introducing anything of interest into this paper):

		<h>1</h><p>3</p><p>10</p><p>21</p><h>64</h><p>129</p>
	    

In the listing, the relationship of the elements is not explicit. An improved version is given in listing 2:

<c><h>1</h><p>3</p><p>10</p><p>21</p></c><c><h>64</h><p>129</p></c>

Someone inspecting this document might not have enough information to deduce the intent of the markup, and we could make the further improvement of listing 3:

<chapter>
    <heading>1</heading>
    <paragraph>3</paragraph>
    <paragraph>10</paragraph>
    <paragraph>21</paragraph>
</chapter>
<chapter>
    <heading>64</heading>
    <paragraph>129</paragraph>
</chapter>

Of course, such lengthy element names may be inconvenient when editing a document. Plausible compromises include converting to an archival format, or making sure that each document includes sufficient information to allow someone to reconstruct this meaning. Of these, the first approach seems more likely to stand the test of time.

Document the significance of markup items

Write clear descriptions of the purpose of each XML element and attribute; be careful not to rely on the element name in the description. For example, do not describe the bulletlist element as containing a bulleted list; describe it as containing a sequence of independent items that form a related group or sequence, and give an example. The meaning and usage of terms such as Paragraph, Section, List, Title, Bullet and Flower, Folio and Explication has not remained constant over the past few centuries, and even today is not constant across all cultures.

A project to archive a body of documents over a Very Long Time might well ensure that the documentation is translated into multiple languages, to increase the chance that a future user will be able to understand what is written. Of course, one might do the same with the actual documents as well as the documentation about the rhetorical and technical contexts of the document. Changing the actual XML Document is outside the scope of this paper, but the possibility of creating an archival format alongside the original document has already been suggested.

Validate the Data

If there is uncertainty about the ability of recipients and users of an archived document to be able to use the XML format, there must be considerably more uncertainty about their ability to cope with errors in the use of the format. Make sure that all XML data is will-formed: not only the XML Document itself but all supporting information. Validate against XML Schemas, DTDs or other test documents wherever possible. Where data fails to validate, the archivist should attempt to ask the supplier to provide corrected content; where that is not possible, the original data must obviously be archived, but the archivist should consider including a corrected version of any documents that do not validate, along with information about the validity problems. Including detailed notes about the format emplyed is of little use if the documents to not correspond to the documentation.

Mean What You Mean To Mean

There is no clear definition of Meaning that satisfies all users of a typical XML document. Some people are interested in denotational semantics from a semiotic viewpoint (what is signified?), some in behavioural semantics (what does it do), some merely in human understanding (what is it?). A consequence of this is that there is no single universal way to denote the meaning of XML markup. W3C XML Schema documents can contain annotation elements for this purpose.

Check Links

If your document format has links, whether explicit or implicit, you should check them before committing the document or documents to the archive. As a minimum, make sure that all link targets exist. Better, make sure that the links go where they should. For example, check the titles of sections that are the targets of links, perhaps to the content of an element in the link that exists only for that purpose. For off-site links, for example to Web sites, consider archiving a surrogate, or including a description of the remote content. Archiving the actual remote Web page may require explicit permission, but for some projects this is worth while. Ensure that you are checking linked documents in your collection that you are about to archive, and that your link checker is not reaching out to a production server elsewhere in your organization! All of this requires provision in the design of an XML document format.

Although the World Wide Web Consortium has produced a specification for marking up hypertext links in XML, xlink, this specification is not likely to be sufficient even for explicit links, as it does not provide for a pattern to match against a given target element in order to validate a link.

The xlink specification does not attempt at all to support implicit links, where, for example, a part-number element within a step of a repair procedure is treated as a link to a database of parts, and for printing is augmented by a description of the part, and for online publication also becomes a hyperlink to an online catalogue. Like implicit content from default elements and attributes, implicit links are difficult to archive. The ISO SGML HyTime specification did have support, using a mechanism termed Architectural Forms, for at least some level of implicit linking, but HyTime has not been adopted by the XML community.

The best approach to linking in documents that are expected to be archived, then, is to include in each document some information about link targets. This could be a short textual description or an entire resource. For the purpose of link checking, static content in the document can be used, such as an attribute whose value specifies an XPath expression, and another attribute giving a regular expression (a text pattern) that the result of evaluating that XPath expression must match. For example, one might say that the nearest enclosing section title of the link target must contain the word Plastic. The utility of this is at archive creation time, and during the regular document maintenance life-cycle.

Provide for Translations

After A Very Long Time, language will have changed. Documents that have been translated into multiple languages clearly have a better chance of bing understood. A single Rosetta Stone is more likely to survive in one place than three separate stones: consider archiving a single file containing all of the translations, or at least a fragment from each language, perhaps first a German paragraph and then a French one and then a Swedish one (in alphabetical order by their own names, but that breaks down for languages written with non-Latin scripts). Remember that any piece of human-readable text may need to contain markup in some language or other, whether to delimit right-to-left and left-to-right components, or for Ruby-style annotations, or for emphasis. As a result, all natural-language fragments should be in XML elements (not attributes), where they can be distinguished by xml:lang values. Designing XML documents for translation has been described elsewhere [e.g. ITS] in detail.

It can be helpful to have block-level translations, so that (for example) a paragraph or list item in one language can be immediately followed by corresponding text in another language. This does not always work, since the rhetorical structures in different cultures may suggest that material be most naturally presented in different sequences.

Provide for Contextualization

Include a place for authors to make explicit not only the purpose of the document, but how it is to be used and how it might fit into the ecosystem of the wider context in which it is created. For example, a dream journal might have an introduction that says the author wrote down memories of dreams each day for a year, but the wider context might include that this was part of a therapeutic exercise in working out resentment towards alien visitors, and that, after a year, the writer's perception about the visitors was changed. This sort of explanation is not generally considered necessary for a work of fiction, but the boundaries between perception and construction become less clearly discernible over time. Harry Potter is a fictional boy, but of course the train from King's Cross station in London, and the oddly numbered platform there, really exist. They have been created after the success of the books, but that small detail may no longer be obvious a Very Long Time from now.

In a business context, the reasons behind particular decisions and documents may not be public; it may be desirable to store two versions of a document, one of which is to be made available only fifty years (say) after its original creation. Some governments have similar policies, although the details vary widely between countries.

As an example of contexts, consider the following amounts of money: all except one represent the same value.

Adding some context would make this clearer:

To belabour the point, even here there is some difficulty. The second item, £-/6/8, is a now-obsolete noation for six shillings and eight pence in the pre-decimal British currency; with twelve pennies in the pound that made 80d (denarii, pennies). Italy, in pre-Euro days, used the Lira, and used the same symbol for currency, although devaluation meant there were of the order of a thousand Italian Lira to the Britsh Pound (at some point in history); similarly, the two dollar figures are from two different countries using the dollar as a currency symbol. Thus, notations and symbols that we take for granted may lose their meaning over time, or not be clear to future readers. There is no clear way to determine how much context to retain, and yet we must persist in claiming that the more context is recorded, the greater the chance of successful communication.

Don't be Inventive

Use existing specifications where possible; if that is impractical, consider copying techniques from existing specifications, which can then be included in the archive. The more people who use a specific technique, the more likely it is to survive.

Summary

Use validated, well-documented XML that wherever possible relies on widespread practices.

Textual Format

As we have already discussed, textual formats tend to be more robust in the face of possible data corruption than compressed or binary formats. If a single storage device block becomes unreadable, the rest of the document after the lacuna will still be readable, albeit incomplete.

Explicit End Markers

Not all text formats have explicit markup to surround objects, or, if they do, may use an ambiguous symbol to mark an ending, such as a close brace or a closing parenthesis. The possibility of data corruption means that the additional redundancy of repeating an element name can help to identify errors and limit the scope of corruption.

Embedded Usage

XML is used in devices ranging from automobile engines to television sets. Its use is very widespread, and it is found in devices expected to last for thirty years or more. These devices cannot easily be changed if XML becomes obsolete and is replaced by (say) Aldus PageMaker files. This means that the chance of XML technology surviving, or being historically retrievable, is statistically significant: 3% of 1000 years is 30 years.

Device Independent

XML Documents do not generally rely on specific hardware. For example, one does not embed Epson dot-matrix printer control sequences in XML documents in order to generate underlining. XML-based graphics formats are generally declarative, and do not generally rely on initializing a plotter or on the size of a sheet of paper.

Open Specification

The XML Specification meets all four of the criteria given in this paper for a specification to be considered fully Open. This is not to say that no other format does: many do.