Balisage Paper: Don't Touch My SGML

SGML Processing

Let's assume we have this rather trivial SGML DTD:

<!-- NOTATIONS -->
<!NOTATION cgm PUBLIC 
"-//USA-DOD//NOTATION Computer Graphics Metafile//EN"               >

<!NOTATION ccitt4 PUBLIC 
"-//USA-DOD//NOTATION CCITT Group4 Facsimile//EN"                   >

<!NOTATION gcf PUBLIC
"-//ATA-TEXT//NOTATION Graphic Companion File//EN"                  >

<!NOTATION pdf PUBLIC
"-//Adobe Inc.//NOTATION Portable Document Format//EN"              >

<!NOTATION png PUBLIC
"-//W3C//NOTATION Portable Network Graphics//EN"                    >

<!NOTATION jpeg PUBLIC
"+//ISBN 0-7923-9432-1::Graphic Notation//NOTATION Joint
 Photographic Experts Group Raster//EN"                             >

<!NOTATION vrml PUBLIC 
"-//ATA-TEXT//NOTATION Virtual Reality Modeling Language//EN"       >

<!NOTATION mpeg PUBLIC
"-//ATA-TEXT//NOTATION Motion Pictures Expert Group//EN"            >

<!NOTATION mp3 PUBLIC
"-//ATA-TEXT//NOTATION Motion Pictures Expert Group, Audio
 Layer 3//EN"                                                       >

<!-- ENTITIES -->

<!ENTITY % id.att   "id ID #IMPLIED" >

<!ELEMENT doc   - - (title, chapter+) >
<!ATTLIST doc   %id.att; >

<!ELEMENT title - - (#PCDATA) >

<!ELEMENT chapter   - - (title, (para | graphic)+, section*) >
<!ATTLIST chapter   %id.att; >

<!ELEMENT section   - - (title, (para | graphic)+, section*) >
<!ATTLIST section   %id.att; >

<!ELEMENT para      - - (#PCDATA) >

<!ELEMENT graphic   - O EMPTY >
<!ATTLIST graphic   href CDATA #IMPLIED
                    name ENTITY #IMPLIED
                    %id.att; >

A valid SGML instance might look like this:

<!DOCTYPE doc PUBLIC "-//CWORD//DTD DOC TEST//EN" "doc.dtd" [
<!ENTITY bill SYSTEM "Bill_Murray.jpg" NDATA jpeg >
]>
<doc>
    <title>My Test SGML Document</title>
    <chapter>
        <title>Chapter 1 Title</title>
        <para>Some content.</para>
        <graphic href="Bill_Murray.jpg">
    </chapter>
    <chapter>
        <title>Chapter 2 Title</title>
        <para>More content.</para>
        <section>
            <title>Subsection 1 Title</title>
            <para>Subsection content.</para>
            <graphic name="bill">
        </section>
    </chapter>
</doc>

This is assumed to use Unicode, but most real-life SGML DTDs will reference external ISO character entity sets since the DTDs were usually conceived before Unicode. For example, ATA DTDs include something along these lines^[9]:

<!ENTITY   % ISOtech     PUBLIC 
"ISO 8879-1986//ENTITIES General Technical//EN"                >
<!ENTITY   % ISOpub      PUBLIC 
"ISO 8879-1986//ENTITIES Publishing//EN"                       >
<!ENTITY   % ISOnum      PUBLIC 
"ISO 8879-1986//ENTITIES Numeric and Special Graphic//EN"      >
<!ENTITY   % ISOgrk1     PUBLIC 
"ISO 8879-1986//ENTITIES Greek Letters//EN"                    >
<!ENTITY   % ISOgrk2     PUBLIC 
"ISO 8879-1986//ENTITIES Monotoniko Greek//EN"                 >
<!ENTITY   % ISOgrk3     PUBLIC 
"ISO 8879-1986//ENTITIES Greek Symbols//EN"                    >
<!ENTITY   % ISOgrk4     PUBLIC 
"ISO 8879-1986//ENTITIES Alternative Greek Symbols//EN"        >
<!ENTITY   % ISOlat1     PUBLIC 
"ISO 8879-1986//ENTITIES Added Latin 1//EN"                    >
<!ENTITY   % ISObox      PUBLIC
"ISO 8879-1986//ENTITIES Box and Line Drawing//EN"             >
<!ENTITY   % ISOamso     PUBLIC
"ISO 8879-1986//ENTITIES Added Math Symbols: Ordinary//EN"     >
<!ENTITY   % ISOamsc     PUBLIC
"ISO 8879-1986//ENTITIES Added Math Symbols: Delimiters//EN"   >

%ISOtech; %ISOpub; %ISOnum; %ISOgrk1; %ISOgrk2; %ISOgrk3;
%ISOgrk4; %ISOlat1; %ISObox; %ISOamso; %ISOamsc;

An entity like — would have a corresponding entity declaration:

<!ENTITY mdash  SDATA "[mdash ]"--=em dash-->

The SGML processing, all of which is in an Ant build file, involves the following:

XML Processing

The well-formed XML needs to be converted to whatever the end vocabulary (DTD, schema...) requires. Whilst the specifics will differ—certainly for the three formats, above—the principle is the same: tweak the XML as necessary to produce valid XML. This involves running a series of XSLT stylesheets, different for each vocabulary, but the pipeline mechanism is the same.

A first step is usually to encode the XML in UTF-8 and get rid of the entity declarations. Another is often to convert any entity-based graphics to direct references; XML processing can do a lot, but entity queries and manipulation remain an absolute pain^[12].

The same Ant build file that converts the SGML to XML calls an XProc pipeline that runs a manifest file listing the required XSLT stylesheets in the order in which they run.

The sgml2xml-tng Repo

The SGML-to-XML conversion repository [SGML to XML TNG] is organised like this:

Figure 4: SGML to XML TNG

Here's the idea:

The step outputs can all be saved to help debugging:

doc/
├── data
│   └── (SGML and graphics input)
└── tmp
    └── 20260311133633
        ├── debug
        │   └── (XProc XSLT pipeline debug output)
        ├── out
        │   └── (valid XML and graphics)
        ├── pre
        │   └── (preprocessed SGML)
        ├── reports
        │   └── (reports)
        ├── spam
        │   └── (spam output)
        ├── sx
        │   └── (sx output)
        └── xml
            └── (well-formed XML)

The Ant build also provides an optional nsgmls validation step.

Libraries and Tools

The SGML processing exists thanks to James Clark's fabulous SP toolkit [SP - An SGML System Conforming to International Standard ISO 8879 -- Standard Generalized Markup Language]. Without it, this paper would not exist.

The XML processing was always going to be a series of XSLT transforms. I much prefer the pipelined approach [Pipelined XSLT Transformations], and the default XML processing assumes a series of XSLTs listed in a manifest.

The pipelines are XProc 3.0 run with Achim Berndzen's MorganaXProc-IIIse [MorganaXProc-III]. The default pipelines use XProc Batch [XProc Batch], but this doesn't have to be the case.

There's also David Maus's SchXslt2 Schematron to XSLT 3.0 transpiler [SchXslt2] to run Schematron validation where needed.

Modelling XML DTDs and Schemas

While XML is the modern standard, SGML’s modeling capabilities were, in some respects, more robust. For example, SGML featured a logical AND group—allowing elements to appear in any order but exactly once. In XML, achieving this level of validation typically requires moving beyond DTDs to Schematron or XSD.

SGML also permitted exclusions and inclusions—convenient shortcuts that often complicate conversion. Exclusions allowed a designer to prohibit specific elements (like a table) from all descendants of an element. Inclusions did the opposite, allowing a specific element to appear anywhere within a hierarchy, bypassing standard model rules.

The inclusion mechanism is a particular challenge for XML DTDs. In ATA iSpec 2200, for example, revision markers (REVST and REVEND) were inclusions. Because they could legally appear at different hierarchy levels, they are impossible to model in an XML DTD without resorting to messy, repeatable OR groups.

Two primary strategies exist for resolving these models during conversion: