Balisage Paper: Migrating DocBook to Uncharted Waters

XSLT Pipelines Overview

Those recollecting previous Balisage papers may remember Ari's fascination with XSLT pipelines [Pipelined XSLT Transformations]. While the paper at hand is not about how XSLT pipelines work, the idea is simple enough:

A sequence of XSLT stylesheets listed in an XML manifest line runs in the order presented, with the previous XSLT's output providing the input to the next. Each XSLT transform handles one thing and one thing only.^[2], and that is really about it. Every XSLT is essentially an ID transform with a few tweaks, and since the full transformation is a sequence of these XSLTs, it is easy to modify the transform by adding, removing, and moving the individual XSLTs. For example, this manifest is the XSLT pipeline part of the DocBook cleanup described in the following section:

<manifest xmlns="http://www.corbas.co.uk/ns/transforms/data" xml:base=".">
    
    <group description="Initial stuff" xml:base="../xslt/common/">
        <item href="idtransform.xsl" description="UTF-8 serialisation to get rid of the character entities"/>
        <item href="remove-wc-protocol.xsl" description="Remove Windchill protocol from hrefs"/>
        <item href="clean-hrefs.xsl" description="Remove unwanted chars in hrefs"/>
        <item href="uri-encode-hrefs.xsl" description="URI-encode refs"/>
    </group>
    
    <group description="Main processing" xml:base="../xslt/db-cleanup/">
        <item href="fix-xi-namespaces.xsl" description="Move any non-standard XIncludes to the right namespace"/>
        <item href="normalize-docbook.xsl" description="Normalise XIncludes, remove namespaces, serialise to UTF-8" enabled="false"/>
        <item href="orphaned-table.xsl" description="Reinsert title and tgroup in table orphaned by Tagsoup"/>
        <item href="convert-table-pi.xsl" description="Table PI handling"/>
        <item href="cleanup.xsl" description="Various cleanup"/>
    </group>
    
</manifest>

Did we mention that it's practically self-documenting?

Encoding and DOCTYPE

Character entities, of course, are only possible to convert if declarations are present. In the case of DocBook 4.0 or any other DTD-validated XML documents, the place to look for character entity definitions in the DTD. A parser normally accesses the DTD by reading the document's DOCTYPE declaration. However, what if the DOCTYPE definition is missing?

In a nutshell, the DocBook 4.0 source consisted of multiple files, with the top-level (root) file containing XInclude links pointing out chapters, which in turn would point out sections or Q and A entries. However, only the top-level XML had a DOCTYPE statement. The lower-level XML filed did not require a DOCTYPE statement to work in Arbortext's ecosystem, creating an interesting chicken-or-egg problem.

See, if a parser finds a character entity it can't resolve, it usually reports an error and exits. The XIncluded files contained character entities but no DOCTYPEs statements resulting in attemptes to open XML files that immediately resulted in errors because the initial cleanup scripts threw errors and exited because entities did not resolve properly.

So, before fixing the encoding, the files needed DOCTYPE declarations where missing, thus ruling out anything XSLT^[5] and most things XProc^[6], so what, then? How about something in an Ant build script that would then run the actual cleanup? Ant, an XML-based pipelining language, offers many tools to process XML, but very few would uncritically open and read an XML file that is not well-formed.

Regular expressions are always an option, of course, but just look at these two variations:

<!--Arbortext, Inc., 1988-2017, v.4002-->
<?Pub EntList alpha bull copy rArr sect trade ordm ohm mu j0 omega
eacute?>
<section id="sectionKASTDE08003" vendor="0000084394">

The previous example has no DOCTYPE to begin with, so it may be possible to match the root element. But what about this one?:

<!-- Fragment document type declaration subset:
Arbortext, Inc., 1988-2017, v.4002
<!DOCTYPE set PUBLIC "-//Arbortext//DTD DocBook XML V4.0 + MathML//EN"
 "axdocbook_math.dtd" [
<!ENTITY minus "&#38;#8722;">
]>
-->
<?Pub EntList alpha bull copy rArr sect trade lbrace rbrace thinsp delta
Delta?>
<?Pub Inc?>
<qandaentry id="qandaentryFEBB06-014" role="1of4" vendor="0000080121">

This one is sneaky because the DOCTYPE is out-commented over multiple lines (and has the wrong document element). You must match the comment markup and then the DOCTYPE inside it to know.

And so on. There are a couple of things to do here:

A few of the XIncluded files had out-commented DOCTYPE declarations, complicating Step 1. Also, getting the root element was not obvious because of the comments, processing instructions, etc, so the regex idea was a no-go.

OK, so maybe an Ant exec task to run external software that adds the DOCTYPE? Most parsers are out of the question because of the basic well-formedness issues, but there is something called Tagsoup [TagSoup - Just Keep On Truckin'], a rather brilliant piece of software intended to fix bad HTML. Among other things, it can add any DOCTYPE declaration to a file, not just HTML, but more importantly, it doesn't care if the file contains unresolved character entities—that is one of the problems it's designed to ignore while fixing other things!

It doesn't do reporting very well, however, so it couldn't reliably check for an existing DOCTYPE and only then, if there wasn't one, add it. For checking for the existence of one, we ended up using xmllint [Installing Libxml/xmllint for Windows]^[7]

XIncludes

With the character encoding handled, the next task was processing the XIncludes. We promptly learned that ACM used a non-standard file protocol:

<xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
href="x-wc://file=0000055442.xml"/>

Handling this was straight-forward in an XSLT step and a regular expression.

While the normalisation of XIncludes (that is, recursively pulling in the XInclude contents into a single file) is not a complex problem, the fact that there was no way to know which of the hundreds of XML files in an input folder was the root complicated matters. Once you have pulled in an XInclude target in the parent file, you no longer have to process the child; you can leave it behind and only process the normalised parent. How do I leave them all behind?

The XSLT pipeline acts on an input directory: it lists the files in that directory and then iterates through the list, applying the pipeline to each one. Here is the clever bit, though: you can apply to include and exclude filters on the operation by using regular expressions.

A pre-processing pipeline step first walks through all of the input files, opening each of them in turn, and compiles a list of the XInclude targets, each separated with a | (pipe) character (whitespace added for readability):

<exclude-filter
    xmlns:axi="urn:x-prefix:xi"
    xmlns:c="http://www.w3.org/ns/xproc-step"
    xmlns:xi="http://www.w3.org/2001/XInclude">
    (Introduction_0000080922.xml|
    Electrostatic_Fields_0000080916.xml|
    Voltage_0000080934.xml|Current_0000080933.xml|
    Magnetic_Fields_0000080930.xml|
    Induced_Voltage_0000080928.xml|
    equationEEEG029_Through_equationEEEG033_Capacitors_0000151486.xml|
    ...
    )
</exclude-filter>

We could extend the exclude filter, but the principle was easy enough except for the filenames.

Filenames

The first few tests had filenames without particular issues, but the first complete book we examined highlighted the weakness of the exclude filter approach described above.

The Arbortext system happily processes filenames with any number of weird characters while some of the pipeline tools choked. For example, some file names contained multiple spaces, parentheses, square brackets, commas, and other characters not typically found in file names.

If you know your regular expressions, you will know that several of these would have to be escaped if used in a pattern. So, when listing the XInclude targets and using the list as an exclude filter (see above), the list would be unusable as a pattern unless every offending filename character was escaped correctly in the regex. This was not an attractive proposition, especially after examining more book sources.

Easier was writing a pre-processing step to rename the files by removing (or replacing) the offending characters and then opening each file and applying the same renaming rules to any referenced files—all this had to be run before the actual cleanup and normalisation to make sure that the XInclude target list used as an exclude filter would not break^[8].

The rename step eventually became a separate XProc library where XSLT stylesheets handled most of the heavy lifting.

4.0 to 4.3 and Cleanup

The actual DocBook 4.0 to 4.3 conversion held very little excitement when compared to the above: a handful of XSLTs to get rid of Arbortext PIs for formatting and other functionality and to unify namespaces—for some reason, authors had sometimes added, um, alternative namespace URLs for MathML and XInclude. The DocBook versions as such are very similar.

The 4.3 DOCTYPE

A final cleanup complication was that the full DOCTYPE for the target files required an internal subset:

[
<!-- Will use "m" as a prefix.-->
<!ENTITY % equation.content "(alt?, (graphic|mediaobject|m:math)+)">
<!ENTITY % inlineequation.content "(alt?, (graphic|inlinemediaobject|m:math)+)">
<!ENTITY % NS.prefixed     "INCLUDE">
<!ENTITY % MATHML.prefix   "m">
<!ENTITY % NamespaceDecl.attrib "
	xmlns:m
		CDATA	#FIXED 'http://www.w3.org/1998/Math/MathML'
">
<!-- Include the MathML DTD -->
<!ENTITY % mathml PUBLIC "-//W3C//DTD MathML 3.0//EN"
                  "http://www.w3.org/TR/MathML3/dtd/mathml3.dtd">
%mathml;
<!ENTITY % xinclude SYSTEM "http://www.docbook.org/xml/4.4/xinclude.mod" >
%xinclude;
]

The XML world has been moving away from DTDs and DOCTYPE declarations ever since XML first came out, 24 years ago. You can't actually output an internal subset or serialise an ENTITY declaration using XSLT^[9]. You can still serialise a basic DOCTYPE, though, with PUBLIC and SYSTEM identifiers.

This was good enough. Remember the Ant build wrapped around everything? Well, we had an external text file with the internal subset, and then, last in the build, after provisioning the processed files with vanilla DocBook 4. DOCTYPE declarations, this added that external text file to the DOCTYPEs:

<target name="insert-doctype">
    <loadfile property="local.doctype" srcfile="${resources.internal-subset}"/>
    
    <!-- Insert the DOCTYPE internal subset -->
    <replaceregexp match="^(.*)(/docbookx.dtd&quot;)(>(.*))$" replace="\1\2${local.doctype}\3" flags="m">
        <fileset dir="${base.out}">
            <include name="*.xml"/>
        </fileset>
    </replaceregexp>
</target>

Cheating, I know^[10].

Normalisation and Split Markup

All this meant that we'd have to start by normalising the sources after all, insert markup to identify where they would be pulled apart again, later in the pipeline, and then add metadata and other structures.

This is where the XInclude normalisation step [section “XIncludes”] to preprocess the XInludes to use the XInclude targets in an exclude filter to the directory listing became a lifesaver. The main book file would grow significantly larger in the first step since everything was pulled in, as seen in this debug listing:

Figure 1: Normalisation

Here, the normalisation is followed by steps to prepare the newly assembled book to be split again. Some conditions had to be done as part of the normalisation because those required examining filenames and containing folders to produce context markup^[12]:

<!-- Find out stuff about the filename to use for later naming and split -->
<xsl:template match="*[not(parent::*)]">
    <xsl:variable name="name" select="name(.)"/>
    
    <xsl:variable name="filename" select="tokenize(base-uri(.),'/')[last()]"/>
    
    <xsl:variable name="label">
        <xsl:choose>
            <!-- Properly marked-up chapter -->
            <xsl:when test="@label and @label!='' and $name='chapter'">
                <xsl:value-of select="@label"/>
            </xsl:when>
            <!-- Diagnostic -->
            <xsl:when test="matches(@id,'diag')">
                <xsl:analyze-string select="@id" regex="^(.*)diag([0-9]+)$">
                    <xsl:matching-substring>
                        <xsl:value-of select="format-number(number(regex-group(2)),'1')"/>
                    </xsl:matching-substring>
                    <xsl:non-matching-substring/>
                </xsl:analyze-string>
            </xsl:when>
            <!-- Is it a front matter chapter? -->
            <xsl:when test="matches(@id,'^(.*)00$')">
                <xsl:value-of select="'0'"/>
            </xsl:when>
            <!-- Gleaned from @id -->
            <xsl:when test="matches(@id,'^chapter[A-Z]+[0-9]+$')">
                <xsl:analyze-string select="@id" regex="^chapter[A-Z]+([0-9]+)$">
                    <xsl:matching-substring>
                        <xsl:value-of select="format-number(number(regex-group(1)),'1')"/>
                    </xsl:matching-substring>
                </xsl:analyze-string>
            </xsl:when>
            <xsl:otherwise>
                <xsl:value-of select="''"/>
            </xsl:otherwise>
        </xsl:choose>
    </xsl:variable>
    
    <xsl:variable name="title" select="string-join(title//text())" as="xs:string?"/>
    
    <!-- ID with 'diag' means diagnostic -->
    <xsl:variable name="dg" select="if (matches(@id,'diag')) then ('diag') else ()"/>
    
    <!-- RM books are identified thusly -->
    <xsl:variable name="rm" select="if (matches($path,'RM/data/$')) then ('rm') else ()"/>
    
    <xsl:variable name="prefix">
        <xsl:choose>
            <xsl:when test="$name=('chapter','bookinfo')">
                <xsl:value-of select="'Chapter'"/>
            </xsl:when>
            <xsl:when test="$name='qandaentry'"/>
            <xsl:otherwise>
                <xsl:value-of select="$name"/>
            </xsl:otherwise>
        </xsl:choose>
    </xsl:variable>
    
    <xsl:variable name="identifier" as="xs:string">
        <xsl:choose>
            <!-- Intro front matter chapter -->
            <xsl:when test="$title='Introduction' or $label = '0.50'"><!-- FIXME -->
                <xsl:value-of select="'0.50'"/>
            </xsl:when>
            
            <!-- Diagnostic test inside the book, chapter label xx.50 -->
            <xsl:when test="$name='chapter' and matches($filename,'_DG_')">
                ...
            </xsl:when>
            
            <!-- Labelled (non-diagnostic) chapter -->
            <xsl:when test="$name='chapter' and $label!='0' and $label!=''">
                <xsl:value-of select="$label"/>
            </xsl:when>
            
            ...
            
        </xsl:choose>
    </xsl:variable>
    
    <xsl:copy>
        <xsl:copy-of select="@*"/>
        <xsl:attribute name="prefix" select="$prefix"/>
        <xsl:attribute name="identifier" select="$identifier"/>
        <xsl:attribute name="dg" select="$dg"/>
        <xsl:if test="$rm='rm'">
            <xsl:attribute name="rm" select="$rm"/>
        </xsl:if>
        <xsl:apply-templates select="node()"/>
    </xsl:copy>
</xsl:template>

Based on this, the next step would then identify splits, like in this example:

<xsl:if test="name(.)=$split-elements and not(ancestor::chapter[@rm='rm'])">
    <xsl:attribute name="split" select="true()"/>
</xsl:if>

A step that named the soon-to-be fragments in a temporary @name attribute then followed. For example:

<xsl:template match="*[@split='true']">
    <xsl:variable
        name="name"
        select="name(.)"/>
    
    <xsl:variable
        name="prefix"
        select="@prefix"/>
    <xsl:variable
        name="identifier"
        select="if (matches(@identifier,'\.50') and @dg='diag')
        then (following-sibling::chapter[1]/@identifier || '.50')
        else (@identifier)"/>
    
    <xsl:variable name="combined-name">
        <xsl:choose>
            <xsl:when test="$prefix='Chapter'">
                <xsl:value-of select="$acronym || '_' || $prefix || '_' || $identifier"/>
            </xsl:when>
            <xsl:when test="$prefix!=''">
                <xsl:value-of select="$acronym || '_' || $prefix || '_' || $identifier"/>
            </xsl:when>
            <xsl:when test="@prefix=''">
                <xsl:value-of select="$identifier"/>
            </xsl:when>
        </xsl:choose>
    </xsl:variable>
    
    <xsl:copy>
        <xsl:copy-of select="@*"/>
        <xsl:attribute name="name" select="$combined-name || '.xml'"/>
        <xsl:apply-templates select="node()"/>
    </xsl:copy>
</xsl:template>

Note that none of this was part of the initial pipeline—the original statement of work left out everything having to do with splitting based on context, type of book.

Metadata

The metadata depended on context and the type of book, essentially the same criteria used to determine where to split the normalised files. For example, the @rm attribute, added during normalisation to identify the Reference Manual book type context, determined if the Q and A entries needed metadata or not:

<xsl:template match="orderedlist[(ancestor::chapter[@rm='' or not(@rm)]) and
    ancestor::question and listitem[@role='correct']]">
    <xsl:copy>
        <xsl:copy-of select="@*"/>
        <xsl:attribute name="class" select="'ktp-answer-set'"/>
        <xsl:apply-templates select="node()">
            <xsl:with-param name="property" select="'ktp:answer'" tunnel="yes"/>
        </xsl:apply-templates>
    </xsl:copy>
</xsl:template>

Another example, also pre-converting selected structures to HTML:

<xsl:choose>
    <!-- Not RM (don't add @class=solution_title before answer) -->
    <xsl:when test="ancestor::chapter[@rm='' or not(@rm)]">
        <xhtml:section
            property="ktp:explanation"
            typeof="ktp:Explanation"
            class="ktp-explanation"
            keep="true">
            <xhtml:section
                property="ktp:explanation-section"
                typeof="ktp:feedback"
                class="ktp-explanation-section"
                data-title="Solution"
                keep="true">
                <xsl:apply-templates select="node()"/>
            </xhtml:section>
        </xhtml:section>
    </xsl:when>
    <!-- RM -->
    <xsl:otherwise>
        <xhtml:span keep="true" class="solution_title t-color--atom-primary t-italic">Solution</xhtml:span>
        <xsl:apply-templates select="node()"/>
    </xsl:otherwise>
</xsl:choose>

What metadata was added to what elements and in what contexts also changed during the project, and again, the pipeline approach proved its worth.

Structural Tweaks

We ended up using seven pipeline steps to add metadata to the required DocBook structures, with another four to tweak them further, and it proved easier to convert these to HTML as part of the metadata additions rather than having xslTNG do it later.

The Q and A entries, for example, use lists for both to list possible answers and for, well, listing things. While it was certainly possible to convert them to HTML using xslTNG, it was easier in a pipeline step:

<xsl:template match="orderedlist[ancestor::question]">
    <xhtml:ol keep="true">
        <xsl:copy-of select="@*"/>
        <xsl:if test="not(@class) and (ancestor::chapter[@rm='' or not(@rm)]) and not(@numeration='upperroman')">
            <xsl:attribute name="class" select="'ktp-answer-set'"/>
        </xsl:if>
        <xsl:choose>
            <xsl:when test="@numeration='upperroman'">
                <xsl:attribute name="style" select="'list-style: upper-roman'"/>
            </xsl:when>
        </xsl:choose>
        <xsl:apply-templates select="node()"/>
    </xhtml:ol>
</xsl:template>


<xsl:template match="listitem[ancestor::question]">
    <xhtml:li keep="true">
        <xsl:copy-of select="@* except @role"/>
        <xsl:if test="not(@property) and (ancestor::chapter[@rm='' or not(@rm)]) and not(ancestor::orderedlist[@numeration='upperroman'])">
            <xsl:attribute name="property" select="'ktp:answer'"/>
            <xsl:attribute name="typeof" select="'ktp:Answer'"/>
            <xsl:processing-instruction name="no-correct-answer-given"/>
        </xsl:if>
        <xsl:apply-templates select="node()"/>
    </xhtml:li>
</xsl:template>

Note keep="true". These were inserted whenever converting to HTML and told xslTNG to not touch that particular node.

Other tweaks included generating cross-reference labels^[13], tweaking footnotes, adding table and equation caption labels and numbering, and adding @class attributes for the publishing CSS. One late-breaking step also replaced .eps image file suffixes with .png when the client changed the image format.

Cleanup and Split

XSLT pipelines tend to end with cleanup steps, from removing processing attributes to getting rid of unwanted namespace prefixes and so on. This one was no exception. However, once the XSLT pipeline, a matter of running XSLTs listed in a manifest file in sequence, was finished, it was still necessary to split the files, which added a separate XSLT file to the XProc itself.

Why not in the XSLT pipeline? The XProc step running the XSLT pipeline manifest is, for practical purposes, a black box that accepts an input folder of files and hands over a converted result folder, all as part of a single XProc step. If we wanted to do secondary output from that final XSLT, we would have to do much tweaking inside that black box. It was far easier to add an XSLT splitting the output of the XSLT pipeline to a result-document and iterating through that in XProc.

Notes on xslTNG

In hindsight, the final pipeline ended up converting more DocBook to HTML than planned; we had simply assumed that xslTNG would do most of the heavy lifting. Ari really wanted to test it in a live setting, having watched Norm's presentation at Balisage in 2020 [XSLT 3.0 on ordinary prose], but doing the entire conversion in the pipeline would have resulted in an easier-to-maintain transform.

There's also the not-at-all-insignificant fact that what we have here is DocBook 4.3, whereas xslTNG much prefers DB 5.0 and later. 4.3 still has INCLUDE/IGNORE sections for SGML, and the DocBook elements do not live in a namespace^[14].

As it stands, the calling XSLT does a few tweaks, from tweaking IDs to changing section numbering. It sets a few variables, and, above all, it makes sure that the @keep attributes are respected.

Final Tweaking

In the end, one additional XSLT, run after xslTNG, removed some final bits and pieces (DocBook attributes that xslTNG didn't seem to recognise, for one thing) and restored HTML wrappers that went missing in xslTNG, either because Ari missed some vital information in the excellent documentation [DocBook xslTNG Reference] or (more likely) the code leading to xslTNG.

XProc, As Ever

Ari's been an XProc convert since before 1.0 came out. The basic idea is fabulous; you black-box everything, connect the boxes, and out comes the result(s). The spec is somewhat less than the sum of it parts; it always gave the impression of knowing about the latest and the greatest while not quite being able to deliver it. You could do amazing things but only within strict confines and never without some pain.

It follows that for all that other processing — the pain — you were resigned to Ant processing or similar, and so it was with the subject at hand. Because of XProc 1.0's strict confines, Ant was a necessity.

This year — this month, actually — XProc 3.0 was finally released. The libraries that do the heavy lifting have all been converted to XProc 3.0 now (again, this month!), and had this project happened a few months later, it would all have been XProc 3.0. Maybe next year's talk.