Balisage Paper: Beyond valid and invalid: implementing complex validation in XProc

History

Bloomsbury Academic began digital publishing in around 2013, later becoming Bloomsbury Digital Resources (BDR).

There was no in-house validation expertise at this time, but the need for a robust set of business rules encapsulated in Schematron schemas was soon recognised.

Likewise, the realisation that an in-house quality assurance team was needed, since not everything could be reliably or definitively validated by machine with the available technology and knowhow. The "QA team" sits within Content Architecture.

A “validation portal” was also set up to carry out delivery package integrity checks, and schema and Schematron validation on receipt from suppliers.

Validation portal 1.0

The first iteration of the validation portal was a bespoke web application developed, hosted and supported by one of our typesetting vendors.

Figure 1: Simplified validation portal workflow

It provided in-built issue tracking for suppliers to file issues for investigation by the CA team, and both documentation and schemas could be updated by us in a self-service manner.

The non-schema validation portions were, however, opaque and not under our control: we neither owned nor had visibility of the source code.

The gratis provision of this service meant that there was also a unreliable lead time for any changes. Support was variable in general, with downtime, ad hoc changes to infrastructure and functional regression all proving disruptive.

Reports resulting from schema validation were purely text-based (i.e., no SVRL^[1] produced by Schematron validation) and a log of only the most recent 200 or so submissions was retained, meaning supplier performance was hard to track over time and analyse for areas for improvement in their work.

Why MorganaXProc-III?

We need to do some work with binaries, e.g., inspecting PDFs for corruption; images for colour space, resolution, etc. For these, we use Python scripts built as executables. We authored a prototype pipeline and put Morgana and Calabash head-to-head, with a Morgana evaluation licence (Calabash supports the p:os-exec step, which we’d need for running our executables, but Morgana’s commercially-licensed “Extended Edition” (MorganaXProc-IIIee) is required for that).

We were impressed by the performance (including its startup time when compared to that of XML Calabash), prompt support, licensing model and pricing.

Once we had established with Morgana’s developer and co-editor of the XProc standard Achim Berndzen that we would be operating within the terms of the EULA, the spend was approved and we were good to go.

One notable difference encountered was that, where our validation logic threw deliberate errors (using p:error), Calabash required these to be explicitly handled (with p:try/p:catch), whereas Morgana did not.

Simplified reports

A BaseX database in the back end stores a digested form of each validation report, fulfilling our desire to monitor supplier performance.

SVRL is verbose, and can run very large indeed, especially for large documents with many failing tests. We recognised that we only need a count of the distinct issue IDs and an example of each, so for efficiency the final output is condensed to e.g.:

<report
    uri="file:///C:/Users/asales/git/content-architecture/xproc/validation-portal/test/warning-scores/suppress-pdf-required/fail/9781350002876_txt_xml.zip"
    timestamp="2026-02-13T15:39:05.4687649Z" warnings="208" distinctIds="41"
    schematronVersion="[schematron-version-info] schematronVersion = 11"
    workTitle="[title-value] Book title = &#34;Shakespeare and Fun&#34;"
    valid="false">
    <warning id="bib-title-end-quote" count="27">
        <svrl:successful-report test="ends-with(., '’') or ends-with(., '”')"
            id="bib-title-end-quote" role="warning"
            location="/*:book[namespace-uri()='http://docbook.org/ns/docbook'][1]/*:part[namespace-uri()='http://docbook.org/ns/docbook'][8]/*:chapter[namespace-uri()='http://docbook.org/ns/docbook'][3]/*:bibliolist[namespace-uri()='http://docbook.org/ns/docbook'][1]/*:bibliomixed[namespace-uri()='http://docbook.org/ns/docbook'][32]/*:bibliomset[namespace-uri()='http://docbook.org/ns/docbook'][1]/*:title[namespace-uri()='http://docbook.org/ns/docbook'][1]">
            <svrl:text>[bib-title-end-quote] title in biblio item ends with single or double quote;
                should it be outside the title? check within "bibliomset" element with id
                "b-9781350002876-2101"</svrl:text>
        </svrl:successful-report>
    </warning>
    <warning id="bib-title-start-quote" count="25">
        <svrl:successful-report test="starts-with(., '‘') or starts-with(., '“')"
            id="bib-title-start-quote" role="warning"
        [...]

since we are mainly concerned with the overall breakdown of issues, rather than the detail of each. In most cases, this reduces the SVRL file size from megabytes to kilobytes. Note that the portal does also record who made the submission (not shown above).

Documaintenance

That list of documentation assets is just a filtered view of a local clone of our GitHub repository, so we decided for ease of maintenance to use an XProc pipeline to generate it (statically; as c:directory, etc.). Again, this means we have complete control over what is included and no software changes are required to effect a change:

                     
<p:declare-step xmlns:p="http://www.w3.org/ns/xproc"
    xmlns:c="http://www.w3.org/ns/xproc-step" version="3.0"
    xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <p:input port="source" sequence="true"/>    <!-- because there is no input -->
    <p:output port="result"/>
    
    <p:option name="git-path" select="'C:/Users/asales/git/'"/>
    
    <p:variable name="schematron-base" select="'file:/' || $git-path || 'content-architecture/content-models/DocBook/Schematron/'"/>
    
    <p:variable name="schema-base" select="'file:/' || $git-path || 'content-architecture/content-models/DocBook/schema/'"/>
    
    <p:variable name="docs-base" select="'file:/' || $git-path || 'content-architecture/content-models/DocBook/doc/'"/>
    
    <p:variable name="schematron-tei-base" select="'file:/' || $git-path || 'content-architecture/content-models/TEI/Schematron/'"/>
    
    <p:variable name="schema-tei-base" select="'file:/' || $git-path || 'content-architecture/content-models/TEI/schema/'"/>
    
    <p:variable name="docs-tei-base" select="'file:/' || $git-path || 'content-architecture/content-models/TEI/doc/Data%20model%20XML/'"/>
    
    <p:directory-list path="{$docs-tei-base}"
        include-filter='^Bloomsbury%20TEI%20XML%20-%20plays%20and%20screenplays%20v.+\.pdf$' detailed='true'/>
    
    <!-- filter out all but the latest (by modification date): some XPath repetition here 
    because p:delete/@match can't contain variable references -->
    <p:delete match="(//c:file[xs:dateTime(@last-modified) lt max(//c:file/xs:dateTime(@last-modified))])"/>
    
    <p:variable name="docs-tei" select="."/>
    
    <p:directory-list path="{$schematron-tei-base}"
        include-filter='^bmyTEI\.sch$' detailed='true'/>
    
    <p:variable name="schematron-tei" select="."/>
    
    <p:directory-list path="{$schema-tei-base}" detailed='true'>
        <p:with-option name="include-filter" select="('^bmyTEI\.rnc$', '^bmyTEI_BASE\.rnc$')"/>
    </p:directory-list>
    
    <p:variable name="schema-tei" select="."/>
    
    <p:directory-list path="{$docs-base}"
        include-filter='^Bloomsbury%20DocBook%20XML%20Data%20Specification%20v.+\.pdf$' detailed='true'/>
    
    <!-- filter out all but the latest (by modification date): some XPath repetition here 
    because p:delete can't contain variable references -->
    <p:delete match="(//c:file[xs:dateTime(@last-modified) lt max(//c:file/xs:dateTime(@last-modified))])"/>
    
    <p:variable name="docs" select="."/>
    
    <p:directory-list path="{$schematron-base}"
        include-filter='^docbook-mods\.sch$' detailed='true'/>
    
    <p:variable name="schematron" select="."/>
    
    <p:directory-list path="{$schema-base}" detailed='true'>
        <p:with-option name="include-filter" select="('^bloomsbury-mods\.rnc$', '^publishers-51cr\.rnc$')"/>
    </p:directory-list>
    
    <p:wrap-sequence wrapper="repo">
        <p:with-input select="., $schematron, $docs, $schema-tei, $schematron-tei, $docs-tei"/>
    </p:wrap-sequence>
    
</p:declare-step>

On upload to the portal, a lambda simply executes the validation pipeline on the zip containing XML and supporting assets (images, PDFs, etc.). This encapsulates the core validation logic, so where we’ll spend a bit of time.

Schemalets (or epischemas)

Although we have our main schema documentation, some content sets require special treatment which may be a one-off, so we typically avoid bloating the schema or documentation with those instructions and constraints.

Instead, we issue additional “conversion instructions” in such cases and historically have applied any programmatic checks in-house, using small, standalone Schematron schemas particular to the case in hand. (We have christened these “schemalets”, as they tend to be small-scale offshoots of the main schema they supplement.^[3])

We would prefer to have these applied at point of receipt, which was not realistically feasible using the legacy portal. So the method we are pursuing is to specify an appropriate <?xml-model?> and have the pipeline parse that to determine which schemalet(s) to use.

This means we can now apply arbitrary schemalets as needed – provided of course that the XML supplier includes the correct <?xml-model?> in their delivery.

Scoring the warnings

Our Schematron schemas distinguish between two basic levels of severity deserving attention: errors and warnings.^[4]

The acceptance criteria are currently that the presence of any errors means the XML is invalid, therefore rejected; but any number of warnings can be present.

However – and this is probably the case more particularly with human-authored, discursive content - some warnings are more important than others. Divining this is a significant part of what the QA team is engaged in.

With the new freedoms afforded to us by the portal 2.0, we resolved to apply a “scoring” algorithm to these warnings, to see which we could instead reject out of hand at submission-time.^[5]

This meant collating information from the team about what would constitute a request for re-work from the supplier, in effect the formal expression of these implicit tolerances.

The approach is to post-process the Schematron validation report. For a given constraint ID, we can specify two parameters: a threshold, or absolute number we can accept of a certain warning; or a proportion, i.e., for what percentage of the occurrences of a particular structure the warning was reported.

The pipeline logic is that a Schematron SVRL document containing warnings, but no errors, is filtered by an XSLT stylesheet, whose main processing template is:

                     
<xsl:template match="/">
    <xsl:variable name="warnings" select="(//svrl:failed-assert|//svrl:successful-report)[@role = ('warn', 'warning')]"/>
    
    <xsl:variable name="grouped-warnings">
        <xsl:for-each-group select="$warnings" group-by="@id">
            <warning id='{current-grouping-key()}' count='{count(current-group())}'>
                <xsl:copy-of select="current-group()[1]"/>
            </warning>
        </xsl:for-each-group>
    </xsl:variable>
    
    <xsl:variable name="scored-warnings">
        <xsl:apply-templates select="$grouped-warnings" mode="score"/>
    </xsl:variable>        
    
    <report uri='{$zip-uri}' timestamp='{current-dateTime()}' warnings='{count($warnings)}' distinctIds='{count($grouped-warnings/warning)}'
        schematronVersion='{$schematron-version}' workTitle='{$work-title}'>
        <xsl:if test="exists($scored-warnings/warning/@invalid)">
            <xsl:attribute name="valid" select="'false'"/>
        </xsl:if>
        <xsl:apply-templates select="$scored-warnings/warning">
            <xsl:sort select="@count" order="descending" data-type="number"/>
        </xsl:apply-templates>
    </report> 
</xsl:template>

where the warnings are decorated with an invalid attribute if the threshold or proportion configured is reached:

<xsl:template match="warning" mode="score">
    <xsl:copy>
        <xsl:copy-of select="@*"/>
        <xsl:sequence select="(
            bmy:threshold-exceeded(@id, @count), 
            bmy:proportion-exceeded(@id, @count, $node-count-by-id?(@id))
        )[1]"/>
        <xsl:copy-of select="*"/>
    </xsl:copy>
</xsl:template>

The function threshold-exceeded() is simple enough:

<xsl:function name="bmy:threshold-exceeded" as="attribute(invalid)?">
    <xsl:param name="id" as="attribute(id)"/>
    <xsl:param name="count" as="attribute(count)"/>
    <xsl:if test="$count > $warning-thresholds?($id)">
        <xsl:attribute name="invalid" select="'thresholdExceeded'"/>
    </xsl:if>
</xsl:function>

while proportion-exceeded() calculates whether the percentage has been exceeded:

<xsl:function name="bmy:proportion-exceeded" as="attribute(invalid)?">
    <xsl:param name="id" as="attribute(id)"/>
    <xsl:param name="warningCount" as="attribute(count)"/>
    <xsl:param name="nodeCount" as="xs:integer?"/>
    <xsl:if test="($warningCount div $nodeCount) ge ($warning-percentages?($id) div 100)">
        <xsl:attribute name="invalid" select="'proportionExceeded'"/>
    </xsl:if>
</xsl:function>

If any warning is marked as invalid in this way, <report valid='false'>...</report> is output, with the upshot that the portal rejects the delivery.

It will be apparent from the snippets above that the configuration of these value limits is currently done using maps. For extensibility and ease of maintenance, we plan to move this into an XML configuration document instead, e.g.:

<config>
    <thresholds>
        <warning id='suppress-pdf-required'>5</warning>
    </thresholds>
    <proportions>
        <warning id='figure-in-label'>100</warning>
        <warning id='table-in-label'>100</warning>
    </proportions>
</config>

where each id corresponds to a Schematron warning ID.

We fully appreciate that greater complexity of algorithms is possible and do plan to exploit this as needed. For now, we need this approach to prove its worth, then we can expand on it.

The other major consideration for us is that having a portal whose validation criteria are stricter than necessary appeals in principle, but when faced with the reality of a fast-paced production environment, the risk to the business of potentially slowing down or blocking supply unnecessarily is often too great.

ONIX metadata cross-checks

While we process a reasonable amount of licensed and acquired content, the majority on balance is indigenous – commissioned by our in-house editors.

For this, our bibliographic metadata system (Biblio) produces ONIX [ONIX], which is syndicated to various recipients, including content aggregators and discovery providers.

An ongoing issue for us has been discrepancies between the metadata held in that system (i.e., as expressed in ONIX) and its equivalents contained in the XML representing the content.

This can arise because of late-breaking changes while the title is in production are not subsequently reflected in what is recorded in Biblio. For example, a work’s subtitle might change after it has been typeset, perhaps because of space constraints on the page.

The intention (not implemented as of this writing) is to take a regular feed of ONIX XML for everything in production. These are very large (sometimes multi-gigabyte) documents, but fortunately record-based XML, so the feed can easily be decomposed into small, individual ONIX Products, then stored and queried efficiently.^[6]

This means for a given ISBN, we can have the validation pipeline query a database of ONIX Products and compare key items of metadata against the XML as captured, flagging or rejecting any mismatch.

This approach also leads the way to potential automated reconciliation: results of validation can be fed back into Biblio, most probably via a human-managed gateway interface.

Testing the pipeline

Testing XProc 3.0 pipelines appears to be a nascent area of the technology, so we were pleased to learn while preparing this paper that support in XSpec for this is being added [XSpec 3.4 highlights].

In the interim, we use a unit-testing approach supported by BaseX's unit module.

Each test is represented by a function with the BaseX-specific XQuery annotation %unit:test. The tests are organised to enable the entire suite to be run at once. There is one passing and one failing unit test for each of the conditions the pipeline can present. For example, this pair of tests relate to the same condition, where media assets not referenced by the XML are present in the delivered package:

                     
declare %unit:test function _:extraneous-media-fail()
{
  unit:assert(
    _:run-pipeline('extraneous-media/fail/9781350996526_txt_xml.zip')/c:errors/c:error[@code eq 'extraneous-media']
  )
};

(:~ clean up extracted files :)
declare %unit:after('extraneous-media-fail') function _:extraneous-media-fail-cleanup()
{
  file:delete(resolve-uri('extraneous-media/fail/9781350996526/', static-base-uri()), true())
};

declare %unit:test function _:extraneous-media-pass()
{
  unit:assert-equals(
    _:run-pipeline('extraneous-media/pass/9781350996526_txt_xml.zip')/c:errors/c:error[@code eq 'extraneous-media'],
    ()
  )
};

(:~ clean up extracted files :)
declare %unit:after('extraneous-media-pass') function _:extraneous-media-pass-cleanup()
{
  file:delete(resolve-uri('extraneous-media/pass/9781350996526/', static-base-uri()), true())
};

There is a performance overhead with this approach, in that we start a new instance of Morgana for every test, invoked via the BaseX function proc:execute, but an XML document representing the result of that invocation is at least returned for inspection.^[7]

We do have some test cases that address individual steps in isolation (enabled, for example, by the appropriate Morgana command line options), but this is less relevant for us as we mostly use built-in steps in the pipeline as currently written. This is where our fail-early philosophy presents a slight disadvantage, because if we want to test in a way that resembles real-world scenarios closely, we have the overhead of maintaining test cases that are valid up to a certain point in the pipeline.

For most tests that is fine, as the criteria are quite stable; but our Schematron schemas do change relatively often, which can mean a pipeline test suddenly fails as a consequence. It would be onerous for us to maintain complete, shared test case documents, so the tactic we have employed here amounts to "mocking", whereby a step option accepts suitable input to produce the desired behaviour. In this unit test, setting the $use-mocking option to true() has the effect of making the step output the input document, just as would be the case if validation succeeded:

declare %unit:test function _:schema-validation-pass()
{
  let $result := _:run-library-step(
    'schema-validation/pass/9780567706652_txt_xml.xml',
      $_:STEPS_LIBRARY_PATH, 
      'Q{http://www.bloomsbury.com/xproc}bmy:validate-rnc-sch',
      map{},
      map{
        'relax-ng-uri':'../../content-models/DocBook/schema/bloomsbury-mods.rnc',
        'schematron-uri':'../../content-models/DocBook/schematron/docbook-mods.sch',
        'use-mocking':true()
      }
  )
  let $output := $result/output=>parse-xml()
  return
  unit:assert-equals($output/c:errors, ())
};

At first this can feel wrong, since we effectively bypass the test, or at least produce an artificial result; but we have reasoned that this is on balance tolerable since a) the main purpose of this test suite is to test the pipeline's behaviour, rather than that of the schemas it references; and b) we already have robust test suites in place elsewhere to test that the schemas themselves are fit for purpose.

Extensibility

Not long after development of the new portal was completed, a new use case for a content type arose. We had handled this content before but on such a small scale that it didn’t warrant building into the portal. Now, however, the scale of the incoming content is ambitious, so we feel vindicated in that we can now easily incorporate a new content type by some relatively self-contained changes to the pipeline.

The changes will amount to adding some minor decision logic to respond to the new markup to select the correct schema, and re-generating the filtered view of our cloned GitHub repository, to make the added validation assets available to users on the portal front end.