Balisage Paper: XSpec for XProc, and XProc for XSLT

Introduction

XProc is a language that excels at performing operation sequences, known as pipelines, to process documents. A pipeline is composed of smaller steps, and a pipeline is itself a step that other pipelines can use. Active support for XProc version 3 includes two processors, a collection of resources at https://xproc.org/, a public mailing list, a series of virtual user group meetings, and a published book [S]. XProc 3 is well positioned for use in applications that have one or more of the following characteristics: large-scale, widely used, enduring, important. Software applications having one or more of those characteristics tend to be good candidates for automated testing and behavior-driven development techniques. And yet, in 2025, I had multiple, unrelated conversations with XProc users who brought up the shortage of tools for testing their XProc code.

XSpec version 4 narrows the testing gap for XProc 3. XSpec [XS] is an open-source software product for testing several languages in the XML space: XSLT, XQuery, Schematron, and now XProc. Support for testing XProc is new in XSpec 4.0, released in June 2026.^[1]

In this paper, I’ll show the XML vocabulary you use to write an XSpec test suite for a custom XProc step or pipeline. I’ll explain some non-obvious semantics related to execution of the test suite, as well as some limitations of the support for testing XProc steps. After that, I’ll dive into an implementation aspect that demonstrates the versatility of custom XProc steps beyond the typical chaining into pipelines. If you’ve found it useful that XProc can call XSLT and wished that it could work the other way, too, you’ll be pleased to know that it can.

Testing background

Imagine that you’ve created an XProc step with type my:count-words (for some namespace binding of the prefix my) that counts words in one or more documents, and you want to test this step. I’ll refer to it as a test target. At a high level, when you test the my:count-words step, you’re doing two things:

You decide what input documents or option values to use when running the my:count-words step. You also decide what aspects of the results are useful to verify. You express each verification in a way that accommodates a pass or fail status.

For instance, suppose you want to verify that exactly one document appears on the result output port of the step and that the document equals a particular one you expect. If exactly one document appears on the result output port but the content of the document doesn’t match your expectation, then the first verification passes and the second one fails.

When you create a set of specific instructions for the call and the verification, you have a test case. For any given test target, you might create multiple test cases, such as one test case that calls the step with one input document, and another that calls the step with two input documents.

Ingredients of an XSpec file

As a software product, XSpec has two main pieces: an XML vocabulary for writing a test file that expresses one or more test cases, and the software that can carry out the specific instructions in your test cases and report the pass/fail status of each verification. When you create an XSpec file, you’re using the vocabulary. When you run your test file to see which verifications passed or failed, you’re using the XSpec software (which in turn calls other software, such as Saxon).

At the highest structural levels of an XSpec test file, testing XProc code is analogous to testing code in the other languages that XSpec supports. For starters, the outermost element of an XSpec file is <x:description>, where the prefix x is bound to the XSpec namespace. The <x:description> element uses an attribute named xproc to point to the XProc step file or library file that contains (directly or through imports) your test target(s). The xproc attribute is analogous to attributes named stylesheet for testing XSLT, query-at for testing XQuery, and schematron for testing a Schematron schema.

If you have one test case, then within <x:description>, you use the <x:scenario> element to contain elements that express how you want to call the test target and what you want to verify. If you have multiple test cases, you use multiple <x:scenario> elements. To simplify explanations in this paper, we’ll assume the file has no nesting of scenarios, so multiple <x:scenario> elements are all siblings of each other.

In a given <x:scenario> element, you use a child <x:call> element to express how you want to call the test target. Following <x:call>, you use a child <x:expect> element for each verification you want to perform. To make test files and reports easier to understand, you also label each <x:scenario> and <x:expect> element with a text description. The label can be an attribute named label or a child element named <x:label>; for simplicity, we’ll use the attribute here.

The described structures look like Figure 1. If fully populated, it would be a test file with two test cases, each of which performs two verifications.

                        
<x:description xmlns:c="http://www.w3.org/ns/xproc-step"
  xmlns:my="http://example.com/syntax"
  xmlns:p="http://www.w3.org/ns/xproc"
  xmlns:x="http://www.jenitennison.com/xslt/xspec"
  xproc="count-words.xpl">

  <x:scenario label="With one document having 3 words">
    <x:call/>
    <x:expect label="there is exactly one output document"/>
    <x:expect label="an XML c:result document with content 3"/>
  </x:scenario>

  <x:scenario label="With two documents, requesting total count">
    <x:call/>
    <x:expect label="there is exactly one output document"/>
    <x:expect label="for the number of words in both documents"/>
  </x:scenario>

</x:description>

In Figure 1, the xproc attribute is the only thing specific to testing XProc. However, populating the <x:call> and <x:expect> elements with details about each test case reveals other ways in which the XSpec vocabulary is tailored for XProc.

Type of step to run

Any XProc step you test using XSpec must have a type attribute on the <p:declare-step> element. To configure the <x:call> element for your test target, you set the step attribute to the type as a qualified name (e.g., my:count-words). The step attribute is analogous to attributes named template and function for testing XSLT.

As usual in XML files, the XSpec file must declare any namespace prefixes that appear.

Input documents and option values

The <x:call> element often needs more configuration than merely the type of step to call. Your test case might require non-default documents on any of the step’s input ports, or it might require non-default values of any of the step’s options. The <x:call> element supports two kinds of child elements in tests for XProc only:

As for how you specify documents or values using these elements, you have several choices. Most of them are analogous to behaviors and markup patterns that XSpec supports across languages. You can embed custom XML content directly within <x:input> or <x:option>; point to an external XML file; provide an XPath expression that derives something from the embedded content or external XML file; or provide a standalone XPath expression that constructs a document or value.

In tests for XProc only, XSpec offers one additional choice, which is the ability to insert one or more XProc <p:document> elements as children of XSpec <x:input> or <x:option> elements. Using <p:document> is especially convenient for specifying multiple documents on an input port, specifying non-XML documents, or modifying the set of document properties of a document.

When you execute the test suite, these <p:document> elements get passed to the XProc processor for evaluation. By contrast, the other ways of specifying input documents and option values are evaluated in XSLT, for reasons I explain later, in the implementation-related portion of this paper.

Figure 2 continues the structural example in Figure 1, by populating the <x:call> elements with the step type, input documents, and an option value.

                        
<x:description xmlns:c="http://www.w3.org/ns/xproc-step"
  xmlns:my="http://example.com/syntax"
  xmlns:p="http://www.w3.org/ns/xproc"
  xmlns:x="http://www.jenitennison.com/xslt/xspec"
  xproc="count-words.xpl">

  <x:scenario label="With one document having 3 words">
    <x:call step="my:count-words">
      <x:input port="source" as="document-node(element(doc))">
        <doc>One two three</doc>
      </x:input>
    </x:call>
    <x:expect label="there is exactly one output document"/>
    <x:expect label="an XML c:result document with content 3"/>
  </x:scenario>

  <x:scenario label="With two documents, requesting total count">
    <x:call step="my:count-words">
      <x:input port="source">
        <p:document href="three-words.xml"/>
        <p:document href="two-words.xml"/>
      </x:input>
      <x:option name="total" select="true()"/>
    </x:call>
    <x:expect label="there is exactly one output document"/>
    <x:expect label="for the number of words in both documents"/>
  </x:scenario>

</x:description>

                     
<x:input port="source" as="document-node(element(doc))">
  <doc>One two three</doc>
</x:input>

Output documents and verifications

A fully configured <x:call> element accomplishes the first part of the two-part testing process, namely, running the test target step with your choice of input documents and/or option values. The second part is to use <x:expect> elements to verify the results of running the step. This element can appear in XSpec tests for any of the supported languages. It can check whether a boolean condition you specify evaluates to true. An example is a condition like exists($x:result//p) using a variable named x:result that XSpec predefines—or exists(//p) for short, if exactly one document appears on a particular output port. Alternatively, <x:expect> can check whether an actual result equals a corresponding expected result that you specify.

Longstanding XSpec features that also work in tests for XProc can help you implement use cases like verifying only a portion of a result and integrating your own filtering or normalization (canonicalization) behaviors into the verification process.

While being largely consistent across supported test target languages, the <x:expect> element also has a few ways in which it is tailored for testing XProc. First, <x:expect> has an optional port attribute whose value is the name of an output port of the step. This attribute says which output port you are interested in for that verification. If you set the port attribute, XSpec predefines its variable named x:result as the sequence of output documents on that port only. Specifying the port also makes certain other aspects of the <x:expect> element more convenient, as the next three subsections describe.

                           
<x:expect label="..." port="result">
  <a/>
  <b/>
</x:expect>

Fully populated XSpec file

We can now complete the syntax example. Figure 4 shows a fully populated XSpec file that calls a step with particular documents and option values and then verifies some results. (The XProc and XML files are not shown, as the purpose of the example is merely to illustrate XSpec syntax.)

                        
<x:description xmlns:c="http://www.w3.org/ns/xproc-step"
  xmlns:my="http://example.com/syntax"
  xmlns:p="http://www.w3.org/ns/xproc"
  xmlns:x="http://www.jenitennison.com/xslt/xspec"
  xproc="count-words.xpl">

  <x:scenario label="With one document having 3 words">
    <x:call step="my:count-words">
      <x:input port="source" as="document-node(element(doc))">
        <doc>One two three</doc>
      </x:input>
    </x:call>
    <x:expect label="there is exactly one output document"
      port="result" test="count($x:result)" select="1"/>
    <x:expect label="an XML c:result document with content 3"
      port="result">
      <c:result>3</c:result>
    </x:expect>
  </x:scenario>

  <x:scenario label="With two documents, requesting total count">
    <x:call step="my:count-words">
      <x:input port="source">
        <p:document href="three-words.xml"/>
        <p:document href="two-words.xml"/>
      </x:input>
      <x:option name="total" select="true()"/>
    </x:call>
    <x:expect label="there is exactly one output document"
      port="result" test="count($x:result)" select="1"/>
    <x:expect label="for the number of words in both documents"
      port="result" select="5"/>
  </x:scenario>

</x:description>

Running this test suite produces an HTML report that looks like Figure 5.

Figure 5: HTML report of test results

Familiar XSpec features

The preceding descriptions of testing XProc steps have pointed out some ways in which the XSpec vocabulary expresses XProc-specific information, as well as some ways in which the XSpec semantics align with XProc handling of data and documents. At the same time, many features of XSpec work the same way in tests for either XProc or other languages that XSpec supports. Here is a non-comprehensive list of such language-agnostic features:

Limitations

You should be aware of the following limitations of XSpec v4.0 support for testing XProc:

Some XProc steps present testability challenges even though XSpec can call the steps. Testing code that has side effects (that is, the code modifies its environment) is tricky. For instance, suppose your step invokes p:file-copy, and you want to verify existence of the copied file. Maybe the copy existed even before your step ran. Maybe calling your step to copy the file interferes with a subsequent test case. XSpec doesn’t give you a way to set up the environment before a test case executes, clean up afterward, or enforce an order of operations; this is an XSpec limitation across languages, not specific to testing XProc.

This concludes the part of this paper about user-facing aspects of XSpec support for testing XProc. Readers not interested in the implementation may skip to the conclusion.

Implementation: XSLT-based test runner

Now that we’ve glimpsed how to write an XSpec test suite for XProc, let’s look more at what happens when you execute the test suite. Somehow, an XProc processor has to invoke your test target with the input documents and option values that each test case specifies.

A precedent worth considering is how XSpec executes a test suite for XQuery. An XSLT stylesheet known as the XSpec compiler generates an XQuery main module known as the test runner. The test runner is what calls the XQuery code being tested and determines which verifications pass or fail.

The XSpec repository’s associated wiki [XW] contains a UML diagram that shows how XSpec works internally, and it appears with minor labeling modifications in Figure 6. In this diagram, the XSpec compiler is the box near the top that says, compile-xslt-tests.xsl (or compile-xquery-tests.xsl). The test runner is the box to which arrow 5 points.

Figure 6: XSpec processing

One logical way to extend XSpec for testing XProc would be to make the XSpec compiler generate an XProc pipeline as the test runner, instead of generating a stylesheet or query. After all, an XProc pipeline is capable of calling the XProc step being tested. XProc is also capable of determining which verifications pass or fail.

Although generating an XProc-based test runner would be a logical implementation path, I chose to do it differently. Before explaining that, I need to share some XProc processor background.

Calling XProc steps as XPath functions

The support for testing XProc in XSpec v4 uses step functions for two distinct tasks that only an XProc processor can accomplish:

Let’s look at these in order. The first case is simpler and helps explain the paradigm, while the second case is more central to the implementation.

Running Tests...
Testing with SAXON HE 12.9 and XML Calabash 3.0.42

                           
<p:declare-step type="pf:system-property" name="system-property">
  <p:output port="result"/>
  <p:option name="property" required="true" as="xs:string"/>
  <p:identity>
    <p:with-input select="string()">
      <p:inline>{ p:system-property($property) }</p:inline>
    </p:with-input>
  </p:identity>
</p:declare-step>

                           
<xsl:value-of select="
  Q{http://www.jenitennison.com/xslt/xspec/xproc/steps/wrap-standard-functions}system-property(
    map{'property': 'Q{http://www.w3.org/ns/xproc}product-version'}
  )?result"/>

                              
<x:scenario label="Call a step">
  <x:call step="my-csv:test-target-step">
    <x:input port="source">
      <p:document href="document.csv"/>
    </x:input>
  </x:call>
  <x:expect label="Verify content" port="result">1,2</x:expect>
  <x:expect label="Verify content type" port="result"
    test="$x:document-properties?(QName('', 'content-type'))"
    select="'text/csv'"/>
</x:scenario>

                              
<p:declare-step xmlns:impl="urn:x-xspec:compile:impl"
  xmlns:p="http://www.w3.org/ns/xproc"
  type="impl:step-runner" name="step-runner" version="3.1">

  <p:input port="step-to-call-test-target" content-types="application/xml"/>
  <p:output port="map-of-outputs" pipe="map-of-outputs@call-test-target"
    content-types="application/json"/>
  <p:option name="map-of-inputs" as="map(*)"/>
  <p:option name="map-of-options" as="map(*)"/>

  <p:run name="call-test-target">
    <p:with-input pipe="step-to-call-test-target@step-runner"/>
    <p:run-option name="map-of-inputs" select="$map-of-inputs"/>
    <p:run-option name="map-of-options" select="$map-of-options"/>
    <p:output port="map-of-outputs" primary="true"/>
  </p:run>
</p:declare-step>

                              
<my-csv:test-target-step xmlns:my-csv="http://example.com/csv"
  name="test-target">
  <p:with-input port="source">
    <p:document
      xml:base="file:/C:/.../example/csv-example.xspec"
      href="document.csv"/>
  </p:with-input>
</my-csv:test-target-step>

                              
<p:identity>
  <p:with-input
    select="map{ 'result':
      map{ 'document': ., 'document-properties': p:document-properties(.) }
    }"
    pipe="result@test-target"/>
</p:identity>

                              
<p:json-merge duplicates="combine" name="merged">
  <p:with-input port="source">
    <p:pipe step="map_result"/>
    <p:pipe step="map_secondary"/>
  </p:with-input>
</p:json-merge>

Conclusion

If you believe that the future of markup includes the XProc language and you want to bolster the future of your own XProc code, then it’s wise to invest in readable, runnable descriptions of how your code behaves. I’m delighted that XSpec now extends its capabilities for automated testing and behavior-driven development into the XProc realm. When designing user-facing aspects of XSpec for XProc, I wanted XProc users to find it sufficiently tailored for XProc, and I also wanted existing XSpec users to find it familiar. The tailored-for-XProc features include dedicated XSpec elements to represent input port documents and option values; the ability to use the XProc <p:document> element within your test suite for full control over input data; access to document properties; and verifications that optionally focus on one output port at a time. XSpec for testing XProc still has an XSpec feel, though, because the way you do things like organize and label test cases, express what you want to verify, and examine your test results is consistent across supported languages.

We saw that the extension of XSpec for testing XProc depends on the ability of an XSLT processor and an XProc processor to call an XProc step as if it were an XPath function. This capability shortened my development time for this XSpec enhancement and, more critically, motivated me to attempt it in the first place. In my opinion, this step function feature is worth knowing about so that you can apply it in situations where the feature fits well. Such situations might even include an XSpec test suite, where <x:helper> integrates your custom XProc step into the testing process (e.g., for normalization, whitespace handling, or filtering) and you then call the step as a step function. More generally, an XProc step that wears an XPath hat can gain entry to—and exhibit its special XProc behaviors in—XPath expressions that reside within XProc or other languages that use XPath as an expression language.