Role of Unit Testing in a Web Application
Suppose you are creating a web application that implements a significant feature set in its initial release, and suppose your design team anticipates growing and improving the features in successive releases. You want to build quality in from the start. Looking ahead, you want to be able to refactor code fearlessly as the design evolves. Unit testing can help you accomplish those objectives. This paper discusses ways to resolve incompatibilities between a particular testing tool and a particular web application development tool.
What Is Saxon-JS?
What Is XSpec?
XSpec is an open-source software product for testing XSLT, XQuery, and Schematron code. An XSpec test suite is an XML document that conforms to the XSpec schema. When writing an XSpec test suite, you set up test scenarios (or "tests") and express the results you expect. When you run an XSpec test suite, the software evaluates the test scenarios, gathers results, and generates a report that compares the actual results to the expected ones. You can view the report as HTML in a browser, or you can programmatically operate on the same information in textual or XML format. An XSpec test suite can include multiple test scenarios, each of which can make multiple assertions about expected results.
Here is a bit more detail about what happens when you run an XSpec test suite to test
XSLT code. First, the XSpec infrastructure uses XSLT to compile your XSpec test suite
new XSLT transform. This compiled test suite incorporates all the logic from your
and also uses
<xsl:import> to import the XSLT you are testing. Next, the
compiled test suite runs and produces an XML report that is further transformed into
report. Execution of the compiled test suite uses Saxon on the Java platform,
The XSpec package also includes functionality for computing and reporting on code coverage.
Motivation: Areas of Incompatibility Between XSpec and Saxon-JS
Case Study for This Paper
While we expect the techniques and commentary in this paper to have broader applicability, we developed them while working on a specific Saxon-JS (version 1.2) web application at a software company. The application displays lists of software language items (functions, methods, and so on) in a suite of approximately 100 products. Each item in the list shows the hyperlinked name of a function or other language item next to a one-line summary. The application enables the end user to:
Vary the scope of the list: across all products, within one product, or in a user-selectable category within one product.
Switch the display between alphabetical and categorized views.
Filter the list to show only language items that support certain data types or extended capabilities.
Switch among functions and other kinds of language items while maintaining prior choices, such as a previously selected product or category.
Any choices that the user can make interactively are also reflected in the URL query string, making the choices easy to access programmatically during testing.
In this application, selecting the correct data to display based on those choices is critical. The graphical controls are important, too, but animation is neither flashy nor the primary focus of the application. These aspects of the application's nature influenced the unit-testing strategy, as later sections describe.
The next screen capture shows a sample in which the application renders a categorized view of a subset of functions within one product [M].
Types of Incompatibilities
During development of the web application, we soon discovered that certain key traits of a Saxon-JS web application cause XSpec to issue error messages. For example:
A recommended programming paradigm in Saxon-JS web applications uses the
<xsl:result-document>instruction to replace or augment the content of an element in the HTML page. The content of the
<xsl:result-document>instruction is evaluated, and the result is inserted into the content of the element in question [S2]. This paradigm poses two challenges for XSpec. First, the XSpec implementation uses
<xsl:result-document>to capture the test results, and
<xsl:result-document>instructions cannot nest. Using
<xsl:result-document>in the XSLT code being tested causes XSpec to issue an error message. Second, XSpec does not have access to the HTML page that the result is supposed to be inserted into. As a result, XSpec cannot act on the
<xsl:result-document>instruction in the manner that Saxon-JS does within the browser.
The interactive XSLT extension functions in Saxon-JS are inaccessible to XSpec. This leads to various problems. For instance, our web application needs to access parameters in the URL query string. Saxon-JS provides the
ixsl:query-paramsfunction for this purpose, as part of the interactive XSLT suite of extensions. However, XSpec has no access to the query string, URL, or the
Like the extension functions, extension instructions are unknown to XSpec. This, too, leads to problems. Our application populates sections of the page incrementally as their content is ready, as opposed to taking the time to compute all the content before starting to render it. Saxon-JS provides the
<ixsl:schedule-action>extension instruction for this purpose. Lacking this instruction, XSpec is unable to schedule actions for later execution and does not even understand the request.
Having successfully used XSpec for testing other parts of our XSLT code base, we wanted to have unit testing for the web application, too. The rest of this paper describes the approach we used for testing this Saxon-JS application using XSpec, an alternative approach that we explored but did not use, and the reasons for our choice.
Primary Approach: Substitute as Needed
In software testing parlance, the term mocking describes a technique in which you provide substitutes for functionality or data that you cannot or prefer not to use in a testing process. For example, suppose you are testing part of an e-commerce application that performs transactions, and you want to check that a customer survey appears after the transaction is complete. You cannot make real purchases in the test environment. Instead, you might substitute a fake account, perform the transaction there, and check that the survey appears. Going one step further from the real situation, you might substitute a fake transaction handler that does not even attempt to access an account but merely sends back a reply indicating what the real handler would have done. This reply might be good enough, because the purpose of this test is to check that the survey appears, not to manipulate a real or fake account. How closely the mock behavior should imitate reality is a choice you make, depending on what you want to accomplish.
We substituted mock behaviors for the behaviors our actual web application performed that XSpec could not perform: creating result documents to change the HTML content of the page, calling interactive XSLT extension functions, and executing interactive XSLT extension instructions. What we were able to test using XSpec included logic and data processing operations. These operations were the core of our application, so we were satisfied that XSpec was applicable there.
High-Level File Structure to Enable Substitution
XSLT import precedence provides a straightforward architecture for many of the substitutions we needed. Instead of making the XSpec file directly test the production XSLT transform, we used the following set of files:
Production XSLT transform, used in the live application.
Test utility file, an XSLT file containing the templates and functions for XSpec to use when substituting for Saxon-JS behaviors. This file can potentially be shared by multiple Saxon-JS applications, if their testing requirements are similar enough.
Test harness, an XSLT file that uses an
<xsl:include>element to include the test utility file and an
<xsl:import>element to import the production XSLT transform.
XSpec file, where the
stylesheetattribute on the top-level element points to the test harness file, not the production XSLT transform. This attribute value and the use of an
<xsl:import>element in the test harness ensure that XSpec uses the templates and functions in the test utility file, not their counterparts in the production XSLT transform.
Alternatively, an XSpec element named
<x:helper>, implemented but not
yet released as of this writing, will enable you to achieve the same result without
requiring a test harness as a separate file.
Case 1: Creating Result Documents
As stated earlier, Saxon-JS applications commonly use
<xsl:result-document> instructions to replace or augment the content of an
element in the HTML page, but XSpec has no HTML page at its disposal. We created a
template that, in production, does nothing but execute
instructions. The production XSLT transform consistently calls this named template
<xsl:result-document> instructions directly. The purpose of
this named template is to provide a level of indirection that enables substitution
Our chosen substitute behavior is to emit a processing instruction followed by the
content that was
passed in. The processing instruction describes what would happen in production. The
next code excerpt
shows the test utility file's code for the
During XSpec testing, any XSLT code that calls the
template executes the testing version. This accomplishes two things: it avoids an
error and it provides information that XSpec can verify. The following two
<x:expect> elements illustrate how XSpec can confirm our expectations
about content that the production XSLT transform inserts into the HTML page.
<x:expect> elements do not directly check that the HTML
<h1> element in the web application gets populated with the correct text.
Instead, they check for indirect indications that the web application behaves as expected.
Designing indirect indications that give you confidence about the production behavior
part of how you apply mocking techniques.
Still, it is possible to have defects in the
<h1> element that these
XSpec assertions would not detect. For example, if the HTML page had no element with
id="listpage_h1" due to a bug, Saxon-JS would not find the spot on the page
to put the text content. XSpec would report that the XSLT code exhibited the expected
behavior, but the web application would still have a defect. Just because a defect
undetected in a unit test, that does not mean the test is not worthwhile. When you
your testing strategy, factors you typically consider include:
How far you can get with unit-level tests that you can run frequently as you change the code.
The risk of gaps due to substitute behaviors.
The options for closing those gaps, such as interactive or system-level testing that exercises the web application in a browser, even if that level of testing is less convenient to perform frequently during code development.
Case 2: Calling an Extension Function
The test utility file contains substitute definitions of these interactive XSLT
functions that we use in production:
ixsl:window. Each substitute definition
has the same signature as the original function, documented at [S3]. We
added each of these substitute definitions to the test utility file as the need
Our substitute behaviors for Interactive XSLT extension functions have these goals:
Avoid an XSpec error.
(Optionally) Report that the function was called.
(Optionally) Return a mock value to use for verification.
For each goal, we offer an example showing where the goal applied and how we used code in the test utility file and XSpec file to accomplish the goal.
Goal 1: Avoid Error
For some extension functions, avoiding an XSpec error is sufficient. For example,
production XSLT transform calls the
ixsl:location function only once, in the
definition of a global variable. The XSpec file overrides the global variable, which
that XSpec does not actually call
ixsl:location (either the Saxonica function
or our substitute version in the test utility file) during testing. In this case,
presence of a substitute function prevents a static error during
testing. The content of the substitute function is irrelevant. Here
is the substitute function's definition:
An alternate approach to error avoidance involves the XSLT 3.0 instructions
<xsl:catch>. In the next example, the
production behavior changes the browser's URL history. We decided not to report or
mimic the URL change in the XSpec environment. The try/catch structure is enough to
a barrier to XSpec testing of the ancestor template.
Another alternate approach is to set a
use-when attribute whose value is
a static parameter. The parameter's value is defined in the production XSLT transfor
true and defined in the test utility file as false.
Goal 2: Report on Function Invocation
For some extension functions, we want XSpec to be able to confirm that the function
was called. For example, in exactly one spot, our production XSLT transform uses the
scenario to check that
ixsl:apply was called, but with only one instance, it
Besides, testing the specific result is a task for browser-based testing because only
following function definition:
This function definition enables an XSpec
<x:expect> element, such as
the following, to verify the presence of the processing instruction. In this case,
the processing instruction happens to
be in the last item in the sequence of XSLT results that XSpec stores in its special
Goal 3: Return Mock Result for Verification
For some extension functions, we want XSpec to obtain a mock return value that we
use to verify correct behavior. For example, in several spots, our production XSLT
transform uses the
ixsl:page function at the start of an XPath expression. In
Saxon-JS, this function returns the document node of the HTML DOM document that the
browser is displaying. Starting an XPath expression from the returned document node
you navigate the HTML tree to query or modify elements.
ixsl:page function is such an important part of our
application, it is not enough to know that it was called. We want our unit test to
to verify what happened next — for instance, whether subsequent code modified the
element. For example, the following XSLT code excerpt uses the
function to identify a particular radio button newly inserted into the page. The code
checked attribute of the radio button. (For more about the template named
mt:set-attribute, see section “Case 3: Executing an Extension Instruction”.)
The XSpec environment has no access to an HTML DOM document. Instead, we need a
ixsl:page function that operates on a substitute document and
returns its document node. Our solution is for the test utility file to define a global
html_el, whose override in the XSpec file captures the relevant
parts of the web application's HTML document. The substitute
function simply returns the root of this parameter value.
Using the substitute
ixsl:page function in the test utility file and a
html_el parameter defined in the XSpec file for this web
application, we can verify that the expected radio button has the
Case 3: Executing an Extension Instruction
Substituting for extension instructions requires a little more work than substituting for extension functions. The goals are similar, however:
Avoid an XSpec error.
(Optionally) Report that the instruction would be executed in production.
(Optionally) Return a mock value to use for verification.
The extension instruction that schedules asynchronous calls to a named template is particularly challenging to integrate with XSpec, and we discuss that in this section.
Goal 1: Avoid Error
We did not have any cases where avoiding an error was the only
goal of substituting for an extension instruction. If we did, we would have used a
try/catch structure with an empty
<xsl:catch/> element, or a
Goal 2: Report on Production Behavior
<ixsl:set-style> extension instruction sets style properties on an
object. Depending on your application's use of this instruction, you might want to
mock results or merely report what happens in production. In our case, the latter
Using an approach similar to how we handled
created a named template that, in production, does nothing but execute
<ixsl:set-style>. The production XSLT transform consistently calls this
named template instead of executing
<ixsl:set-style> directly. The purpose
of this named template is to provide a level of indirection that enables substitution
Our chosen substitute behavior is to emit a processing instruction that says what
production. The next code excerpt shows the test utility file's code for the
During XSpec testing, any XSLT code that calls the
executes the testing version, avoiding an XSpec error and providing some verifiable
information. The following
<x:expect> element illustrates how XSpec can
confirm our expectations about a style change that the production XSLT transform makes.
this case, the processing instruction about the style change happens to be the first
processing instruction in the XSLT
Goal 3: Return Mock Result for Verification
<ixsl:set-attribute> extension instruction sets an attribute on
the context node in the HTML DOM. Once again, depending on your application's use
instruction, you might want to obtain mock results or merely report what happens in
production. In our case, we chose to obtain mock results.
We created a named template,
mt:set-attribute. In production, this
template behaves and is used like the
mt:set-style template described
Our chosen substitute behavior in the XSpec environment is to produce an element with
the specified attribute set to the specified value. The next code excerpt shows the
utility file's code for the
mt:set-attribute template. It uses an additional
html_el, that acts as a mock context node.
During XSpec testing, any XSLT code that calls the
template executes the testing version, avoiding an XSpec error and mimicking the
production behavior. Because the XSpec file for this web application defines a
html_el parameter, we can verify that a particular element has
a particular attribute set.
The advantage of mimicking the production behavior is that, in isolation, it is
somewhat more realistic than merely reporting what happens in production. But in a
can be less realistic when combined with the other behaviors of the
template or function you are testing. The browser behavior is to modify an HTML element
place, not to create a duplicate element with the new attribute value. If your template
mt:set-attribute twice in succession to set different attributes, the
output to test for would be two modified elements: one with the first attribute and
with the second attribute. When testing your own application, you can decide whether
testing needs are better served by an indication of the production behavior, this
particular way of mimicking the production behavior, or something else.
Additional Challenges in Scheduling Template Calls
<ixsl:schedule-action> extension instruction schedules a call to a
named XSLT template. Scheduling the call, instead of executing it immediately, is
because it lets the browser regain control and render whatever results it has so far.
with other interactive XSLT extension instructions, XSpec does not recognize
<ixsl:schedule-action>. XSpec cannot schedule actions for later
execution. Beyond that, this instruction presented some additional challenges for
The first challenge was executing one behavior in production and a substitute behavior
in the test environment. The content of
restricted to one
<xsl:call-template> instruction. A thin wrapper similar
to the ones around
<ixsl:set-attribute>, described earlier, would not support arbitrary
template calls having arbitrary template parameters. On the other hand, if the wrapper
<ixsl:schedule-action> were complicated, the complexity would be
located in the production XSLT transform. That extra complexity in
production seems undesirable, especially if one purpose of using
<ixsl:schedule-action> is to make the web application run faster.
As an alternative to the wrapper approach, we use
<xsl:catch>. In production, the
<xsl:try> block executes
<ixsl:schedule-action> successfully, assuming there is no unintended
error condition. In XSpec, the
<ixsl:schedule-action> instruction cannot
execute, and the processor falls back to the
<xsl:catch> block. The
<xsl:catch> block produces the desired substitute behavior. This
structure achieves the goal of varying the behavior in production and the test
environment, albeit with the disadvantage of requiring some test-oriented XSLT code
located in the production XSLT file.
In some situations, we make the
<xsl:catch> block call the same
template named in the scheduled action. In the next example, the content of
<ixsl:schedule-action> is identical to the content of
<xsl:catch>. That way, calling the template that contains this
<xsl:try> element returns the same content in the browser as in the
XSpec environment, although the manner of returning the content is different.
A second challenge for
<ixsl:schedule-action> involves a looped series
of actions, where each action schedules the next action. In effect, XSpec can get
executing the loop recursively, although the browser executes the loop serially with
problem. Here is how that can happen. One behavior of our web application involves
software language items across many products. The production XSLT transform contains
named template that creates the list for the first product, and then uses
<ixsl:schedule-action> to call the same template recursively for the
next product. In the browser, the application computes the first product's list, schedules
the next template call, returns from the XSLT transform and renders the first product's
list, and then eventually re-enters the XSLT transform to compute the second product's
list. In particular, in the browser, the scheduled call executes after the end of
prior call; in effect, the template executes many times serially. Outside the browser,
however, the template's
<xsl:catch> block executes during the template's
execution. If the
<xsl:catch> block mimics the browser behavior by
actually calling the same template as in the
the template executes recursively. The try/catch structure interferes with tail recursion
and makes the recursion unsustainable.
In this situation, instead of making the
<xsl:catch> block call the
same template recursively, we make the block merely report what happens in
Making the XSpec behavior nonrecursive was not ideal, as it increased the gap between the browser behavior and the test environment behavior.
Alternate Approach: Use the Browser
This section describes an alternative to the preceding discussion of making XSpec test scenarios runnable in Saxon-EE. Given that the Saxon-JS web application runs XSLT in a browser, it is natural to ask whether an XSpec test scenario can use a browser to run the web application being tested. Running the web application in the browser reduces the need for substitution and might remove that need altogether. Hypothetically, the browser could also render the report of XSpec results. This section describes how such an approach might work. An important caveat in this discussion is that we did not actually code the necessary XSpec infrastructure modifications. Perhaps someone will be inspired to add an enhancement like this to the XSpec infrastructure, which is open source. For our immediate exploratory purpose, we manually coded or modified certain files that would be generated by the XSpec compiler if the browser-based approach were an implemented feature. The examples use a simple application instead of our actual case study.
Consider a simple Saxon-JS application that operates on an HTML page whose body contains an unordered list.
Suppose you have an XSLT template that highlights one of the list items by coloring
text red and its background yellow. To demonstrate multiple approaches, the template
Consider the following XSpec test scenarios for the
template. The last
<x:expect> element's assertion is false, because the text
turns red, not blue.
In the existing architecture for an XSpec test suite for an XSLT transform, the XSpec infrastructure compiles your XSpec file into a generated XSLT file that imports the XSLT stylesheet you are testing, runs your code according to the test scenarios, and compares the results with your expectations. The XSpec infrastructure also formats the results into an HTML report. In this example, we envision an enhanced XSpec infrastructure using multiple files:
XSLT file, which runs your code according to the test scenarios, compares the results with your expectations, and returns pass/fail information. This file is similar to the one the actual XSpec compiler now creates for non-browser XSLT transforms. In a full implementation, this file would be generated from the XSpec test file, but in this example, it was manually modified based on the actual XSpec compiler's output.
Stylesheet export file, which Saxon-EE compiles from the XSLT file in item A. This file is always generated.
HTML file that you open in a browser whenever you want to execute your XSpec test suite and display a report of the results. This HTML file is similar to the real application's HTML file. In particular, it contains the same document markup that the web application looks for (such as
If variations in the production HTML files for Saxon-JS web applications make it impractical to generate this HTML file, perhaps test developers would create and maintain it manually.
While developing this example, we wanted to explore how closely we could align with
existing XSpec infrastructure. We found that the mapping of basic XSpec code units
(scenarios and assertions) to compiled code units could be the same, but the way the
of compiled code call each other had to be different. For a simple XSpec file structured
in this example, the existing XSpec compiler produces one named template per scenario
one additional named template per
<x:expect> assertion. Such a mapping works
well in this example. However, whereas the existing XSpec compiler checks the assertions
immediately after running the scenario, the test for the Saxon-JS application should
browser regain control and render the scenario behavior before the test starts to
assertions. In other words, you should not run the code being tested and check assertions
the same transform operation.
In Saxon-JS version 1.2, the
SaxonJS.transform method, which invokes
Saxon-JS, lets you specify a callback function that executes when the browser regains
control after processing an XSLT transform. Typically, this callback function is a
vehicle for calling Saxon-JS again to check the assertions (although scheduled template
below, the functions named
scenario2done are all callback
functions for invocations of
SaxonJS.transform. Note that
SaxonJS.transform does not support this callback syntax in Saxon-JS version
2; the same underlying requirement can probably be satisfied using different syntax,
did not pursue that.
Another question that arose during development of this example was how different parts
of the process would transfer data, without writing intermediate files for a subsequent
to read. What data does the transform that checks assertions need to obtain from the
transform that executed the parent scenario? How do we store results of each assertion
can later format the report of the results? Or can we avoid the need for storage by
incrementally building the report? Our approach in this example uses an XSLT global
slabel, for scenario label) to transfer data from the scenario
transform to the corresponding assertion transform. Also, the example uses the
code captures using the
deliverMessage option that is part of the
SaxonJS.transform method. The messages from various assertions accumulate in
alltestresults, whose content is eventually passed
to the code that formats the report.
When you open the HTML file for this example in a browser, it briefly flashes the first two screens below to run the two scenarios, and then settles on the report.
Some Ingredients for Browser-Based Approach
Extrapolating from the example, we conjecture that browser-based XSpec testing for Saxon-JS applications might require the following ingredients. This list is not comprehensive.
If the implementation uses the
deliverMessageoption as this example does, then another change in the XSpec compiler involves using the
<xsl:message>instruction only for the exact data we want to capture and pass downstream. To avoid interference from
<xsl:message>instructions in the XSLT code being tested, it would be useful to look for alternatives to
deliverMessageor more robust ways to use it.
Either the XSpec infrastructure must produce an XSpec-oriented HTML file, perhaps derived from the real application's HTML file, or the documentation must describe how the test developer should create the XSpec-oriented HTML file.
The XSpec report formatting code needs minor changes to create the rudimentary report shown in this example. To create a report closer to what XSpec currently produces for non-browser XSLT transforms would require more work in both the compiler and the report formatter.
In general, supporting multiple scenarios in one XSpec file requires some way of restoring the web application to a known state between successive scenarios — or enabling the test developer to restore it. If test developers are responsible for restoring the state, they might need enhancements in the XSpec schema to indicate setup or cleanup code. A scenario that calls setup or cleanup code without making any assertions might suffice, in a pinch, but it would be better to mark setup or cleanup code explicitly. One possibility is to make the XSpec schema allow something like
<x:cleanup-call template="my_named_template_for_cleanup"/>at the beginning or end, respectively, of a scenario. The compiler would need to call the specified templates from the generated XSLT.
Capturing results programmatically, rather than just displaying them, would be desirable and would require infrastructure modifications. The example here does not include a way to capture results programmatically.
If it is possible to get code coverage data from Saxon-JS, reporting on code coverage would also require infrastructure modifications.
Benefits and Costs
Benefits and Costs of Mocking Approach
In our experience, benefits of the mocking approach include:
Simplicity. The templates and functions in our XSpec test harness are all fairly short and simple. In most of the cases where we used mocking, we planned for that architecture early in the development process, which reduced the need to refactor code afterward.
Immediate availability. The approach does not require enhancements in either XSpec or Saxon-JS.
Potentially faster execution of tests. Substitute behaviors can run faster than the actual browser behaviors.
Costs of the mocking approach include:
Some loss of fidelity. The behavior in the test environment is necessarily different from the behavior in production. If bugs go unnoticed due to those differences, we might find that our tests pass but the actual web application does not behave correctly.
Benefits and Costs of Browser Approach
Benefits of the browser approach include:
Costs of the browser approach include:
Need for XSpec infrastructure enhancements. The prototype shown here was coded manually for exploratory purposes, not generated by the XSpec compilation and reporting infrastructure currently on GitHub. Modifying the XSpec infrastructure to support the browser-based approach described in this paper would likely require significantly more design and implementation work than using the mocking approach.
Unclear limitations. We have not pursued this approach enough to know where the realism of the browser-based approach would fall short, either in a particular application or in an effort to implement the approach generically. For example, if an application relies heavily on client system events, how do you simulate an event well enough to test the event handler? If an application relies heavily on asynchronous, scheduled template calls, when do you verify results and how do you know when the template calls are complete?
Potentially slower execution of tests. The cost of higher fidelity is needing to wait for the browser to execute your test scenarios, including setup code, cleanup code, and rendering. Delays can make the development process less agile.
Our Decision and Results
[F] Feathers, Michael C. Working Effectively with Legacy Code. Prentice Hall: Upper Saddle River, N.J., 2005.
[M] MathWorks documentation, "Reference List - MATLAB & Simulink" (sample URL), https://www.mathworks.com/help/matlab/referencelist.html?type=function&category=2-and-3d-plots
[S1] Saxon-JS documentation, "About Saxon-JS," http://www.saxonica.com/saxon-js/documentation/index.html#!about
[S2] Saxon-JS documentation, "Result Documents," http://www.saxonica.com/saxon-js/documentation/index.html#!browser/result-documents
[S3] Saxon-JS documentation, "Extension functions," http://www.saxonica.com/saxon-js/documentation/index.html#!ixsl-extension/functions
[XS] XSpec. https://github.com/xspec/xspec