XML Pipeline Processing in the Browser

Vojtěch Toman

Consultant Software Engineer

EMC Corporation

Copyright © 2010 EMC Corporation. All rights reserved.

expand Abstract

expand Vojtěch Toman

Balisage logo

Proceedings

expand How to cite this paper

XML Pipeline Processing in the Browser

Balisage: The Markup Conference 2010
August 3 - 6, 2010

Introduction

Looking back at the past couple of years, it is clear that web browsers have come a long way and have evolved tremendously. With increasing support for open standards and rapid performance improvements, modern web browsers are no longer just tools for viewing web content - they have become complete platforms for developing complex applications.

Traditionally, the programming model in the web browser environment has been based primarily around HTML and JavaScript – and surprisingly, not much has changed in this regard: we still use (more or less) the same markup language for describing the structure and content of web pages, and implement the dynamic logic in the same scripting language. What has changed, however, is how we combine these two technologies: from the first static web pages with only a minimum of scripting to the highly dynamic and interactive applications of today, often built on top of entire JavaScript frameworks and libraries.

With the major browsers gradually working out the performance and compatibility issues, JavaScript has become powerful (and usable) enough not only for web pages scripting, but also as an enabling tool for other web technologies for which there is no (or very little) native browser support. So we now have, for instance, a number of JavaScript XForms implementations[1], libraries for cross-browser rendering of SVG[2], or even a JavaScript implementation of Flash[3]; and many more.

JavaScript has also brought XML to the web applications world, in particular thanks to the AJAX programming model that introduced an XML-based communication mechanism between the client-side JavaScript and the server-side. Obviously, the ability to deal with XML data relies on the availability of XML processing APIs: DOM-based access has always been the standard in JavaScript, and most modern browsers also support other technologies such as XSLT or XPath.

With the recent interest in the XRX (XForms/REST/XQuery XRX) architecture and native, end-to-end XML environments in general, web applications (both on the server-side and client-side) are becoming more centered around the XML data model; modern XForms-driven user interfaces are an example. However, JavaScript is still often used to implement the client-side XML processing logic (parsing of XML documents, extracting relevant information etc.) and to bind the various XML-based components together (populating an XForms instance etc.). While this approach works, it often requires hard-coding the processing logic in JavaScript and writing plumbing code (often browser-specific). This can make the applications difficult to develop and maintain, and to be interoperable with different web browsers.

But integrating and orchestrating XML processes is exactly what XProc: An XML Pipeline Language W3C XProc is trying to address. The declarative, pipeline-oriented approach to XML processing in XProc provides a flexible integration layer on top of other XML technologies (such as XQuery, XSLT or, for instance, schema validation) that makes developing complex XML processing flows easier and more transparent.

Although XProc is probably viewed as a primarily server-side technology, we believe that it can have useful applications on the client-side, too. Based on our recent experimentation with porting Calumet, EMC's XProc implementation to JavaScript, this paper discusses some examples of using XProc in the web browser environment.

Calumet: Sharing the XML Peace Pipe with JavaScript

To our knowledge, all currently available XProc implementations are essentially server-side applications. This is definitely the case with XML Calabash XML Calabash and EMC's Calumet EMC Calumet, which are both Java-based. Other projects that we are aware of also make use of server-side technologies.

At EMC, when we started thinking about bringing Calumet to the browser, we were basically facing two options: either re-implement the processor in JavaScript from scratch, or port the existing code base to JavaScript somehow. (As our objective was a truly cross-browser solution with no additional requirements on the client-side, we didn't want to go the route of writing an XProc browser plug-in.) Eventually, we decided for the latter option: porting existing Java sources to JavaScript.

This decision was motivated mainly by our previous experience with the Google Web Toolkit, or GWT Google Web Toolkit, a framework for building dynamic web applications - in Java, without having to write any JavaScript[4]. The central component of GWT is the GWT compiler that takes the application Java sources and converts them into highly optimized JavaScript that runs in a variety of browsers, without any need for extension plug-ins or presence of Java runtime on the client machine. GWT supports only a subset of the JRE functionality as not everything that Java provides can be mapped to JavaScript; however, the set of supported features is still comprehensive enough to bring most of the power and benefits of Java to the web application development context.

Porting Calumet to GWT turned out to be mostly a mechanical process: refactoring out dependencies on features not supported by GWT from the original Java code, and implementing adapters for accessing the browser DOM functionality. Especially the latter turned out to be critical for the overall function of the XProc engine and its interoperability with different browsers.

The result is an XProc processor with a standard server Java version and a JavaScript version that can run in a web browser. Figure 1 shows a screenshot of a simple demo application built on top of the GWT version of Calumet. The demo was presented at the XML Prague '10 DemoJam event.

Figure 1: GWT Calumet Demo Application

png image ../../../vol5/graphics/Toman01/Toman01-001.png

XProc Compliance

A large part of the code base is shared between the Java and GWT versions. This not only reduces code duplication, but also ensures that both versions of Calumet are aligned in terms of functionality and the level of XProc support. As of July 2010, the GWT version supports 34 out of the total 41 steps from the standard XProc step library W3C XProc (the Java version supports all steps); Table I provides an overview of the missing functionality.

Table I

XProc Standard Step Library - Missing Functionality

StepSupportedRemarks
p:directory-listNo
p:http-requestYesOnly simple GET requests
p:loadYesDTD validation not supported
p:unescape-markupNo
p:xincludeYesXPointer not supported
p:xsltYesOnly XSLT 1.0 supported (browser-native)
p:execNo
p:hashYesCRC32 not supported
p:validate-with-relax-ngNo
p:validate-with-xml-schemaNo
p:xqueryNo
p:xsl-formatterNo

The GWT version has most of the features of the original Java version: from high compliance with the XProc specification to extensibility and customizability. For instance, it is possible to register extension steps or custom URI handlers with both the Java and GWT versions.

Table II below summarizes the current results of running the GWT version of Calumet against the XProc Test Suite XProc Test Suite. The XProc Test Suite comprises four main categories of tests:

  • Required tests (that all conformant XProc processors must pass)

  • Optional tests (that conformant XProc processors are not required to pass)

  • Serialization tests (that test the XML Serialization W3C Serialization features)

  • Extension tests (that test the support for XProc extensibility)

Table II

GWT Calumet XProc Test Suite Results

CategoryPercent passed
Required81.85%
Optional45.45%
Serialization68.00%
Extension100.00%

As can be seen from Table II, GWT Calumet scores relatively well with the required tests (passing over 80% of the tests), as well as with the serialization and extension tests. The biggest gap is in the optional tests that depend on functionality that is difficult to implement in JavaScript (for instance, XQuery or schema validation) and some of which will probably not be supported in the GWT version of Calumet any time soon. However, this limitation can be overcome easily in the web browser environment by using the standard XProc HTTP Request step and invoking a server-side service that provides the missing functionality. Alternatively, the missing steps themselves can be implemented as callbacks to the server-side if necessary.

Preliminary Performance Results

The GWT version of Calumet has been tested successfully with most of the major browsers. Work is currently underway on implementing the gaps in the standard XProc step library support and also on improving the performance and reducing the size of generated JavaScript. The current size of the JavaScript code (without compression) is about 550 KiB, but we are confident that this can be brought down - the original Java code was not written with GWT in mind and many of the programming constructs used in the code are perhaps too generic for efficient translation to JavaScript.

Regarding the performance of the GWT version of the processor, there are noticeable (and expected) differences between different browsers, especially with more complex pipelines that involve evaluating many XPath expressions or creating large numbers of intermediate XML documents. But surprisingly, the most expensive part of running an XProc pipeline turned out to be the initial phase: parsing the pipeline source, resolving imports, performing the static analysis, and establishing the evaluation order of the steps in the pipeline. While executing the pipeline itself generally requires only milliseconds or tens of milliseconds (in Gecko- or WebKit-based browsers on a 2.33 GHz dual-core workstation with 4 GB of RAM), preparing and statically checking the pipeline often takes considerably more time (from 50 to 300 milliseconds depending on the web browser and the complexity of the pipeline); a clear area for further performance optimizations.

Having the possibility to run - and test - the same code base in two completely different environments (Java and in-browser JavaScript) actually resulted in an interesting synergistic effect between the two versions of the processor: code that performs reasonably well in Java may prove to be a performance bottleneck in JavaScript (or the other way around); but fixing the issue usually has a positive effect in both environments.

Applications of Client-side XProc

From the start, our work on the JavaScript port of Calumet was driven by a number of use cases that we were trying to address to validate that the whole concept is viable. This section describes some of these use cases.

AJAX and Dynamic Web Applications

Modern dynamic web applications often rely on heavy use of JavaScript and AJAX for interacting with the server-side. In the XML-based model, the result of an AJAX request is an XML document that needs to be processed in some way; most often by traversing the XML structure using the DOM API. For example, in a web shop application, an AJAX callback might be used to call a productlist service and to display the results in a dynamically constructed HTML table. This would typically involve iterating over the result elements in the returned XML document and creating a table row for each result.

While this approach works, hard-coding the AJAX XML response processing in JavaScript may not always be the best option. First, it requires knowledge of the AJAX and DOM APIs as well as awareness of various browser quirks, and second, the processing model becomes set in stone, often hard to change or even understand. This may be critical for larger-scale enterprise applications, or in any application in general that requires a flexible (and maintanable) processing model the developers can build on.

The example below attempts to translate the usual AJAX request - process response - update host page pattern to XProc. The XProc pipeline starts with an HTTP request to the productlist service. The XML document returned by the service is then processed by creating an HTML table row for each product element in the document. After that, all table rows are inserted into a table wrapper which is then injected into the host page. In the example, a custom extension is used that makes it possible to address elements of the host page using their ID. In our case, the p:store step effectively replaces the element with the ID search-results by the generated HTML table.

<p:declare-step xmlns:p="http://www.w3.org/ns/xproc"
                xmlns:c="http://www.w3.org/ns/xproc-step"
                xmlns:xhtml="http://www.w3.org/1999/xhtml"
                version="1.0">

  <p:http-request>
    <p:input port="source">
      <p:inline>
        <c:request method="GET" href="/productlist"/>
      </p:inline>
    </p:input>
  </p:http-request>

  <p:for-each name="for">
    <p:iteration-source select="//product"/>
    <p:output port="result" sequence="true"/>

    <p:variable name="product-name" select="/product/@name"/>

    <p:string-replace match="xhtml:td/text()"
                      replace="concat('&quot;', $product-name, '&quot;')">
      <p:input port="source">
        <p:inline>
          <xhtml:tr>
            <xhtml:td>tmp</xhtml:td>
          </xhtml:tr>
        </p:inline>
      </p:input>
    </p:string-replace>
  </p:for-each>

  <p:insert match="xhtml:tbody" position="first-child">
    <p:input port="source">
      <p:inline>
        <xhtml:table id="search-results">
          <xhtml:tbody/>
        </xhtml:table>
      </p:inline>
    </p:input>
    <p:input port="insertion">
      <p:pipe step="for" port="result"/>
    </p:input>
  </p:insert>

  <p:store href="domid:search-results"/>

</p:declare-step>

XML is not the only format that can be used in AJAX environments. JSON, or JavaScript Object Notation JSON, is an increasingly popular, text-based format for data interchange that is both light-weight and very easy to work with in JavaScript (on the syntactic level, JSON is just a subset of JavaScript). In the web applications world, JSON and XML can be seen as competing formats where JSON is often presented as the simpler and more efficient alternative to XML. Leaving aside the (often emotional) arguments between the JSON and XML proponents as to which format is better, it is obvious that both JSON and XML have much in common: they are both open formats, readable by machines and humans; they are self-describing; and they are well supported by a wide variety of tools. The strength of JSON lies mainly in representing simple data structures, where XML is often seen as too heavy-weight; XML, on the other hand, is generally better suited for semistructured data.

For the use case discussed in this section, one could argue that using AJAX and JSON would be a much better fit than XProc. Indeed: the 40 lines of XProc code (plus the necessary GWT or JavaScript code for actually executing the pipeline) could probably be replaced with only a handful of lines of simple JavaScript. The overhead of an XProc solution is also likely to be much higher compared to plain JavaScript: the XProc pipeline needs to be parsed and analyzed first, and the data is then processed using relatively expensive (at least in the browser) XML manipulations.

So is there a benefit in adopting the inherently heavier XProc approach where a simpler and more efficient alternative exists? We argue there is, although it depends strongly on the particular use case. XProc is not a hammer for everything: it is first and foremost an XML processing language, and it should be used as such. Client-side XProc therefore makes most sense in user interfaces that are XML-driven, consume or produce XML data, or require non-trivial XML processing. In other situations, other approaches may be more appropriate.

The XProc pipeline above is admittedly very minimal, and from the dynamic web applications perspective it does not show anything new that could not be done with existing approaches. The interesting part lies in what XProc can offer beyond this point. While the example pipeline may seem as unnecessary overhead for the simple problem (populating an HTML table based on data returned from the server), the perspective begins to shift when further processing of the server-side results is needed. The XProc pipeline can query or transform the data easily, enrich it, or combine it with data obtained from other services; all using a unified, declarative, and flexible XML processing model.

Client-side XML Presentation

A common task in XML-based web applications is presenting the XML data (either static or dynamically generated) to the user. Typically, this is done by transforming the XML data on the server-side to HTML or some other format understood by the browser. Most of these transformations can be expressed in the form of XProc pipelines, and with a reasonably compliant client-side XProc implementation, it should be possible to move the processing to the client-side completely. Delegating the rendering to the client-side can not only reduce the load on the server, but in many cases, it can also simplify the server-side functionality in general and make it less coupled with the front-end technology.

The pipeline below takes an XML document, resolves possible XInclude references, and applies an XSLT stylesheet to the resolved document; all with standard XProc and completely in the browser.

<p:pipeline xmlns:p="http://www.w3.org/ns/xproc"
            version="1.0">

  <p:xinclude/>

  <p:xslt>
    <p:input port="stylesheet">
      <p:document href="style.xsl"/>
    </p:input>
  </p:xslt>

</p:pipeline>

The previous pipeline is obviously a very simple one, but the potential of using XProc as a client-side XML rendering tool is obvious: one can imagine using XProc for bringing popular XML vocabularies like, for instance, DocBook DocBook or DITA DITA to the browser.

The example below shows a pipeline that takes a DITA topic and returns its HTML rendition. While the pipeline itself is trivial, the dita:topic-to-xhtml step will most likely perform rather complex XML processing: from resolving the various DITA link types to content filtering to applying an XSLT stylesheet. Or... maybe not: the black-box nature of XProc steps provides great freedom by allowing different implementations of the same step interface - which is not only convenient when writing (and testing) the pipelines, but it also makes it possible to adapt the pipelines to the needs of a particular user audience or to different browser environments. Thus, the pipeline below can, for instance, do full-blown client-side DITA processing in web browsers that are known to be fast (or compliant) enough, and delegate to the server-side in other cases.

<p:pipeline xmlns:p="http://www.w3.org/ns/xproc"
            xmlns:dita="http://example.org/ns/dita"
            version="1.0">

  <p:import href="dita-lib.xpl"/>

  <dita:topic-to-xhtml/>

</p:pipeline>

Interactive Processing

An interesting application of XProc pipelines in web browser is using them as a driver for interactive client-side procedures. In such an obvious user interface-oriented platform as a web browser, it is possible to imagine XProc extension steps that would add interaction with the end-user to the XML processing logic - for example, by displaying dynamically generated dialogs or messages on the screen. This would make XProc a simple (yet powerful) alternative to other approaches for representing interactive processes: from simple data collection procedures to complex and often highly dynamic maintenance and diagnostic procedures found, for instance, in the military and in the aerospace industry.

In the area of interactive processing, client-side XForms is a technology that combines exceptionally well with XProc: as an XML-based standard, it is very easy to load - or even dynamically generate - XForms-based dialogs in XProc pipelines. Similarly, the XForms submissions, which are XML documents as well, can be processed naturally in XProc.

The pipeline below shows how XForms could be used with XProc in an imaginary aircraft maintenance system; in this case, the example is a simple circuit breaker check procedure. The pipeline has two options, an aircraft model number and its variant. Depending on a specific combination of the model and the variant, the maintenance mechanic is presented with a dialog (an XForm) that allows him to enter information about the state of the circuit breaker. When the mechanic submits the dialog, the pipeline displays a message that tells the mechanic to switch on the breaker if it was in the OFF position.

<p:declare-step xmlns:p="http://www.w3.org/ns/xproc"
                xmlns:am="http://example.org/ns/aircraft-maintenance"
                xmlns:xf="http://www.w3.org/2002/xforms"
                version="1.0">

  <p:option name="model" required="true"/>
  <p:option name="variant" required="true"/>

  <p:import href="aircraft-maintenance-lib.xpl"/>

  <p:choose>
    <p:when test="$model = '777' and ($variant='200' or $variant='200ER')">
      <am:dialog href="forms/777-200-circuit-breaker-check.xfm"/>
    </p:when>
    <p:when test="$model = '777'">
      <am:dialog href="forms/777-circuit-breaker-check.xfm"/>
    </p:when>
    <p:when test="$model = '787'">
      ...
    </p:when>
    ...
  </p:choose>
  
  <p:choose>
    <p:when test="/state = 'OFF'">
      <am:message>
        <p:input port="source">
          <p:inline>
            <p xmlns="http://www.w3.org/1999/xhtml">                                                              
              Switch the circuit breaker to the ON position.
            </p>
          </p:inline>
        </p:input>
      </am:message>
    </p:when>
    <p:otherwise>
      <p:sink/>
    </p:otherwise>
  </p:choose>

</p:declare-step>

Conclusion and Future Work

Although XProc is still a relatively new technology, it is already finding its way into the XML application developers' tool set. Extending and complementing the family of established XML processing languages such as XSLT and XQuery, XProc provides a unifying and flexible integration layer that makes orchestration of XML processes easier.

In the web applications world, XML processing has traditionally been done primarily on the server-side. However, with the recent advances in modern web browsers and the growing interest in end-to-end XML application architectures, we can see that more and more XML processing is being done on the client-side. We believe that the problems that XProc is attempting to address - the impedance mismatch between different XML processing models, and the need to write plumbing code - apply equally to the server-side and the client-side (although on the client-side this may not be that visible - yet).

This paper discussed some of the possibilities of using XProc pipelines in the web browser environment, motivated by our recent work on porting EMC's Calumet XProc processor to JavaScript (using the Google Web Toolkit). What initially started more as a proof-of-concept effort has lead to some interesting outcomes. Not only did it prove that performing complex XML processing in JavaScript in the web browser is possible, it also showed that client-side XProc can have useful practical applications that are worth exploring.

The current GWT port of Calumet is still very much work-in-progress. While it is reasonably stable already and supports a relatively large portion of the XProc specification, the code still needs to be optimized and fine-tuned for different web browsers. Also, work needs to be done on providing convenient XProc processing APIs in JavaScript. In GWT-based applications, the native Java API of Calumet can be used, but for the case of traditional HTML/JavaScript applications, a JavaScript interface will be necessary. An additional option that we are considering is to support embedding XProc pipelines in the HTML script element; however, that direction still needs to be researched.

Future versions of the Calumet XProc processor will most likely be distributed as a dual Java/JavaScript package; work is still ongoing on the JavaScript version, but the progress with the development and the results achieved so far are encouraging.

Bibliography

[DITA] Michael Priestley, JoAnn Hackos, eds. Darwin Information Typing Architecture (DITA) Architectural Specification v1.1. OASIS Standard. 1 August 2007. http://docs.oasis-open.org/dita/v1.1/OS/archspec/archspec.html

[DocBook] http://www.docbook.org/

[EMC Calumet] http://developer.emc.com/xmltech/

[Google Web Toolkit] http://code.google.com/webtoolkit/

[JSON] http://json.org/

[W3C Serialization] Scott Boag, Michael Kay, Joanne Tong, Norman Walsh, and Henry Zongaro, eds. XSLT 2.0 and XQuery 1.0 Serialization. W3C Recommendation. 23 January 2007. http://www.w3.org/TR/xslt-xquery-serialization/

[W3C XForms] John M. Boyer, ed. XForms 1.1. W3C Recommendation. 20 October 2009. http://www.w3.org/TR/xforms11/

[W3C XProc] Norman Walsh, Alex Milowski, and Henry S. Thompson, eds. XProc: An XML Pipeline Language. W3C Proposed Recommendation. 9 March 2010. http://www.w3.org/TR/xproc/

[XML Calabash] http://xmlcalabash.com/

[XProc Test Suite] http://tests.xproc.org/

[XRX] http://en.wikibooks.org/wiki/XRX/



[2] See, for example, http://www.amplesdk.com/

[4] Of course, it is possible to combine JavaScript with GWT quite easily; GWT-based applications can invoke external JavaScript functionality and vice versa.

Author's keywords for this paper: XML; XProc; JavaScript; GWT; client-side; XRX