Balisage Paper: Uphill to XML with XSLT, XProc … and HTML

Note

Note

Many of the more challenging problems in data conversion are much more tractable when broken into parts or phases, than they are when conceived as single operations. Having an nominal carrier format already designed – even before other parts of a system are yet fully mature – is really helpful when we turn to building tools and processing, since one aspect of that problem at least (namely, what sorts of tags to use to express things), has been reduced, making it possible to focus on the more essential parts (what is the information being represented and processed).

Thus, even when HTML solves no problems for a conversion pipeline, it can expose them. This is itself is really useful.

So for example, a huge problem in data conversion is structural inferencing – how to determine appropriate structures latent in unstructured data: in transformation terms, allocating contents to structures properly in a result, even when the input is "soup". It is convenient, though not essential, that HTML can be used easily to represent data in both forms, before ("soupy") and after ("structured"), using HTML div or section elements to represent the structure.

It's getting from one form, to the other, that is the problem. Producing the soup from the structure is easy: that's why we call it "down". But to get structure from the soup is comparatively difficult, and solutions tend to be accordingly brittle, partial and limited. The process requires two distinct operations, not just one. First, the "signaling elements" that are taken to represent structural boundaries (in this case, section titles or other section demarcators), must be recognized. (For most purposes in structured documentary data, section titles serve as effective signals regarding at least some salient document structures. But this is not always the case.) Only once this has occurred (logically or temporally) does it become possible to render the structure itself. And rendering the structure is only possible if the signals line up.

This two-phase approach plays out easily in HTML: in the first phase, paragraphs that are "actually headers" can be promoted to h1-h6. In the second phase, the section structure is built up from the headers. Each of these is significantly easier, when it can be isolated from the other.

What's more, we can explain it this way, and anyone familiar with HTML can understand it. We don't need to educate them in a new data representation to make our methods explicable.

Note

A so-called "meta-pipeline" or "meta-transformation", loosely, might be any application of a transformation technology that is anything but the simple three-part source/transformation/result arrangement. Yet even within a simple architecture, pipelines will typically be made of pipelines, and transformations will include multiple "logical" and even "temporal" steps or stages, within both their specification(s) and their execution. More complex arrangements are possible and sometimes useful. These include not only pipelines of transformations in sequence (each one consuming the results of the precedent one) but also pipelines with extra inputs, spin-off results, or loops, wherein (for example) logic is produced in one branch that is then used to transform the results of another branch.

Because XSLT is syntactically homoiconic (canonically expressed in the same notation that it reads and produces, i.e. XML), it is a straightforward exercise to construct a pipeline whose transformation is itself generated dynamically. This is useful if we don't know what XSLT we will want, until runtime. If we can specify inputs to produce a transformation programmatically, we can delay its actual production until we have the data.

An example is the header promotion transformation as described above – a transformation of HTML data in which paragraphs (p elements) can be mapped into h1-h6 based on properties either assigned to them (in the data) or accessible and measurable. This is not a trivial operation, but it can be achieved using pipelines in and with XSLT.

The difficulty is that such a transformation depends on an assessment of which properties assigned to which paragraphs, separately and together, warrant promotion for that (type) of paragraph. The particulars of this assessment may only be fully discovered in view of the data itself. So a pipeline has to "inspect" and "assess" the data itself before it can produce its set of rules for handling it.

Thus, in a pipeline, header promotion can proceed in three steps: in the first step, and analysis of the data is conducted in which candidate (types of) block-level or p elements are selected and bound to (different levels of) header elements. In a second step, this analysis (result) is fed to a generic "meta-transformation" XSLT that produces a one-time use XSLT specifically for the data set. The third step is the application of this one-time custom-fit XSLT to the data, matching elements appropriately to produce headers from the p elements as directed.

As noted, HTML's lack of any kind of structural enforcement over its element set, is very advantageous here. A header promotion transformation can litter the result file with h1 - h6 elements, all without (much) concern either for formal validation or for predictable behavior in tools.

To be sure, such raw data may not be ready to bring into a structured environment, which will not permit such a free representation: but then, that is the point. The inference of div or section boundaries, once headers are in place, is another fairly straightforward operation – when the data warrants it.

Other similar examples of pipelines, metapipelines and multi-stage pipelines can be mentioned, including pipelines

Note

One interesting and unexpected consequence of distinguishing our temporary carrier format from our long-term hub format, is that it becomes possible to mix them on the way from one to the other. This may be regarded as cheating - certainly it feels a little different to write XSLTs that expect tagged inputs to be mixed, and to behave accordingly. Yet since the entire purpose of such XSLT is to clean up such tagging (i.e. to reduce the use of the uphill carrier format in favor of the hub format), this isn't actually a problem. It's something we can get used to.

In such murky worlds, the introduction of ad-hoc validation steps for even intermediate results, is sometimes useful. For example, a Schematron can tell whether an HTML file's headers (h1-h6 elements) are in "regular order", thus susceptible to serving as markers of a regularly-nested div or section structure. (One implementation of a rule enforcing regular order is that each header be either an h1, or not more than one level below the immediately preceding header. So an h4 can directly follow an h5 or h3, but not an h2.) Naturally, much emphasis is placed on assigning and managing values to @class and thereby abstracting semantic categories away from the literal code.

Work on validating profiles of HTML is critical to this. Schematron is one option. So is Gerrit Imsieke's Epischema https://www.xml.com/articles/2017/04/29/epischemas/. In any case, when the goal remains clear – both valid and optimal expression in the target language – much remains forgiveable along the way. The relevant principle here is "Validation is in the eye of the beholder".

Another consequence of the "murky worlds" is that it becomes possible to implement features such as "passive upconversion". For example, a system that produces JATS from arbitrary HTML can easily be "sensitized" to produce certain JATS elements as soon as their names are found on HTML inputs, assigned via @class. This makes the implementation of mappings from any HTML into these JATS elements, as easy as seeing to it that the appropriate class value is provided in an HTML → HTML transformation – something relatively easy to configure and implement. (Especially using another meta-XSLT.)