Balisage Paper: DITA Document Types: Enabling Blind Interchange Through Modular Vocabularies and Controlled Extension

Introduction

SGML and XML have always been about interchange, starting with the business problem of interchanging typesetting data sets between publishers and printers and evolving into the promise that document data could be interchanged widely among different users of the data because it was in XML.

The implicit, and sometimes explicit, promise, or at least vision, of SGML and, later, XML, was blind interchange: the ability for one party of an interchange to use data provided by the other party using only the information in the document itself along with shared common knowledge, such as understanding of the standards to which the data conforms, without the need for any additional knowledge to be directly communicated from the sending party to the receiving party. In other words, the sending party can, for example, make data available for licensing to anyone who wants to license it and licensees can get the data knowing that they will be able to use it immediately, without the need to adapt either the data or their processing infrastructure to accommodate the data received.

This vision has gone largely unrealized because the SGML and XML community had not devised a way to allow local definition of specific vocabulary while ensuring both understandability and processability of data. You could either have precise markup optimized for local requirements or you could have general-purpose markup that could be processed by many potential consumers, but you couldn't have both. This was, and continues to be, a problem that limits the value of XML because it adds significant cost to the use of XML where interchange is a requirement (and interchange is always a requirement, even if that interchange is with only with your future self). An obvious example of this cost is the complex system of validation and data normalization transforms created and maintained by PubMed Central in order to manage the publishing of journal articles from many publishers into the PubMed repository. This cost is entirely avoidable.

The DITA architecture provides a general way of thinking about and using modular vocabulary that enables interchange at the lowest cost (and lowers the cost of new vocabulary development and support of that vocabulary) in a way that no other standard for XML documentation does.

DITA's general approach could be adapted to other XML vocabulary domains and thus give them the same advantages. As discussed in the last section of this paper, while it is technically possible to apply DITA-based markup to essentially any set of documentation structuring requirements, there are many reasons why that is not going to be a viable solution for many communities. But DITA's architectural ideas can, and I assert should, be applied to non-DITA-based XML vocabularies in order to realize the same value that DITA provides in terms of reducing the cost of interchange and generally minimizing the cost of using XML. That is, I am not suggesting that DITA is the solution for everyone, I am only suggesting that DITA's way of thinking about document types and vocabularies is powerful and interesting and useful. I am asserting that DITA's approach to enabling interchange works where no other approach does.

In this paper, the term "community of interchange" means the expected set of partners involved in the interchange of a specific set of documents or document types. Different communities have different scopes, from small communities limited to a few authors or products within a small software company all the way to essentially unbounded scopes, such as all potential users of journal articles.

From a business standpoint, my day-to-day concern is with the business challenge of very wide scope interchange as done or desired by Publishers, where the value of a given unit of content is determined largely by the number of different consumers who are willing to buy it or the number of different channels through which it can be published. From this standpoint, the cost of interchange is an important determiner of overall value of Publisher's primary business output, information. The less it costs others to acquire and use the content, the greater the value of that content and the larger the potential market for it.

This paper focuses in particular on "blind" interchange. By blind interchange I mean interchange that requires the least amount of pre-interchange negotiation and knowledge exchange between interchange partners. In the ideal scenario, Parter A is able to say to Partner B "here is data in format X, a format we both know and understand" and Partner B is able to reliably and productively use the data from Parter A in non-trivial ways without the need for any more communication with Parter A.

Why Traditional Approaches to Interchange Do Not Work

SGML introduced the formal "document type definition" (DTD), which allowed communities of interchange to define sets of elements that would be allowed in the documents used by that community. Because DTDs are formal, machine-processable definitions, documents could be validated against a document type to determine conformance to the rules of the document type. The general assumption or hope or implication was that documents that conformed to a given DTD would then be reliably interchangeable within the community.

This approach to interchange does not work and cannot work because it is too simplistic. It ignores a number of factors that serve to impede interchange over any non-trivial scope, including:

In short, the idea that you could define single DTDs for non-trivial documents that would then enable blind interchange was naive at best and disingenuous at worst. The SGML and XML community has proven through painful experience that monolithic "interchange" DTDs simply do not work to enable smooth, blind interchange. Interchange can be done but at significant cost and impedance. With transforms and documentation and countless person hours spent in analysis and DTD design, people have been able to get documents interchanged but not in a way that could be in any way characterized as "blind". In general where interchange has been made to work, it has involved some combination of:

All of these reactions tend to limit the utility of interchange DTDs relative to the cost of simply mapping to and from the individual document types of the various interchange partners. Where interchange has been made to work at all it is usually because one of the trading partners has a significantly more powerful role relative to the other partners, for example Boeing or Airbus relative to their suppliers or PubMed Central relative to individual publishers. These large players can both impose constraints and business rules and also fund and maintain the tooling needed to enable interchange.

But where the interchange community does not have such a major player, then the value of the interchange DTD is much lower. The example here would be DocBook, where you have many relatively small users using DocBook with no single large player imposing some particular consistency of use across the many users of DocBook. Because DocBook both reflects a wide range of requirements and a general approach of generality over specificity, the range of application of DocBook to the same information challenge is quite wide. In addition, DocBook allows unilateral and unconstrained extension from the base DocBook DTD. This means that there is no formal or automatic way to know how any two DocBook-based documents relate to each other. (To be fair, blind interchange was never an explicit or primary requirement of DocBook as a standard.)

While an invariant monolithic DTD that satisfies all requirements of all interchange partners would enable this form of blind interchange, it is obvious that such a DTD is impossible in practice. Likewise, unconstrained extension as provided by DTDs like DocBook and NLM don't enable blind interchange because at a minimum Partner A has to transmit knowledge to Partner B about how Partner A's DTD differs from the base standard and, quite likely, provide processing components for processing that content. At that point, there is no useful difference between using the standard for interchange and using arbitrary DTDs, as the difference is only one of degree, not kind (do I have to transmit a lot of knowledge or a little knowledge?): in both cases you have to describe the markup details and, usually, provide supporting code or, lacking code, impose on Partner B the cost of implementing support for the new or different markup. Likewise, there is no guarantee that the data from Partner A will be composable with the data from Partner B such that B could re-use data elements from A directly in their existing documents, either by copy or by reference.

DITA's Approach: Modular Vocabulary Composition and Controlled Extension

The DITA approach flips the solution around. Rather than attempting to enable interchange by imposing constraints top-down through a monolithic document type that attempts to reflect the union of known requirements that can be met within the scope available, it imposes constraints bottom up in a way that enables local extension and configuration while ensuring that all conforming DITA documents can be usefully and reliably processed by all conforming general-purpose DITA processors. In the best case, interchange among partners requires nothing beyond the agreement to use conforming DITA documents. In the usual case, partners only need to interchange knowledge of locally-defined markup where that markup is of interest to the receiving partner and only need to interchange supporting processing where the locally-defined markup requires unique processing (that is, its semantic distinction also implies or requires a processing distinction of some sort).

DITA does this by moving the focus from document types to vocabulary "modules" assembled or composed into distinct document types, coupled with two controlled extension mechanisms: constraints and specialization.

DITA defines a general architecture to which all DITA documents must conform and from which all DITA vocabulary must be derived. However, DITA does not limit the ability to define new vocabulary or combine existing vocabulary modules together in new ways, as long as the new vocabulary or module combination conforms to a few constraints.

A document is a conforming DITA document if the vocabulary it uses conforms to the DITA architecture, irrespective of the specific element types used in the document. There is no sense in which DITA defines "a" document type. Rather, it enables an infinite set of interoperable document types based on a base architectural model. It also enables automatic comparison of document types to determine "compatibility" and therefore the degree to which content from one DITA document may be directly used by reference from another DITA document.

All DITA vocabulary is based on a small set of base types that define the basic structural and semantic rules for DITA documents. The base types are sufficiently general to allow almost unconstrained variation in how they are adapted to specific markup requirements so that there will always be a way to meet any reasonable markup requirement. In order to make the mechanism work the DITA architecture must impose a few arbitrary rules but those rules are not particularly limiting.

While the DITA standard defines a number of vocabulary modules in addition to the base DITA types, these modules are not mandatory (no conforming DITA document is required to use or reflect any of these modules). Likewise, the various document type definitions defined by the DITA standard or provided by the DITA Technical Committee are not mandatory and simply serve as examples or convenience for DITA users. So while the DITA standard may appear at first glance like just another monolithic DTD standard, it is absolutely not one.

<bar class="- topic/p module-1/foo module-2/bar ">
 <a>A bar</a>
</bar>

Reuse and Document Type Compatibility

The point of interchanging DITA content is not just to be able to process it but to do something new and interesting with the content interchanged. That is, we can presume that you have acquired the DITA content so that you can then do something more with it than simply generate various outputs, since you could have simply requested the outputs themselves and saved the expense of generating them.

DITA provides two mechanisms for combining content into new publication structures: maps and content references.

DITA's map mechanism uses documents consisting of only hyperlinks (DITA maps) to organize maps and content objects ("topics") into arbitrary hierarchies. The only requirement for the use of a given topic by a given map is that the topic be a valid DITA topic document. The vocabulary and constraint details of the topics used by the map are not relevant. (Maps can also use non-DITA resources, whether XML-based or non-XML but of course DITA can only guarantee interoperability of DITA documents.)

DITA's content reference mechanism ("conref") uses direct element-to-element links within topics (or maps) to establish use-by-reference relationships such that the referenced element becomes the effective value of the referencing element. It is similar to, but more sophisticated than, XInclude and similar link-based use by reference mechanisms.

Because conref results in new effective documents (either virtual or literal depending on how the processing is implemented), DITA imposes constraints on what may reuse what in order to ensure, as much as possible, that the effective result is valid in terms of the constraints governing the referencing context. In particular, a document can only use content that is at least as constrained and at least as specialized as itself (meaning that a more-constrained document cannot use content from a less-constrained or less-specialized document). This rule is defined in terms of the vocabulary modules used by both documents, not in terms of the individual content models or instance data involved.

Thus, while there may be cases where less-constrained data would in fact be valid in a particular use instance, DITA does not attempt to determine compatibility on a per-use-instance basis. Not only would that require access to formal definitions of all the content models involved, it would require complex and expensive processing for only marginal gains in practice.

Rather DITA uses the module use declarations in the @domains attribute to compare the modules used in two documents to determine if they are compatible for the purposes of content reference.

DITA Self Description and Processing

In the most extreme case of blind interchange a system is presented with a document, with no advanced preparation or configuration for the document's specific document type, with the expectation that the system can first determine what type of document the document is and second apply appropriate processing or determine and clearly report that it is either not able to determine the document type or recognizes the document type but determines that the document uses facilities or has content that it does not know how to process.

For example, XML content management systems should be capable of attempting ingestion of any XML document of any sort and, when the document is recognized as being of a known type, automatically do whatever is necessary to ingest and manage that document.

For this level of automation to be possible, documents and their components must be sufficiently self descriptive so as to allow a system to determine the document's type and, where appropriate, understand how to handle individual elements and attributes within the document. XML documents are inherently self-describing as XML as long as they have an XML declaration but beyond that more is required. If we accept that requiring the use of a DOCTYPE declaration or XSD schema reference or any other form of reference to an external schema is neither acceptable nor possible to enforce in the general case, then it follows that any self-description mechanism must use normal document markup.

All DITA documents are self describing in several important ways, none of which depend on the use of any form of document schema or grammar declaration. The self-description features are:

Taken together, these three self-description mechanisms mean that general-purpose processors can be constructed without too much effort that can do the following when presented with any conforming DITA document with all attributes explicit (or defaulted via a DTD or XSD), no matter how specialized or what set of modules it uses:

With the possible exception of HTML, there is no other XML application for documentation that allows both action (3) and allows definition of arbitrary new vocabulary.

Applying the DITA Approach More Widely

The DITA modular vocabulary and specialization facilities could be applied to any XML application regardless of domain of application. While the particular implementation details used in DITA 1.x of @domains and @class may not be appropriate for other XML applications, the concept of modular vocabularies and controlled extension could be applied. In particular, the DITA 1.x @class attribute syntax does not accommodate namespaces in any satisfying way (other than requiring the use of specific prefixes). Any more general class-like mechanism would need to use a syntactic approach that fully supports namespaces. Designing such a mechanism will be a major focus of the DITA 2.0 activity (unless the DITA Technical Committee arrives at a solution that is backward compatible with the current 1.x syntax as part of the DITA 1.3 development activity under way now).

The main practical challenge is applying the necessary constraints to existing markup design where the markup was likely not originally designed either as a hierarchy of strictly-related types or to enable specialization by avoiding unnecessary constraints in base types. Because DITA must impose some constraints on content model design, it is usually not possible to take non-trivial document types and simply add @class attributes to the existing element types and make them conforming DITA documents—there will almost always be some amount of adjustment to structural models required. Even in the case where the requirement is not to conform to existing DITA structural models but simply to enable specialization from some set of base models reflecting the design of the existing document type, it will likely be the case that element types that are conceptually specializations of a newly-defined based type will not be structurally compatible with the base types, requiring adjustment or normalization of content models. This means that applying DITA directly or a DITA-type architecture will usually require creating a new non-backward compatible version of the existing document type. As in any such endeavor, the cost must be balanced against the benefit, but I assert that the potential benefit is quite large.

DITA's out-of-the-box vocabulary, while well adapted to the specific requirements for which DITA was originally designed (authoring and production of modular technical documentation optimized for online and dynamic delivery), is certainly not limited to that particular use and can be usefully applied to essentially any type of document or documentation. Therefore, new XML standards could be DITA-based without limiting their ability to define appropriate vocabulary. That is, one can assert that there is no reason not to make all new documentation-related XML document types DITA-based.

However, there are well-established applications in specific subject or application domains, such as NLM, that reflect person-decades of markup design and that are well established within their communities of use. It would be counter-productive to suggest or require that those communities replace their existing document types and infrastructure with a new DITA-based application just to get the interchangeability of DITA. It makes more sense to adapt the DITA concepts of modular vocabulary and specialization to those existing applications, preserving the existing investment in knowledge and infrastructure while lowering the cost of implementation and interchange, raising the value of the content that uses those vocabularies.