Balisage Paper: Extending Vocabularies: The Rack and the Weeds

Planned-for Extensions

The creators of a vocabulary foresaw a need but not the specifics, and so provided mechanisms to allow the vocabulary to be extended.

Unanticipated Extensions

The creators of the vocabulary did not foresee the need for extensions, or not of the kinds that users of the vocabulary wanted or needed.

Hybrid and Absorbed Extensions

Extensions are sometimes adopted back into a vocabulary; in most cases this is done in such a way that people previously using the extension have to change their usage to conform, because people making extensions usually do not share exactly the same constraints and perspectives as the vocabulary’s creators.

An absorbed extension, then, is one that was originally an extension but became part of the vocabulary. A hybrid extension shares characteristics of planned-for and unanticipated extensions and may also be, or become, officially absorbed.

Committee Proposals

Very often a new feature starts out as a proposal from someone already participating in whichever group or committee maintains a particular vocabulary. Such an extension may go into a future version of the vocabulary, in which case it people using that next version do not generally consider it to be an extension. Sometimes the committee will reject the proposal, and in that case it may later become part of some third-party extension. Eventually it may return to become part of the main specification, as SVG did with HTML 5.

The important thing about proposals from within the committee is that because they are very often developed in a context of what Applen and McDaniel refer to as tacit knowledge [Applen & McDaniel 2009], they tend to fit in well with the overall design of the vocabulary or specification in question.

Community Proposals

Sometimes people who are on the periphery of a committee, whether outside but following closely or inside but not part of the cognoscenti or not well respected, will come up with a proposal; at other times it’s committee members but the proposal falls outside the scope of the committee work, or is of a nature that means the details could not easily be agreed upon within the group.

At other times, a user or implementer group outside the original or main committee decides to extend the specification. This can happen through dissatisfaction with the main group (as, for example, with HTML 5 and the WHAT WG) or a need for something faster than full consensus allows, or sometimes simply because the outsiders did not understand that they could have been more closely involved.

Forks

Strictly speaking a fork can happen from within a committee or from the outside, or even a combination of both. The term comes from open source programming: a fork of a piece of code (whether a complete application or just a single library) comes when someone copies the original, changes it, and starts redistributing their changed version. This can be for several reasons: the original maintainer might have wandered off, leaving the work orphaned; the maintainer might have refused to make changes someone wanted; sometimes the original maintainer passes on the flag to someone else, or agrees there will be two versions with two different areas of focus. Thus a fork can be amicable or can be hostile.

In the world of markup vocabularies and specifications a fork most often happens when the original standards committee doesn’t recognise a particular need as valid (rightly or wrongly). It can also happen if a group needs a smaller subset of a specification, as happened with the Mallard subset of DocBook for the GNOME project. Usually in the case that the new fork is not intended to replace or supplant the original specification there is no need for hostility: Mallard was made for use by a specific community, for example.

Merging

Sometimes two specifications merge into a single larger one^*; usually the resulting vocabulary is the union of the original specifications before the merge, often with some additions since if one is revising a vocabulary it can be hard to argue with people who want to add to it.

A merge can be done to include one vocabulary inside another, such as HTML 5 incorporating MathML and SVG, rather in the manner of a shark eating a jellyfish. The result can remain separate specifications or can become one larger one. Another reason for a merger is when there are variants of the original specification in use and incorporating the variations seems best for everyone.

After the Work Ends

Sometimes a maintainer wanders off, loses interest, loses the ability to continue the work, or even dies. An organization can be taken over (such as Sun Microsystems by Oracle), or can cancel a project (such as Oracle canceling Solaris). Sometimes the specification may remain frozen, and may even become difficult or impossible to obtain. But if the vocabulary is in widespread use then new needs will emerge, and a new group will probably carry the torch forward.

Sometimes a committee will mark a particular vocabulary or version as deprecated. On other occasions a specification may be actively withdrawn by its publisher, for example for legal reasons. And of course at times a specification is perfect: the committee can be disbanded because the work is finished.

Functional Extensions: New Behaviour

A behavioural extension is one that changes the behaviour, or enables such changes, in software processing marked-up documents.

In HTML, for example, code running in the client (that is, in a Web browser) makes use of the extensions: for example, a browser might interpret rel=toc to provide a toolbar button or keyboard shortcut to access a table of contents entirely outside the containing document. Markup to extend behaviour is often very specific to the behaviour however: early HTML examples included blink and marquee elements whose purpose was to affect the display of contents rather than to indicate meaning.

A vocabulary or a system using a vocabulary might also be extended by missing or adding entirely different languages. For example, one might include OpenGraph or Schema.org features in an XML vocabulary, and these might be expressed in a JASON-LD syntax inside XML elementsor attributes. Or, a system might be extended by supporting scripting, and this might become visible inside documents. Such extensions can be pernicious, tying documents down to use by specific software in specific contexts and limiting reuse.

Semantic Coverage: New Meanings

In the semantic coverage case, the purpose of the extension is to represent information in documents. For example, in HTML, one might use a span element with a class attribute value of place to mark up places mentioned in a document. Although this information might then enable new functionality, such as connecting the prose to a map view, the markup is not tied to any particular behaviour.

Implicit Extensions

Sometimes when we use a specification and share our documents or data, we do not realize that we have created an extension. For example, people marking up HTML documents might find they have used common class attribute values, or people might take a specification like SCXML and use it for subject domains that the Working Group that developed it never envisioned, and to which the prose in the specification applies at best poorly. Over time the result can be to broaden the scope of the original language.

An implicit extension, then, is one where the fact that a vocabulary has been extended is not necessarily obvious to an observer.

Explicit Extensions

Many specifications provide methods for extension, some of which are covered later in this paper. In most cases the fact that something is an extension is made explicitly visible: for example, by the use of an XML namespace, or by including a module or library.

An explicit extension, then, is one whose use (and not just whose description or specification) makes it clear both to software and to any human working with the vocabulary that an extension has been used.

Usage Conventions

Sometimes it seems easier to decide on a particular way of using a vocabulary than extending it. With a vocabulary that does not include section titles, one might decide to use the first paragraph of each section as a title, even if formatting software does not embolden it. Strictly speaking a usage convention is not an extension to a vocabulary, but it extends the scope of the vocabulary without introducing any new terms or markup.

Adding New Elements

One of the most obvious ways to extend an XML vocabulary, or one in any similar language, is to add new terms to the vocabulary. We might, for example, decide that the title element of an HTML document is insufficient for our purposes because it does not allow nested elements within it; instead, we add a pagetitle element that’s richer.

One obvious problem with this is that existing software doesn’t understand it. The second is that we still need a title element in each document for existing software to use, so now there is duplicated information. The new element reduces interoperability of documents, but if the usage is confined to a well-defined group then this is not a problem.

The use of XML namespaces is the most common way to identify extension elements. Namespaces are a fragile mechanism, often failing silently if there’s a typo in the namespace name (the URL) and in some implementations even failing if a document uses a different prefix. However, the fragility seems more than compensated for by avoiding conflicts, where two groups add elements of the same name. One of the original use cases for XML namespaces was to allow the mixing of vocabularies in this way.

In all cases, anyone adding an element of their own to documents that otherwise conform to someone else’s vocabulary needs to ensure that receiving software that does not understand the new element will behave sensibly: this is known as fallback. For example, a Web browser receiving a document containing a dblookup element will (in the absence of scripting) simply display the contents of the element. The document author therefore needs to include appropriate fallback contents. The design of extensibility in HTML places burdens on document authors.

Adding New Attributes

Very often, software processing a vocabulary will ignore attributes that are not recognized. Schemas need to be modified, but that’s true of any change. XML extension attributes can be associated with an XML namespace to avoid conflicts, as with elements.

A benefit of using attributes for extensions is that they tend to be less disruptive than elements. On the other hand they are restricted to simple string content and cannot be marked for language or text direction (e.g. RTL). Attributes are therefore not in general suitable for human-readable content: for example, you can’t easily have Taiwanese alt text for an SVG image in a Chinese HTML page, and this matters because Unicode code points are shared between those languages, so that language marking is needed for the text to be readable. In addition, there can only be one attribute of a given name on any particular element, limiting some sorts of extensibility.

The HTML 5 specification reserves attributes whose names begin with data- to be extension attributes, but this naming convention is not always acceptable to other groups extending HTML.

Adding New Content

Additional content is not usually considered to be an extension, since it does not affect the vocabulary itself. But consider including multiple translations of each paragraph of a document, one after the other; a usage convention might be used to say that the first paragraph is Romanian and the second the Italian translation, marked as Italian with xml:lang but not otherwise as a translation.

Adding New Values

New attribute values or element contents make an obvious way to extend many specifications. Attributes with names like role seem good candidates. It can be difficult to avoid collisions here, however, and there can be problems with fallback.

Examples in HTML include adding new meta or rel values, using data:* attribute values instead of linking to external resources, or using non-standard ARIA role attribute values. Note that this is different from extending HTML using Custom Elements or new class attribute values, because those are intended to be used for customization.

Subtractions

It may seem odd to consider removing part of a vocabulary as an extension. Such a change, however, can greatly facilitate implementation and can also help with authoring (by reducing choices). A diminished version of a vocabulary is sometimes known as a subset and sometimes as a profile, depending mostly on whether the speaker approves of it or not. Subsets (or profiles) can reduce interoperability, because an implementation might support one dialect and not another. They are therefore most suited for well-targeted use cases and communities.

One well-known example of a profile, or subset, is XML: every well-formed and DTD-valid XML document is also a valid SGML document. Admittedly this took a change to SGML to achieve, but the change (or rather, set of changes) was not unreasonable. Although XML was originally made by a group who did not think they could get the SGML committee to make changes in a timely fashion, if at all, in the end the committee turned out to be generally (overall) amenable to changes, and the design of XML could have been somewhat simplified had this been anticipated.

Combining Vocabularies: Xreole

Merging specifications to make a superset has already been discussed above. Another possibility when merging is to pick and choose, resulting in what is perhaps best considered to be an entirely new markup vocabulary, a sort of XML Creole, that was influenced by its ancestors but is not compatible with them.

This may sound the province of Igor in the basement, but can have the advantage of reduced training costs and sometimes even reduced tooling costs. Consider a vocabulary that uses DocBook element names for structure, HTML names for paragraphs and below, and DITA-style assembly from fragments. We could call it DitaWebBook. The HTML names for italic and bold, the accessibility attributes, the p element, all add a (perhaps false and misleading) sense of familiarity. Authors may then be surprised when MathML or SVG or JavaScript are not supported.

Adapting Existing Markup

When you don’t have an element to mark up a foreign phrase that’s to be italicized, and there’s no element for meaningless (semantically unweighted) italics, what’s an author to do except look for some other element that displays in italics? Emphasis, perhaps, resulting in documents in which text-to-speech software reads out foreign phrases (or book titles, perhaps) in a louder or higher-pitched voice as if they were really important.

More pernicious them poorly-accessible italics, hover, are values that are interpreted by software: our vocabulary didn’t have an element for postcode, so we used email address, because we aren’t allowed to store those. This is payback for vocabulary designers who did not allow for extensibility. The three-level postal address that doesn’t work in other countries; the telephone number field that doesn’t allow for an office extension number. Every example represents a design failure.

Adapting markup is sometimes derogatorily called tag abuse, although it can also be a form of usage convention.

Scripting

It is often tempting to make a system user-extensible by incorporating a scripting language. The result, as suggested above, can be that documents become tied to a particular system used in a particular configuration, because they contain fragments of programs or hooks for extension scripts to use.

An example is HTML Custom Elements, where the language is extended not by editing the grammar in some way, but through a JavaScript API which itself is subject to change.

Version Numbering

if an XML vocabulary includes its version number in its namespace, any change to the version number will generally break all processing tool chains. This is appropriate if it would always be an error for version N software to attempt to process version N + 1 input, but more often there are compatible changes, or new features added to the vocabulary such that every version N+1 document that does not use the new features is also a conforming version N document. This can be managed by separating the namespace (if used) from a version attribute on the top-level element, as is done by XSLT and DocBook 5.

It also helps to use a version number scheme that says that minor revisions are compatible in the way mentioned above; this is often done using a decimal point in the version number, so that a processor for version 3.2 of a vocabulary can process input marked as 3.* (where * represents any number, such as 3.9), but would report an error if given a version 4 document, where the first part of the number, before the dot, was higher than the processor understood. The 4 here is called a major revision number and the par after the dot (or the entire number) a minor revision number.

Allowing Mixed Namespaces

Allowing foreign, or secondary, namespace can help demarcate extensions from the primary vocabulary, and can make sure there are no conflicts. For example, both DocBook and SVG have title elements, but DocBook documents that use SVG elements associate them with the SVG namespace, so there is no conflict. However, the DocBook 5 specification indicates where SVG elements are allowed to appear.

Fallback

One of the places where CSS design has improved upon HTML design is the notion of fallback; that is, in considering what an implementation will do if it encounters CSS it does not understand, and making sure the base language is designed so that a sensible fallback is always possible, meaning that the document should always be readable even if some features (such as coloured borders, for example) are not rendered.

Constraints such as CSS FallBack places on designers of language extensions can be very helpful to user communities.

Extension Attributes and Namespaces

The HTML 5 specification allows any number of attributes whose names start with data- to appear on any element as an extension. In an XML environment one might supply a specific extension namespace, or one might say, as XSLT says, that attributes in any namespace other than that of XSLT are extension attributes. The goal is to make sure there can never be conflict between extensions and the original vocabulary as it grows and changes over time. Requiring people writing extensions to use their own namespaces means that any two different extensions will not conflict either.

The same techniques can be used with any names, including elements.

It should be noted that a large proliferation of XML namespaces can cause problems with implementations; there have been XSLT engines, for example, with limits of 256 namespaces per element, or even per document. In addition, users can find it confusing to remember which namespace to use. A possible strategy is to stick to one for the main vocabulary, and one for each organization making extensions, rather than one per extension.

Communication

The single most important factor in writing a successful language extension is to be in communication with both the original language maintainers and the primary user community. Therefore, a wise vocabulary designer will provide a place for people to get in touch at an early stage both with the developers of the vocabulary and with users.