Balisage Paper: Semantics and the Web

Introduction

Markup is powerful stuff, able to solve problems at scales from the personal to the multinational, across human languages, operating systems, programming languages, and a wide variety of approaches. We gather at Balisage to celebrate the many things it can do, to figure out new ways to apply it, and to figure out how best to make it interoperate with competing (and complementing) approaches.

Unfortunately for the amazing people of Balisage, our markup appears to solve problems that most people and organizations don't want to solve with markup. Markup enthusiasts still exist in the world, but their share of the broader conversation keeps shrinking. Some large organizations and some smaller ones still love our work, but broader public interest keeps declining. The vast bulk of the markup that gets sent out into the world over networks keeps getting simpler and simpler, using markup with fewer features than was common in the 1980s or 1990s, with support from non-markup technologies that provide meaning.

Are semantics receding from markup? Was this inevitable?

A Note on Semantics

Semantics is a word whose meaning wanders widely. The meaning of meaning and what is meaning and how does meaning connect to a word meaning meaning... it's all circular, at the best of times. If someone tells you that they have a neatly pinned-down definition of semantics, they are trying to sell you something. However, for the purposes of this paper, the exact meaning of semantics and the ways we attempt to convey it are less important than where we attempt to do that.

This paper focuses first on moves toward putting semantic indicators into documents and then on the later move toward using generic structure in documents. Semantics have moved into the hands of either procedural programming languages that operate on markup documents, into non-markup declarative formats, or both. While various groups have proposed different approaches to semantics in documents, from named elements and attributes to constructions built on URIs, semantics are, at least on the Web, mostly retreating from markup overall.

New Magic

Markup appeared and was reinvented across multiple systems. Many of them used procedural markup, embedded commands in the documents, fancy versions of the "turn on bold" and "turn off bold" control characters I used to make the output of a C. Itoh Prowriter dot matrix printer look more appealing. Others, like LaTeX, separated markup from presentation, with macros that operated on the markup. Some developers, notably Ted Nelson, found the very idea of adding markup to pristine content repulsive. (Nelson 1997)

Out of this early brew of possibilities, one family of markup approaches grew to dominate the rest. There are still many people using LaTeX and even troff, though probably not many operating Prowriters. The "angle brackets" family of markup technologies sprang from the Generalized Markup Language, GML, though the brackets appeared later.

GML also provides our hero, the h1 element, defined first as an IBM formatting option and then in Annex E of ISO 8879, SGML, and then in HTML, XHTML, and HTML again. GML was used at IBM as part of the Document Composition Facility, and included a starter set with a core vocabulary that got documentation writers started. Page 12 of their documentation introduced the h1 tag:

In its earliest days, H1 (or h1) provided structure to documents that formatting tools could use for presentation, cross-referencing, and more. It wasn't intricate descriptive markup, but it didn't require the services of a (marvelous) pizza chef (TEI Pizza Chef) to do its work.

The early GML documentation, particularly for the starter set, focused on formatting documents rather than describing their structure. The two certainly blended, and from this potent mix the next generation of markup would emerge.

As GML work shifted toward SGML work, the familiar angle brackets - < and > - appeared, and the scope and ambition of this family of markup grew. H1 remained, but now in Annex E of SGML, while most of the specification strove toward more generalized and more powerful possibilities. The memory of its publishing foundations remained strong, an early pointer toward greater things:

Annex A.1 of the specification provided more detailed explanation of what this looked like, called Generalized Markup and Descriptive Markup:

Over the following years, postulate (a) would lead markup into projects of broad data interchange as well as document structure identification, while (b) would lead developers toward creating tools mapping document structures to a variety of processing structures. (In the Annex, the definition of 'rigorous markup' focuses on markup minimization through the understanding of structural rules, something that continues in HTML5 but is rarely described as 'rigorous' today.)

SGML transformed some corners of the world, though as late as 1997, Chet Ensign was able to title a book $GML: The Billion-Dollar Secret (Ensign1997), and have the "secret" part still be plausible. SGML's most widely used descendant, however, grew from the specific markup vocabulary defined in its Annex E, "Application Examples". The definition for the h1 element appears on page 532 of Goldfarb 1990. The AAP DTDs that built on this, eventually ISO 12083 (ISO 12083 DTD), influenced the CERN SGMLguid system that Tim Berners-Lee used and then simplified for his World Wide Web.

The earliest HTML, seen here in this fragment from the original CERN web page (First Web), was built using pieces of that Annex E vocabulary, plus additional markup for hypertext:

<BODY>
<H1>World Wide Web</H1>The WorldWideWeb (W3) is a wide-area<A
NAME=0 HREF="WhatIs.html">
hypermedia</A> information retrieval
initiative aiming to give universal
access to a large universe of documents.<P>
Everything there is online about
W3 is linked directly or indirectly

BODY and A are new, but H1, P, and the later DL, DT, and DD are all capitalized descendants of that original Annex E markup.

HTML documents inherited from SGML's approach, and a burst of early browsers demonstrated that the same documents could be presented by many independently-developed tools. The vision of declarative markup wasn't just a "standards geek" thing at the beginning. Technical books aimed at introducing HTML to a broad audience included claims like:

However, early HTML practice quickly moved using markup targeted exclusively at presentation. Initially, the limited number of options and their simple structure meant that HTML practice still focused on structuring documents. However, as HTML standardized its own extensions beyond the initial set of tags, more and more of those were either created for or repurposed to very specifically presentation purposes. FONT, of course, is purely presentation. IMG combines presentation and transclusion, but with the use of spacer GIFs, was often far from semantic. Tables are, of course, the classic example of markup created for a structural purpose which was then (ab)used for presentation purposes. (Siegel 1997) The A tag shown here, of course, offered the most minimal hypertext possibility, ignoring the more sophisticated possibilities HyTime had shown the SGML world. (DeRose 2018)

While HTML was spreading quickly, a group of SGML experts converged to create a different kind of simplified markup. Rather than focusing on a single simplified vocabulary, the group that created XML focused on shrinking SGML's syntax to a smaller set that was easier for computers and humans to parse unambiguously. As two of the leaders of that project described its purpose:

Between the release of the XML 1.0 Recommendation (XML 1.0) in 1998 and the release of XML Schema 1.0 in 2001, the computing world seemed eager to absorb as much XML goodness as it could. The W3C, founded to shepherd HTML, rapidly expanded its XML work. The GCA was reinvigorated and became IDEAlliance. SGML Open renamed itself OASIS Open and started work with UN/CEFACT to reinvent business communications through ebXML and SOAP-based Web Services specifications (sometimes referred to as WS-*).

At the center of these dreams, and most visibly, sat Tim Berners-Lee, now Director of the World Wide Web Consortium (W3C). In 1999, he wrote:

The Semantic Web and its competitor (largely from the HyTime SGML world) Topic Maps has fueled many talks at this conference and its predecessor, Extreme Markup Languages, but still seems off in the distance. We are still near the bottom of the "layer cake diagram". (Semantic Web Architecture)

While those big dreams promised big things, the Web Standards Project (WASP 2013) was fighting smaller but crucial battles. The "Browser Wars" had left users with incomplete and incompatible browsers supporting various parts of standards. Even as that situation improved, thanks to time, pressure, and the slow end of Internet Explorer's dominance, many HTML developers' tools were still using old techniques like table-based layout. A huge education process strove to move developers to put their formatting in stylesheets, in CSS for HTML, that were more easily maintained and extended. Sites like CSS Zen Garden showed how to separate presentation from content, leaving clean accessible semantic HTML in the markup documents, and putting presentation in the stylesheet. Utility sites like Cleaner Site even automatically replaced table markup with div elements for easier styling. H1 elements could now take on different formatting than what had been built into the browser.

The W3C attempted to clean up the HTML mess with XML as well, developing XHTML as a way to combine the HTML vocabulary with XML structure, creating XHTML. At this point, H1 became h1 again. At the same time, the W3C specified a Document Object Model (DOM), making it possible to manipulate HTML, XML, and XHTML with JavaScript or Java. While XHTML 1.0 was mostly a syntax cleanup and documentation, XHTML 1.1 was itself extensible. Its crowning achievement was a set of DTDs that used entity references to allow developers to add more elements and attributes to XHTML, complete with support for namespace prefixes. (I believe that Murray Altheim presented on them at Extreme Markup Languages, but can no longer find records of it.) In 2010 the W3C added an XML Schema version as well.

At this point, a popular consortium was combining its early success with a particular markup language, HTML, with a more general markup language, XML, pursuing a grand vision of a Web shared by humans and computers. Companies and other standards bodies were racing to build on top of these components. What could possibly go wrong?

Old Magic Defeats New Magic

While WS-* specifications kept sprouting for years, and REST began its rise, XHTML hit walls early. Some of the issues resulted from years of celebrating markup as text, and the expectation (dating back to CERN and SGML before that) that many tags were optional. As browsers competed for the affections of developers and users, they had become ever more forgiving, each in their own ways, of tagging errors and errors within the tags. If the results looked right in a browser, they were right, and billions of lines of legacy code (and even fresh code) were unlikely to change. The dot-com bust also limited resources for cleaning up this "tag soup", even where companies were interested.

While HTML validators had long been available, they had never bonded with HTML culture they way they did with SGML or XML culture. A List Apart, a longtime bastion of web best practices tied to the Web Standards Project, did offer an article (Koch 2005) on extending XHTML DTDs to add attributes. By 2008, however, even ALA could no longer wholeheartedly support the HTML validation it had recommended in articles over the years. (Marcotte 2008) XHTML syntax was only part of the problem.

XHTML standards also ran into roadblocks. While I have heard it reported in the halls of XML conferences that the browser makers actively wanted XML's strict syntax so they could reduce maintenance costs on their tag soup code and compatibility, when it came to actually implementing it, they were... less eager. XML parsers did become a normal component of browsers, but weren't regularly called upon. The XHTML 1.0 specification (XHTML 1.0) had listed ways to make XML syntax acceptable to existing browser engines, and that was about as far as common practice got. Even those were too much for some HTML tools. I remember having to convert <br /> back to <br> in a Java-based documentation system to avoid slashes appearing all over the documents.

Opposition to XHTML arose inside and outside of the W3C, with browser vendors creating the less formal WHATWG to propose other paths. Initially intent on creating JavaScript-based alternatives to XForms, the WHATWG moved more boldly toward a full HTML5, explicitly not XHTML. In a distant echo of SGML's "rigorous parsing", HTML5 included its own parsing model (WHATWG parsing), offering more relaxed syntax options based on an understanding of the HTML vocabulary. Beyond that, however, they largely kept changes to the HTML vocabulary small, allowing for some extensibility through data-* attributes. (HTML5 Data Attributes) XHTML 2.0 was ended and its proposals effectively orphaned in 2010 (XHTML 2.0), when the XHTML Working Group was closed. (XHTML 2.0 WG)

At the same time that HTML5 was firing up, limitations in CSS were shifting the way developers created markup. In the early days of HTML, tag choice was central, as the tags determined the result. As formatting moved into separate stylesheets, the markup itself could be far more generic. With the demise of table-based layouts and the general failure of frames, both of which created major accessibility issues, float-based CSS layouts took over. Even the best float-based layouts (Levine 2006) used div elements as generic containers, receiving formatting instructions from the stylesheet without contributing to the content of the documents. Some layouts required multiple levels of divs. "Div-itis" became a common diagnosis (McDermott 2011), but there was little developers could do to avoid it until the appearance of the CSS Flexbox and Grid specifications a decade later. Even when developers used more semantic elements, they often used CSS resets (Meyer 2011) that removed the formatting reminders of what those elements had been, simplifying their work tremendously. (Meyer 2007) On the more semantics-friendly side, WAI-ARIA created descriptions to provide better accessibility to increasingly generic markup. (WAI-ARIA)

CSS wasn't the only aspect of Web development fond of the div element. JavaScript had been on the Web since 1995, and had grown from a supplemental scripting language to a powerful controller of content and interaction. Standardization of the Document Object Model (DOM) gave JavaScript the ability to listen for activity and respond to it by modifying documents. The object-oriented model of named generic containers holding state as values was a good fit for div elements with id, data-*, and sometimes class attributes. Recent CSS frameworks like Bootstrap and Tailwind CSS are designed to mesh well with this div-centric model.

Even within the XML community, developers were looking for other approaches. As early as 1999, a group of developers on the xml-dev mailing list started searching for an even simpler XML, SML. (La Quey 1999) Those efforts eventually bore long-lasting fruit in YAML, "YAML Ain't Markup Language" (YAML), which is commonly used for configuration on large software infrastructure projects. At the same time, Douglas Crockford was extracting a JavaScript data structure from that language, and called it JavaScript Object Notation, or JSON. (JSON) By happy coincidence, JSON was easy to make into a clean subset of YAML, letting XML's two primary text-based competitors work together.

XML got another burst of attention when Asynchronous JavaScript with XML (Ajax) appeared. (Garrett 2005) Rather than emerging from a standards process, Ajax was a usage pattern that emerged from complex projects. Instead of refreshing documents constantly in a formal client-server conversation, a single document served as the frame for multiple changes of content and structure inside of it. JavaScript programs could use XMLHTTPrequest object to send multiple requests to servers, rather than having to do a complete cycle of change in order to have a conversation. Unfortunately, since binding JSON to JavaScript object was naturally trivial, most of this traffic migrated to JSON rapidly. At the same time, other APIs were also migrating away from XML to JSON. (DuVander 2012)

While none of these setbacks meant the end of XML, the spark had faded. XML had opened the door, convincing people that text-based data and formal document interchange was possible, but both the data world and the Web world were shifting away from XML approaches. While most XML-focused working groups, with the notable exception of XHTML 2.0, were able to complete their work and achieve Recommendation status for their projects, the W3C's XML Core Working Group closed in 2016 (W3C 2016) and the XSLT/XQuery Working Group closed in 2018. (Jia 2018) The RDF Working Group, which had focused on semantic possibilities above the level of markup, had previously closed in 2014. (W3C 2016)

XML had largely vanished from the browser-based Web at this point. Maybe it wasn't needed? Existing standards used in web browsers provided multiple levels of support for custom yet shareable semantics. (St. Laurent 2013) The W3C took another shot at specifiying a coherent component model, now combining semantics and behavior. Web Components suggested a path forward, adding features to the DOM that allowed custom markup and its supporting code to have more private state, making it easier to mix and match content and behavior specified in different places. However, while component models have indeed taken off, they tend so far to mostly operate in the contexts of specific frameworks (notably React) rather than as generic containers that can migrate freely.

The markup used in those components, and in web pages and applications generally, has continued its march toward generic <div> elements. The 2005 data from Google that justified many HTML5 priorities has unfortunately disappeared from the Web, but a 2016 study showed that <div> was the most popular tag used in the body of websites, with <p> still used in 81.5% of sites. Our erstwhile hero <h1> was in 55.8% of sites, with <h2> close behind and <h3> down at 43.4%. List structures, commonly used for navigation menus, did okay around 74%, but most inline markup is much further down the list. (Except the generic <span>, at 75.6%. However, looking at the usage of tags inside of sites, <div> is responsible for 55.8% of text content markup, and <p> for only 12.1%. (Rosu 2016, with an updated version at Advanced Web Ranking. Going forward, the Web Almanac (Meiert 2020) is probably the best source.)

More and more, those <div> elements get their content through JavaScript. While sometimes the (new and improved) Server-Side Rendering approaches send the first version of a document or application with its content already in place, many times a blank template is sent with JavaScript, which retrieves the actual content on a separate channel. Sometimes that content still arrives as HTML or XML, but often it arrives as JSON. The latest popular approach to connecting the client to the server, GraphQL, uses JSON to specify both its requests and its responses. Markup languages can be part of the conversation for specific implementations, but are left out of the core specification. (GraphQL 2018)

Markup still has a strong core of supporters, but is falling into secondary or specialist use. Keepers of the semantic markup flame in HTML still encourage learners to master the core HTML vocabulary before leaping to CSS and JavaScript frameworks. XML still gets endless use in sophisticated documentation projects, and "on the wire" in protocols built before and sometimes after the rise of JSON. (And YAML is taking a share of JSON's more complex projects.) As a share of visible projects, however, markup's star has decidedly dimmed.

Is there hope for large-scale use of semantic markup?

Some people and projects still need shared semantics, and semantic tools remain available on the Web, even in the lands of divs. In the late 1990s, XML Namespaces (Namespaces) seemed to win a crushing victory over architectural forms (XML Architectural Forms), at least within the W3C. URIs were to be the glue between markup vocabularies and meaning, rather than mere attributes offering other semantics that might apply to an element. In the 2020s, that victory seems to have been reversed. Namespaces are rarely used for the free mixing of vocabularies they were supposed to support, while millions of HTML developers use architectural forms without being aware of it. (The concern many of us had about "what does a namespace URI mean?" RDDL 2002 seems quaint today.)

HTML5's decision to base its parsing model on an approach that required knowledge of the HTML vocabulary limited its extensibility. CSS and JavaScript do work with other elements dropped into HTML5 documents, just slightly less predictably. Markup hygiene matters more for element names that HTML5 doesn't already understand. Despite claims like "the web ecosystem routed around the damage of XML's influence by making HTML better suited for extensibility than ever before" (Denicola 2014), HTML's extensibility remains limited.

HTML5 does slightly expand the attribute-based extensibility that HTML had long provided. The data-* attributes (HTML5 Data Attributes), build on the continuing existence of id and class, so that there is frequently a home for semantic information identifying the "real" purpose of even a generic element. While it is possible to tell that class="headline" means the same thing that <h1> once did, its use is mostly limited to individual cases. It is chaotic, cloudly, often duplicating, and case by case, but it still lingers, while namespace URIs have vanished to specialist schema zones. Most of the meaning now is kept in the JavaScript code used to process this markup, so the markup's use is possible but often limited. It can be used like Architectural Forms, but isn't as general.

While JavaScript is mostly available in the browser, it's not convenient to run massive quantities of it across thousands of documents while indexing documents. (Though Google indexing does run some JavaScript. (Google SEO 2021)) Sites indexing and sharing content often suggest additional markup from Web content that simplifies the work of sharing it. Social media sites often suggest header meta markup, like Twitter Cards (Twitter Cards) and Facebook Open Graph (Facebook Open Graph), to make it easier for their tools to present shared pages to users. Google supports Structured Data (Google Structured 2021), built with RDF or more frequently JSON, for similar purposes. All of these are included within documents, using minimal markup, and often duplicate information that is already present in the document.

Developers who want to extend the HTML element vocabulary can use Web Components, which support hyphenated element names. Though this URI-free approach to prefixes has been described at Balisage as "the coming namespace winter" (Miłowski and Walsh 2014), it does allow developers to create custom elements. It is more direct than the attribute approach, it is trapped somewhat by the slow and often halting emergence of Web Components specifications and support. (Web Components)

There may yet be hope for large-scale use of semantic markup, but it would require a drastic turnaround from the trends of the last two decades. If it arrives, perhaps we're better prepared for it this time.