Balisage Paper: Language Identification for Program Code in Documents

Introduction

HTML, XML, and many other systems provide ways to mark the natural language of text portions, such as by defining language, lang, or xml:lang attributes. The values are typically expected to be natural language codes, often with a region-code suffix to distinguish variants (see ISO 639 and RFC 5646). Typically, such codes consist of a 2- or 3-letter language code, plus sometimes a 2-letter region code. For example:

<div lang="en-uk">Put the portmanteau in the boot.</div>

Such identification is useful for many kinds of processing:

Technical documentation, tutorials, and other documents commonly contain code as well as natural-language text, such as for examples. Those content portions have similar needs for language-dependent processing:

Language specification methods (most obviously ISO 639) have excellent coverage of natural languages, including ancient ones and even extending to constructed languages such as Esperanto and Klingon. However, they do not provide for programming languages, making it harder to gain similar benefits. There are reasons for this: programming languages are invented far more often than natural languages; they are used in different ways by different communities; and they differ typologically.

Despite many differences, there are reasons that natural and programming languages are both called languages. Both are generative systems that allow expressing an unbounded range of information in interpretable ways. More relevant for the current purpose, text in both kinds of languages frequently occurs in documents, and knowing which kind and which specific language a part is in affects many aspects of how it should typically be processed.

To help gain similar benefits to those of labelling the applicable natural language of content it would be useful to enable labelling the programming language of content where applicable. Data formats might similarly be of value when data in them is likewise embedded.

A given piece of content is normally only in one language (whether natural or program), barring unusual edge cases such as code carefully crafted to conform simultaneously to multiple syntaxes, or poetry carefully crafted to also be executable. This proposal does not addess such cases. The usual case, that content is generally not in a natural language if it is program code and vice versa, means that the very same labeling mechanism can be used in both cases so long as the specific codes do not collide. Indeed, this complementary distribution suggests that is desirable.

Therefore this proposal does not suggest any new attributes or other constructs, but an extension to the set of permissible values for existing ones. That is, we propose using contructs such as the lang, xml:lang, and other constructs in the usual manner but with additional values, to identify document portions such as program code. This seems clearly better than leaving such portions unidentified, which leads to poor results from spelling and grammar checkers, inappropriate formatting, and sometimes active corruption of code by applying language-inappropriate corrections or enhancements. It also seems better than the epub solution of labelling them merely zxx or und (see below).

Leveraging existing language indicators has the advantage that inheritance works as expected: for example, if a chapter in Japanese contains a code snippet in Java, setting the language to Java locally overrides the containing language. If a separate indicator were used instead it would suggest it is orthogonal, which it is not. Even if identifiers within that Java code uniformly use Japanese identifiers, the code remains in Java not in the natural language Japanese – its syntax, even down to what strings are permissible identifiers, is defined by the rules of Java not of Japanese per se. Once back in the non-code text the rules of Japanese apply again.

The lang attribute in HTML and xml:lang in XML provide standardized mechanisms for language identification, but they and most other mechanisms use language codes that cover only natural languages and certain variants (plus a few edge cases such as constructed languages). Obvious approaches to having normative names for specific programming or other languages, would be to add them to the ISO 639 space or to separately define non-conflicting extensions. However, because programming languages occupy a different conceptual space, are used by a smaller community, and have a much higher rate of additions, the high level of co-ordination required for such approaches seems impractical.

Instead, this proposal takes a lesson from how ISBNs were incorporated as a sub-space within product codes. Barcodes for retail products begin with a country code. ISBNs were inserted as a single new country called Bookland, with code 978 (see Barcoding Guidelines). This approach neatly separated concerns: Books could do their own thing within their own space (and what they do is use ISBNs), and not trip over other product codes (or vice versa).

Similarly, we propose a single top-level language code, qpr, to cover the space of all programming languages. ISO 639 reserves codes beginning with q for extensions, private use, etc. The particular programming language is then specified by and additional code field, using the already familiar rules of RFC 5646. Because programming languages rarely have regional dialects or variants per se, this shift in the region semantics seems fair. Grouping programming languages under one code makes it trivial to tell whether given content is natural language text or program code. Some processing such as turning off spelling correction or fancy quotes may only need that distinction, while other processing such as syntax highlighting would leverage the more specific sub-codes in the remainder.

There does not appear to be a normative registry of programming languages, although many have language definitions promulgated via ISO, IETF, W3C, or other organizations. This proposal does not require creating a registry (though it also has no objections should one arise). Rather, it suggests using the widely-used de facto codes embodied in file extensions: py for Python, lisp for LISP, and so on.

For practical reasons these tend to be nonconflicting. However, there are exceptions such as dot and m. In such cases more is needed. This proposal thus also includes means for declaring unambiguous mapping of language codes to URLs, as is typically desired for Web vocabularies. We illustrate a mechanism for this in XML that leverages already-existing mechanisms and requires no additions to parsers, DOM implementations, etc. We also give possible approaches to achieve the same kind of mapping in HTML. Similar approaches could be applied to any document format or processing system that supports language identification codes, from markup languages to content management systems and beyond.

Background and Related Work

Proposed Approach

                     
<code xml:lang="qpr-x-py">
def fibonacci(n):
    return n if n <= 1 else fibonacci(n-1) + fibonacci(n-2)
</code>

                     
<data xml:lang="qpr-png-base64">
iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8...
</data>

Resolving Codes to Definitions

An important characteristic of names on the Web is that they eventually map to URLs. This can be accomplished through various mechanisms: normative registries (like ISO 639 for languages, ISO 15924 for scripts, and ISO 3166 for country codes), local declarations (like XML namespace declarations), de facto conventions (like file extensions), or other means.

This proposal defines a single top-level language code qpr to conform to ISO 639, then creates an extension space for particular programming languages. This subspace does not, at present, have a formal registry — our research found no existing ISO or ANSI enumeration of programming language codes, despite the extensive work of ISO/IEC JTC 1/SC 22 on individual language specifications.

However, this proposal provides two mechanisms for associating codes with definitive specifications: (a) strong recommendation to use existing file extensions with their established meanings, and (b) declaration-based association of codes with authoritative URLs. However, this proposal does not itself require use of such declarations (of couse, any particular use of the specification could add such a requirement if desired in its own context).

                     
<!NOTATION png SYSEM "https://www.w3.org/TR/png-3/">

                     
<!ENTITY figure_12 SYSEM "pix/fig12.png" NDATA png>

<!ATTLIST figlink
    href    CDATA    #REQUIRED
    fmt     NOTATION #REQUIRED
    alt     CDATA    #REQUIRED>

&figure_12;

<figlink href="http://example.com/globe.png" fmt="png" alt="A dog."/>

                     
<!NOTATION qpr-x-py PUBLIC "-//Python Software Foundation//Python 3//EN"
                    "https://docs.python.org/3/reference/grammar.html">
<!NOTATION qpr-x-rust PUBLIC "-//Mozilla Foundation//Rust Language//EN"
                    "https://doc.rust-lang.org/reference/">
<!NOTATION qpr-x-js PUBLIC "-//ECMA International//ECMAScript 2023//EN"
                    "https://tc39.es/ecma262/">

<!ENTITY example_1 SYSEM "snippets/ex1.py" NDATA qpr-x-py>

<!ATTLIST pre
    lang    NOTATION #IMPLIED>

&example_1;

<pre lang="qpr-x-py">
    import sys
    print(sys.version)
</pre>

                     
<meta name="lang-qpr-x-py" content="https://docs.python.org/3/reference/">
<meta name="lang-qpr-x-js" content="https://tc39.es/ecma262/">

                     
<style type="text/css">
    pre[lang="qpr-x-py"] { font-family:"Courier", monospace; }
    pre[lang="en-us"]    { font-family:"Garamond"; }
</style>

Implementation Considerations

                     
<!ATTLIST code xml:lang (en|fr|qpr-py|qpr-js|qpr-rust) #IMPLIED>

Example language codes

Usage examples

                     
<article xml:lang="en">
  <title>Getting Started with Rust</title>
  <para>Here's a simple Rust program:</para>
  <programlisting xml:lang="qpr-rs">
fn main() {
    println!("Hello, world!");
}
  </programlisting>
  
  <para>To compile it, run:</para>
  <screen xml:lang="qpr-bash">
rustc hello.rs
./hello
  </screen>
</article>

                     
<function xml:lang="en">
  <funcsynopsis><funcdef>authenticate</funcdef></funcsynopsis>
  <para>Authenticates a user via JWT token</para>
  <example xml:lang="qpr-py">
import jwt
token = jwt.encode({"user": "alice"}, secret, algorithm="HS256")
  </example>
  <example xml:lang="qpr-js">
const jwt = require('jsonwebtoken');
const token = jwt.sign({user: 'alice'}, secret, {algorithm: 'HS256'});
  </example>
</function>

                     
<section>
  <title>Configuration</title>
  <programlisting xml:lang="qpr-docker">
FROM python:3.9
COPY requirements.txt .
RUN pip install -r requirements.txt
  </programlisting>
  
  <programlisting xml:lang="qpr-yaml">
version: '3.8'
services:
  web:
    build: .
    ports:
      - "8000:8000"
  </programlisting>
</section>

Remaining issues

In some cases it is important to specify a particular version number for programming languages. This could be done by defining separate extension codes (say, f66 vs. f90 above), or perhaps better, appending a field specifically for the version number, for example qpr-py-3.11.

Data formats such as CSV, png, and countless others as well as templating, configuration, and other systems, may be considered another category(s) of language. So considered, they are largely complementary to natural and programming languages. Similar declaration functionality might be useful for them as well. In that case, the same approach could potentially be applied (say, with prefix qda.).

The boundaries between programming languages, data representation languages, and markup languages are imprecise. Some interesting boundary cases include XSLT, PostScript, and regexes.

XSD does support the NOTATION datatype for attributes. However, it does not provide an equivalent for NOTATION declarations.

Conclusion

The approach requires no formal standardization or updating of applications for immediate use or adoption. Organizations or individuals can begin using the conventions immediately while building consensus around language code assignments.

This paper presents a backward-compatible approach to programming language identification in documents, that leverages existing standards (natural language tags and XML NOTATION) to provide both machine-readable identification and dereferenceable specifications.

The approach offers several advantages:

By building on existing language identification infrastructure rather than introducing new mechanisms, the approach provides a path toward better programming language support in technical documentation while maintaining full compatibility with existing toolchains. Using the same mechanism also properly captures the complementary use of natural vs. programming language syntax.

The combination of familiar file extension conventions with formal NOTATION declarations strikes a good balance between developer usability and semantic precision, enabling both human authors and automated tools to work more effectively with diverse technical content.

Acknowledgments

The author thanks Claude (Anthropic) for collaborative development of these ideas and significant contributions to the analysis and presentation in this paper.