Balisage Paper: Markup to generate markup to generate markup

Packages and Classes

A document class is a collection of macros providing both formatting and markup for a specific class of documents, such as the articles for a particular journal, the books by a particular publisher, the theses for a particular university, or any of over 400 other types of document. It is broadly equivalent to a DTD or Schema, although without prescription, and with formatting specifications embedded. The default document classes (report, book, article, and letter) are stylistically minimalist but provide sufficient markup for draft purposes.

A style package is a collection of macros providing a specific variant on formatting, such as hyperlinks in a PDF, the styling of footnotes or references, the use of additional typefaces, or any of over 3,600 other typographic or markup possibilities. There is no direct equivalent in the XML field, but a package can be regarded as broadly equivalent to a CSS or XSLT2 fragment, implementing a particular formatting requirement.

Document classes and packages are typically distributed as DocTeX (.dtx) files, which contain the LaTeX code implementing the features, interleaved with annotation in a literate programming manner, plus user documentation about how to use the additional markup provided (Carlisle2007). An installer (.ins) file uses LaTeX to extract the code as a class (.cls) or style package (.sty) file from the .dtx file, and LaTeX can then be run on the .dtx file directly to produce both user documentation and code annotation.^[1]

This method has proved a very reliable and compact means of distribution, but at the cost of some additional complexity in the construction of the master .dtx file:

Against this must be set the advantages of robustness once constructed; the availability of all LaTeX facilities for writing and formatting the documentation; some added document-management features (version control, change-recording, checksumming, indexing of commands used in the code, etc), and the extensive supporting documentation (LaTeX2006, Mittelbach2004, Lehmann2011).

Automation requirement

In 2005, I undertook to create a new thesis document class in my university which would implement stricter controls on the content and sequence of front matter (title page, legal, table of contents, declaration of originality, etc), and particularly on the naming and identity of schools, departments and research centers, and the bibliographic reference format used by each. Many users had become accustomed to designing their own title page, and to the re-wording of the names of their unit to suit their own perceptions or requirements. In some cases this involved inventing entirely new department names or descriptions of their degrees, which conflicted with the university's statutory requirements. While the new class would initially only affect the title page and preliminaries of a thesis, this is exactly where the Library catalog staff look for the metadata (in the case of electronic submissions, the PDF metadata is also required to provide the same information).

The data on course names and codes, the abbreviations and full titles of degrees, and the official names of departments and centers were all available from the institutional database, but were subject to annual change, as there were complex and overlapping administrative and pedagogical requirements to be satisfied. This data needed to be converted to the parameter syntax used by LaTeX on an ongoing basis to make it usable as selectable options by users, so a more robust and programmatic solution was needed to automate the process. The data was already available in a consistent XML format, so XML and XSLT2 were obvious candidates for the task. As a long-time user of XML for documentation, I felt it would be an advantage from the maintenance point of view to use the same syntax and method for writing the documentation, and this led to the experiment in using DocBook and generating the .dtx file with XSLT2.

Beyond the title page and the settings for margins and type size, the remainder of a student's thesis document would be largely unaffected, as LaTeX's report document class and existing packages already provided all the facilities needed. However, it had become clear from local LaTeX training sessions that some requirements of thesis writing would benefit from more automation, and that better use could be made of the layout specifications, which were, and remain, relatively lax (Flynn2012), so the decision was taken to experiment with using DocBook for the whole project.

Metadata

A .dtx file is made up of a number of well-defined sections:

The design of a DocBook document does not of course align directly with this, but there is provision in one form or another for most of it, and XSLT2 can easily vary the order of processing. The initial metadata (mostly effectivities) is stored in the book root element start-tag:

<book xml:id="uccthesis" version="1" revision="03" xml:lang="en"
  xml:base="ucc" remap="a4paper,12pt" arch="class" audience="lppl"
  condition="2009/09/24" conformance="LaTeX2e" os="all"
  security="2070" userlevel="cls" vendor="UCC" status="beta">

The info/cover element type was used to hold the document management data, principally the metadata, the lists of packages required by both the documentation and the class or package file itself; a file list for the manifest; and any setup commands for the documentation. The title, author, contact details, Abstract/Summary, and revision history are in the info container in the conventional manner.

Working from the DocTeX and ltxdoc specifications, with existing classes as examples, it was then possible to construct the .dtx initialization block as a literal result template, using the ID and version values from the book element's attributes. The preliminary LaTeX comments and the ‘driver’ block are shielded from processing by a conditional which always evaluates to false:

% \iffalse meta-comment
%
% Extracted from uccthesis.xml
[...licensing and descriptive comment...]
% \fi
% \iffalse
%<*driver>
\ProvidesFile{uccthesis.dtx}
%</driver>
%<class>\NeedsTeXFormat{LaTeX2e}[2009/09/24]
%<class>\ProvidesClass{uccthesis}[2012/12/18 v1.03 Typesetting a UCC thesis with LaTeX]
...
% \fi

Annotated code

The annotated code is stored in chapter elements in a part element with an ID of code. These can be subdivided into sections and subsections according to the modularity and complexity of the code. The annotations get output as part of the formatted documentation: the code gets extracted to the class or package file. The ltxdoc package uses LaTeX \sections as its top level, so a DocBook chapter is mapped in the XSLT to a LaTeX section, a sect1 to a \subsection and so on.

<methodsynopsis xml:id="physio" arch="med">
  <methodname>Vancouver</methodname>
  <methodparam>
    <parameter role="department" remap="Department of">Physiology</parameter>
    <initializer>vancouver</initializer>
  </methodparam>
</methodsynopsis>

%␣␣␣␣\begin{macrocode}
\DeclareOption{physio}{%
  \department{Physiology}
  \@usebib{vancouver}{Vancouver}{}
}
%␣␣␣␣\end{macrocode}

<info>
  <cover>
    <constraintdef xml:id="clspackages" linkend="options">
      <segmentedlist>
        <segtitle>Packages needed for this class</segtitle>
        <seglistitem>
          <seg>fix-cm</seg>
        </seglistitem>
        <seglistitem>
          <seg role="textwidth=159mm,textheight=229mm">geometry</seg>
        </seglistitem>
        <seglistitem>
          <seg>graphicx</seg>
        </seglistitem>
        [...]
      </segmentedlist>
    </constraintdef>
    [...]
  </cover>
</info>

\usepackage{fix-cm}
\usepackage[textwidth=159mm,textheight=229mm]{geometry}
\usepackage{graphicx}
...

Modular code

Code can be given in programlisting elements interspersed with para and other documentary elements of annotation. The amount of annotation and frequency of interruption is unrestricted: the ltxdoc extraction process simply stitches together all the code and outputs it; and the documentation formatting treats the code as verbatim blocks (line-numbered for convenience).

However, the literate-programming format for the uses annotation elements to define the LaTeX commands and environments being provided. The role attribute defines the class of object being annotated, and the xreflabel attribute gives its name. Each such annotation element can contain paragraphs, lists, etc, plus the programlisting code, broken into whatever granularity is needed to explain what is being done.

  <annotation role="environment" xreflabel="epigraph">
    <para>Define an environment for Epigraphs. These would normally go immediately after the
      <command>chapter</command> command. This is basically the <envar>quotation</envar>
      environment modified, but it has to allow for <emphasis>either</emphasis> manual
      <emphasis>or</emphasis> automated citation (because it may just be a phrase needing 
      no citation), whereas a normal quotation <emphasis>must</emphasis> be cited. It 
      therefore has <emphasis>two</emphasis> arguments, described below:</para>
    <remark version="0.92" revision="2011-05-31">Added Epigraphs.</remark>
    <programlisting>
\newenvironment{epigraph}[2][\relax]{%
    </programlisting>
    <para>Record the argument values now, because they are needed in the end of the 
      environment, so they have to pass across the group boundary. The compulsory 
      argument is for a &BiBTeX; citation key, so that a proper citation can be 
      formatted; the optional argument is for when a pre-formed, 
      <wordasword>full</wordasword> (actually often simpler, non-rigorous) citation
      is wanted.</para>
    <programlisting>
  \gdef\@fullcite{#1}%
  \gdef\@quotcite{#2}%
    </programlisting>
 ...
  </annotation>

The remark element is used for noting updates: these get extracted to the revision history. The annotations are output using the armored ltxdoc code; the actual lines of code from the programlisting elements are output unarmored for extraction. This results in LaTeX code in the .dtx as shown below:

% \begin{environment}{epigraph}
% Define an environment for Epigraphs. These would normally go immediate after the
% \DescribeMacro{\chapter}\verb`\chapter` command. This is basically the
% \DescribeEnv{quotation}\texttt{quotation} environment modified, but it has to 
% allow for \emph{either} manual \emph{or} automated citation (because it may 
% just be a phrase needing no citation), whereas a normal quotation \emph{must} be 
% cited. It therefore has \emph{two} arguments, described below:\par
% \changes{v0.92}{2011/05/31}{Added Epigraphs.}
%    \begin{macrocode}
\newenvironment{epigraph}[2][\relax]{%
%    \end{macrocode}
% Record the argument values now, because they are needed in the end of the environment, 
% so they have to pass across the group boundary. The compulsory argument is for a 
% \BibTeX{} citation key, so that a proper citation can be formatted; the optional 
% argument is for when a pre-formed, `full' (actually often simpler, non-rigorous) 
% citation is wanted.\par
%    \begin{macrocode}
  \gdef\@fullcite{#1}%
  \gdef\@quotcite{#2}%
%    \end{macrocode}
...
% \end{environment}

The formatted result in the documentation PDF is shown in Figure 1, where the marginal annotation of the commands being documented can be seen.

Figure 1

The ltxdoc package provides only two documentary environments for annotated code: macro and environment. The dox utility package has been used to provide additional environments for other declarations such as counters, classes, options, templates, etc.

User documentation

User documentation is similarly stored in a part element, this time with the ID of doc. In the .dtx file, the user documentation starts with an unarmored LaTeX Preamble where settings and packages needed for formatting the documentation are specified, followed by a self-reference to the same .dtx file in place of the actual text. This enables LaTeX to read the Preamble and then switch to armored mode to input the same document to process the armored documentation at high speed (doing it all in a single pass would entail a more computationally-intensive process).

%<*driver>
\documentclass[a4paper,12pt]{ltxdoc}
\usepackage[utf8x]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[textwidth=159mm,textheight=229mm]{geometry}
\usepackage{graphicx}
\usepackage{fancyvrb}
[...]

[...]
\EnableCrossrefs
\CodelineIndex
\RecordChanges
\begin{document}
\raggedright
\DocInput{uccthesis.dtx}
\end{document}
%</driver>

<variablelist>
  <varlistentry>
    <term><envar>dedication</envar></term>
    <listitem>
      <para>The <envar>dedication</envar>
	environment is for you to add a dedication.</para>
      <programlisting annotations="dedication" language="LaTeX">
\begin{dedication}
...
\end{dedication}
      </programlisting>
    </listitem>
  </varlistentry>
[...]

% \item[Dedication:] The \texttt{dedication} 
% environment is for you to add a dedication. 
% \iffalse 
%<*ignore> 
% \fi
\begin{lstlisting}[language={[LaTeX]TeX},emph={dedication}]
\begin{dedication}
 ... 
\end{dedication} 
\end{lstlisting} 
% \iffalse 
%</ignore> 
% \fi

Development from pilot to production

The original thesis document class was successfully implemented, and the XML-based system as described is used to maintain it. The 50 or so class options specifying department and degree are used to simplify and rationalize the setup for the department name, title-page layout, and style of references, while the class itself presets the rest of the formatting; see Figure 2).

Figure 2: Thesis set-up

\documentclass[history,phd]{uccthesis} \begin{document} \title{The Application of XML to the Lexicography of Old, Middle and Early Modern Irish} \author{Julianne Nyhan} \qualifications{BA} \professor{Prof Dermot Keogh} \supervisor{Prof Donnchadh Ó Corráin} \date{June 2005} \maketitle ... \end{document}

However, that class was a pilot: the result is that this XML-based mechanism is usable for the creation and maintenance of almost any LaTeX class or package. The system is used for all the author's classes and packages, and has significantly reduced development time on a new class or package. In the development of additional classes or packages in a series or suite (such as occurs in corporate use) the reduction is greater because of the ease and reliability with which modules of code can be included (as entities or XIncludes). The reuse of imported data specifications also has an important place in industrial documentation, where sets of part numbers or known production components need to be pre-specified, and the system has now been adapted twice to use this method.

Markup load

Many of the templates in the XSLT2 program make decisions about the markup they should emit according to the content of the element type they match. As an example, a firstterm element type can be made to identify from its position if it is indeed the first occurrence, and if so, to add a bold LaTeX \index entry rather than a plain one. The careful author can add an attribute to suppress this behaviour in cases where a first or early occurrence may be used en passant.

In a more complex environment, such as a footnote or the term element of a variablelist containing code requiring a monospace font and LaTeX's verbatim formatting, the template will choose not to use LaTeX's \verb command because of its fragility inside other markup, and to use \texttt (simple monospace) instead, or even \url, according to content. This is something which would otherwise require the author to remember that certain special characters cause LaTeX problems when treated verbatim.

Cross-references which cannot be automated by LaTeX's otherwise excellent varioref package (such as references to an unnumbered list item, where by definition no reference number exists) are pre-empted in the XSLT2 code and the reference switches to the fmtcount package, which phrases a counter value as a spelled-out ordinal: see the third item in the list on p.42.

The objective in all these cases is to relieve class and package authors of the need to work manually around LaTeX's oddities and allow them to write unhindered, for example, by the need to remember that such-and-such a reference was to a table, or a figure, or a subsubsubsection, or a call-out; and to have the reference auto-adjust its semantics if the target element type gets changed.

As an example of the use of markup, the formatted annotation output (code documentation) usually requires a wider left margin than the user documentation because code fragments are identified by a marginal note showing the LaTeX command name. In order to accommodate the widest name used, a new value for the margin is calculated in the XSLT2 program, using the longest value of the various commands explained in the annotations. This ensures that an unexpectedly long command name will not extend beyond the left-hand edge of the page. This calculation, straightforward in XSLT2, would be computationally challenging in LaTeX and would need to be written to use the second pass of the document normally associated with LaTeX tables of content and cross-references. This calculation can therefore be done first, before processing the content of the part element for annotated code.

The use of XML also makes it straightforward to query the document structure for control purposes. For example, using standard command-line tools such as the LTxml toolkit provides, a list of macros and environments defined can be extracted, or a list made of the packages used:

$ lxprintf -e annotation "%s (%s)\n" @xreflabel @role uccthesis.xml | sort
ackname (macro)
acknowledgements (environment)
author (macro)
bibliography (macro)
bibname (macro)
cjk (option)
dedication (environment)
department (macro)
draft (environment)
epigraph (environment)
...
$ lxprintf -e \
'constraintdef[@xml:id="clspackages"]/segmentedlist/seglistitem' \
"%s\n" seg uccthesis.xml
inputenc
fontenc
geometry
lmodern
url
graphicx
array
calc
soul
textcomp
ucccrest
setspace
float
$

Tag abuse

We said earlier in section “Annotated code” that some element types have been used for purposes not envisaged by DocBook, and that part of this experiment was to identify what the nature of these use cases in class and package maintenance was likely to be. As there are areas of DocBook into which the present author has never had need to stray, suggestions are welcomed for element types with a better fit. A future task is to write an RNG specialist modification layer for the DocBook schema to create some additional element types to avoid the current level of abuse.

At the moment, the XSLT also generates a shell script file which can be used to build the relevant LaTeX distribution package (a specially-formed zip file). This needs to be replaced by a parameterised Makefile, using the latexmk script.