Vast collections of information are stored in HTML files distributed over millions
pages through the Internet. Among these quite valuable data can often be found; however,
does not offer a large pool of semantically motivated elements or attributes for annotating
arbitrary data, since the language was originally created for hypertexts. Although
microformats Suda (2006) may be used to add semantic value to structuring
span), most information is buried underneath a
"tag soup" of
div elements that allow no
inference about their content. In contrast, we can have information that is highly
terms of very specialized XML markup using a document grammar (DTD SGML, ISO 8879:1986,
XML 1.0, XSD XML Schema Part 0: Primer or RELAX NG RelaxNG, ISO/IEC 19757-2:2003)) that allows for easy retrieving of very specific information. A real
world example where the origin of our data is a collection of (sometimes even invalid)
HTML 4.01 Web pages storing documents of video game reviews is a good
candidate for demonstrating how value can be added through better markup. Our goal
transform these into fully structured and valid XML instance documents that allow
queries about the information. Since we are confronted with several hundred reviews,
conversion process is valuable. As an additional goal, we would like to stay in the
realm of XML
techniques; for example, we would like to avoid using non-XML-aware software such
general-purpose scripting languages (e.g. Perl, Python).
Video games are a part of today's culture and are available in a huge variety in terms of supported game system, genre and — of course — quality. Finding a game that fits both one's hardware requirements and favored genre is a relatively easy task to accomplish, but basing the decision to buy a specific game only on the text written on the back of its case is daring at least. Impartial (more or less) reviews of video games may help to clarify if the money is well spent in the long run by providing rating systems for features such as graphics, sound, atmosphere or overall score (usually higher scores are better). The team of the German Mag'64 Web site  has tested video games for over eight years, gathering over 1500 reviews, each consisting of a single HTML Web page. Each document contains information about the game being tested, the review, including a general judgement, and images and screenshots. This information is quite valuable since among the provided items are general ones such as the title, system, or publisher, but in addition more specific items such as number of players, genre, age rating and difficulty. The review consists of running text' while the final verdict and pros and cons are summarized in a tabular view. The data we have to deal with consists generally of two types of reviews, which we call "Type A" and "Type B". Type A was used during the years 2001 through 2004, while Type B was introduced in the Autumn of 2004.
From a technical point of view the data is stored in HTML Web pages. Because HTML's
original task is to structure hypertexts, it lacks specific elements and attributes
annotating the information we are interested in. Furthermore, the markup of our test
very focussed on presentation, that is, general HTML elements such as
td are used for physically structuring the information according
to a given layout. While the two review types, A and B, do not differ regarding their
information content, there are differences in the markup techniques used.
The Type A review was originally used as part of an HTML frameset. While one frame
contained a menu for navigating through the whole service, the second frame stored
review in the form of a HTML Page. This page lacks an HTML Doctype declaration, and
copy and paste errors can be found, including end tags without preceding start tags,
attributes, etc. The
img element for embedded graphics lacks the required
alt attribute.  Furthermore, no information about the character encoding is given, which leads to
encoding errors since German umlauts and other special characters were used.
Figure 1 shows an excerpt of an Type A review.
This markup we have to deal with is very presentation-focussed: semantic markup such
h2 that could be used for structuring the text is not used at
all. The title of the game can only be found in the running text or in the graphic
referred by the
img element — and sometimes in external cheats or tricks documents
that are referred to from the review page (the term "CHEATS: JA" in Figure 2). All useful information is buried deep inside HTML's
table elements, and the
page lacks any
meta elements for storing additional information. Spacing between
different parts of the text was introduced by using HTML's entity, while the
markup is layout oriented, using
Sometimes font elements with identical formatting options are embedded into each other
resulting in a tag soup. Emphases are arranged solely by selecting "size 3" fonts.
The running text of the review is distributed among different
establishing a print-like layout. Each review begins with two blocks containing
meta-information, such as system, genre, number of players, etc.
The Type A review ends with a tabular overview, consisting of the "pros" and "cons" of the game.
The Type B reviews were established in the Autumn of 2004, coinciding with the release of the Nintendo DS® handheld console. Since this videogame console introduced some features that were unknown before (e.g. split-screen and the stylus input device), a new HTML template for reviewing video games was adapted. As a new meta-information item, an age rating was added, and the running text was subdivided by headings.
Most of the HTML pages contain a doctype declaration (incorrect for HTML 4.01), a
reference to an externally declared CSS stylesheet and information about the character
(ISO-8859-1 — although the specified encoding is sometimes not correct, since some
are encoded using the Windows-1252 charset or even UTF-8). In addition to the external
file, local formatting using attributes such as
border can still be found. In general, the HTML pages are not valid according
to the W3C validation service. Figure 3 shows the mixture of different
formatting options used.
A positive difference from the Type A is the fact that the title of the game appears
(together with the platform it was released for) in HTML's
Important information such as price or age rating are hidden inside a single
element (Figure 4), divided by line breaks.
In contrast to the Type A reviews, subheadings are included; however, these are not
up by HTML's inherent
h6 elements but by using formatting
elements such as
Both review types show HTML's inherent lack of support for highly structured data. Although our example application deals with document-centric texts, the data under observation contains important information that should be marked up explicitly.
Highly structured data
Our goal is to create an XML markup language capable of structuring the video game
of both Type A and B that have been discussed. This format should be used as representation
format for the output of the conversion process that will be presented in the section “Upconversion” and could be used as a storage format for future review applications.
Since we have already stated the input documents are often invalid (sometimes even
well-formed) and important information is buried inside HTML
table elements, having
a document grammar for both validating the conversion process's output format and
explicit markup of the important information is quite important for us. For these
use of a capable of full text search engine was not taken into account. We have chosen
in favor of XML DTD because of its datatype library and especially for its support
user-defined simple and complexTypes Walmsley (2002). A RELAX NG schema (in
combination with the XML schema datatype library) would have been another option,
broader support for XML schema supplied by the XSLT processor used during the conversion
tipped the scales for us (Figure 5).
Each game can be identified by a unique
xml:id attribute, further optional
attributes correspond to
subgenre, supporting an enumerated
list of possible values which should help avoiding typical errors such as typos. Children
game element are the
platforms elements, the
latter consisting of at least either one
videoGameConsole, allowing to combine reviews of the same video game released on
multiple platforms . Both elements are derived by extension of the globally declared complexType
consoleType, sharing common information present in stationary and handheld game
consoles (see Figure 6 for a graphical overview of the shared
release element stores information about the date of release (using an
xs:date Type Attribute), the different languages and price. Children of the
languages element are
handbook elements, depicting information about the parts of the game that have been
price element has a
currency attribute that uses an
enumerated list of possible values according to ISO Country Codes, ISO 4217:2008.
image element can be used to represent box pictures or screenshots
of the game reviewed.
As mentioned above, the
elements are derived from the complexType
consoleType by extension. Although the
saving use the same names, their
content models are different with respect to the video game console, since, for example,
requirements for storing save games are different between handheld and stationary
videoGameConsole element allows for the
child element. Most of these elements use enumerated lists to eliminate possible typos
ease the acquisition of new reviews.
The main part of the review is stored underneath the
review element that
consists of the
conclusion elements and further optional
screenshots and that has a
date attribute and an
group. The running text is subdivided into optional headers and paragraphs, allowing
grained division of text parts and representing both review types.
conclusion element is used to store both further text (e.g. in a form of a
final verdict similar to the Type B reviews) and the tabular-like lists of pros and
followed by the final
score element. Scoring can be expressed either via numeric
values (using the
percent child element with its attributes
multiplayer (optional) and
through text, since both variants can be found in our sample data.
This grammar can not only be used to store the information coded in both review types but also is highly flexible for future extensions. Possible future extensions of the schema may include XSD 1.1 assertions, for example, to ensure that multiplayer scoring information is only allowed when the maximum number of players is greater than "1".
Our upconversion process begins in the typical manner by using XSLT 2.0 / XPath 2.0 Kay (2008). Because it requires multiple steps and must be applied to many files, we have encapsulated it in XProc.
XSLT 2.0 benefits
In his paper "Up-conversion using XSLT 2.0" Michael Kay points out the great advances XSLT made when shifting to XSLT 2.0, and he provides a real-world example that makes heavy use of the new features. The key features which produce benefit for upconversion are in short schema-awareness, support for regular expression processing, better manipulation of strings, and advanced grouping possibilities. So tasks that formerly were often solved by using a general purpose scripting language like Perl or Python, by loading XML modules can be done equally well or better with XSLT 2.0 [See Kay (2004) for an elaborated example]. Our upconversion of the reviews mostly makes use of regular expression processing and string manipulation.
The documents are preprocessed into well formed XML using HTML Tidy.  For the upconversion, both functions as well as named templates are used widely.
following snippet demonstrates the massive clean-up the stylesheet performs. It is
main template, which uses a variable to hold the string with
information about the genre of the reviewed game (Figure 7). This
string is checked for both Type A and Type B data equally but it is applied differently
respect to the structure.
This variable is then checked against regular expressions to assign the respective
value from the defined enumerated list. Figure 8 demonstrates the
assignment of some genres and a sub genre, implemented using case differentiation
advantage of the order of the test expressions.
Because the data varies a lot throughout the transformation, many case
differentiations are used. To find the title of some documents information stored
documents has to be taken into account. In Figure 9, a linked "cheats"
or "tips" document is accessed to extract the game title that is hidden in the backlink
Throughout the transformation many more requirements are met in carrying out the
upconversion. The examples above are simply illustrative of the process without going
Pipelining with XProc
XProc a new standard for automating processes like ours through an XML pipeline has been developed by the W3 working group XProc. It has reached the status of W3C Recommendation on 11 May 2010 after being advanced to Proposed Recommendation in March 2010. The specification had been downgraded from Candidate Recommendation to Working Draft again in January to solve some issues. It has reached a fairly stable level now, and a book on XProc by Norman Walsh is in progress. For our desired all-in-one XML solution, XProc is first choice to handle the pipeline.
The pipeline should process the documents that are stored locally in the filesystem recursively (Figure 10). There are documents other than game reviews (e.g. cheats and tricks), and we need some of them to extract the titles of games, but most of these documents are discarded. One problem here is that while we can say from the filename what is most likely not a test, but not what actually is. The pipeline will apply the following tasks to each HTML document:
Use HTML Tidy to transform the HTML input into well-formed XML
Apply the XSLT script to the output of the former task using an XSLT 2.0 processor
Validate the output files according to the XML schema
Separate valid from invalid documents
Provide a log of valid documents
p:declare-stepas root element for good control of input and output ports. Both are set to allow any number of documents. Since parameters are to be used for XSLT transformation, we need the optional input port "parameters" - because it is the only parameter port in the pipeline it is primary by default. The source directory HTML is bound to a variable and made accessible for the step
p:directory-list, which here returns the system-folders in c-namespace (Figure 12). To advance deeper into the structure we use nested
p:for-eachloops; of course, the output port needs to be set to accept sequences. Next we list the subdirectories, consisting mainly of game-folders (Figure 13). Now we loop over the game-folders (not shown due to space restrictions) and prepare the files for accessibility. First we add the base-uri to get the complete filepath using
p:make-absolute-uris. Then we add slashes using
p:string-replaceto ensure accordance to the file protocol. To make sure the file is accessible for the
p:http-requeststep we rename the element
c:request. Furthermore, we need to add the proper attributes for the
p:http-requeststep to work. Since there is no server involved and we do not want to work with binary data, we need to add the attribute
override-content-typeand attach the value
text/html(Figure 14). Now we can process the HTML documents in sequence. We use a filter to exclude documents which are not reviews and will not help us to find game titles (Figure 15). These documents may be reader reviews that follow no certain structure, hardware reviews, or other texts. Files that may help us to find missing game titles contain these abbreviations:
opt|chea|tipp|herz|guid|pass. For the filtered documents, the second and, therefore, the main part of the pipeline is initiated (Figure 16). If something goes wrong during the upconversion, we want to be able to check in which step and what the reason may be, so each of the main steps has its output stored apart from each other. We nest
try-catchclauses to ensure the flow of the pipeline. The variable
fileholds the URI of each file. It will be available throughout the loop and not only serve to get each file but to store each file in its given folder. So first we convert these files that pass the filter through HTML Tidy via
p:exec, which can take non-XML input and provides safety (Figure 17). We could use
p:unescape-markupin conjunction with Tagsoup 1.2 or HTML Tidy as an alternative solution here, but as XML Calabash so far only implemented Tagsoup for reading HTML and the results from HTML Tidy and Tagsoup differ slightly, we stick to
p:exec. Calumet supports both HTML Tidy and Tagsoup for this step, but as we are using XPath 2.0 we cannot use this option. We set
source-is-xmlto false and
result-is-xmlto true. By default, result lines are wrapped, and the output of this step is also wrapped to ensure wellformed XML documents on the output port. We negate
wrap-result-linesand unwrap the output of the step. (Note that the arguments for HTML Tidy need to be in a single line.)
The output of this step is saved to folder "Tidied" as "filename.xml" and chained
p:xslt. As a precaution, this step along with the connected saving
procedure is encapsulated into a try group. If any of this fails, we record the tidied
the folder "Transform-failed". The
p:xslt step takes three input ports, one for the
stylesheet, one for the XML document and one for parameters (Figure 18). The filepath needs to be provided to the stylesheet to ensure reaching the documents
will be consulted for missing titles. The filename and system folder are processed
transformation as well.
If the transformation and the saving process can be executed successfully, the output
this step serves as input for
p:validate-with-xml-schema (Figure 19). Depending on the output of this step, the documents are
saved separately. Valid documents can be found in the 'Schema-Valid' folder and the
the 'Schema-Invalid' folder. (During the programming of the XSLT-Transformation, invalid
documents give hints for expressions in need of improvement.)
The last steps of the pipeline follow after the loops and take the result of the
started in Figure 13. Here we create an XML document which takes the
c:result elements returned by the step
directoryloop and lists them
for an overview (Figure 20).
This pipeline takes approximately half an hour to process the data, and is relatively independent of CPU speed on an average actual system. It results in 1573 schema-valid files.
The result of the upconversion process
Figure 21 shows an excerpt of an instance coded in the target output format according to the XML schema. The critical information is marked up with the help of appropriate elements or attributes. Conversions of a game (i.e., the release on different platforms) are supported, as well, by separating the general information such as title and genre from the platform for which the review is written. The verdict contains the list of "pro" and "con" items and the score (depending on the input review type, subdivided into single figures for game graphics, sound, multiplayer and overall) in a highly-structured form that allows easy access to relevant criteria.
Benefits of highly structured data — searching for the game according to your flavour
The result instances of the automatic upconversion process discussed in the section “Upconversion” contains highly structured information. All relevant and important data
that was formerly hidden inside HTML's
table element or as part of the running text
can be accessed via XPath or XQuery expressions Chamberlin et al. (2004), allowing for
easy retrieval of reviews of games of certain types or according to certain criteria
genre, price, and score. While the original structure of the Mag'64 Web site offered access to the review based on either the video game system
or the name of the game, a full-text search engine was not implemented. We have
developed some sample XQuery queries that allow for a different kind of retrieval
Alternative access to the reviews
genres.xq uses two parameters, genre and platform, to search for
games of a certain genre on a specific platform by using a collection of all valid
documents. Figure 22 shows the output of the
with the value "Wii" for the platform parameter and the value "Puzzle"
supplied for the genre paramater. Since this query was originally developed as a alternative
access mechanism, the information returned is very sparse. However, in combination
(X)HTML output containing hyperlinks to the respective review page, it would be
Finding a game according to specific features
Sometimes a user searches for games that support certain technical features, such
content, multiplayer, etc. The
techspecs.xq query uses the parameter platform and
techspec to retrieve only the reviews of games that include the provided feature.
Figure 23 shows an example result.
A more elaborated example: a wish list
Kids love video games these days, and often they leave their parents behind when it comes to choosing the right game for a present. We will demonstrate the benefits of highly structured data in this example. Consider a seven-year-old child with a Nintendo DS® who wants to get a racing game for his system. The parents might agree but formula additional constraints: the game to be bought should have a score of at least 70% and should be appropriate for kids of his age. Furthermore, the difficulty should not be too high.
For this query different parameters have to be taken into account: the platform, the
age rating, score, and difficulty. The
shoppingList.xq query provides all these
parameters (Figure 24). Using Saxon as XQuery processor with the
following call results in the output shown in Figure 25.
The results are sorted according to the score in descending order (with 100 representing
the best value). Each
instance element contains the age rating, score, and
information about the difficulty, encoded in attribute values. Child elements are
the title and
the review notes, consisting of the "pros" and "cons" of the game. The
element, in particular, may contain information that is subjective; it may occur that
example parents will judge a certain feature higher or lower than the reviewer did
think of a "con" as a "pro").
The results of our work are of many kinds: first, the newly introduced features such as regular expressions and string manipulations qualify XSLT 2.0 as a full-fledged conversion tool for transforming weak structured data into a highly structured format. Second, if a transformation process has to be carried out multiple times and if other processing is involved, automation by using the XProc pipelining language is highly recommended. Both the XProc specification and the supporting software tools are ready for a productive environment. Furthermore, the output of the upconversion clearly shows a high potential in terms of flexibility and of the ability to retrieve certain information, as shown by our example applications using XQuery.
We are certain that minor problems such as the one caused by the character encoding will be fixed during the ongoing development of XProc software. From our point of view, future modifications could result in a XSD 1.1 compatible XML schema supporting more video game systems or textual content that is not review related, such as cheats, hints, or walk-throughs. Both the XSLT script and the XQuery queries could be modified in how they interact with each other. For example, the distinction of different cases that is carried out by the XSLT script could be reformulated as pipeline step, allowing for a more maintainable XSLT script.
In general, the realization of the pipeline and query system as a Web service in conjunction with a native XML database would result in an alternative search and retrieval mechanism that would indeed search for the game according to your flavour.
[Chamberlin et al. (2004)] Chamberlin, D., D. Draper, M. F. Fernández, M. Kay, J. Robie, M. Rys, J. Siméon, J. Tivy, and P. Wadler, XQuery from the Experts: A Guide to the W3C XML Query Language. Pearson Education. Addison-Wesley, Boston, 2004.
[HTML (ISO), ISO/IEC 15445:2000] Information technology — Document description and processing languages — HyperText Markup Language (HTML). ISO/IEC 15445:2000, International standard, International Organization for Standardization, Geneva, 2000.
[ISO Country Codes, ISO 4217:2008] Codes for the representation of currencies and funds. ISO 4217:2008, International standard, International Organization for Standardization, Geneva, 2008.
[Kay (2004)] Kay, M. "Up-conversion using XSLT 2.0." http://www.saxonica.com/papers/ideadb-1.1/mhk-paper.xml, 2004.
[Kay (2008)] Kay, M. XSLT 2.0 and XPath 2.0 Programmer’s Reference. Wiley Publishing, Indianapolis, 4th edition, 2008.
[RelaxNG, ISO/IEC 19757-2:2003)] Information technology - Document Schema Definition Language (DSDL) — Part 2: Regular-grammar-based validation — RELAX NG. ISO/IEC 19757-2:2003, International standard, International Organization for Standardization, Geneva, 2003.
[SGML, ISO 8879:1986] Information Processing — Text and Office Information Systems — Standard Generalized Markup Language. International standard, International Organization for Standardization, Geneva 1986.
[Suda (2006)] Suda, B. Using microformats. O'Reilly, Sebastopol, CA, USA, (2006).
[Walmsley (2002)] Walmsley, P. Definitive XML Schema. Prentice Hall PTR, Upper Saddle River, NJ, USA, 2002.
[XML 1.0] Bray, T., J. Paoli, and C. M. Sperberg-McQueen, Extensible Markup Language (XML) 1.0. W3C Recommendation 10 February 1998. World Wide Web Consortium. http://www.w3.org/TR/1998/REC-xml-19980210, 1998.
[XML Schema Part 0: Primer] Fallside, D. C., and P. Walmsley, XML Schema Part 0: Primer Second Edition. W3C Recommendation 28 October 2004, World Wide Web Consortium. http://www.w3.org/TR/2004/REC-xmlschema-0-20041028/, 2004.
 The optional merging of different game instances can be carried out by an XQuery script.
 We will need
p:http-request, although we work on the filesystem. This is
p:data, which one could expect here, is not a step and therefore does not