Balisage Paper: SOCRview: a case study of RESTful service development for publishing

Context and Goals

SAGE is an academic publisher whose content is marked-up in XML and stored in an Content Management System (CMS) known internally as SOCR (SAGE Online Content Repository). Many types of content are stored in SOCR but this paper will focus on journals.

SOCR is a CMS with typical characteristics: content comes in (ingestion, validation), goes out (reports, searches, delivery) and is stored/archived. At its core SOCR consists of two applications: a front-end that provides user access and also contains a workflow engine; and a back-end XML database.

This paper will present how content goes out, is accessed through a service, SOCRview, running on an XML database; starting from initial motivations and an XML database: services, URIs, and a REST framework will be sequentially added to the mix.

XML Database Services

To set the stage for what follows it necessary to understand a little about services written in XQuery. A minimum configuration can consist of specifying a port and location for XQuery files. The examples below demonstrate a simple content query and how to access the requesting URL.

A service is written like an XQuery program where the input context is all the documents in the database, as if the database was one giant root document and each actual document a child of the root document. This example illustrates a service running on port 8123 that returns an arbitrary article. The following is placed in a file, one-article.xqy:

(/article)[1]

http://localhost:8123/one-article.xqy

<article article-type="research-article" dtd-version="1.1d1" r:rsuiteId="6536723" xml:lang="EN">
  <front>
    <journal-meta>
      <journal-id journal-id-type="publisher-id">EPM</journal-id>
  ...
</article>

There was one word excluded from the stated goal of accessing content using URIs: "directly." We want to use the URI directly and not as a unique id in a parameter.

Not like this:

http://localhost:8080/goGoGadget.xqy?uri=/a/b/c

Yes, like this:

http://localhost:8080/a/b/c

This can be achieved by configuring the service to redirect requests. Below is an example of redirecting all requests to XQuery file simple-service.xqy that contains the following:

let $url := xdmp:get-original-url()

return
  if ( $url eq '/one-article' ) then
    (/article)[1]
  else
    fn:concat("URL: ",$url)

http://localhost:8080/one-article

http://localhost:8080/a/b/c

URL: /a/b/c

The XQuery program above, simple-service.xqy, shows how an HTTP service can interpret a request URL and access content. The next step would be to use regular expressions to match the request URL, isolating the object URI from modifiers that will indicate which aspect of the object is to be returned: (e.g. the object itself, its metadata, a transformed version of the object, etc.)

URI - Initial Analysis

The catalyst for introducing URI design for journals came from a 2012 MarkLogic Users Group London (MUGL) meeting where Jeni Tennison presented her technical approach and architecture for UK legislationMUGL2012; in particular the utility of meaningful persistent URIs and how modifiers could be applied to view different aspects of an object. This presentation at MUGL led to combining an analysis of our journal content and examples of how other systems provided URI-based access to journal content into an initial URI design.

http://localhost:8080/rsuite/rest/v1/content/38024?skey=1345

http://aas.sagepub.com/content/25/4/395

http://aas.sagepub.com/lookup/doi/10.1177/009539979402500401

/AAS/25/4/10.1177/009539979402500401

/AAS/25/4/10.1177_009539979402500401

10.1597/1545-1569(1995)032<0206:pfaoat>2.3.co;2

/journal/AAS/25/4/10.1177_009539979402500401

/journal/AJS/44/9/10.1177_0363546515618372.xml

/journal/AJS/44/9/10.1177_0363546515618372/10.1177_0363546515618372.xml

/journal/AJS/44/9/10.1177_0363546515618372/foobar.xml

SOCRview Proof Of Concept (POC)

Full details of the POC would show little but two aspects have bearing on what follows: most importantly it successfully accomplished most of the stated goals and demonstrated what was possible in a way that words by themselves did not; the approach taken to matching an parsing URI was not ideal.

Production SOCRview using RXQ

An alternative approach to matching and parsing URIs presented itself at another MUGL where meeting where Jim Fuller presenting his RESTXQ library which use XQuery function annotations to expose RESTful services in MarkLogicMUGL2014. Jim's RESTXQ library is based on Adam Retter's RESTXQ draft RESTXQspec presented at XML Prague 2012XMLPrague2012.

The RXQ library makes use of XQuery annotationsXQuery3 on function declarations. Every entry point (endpoint) of the service will have function declared as in the example below. This example shows the default behaviour when no URI is provided, the root URI '/', return a static table of contents XML document:

declare
%rxq:produces('text/xml')
%rxq:GET
%rxq:path('/')
function toc() { static:toc() };

Before showing the %rxq:path annotations that would match URI, as proposed above, it is necessary to explain an enhancement made to RXQ. As ubiquitous and powerful as regular expressions are they can be cryptic--especially for complex patterns--and difficult to understand or modify; more, a programmer should be able to look at a regular expression in an annotation and understand the URI it is intended to match. An abstraction layer was added to add symbolic patterns/pattern variables. Pattern variables are defined in a map:

let $m := map:map()
let $_ := map:put($m,'$doi','(10\.\d{4,5}_[^/]+)')
let $_ := map:put($m,'$tla','([A-Z]*)')
let $_ := map:put($m,'$vol','([^/]+)')
let $_ := map:put($m,'$iss','([^/]+)')
let $_ := map:put($m,'$obj','([^/]+\.$objext)')
let $_ := map:put($m,'$objext','([a-z]+)')
...

declare
  %rxq:GET
  %rxq:path('(/journal(/$tla(/$vol(/$iss)?)?)?(/$obj|/$doi)?)($filter)?')
function jrnlObject(
  $socrUri, $_1, $_tla, $_2, $_vol, $_3, $_iss, $_4, $_obj, $_objext, $_doi, $_5, $filter
)
{ uf:applyFilters($filter,_getObject($socrUri)) };

In the above function declaration:

Parentheses in regular expressions are used to isolate sub-expressions and capture text. These capture groupsRegular Expressions are assigned to a corresponding variable in the declared function. In the example most of the capture patterns are not used; only the URI and filter are used. A future enhancement could implement non-capture groups so that only required capture groups are assigned to variables. A future enhancement might also disallow capture groups inside pattern variables so that what is captured can be understood just from reading the %rxq:path.

Using RXQ allows for better organization and maintenance of service endpoints. Functions that match URI with complex patterns can be created that act upon the URI, applying any modifiers.

Views

So far all examples of URI and corresponding endpoints have corresponded to objects (container, non-XML or XML nodes); views are anything the can be derived from an URI and some modifying suffixes. Here are some examples of views:

/journal/AJS/44/9/localDelivery.zip

<deliveryConfig id="localDelivery">
  <pkgList type="xslt" uri="/deliver/localDelivery.xsl"/>
</deliveryConfig>

/journal/AJS/44/9/localDelivery.zip

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  exclude-result-prefixes="xs"
  version="2.0">

  <xsl:template match="container">
    <transform type="zip">
       <xsl:apply-templates select=".//object"/>
    </transform>
  </xsl:template>

  <xsl:template match="object[@type ne 'nonxml']">
    <transform name="{util:getObjName(.)}" type="xqyfn" fn="serialize">
      <param name="addHeader"/>
      <param name="removeRsuite"/>
      <object uri="{@socrUri}"/>
    </transform>
  </xsl:template>
  
  <xsl:template match="object[@type eq 'nonxml']">
    <object name="{@@name}" uri="{@socrUri}"/>
  </xsl:template>

</xsl:stylesheet>

<transform type="zip">
  <object name="10.1177_0218492315603212.pdf" uri="/journal/AAN/25/3/10.1177_0218492315603212.pdf"/>
  <object name="10.1177_0218492315603212-fig3.tif" uri="/journal/AAN/25/3/10.1177_0218492315603212-fig3.tif"/>
  <object name="10.1177_0218492315603212-fig2.tif" uri="/journal/AAN/25/3/10.1177_0218492315603212-fig2.tif"/>
  <transform name="10.1177_0218492315603212.xml" type="xqyfn" fn="serialize">
    <param name="addHeader"/>
    <param name="removeRsuite"/>
    <object uri="/journal/AAN/25/3/10.1177_0218492315603212.xml"/>
  </transform>
  <object name="10.1177_0218492315603212-fig1.tif" uri="/journal/AAN/25/3/10.1177_0218492315603212-fig1.tif"/>
</transform>

Archive:  localDelivery.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
    17111  07-09-2017 17:42   10.1177_0218492315603212.xml
  1737310  07-09-2017 17:42   10.1177_0218492315603212-fig1.tif
   303239  07-09-2017 17:42   10.1177_0218492315603212.pdf
  5352824  07-09-2017 17:42   10.1177_0218492315603212-fig2.tif
  1612126  07-09-2017 17:42   10.1177_0218492315603212-fig3.tif
---------                     -------
  9022610                     5 files

/journal/AAN/25/3/10.1177_0218492315603212/uniqueDoi.rpt

<deliveryConfig id="uniqueDoi">
  <report>
    <function
    fnName="uniqueDoi"
    fnNamespace="http://sagepub.org/socrview/report"
    fnLocation="/modules/report.xqy"/>
  </report>
</deliveryConfig>

declare function uniqueDoi(
  $socrUri as xs:string
, $refxml as node()
)
{...};

<socrUris/>

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0" 
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xlink="http://www.w3.org/1999/xlink"
>

<xsl:template match="*">
  <wrapperOne>
    <xsl:copy-of select="."/>
  </wrapperOne>
</xsl:template>

</xsl:stylesheet>
         

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xlink="http://www.w3.org/1999/xlink"
>

<xsl:param name="id" select="'Default'"/>

<xsl:template match="*">
  <xsl:element name="{concat('wrapper',$id)}">
    <xsl:copy-of select="."/>
  </xsl:element>
</xsl:template>

</xsl:stylesheet>

/journal/AAN/25/3/10.1177_0218492315603212.xml/__filter/xslt/wrapper-one.xsl

<wrapperOne>
  <article article-type="case-report" dtd-version="1.1d1" r:rsuiteId="167632474" xml:lang="en">
    <front>
...

/journal/AAN/25/3/10.1177_0218492315603212.xml/__filter/xslt/wrapper-one.xsl/xslt/wrapper-id.xsl?id=Two&dummy=Null

<wrapperTwo>
  <wrapperOne>
    <article article-type="case-report" dtd-version="1.1d1" r:rsuiteId="167632474" xml:lang="en">
      <front>
...

/journal/AAN/25/3/10.1177_0218492315603212.pdf/__filter/xdmp:md5(binary()).xpath 

12130105eaeaf74a21cbe457b8b70bd0

/journal/AAN/25/3/10.1177_0218492315603212.xml/__filter/xdmp:binary-size(xdmp:unquote(xdmp:quote(.),(),"format-binary")/binary()).xpath

17061

/journal/AAN/25/3/10.1177_0218492315603212.xml/__filter/descendant::abstract.xpath

<abstract>
  <p>A 51-year-old woman with exercise angina and a history of pulmonary artery hypertension ...</p>
  <p>After a multidisciplinary evaluation,...</p>
</abstract>

Summary

The initial motivation of exposing SAGE's journal content through a simplified interface and the goals of building this interface through an HTTP service utilizing persistent, readable, meaningful and succinct URI was achieved. The usefulness of approach has so far mostly been seen in redesigning SOCR as multiple services but the browser interface has also proven popular and useful for technical users in our publishing systems group--who created and maintain SOCR--and the production group--who use SOCR. There has also been a gradual increase content accesses through scripts (e.g. data scientists using Python). URI design for journals has met all stated goals but URI design for non-journal content has been less satisfying because of tendency to view content base on its form (i.e. markup – e.g. TEI, DocBook, etc.) rather than its function (e.g. a book)--a salutary lesson that the time spent thinking about journal URI design was well spent. The use of RXQ has allowed easy additions of new, non-journal, content types. Having multiple levels of configuration for deliveries has allowed simple new deliveries to be created without the intervention of a developer or administrator and complex deliveries, requiring development, to be created faster.