Nordström, Ari. “Eating Your Own Dog Food.” Presented at Balisage: The Markup Conference 2019, Washington, DC, July 30 - August 2, 2019. In Proceedings of Balisage: The Markup Conference 2019. Balisage Series on Markup Technologies, vol. 23 (2019). https://doi.org/10.4242/BalisageVol23.Nordstrom01.
Balisage: The Markup Conference 2019 July 30 - August 2, 2019
Ari Nordström is a Senior Content Architect (fancy speak for "markup geek") at
Karnov Group, a Scandinavian legal publisher. He is based in Göteborg, Sweden,
but has been known to provide angled brackets across a number of borders over
the years.
Ari is the proud owner and head projectionist of Western Sweden's last
functioning 35/70mm cinema, situated in his garage, which should explain why he
once wrote a paper on automating commercial cinemas using XML.
XML and markup technologies are a wonderful thing, but only lately has the author
had the confidence to really and truly eat his own dog food, trusting entirely on
the technologies familiar to him. He's learned to ignore the calls for real
programmers while discovering how to copy and paste from
Stack Overflow and the like in moderation. XForms, XQuery, XSLT, XSL-FO, and a
proper XML database might just be all you need, at least if you know how to Google
and ask for help.
This, then, is nominally about building a conference registration application
ensuring that we get paid for every delegate, but really all
about the fact that we can now rule the world without bothering with the
programmers.
If you're a programmer, all kinds of things are available to
you.
Or that, at least, is how it all appears to me when I peek in from the outside. You
have a requirement, a list of specs, something, and you use magic
to invoke that requirement, list of specs, or something. There's the infrastructure
to
do the cool things under the bonnet, there's a graphical user interface that presents
a
selection of those cool things to your end user and allows her to interact with them,
and there's a processing layer that does all the things the user said your application
should.
I always found this awesome.
Me, I was always a lowly markup geek. I had the tools to produce the basics of those
cool things—the schema to govern the input data, the stylesheets to transform that
data
to something else, the queries to locate things—but I could never bring them all
together to give you, the end user, everything. At least not without becoming a
programmer.
Until fairly recently. Now, at long last, markup technologies promise, and, in my
mind, deliver change.
An Itch to Scratch, Pt 1
At Balisage 2012, I presented a paper (id-xml-for-xml) where I
suggested a way to use an XProc pipeline—then a brand-new technology promising to
bring together all of those cool markup technologies—to generate a customised user
interface for user-specified XProc task. It wasn't much; it merely demonstrated my
stated independence from a bunch of programmers at work to
configure and carry out publishing processes without having to ask them for help.
It
was an eye-opener. I insisted I wasn't a programmer, equating programming with
C# and Java, but the one thing I
remember from that presentation is a member of the audience stepping up to the mic
to say but you are a programmer.
An Itch to Scratch, Pt 2
Fast-forward to April, 2018. I'm in a hotel room in Copenhagen after my very first
day on a new job, and I'm panicking. Two months earlier, we had announced the very
first edition of Markup UK, a conference on XML and markup, to
be held in June, less than two months away. We were a bunch of happy amateurs with
ideals, so while we had plenty of connections and the promise of a couple of
interesting papers, we had no way of getting paid.
So I had a think. What would it entail?
People registering their info.
People paying.
For #1, I immediately thought XForms and eXist-DB. I had used both
in the past, the former starting with the 2012 Balisage paper about that XProc-based
UI and the latter for many clients requiring storage for their XML data. A match
made in heaven!
For #2, I panicked again—I had no idea of how to connect XForms and eXist-DB to a
payment service, thinking it took a programmer. Then I realised
that the other way to do it was to simply send an invoice to the delegate and have
others worry about the money reaching our bank account.
This I knew how to do: Enter the invoice data in an XForm, store the output in
eXist-DB, apply XSLT and XSL-FO, both cool XML technologies, on the saved data to
generate a PDF, and finally use XQuery—another cool XML technology—to email the PDF
to the delegate, using the email address provided by the form.
My somewhat crude registration app was running just a few short days later. People
were able to register, we had a successful first conference—and someone else had to
worry about actually getting paid.
OMG, It's an Eczema!
Surprisingly, the registration problem didn't go away for this year's Markup UK.
The person worrying about us getting paid complained. Really, he said, this was no
way to live. I agreed. Getting paid there and then, when registering, seemed like
a
reasonable addition to the app.
Now, how hard could that be?
Getting Paid
While crude, the Markup UK 2018 registration app was a success. We got paid, in the
end, and the conference was a success. I had proved that the XForms/eXist-DB/XQuery/XSL
combo worked. Rather than start anew, I figured I would build on the previous
app.
Having looked at various online payment options, it seemed to me that PayPal was the
easiest by far. If their site were to be believed, hooking PayPal to our registration
page was a matter of copying and pasting, a time-honoured way of developing code.
A look
at PayPal's developer documentation suggested the following registration flow:
The delegate enters her details in the form and hits
Register. This saves the registration information, and
generates a web page summary with the registration info and a PayPal
button.
The delegate hits the button and PayPal takes over, providing the
PayPal experience,[1] going through the payment either by logging in or by entering her
credit card details directly. PayPal provides both options.
Once done, PayPal confirms the payment and returns the delegate to a
predefined thank you page. That page also saves the transaction information in
eXist-DB.
A complicating factor is that we need to charge several different amounts. For
example, there is an early-bird rate, a student rate, and
so on, but we also need to be able to define discounts for various purposes such as
large groups.
The XForm
The registration form is built using XForms and looks like this:[2]
Figure 1: Registration Form
It's as simple as they come, really, with a couple of text fields for name and
address, an email field with a basic email format validation, and a long list of
countries fetched from an XML lookup file. There is also a mechanism for registering
VAT for companies, with logic to handle the differing requirements depending on
country, a drop-down with the available registration types, and a text field for
entering a discount code. The types are fetched from a config file with prices
including and excluding VAT, and any entered discount is checked against
approved codes in the config file (more on this in section “The Admin Page”).
Hitting Register calls an XQuery that saves the registration
information in eXist-DB and runs an XSLT that produces a summary page with the
registration information and a PayPal button.
The Button
At first, I toyed with the idea of embedding a PayPal button directly in the
XForm, but it seemed to me that it was undesirable to mix XForms with HTML Forms[3]. I decided to embed the button on a registration information summary
page instead.
Version One
The very first version of the PayPal button took me all of fifteen minutes to
copy, paste, and add to the summary HTML.[4] The code example I grabbed from the PayPal site added
script tags:
Unfortunately, after spending some time trying to add the amount to a hidden
variable, I decided to read the documentation and realised that was one of the
things you couldn't add.
Version Two
As an alternative, Paypal offers an online tool that can be used to generate
custom PayPal button code. All you have to do is define your parameters,
including amount, currency, etc, hit Generate, and this
sort of thing will result:
While easy enough to use, this resulted in a fair number of buttons; for
example, a £150 amount payable would require generating two sets of code, one
for the sandbox and one for the live environment. Every new amount, then, will
result in two additional pieces of code, one for testing and one for
live.
The generated buttons also didn't really solve the problem with custom
discounts—we'd have to generate a button for each and every discount when the
need arose. Nevertheless, I wrote some XQuery that used XPath expressions to
pick the right button element from a button lookup XML document
containing every possible button type and environment combination. Here, for
example, are the buttons required for the sandbox:
This was cumbersome, to say the least. Thankfully, I found a third
option.
Version Three
In the documentation, there is a reasonably thorough, step-by-step description
of how to integrate PayPal into your website, complete with code examples in a
number of languages[5] (see PayPal Checkout - Basic Integration). What tickled my imagination was that the example
code included an amount:
<script>
paypal.Buttons({
createOrder: function(data, actions) {
// Set up the transaction
return actions.order.create({
purchase_units: [{
amount: {
value: '0.01'
}
}]
});
}
}).render('#paypal-button-container');
</script>
Now, I don't do JavaScript so a lot of what's in there is magic to me,[6] but clearly, it should be possible to express 0.01 as
a variable. Exciting! Shouldn't it be possible to inject
some XQuery into that thing, like so:[7]
Following the example, I added return pages (the
actions.redirect() calls) and various other stuff that made
testing and deploying the code easier (see section “Sandbox Vs Live” and
section “The Admin Page”).
This is what my JavaScript/XQuery end result looks like:
Note the custom: {string($uid)} key-value pair that sends the
user ID of the registration as a custom variable to PayPal; this is returned to
the summary page (thanks.xquery, above) when confirming, along with
the PayPal transaction ID.
And this is actually most of what we need, except for a tiny little detail
where we find out if the payment was successful.
How Do I Know They Paid?
The PayPal documentation hints at (well, screams, if you are
a programmer) ways to find out if the payment went through. The
basic flow is something like this:
You click the PayPal button and complete the payment on PayPal.
PayPal sends a confirmation, known as the IPN (Instant Payment
Notification).
A listener catches the IPN, and sends back an empty 200 POST followed by a
string and a verbatim copy of the IPN.
The rest is really about logging the transaction details in eXist-DB, generating
an invoice, and sending a confirmation email. I thought that once I get hold of that
IPN string, the rest should be a matter of writing some XQuery[8] and closing the deal.
So, what's a listener, anyway?
You Talk and I Listen
A listener, apparently, is something that
listens to incoming calls and, when catching one, does
things to the call and responds in some predetermined way. A
call, in this case, is an http POST request that we, after
acknowledging it with an empty POST, return to the sender just the way we found it.[9]
And this is where my ignorance started to show.
PayPal, it turns out, provides an IPN Simulator, essentially something that
sends an IPN string to your listener and checks if the response is what it
should be. It's tremendously useful. The problem was that I had no idea what a
listener looked like.
I tried a few variants, using this web form:
<form
target="_new"
method="post"
action="http://localhost:8080/exist/rest/db/apps/test/test.xquery"
enctype="multipart/form-data">
<input type="hidden" name="payer_id" value="LPLWNMTBWMFAY"/>
<input type="hidden" name="payment_status" value="Completed"/>
<!-- code for other variables to be tested ... -->
<input type="hidden" name="custom" value="my_info"/>
<input type="submit"/>
</form>
The URL pointed to this test XQuery that would simply echo what it
caught:
I was quickly able to catch whatever the web form sent to the XQuery. However,
when pointing the IPN Simulator to the XQuery, I registered nothing and the
simulator reported a fail. No matter what I did, it always failed.
Desperation
In my desperation, I did what people do,[10] I asked for help. Adam Retter, one of the core eXist programmers,
the inventor of RESTXQ, and an all-around good guy who will forget more about
XQuery than I will ever know, replied to my email within hours, attaching an IPN
listener. He didn't have an XQuery parser available but nevertheless provided
something that, with minor fixes, parsed and worked.
Adam's listener, of course, was RESTXQ-based, and while I knew about the
concepts behind it, I had never written any myself and it took me a few days of
testing to understand how it worked.
I hooked up the listener to my eXist-DB test setup and managed to get the IPN
Simulator to complete successfully. In other words, Adam's listener quite
clearly responded with the required two messages—an empty POST 200 followed by a
copy of the IPN message.
However, I still couldn't extract any part of the IPN message locally.
A Simpler Solution
After some (= a lot of) research,[11] it dawned on me that the difference between my form and the IPN
Simulator is what they send. PayPal sends an
application/x-www-form-urlencoded POST request, which
apparently means that the message body is a bunch of parameters, like this:[12]
Now, if I had read up on the subject properly, I might have been able to solve
the problem earlier. The fact is that the message is a bunch of key-value
parameters, not a body per se. I'm not that clever, however, and
so understanding what actually went on took a far longer time than I
thought.
To test my understanding, I dabbled in ways to read the parameters and
eventually wrote a minimal listener to catch the PayPal confirmation only,
ignoring the intricacies of the IPN process and thinking that the reason I
wasn't able to catch the message was due to the inherent asynchronicity in http
communication:
declare
%rest:POST("{$body}")
%rest:path("/paypal-transaction-complete")
function pay:listener($body) {
let $json := fn:parse-json(util:base64-decode($body))
return
xmldb:store($muk:transactions-collection-uri,concat('uid-',$json?custom,'.xml'),
<transaction tstamp='{current-dateTime()}'
orderID='{$json?orderID}'
userID='{$json?custom}'/>),
pay:all-good()
};
declare function pay:all-good(){
(: Reply with an empty 200 response to indicate to paypal the IPN was received correctly. :)
(
<rest:response>
<http:response status="200" reason="OK"/>
</rest:response>,
""
)
};
This is a ridiculously simple listener; all it does is to save the message
locally in XML (rather than JSON) format and send back a POST OK 200. It is the
minimum functionality, what I had to have[13], and it tied beautifully to the summary page with the button.
Becoming
(With apologies to Michelle Obama for shamelessly borrowing her book
title.)
All Together Now
So, now I had a registration summary HTML page[14] that presented the registration information from the XForm and
integrated a PayPal button with a custom amount and the current user ID as
input, and also included return pages—functionality to be triggered after
completing (or cancelling) the transaction.
All I had to do now was to tie it all together:
Functions to find out the type of registration from the XML from the
XForm and, from it, calculate the actual amount to be provided to the
PayPal button. This, basically, involved a lookup to a list of
registration rates and discount codes, but also to provide the amount
with or without VAT, depending on the delegate's country of
residence.
A thank you HTML page, triggered upon transaction completion, that
would insert the transaction details and payment confirmation into the
registration XML and save it with other confirmed registrations.
Functions to generate a PDF invoice from the registration XML using
XSL-FO, and to send a confirmation email to the delegate with the
invoice attached.
Sandbox Vs Live
As it is a really bad idea to develop and test all this in a live environment,
PayPal provides a sandbox environment, essentially a copy of the live
environment but with means to create fake sellers and buyers, complete with fake
credit cards and other means of paying.
Figure 2: The Sandbox Is Making Some Real Money
The way to keep the two separate is simply to point the button to two
different URLs using a variable, $url:
Locally, I also needed to be able to store the registration data in two
separate locations in eXist:
Figure 3: Locations for Registration Data
When switching between live and sandbox, I had to change the button URL but
also redirect any read and write operations to the appropriate locations. I was
willing to bet that I'd forget at least one sooner rather than later, so I added
a config file that provided paths and settings for both environments,
parameterising every path in every function so they would always be able to look
up the currently applicable paths:
<config>
<!-- This is a MUK Registration app config file -->
<!-- MUK info -->
<business>
<name>Markup UK Conferences Limited</name>
<email>info@markupuk.org</email>
<pid>...</pid>
<bank>HSBC</bank>
<account>...</account>
<client-id>
<sandbox>...</sandbox>
<live>...</live>
</client-id>
</business>
<!-- eXist-DB Domain -->
<domain>http://localhost:8080</domain>
<!-- Data root for MUK-data and subcollections -->
<data-root>/db</data-root>
<!-- Collection names in eXist -->
<collections>
<root>MUK-data</root>
<sandbox>sandbox</sandbox>
<live>live</live>
<sub>
<tmp>tmp</tmp>
<transactions>transactions</transactions>
<data>data</data>
<pdf>pdf</pdf>
</sub>
</collections>
<!-- Current env -->
<current>sandbox</current>
<!-- The currently *selected* path (live or sandbox, basically) -->
<data-uri>/db/MUK-data/sandbox</data-uri>
...
</config>
Now, all I had to do was to change a single field in the config
(<current>sandbox</current>; the sharp-eyed reader will
wonder about the data-uri field in the config fragment, but this is
changed inline by an XQuery Update instruction, triggered by an update in the
admin XForm described in section “The Admin Page”).
In the HTML/XQuery in eXist-DB, $url is set like this:
let $clientid := if ($muk:config//current='sandbox')
then (string($muk:config//client-id/sandbox))
else (string($muk:config//client-id/live))
let $url := concat('https://www.paypal.com/sdk/js?client-id=',
$clientid,
'&currency=GBP&intent=capture&commit=true')
Here $clientid is the ID that PayPal uses to identify where the
URL points—sandbox or live. $muk:config is the above config file,
and so, $muk:config//current tells us what environment the app is
currently running in.
Note
A review correctly pointed out that the above uses LIVE unless the
appropriate field is set to “sandbox”. So if you accidentally write
“sanbdox” in the field, you are running live. Oops. The reviewer
then goes on to suggest that I reverse the logic to avoid that particular
mishap. This is true and will be fixed for next year, along with other
fixes. However, I do wish to point out that the field is never entered
manually, but instead chosen from a drop-down menu in an admin interface,
described in section “The Admin Page”.
The Admin Page
Yet another complication was that if I wanted to move the application between
servers—for example, from http://localhost:8080 on my local PC to
the Markup UK server URL—it also had to go into the config file, and it also had
to be changed when going live. As shown in the config file fragment, above, the
domain is set in
<domain>http://localhost:8080</domain>.
And of course, I also needed the config to include the registration types and
amounts, as well as any special discounts:
Finally, rather than updating the config file directly, I added an XForm for
the purpose. The following sets the domain, environment and the collection
names:
Figure 4: Admin Page, Basic Configuration
And here's a GUI for setting the pricing:
Figure 5: Admin Page, Pricing Settings
Note the buttons that allow us to add and delete custom discounts.
Programming
I still claim I'm not a programmer, but I can't deny I'm leaning more towards
an, um, developer kind of mentality. For this paper, I was tempted to add a
section on modularising, testing, and refactoring, all of which I began to
appreciate in earnest while writing the app, but really, that section would have
been out of scope for this paper.
You could easily argue that XSLT, XProc, and XQuery are all about programming,
just as writing schemas is, but the difference to me is about the basic
paradigm—the core technologies used here are all based on a single paradigm, a
declarative approach common to pretty much every XML markup technology I
use.
Odds and Ends
A couple of odds and ends:
Last year's invoice XSL-FO saw a few tweaks, mainly to better comply
with limited company regulations and to play nice with the config file
that governs most things about the app, but most of the publishing end
was really about making the transform and PDF rendering into XQuery
functions.
Similarly, the email code that sent a thank you note and the invoice
to the delegate stayed pretty much the same.
The whole registration app is for a single
delegate registering, which is not always true. Some organisations
prefer a single invoice for all of their delegates, which is
understandable but more or less broke my app. I ended up writing a
registration XML by hand, outside the registration form, tweaking the
XSL-FO to handle multiple delegates, and finally call directly the
XQuery that generates the PDF and sends the email with that manually
created XML as input.
Speakers attend the conference free of charge, so while they should
register, they shouldn't have to pay. I ended up bypassing the PayPal
code again, but this time to never generate the button to begin with,
skipping straight to the code generating the email.
Getting Cocky
Another hack for 2018 was a registration and voting application for the conference
demojam.[15] Usually, voting is done using a sound level (dB) meter, but two days before
the event, we thought wouldn't it be cool to register the participants using an
XForm and eXist-DB, and then allow people to vote for their favourite using another
XForm? I wrote some XQuery and an XForm to add participants to a list, and
then another XQuery and XForm to register a vote and add that to a total.
Figure 6: Demojam Voting Page
As you may have guessed, this is a screen capture of my test voting page, not the
actual list of participants.
Basically, we shared the voting URL with the audience right before the demojam, and
people could then vote for their favourites using their mobile phones. This worked
surprisingly well, with one caveat: you could vote as many times as you liked. The
Russians couldn't have done it better.
This year, I planned to use my new-found powers to rewrite the application so it
generated a unique code for every delegate, allowing them to only vote once. In the
end,
however, I decided voting fraud is more fun and opted for further registration app
improvements instead, adding reporting functionality to the registration app so we
could
easily keep track of registrations, allowing us to speed up bookkeeping and handle
various special requirements.
See? I am on my way to becoming a real programmer!
But It's Not Just That
So far, this has very much been a story about me realising that I might actually be
a
programmer after all. But really, is that what I am actually becoming?
Declarative (Markup) Languages
The registration app was made possible using XML technologies. Well, almost;
please ignore the XQuery injections to PayPal's JavaScript examples. It's not just
about the angled brackets, though. XQuery doesn't use any, it simply ties into the
mindset, allowing you to use a declarative language to solve your problems.
XSLT, of course, used to be exclusively about angled brackets, but these days, it
treats JSON as a first-class citizen, too. I'm not really a fan but JSON has its
uses, for example, when handling those pesky key-value pair POST messages (see section “A Simpler Solution”).
I write a lot of documentation, and these days, when writing a README file or
producing a simple User's Guide, I tend to use Markdown. It's a brilliant format,
very intuitive and easy to learn, and it is supported all over the place. However,
it's too simplistic for most of what I do for my end users—both technical manuals
and legal documents tend to be far too complex for Markdown and practically require
XML—but for lighter documentation such as README files and meeting minutes, it is
perfect!
Another case in point: the last few years, I've avoided the likes of PowerPoint
and Keynote:
There is the open-source Sozi (see Sozi), an SVG-based
presentation tool that allows you to zoom, rotate, and track across your SVG, much
like the commercial equivalent, Prezi. Sozi is a lot of work, though, and all I need
is something to produce slides with.
For this, I'm now partial to Reveal JS (see Reveal.JS), a Node
JS-based kit that allows you to write your slides in HTML or even Markdown. The Node
JS is far beyond me, of course (I'm not THAT kind of programmer), but I don't have
to actually understand any of it. It's enough to know how to write HTML or Markdown,
with some minor tweaks to add class attributes to trigger the Node JS
functionality.
So the cool thing here is that I'm not actually branching out to programming;
rather, the maturity, stability, and dare I say omnipotence, of markup technologies
now allow me to do what I want using markup technologies only.
More!
The markup technologies-only approach was something I first started to focus on
with my 2012 Balisage paper about XProc pipelines to tie together publishing
processes. XProc, then, was the glue I needed.
XProc 1.0 was somewhat crude, though, and wasn't really in sync with XPath 2. It
also had a significant learning curve, partly because of that non-sync problem, and
so it remained a language for markup geeks, first and foremost. You already had to
be convinced of the usability of XML to bother to learn it. Also, XML was pretty
much it for XProc 1.0; it couldn't handle anything else flowing through a
pipeline.
Now, of course, there's an XProc 3.0 on the way (XProc 3.0: A Pipeline Language), with numerous
improvements allowing me to do more generic manipulation of
data. Why should I bother with non-XML technologies?
And let's not forget, there is an XForms 2.0 on the way, too (XForms 2.0). Some
people claim XML is dead. Really?
The Community
Which brings me to an important topic: the markup technologies community. We're
all part of it, of course, which is why you're reading this and—hopefully—listening
to my presentation. Today, the community is actually driving much of the development
of new XML technologies. Both XProc 3.0 and XForms 2.0 are being developed by
W3C community groups (see XProc Next Community Group and XForms Users Community Group, just to
pick two). We are keeping it all alive, all of us.
There is also another aspect of the markup community—it's an incredibly helpful
bunch. Witness, for example, how Adam Retter not only answered my questions but also
wrote an entire listener module. Adam is also active on the eXist mailing list and
frequently answers questions on—you guessed it—Stack Overflow.
And he is not alone. A lot of people, too many to list here, have helped me with
my projects over the years, and I am trying to return the favour whenever I
can.
Literally, Future Work
So, to sum it all up...
Now That I'm a Programmer
So, now that I am a programmer, how about writing an open-source
document management system? The proper kind, with proper document
version management and the works? I've been thinking about this for several years.
In 2014 at Balisage, I presented a paper on XML-based multi-level versioning
management (Multilevel Versioning), and, of course, there is the XProc thing I
spoke about in 2012 (Balisage, id-xml-for-xml) and 2014 (XML Prague, id-proxist, where I
wanted to use my XProc pipelines to generate a customised user interface for
carrying out a user-specified XProc task.
This paper is far too long already, but this, really, is what I wanted to end
with—using markup technologies only to write that open-source document management
system. My registration app, nominally the subject of this paper, was just the
beginning. Want to join me?
In Closing
My heartfelt thanks to...
My friends in the Markup UK organising committee
My bosses at Karnov Group, who allow me to do all this and actually
encourage me to do it
The fantastic XML community, in particular, Adam Retter
The anonymous Balisage reviewer who spent time copy-editing this
paper
And, of course, my beta testers, um paying conference delegates. Every time
someone registers, my heart still skips a beat, both out of worry and because
it's exciting!
[13] Adam's listener did a lot more, of course, since the idea with the IPN
workflow is for PayPal to keep sending the message until confirmed. Mine
only did a one-time catch of the message, hoping that everything would
work out during that particular split second.
[14] Actually an XQuery that ran an XSLT to output the summary and injected
some XQuery code into an HTML fragment featuring JavaScript code for the
button.
[15] Where participants get five minutes each to present something markup-related
and a winner is chosen among them by the audience.
