How to cite this paper
Case Study: Applying an Agile Development Methodology to XML Schema Construction
Balisage: The Markup Conference 2016
August 2 - 5, 2016
The Basics; Brands, Brand Identity and the Brand Print Quality Ecosystem
Brand is a name, term, design, symbol or other feature that distinguishes one seller's product from those of others. Brands you might recognize are Tiffany, Coca-Cola, Ford Motor Company and P&G. For each brand, the reproduction of its name and symbol or logo is of utmost importance as the printed reproduction is a measure of brand quality. While brand names and logos are reproduced in many formats, print reproduction is high on the list. From billboards to packaging to advertising to the print representation of a brand on a targeted mailing piece, the quality of each printed item is a reflection on the quality of brand itself.
Since brands spend billions on print each year, it should come as no surprise that brands wish to automate tracking the print quality of their suppliers on an ongoing basis. Today this is difficult and expensive because printers use many different quality measurement tools. As a result, they send print quality reports to the brand in a wide variety of formats. And since a standard industry reporting format does not exist, quality data cannot be directly utilized or imported into the brand's quality tracking systems. As a result, when brands receive print quality data from their printers, they spend a great deal of time and money to have that data converted or keyed into their quality management systems. Today, collecting print quality data can be difficult and expensive because different printers use different measurement tools. Tools from different manufacturers do not interconnect and do not report print quality in the same format. In some cases, printers must use multiple sets of tools to service different brands, which have different specifications and requirements, reducing printer's efficiency and increasing their costs.
Brands judge print quality based on color fidelity (the accuracy of the color of a printed image based on spectral measurements), registration (the accuracy of the alignment of inks on a printed image) and absence of defects such as lines, smudges, streaks, etc.). Since the methods to score quality vary significantly by printing type/sector and vary from brand to brand, those developing PQX were highly challenged to create a model that would work equally well for those printing packages, those printing signage and those printing marketing materials.
Adopting an XML-based standard print quality reporting message would would be beneficial for both the printer and for the brand. Having a single message format would enable supply chain automation because new tools or plug-ins for existing tools could be developed to streamline the print quality reporting and analysis process. In 2015 Idealliance began working with its printer and brand members, along with the larger ISO Graphic Technology community to develop an XML print-quality exchange specification to serve as the standard message format for printers to report print quality to Brands (print buyers).
The print quality ecosystem is made up of print service providers (Printers), print buyers (Brands) and third-party print quality service providers employed by the brands to act as intermediaries to manage print quality on behalf of a brand. Because the print quality requirements and reporting criteria are quite varied, we believe that two standardized messages are required. The first is a message from the brand to the printer that defines the requirements for the printer to conduct press runs to report quality. This message is the Print Requirements eXchange message or PRX. The second standard message goes from the printer back to the brand. This message, the Print Quality eXchange, or PQX, delivers raw print quality data according to the parameters set by the first message.
Figure 1: Print Quality Message Ecosystem
PQX is an Idealliance Specification. In the fall of 2016, it will be fast tracked into ISO TC130 WG2 to become ISO 20616 Part 2.
The Basics; Agile Software Development
Agile software development is a software development methodology in which the requirements and solutions evolve through collaboration between development partners. Typically agile software development methodology involves incremental, evolutionary iterations of code.
Iterations between versions of the software often take place over a short time frame. A highly engaged cross-functional team participates in each iteration. Typically the tasks that occur within an iteration include:
The goal of agile development is to have a new version of software produced for each iteration. Clearly to enable a short feedback loop and adaptation of software it is critical to establish an efficient communication / collaboration mechanism. Following an agile software development methodology means that the software can evolve and yet be delivered rapidly with consensus of all stakeholders.
Why Use Agile Development for PQX?
Several factors pushed the working group toward employing an agile software development methodology to develop the XML schema. First was the urgency to develop a workable standard for the industry. Today, with the decreasing volumes of printing being purchased, both printer and print buyers are looking to automation to improve the bottom line. Second was the urgency to move from the development of the XML message to immediate implementation of that message. In the case of PQX, all parties were eager for immediate implementation. Technology stake holders did not see new products being developed to support PQX-based communication, but rather believed that PQX would be implemented by a set of transforms from their proprietary reporting formats into the standard PQX format. So our project was highly influenced by the technology vendors' programmer staffs, who are accustomed to using agile techniques as they develop their own products. They insisted that iterations and testing of transforms to PQX occur as the message was developed to assure that all current reporting features in their commercial products be included in PQX and to assure that writing transforms required to implement PQX was as straightforward as possible.
Another factor that made our iterative approach important was that we were bringing together experts from different printing commuities. While to outsiders, all types of printing may appear to be quite similar, in reality this is far from the truth. Printing cartons is different than printing on items such as a can or a bottle. And printing labels to be adhered to a can or bottle is different from directtly printing on those items. In each case the sampling techniques, measurement techiques and observational techniques vary. We found that the team participation for any one development session brought their print-sector bian to the model. Only by showing the model and samples following each session could we solicit input from other sectors to assure that the model would work across all print industry sectors.
Step 1: Developing a
Terms of Reference
The first task of the PQX Working Group was to define what we call the Terms of Reference. Basically, the ToR is a statement of scope and mission of the working group. According to Wikipedia, Terms of Reference is defined as:
Terms of reference describe the purpose and structure of a project, committee, meeting, negotiation, or any similar collections of people who have agreed to work together to accomplish a shared goal. The terms of reference of a project are often referred to as the project charter.
It took several working group calls to develop and refine the Terms of Reference for PQX. We wanted to develop a concise statement for the reference of the working group and to communicate the intent of the project to the outside world. Over the past year, we have revisted the Terms of Reference any time there was confusion about the scope and mission of PQX. Each new version of the ToR included a more precise description of our charter. It is interesting to note that although the statement has been updated over time, the essence of the ToR did not change.
The current Terms of Reference:
PQX is intended to facilitate the one-way transmission of performance data for one or more printed samples from a single press run from print service providers to relevant stakeholders and brand owners; thus allowing brand owners to assess and track relevant business, production, color and quality data of printed materials of all forms. PQX is only intended to transmit raw quality data. The PQX message intentionally excludes tolerance and evaluative information, allowing the receiver to determine acceptability by applying their own scale and tolerance values. PQX incorporates color using the same data containers that are defined in ISO 17972 (CxF). While PQX and CxF are different formats with different parsing requirements, developers can use the same strategies for reading and writing color data in a PQX file that they use for reading and writing color data in a CxF file. PQX is compatible with both spectral and non-spectral color data.
Step 2: Developing Formal Design Principles
Since employing an agile software development methodology means that we will be producing new versions of the XSD as a series of rapid iterations, it was critical that we formalize the design principles for the schema. Looking back, this was one of the most critical factors that enabled us to succcessfully employ agile techniques. As development proceded, there were times when we developed structures that seemed like a good idea at the time, but that violated the design principles. In such cases, a disciplined habit of referencing the design principles brought the direction of our schema development efforts back on track.
The design principles for the development of the PQX schema were divided into sections to assist with referencing and understanding the principles. All principles were developed by consensus of the stakeholders. Again, it took some time to develop and agree upon the design principles. But our working group agree that developing the principles was an important ingredient in enabling agile schema development.
Not all our design principles were developed before schema development began. Some were developed along the way to guide us as we tackled a new block of the message model. For example, when we decided to use the ISO color data specification, CxF, we developed the principles for how we would employ CxF before we began to develop that part of the schema.
PQX Design Principles
Structure of the PQX Message will conform to the Terms of Reference as defined by the WG
Elements defining Tolerances and Scoring are out of scope
The PQX message will be limited to the reporting of press run results, not for returning the requirements or specs for that press run.
Standard ANSI/ISO nomenclature (terms/definitions) will be used whenever possible
Element/attribute names will self-documenting; i.e. names will not be abbreviated
Each element and attribute will be documented in place using annotation and documentation XSD structures
Each element and attribute will documented as development occurs to enable automatic generation of a documentation for each schema iteration
The namespace for all PQX elements will be pqx:
CxF elements will be imported from the CxF3_core schema and will retain their CxF cc: namespace
Either CxFX1 or CxFX4 may be used for color data. This means while spectral data is allowed (for packaging) it is not required.
The root element for the Print Quality Report will be the PQX message
The Schema will be prescriptive and strictly enforceable, rather than flexible, to allow for little, if any programmer interpretation of intent
Ordering of fields within PQX will be absolute
Cardinality will be determined by consensus of the stakeholders
Required elements/attributes must be agreed upon by all
Optional elements, within the scope of the project, will be included at the request of any stakeholder
Employing CxF Data
CxF will be employed as the data store for reference and sample color data
The PQX message will have an attribute of CxFVersion to allow for inclusion of any version of CxF that may be supported by a printer's color measurement tools
Only Core CxF will be employed as the data store for color data
The CxF schema will be imported into the PQX schema so that CxF elements can be included in the PQX message in their native cc: namespace
CxF will be employed as a complete CxF hierarchy (blob) with the cc:CxF element as the root to ensure direct importability from color measurement devices
No fragment of CxF will be allowed within the PQX model
CxF elements will be employed for only those mechanisms where the intent of CxF is a match for the intent of PQX
The use of CxF Tags to customize CxF to fit the intent of PQX is not appropriate
The use of CxF CustomResources and CustomAttributes is not appropriate
Valid but non-appropriate CxF elements may be written into a PQX message by sending software systems but will be ignored by receiving systems
Quality reporting objects not explicitly supported by CxF Core will be implemented in PQX outside of the CxF data store
CxF is ISO 17972. It is the standard data transport for color data. It was very important to the working group to define exactly how PQX would carry color data using CxF. Hence this section of design principles was developed.
PQX Iteration Process
About Our Process
Once we had the
Terms of Reference and the
Design Principles in place, we began the iterative process. Since one of our motivations for employing an agile methodology was to develop the message in a relatively short timeframe, we began with two working group sessions each week. Once we began working on detailed models for color reporting we met only once each week with a break for winter holidays. In the period of 12 months we completed 47 iterations before posting the schema and its documentation for public comment as an Idealliance Specification.
The PQX Working Group is made up of Idealliance members and invited experts from ISO Technical Committee 130 (Grapic Communications). The Working Group was made up of two Co-Chairs and 47 members, representing the United States, Canada, Great Britain, Germany, France, Italy and Japan. I serve as the Idealliance Program Manager and Editor of the PQX Specification.
Each working group session was conducted using the Idealliance Webex service. Each session was recorded so that any participant that missed a session could be brought up to speed for the next session. Each iteration consisted of the following steps:
The latest version of the PQX XSD structure was reviewed by the facilitator. Changes from the previous version were highlighted.
test data sample was reviewed so all could understand how their quality data would be encoded for exchange. Transformations of the test sample to/from any technology vendor’s proprietary format could be tested by individual software vendors participating in the effort and faults were reported.
The schema was advanced by modification of structures in the current version or the addition of new fields/functions.
Documentation about new elements and attributes were developed by the group to assure a stable, working nomenclature was in place.
Following each working session, the facilitator produced a new version of the PQX XSD, complete with inline documentation.
The facilitator also produced 1 or more samples to demonstrate the latest schema version.
Complete documentation was generated from the schema by the facilitator.
The new version of the XSD, its documentation and related samples were posted in the library of the PQX online collaboration website, Idealliance Connect (http://connect.idealliance.org) where all members of the working group could access the materials for review and comment before the next scheduled session. Minutes were posted to the library each week. Connect was also used for group discussions to advance our efforts.
The facilitator prepared a slide deck to lead the discussion and advance the XSD for the next working session.
The following graphics illustrate changes and advances in the PQX schema over the development lifecycle.
This was our first attempt to model what we imagined might be in the PQX message. This version was developed by the facilitator based on a free-ranging discussion from our first several sessions.
In the second version of the schema we continued our work on the high level structure. As you can see the fields changed and the idea of reporting color measurements, registration and visual (defects) began to emerge.
During the development of Versions 3-10, we focused our work on modeling the business data that must be passed between the printer and the brand. We developed separate, clearly defined fields for the party the evaluation samples were printed for, printed by and which party did the measurements of these samples. At this time we were working primarily with sheetfed offset printers and the assumption was that quality data would be based on press sheets pulled randomly during the press run. During this time we also decided to report color from sample measurements and began the task on how to report color accuracy in XML. That task would be our focus for the next 12 versions.
In Version 22 we added a block for reporting production information as an additional business information field. Because when printers and technologists from the packaging and labeling printing sector were added to the working group, we came to see that quality might be judged from more than a randomly pulled press sheet. Instead we might be reporting print quality from labeling printed on cans or bottles. Or we might be reporting quality from labels for packaging, where it is common pracctice for labels for several products are printed simultaneously across the substrate base. All these new considerations needed to be factored into the iterative schema models we produced each week.
We also discovered that a brand might require that on each printed sample, measurements should be taken at several locations. As a result, we would need to create a model to report the quality from each location on a sample individually. Additionally, by the time we developed Version 22, we added a block to hold ISO standard color data that would be dumped directly from color measurement systems into the PQX message to send to the brand.
During the development of Versions 23-30 the group focused on how to report color measurements. It turns out that this is quite a complex task. As part of this task we needed to add the ability to report the ink laydown order on the press, the substrate color information, patch information as well as tints and the color measurement values for the parent ink solids. Because there is no single, standard formula for calculating and reporting tone values, we had to assure that we included all color factors in the PQX message so that a receiving system could calculate tone values using their formula of choice. All this was a challenge for print technologists to model for the PQX XML message.
Version 31 - 47
Once the issue of modeling color reporting was complete, the working group turned its focus to developing methods to report registration (data about the accuracy of the alignment of inks on a printed image) and how to report printing defects such as lines, smudges and something printers call "hickeys". We were additionally challenged to support the transport of data from the new on-press quality systems that report on every print impression using averaging techniques to minimize the reporting message size.
Because we employed an agile development strategy to develop the PQX message, programming teams of print quality reporting products stand ready to commercialize PQX immediately following the publication of the Idealliance / ISO Standard. Without having engaged these programming teams from the beginning and employing agile techniques, implementation might not have happened for some time. As a result of the experience gained in developing the PQX schema, our team believe that using agile techniques to develop XML schemas can serve as a best practice for the development of many other XML schemas where immediate software implementation is a goal.
Appendix A. Tagged Example
In order for readers to understand more about data exchanged using PQX, reviewers have requested the inclusion of a sample. In this sample, a press run at Printer ABC in Queens, New York is conducted for Jupiter Candy, Inc. The product is the 12 oz WoWs bag with the SKU 1234. The measurements are not being taken by the printer. Measurements is being handled by their contractor, Q-TraX in Cincinnati. The job was run on November 15, 2015 under the printer’s LOT Id 701-123-3331. There are 5 ink channels on the press that lay down ink in this order; Cyan, Yellow, Magenta, Black and the “Wow Red” spot color. The data capture method for both color and registration was by manual pressman pulls throughout the run. Defects were reported from an automated camera system, with a defect reporting 80 percent.
Color quality is reported in the Color Report block and linked back to the Ink Channel information using the InkChannelIdLink attribute. The Color Report consists of a Measurement Set at position “1” on the package. In a full PQX report, multiple Measurement Sets at Brand-specified positions on the printed item would be sent. This sample illustrates only a single measurement set. Each Measurement Set is made up of measurements of one or more patches. Each measurement has an indication of the patch type being measured. All patches except the substrate patch are linked pack to ink channels using the InkChannelIdLink attribute. Patches that are made up of overprints of more than one color are linked back to multiple ink channels. Tint patches have tint values specified and are linked back to their parent ink channel. This example shows coding for measurements of the substrate, 4 solids, 2 tints, an overprint, a graybalance and a build patch.
Within a PQX message, color data must be expressed as CxF. Rather than embedding CxF object into a color measurement, the CxF data is stored, in its entireity within separate CxF blocks of the message. So each PQX Measurement element in the Color Report must be linked to the Object in the CxF Sample data block. Links for each measurement may also be made back to standard (aim) CxF data if the Brand requires a return of that data.
The quality of print registration is reported in the Registration Report element. The data capture method for registration was by manual pressman pulls throughout the run. The registration reporting method used was the “Channel Registration” method where the alignment of each ink channel is compared to the reference ink channel. In this example the reference ink channel is the “Cyan01” which can be referenced back as the first in print order in the Press Information. The Registration Report in the sample is made up of two Registration Sets. In each set the alignment of non-reference ink channels is compared with the reference on both the X (horizontal) and Y (vertical) axis.
Reports about any printing defects are carried in the Defect Report element. . The report was generated from an automated system that has reported at a 100% rate of inspection. Two defects were reported. Both were determined to be defects when compared to the proof. The first defect was a hickey in the center zone with a severity of “4”. The count for this defect was 200. The second defect was a line in the top margin zone with a severity of “1”. The count for this defect was 1000.
<?xml version="1.0" encoding="UTF-8"?>
<!-- Defect Report from inline system reporting on 100 sample taken every 15 minutes -->
xmlns:n1="http://www.altova.com/samplexml/other-namespace" PQXVersion="Draft44" PQXDate="2016-08-23" PQXMessageID="pqx010" xsi:schemaLocation="http://idealliance.org/pqx PQX_Draft44.xsd">
<CompanyName>Jupiter Candy, Inc.</CompanyName>
<Product>WoWs 12 oz bag</Product>
<Printer>Printer ABC </Printer>
<Location>Queens, NY </Location>
<PrinterJobDescription>WoWs 12 oz bags</PrinterJobDescription>
<InkChannel InkChannelId="Cyan01" InkChannelPrintOrder="1" InkChannelName="cyan" InkName="Cyan224"
<InkChannel InkChannelId="Magenta01" InkChannelPrintOrder="2" InkChannelName="magenta" InkName="Magenta224" InkType="process"/>
<InkChannel InkChannelId="Yellow01" InkChannelPrintOrder="3" InkChannelName="yellow" InkName="Yellow224"
<InkChannel InkChannelId="Black01" InkChannelPrintOrder="4" InkChannelName="black" InkName="Black224"
<InkChannel InkChannelId="Spot01" InkChannelPrintOrder="5" InkChannelName="spot" InkName="WoW Red" InkType="spot" />
<RunLength TotalRolls="8" TotalMeterage="16000"
<Measurement MeasurementName="My Substrate"
<Measurement MeasurementName="My Cyan" MeasurementId="M001"
<InkChannelInfo InkChannelIdLink="Cyan01" TintValue="100"/>
<Measurement MeasurementName="My Magenta"
MeasurementId="M002" PatchType="solid" PQXSubstrateIdLink="M00S"
<InkChannelInfo InkChannelIdLink="Magenta01" TintValue="100" />
<Measurement MeasurementName="My Yellow" MeasurementId="M003"
<InkChannelInfo InkChannelIdLink="Yellow01" TintValue="100"/>
<Measurement MeasurementName="My Black" MeasurementId="M004"
<InkChannelInfo InkChannelIdLink="Black01" TintValue="100" />
<Measurement MeasurementName="My Cyan Tint50" MeasurementId="M005"
<Measurement MeasurementName="My Cyan Tint25"
MeasurementId= "M006" PatchType="tint" PQXSubstrateIdLink= "M00S"
<InkChannelInfo InkChannelIdLink="Cyan01" TintValue="25"
<Measurement MeasurementName="Blue Overprint" MeasurementId="M007"
<InkChannelInfo InkChannelIdLink="Magenta01" TintValue="100"
<Measurement MeasurementName="My50Gray" MeasurementId= "M008"
<InkChannelInfo InkChannelIdLink="Cyan01" TintValue="50"
<InkChannelInfo InkChannelIdLink="Magenta01" TintValue="40"
<InkChannelInfo InkChannelIdLink= "Yellow01" TintValue="40"
MeasurementId= "M009" PatchType="build"
PQXSubstrateIdLink="M00S" CxFSampleObjectIdLink="xxx" >
<InkChannelInfo InkChannelIdLink="Cyan01" TintValue="41"
<InkChannelInfo InkChannelIdLink="Magenta01" TintValue="33"
<InkChannelInfo InkChannelIdLink= "Black01" TintValue="37"
<MeasurementSet> . . . </MeasurementSet>
<RegistrationSet PositionOnSample = "left" >
<ChannelReport ReferenceInkChannelIdLink="Cyan01" >
RegistrationUoM="mm" XPosition="-.02" YPosition="0" />
<MeasuredChannel InkChannelIdLink="Black01" RegistrationUoM="mm"
XPosition="0" YPosition = ".01" />
<MeasuredChannel InkChannelIdLink="Spot01" RegistrationUoM="mm"
XPosition="0" YPosition="0" />
<RegistrationSet PositionOnSample = "right" >
<ChannelReport ReferenceInkChannelIdLink="Cyan01" >
RegistrationUoM="mm" XPosition="0" YPosition="0" />
RegistrationUoM="mm" XPosition="-.01" YPosition="0" />
<MeasuredChannel InkChannelIdLink="Black01" RegistrationUoM="mm"
XPosition="0" YPosition = ".02" />
<MeasuredChannel InkChannelIdLink="Spot01" RegistrationUoM="mm"
XPosition="0" YPosition="0" />
<DefectEntry BasisOfReference = "proof" DefectZone = "center"
DefectName = "hickey" DefectSeverity = "4" DefectXMeasure = ".2"
DefectYMeasure = ".2" DefectUoM = "mm" DefectDescription = "This is a donut
hickey with a diameter of .2mm" DefectCount = "200" />
<DefectEntry BasisOfReference = "proof" DefectZone = "top margin"
DefectName = "line" DefectSeverity="1" DefectXMeasure = ".1"
DefectYMeasure = "3.0" DefectUoM = "mm"
DefectDescription = "This is a very thin black hairline near the top margin"
DefectCount = "1000" />
<cc:CxF> . . . </cc:CxF>