Developing Low-Cost Functional Class 3 IETM

Copyright © 2012 Electronic Commerce Connection, Inc.

expand Abstract

expand Betty Harvey

Balisage logo

Proceedings

expand How to cite this paper

Developing Low-Cost Functional Class 3 IETM

Balisage: The Markup Conference 2012
August 7 - 10, 2012

Introduction

Note

The MIL-STD-38784 data used as examples in this paper and demonstration is SGML data that is provided as part of MIL-STD-38784 (BV8). The graphics used in the manual are generic bicycle graphics modified to support the functionality. The graphics and SGML/XML tagging have no correlation except they have been arbitrarily linked for demonstration purposes..

IETM Background and History

U.S. Department of Defense (DoD) started down the path of Interactive Electronic Technical Manuals (IETM) in the early 70’s.  The first initiative was the “Paperless Ship” initiative.  The original idea behind the Paperless Ship initiative was to digitize all paper manuals and provide them in electronic form.  The first initiative took paper manuals, scanned and created microfiche images from the pages of the manuals.  Every ship was provided microfiche readers.  Many of the microfiche images were very poor quality. Scanning documents was new technology at the time and not perfected.

In practice, most technicians printed the relevant pages from the manuals they were interested in. After all the relevant pages were printed, the microfiche readers became big paperweights and gaithered dust.  It wasn’t feasible for technicians to take the microfiche readers to their maintenance areas or fit a microfiche reader to their bunk to study. So much for microfiche!

The next phase of the Paperless Ship initiative was electronically indexed page images which began in the early 80’s.  Maintenance manuals were digitized into TIFF graphics.  Indices were created in the background which allowed readers to search and navigate to the point in the graphical representation of the document.  This obviously was a vast improvement over microfiche.  The original Navy Advanced Technical Information Support (ATIS) program was a fairly successful initiative.

The Computer Aided Logistic Support (CALS) initiative began in 1985.  In later years the CALS acronym morphed into "Continuous Acquisition and Lifecycle Support" and finally "Commerce at Light Speed". The CALS initiative resulted in numerous military specifications for graphics, publishing, integrated database and IETMs.  Tri-Service working groups were created which consisted of Army, Navy and Air Force representatives, as well as government contractors and interested software vendors.  These working groups cooperated to create joint specifications which were designed to be used by all U.S. DoD organizations.  These specifications were also used as contractual requirements for DoD logistics suppliers. 

DoD SGML Publishing and Specifications

The specification MIL-PRF-28001 (MARKUP REQUIREMENTS AND GENERIC STYLE SPECIFICATION FOR EXCHANGE OF TEXT AND ITS PRESENTATION) was originally published in 1992.  The last printing was MIL-PRF-280001C which was published in 1997.  MIL-PRF-28001 specified the use of SGML for all new technical manuals within DoD.  Each branch of the service developed DTD’s for use within their individual organizations based on their specific requirements and adhering to the requirements in MIL-PRF-28001. 

In addition to specifying the SGML constructs for developing DTD’s, MIL-PRF-28001 provided a DTD and specification for applying styling to the SGML.  The styling specification was called Formatting Output Specification Instance (FOSI).  Appendix B of the specification contained the DTD that supported the use of FOSI for presentation.  Several SGML vendors were part of the working group and were working toward developing a FOSI-based publishing system.  Two vendors DataLogic and Arbortext successfully developed FOSI-based formatting within their product.  However, the implementation was slightly different in both systems based on different interpretations of the specification and ambiguity of the DTD.  The result was that a FOSI developed for one system could not be used in other systems.

About the time the FOSI was being finalized, the JCALS (Joint Computer Aided Logistics) program began which implemented an integrated publishing system for the components of CALS integrated into the hardware/software platform. JCALS included SGML authoring, editing and publication components.   Arbortext was the editor and DataLogic performed the publishing in JCALS.  The system was available and started being used and mandated for technical publications in the 1994/1995 timeframe.

IETM Specifications

The CALS initiative had three IETM specifications.

  • MIL-PRF-87268 - General Content, Style, Format and User Interface (GCSFUI)

  • MIL-PRF-87269 - Data Base, Revisable: Interactive Electronic Technical Manuals (IETM) for the support of (Contains the IETM DTD

  • MIL-Q-87270 - Quality Assurance (rescinded in 1994 because the specification was hard to interpret and there weren't any implementation of this specification)

The IETM user interface (GCSFUI) specification was developed in 1992.  The presentation was very simplistic.  Window-based systems were in their infancy and the user interface, even at the time of publication, was very linear. 

The SGML specification for the data was based on SGML and HyTime.  The IETM systems were custom developed and were costly.  Getting data to valid MIL-PRF-87269 SGML was very expensive.  In 1995 the estimate for authoring or converting data was $100 a page.  Once the data was in an 87269 SGML format it was very difficult to produce written pages because of the fault isolation and navigation linking that took place.

Small programs and component manufacturers could not economically afford such an expensive endeavor.  If and organization could afford getting the data into a compliant 87269 format, obtaining knowledgeable personnel was difficult to find.  Most knowledgable individuals were employed by the large defense contractors and smaller companies could not compete at that level.

DoD and the Navy in particular realized that a ‘one-size fits all’ approach to IETM’s wasn’t realistic.  The Navy adopted a class system for discussing and identifying the types of IETM’s in 1994[1] .  The Navy defined 5 classes of IETMs. 

Note

The class definitions only include SGML but XML is a subset of SGML and there is a direct correlation.

  •  Class 1 - Page Turner (limited Navigation)

    • Raster Scanned, i.e., PDF

  • Class 2 - Electronically Scrolling Documents - Navigation

    • May be SGML but not necessarily

  • Class 3 - Linearly Structured IETM Documents - SGML

  • Class 4 - Hierarchical Structured IETM

    • Conforms to MIL-D-87269

  • Class 5 - Integrated Electronic Information System

    • Expert System, Artificial Intelligence, Integrated System Diagnostics

About the time the U.S. DoD IETM specifications were published the AeroSpace and Defence Industries Association of Europe published the S1000D specification.  S1000D was in direct competition with MIL-PRF-87269.  S1000D is being used on many European and now U.S. DoD projects.  Today we see limited 87269 implementations.

Fast Forward to 2012

Today U.S. DoD is still requiring organizations to deliver technical data in SGML.  Some contracts still stipulate that the documents have to be published using the FOSI’s that were developed for the JCALS systems.  JCALS software and the hardware that supports it is no longer available.  JCALS ran on Sun Microsystems platform.  The software components were tightly integrated. 

Companies who have had long-standing contracts (5 years or more) with DoD to supply manuals in whatever format that was specified, i.e., Word, WordPerfect, etc. are receiving new contracts that have contractual language requiring SGML and published PDF using JCALS FOSI's. Most of these organizations have never heard of SGML or FOSI's. The documentation part of the contract is a small component of their deliverables.

This was case the for Cobham.  They Air Force modified the documentation portion of their contract to deliver technical manuals in SGML conforming to MIL-STD-38784C or MIL-PRF-87269.  Initially there was a stipulation that a PDF for each manual be delivered using the JCALS FOSI (DataLogic).   These requirements are impossible for them to achieve, especially with MIL-PRF-87269.  During the initial phase of the requirement by the Air Force to move to SGML, Cobham management and staff received an education in SGML and surrounding technologies. SGML, FOSI, and XML. This was brand new territory for them.   In order to understand the nuances of the requirements of the contract they needed to be made aware of the realities of modern-day SGML publishing and it's challenges and limitations, i.e., the lack of available software.  Through negotiation the Air Force agreed to the following deliverables:

  1.  SGML Manuals

  2. TIFF graphics

  3. Class 3 IETM

Solution

The first step was to educate the staff in SGML authoring and MIL-STD-38784C in particular. Their manuals were in Word and conformed to the Air Force format as outlined in the Technical Manual Contract Requirements (TMCR) The staff was also provided Arbortext Editor training.   The staff involved in the editing of manuals included a team of 3 people. 

Jointly we  were able to create a Class 3 (with some Class 4 functionality) IETM at a relatively low cost.  We developed a product that the Air Force was very happy with and accepted in lieu of PDF.  The IETM met and exceeded the following capabilities:

  1. Navigation bar for Table of Contents

  2. Inline graphics with links to PDF file with full-size  graphics. Graphics and links can be SVG if hot spots are provided.

  3. Integrated Parts Breakdown (IPB) with linking to specific graphics. Bi-directional linking is available with SVG graphics and hotspotting.

  4. Full-text search.

  5. Consolidated list of tables with links to appropriate locations

  6. Consolidated list of figures with links to appropriate location.

Software Used

This section will describe the software that we used to create the Class 3 IETM.  All the software is either open source or commercial-off-the-shelf (COTS) products.  The cost of the software was relatively inexpensive.

Apache Subversion (SVN): SVN is an open-source software versioning and revision control system.  SVN allowed Cobham personnel and ECC to exchange files and configurations without the necessity of being co-located.  SVN was critical during the training process for Cobham SME's.  It also provided Cobham the ability to remotely test the development of the IETM and provide real-time feedback.  An SVN repository was established on a Linux (CENTOS) box that was accessible to both Cobham and ECC.

Tortoise SVN Client:  Tortoise SVN client was used to communicate between the SVN repository and local Microsoft Windows computers.  Tortoise allowed individuals to check-in and check-out data.

Editor:  Arbortext Editor – Arbortext Editor was selected because it is one of the only editors that supports SGML.  Most XML editors only support XML.  JustSystems XMetaL also supports SGML editing but we decided to use Arbortext to take advantage of Arbortext included 38784 stylesheets. 

XSLT:  Custom XSLT scripts were created to create the customized HTML and XSL-FO files.  Saxon is used as the XSLT transformation engine. 

XSL-FO Engine:  RenderX

Search Engine:  Zoom Search Engine by Wrensoft.  Zoom is a low-cost search engine that provided quite a bit of functionality and allowed the Javascript that controls the search to be bundled with each book.

SVG Editor:  Inkscape. SVG was only used to test the feasibility and currently isn't being used.

The SVN repository was created as a 'self-contained' environment.  This provided Cobham with the ability to add manuals and convert them using a single batch file.  The figure below shows the structure of the SVN repository.

image ../../../vol8/graphics/Harvey01/Harvey01-001.jpg

The green check mark on the folders shows that the directory is up to date.  When a red exclamation mark is displayed in the file navigator it is a clue to the editor they need to update the SVN repository with their modifications.  I will describe the contents of the directories later in this paper.

Conversion from Word to MIL-STD-38784C

The manuals were initially written using Microsoft Word.  An XSLT script was written to initially convert the Microsoft Word manuals to MIL-STD-38784C.  The manuals that Cobham had written were very well formatted so the conversion was better than most Word to XML conversions.  Cobham authors were consistent in the use of styles within the Word file.  Consistency in use of styles within a desktop editor file makes conversion to XML much easier and provides a better end product.

The initial conversion was to XML because XSLT only works on XML. Another option would have been to use Omnimark and go directly to SGML but there are more XSLT developers then Omnimark developers and if modifications were required any XSLT developer could make the modifications. Arbotext Editor supports both SGML and XML so the obvious choice was XML.  The data was ‘saved as’ SGML after the final edits were made to the manual and it was ready to be delivered.  The SGML version of the file is the official deliverable.  The XML version is used to create the IETM version.

There were some manual edits that were required after the Word to XML conversion.  These edits required some reorganization of text. Most military technical manual specifications are designed to be very hierarchical and are very stringent in the organization of content.

Some of the areas that needed to be reorganized or rewritten by a subject-matter expert (SME) was:

  1. Multiple paragraphs in sections - MIL-STD-38784C only allows 1 paragraph in a hierarchical structure, i.e., para0 (equivalent to a section).  The SME was required to evaluate the text and determine the best approach for making the SGML valid according to the DTD and not affect the functionality of the manual negatively.

  2. Alerts  –  MIL-STD-38784C specifies that alerts (warnings, cautions and notes) must appear before the text the alert refers too.  Alerts also must appear in the “Warning, Caution, Notes” sequence.  The Word manuals followed this construct for warnings and cautions, however, notes occasionally were located after paragraphs.  The conversion tried to capture these occurrences but wasn’t 100% successful because it required evaluating text to determine if a paragraph inside the Word file was a note.

  3. Tables – Tables converted fairly well.  A few of the complicated tables required modification of spans and column and row borders.

  4. Figures – Figures also converted fairly well.  Problems arose when multiple graphics were included in a single figure.  Technical manuals tend to have ‘sheet’ information

Integrated Parts Breakdown (IPB)

The presentation of the IPB is in table form in the printed manual.  The only differentiating characteristic to identify the hierarchy of assemblies is leading dots (.) in front of the description.  Below is a fictional example of a printed IPB.   The periods (.) in front of the part description defines the hierarchy.   Getting the hierarchy correct in the XML required analyzing the text in the description before outputting the assembly level.

image ../../../vol8/graphics/Harvey01/Harvey01-002.jpg
FIGURE & INDEX NO. PART NUMBER CAGE  DESCRIPTION 1  2  3  4  5  6  7 UNITS PER ASSY USABLE ON CODE SMR CODE
6-2- P-1234-1 04577 TOP LEVEL ASSEMBLY ................................ REF   EX001
-1 P-1234-1-1 04577 .  SUB 1, Assy of ........................................... 1   EX002
-2 P-1234-1-2 04577 .  SUB 2, Assy of ......................................... 1   EX003
-3 P-1234-1-3 04577 .  SUB 3, Desc. ............................................ 2   EX004
-4 P-1234-1-3-1 04577 .  .  SUBSUB 1 ............................................. 1   EX005
-5 P-1234--1-3-2 04577 .  .  SUBSUB 2 ............................................. 1   EX006
-6 P-1234-1-3-2-1 04577 .  .  .  SUBSUBSUB 1 .................................. 1   EX007
             

A separate XSLT was written for the IPB after the entire manual had been converted from Word to 38784C.  The IPB section was initially converted to a CALS table model during the Word to XML conversion.  The IPB table was converted in a second step to create a valid <ipbpl> structure from the converted Microsoft Word IPB table.  Below is the above sample IPB tagged according to MIL-STD-38784C:

<ipbpl>
    <ipbplgrp>
        <figure id="Examplefig6-2">
            <title>TOP LEVEL ASSEMBLY</title>
            <graphic boardno="Example-fig6-2" reprowid="5.0in" reprodep="8.0in"/>
        </figure>
        <ipbpl0tbl>
            <ipbpl0>
                <ipbfigindex> 6-2- </ipbfigindex>
                <partno>P-1234-1 </partno>
                <ipbcage>04577</ipbcage>
                <ipbdesc>
                    <nomen>TOP LEVEL ASSEMBLY</nomen>
                </ipbdesc>
                <ipbunits> 1 </ipbunits>
                <ipbuseoncode/>
                <ipbsmrcodes>
                    <ipbsmrcode>EX001</ipbsmrcode>
                </ipbsmrcodes>
                <ipbpl1>
                    <ipbfigindex> -1 </ipbfigindex>
                    <partno> P-1234-1-1</partno>
                    <ipbcage>04577</ipbcage>
                    <ipbdesc>
                        <nomen>SUB 1</nomen>
                    </ipbdesc>
                    <ipbunits> 1 </ipbunits>
                    <ipbuseoncode/>
                    <ipbsmrcodes>
                        <ipbsmrcode>EX002 </ipbsmrcode>
                    </ipbsmrcodes>
                </ipbpl1>
                <ipbpl1>
                    <ipbfigindex> -2 </ipbfigindex>
                    <partno> P-1234-1-2</partno>
                    <ipbcage>04577</ipbcage>
                    <ipbdesc>
                        <nomen>SUB 2</nomen>
                    </ipbdesc>
                    <ipbunits> 1 </ipbunits>
                    <ipbuseoncode/>
                    <ipbsmrcodes>
                        <ipbsmrcode>EX003 </ipbsmrcode>
                    </ipbsmrcodes>
                </ipbpl1>
                <ipbpl1>
                    <ipbfigindex> -3 </ipbfigindex>
                    <partno>P-1234-1-3</partno>
                    <ipbcage>04577</ipbcage>
                    <ipbdesc>
                        <nomen>SUB 3, Desc </nomen>
                    </ipbdesc>
                    <ipbunits> 1 </ipbunits>
                    <ipbuseoncode/>
                    <ipbsmrcodes>
                        <ipbsmrcode>EX004</ipbsmrcode>
                    </ipbsmrcodes>
                    <ipbpl2>
                        <ipbfigindex> </ipbfigindex>
                        <partno>P-1234-1-3-1</partno>
                        <ipbcage>04577</ipbcage>
                        <ipbdesc>
                            <nomen>SUBSUB 1</nomen>
                        </ipbdesc>
                        <ipbunits> 1 </ipbunits>
                        <ipbuseoncode/>
                        <ipbsmrcodes>
                            <ipbsmrcode>EX005</ipbsmrcode>
                        </ipbsmrcodes>
                        <ipbpl3>
                            <ipbfigindex> -4 </ipbfigindex>
                            <partno>P-1234-1-3-2 </partno>
                            <ipbcage>04577</ipbcage>
                            <ipbdesc>
                                <nomen>SUBSUB 2 </nomen>
                            </ipbdesc>
                            <ipbunits> 1 </ipbunits>
                            <ipbuseoncode/>
                            <ipbsmrcodes>
                                <ipbsmrcode>EX006 </ipbsmrcode>
                            </ipbsmrcodes>
                        </ipbpl3>
                        <ipbpl3>
                            <ipbfigindex> -5 </ipbfigindex>
                            <partno>P-1234-1-3-2-1</partno>
                            <ipbcage>04577</ipbcage>
                            <ipbdesc>
                                <nomen>SUBSUBSUB 1  </nomen>
                            </ipbdesc>
                            <ipbunits> 1 </ipbunits>
                            <ipbuseoncode/>
                            <ipbsmrcodes>
                                <ipbsmrcode>EX007 </ipbsmrcode>
                            </ipbsmrcodes>
                        </ipbpl3>
                    </ipbpl2>
                </ipbpl1>
            </ipbpl0>
        </ipbpl0tbl>
    </ipbplgrp>
</ipbpl>

               

IETM Development

The development of the IETM code was developed using an agile, iterative process.  Cobham staff was involved in each step of the development.  They provided feedback and guidance on the look and feel of the IETM throughout each iteration.  Creating HTML from the XML version of the manuals was relatively straight-forward, however, decisions had to be made about the necessary functionality and look-and-feel.  MIL-STD-38784C was developed with paper or PDF output in mind.  It is not a true IETM specification.  The following decisions were required:

  1. 1.       Navigation from components

  2. 2.       Level of granularity of text (chunking)

  3. 3.       Specialty sections, i.e., List of Tables.

  4. 4.       Handling graphics

Navigation From Components

We decided to use a framed-approach for the IETM.  A navigation panel is on the left-side of the screen.  The user can navigate using a hierarchical table of contents.  We also added widgets for expanding and collapsing navigation hierarchy. We also added a button for searching the entire contents.  Below is the start page of the sample IETM.

jpg image ../../../vol8/graphics/Harvey01/Harvey01-003.jpg

Search Engine

We selected a very inexpensive search engine called Zoom from Wrensoft. Zoom will index directories and subdirectories.  It is highly configurable and had all the functionality required to be able to search and navigate to appropriate locations in the IETM. The search engine can be used to create indexes of websites or documents for CD.  Currently each manual is indexed separately however, if a family of manuals were included the search engine index session the product could index more than a single manual.  The process for creating the index is manual and is performed by Cobham SME's after the book has completed the editorial process and the IETM has been generated and is ready to be delivered.

Zoom creates 5 files that get included into the root directory where the indexed files.  The conversion of XSLT scripts create the interface that enables the search functionality during conversion from XML to HTML.

jpg image ../../../vol8/graphics/Harvey01/Harvey01-004.jpg

Graphics

Graphics we problematic.  Figures in a MIL-STD-38784C paper manual are full-page graphics.  Large, highly detailed graphics do not scale well in a viewing pane. During initial design we provided the graphics in-line but they really weren't acceptable because of the sheer size of the graphics. We solved this problem by providing links in the IETM to the graphic.  A PDF file was created for every figure in the manual using XSL-FO.  A XSL-FO (.fo) file is created int he graphics directory for each graphic in the manual. The XSLT creates a separate Windows batch file that get run separately to call RenderX and create the PDF file of the graphic. The PDF file contained the figure number, the title of the figure and the graphic itself.  All the graphics in the manual are currently in TIFF formatted files.

SVG Graphics

We did a proof of concept for creating SVG.   We tested putting warnings, cautions and notes into a presentation defined in GCSFUI.  The challenge was ensuring that the  text not overflow the boundary of the box.

SVG Generated Warning from XSLT (NOTE:  Text replaced - actual graphic will be used once non-proprietary data is converted)

image ../../../vol8/graphics/Harvey01/Harvey01-005.jpg

We also tested putting overlays on the on the bitmapped graphics.  We successfully created hotspots on the graphics themselves and linked them bi-directionally to and from the IPB to the PDF graphic.  This functionality is very easy to obtain during the conversion by using id/idref functionality.  The time consuming part of this approach was having manpower to manually create the hotspots. This can be very time consuming when there are hundreds of call-outs in a graphic. There were no requirements for graphic hotspots so this functionality was put on the back burner.

None of the SVG graphics were used in the final product but we did prove that it could be done and if the hotspots were available in the graphic during development of the product it is relatively easy to accomplish. 

On-site (Cobham) Production of ETM

After the first few manuals Cobham was off and running on their own.  They were authoring new manuals to MIL-STD-38784 and creating the ETM version based on the environment we setup in SVN.  We have since broken the SVN connection because they don't require support.

Conclusion

This project was very successful.  The Air Force was happy with the deliverables.  Cobham was able to deliver a quality product to the Air Force.  The challenge is still obtaining necessary technical editors who are knowledgeable about SGML (and XML). Whenever there is a turnover in personnel new people need to either have an SGML/XML background or be trained in the technology. 

This project could not have been as successful without the vision of Cobham and their dedication to producing a quality product.  The military SGML specifications are antiquated but with a little ingenuity and knowledge of the technology it is possible to make them work with today's technology without costing the government and creating organizations millions of dollars.

References

  MANUALS, INTERACTIVE ELECTRONIC TECHNICAL - GENERAL CONTENT, STYLE, FORMAT, AND USER-INTERACTION REQUIREMENTS, October 1, 1995, http://www.navsea.navy.mil/nswc/carderock/tecinfsys/etm/sta-spe/pdf/S87268A.PDF

MIL-PRF-87269A, DATA BASE, REVISABLE - INTERACTIVE ELECTRONIC TECHNICAL MANUALS, FOR THE SUPPORT OF, October 1, 1995, http://www.navsea.navy.mil/nswc/carderock/tecinfsys/etm/sta-spe/pdf/S87269A.PDF

THE MOVE TO PAPERLESS TECHNICAL MANUALS IN THE US DOD by Eric L. Jorgensen, Naval Surface Warfare Center, Carderock Division, U.S. Department of the Navy (http://www.navsea.navy.mil/nswc/carderock/tecinfsys/etm/rep-pap-pre/pdf/calseuro.pdf)

MIL-PRF-28001C, MARKUP REQUIREMENTS AND GENERIC STYLE SPECIFICATION FOR EXCHANGE OF TEXT AND ITS PRESENTATION, (http://www.navsea.navy.mil/nswc/carderock/tecinfsys/sgm/sgm-std/doc/28001c.pdf)

DoD CLASSES OF ELECTRONIC TECHNICAL MANUALS, Eric L. Jorgensen, http://www.navsea.navy.mil/nswc/carderock/tecinfsys/etm/rep-pap-pre/pdf/CLASSES.PDF

STANDARD PRACTICE FOR MANUALS, TECHNICAL: GENERAL STYLE AND FORMAT REQUIREMENTS, July 2, 1995, https://acc.dau.mil/adl/en-US/159776/file/29704/MILSTD%2038784.pdf

S1000D, International Specification for Technical Publications utilizing a Common Source Database , http://public.s1000d.org/Pages/Home.aspx



[1] DoD CLASSES OF ELECTRONIC TECHNICAL MANUALS, Eric L. Jorgensen, http://www.navsea.navy.mil/nswc/carderock/tecinfsys/etm/rep-pap-pre/pdf/CLASSES.PDF