Balisage Paper: Heart of Dorkness

1.1. Ars Magica History

Ars Magica is a tabletop roleplaying game about wizards in medieval Europe, although that description understates both the game and the bookkeeping involved. Its "Mythic Europe" treats medieval beliefs as substantially true: saints perform miracles, faeries haunt the wilderness, demons tempt the unwary, and scholars debate Aristotle while being menaced by dragons. Its central characters are magi of the Order of Hermes, who live in communities called covenants and devote themselves to magical research, political infighting, and explaining why this season’s laboratory explosion was entirely foreseeable.

Figure 1: The Ars Magica Logo

Image description

The words "Ars Magica" appear in an elaborate medieval-style display face on a white background. The enlarged capital A and M are red illuminated-letter forms with knotwork and gold outlining; the remaining letters are blue with red and gold accents.

Jonathan Tweet and Mark Rein-Hagen created the game, which Lion Rampant first published in 1987. It later passed through White Wolf and Wizards of the Coast before Atlas Games acquired it in 1996. Atlas published the fourth edition that year and the fifth in November 2004 [Atlas Games 2024a; Atlas Games n.d.].

Rather than replace the fifth edition every few years, Atlas supported it with a large, interconnected library. A character may possess a host of defining characteristics, associated items and locales, and decades of advancement history. All of these characteristics, items, and locales have their own structures and relationships.

In October 2024, Atlas released the text of the fifth edition line under the Creative Commons Attribution-ShareAlike 4.0 International licence. The Ars Magica Open Licence covers 53 books, but not their artwork, cartography, logos, trade dress, or graphic design [Atlas Games 2024b]. The books themselves were not freely reproducible, but their textual rules and game content could now be reused in software.

This looked like a job for XML!

There had already been a substantial character creation program for the game. Alter Ego Software released MetaCreator in 1994, a generic program designed to be customized for a variety of roleplaying game systems. The Windows application supported many such roleplaying games through content-specific data sets, and its Ars Magica implementation went far beyond generating a starting character [Alter Ego Software n.d.; Kvam 2002; Project: Redcap 2022]. It was powerful and comprehensive, but visibly belonged to an earlier generation of Windows software.

Peggy and Bruce Kvam retired in 2022 and discontinued MetaCreator sales. The company website and email addresses disappeared, and Atlas could not assume distribution or maintenance because it neither owned the software nor employed software development staff [Nephew 2022].

Ars Magica therefore had a newly open textual corpus but no modern, maintained character-management program. I had played the game for decades, built data-driven software, and remained vulnerable to projects beginning, "Wouldn’t it be cool if I...?"

I foolishly decided to build a free, open-source replacement: not just a character generator, but eventually a tool for managing the accumulated debris of an immensely supported roleplaying game and the adventures arising from it. I called it Hermetic Foundry: "Hermetic" for the Order of Hermes, and "Foundry" for a place where characters and stories could be constructed.

It sounded much better than Ben’s Newest Enormous Time Suck.

1.2. The Skill Prosthetic

There was one immediate obstacle: I am bad at user interface design.

This is not false modesty. I have programmed for four decades, mostly in data processing, business logic, databases, and back-end systems. When a project required polish, somebody with visual talent handled it. My design instincts are practical in the sense that a concrete bunker is practical. I have the artistic sensibility of a rock. People recoil from my stick figures.

I delegated from weakness. I described the application, its information, and the atmosphere I wanted. Ars Magica evokes medieval scholarship, illuminated manuscripts, alchemy, and laboratories full of improbable apparatus. The interface should reflect that setting without becoming unreadable or descending into faux-medieval crimes.

ChatGPT produced a static mock-up with a parchment-like workspace, navigation for major record types, an information pane, and a visual language suggesting manuscripts and alchemical diagrams without requiring candlelight.

Figure 2: The Original ChatGPT-Generated Hermetic Foundry Interface Mockup

Image description

A wide desktop-application mock-up for Hermetic Foundry. A dark navigation column on the left lists Ars Magica record types, while the main pale workspace shows the Guided Creation workflow for the character Guillaume de Mortemer. The centre contains numbered creation steps, biographical fields, a portrait, summaries, and linked content. A contextual information pane and recent-changes panel appear on the right.

I expected generic panels wearing parchment texture as a hat. Instead, the design was remarkably close to what I wanted. After a few adjustments, it was more than 80% of the way to the visual identity Hermetic Foundry still used at the cutoff date. Its broad layout, colour scheme, typography, and division into navigation, content, and contextual information all survived.

I still regarded ChatGPT as a prosthetic rather than a development partner: a ruler supplying straight lines to somebody incapable of drawing one. Architecture, data modelling, and XML would remain mine. The image would be a target for the interface I eventually built.

Then I wondered whether Codex could turn it into a prototype.

ChatGPT recommended Tauri, SvelteKit, TypeScript, and Rust, all largely new to me. That seemed like a benefit rather than a warning: the project was for enjoyment, and learning a new stack sounded better than repeating familiar work. ChatGPT also proposed a structure and generated detailed instructions for Codex.

I expected rough scaffolding that I would later replace. Instead, Codex produced a working desktop application with menus, navigation, reusable components, styling, and the beginnings of real interaction.

Figure 3: The First Working Codex Implementation of the Hermetic Foundry Interface

Image description

An early working Hermetic Foundry window implementing the mock-up’s three-column layout. The left side contains navigation for characters and other record types, the centre shows Guillaume de Mortemer’s guided-creation workflow and a large placeholder portrait, and the right side contains information, linked-content, and recent-changes cards. The implementation is recognizably based on the mock-up but is more crowded and visually rough.

The prototype needed refinement, but not wholesale replacement. This was not slop. It preserved the mock-up’s visual character while expanding it into an application capable of holding Ars Magica’s many kinds of information. The code was organized and comprehensible despite using a stack I had never worked with.

The first important result was not that ChatGPT and Codex replaced something I did well. They extended me into an area where I was weak. ChatGPT supplied possibilities I would not have proposed; Codex made them work; I supplied enough experience to understand, test, and revise the result.

I considered it a successful rapid prototype.

My eldest child had another name for it.

1.3. In Which I Discover How Uncool I Am

Figure 4: A Horrified Teenager Recoils

Image description

A black-and-white cartoon in a living room. Ben, bald and wearing glasses and a goatee, holds an open laptop and looks sheepish; his T-shirt reads "LAME." His stocky, bespectacled eldest son recoils in theatrical horror with both hands raised; his shirt reads "SO COOL." A large speech bubble from the son reads, "Are you...vibe coding?!"

I was showing the prototype to my partner, who has considerable artistic ability and far more patience for interface details than I do. Our eldest child wandered in, looked at the application, looked at me, and asked with all the contemptuous power he could muster:

"Are you...vibe coding?"

Being old and not cool, I had no idea what he meant. He explained it to me.

I indignantly denied the charge. I was merely using generative AI to design an interface, choose a stack, write implementation instructions, and generate most of a working prototype from natural-language descriptions.

I was rapid prototyping.

Andrej Karpathy had introduced "vibe coding" earlier that year to describe a carefree style of AI-assisted programming: state what you want, accept generated changes with little inspection, paste errors back into the model, and gradually lose track of the code [Karpathy 2025]. The programmer decides what the application should do and whether it appears to work, but surrenders much of the implementation.

The term quickly broadened to cover almost any process in which a programmer described an outcome and an LLM generated substantial code. The boundary became unclear. Was it vibe coding only if nobody read the code? What if the developer understood the architecture, reviewed changes, ran tests, and intervened? At what percentage of generated code did respectable engineering acquire vibes?

Sarkar and Drosos describe a broader conversational practice: developers prompt an AI system, inspect or test the result, revise instructions, and sometimes edit directly. Their study found that expertise remained important even when the programmer typed less code; it moved toward specification, context management, evaluation, and debugging [Sarkar and Drosos 2025].

By that definition, my child had me dead to rights.

I was expressing goals in prose, asking ChatGPT to translate them into development instructions, and giving those instructions to Codex. I retained control over the product, architecture, data model, and acceptance criteria, but no longer constructed each component myself.

I still considered the generated code scaffolding. Surely the illusion would collapse when the project reached XML Schemas, package architecture, validation, document relationships, and complicated rules. I would then take over the real development.

My denial was sincere. Vibe coding, as I understood it, meant abandoning comprehension and trusting the machine. I was merely using the machine to do almost everything while firmly intending to understand it later.

Having established that distinction to my own satisfaction, I wondered what would happen if I stopped making it.

1.4. The Descent Begins

The accusation that should have ended the matter led me to ask: what if I actually tried to vibe code the entire application? How terribly wrong would it go?

Until then, ChatGPT and Codex had been prototyping tools. Their work exceeded expectations, but the deeper architecture would surely expose their limits. Data modelling, validation, file formats, cross-document relationships, and Ars Magica’s interlocking rules seemed likely to produce nonsense – or "slop," in the common parlance.

Or would they?

I decided to use ChatGPT and Codex for as much development as possible, intervening directly only when they became hopelessly lost or when correction took longer than implementation. I expected failure; the interesting questions were where, how badly, and what might remain afterward.

XML had always been central. Other roleplaying tools had demonstrated its usefulness: HERO Designer used a kludgy pseudo-XML, while Aurora Builder used XML for extensible Dungeons & Dragons content and character sheets [HERO Games n.d.; Aurora Builder n.d.]. Their files worked, but often resembled serializations of internal application state rather than independently designed vocabularies. Elements existed because the software needed somewhere to put a value, not because the document expressed its domain particularly well.

That was moderately acceptable engineering. Most users would never inspect the files. To somebody who cares perhaps too much about XML design, however, they were singularly unimpressive.

Hermetic Foundry’s XML should remain intelligible outside the application. Its structures should reflect Ars Magica, not the incidental arrangement of a user interface. It should support validation, extension, provenance, links between records, and exchange between installations. A character file should describe a character, not resemble a data object tipped into angle brackets.

This made the experiment more demanding. A generated button could be judged by clicking it. Generated XML architecture had to preserve structural and semantic decisions across schemas, documents, application code, imported content, and repeated revision. A locally plausible change could remain well formed while damaging the vocabulary as a whole.

I expected the LLMs to fail here. They could imitate tags, but imitation was not the same as maintaining a coherent vocabulary. I anticipated malformed documents, invented structures, inconsistent names, broken references, and schemas that looked convincing until a validator met them.

Balisage work already suggested a more complicated picture. DeRose found that LLMs knew a surprising amount about XML and could often recognize or generate markup, although performance remained inconsistent and prompt-sensitive [DeRose 2024]. I had not yet appreciated the difference between "often wrong" and "reliably wrong."

I also expected application quality to deteriorate as the repository grew. Small prototypes are forgiving; large programs accumulate dependencies, conventions, and decisions whose importance is not visible in the file being edited. An agent might write excellent local code while steadily making the system less coherent.

The experiment therefore had a simple horizon: continue until AI-assisted development became slower than writing the code myself, defects overwhelmed progress, or the XML became intolerable. I expected one threshold to arrive quickly.

In the meantime, it might save work, and it would certainly be interesting.

1.5. The Ninety-Ninety Rule

Tom Cargill’s Ninety-Ninety Rule observes that the first 90% of a program consumes the first 90% of development time, while the remaining 10% consumes another 90%. The arithmetic is impossible; the scheduling implications are distressingly accurate [Bentley 1985].

Hermetic Foundry was nowhere near either 90% at the time of this writing

At that time, the repository contained 262 tracked text files and 77,759 nonblank lines, including 57,135 lines of Svelte, TypeScript, Rust, and CSS. The rest comprised XML Schemas, rules and application data, tests, fixtures, scripts, documentation, and configuration. The count excluded dependencies, compiled output, lockfiles, images, fonts, binaries, and ZIP archives.

This was not an authorship count. Git could not reliably distinguish Codex-generated lines from those I wrote or revised, which only accounted for about 200 lines of code. It measured the project produced through this process.

The application was roughly 25% complete. It had a desktop shell, XML entity architecture, package loader, character sheets, and alpha versions of several record types. It could create, edit, save, reload, and validate meaningful Ars Magica data.

Still ahead were guided creation and many game systems; a custom-content workbench for users and publishers; XML help; print and PDF output; usability and performance work; deep rules automation; and the testing, packaging, and release effort needed before inflicting it on an unsuspecting world.

The development plan suggested a final version near 250,000 nonblank tracked lines, although projecting completed size from unfinished software is an excellent way to produce a precise-looking number with no obligation to be correct [Wolfe 2026].

The significant point was not that Hermetic Foundry had become large. It had survived long enough to become a large, incomplete software project instead of a small, complete failure. ChatGPT and Codex had carried it far beyond the point where I expected to take over or abandon it.

Whether that represented a repeatable method, extraordinary luck, or merely the first 90% remained to be seen.

2.1. How Much to Code?

Using Codex for isolated functions would prove little. I had used development assistance tools for years. The useful question was whether an LLM agent could perform a substantial share of real development without reducing me to a full-time janitor for spaghetti code.

I fully committed. ChatGPT and Codex would implement as much as possible. I would supply product vision, domain knowledge, architecture, requirements, and acceptance decisions, but fastidiously avoid coding merely because I could do it faster or more comfortably. Direct intervention was reserved for cases where Codex became lost, repeatedly failed, or required more explanation than implementation.

This did not test whether expertise was unnecessary. I inspected designs and important code, ran the application, validated XML, tested behaviour, and rejected inadequate work. The restriction concerned who implemented it, not whether it received scrutiny. I would treat Codex like a junior developer.

I expected that division to change as the project became technical. Codex might manage the shell and ordinary interface work, but I assumed I would eventually take over the XML architecture, schema revisions, package format, and corpus conversion. Small modelling errors in those areas could spread throughout the application.

I also expected to replace much generated code once Codex reached its limits. I did not know whether that would happen after a week, a month, or the first complicated schema revision. Beyond my conviction that the XML would eventually be terrible, I had no strong prediction.

I asked ChatGPT what to expect and how best to work with Codex. It recommended conventional engineering practices for a synthetic developer: define goals, provide relevant architectural context, divide work into bounded tasks, identify systems that must not change, specify user-visible behaviour, require tests, and demand a report of changes. It also supplied sources so I could check its advice.

Large, vague requests were discouraged. Work should proceed in increments that could be built and reviewed; Codex should inspect the repository first, follow existing patterns, and avoid redesigning unrelated systems. These recommendations entered both the development plan and my recurring Codex instructions [Wolfe 2026].

This resembles what surveys describe as planning-driven, test-driven, and context-enhanced vibe coding: the agent receives explicit plans, repository context, constraints, and feedback from compilers, tests, execution, and human review [Ge et al. 2025]. Results depend not only on the model but on the environment and the human’s ability to manage context.

My workflow was therefore more structured than "make me an app" followed by repeated clicks on "Accept." I described features in detail, ChatGPT converted them into implementation tasks, and Codex worked inside a repository governed by schemas, tests, version control, and an expanding development plan. Like a junior developer fresh out of school, it was skilled, eager, and required adult supervision.

Codex still had freedom. I specified behaviour rather than exact code. It could inspect the repository, choose an approach, create or modify components, revise permitted data models, and add tests. I retained the right to reject the result without micromanaging every function.

This was not a controlled experiment, nor a test of whether Codex could replace a programmer. It was a case study of how much implementation an agent could perform when directed by an experienced programmer who deliberately refrained from doing it themself. Evaluation combined elapsed time, code volume, build functionality, validation, and completed features with professional judgement about maintainability and correctness.

The rule was simple: let the machine code until letting the machine code became the stupid option.

2.2. Deciding on the Platform

Hermetic Foundry would use XML as the canonical format for saved records, rules content, custom material, manifests, help files, and interchange.

This was not only because I really, really like XML. Ars Magica is document-oriented. All of its objects like characters, items, etc. are recognizable documents containing nested, ordered, repeatable, and optional structures. They require provenance, extensibility, validation, and links to other records. A character is not merely a row with an alarming number of columns.

Other roleplaying tools showed XML’s practicality. HERO Designer saved characters in XML-based files and supported pseudo-XML export templates; Aurora Builder used XML to assemble extensible Dungeons & Dragons content [HERO Games 2021; Aurora Builder 2020]. Aurora’s generic <element type="..."> vocabulary was flexible, but semantically different objects shared one container and relied heavily on identifiers, free-form expressions, and imperative-looking rules. HERO Designer’s format likewise resembled a serialization of its internal model.

HERO Designer Snippet

                     
<?xml version="1.0" encoding="UTF-8"?>
<CHARACTER version="3.0" TEMPLATE="builtIn.Superheroic.hdt">
  <BASIC_CONFIGURATION BASE_POINTS="250" DISAD_POINTS="150" EXPERIENCE="185" EXPORT_TEMPLATE="/home/ben/HeroDesigner/ExportTemplates/DeluxeChampions.hde" RULES="Default" />
  <CHARACTER_INFO CHARACTER_NAME="Rebel" ALTERNATE_IDENTITIES="Max Gable" PLAYER_NAME="NPC" HEIGHT="75.98425196850394" WEIGHT="251.32696226832724" HAIR_COLOR="Dark Brown" EYE_COLOR="Blue" CAMPAIGN_NAME="Brigade Universe" GENRE="Modern Champions" GM="Ben Wolfe">
    <BACKGROUND />
    <PERSONALITY />
    <QUOTE />
    <TACTICS />
    <CAMPAIGN_USE />
    <APPEARANCE />
    <NOTES1 />
    <NOTES2 />
    <NOTES3 />
    <NOTES4 />
    <NOTES5 />
  </CHARACTER_INFO>
  <CHARACTERISTICS>
    <STR XMLID="STR" ID="1067075197597" BASECOST="0.0" LEVELS="10" ALIAS="STR" POSITION="1" MULTIPLIER="1.0" GRAPHIC="Burst" COLOR="255 255 255" SFX="Default" SHOW_ACTIVE_COST="Yes" INCLUDE_NOTES_IN_PRINTOUT="Yes" NAME="" AFFECTS_PRIMARY="Yes" AFFECTS_TOTAL="Yes">
      <NOTES />
    </STR>

Aurora Builder Snippet

                     
<build>
  <input>
    <name>Aerya Windspun</name>
    <gender>Female</gender>
    <player-name>NPC</player-name>
    <experience>34000</experience>
    <attacks>
      <description><![CDATA[]]></description>
      <attack identifier="a0fdffa5-b03a-4293-bcfb-3ead0641bfb8" name="Rapier" range="5 ft" attack="+7 vs AC" damage="1d8+4 piercing" displayed="true" ability="Dexterity">
        <description><![CDATA[Finesse]]></description>
      </attack>
    </attacks>
    <backstory><![CDATA[You thrive in front of an audience. You know how to entrance them, entertain them, and even inspire them. Your poetics can stir the hearts of those who hear you, awakening grief or joy, laughter or anger. Your music raises their spirits or captures their sorrow. Your dance steps captivate, your humor cuts to the quick. Whatever techniques you use, your art is your life.]]></backstory>
    <background-trinket>
    </background-trinket>
    <background-traits>Nobody stays angry at me or around me for long, since I can defuse any amount tension. I’m a hopeless romantic, always searching for that "special someone."</background-traits>
    <background-ideals>Beauty. When I perform, I make the world better than it was. (Good)</background-ideals>
    <background-bonds>My instrument is my most treasured possession, and it reminds me of someone I love.</background-bonds>
    <background-flaws>I’m a sucker for a pretty face.</background-flaws>
    <background>
      <feature name="Backstage Pass">
        <description><![CDATA[You’ve learned that most of the real business of entertainment (or any other venture) happens behind the scenes. It’s easy for you to case what sorts of audiences attend what venue – like how toughs gather at the Blushing Mermaid or how brash patriars congregate at the Helm and Cloak. After a successful performance, you may meet an enthusiastic member of the crowd – someone of an occupation or social class that frequents the establishment. This contact is delighted to talk with you, and to listen.]]></description>
      </feature>
    </background>

I wanted stronger domain claims. A spell should be a spell, not a generic element whose type happened to say Spell. Character-instance values should be distinct from package definitions. References should be explicit, portable, and accompanied by enough embedded information that a missing source would not turn a sheet into opaque identifiers.

XML was text-based, inspectable, diffable, transformable, and independent of the application. It handled descriptive and structured content together, supported namespaces, and had mature parsers and validators. Most importantly, documents could be checked against a published contract rather than handed back to the creating application and judged by whether it complained.

The files should outlive implementations. A character should remain intelligible if Hermetic Foundry disappeared or changed technology stacks. Users should be able to inspect, transform, validate, archive, and repair their data without reverse-engineering binaries or depending on a cloud service.

XML did not have to enable the fastest runtime index. Large content sets would eventually benefit from caches or a local database. XML remained the canonical portable representation; disposable indexes could be rebuilt from it.

DTDs were not sufficient for the schema layer I wanted. Hermetic Foundry needed namespaces, reusable complex types, datatypes, facets, enumerations, identity constraints, and documentation on schema components. I chose W3C XML Schema. XSD could distinguish identifiers, dates, integers, scores, categories, and descriptions; reuse model and attribute groups; and attach xs:documentation and xs:appinfo. XSD 1.1 added assertions and conditional type assignment, although validator support remained uneven [W3C 2012a; W3C 2012b].

Schema validity would not encode all Ars Magica rules. Some constraints were contextual, procedural, package-dependent, or cross-document. Schematron, application validation, and later automation would still be needed. The schema’s role was structural and lexical integrity, not the entirety of medieval magic.

For the application, I wanted local files, offline use, private roleplaying campaign data, package import, and no dependence on paying my server bill, without building separate native interfaces for each operating system. ChatGPT recommended Tauri: a web interface in the operating system’s webview with a Rust native layer and a defined message boundary. It suited an application dominated by forms, lists, inspectors, previews, and data-entry workflows while retaining native filesystem and packaging support [Tauri 2026].

SvelteKit supplied reactive components, routing, and build structure [Svelte 2026]. TypeScript supplied the main application language and a static type system that gave Codex fast feedback when it misunderstood interfaces or object shapes [Microsoft 2026b]. Rust handled the native side, bringing strong compile-time protections to files, archives, checksums, and operating-system interaction [Rust Project 2026]. All were largely new to me, maintaining the project’s tradition of making every choice slightly more alarming than the last.

Extensibility required more than individual XML files. Users and publishers needed to create material with dependencies, licences, documentation, media, schemas, versions, and provenance. I devised a ZIP-based XML package containing a manifest, licence, content files, and optional help and media. The manifest identified the package, version, files, checksums, and dependencies; Hermetic Foundry could validate the archive, detect conflicts, and request confirmation before installation.

The result was recursively XML-like: XML described game entities, XML Schemas described those documents, and another XML document described the package containing them.

All I needed was for an LLM to implement it without becoming confused.

2.3. Toolbox

ChatGPT and Codex did the conspicuous work, but a collection of less glamorous tools enforced consequences.

Visual Studio Code was my main repository view – not Visual Studio, because Microsoft apparently considers removing one word sufficient differentiation. I used it to inspect Codex’s changes, search the codebase, compare patterns, and make the occasional permitted edit. Its terminal and Git integration kept code, commands, diagnostics, and history together [Microsoft 2026a]. The Codex extension also gave the agent repository and editor context [OpenAI 2026a].

PowerShell 7 supplied the Windows command line for dependencies, scripts, validation, inspection, metrics, and diagnosis [Microsoft 2026c]. The standard quality-assurance cycle became:

pnpm check
pnpm test
pnpm tauri build

These commands checked Svelte and TypeScript, ran tests, and built the packaged desktop application. Development mode was not always representative, so a feature had to survive the built application. A successful build did not prove correctness; a failed one was persuasive evidence of a problem.

GitHub provided source control, while GitHub Desktop handled branches, diffs, commits, and merges [GitHub 2026]. I gave Codex bounded branch work, reviewed it, ran quality assurance, and merged only after manual acceptance. Version control mattered because Codex could distribute a misunderstanding across components, tests, models, and styles before I finished making tea. Checkpoints kept the damage inspectable and reversible [OpenAI 2026b].

I used XML Copy Editor for direct inspection of XML and XSD [Schmidt et al. n.d.]. Apache Xerces2 Java provided independent XML Schema validation, including XSD 1.0 and 1.1 support [Apache Software Foundation 2026]. Neither ChatGPT, Codex, nor Hermetic Foundry could declare their own XML correct merely because it looked convincing. Well-formedness, schema validity, semantic correctness, and compliance with Ars Magica rules were separate questions.

Google Docs held the development plan. By the cutoff, version 1.5 had reached 103 pages, recording vision, terminology, architecture, milestones, exclusions, definitions of done, and the division of labour [Wolfe 2026]. It became the project’s external memory as conversations ended, context windows filled, and accepted decisions became surprisingly forgettable. Version history also preserved earlier revisions [Google 2026].

Together, these tools formed the containment system around the generative models.

2.4. The Pass/Fail Point

An experiment needs a way to fail; otherwise it is just a project with unusually self-important documentation.

The clearest failure condition was economic: if directing, correcting, and testing ChatGPT and Codex took longer than doing the work myself, the process had failed. Generation speed meant nothing if I then spent days untangling the result. The useful measure was total human effort required to reach an acceptable implementation.

My experience supplied an imperfect baseline for how long substantial features normally took me. The second condition was less precise: the software could become too buggy to justify continuing. Ordinary interface defects were expected. Architectural damage, silent data loss, irreproducible behaviour, or repeated regression would indicate something worse.

Cumulative failure worried me most. A feature could appear correct while depending on a wrong data-model assumption. Codex might then copy that pattern into later work, producing an internally consistent implementation of the wrong idea with industrial efficiency.

XML supplied another threshold. I could tolerate rough interface work because it was revisable. Bad XML architecture would poison nearly every feature and become more expensive with every saved record and package. Invalid schemas would be obvious, but validity alone was insufficient. I expected coherent naming, sensible type reuse, meaningful structure, appropriate constraints, clear ownership, and extensibility. If ChatGPT or Codex could not maintain those qualities, I would take over the XML even if the broader experiment continued.

Success meant software as good as, or better than, what I could reasonably produce, reached with less of my time. "Better" could mean an interface beyond my design ability, broader tests, stronger documentation, or more features in the same period. The comparison was with my real practice, not an imaginary version of me who never tired, became distracted, or decided comments could wait.

The project still depended on my knowledge of Ars Magica, XML, architecture, and management. The question was whether the tools could multiply that expertise through implementation.

My practical criteria were:

If the answers became "no," I would stop and salvage what I could.

3.1. Telling It to Tell That to Do the Thing

When I first asked ChatGPT how to turn the mock-up into a working prototype, it produced instructions I could paste into Codex. I treated that as a convenient shortcut. The pattern repeated whenever I used ChatGPT to investigate unfamiliar technologies or compensate for unclear documentation: its answers often arrived as implementation tasks rather than general advice.

Eventually, the significance became obvious. I did not need to write detailed Codex instructions for every change, bounding with limits and expressing what I needed it to create, while also making certain it didn’t create regression errors. I could describe behaviour in ordinary prose using the vocabulary of the application and game. ChatGPT could organize that into files, architecture, constraints, tests, and expected results.

A request might say:

"The second Notes field is redundant. Remove it. All Quick Items should support an optional image. Remove the Entry Mode field. Preserve existing save behaviour and do not redesign unrelated parts of the sheet."

ChatGPT would tell Codex to inspect established Quick Sheet patterns, locate the relevant Svelte components and XML mappings, preserve ownership rules, update tests, run quality assurance, and report changed files.

My Instructions to ChatGPT

The next round of changes for Codex to the Quick Creature are:
1. "Appearance" and "Behavior" should have rich text editing.
2. In "Characteristics," each stat should only accept the following: negative numbers denoted with a negative sign; positive numbers with or without a positive sign; zero; or zero with a positive sign in front of it. Each stat should have an optional "Description" field associated with it. This may require re-organizing how the Characteristics are displayed.
3. "Summary" is meaningless. Remove it.

ChatGPT’s Resultant Instructions for Codex

Codex Instructions — Quick Creature Bug Pass — Rich Text and Characteristics
1. Short explanation
Continue work on Milestone 3.7.2 — Quick Creature Alpha.
This is a narrow Quick Creature bug pass covering:
1. rich-text editing for Appearance and Behavior;
2. stricter input validation for Characteristics values;
3. optional Description fields for each Characteristic;
4. removal of the meaningless Summary field.
Do not work on unrelated Quick Creature panes in this pass.

2. Branch
Continue on the active Quick Creature 3.7.2 branch if this work is already in progress.
If starting fresh from main:
git checkout main
git pull
git checkout -b spiral-3-milestone-3.7.2-quick-creature-characteristics-fixes

3. Required pre-coding inspection report
Before coding, report:
1. current branch;
2. git status;
3. where the Quick Creature Concept and Description pane is implemented;
4. how Appearance and Behavior are currently stored;
5. whether they already use plain strings, HTML, Tiptap JSON, or some existing rich-text wrapper;
6. where the Quick Creature Characteristics pane is implemented;
7. how each Characteristic value is currently validated;
8. whether the current schema/model supports per-Characteristic descriptions;
9. where the current Summary field is stored and displayed;
10. whether removing Summary from the UI has any save/load implications.
Do not discard or overwrite existing work.

4. Implementation requirements
4.1 Appearance and Behavior must use rich-text editing
In Quick Creature’s Concept and Description pane, change:
• Appearance
• Behavior
so that both use the same rich-text editing pattern used elsewhere in the app for active multi-line rich-text fields.
Requirements:
• Use the shared Tiptap editor/component already used in Quick Character and Quick Creature rich-text fields.
• Preserve EB Garamond body styling.
• Preserve formatting on Apply and save/reload.
• Do not expose raw HTML or raw Tiptap JSON in ordinary UI.
• Use richTextToPlainText or the existing equivalent for summaries, previews, search, logs, validation, and exports.
• Do not use Tiptap for single-line fields or generated summaries.
Manual formatting controls should match existing Quick Sheet rich-text controls.

4.2 Characteristics value validation
Each Quick Creature Characteristic stat field must only accept integer values in the following forms:
Valid examples:
• -3
• -2
• -1
• 0
• +0
• 1
• +1
• 2
• +2
Invalid examples:
• --1
• ++1
• 1.5
• +1.5
• abc
• 1a
• empty value, if the stat is required
• -0
The intended rule is:
• negative numbers must have a single leading minus sign;
• positive numbers may have a single leading plus sign or no sign;
• zero may be written as 0 or +0;
• decimals are not allowed;
• non-numeric text is not allowed;
• duplicate signs are not allowed;
• -0 is not allowed.
A safe validation pattern would be equivalent to:
/^(?:-[1-9]\d*|\+?[1-9]\d*|\+?0)$/

Trim leading/trailing whitespace before validation, but do not allow internal whitespace.
If the schema or existing app rules impose a valid numeric range for creature Characteristics, preserve that range after the format check.
Use user-facing validation messages. Do not expose regexes, schema internals, or raw validation output.

4.3 Characteristics display and optional descriptions
Each Characteristic should have an optional Description field associated with it.
This may require reorganizing the Characteristics pane.
Implement a clean display that remains consistent with Quick Character styling. Acceptable layouts include:
• one card/row per Characteristic with:
  • Characteristic name;
  • stat input;
  • optional Description field;
• or a compact two-column layout, if it remains readable and the Description fields do not feel cramped.
Requirements:
• Description is optional.
• Description should be editable.
• If the description is multi-line, use the shared rich-text editor only if consistent with current rich-text conventions and not visually excessive.
• If a plain multi-line text area is more appropriate for compact per-stat notes, use that and keep it visually consistent.
• Values and descriptions must preserve state between tabs/panes.
• Values and descriptions must persist on save/reload if the schema/model supports them.
Important: do not fake persistence.
If the current schema/model does not support per-Characteristic descriptions, report that clearly before making schema changes. Do not silently add UI-only description fields that disappear on reload.
If schema/model support exists, implement the fields fully.
If schema/model support is missing but a safe existing generic notes/custom-metadata mechanism already exists for this exact purpose, use it only if it persists cleanly and does not expose internals.

4.4 Remove Summary
Remove the visible Summary field from the Quick Creature Characteristics pane.
Requirements:
• The user should not see or edit Summary.
• Summary should not be required.
• Removing it should not break existing creature load/save.
• If existing XML files contain Summary data, do not destroy it unless the current model already discards unknown/unused fields.
• Do not remove or alter the XML schema just to remove the visible UI field.
If there is more than one visible field named Summary in Quick Creature, inspect and confirm which one is meaningless in this context before removing. The target for this pass is the Summary field associated with the Characteristics work.

4.5 Apply and save behavior
Preserve the recently fixed Apply behavior.
Requirements:
• Apply Changes must not save files.
• Apply Changes must not trigger File Error #1007.
• Apply Changes must be disabled when there are no unapplied changes.
• Apply Changes must become enabled after edits to Appearance, Behavior, Characteristics values, or Characteristic descriptions.
• Apply Changes must preserve the current pane’s data in the in-memory draft.
• Fields must remain editable after Apply.
• Save must still validate and persist correctly.

5. Manual QA checks
Appearance and Behavior rich text
1. Open Quick Creature.
2. Go to Concept and Description.
3. Enter formatted text in Appearance.
4. Enter formatted text in Behavior.
5. Apply Changes.
6. Confirm fields remain editable.
7. Save.
8. Reload.
9. Confirm formatting persists.
10. Confirm no raw HTML or Tiptap JSON appears.
Characteristics validation
For each Characteristic field, test:
• -3 — accepted.
• 0 — accepted.
• +0 — accepted.
• 3 — accepted.
• +3 — accepted.
• -0 — rejected.
• ++1 — rejected.
• --1 — rejected.
• 1.5 — rejected.
• abc — rejected.
Confirm invalid values produce user-facing validation messages.
Characteristic descriptions
1. Enter a Description for at least two Characteristics.
2. Leave at least one Description blank.
3. Apply Changes.
4. Switch panes and return.
5. Confirm descriptions remain.
6. Save and reload.
7. Confirm descriptions persist if schema/model support exists.
If persistence is not supported, report that before implementing UI-only fields.
Summary removal
1. Open the Characteristics pane.
2. Confirm Summary is no longer visible.
3. Confirm Summary is not required.
4. Confirm existing creature files still load.
5. Confirm save/reload still works.
Apply behavior regression
1. Open Quick Creature.
2. Make no changes.
3. Confirm Apply Changes is disabled.
4. Edit a Characteristic value.
5. Confirm Apply Changes enables.
6. Apply Changes.
7. Confirm Apply Changes disables.
8. Continue editing.
9. Confirm fields are not locked.

6. Automated checks
Run:
pnpm check
pnpm test
pnpm tauri build

Do not run pnpm lint.
If pnpm tauri build emits only the known Vite chunk-size warning, treat it as non-fatal.

7. Completion report
Report:
1. branch name;
2. files changed;
3. how Appearance and Behavior were converted to rich text;
4. how Characteristic value validation was implemented;
5. whether any existing range validation is preserved;
6. how per-Characteristic Description fields are stored;
7. whether Description fields persist on save/reload;
8. what happened to the Summary field;
9. manual QA results;
10. automated check results;
11. any remaining schema/model limitations.

8. Do-not-change boundaries
Do not change unrelated Quick Creature panes.
Do not weaken save validation.
Do not break the recently fixed Apply behavior.
Do not make Apply perform file persistence.
Do not implement autosave.
Do not expose raw HTML, Tiptap JSON, XML, schema internals, validation internals, IDs, or diagnostics in ordinary UI.
Do not change XML schemas unless the per-Characteristic Description requirement is blocked and Ben explicitly approves the schema change.
Do not change package content.
Do not redesign the app shell.
Do not change the app-wide brand/title font.

The workflow became:

Information also moved backward. Codex could identify a schema limitation, architectural dependency, or repository conflict invisible from the interface. I returned that report to ChatGPT, which helped interpret it and revise the task. Build failures, regression reports, and screenshots entered the same loop.

ChatGPT therefore translated between product language and repository language. I spoke in terms of users, Ars Magica, and project goals; Codex worked in components, types, schemas, tests, and repository state.

The tasks acquired a regular structure – context, goal, requirements, prohibitions, tests, and expected output – which eventually became a template in the development plan [Wolfe 2026].

The discovery was not that prose could replace code. Carefully written prose could act as a specification layer. Precision moved earlier: vague descriptions produced vague tasks, while detailed descriptions usually produced executable ones.

For a while, I concluded that more detail was always better.

It was not.

3.2. ...Not That Much

The first weakness was specification bloat.

ChatGPT preserved earlier constraints as a precaution. Useful global rules became mixed with obsolete branch assumptions, completed milestones, and implementation details no longer relevant to the task. At first, I assumed these references mattered. Repository inspection showed that many were simply stale.

More detail was not always more context. Sometimes it was noise. Codex spent time examining unrelated systems and risked preserving old workarounds or touching features outside scope.

I began removing outdated instructions and retaining only current requirements, directly affected systems, and genuinely global rules. Tasks became narrower: preserve established patterns, do not alter unrelated schemas, run standard checks, and stop.

The opposite problem was underspecification. I had described individual screens in detail but never defined an application-wide style guide. Codex solved each interface problem locally; the screens worked, but layouts, controls, spacing, and interactions drifted. Nothing in a single task was obviously wrong. The missing requirement existed between tasks.

ChatGPT also encouraged scope creep. A request to correct one field could expand into restructuring a pane, revising validation, updating documentation, and reconsidering the data model. Some additions were sensible. They were also not the task.

Negative requirements became nearly as important as positive ones:

Do not redesign unrelated systems. Do not change the schema unless separately approved. Do not alter save behaviour. Do not invent a new interaction pattern when one already exists.

This reflects a wider problem in agentic development: relevant context helps, but irrelevant or conflicting context can distract the model [Ge et al. 2025]. Instructions required curation, not accumulation.

The surprise was what I did not encounter. I found no clear case where ChatGPT or Codex invented a library, fabricated a schema feature, cited an imaginary repository component, or claimed success despite contrary evidence. They misunderstood requirements, chose poor interfaces, preserved obsolete assumptions, and fixed symptoms rather than causes. Those were serious development errors, but not hallucinations in the ordinary sense.

The tools behaved less like fountains of plausible fiction than overenthusiastic junior developers: carrying too much context, missing unstated requirements, widening tasks without permission, and needing someone to say when enough was enough.

The answer was not maximum information. It was the right information for the current task.

3.3. Finding a Balance

The workflow became reliable when I stopped treating the ChatGPT/Codex combination as a tool and started treating it as a junior developer.

Clear goals and context were necessary, but so were boundaries and review. ChatGPT could produce detailed, professional-looking instructions that still contained stale assumptions, excessive scope, or the wrong branch. The speed and polish of the output made complacency easy.

The times I failed to review instructions carefully caused some of the project’s costliest mistakes. Codex once created a new branch before prior work had been reviewed and merged; elsewhere, it followed obsolete assumptions or widened tasks because ChatGPT’s draft told it to. The remedy was simple: treat generated instructions as draft specifications. I removed stale requirements, tightened scope, corrected assumptions, and added prohibitions before Codex saw them.

For larger tasks, I also added read-only reconnaissance. Codex first inspected the repository and reported the relevant files, patterns, data flow, tests, and likely schema or architectural issues without changing anything. This exposed misunderstandings and dependencies while they were still sentences rather than code.

The fuller cycle became:

Small visual corrections did not need every step. Schema, save/load, shared-component, and cross-record changes usually did. The two pre-coding reviews reduced apparent speed but saved time by catching faulty assumptions before implementation.

The roles remained distinct. I controlled product, architecture, domain model, and acceptance. ChatGPT turned those decisions into specifications. Codex modified the repository. Deterministic tools and manual review judged the result. ChatGPT knew the conversation but not always the repository; Codex knew the repository but not always the larger intent; I understood both and was the only participant responsible for usefulness.

The successful workflow was supervised delegation with unusually fast implementation.

The models worked best with the management competent junior developers need: clear work, adequate context, permission to raise problems, and the knowledge that somebody would read what they produced.

4.1. XML Schemas

The XML Schemas were the part of Hermetic Foundry I most expected to rewrite myself. Instead, they became some of the strongest work produced during the experiment.

As of this writing, the integrated schema package had reached version 0.3.10. The project records document numbered revisions from v0.1 through v0.3.10. These were not cosmetic increments. Successive releases added various Ars Magica rules structures with increasing complexity [Wolfe 2026].

The schemas were organized as a modular suite rather than a single large grammar. Shared types handled common entity metadata, provenance, timestamps, identifiers, references, and embedded snapshots. Domain schemas then defined managed records for various classes of documented game aspects.

Several design choices were particularly important:

The schemas also encoded ownership decisions. A character could link to a laboratory or familiar, for example, without absorbing all of that entity’s data. Laboratory facts remained in the laboratory document; character-specific laboratory totals remained with the character. These distinctions became increasingly important as Hermetic Foundry moved from isolated records to a network of related documents.

Hermetic Foundry Reference Example

                     
<hf:references>
  <hf:entityReference relationship="knownSpell">
    <hf:stableId>spell.pilum-of-fire</hf:stableId>
    <hf:canonicalId>
      spell.pilum-of-fire@sha256-3f6b1e1d4a9300c70e0f73adceac00a6a8c29b0cb6e2a2aa97f51ce1f4c59a62
    </hf:canonicalId>
    <hf:entityType>Spell</hf:entityType>
    <hf:displayName>Pilum of Fire</hf:displayName>
    <hf:checksum algorithm="sha256">
      3f6b1e1d4a9300c70e0f73adceac00a6a8c29b0cb6e2a2aa97f51ce1f4c59a62
    </hf:checksum>
    <hf:referenceStatus>snapshotOnly</hf:referenceStatus>

    <hf:source sourceType="xml-file">
      <hf:work>Core spell list</hf:work>
      <hf:title>spells.xml</hf:title>
      <hf:license>Project demo data</hf:license>
    </hf:source>

    <hf:packageProvenance>
      <hf:packageStableId>core.spell-data</hf:packageStableId>
      <hf:packageCanonicalId>
        core.spell-data@sha256-8fb9c3d59db0df3e9ca56f2849ed313d30f7c625f04f5c19bd22081f9020f3c4
      </hf:packageCanonicalId>
      <hf:packageName>Core Spell Data</hf:packageName>
      <hf:packageKind>core</hf:packageKind>
      <hf:packageVersionUnix>1780305720</hf:packageVersionUnix>
      <hf:contentPath>spells.xml</hf:contentPath>
      <hf:importedAt
        unix="1780305720"
        iso="2026-06-01T14:42:00.000Z"/>
    </hf:packageProvenance>

    <hf:embeddedSnapshot>
      <hf:stableId>spell.pilum-of-fire</hf:stableId>
      <hf:canonicalId>
        spell.pilum-of-fire@sha256-3f6b1e1d4a9300c70e0f73adceac00a6a8c29b0cb6e2a2aa97f51ce1f4c59a62
      </hf:canonicalId>
      <hf:entityType>Spell</hf:entityType>
      <hf:displayName>Pilum of Fire</hf:displayName>
      <hf:description>
        A Creo Ignem spell that hurls a spear of fire at a target.
      </hf:description>
      <hf:checksum algorithm="sha256">
        3f6b1e1d4a9300c70e0f73adceac00a6a8c29b0cb6e2a2aa97f51ce1f4c59a62
      </hf:checksum>
      <hf:snapshotCreatedAt>
        2026-06-01T14:42:00.000Z
      </hf:snapshotCreatedAt>
      <hf:snapshotFormatVersion>0.1</hf:snapshotFormatVersion>
    </hf:embeddedSnapshot>
  </hf:entityReference>
</hf:references>

The most surprising fact was that none of the delivered schema versions were invalid. Every version I received validated with Xerces. I found modelling decisions that needed refinement, and new application features regularly exposed structures that needed expansion, but I never had to repair malformed XSD or explain basic schema syntax to either ChatGPT or Codex.

Validity alone is a low bar. An XSD can validate perfectly while expressing a poor vocabulary. I reviewed naming, type reuse, cardinality, ownership, extensibility, and whether the structures reflected Ars Magica rather than the current user interface. The schemas changed because the model evolved, not because the LLMs were unable to produce legal XML Schema.

This result is consistent with emerging work on LLM-generated schemas. Zhang et al. found that LLM-based pipelines could generate high-quality ShEx schemas for large knowledge graphs when supplied with appropriate local and global context, although evaluation remained necessary to distinguish syntactic success from useful modelling [Zhang et al. 2025]. My experience involved XSD rather than ShEx and a smaller but continuously evolving domain, yet the same general lesson applied: the models were unexpectedly capable of structured schema generation, provided that domain expertise and deterministic validation remained outside the model.

I was surprised that ChatGPT and, later, Codex could revise a connected schema suite repeatedly while preserving its overall design, with zero validation issues.

4.2. XML Packages

The schemas defined what Hermetic Foundry could represent. The next task was filling them with the openly licensed Ars Magica corpus.

Hermetic Foundry distributes rules content through ZIP-based XML packages. A package contains a manifest, a licence document, content files, and optional supporting resources. The manifest records the package identity, version, checksums, dependencies, and included files. During installation, the Foundry validates the archive and its XML, checks identifiers and checksums, records provenance, and builds an index of the entities the package provides [Wolfe 2026].

This architecture allowed the core rules to use the same mechanism intended for future third-party and user-created material. Virtues, Flaws, Abilities, Spells, Weapons, Armor, Creatures, Books, Laboratory Features, and other selectable objects were not hard-coded into the application. They were imported content marked-up as first class objects.

Initially, I did not know that Atlas Games had published a Markdown version of the open-licence corpus. I asked ChatGPT to extract material from the PDFs.

This worked better than it had any right to, but PDFs were a poor source format. Page headers, footers, columns, tables, page breaks, and changes in typography carried structural information that did not always survive text extraction. ChatGPT sometimes had difficulty determining where one entry ended and another began. Human intervention was required to compare the output with the source and correct boundaries that had been inferred incorrectly.

The big surprise was that ChatGPT also found errors I had not asked it to find. While extracting the rules, ChatGPT identified spelling mistakes, grammatical problems, and references that appeared to point to the wrong section or game concept. These observations still required confirmation against the source, but they demonstrated that the model was doing more than copying strings into elements. It was interpreting the text closely enough to notice internal inconsistencies.

Once I discovered the Markdown corpus, extraction became much easier. Headings, paragraphs, lists, and tables were represented explicitly rather than implied by page layout. Markdown was not as semantically rich as the XML I wanted to produce, but it preserved enough structure to make entry boundaries and document hierarchy less ambiguous.

The most important failure was semantic rather than syntactic. ChatGPT sometimes promoted examples into first-class entities. A worked example or illustrative game object could resemble a formal spell, item, creature, or rule entry closely enough that the model packaged it as selectable content. The resulting XML was well-formed and schema-valid. It was also wrong.

This distinction is central to evaluating AI-generated structured content. Validation can establish that a document follows its grammar; it cannot establish that every valid record deserves to exist. Gross similarly notes that XML collections may parse and validate while still containing incorrect identifiers, references, metadata, or structural interpretations that reduce their usefulness [Gross 2024]. Waldt’s experiments with automatic tagging reach the same practical conclusion: AI can produce surprisingly good markup, but the result must still be checked for lost or incorrectly classified content [Waldt 2025].

I responded by narrowing the extraction scope. The package should contain reusable game entities and data required by character sheets and lifecycle workflows, not every named thing in the books. Examples, explanatory prose, rules procedures, and gamemaster advice remained source text unless the application had a concrete reason to model them. That boundary was eventually formalized in the development plan [Wolfe 2026].

The result was not an automatic conversion pipeline that could be trusted without review. It was a productive extraction process in which the LLM performed much of the identification, transcription, and initial structuring, while schemas and human review caught different classes of error.

The XML was usually valid. Determining whether it was true to the spirit of the text remained my job. I have found no useful way to automate that understanding from the source material. Context, it seems, remains outside the ability of LLMs.

4.3. XML Character Sheets

The schemas and core package came together in the character files produced by Hermetic Foundry. The current version plus documentation can be found in a zip file here: https://hf.wolfshafenpress.com/xml-schemas/hf-schemas-docs.zip

A saved character was not simply a dump of values from the interface. The XML distinguished between data owned by the character and data referenced from elsewhere. A character’s Ability score, specialty, and experience belonged to the character document; the definition of that Ability belonged to the rules package. The same distinction applied to Spells, Equipment, Virtues, Flaws, and other package-backed content.

References included provenance and an embedded snapshot of the selected material. If a character knew a spell from the core package, the file recorded the source reference and enough descriptive information to remain understandable if the package was unavailable. This allowed the application to preserve both identity and portability: it could recognize the original package entity when present without reducing the saved character to a collection of opaque identifiers when it was not [Wolfe 2026].

The character files also linked to other first-class documents. A magus could reference a separate laboratory or magical item record without copying all of that entity’s data into the character. Objective laboratory information remained in the laboratory XML, while character-specific lab totals and notes remained in the character XML. The same ownership principle prevented changes made through one record from silently rewriting another.

Hermetic Foundry Character Sheet Example

                     
<hf:characterData xmlns:hf="http://hf.wolfshafenpress.com/xml-schemas">
  <hf:magicProfile>
    <hf:spellsKnown>
      <hf:spellKnown>

        <hf:spell
          stableId="spell.pilum-of-fire"
          canonicalId="spell.pilum-of-fire@sha256-3f6b1e1d4a9300c70e0f73adceac00a6a8c29b0cb6e2a2aa97f51ce1f4c59a62"
          entityType="Spell">

          <hf:displayName>Pilum of Fire</hf:displayName>

          <hf:checksum algorithm="sha256">
            3f6b1e1d4a9300c70e0f73adceac00a6a8c29b0cb6e2a2aa97f51ce1f4c59a62
          </hf:checksum>

          <hf:referenceStatus>resolved</hf:referenceStatus>

          <hf:packageProvenance>
            <hf:packageStableId>core.spell-data</hf:packageStableId>
            <hf:packageCanonicalId>
              core.spell-data@sha256-8fb9c3d59db0df3e9ca56f2849ed313d30f7c625f04f5c19bd22081f9020f3c4
            </hf:packageCanonicalId>
            <hf:packageName>
              Ars Magica Definitive Edition Core Rules
            </hf:packageName>
            <hf:packageKind>core</hf:packageKind>
            <hf:packageVersionUnix>1780305720</hf:packageVersionUnix>
            <hf:contentPath>entities/spells.xml</hf:contentPath>
            <hf:importedAt
              unix="1780305720"
              iso="2026-06-01T14:42:00.000Z"/>
          </hf:packageProvenance>

          <hf:embeddedSnapshot>
            <hf:stableId>spell.pilum-of-fire</hf:stableId>
            <hf:canonicalId>
              spell.pilum-of-fire@sha256-3f6b1e1d4a9300c70e0f73adceac00a6a8c29b0cb6e2a2aa97f51ce1f4c59a62
            </hf:canonicalId>
            <hf:entityType>Spell</hf:entityType>
            <hf:displayName>Pilum of Fire</hf:displayName>

            <hf:description>
              A Creo Ignem spell that hurls a spear of fire at a target.
            </hf:description>

            <hf:packageProvenance>
              <hf:packageStableId>core.spell-data</hf:packageStableId>
              <hf:packageCanonicalId>
                core.spell-data@sha256-8fb9c3d59db0df3e9ca56f2849ed313d30f7c625f04f5c19bd22081f9020f3c4
              </hf:packageCanonicalId>
              <hf:packageName>
                Ars Magica Definitive Edition Core Rules
              </hf:packageName>
              <hf:packageKind>core</hf:packageKind>
              <hf:packageVersionUnix>1780305720</hf:packageVersionUnix>
              <hf:contentPath>entities/spells.xml</hf:contentPath>
              <hf:importedAt
                unix="1780305720"
                iso="2026-06-01T14:42:00.000Z"/>
            </hf:packageProvenance>

            <hf:checksum algorithm="sha256">
              3f6b1e1d4a9300c70e0f73adceac00a6a8c29b0cb6e2a2aa97f51ce1f4c59a62
            </hf:checksum>

            <hf:snapshotCreatedAt>
              2026-06-01T14:42:00.000Z
            </hf:snapshotCreatedAt>

            <hf:snapshotFormatVersion>0.1</hf:snapshotFormatVersion>

            <hf:snapshotData>
              <hf:dataField key="technique">Creo</hf:dataField>
              <hf:dataField key="form">Ignem</hf:dataField>
              <hf:dataField key="level" valueType="number">20</hf:dataField>
              <hf:dataField key="range">Voice</hf:dataField>
              <hf:dataField key="duration">Momentary</hf:dataField>
              <hf:dataField key="target">Individual</hf:dataField>
            </hf:snapshotData>
          </hf:embeddedSnapshot>
        </hf:spell>

        <!-- Character-sheet projection used for display and calculations. -->
        <hf:displayName>Pilum of Fire</hf:displayName>
        <hf:technique>Creo</hf:technique>
        <hf:form>Ignem</hf:form>
        <hf:level>20</hf:level>
        <hf:range>Voice</hf:range>
        <hf:duration>Momentary</hf:duration>
        <hf:target>Individual</hf:target>

      </hf:spellKnown>
    </hf:spellsKnown>
  </hf:magicProfile>
</hf:characterData>

Once again, the surprising part was how little XML repair was required. The generated character documents were well-formed, schema-valid, and able to survive save-and-reload testing without losing the structures under examination. Problems occurred in application behaviour and data mapping, but not because ChatGPT or Codex could not generate proper XML.

The files nevertheless became more complex than I had anticipated. A mature character could contain art, rich-text notes, advancement history, embedded snapshots, and references to Laboratories, Familiars, Spells, Equipment, and other records. A single XML document could preserve much of this, but distributing or archiving the complete character eventually required more than one file.

This led to the same solution used for rules content: a ZIP-based package with an XML manifest, based off a similar concept used in EPUB [W3C 2026]. The manifest could identify the principal character document and enumerate associated media, linked records, checksums, and other resources required for a portable exchange package. The XML remained the canonical record; the archive supplied the container.

The need for that container was not evidence that XML had failed. It showed that a useful XML document rarely exists alone: once the character sheet became part of a larger document system, packaging, provenance, and dependency management became part of the design as well.

4.4. Codex Takes Over

For roughly the first half of the project, I kept the XML work separate from the application repository. ChatGPT generated or revised the schemas and core package, I reviewed and validated them, and Codex implemented the application against the resulting files.

That division ended accidentally.

During one implementation task, Codex inspected the repository and reported that the requested behaviour could not be represented properly by the existing schema. It recommended changing the schema and updating the affected package content before modifying the application.

Until then, I had treated Codex as an implementer working within an XML architecture produced elsewhere. Allowing it to change that architecture seemed considerably riskier. A schema revision could affect serialization, validation, package ingestion, test fixtures, and previously saved records. It was also the sort of intervention I had expected to perform myself once the project became complicated.

I let Codex proceed.

It updated the schema, adjusted the related package structures, modified the application code, and revised the tests. The change preserved the existing design while adding the required capability. The resulting schemas and XML remained valid, and the feature worked correctly.

This was important for two reasons. First, Codex had recognized that the problem could not be solved cleanly in the interface alone. Rather than forcing the new behaviour into an inadequate model, it identified the architectural dependency and pushed back on the task. That was the behaviour I would have wanted from a human developer.

Second, Codex had one advantage over ChatGPT: it could inspect the current repository directly. It could see the schemas, package files, TypeScript models, serializers, tests, and application components together. This made it better placed to perform coordinated changes across them.

After that point, schema and package modifications increasingly became part of ordinary Codex work. I still reviewed the modelling decisions and validated the output independently, but I no longer assumed that XML changes had to be produced outside the repository and handed to the implementation agent.

This created a new context problem. The ChatGPT sessions I used to prepare instructions did not automatically know that Codex had revised the repository. If a schema or package changed, I had to provide the new files, summarize the change, or carry the relevant information into the next conversation. Otherwise, ChatGPT might prepare instructions based on an obsolete version of the architecture.

As the project grew, maintaining that shared context became a substantial part of the workflow. I kept handoff notes, development-plan revisions, current file versions, and lists of accepted decisions so that a new conversation could be brought up to date. Codex had repository context but limited awareness of the long-term conversation. ChatGPT had the design conversation but not an automatically synchronized view of the repository.

The transition nevertheless marked an important change in the experiment: Codex was no longer merely implementing an architecture I had specified elsewhere. It was participating in the evolution of that architecture and doing it well.

4.5. The Human Comparison

By this point, I had strong impressions about Codex’s productivity, but no direct comparison with my own work on the same part of Hermetic Foundry. I took on one substantial task myself: refactoring the Quick Creature functionality, where a user with significant rules knowledge of the game could bypass the wizard-style guided creation and just fill out a sheet. This guided creation is a centerpiece of the application, but so complex that i decided to tackle the Quick Sheets first, knowing they would inform the look, feel, and functionality of the guided creation process.

In programming languages I know well, I typically complete about one significant feature in a working day and produce roughly 400 lines of finished code. This varies widely by task, and line count is a poor measure of software quality, but it provided a familiar baseline.

I worked on Quick Creature for 7 hours and 32 minutes. In that time, I wrote 387 lines and produced a thin mock-up of one screen. It established part of the layout, but not the complete interface, data handling, XML integration, or save-and-reload behaviour.

Figure 8: My Quick Creature Implementation After 7 hours and 32 minutes

Image description

A Hermetic Foundry screenshot showing Ben’s limited Quick Creature implementation. The main workspace contains selectors for Quick Character, Quick Creature, Quick Covenant, Quick Laboratory, and Quick Item. The creature form has only four tabs — Identity, Stats, Powers, and Notes — and a small set of basic fields for name, might, type, size, realm, description, and notes. The surrounding application navigation and information panes are present.

I then gave Codex the same overall refactoring task. The comparison was not a controlled experiment: Codex could inspect the existing repository, and the work required a mixture of frontend, backend, XML, and test changes. It nevertheless provided the most concrete comparison available within the project.

Codex’s strict bounded working time was 9 hours, 41 minutes, and 55 seconds. The longer raw session span included large periods when it was idle or waiting for review, so I excluded those gaps. During that bounded runtime, Codex rebuilt the full Quick Creature functionality across the frontend and backend at an average rate of approximately 1,174 lines of code per hour.

My active contribution was 41 minutes. That included preparing instructions, reviewing reports, testing the result, and requesting corrections.

Figure 9: The Completed Codex Quick Creature Implementation

Image description

A polished Hermetic Foundry Quick Creature screen produced by Codex. A metadata strip and ten task-specific tabs appear above a structured Identity pane. The form includes creature name, type, species, realm, might score, might form, size, and source, with consistent two-column alignment. The established navigation, information, recent-changes, validation, and unsaved-state areas remain integrated around the form.

The raw line counts do not establish that Codex was twenty-two times as good because it produced roughly twenty-two times as many lines per hour. More code can be duplication, unnecessary complexity, or an automated path to a larger maintenance problem. The meaningful difference was scope. My working day produced part of one screen. Codex’s work produced the functional record type, including its interface, application logic, XML handling, and tests.

Codex’s working time did not require my continuous attention. While it was operating, I did two loads of laundry, walked the dog twice, made two lunches for my children, cooked dinner for my family, and read an entire novel. I monitored progress and responded through the OpenAI phone application, returning to the computer only when closer inspection was necessary.

This separation between machine runtime and human attention was as important as the raw speed. Conventional programming occupies nearly all of the programmer’s working time. Agent-assisted programming allowed implementation to continue while I performed unrelated work, provided that the task was sufficiently well specified and the agent did not require intervention.

The comparison still had limits. I knew that my own code would be easier for me to understand immediately, and I had not attempted to optimize either approach for a formal benchmark. Codex also benefited from the architecture, patterns, schemas, and earlier components already present in the repository. It did not create Quick Creature from nothing, but then again, neither would I have done so.

The result was sufficient for the question I had actually asked at the beginning of the experiment: was this faster than doing the work myself? For this task, the answer was that my speed was not even remotely close to the LLMs.

5.1. Software Development

My software development experience shaped nearly every part of the experiment.

I have been programming since childhood, mostly in data-heavy applications. That background let me choose an architecture suited to Hermetic Foundry rather than merely accepting the first stack ChatGPT suggested. Tauri, SvelteKit, TypeScript, Rust, XML, and a package-based content system formed a coherent application architecture because I could evaluate how each component would serve the product. While I was unfamiliar with aspects of the stack, they were all built on foundations with which I was intimate.

The same experience allowed me to review Codex’s work intelligently. I could inspect the code, follow data through the application, recognize duplicated logic, and determine whether a problem was local or architectural. When Codex reported that a feature required a schema change, I could judge whether that conclusion was reasonable. When it added a field in the interface but failed to preserve it through save and reload, I knew where to look.

This distinction matters because a generated implementation can appear successful while remaining technically weak. A button may work in the current screen while bypassing the shared state model. A saved document may reload correctly in one test while discarding unknown content. A new component may solve the immediate problem while duplicating an existing pattern and making future maintenance harder. Evaluating these outcomes requires more than observing that the application runs.

My experience also helped me choose work that suited the agent. Codex was particularly effective at tasks that could be defined through existing patterns: adding a new pane, extending a record type, wiring a field through the data model, updating tests, or applying a known interaction pattern elsewhere in the application. It was less reliable when the task depended on an unstated design principle or required deciding what information a user should see.

The literature on agent-assisted development similarly treats human expertise as a major part of the surrounding system. Planning, context selection, validation, and review substantially affect the quality of the generated result [Ge et al. 2025]. The model supplies implementation capacity, but the human still determines whether the implementation belongs in the product.

This was especially important because ChatGPT was consistently and cheerfully encouraging. It would readily endorse an idea, elaborate on it, and produce instructions for implementing it. That behaviour made it useful as a brainstorming partner, but not as an independent source of judgement. Agreeable support is not the same as good architecture. Good development partnering requires an occasionally adversarial nature: engineers argue to refine.

As an example of how agreeability was problematic: I initially continued building new screens without a formal user interface style guide. ChatGPT helped specify each screen enthusiastically, but did not challenge the absence of an application-wide design system until I explicitly asked why the interface had become inconsistent. A more skeptical review earlier in the process would have saved substantial rework.

My role was not merely to approve or reject generated code: it was to question the premise of the task itself. Did the feature belong in this milestone? Was the architecture ready for it? Was Codex extending an established pattern or inventing a new one unnecessarily? Was ChatGPT solving the problem I had described, or expanding it into a different problem because expansion sounded helpful?

The tools made implementation dramatically faster. They did not relieve me of deciding what should be implemented, how it should fit the system, or when a plausible suggestion was a bad idea.

5.2. XML

My XML experience was even more important than my programming experience because XML sat at the centre of the application’s architecture.

I could describe the vocabulary Hermetic Foundry needed before asking ChatGPT or Codex to build it. Characters, Spells, Laboratories, and other first class data models all had different ownership rules, relationships, and validation requirements. That let me give the LLMs precise instructions about document structure instead of asking them to "make some XML" and hoping they inferred the domain correctly.

It also allowed me to evaluate the results. Schema validity was necessary, but not sufficient. I could judge whether types were reusable, whether cardinalities made sense, whether package content was being confused with character-instance data, and whether references preserved provenance without making documents dependent on an installed package. When the generated XML was strong, I knew why it was strong. When it needed revision, I could usually identify whether the problem lay in the schema, the instance document, or the application’s interpretation of either.

That distinction mattered throughout development. Many apparent interface problems were actually modelling problems. A field that could not be saved cleanly often indicated that the schema lacked the right structure. Conversely, some requests did not justify a schema change at all and belonged in the application layer. XML expertise helped me decide where the boundary should be.

It also let me recognize when the existing schemas needed to evolve. New features exposed new requirements: richer item categories, linking laboratories to their wizard character sheets, embedded snapshots, and parameter-aware rules all required controlled schema revisions. I could review those changes without treating every new requirement as an excuse to redesign the vocabulary.

Finally, I knew when XML alone was not enough.

A character document could represent the character perfectly and still fail as a portable package if it depended on images, linked records, or other resources. That led to the ZIP-and-manifest approach used for both rules packages and future character exchange. XML remained the canonical representation, but the package supplied the container, identity, checksums, dependencies, and associated files.

Without that background, I might have accepted valid XML that was poorly modelled, or rejected a sound design because it looked unfamiliar. The models were capable of producing excellent structured documents, but they still needed someone who could distinguish syntactic correctness from a vocabulary worth keeping.

The XML did not succeed because the LLMs understood the domain unaided. It succeeded because I could tell them what the domain required and recognize when they had actually delivered it.

5.3. Project Management

My project management experience gave the experiment structure.

Hermetic Foundry was too large to develop as a sequence of isolated prompts. It needed a product vision, milestones, dependencies, definitions of done, and a way to preserve decisions across many ChatGPT conversations and Codex sessions. I had to maintain a detailed development plan describing the application’s architecture, terminology, workflows, current status, deferred work, and testing expectations [Wolfe 2026].

The plan helped me see the application as a whole, including areas where I lacked direct expertise, as such plans are supposed to do. A new character sheet feature might affect package content, XML serialization, or other aspects of the application. That did not mean every task should address all the related systems immediately. It meant I needed to understand the dependencies well enough to decide which belonged in the current milestone and which should be deferred.

This was also how I treated ChatGPT like a junior developer rather than a full partner in the process. I gave it bounded problems, relevant context, expected behaviour, and explicit exclusions. I reviewed the resulting Codex instructions before implementation and required Codex to explain its changes and tests afterward. For larger tasks, I added a read-only reconnaissance pass so that architectural conflicts could be identified before code was written.

The development plan became especially important because neither LLM system had complete continuity. Codex knew the repository it was currently inspecting but not the full history of product decisions. ChatGPT knew the conversation but could be working from outdated assumptions about the repository. Written specifications, milestone notes, and handoff summaries kept those two partial views aligned.

Project management experience also helped me judge the real effect on productivity. Generated code volume was interesting, but it was not the main measure. I tracked how much of my own time was spent specifying, reviewing, testing, and correcting the work. A feature was only faster if the total supervision cost remained lower than implementing it myself.

This perspective prevented me from confusing machine activity with progress. Codex could work for hours and produce many changes, but the useful output was the portion that met the milestone, passed validation, and survived review. Conversely, a short task that clarified an architectural dependency could be more valuable than a large implementation.

The process therefore depended on familiar management practices: break work into coherent units, define acceptance criteria, preserve decisions, review before merging, and stop scope creep before it became architecture.

The tools did not change the need to manage the project. In fact, they underscored it.

5.4. Lack of Expertise

The clearest evidence that expertise still mattered was what happened in the area where I had the least of it: user interface design.

The initial interface mock-up had been excellent, but a mock-up is not a design system. As Hermetic Foundry grew, each new screen introduced fresh decisions about the user interface. I could recognize obvious ugliness, but I did not always know which underlying design principle had been violated.

Codex therefore solved many interface problems locally. Each screen was usable in isolation, yet the application gradually lost consistency. Similar lists behaved differently. Buttons appeared in different places. Modals used different structures. Some panes displayed too much implementation detail, while others omitted information the user actually needed.

The drift was not caused by Codex ignoring instructions. It was caused by my failing to provide the right instructions. I had specified individual features without defining enough application-wide patterns for them to follow.

By the time the inconsistency became impossible to ignore, much of the interface already existed. I stopped forward development, stepped back several milestones, and created a formal style guide. Then I had Codex refactor the existing user interface to ensure everything followed the style guide.

The guide improved later work immediately. Codex could now reuse established patterns instead of inventing a new solution for each pane. It also made review easier because deviations were no longer merely matters of taste; they could be compared against written standards.

The refactoring process exposed further omissions, so the guide itself evolved as the interface was corrected. That was expensive, but less expensive than allowing the inconsistency to spread through the rest of the application.

The problem has not disappeared. Codex remains much better at implementing a specified interface than deciding what the interface should expose. It often displays data because that data exists in the model, not because a user needs to see it.

This weakness differs from a hallucination. The information Codex presents to the user is real. The mistake is one of relevance and presentation. The agent knows that a value exists, but lacks a reliable sense of whether showing it helps the person using the program.

My lack of interface expertise therefore caused the largest disruptions in the project. ChatGPT and Codex compensated for that weakness well enough to begin the application, but not well enough to eliminate the consequences of my own underspecification.

The lesson was not that LLMs cannot help outside a developer’s area of expertise. They clearly can. The initial interface was far better than anything I would have designed alone. The lesson was that assistance is not the same as supervision. Where I lacked the knowledge to define standards early, problems accumulated until I had to stop, create a specification, come back to prior work, revise the specification, and move forward.

The lesson is clear: without expertise, you really will get "vibe coding" in the worst sense of the term. GIGO continues to be a relevant acronym.

6.1. Instruction Clutter

As Hermetic Foundry grew, ChatGPT’s Codex instructions began carrying too much history.

Constraints from earlier milestones remained in later tasks long after they had stopped being relevant. Old branch names, superseded implementation details, and warnings tied to completed features appeared alongside current requirements. Because the instructions were coherent and professionally phrased, I initially assumed those details still mattered.

Eventually, Codex began taking noticeably longer to complete comparable tasks. I inspected the instructions more carefully, then checked the repository to see whether the older references were actually connected to the requested work. Often, they were not.

The problem was not simply length. Some large tasks genuinely required extensive context. The problem was undifferentiated context: useful architectural constraints mixed with obsolete instructions, historical explanations, and precautions that applied to other parts of the program. Codex had to inspect and reason about all of it before deciding what was relevant.

This created two risks. First, execution slowed as the agent investigated unnecessary files and systems. Second, outdated context could influence the implementation itself. A task might preserve an old workaround, follow a superseded pattern, or touch an unrelated feature because the instructions implied that it remained in scope.

I began editing ChatGPT’s output more aggressively before passing it to Codex. Instructions were limited to the current milestone, the systems directly affected, and the global rules that genuinely applied everywhere. Historical context was included only when it explained an otherwise unclear architectural decision.

I also started explicitly telling ChatGPT to drop superseded instructions rather than assuming it would recognize that they had expired. This became another form of specification maintenance. Requirements did not merely need to be added; they needed to be retired.

Because reducing context created a theoretical risk of omitting an important dependency, I strengthened the review process around each feature. Codex was asked to inspect the relevant code before implementation, and completed work received a separate quality-assurance pass. The narrower instructions reduced unnecessary exploration, while repository inspection and testing supplied safeguards against oversimplification.

The effect was substantial. Comparable tasks completed roughly 30% faster after I began removing stale context. That figure is approximate rather than the result of a controlled benchmark, but the change was consistent enough to alter my standard workflow.

This exposed an important limitation in long-running LLM-assisted development. Context is often treated as an unqualified resource: more context should produce better work. In practice, context behaves more like code or documentation. It accumulates technical debt. Old assumptions remain visible, temporary rules become permanent, and nobody removes them unless doing so becomes an explicit task.

In short, one must keep the instructions current.

6.2. Tool Failures

Some disruptive problems came not from the generated code, but from the LLM itself, as a tool.

Codex regularly had difficulty accessing the Windows sandbox required to run automated checks. Commands that worked in one session might fail in the next because the environment could not see a directory, execute a process, or use a permission that had previously been granted. This weakened one of the main safeguards in the workflow: the agent was supposed to build and test its own changes before returning them for review.

The build process itself was also unreliable. A Tauri build would sometimes fail for no apparent project-related reason and then succeed unchanged on the second attempt. I learned not to treat every failed build as evidence that the implementation was defective, but neither could I safely ignore it. Each failure had to be examined, retried, and distinguished from an actual code problem.

Permission handling created another class of interruption. Codex could forget that I had already authorized a command or process and pause while waiting for the same approval again. Because much of its work ran while I was away from the computer, these pauses were expensive. An agent that might otherwise have completed a task unattended could spend an hour doing nothing because it was waiting for a response I did not know it needed.

LLM uptime was an even more basic weakness. During one phase of development, both ChatGPT and Codex were intermittently unavailable for periods of several hours. My specifications, implementation workflow, and much of the project’s practical momentum depended on services I did not control. When they were unavailable, development stopped.

A further complication is that OpenAI releases new versions of their software rather haphazardly, and with very little documentation. I could sit down at my desk one day, and ChatGPT suddenly changed the way a user specified which model they were using, or Codex suddenly changed what it showed about Git interactions. It was maddening to have to re-teach myself aspects of the tools once or twice a week.

These failures were not evidence that LLM-generated code was poor. They were evidence that the surrounding development environment remains immature and buggy, a clear sign of rushed development by OpenAI. The workflow depended on several layers working together: conversation, repository access, command execution, sandbox permissions, build tools, and remote service availability. A failure in any one of them could interrupt the entire process.

They also complicated productivity comparisons. Codex might require only a small amount of my active attention, but elapsed time could expand considerably when a session stalled, a build needed to be repeated, or access had to be restored. For this reason, I distinguished between active or bounded working time and the full calendar span of a task when comparing Codex’s performance with my own.

The failures did not outweigh the productivity gains, but they changed my understanding of the experiment. I had begun by asking whether the model could write acceptable software. By this stage, the more immediate question was often whether the infrastructure around the model would let it finish the job.

6.3. Refactoring Due to Ignorance

The largest refactoring effort was caused by something neither ChatGPT nor Codex could supply automatically: design knowledge I did not possess.

As noted earlier, Hermetic Foundry’s interface had developed one screen at a time. I described the immediate requirements, ChatGPT expanded them into Codex instructions, and Codex implemented them. Each result was usually functional, but the application as a whole was becoming inconsistent.

I initially treated each inconsistency as a local defect, but they soon became cumulative. Fixing them one at a time did not solve the underlying problem because Codex still lacked a common standard to follow. I eventually asked ChatGPT why the interface kept drifting. Its answer was obvious in retrospect: Hermetic Foundry needed a style guide.

That realization forced me to stop forward development and step back through several completed milestones. I documented the acceptable design patterns. I also defined which information belonged in ordinary user-facing screens and which should remain hidden as implementation detail.

Codex then used that guide to refactor the existing interface. The process uncovered additional cases the first version had not considered, so the guide evolved alongside the corrections. This made the work slower than if I had established the standards at the beginning, but continuing without them would have multiplied the cost.

The failure was mine. ChatGPT had not warned me early enough, and Codex had not inferred a coherent design system from the prototype, but the real issue was that I had never asked either of them to do so. I had supplied detailed requirements at the feature level while leaving the application-wide interaction model largely unstated.

This is one of the clearest limits of compensating for a lack of expertise with an LLM. The tools could produce a better interface than I could design alone, but they did not reliably identify the standards I should have established before development began. They filled the gaps in my instructions with locally plausible decisions, and those decisions diverged when moved out of the immediate locality.

The refactoring succeeded because I eventually recognized the pattern, researched the missing discipline, and converted what I learned into explicit rules. Had I lacked enough software experience to notice that the problem was systemic, Hermetic Foundry might simply have continued accumulating inconsistent screens.

6.4. Ongoing user interface Issues

The style guide reduced inconsistency, but it did not solve Codex’s broader weakness with user interface judgement.

Codex could usually reproduce an established component, layout, or interaction pattern. It was much less reliable at deciding what information a user actually needed, or how it should be displayed in a pleasing fashion. Codex often incorrectly assumed system-level information about things like document validity should be displayed to the user.

This became a recurring part of review. I repeatedly removed information that was technically correct but irrelevant to ordinary users. The problem was not that Codex invented these values. It found real data in the application and exposed it without understanding the difference between information useful to the program, information useful to a developer, and information useful to an Ars Magica player.

The same issue appeared in layout decisions. Codex sometimes gave equal visual weight to fields of very different importance, placed diagnostic summaries beside primary controls, or created interface elements because a property existed rather than because the user needed to manipulate it. These choices rarely violated the style guide. They violated a less easily formalized principle: the interface should reflect the user’s task, not the internal shape of the software.

I gradually added more explicit instructions:

These rules improved later work, but they did not remove the need for close review.

Again, one cannot classify these failures as hallucinations. Codex was not fabricating nonexistent data or claiming that an unsupported feature existed. The information was generally real, and the implementation usually worked. The errors were about relevance, hierarchy, and audience.

That distinction is important. LLM failures in software development are often discussed as if the central risk were invented APIs or impossible code. In Hermetic Foundry, the more persistent problem was perfectly real information placed in front of the wrong person or displayed in a fashion that no human would enjoy.

Codex remained capable of building interfaces much faster than I could. It also remained capable of producing screens that were technically complete, conceptually tone-deaf, and ugly.

While the style guide taught it how Hermetic Foundry should look, I still had to tell it what Hermetic Foundry’s users should see, and each new feature had to be verified that it fit in in the style guide, or be defined there before coding could continue.

7.1. Overall Assessment

The experiment succeeded more completely than I expected.

I was very surprised to discover that working in a pipeline from myself to ChatGPT to Codex and back again did not merely accelerate isolated programming tasks. Together, the LLMs contributed to project planning, technology selection, interface design, XML Schema development, rules-package construction, application code, tests, documentation, and debugging. In several of those areas, particularly user interface design, the result was better than what I would have produced alone, even though it was rife with issues due to my own underspecification.

The productivity difference was unbelievable. The Quick Creature comparison showed Codex completing a broad frontend-and-backend refactoring while requiring less than an hour of my active attention. Similar gains appeared throughout the project. The time moved from implementation to specification, review, and correction, but the total human effort remained markedly lower than conventional development.

The XML results were equally significant. I had expected malformed schemas, inconsistent vocabularies, weak modelling, and repeated manual repair. Instead, the schemas remained valid across multiple revisions, the documents round-tripped successfully, and the architecture supported provenance, package-backed references, embedded snapshots, linked entities, and ZIP-based exchange packages. The XML was not merely syntactically acceptable; much of it was well designed.

That does not mean the tools operated independently. Hermetic Foundry depended on my experience in programming, XML, software architecture, and project management. I defined the product, selected or approved the architecture, reviewed the schema designs, judged the user-facing behaviour, maintained the development plan, and decided when generated work was acceptable.

The tools also failed in ways I had not predicted. ChatGPT preserved obsolete instructions, encouraged scope expansion, and did not always identify missing specifications. Codex reproduced weak patterns, exposed internal data in user-facing screens, and occasionally solved the visible problem rather than the architectural one. Neither system reliably knew which requirements I had forgotten to state.

For these reasons, I do not regard the project as evidence that non-experts can safely replace software development with prompting. The models were most effective when operating inside a system of written specifications, existing patterns, version control, automated tests, XML validation, and expert review. Without those constraints, the same speed could have produced a large amount of superficially convincing but ultimately fragile software.

I supplied intent, domain knowledge, architecture, judgement, and accountability. ChatGPT helped turn those things into instructions. Codex converted the instructions into implementation. Deterministic tools tested whether the result compiled, validated, and behaved as required.

Within that arrangement, the models were far more capable than I had believed. They did essentially everything as well as I could have done, and several things better, while allowing the project to advance much faster than I could have managed alone.

I may not be out of a job, but the job has changed more than I could have predicted. The vistas of productivity that LLMs have opened are vast – one experienced programmer is now capable of creating products that would have formerly required entire teams.

7.2. Evolution of LLMs and XML

The change in LLM capabilities over the previous three years is striking, although the evolution involved development tools as much as the models themselves.

In 2023, Balisage papers were still asking whether general-purpose LLMs could work with XML at all. Ogbuji found that models without specialized XML training could perform a surprising range of markup tasks, while still exhibiting clear limitations [Ogbuji 2023]. Prescod et al. proposed benchmarks for automatic document markup, but deliberately excluded componentization, internal linking, metadata generation, and schema-directed reordering from their initial scope. They considered these whole-document tasks difficult enough to leave for later systems [Prescod et al. 2023].

Beshero-Bondar’s experiments provided an important warning. Her 2023 models repeatedly failed at exact textual comparison, dropping or inventing differences even when given relatively short passages. The problem was not producing angle brackets; it was reliably preserving and expressing distinctions that declarative markup could represent precisely [Beshero-Bondar 2023].

By 2024, DeRose found LLMs useful for analysing existing markup, translating between notations, identifying textual structures, and inserting XML. His principal concerns were stability, inconsistent application of instructions, and the difference between plausible output and accurate output. His conclusion was positive but qualified: LLMs were already useful to the XML community, provided that their work was checked carefully [DeRose 2024].

Waldt’s 2025 experiments moved further into practical publishing work. ChatGPT successfully generated summaries, checked content conventions, performed simple automatic tagging, and produced valid XHTML tables. A larger request to convert an entire Word document into Balisage DocBook, however, failed to complete. His recommendation remained to treat AI as a capable assistant and verify everything it produced [Waldt 2025].

Hermetic Foundry extends this progression from individual markup tasks to sustained development of an XML-dependent system. ChatGPT and Codex generated and repeatedly revised a connected XML Schema suite, transformed source material into reusable package entities, created cross-document references and embedded snapshots, and modified the application code that consumed and produced those documents. Several features that the 2023 Auto-Markup Benchmark excluded – notably componentization, internal linking, metadata, and schema-directed restructuring – became routine parts of the work.

This is not a controlled benchmark of model improvement. The systems, prompts, tasks, and human supervision differed. More importantly, Codex was not merely a chatbot producing XML in a response window. It could inspect a repository, follow existing patterns, edit multiple related files, run validators and tests, and revise its work. The practical advance came from the combination of stronger models, larger usable contexts, agentic tooling, and deterministic feedback.

The experience also suggests that LLMs do not make XML obsolete. They make its constraints more valuable. XML Schemas, validation, explicit references, and documented ownership rules supplied a declarative framework around probabilistic implementation. The LLMs generated possibilities quickly; XML established which structures were permissible and made their output inspectable.

The relationship is now complementary in a fashion that it has not been previously, due to the evolution of LLMs. LLMs reduce the effort required to create and manipulate structured information. XML can supply the precision, semantics, and validation that LLMs still cannot guarantee.

7.3. Lessons Learned

The most important lesson was that I should have designed the collaboration before beginning the implementation.

I discovered the "me to ChatGPT to Codex" workflow accidentally and refined it while the project was already underway. A better approach would have been to describe the proposed application to ChatGPT at the outset and ask it to help identify the specifications, architectural decisions, coding standards, interface conventions, validation requirements, and review procedures that Codex would need. Several later problems, particularly interface drift and instruction clutter, could have been reduced by making those assumptions explicit before the repository became large.

The second lesson was to review instructions as carefully as code. ChatGPT’s Codex tasks often looked authoritative because they were detailed and well organized. They could still contain obsolete assumptions, unnecessary scope, or a misunderstanding of the current branch. A few minutes spent correcting the task before implementation was consistently cheaper than repairing the resulting code afterward.

Pre-coding reconnaissance also proved valuable. Asking Codex to inspect the repository and propose an approach before making changes exposed schema dependencies, ownership conflicts, and misunderstandings while they were still easy to correct. For significant tasks, the most efficient sequence was not "prompt, accept, and test." It was "specify, inspect, revise, implement, and verify."

Context also required maintenance. More context was not always better. Long-running instructions accumulated expired constraints and irrelevant history, slowing Codex and occasionally steering it toward unrelated systems. Useful project memory had to be curated: stable architectural decisions belonged in the development plan or style guide, while task instructions should contain only the context needed for the current work.

The project also confirmed my skepticism about relying on third party services. Reliance on hosted services created avoidable risks. Interfaces changed, tools moved, permissions behaved inconsistently, and service outages could halt development entirely. OpenAI updates ChatGPT and Codex with alarming frequency, often pushing out buggy programs with minimal documentation. As I’ve often said, Agile methodology has a lot to answer for.

For sustained professional use, a capable locally controlled LLM and coding agent would offer greater stability, privacy, and control over versions and configuration. The financial cost would be substantial, but so would be the value of the time saved and the predictability obtained.

Finally, the process became an unexpectedly effective way to learn unfamiliar technologies. Before Hermetic Foundry, I had not used Rust, Tauri, or SvelteKit professionally. ChatGPT explained unfamiliar concepts in the context of a real problem, while Codex supplied working examples inside the application. Reviewing, testing, and correcting that work taught me the stack far more quickly than disconnected tutorials would have.

This does not mean that generated code removes the need to understand the technology. It means that the implementation itself can become instructional material. By the time of this writing, I was able to reason about the architecture, diagnose problems, and review changes in technologies that had been unfamiliar to me when the project began.

The general lesson was simple: LLM-assisted development worked best when treated as software development rather than magic. It required specifications, boundaries, review, tests, current documentation, and someone willing to question a confident presentation and expert enough to evaluate the answer.

7.4. All Magic Has a Price

The gains described in this paper do not make broad LLM adoption an unqualified good. The technology carries costs that are easy to ignore because they occur outside the conversation window.

The most immediate is environmental. AI systems require data centres, electricity, cooling water, specialized hardware, and rapid equipment replacement. The International Energy Agency projected that global data-centre electricity consumption could more than double to approximately 945 terawatt-hours by 2030, with accelerated servers driven largely by AI accounting for much of the increase [International Energy Agency 2025]. Environmental assessments also tend to focus on electricity and carbon while neglecting water use, electronic waste, mineral depletion, and negative impacts on biodiversity [International Telecommunication Union 2025]. The hardware depends on critical minerals and rare elements whose extraction and processing may involve unsustainable practices and environmental injustice [United Nations Environment Programme 2024]. The prompt may feel weightless. The infrastructure is not.

Corporate control presents a different problem. Hosted models operate under guardrails chosen by companies rather than users, and those rules reflect corporate liability concerns, brand protection, and cultural assumptions. In my experience, they often lean towards American puritanism and fear of lawsuits. ChatGPT’s restrictions on direct quotation, for example, are considerably more rigid than American fair-use doctrine itself, which permits limited quotation for purposes including scholarship and commentary and deliberately establishes no fixed word or percentage limit [United States Copyright Office n.d.]. A private company is entitled to impose stricter limits, but the user may encounter them suddenly and without meaningful appeal. One can be poodling along productively and abruptly receive the equivalent of, "I’m sorry, Dave. I’m afraid I can’t do that."

The changing interfaces, shifting product features, service interruptions, and opaque policy decisions also reinforce the case for locally controlled models. My development process depended on systems that could be altered or withdrawn without my consent. Convenience came with dependency.

There are more direct software risks. Vibe coding allows people with little development experience to produce applications that appear functional without understanding security, data ownership, deployment, or maintenance. The SusVibes benchmark found that, in one tested configuration, 61% of agent-produced solutions were functionally correct but only 10.5% were secure [Zhao et al. 2025]. This is the software equivalent of a house that looks finished because the wallpaper is up, while the wiring quietly smoulders behind it.

Making software easier to generate does not make it easier to evaluate. The same mechanisms already producing floods of disposable images and disinformation prose will produce floods of specious programs: convincing on the surface, poorly understood underneath, and potentially handling other people’s private information or mission critical applications incorrectly.

Even expert partnership is dangerous when the stakes become high and oversight becomes perfunctory. The February 2026 strike on a girls’ elementary school in Minab, Iran, is an important caution, although it would be irresponsible to reduce it to "an LLM bombed a school." Reporting at the paper’s cutoff indicated that the school had appeared on a U.S. target list and raised questions about AI-assisted target identification; a questionable military investigation instead pointed to obsolete intelligence coordinates supplied by the Defense Intelligence Agency [Copp et al. 2026; Madhani et al. 2026]. The precise causal role of AI remained unresolved. The broader danger was clear: automated systems can accelerate decisions based on stale or incorrect data while reducing the time and institutional capacity available for human challenge. At that scale, "the output looked plausible" becomes a lethal standard.

LLMs also lower the cost of producing misinformation. They can generate persuasive falsehoods, synthetic evidence, and tailored variations faster than human fact-checkers can examine them. UNESCO has warned that generative systems can both intentionally spread disinformation and unintentionally invent false historical claims [UNESCO 2024]. Automated accounts can then amplify such material while appearing to be human participants, distorting the apparent popularity of ideas during elections and crises [OECD 2024].

More surreal is the willingness of actual humans to act as intermediaries. People now ask an LLM to formulate an argument, paste that argument into social media, and wait while another person uses another LLM to answer it. Humans are voluntarily serving as meat puppets for glorified pattern-matching systems.

A smaller but personally significant cost was laziness. The tools were so effective that I became increasingly tempted to delegate tasks I could perform faster or better myself. Delegation was the point of the experiment, but convenience can erode judgement. The question gradually shifts from "Would the model improve this work?" to "Why should I bother doing this?" Those are not equivalent.

LLMs can extend expertise, accelerate development, and make ambitious projects possible. They can also concentrate power, consume resources, magnify bad information, conceal weak work beneath polished output, and encourage users to stop exercising abilities they still need.

All magic has a price, and I have come to believe we should approach the magic of LLMs with cautious optimism and a keen eye on corporate and governmental accountability.