ColorPages · Built to learn
Designing Around the Limits of Generative AI
Generating a drawing was easy. Keeping characters consistent across a book took more. I designed and built ColorPages around selecting and repairing drawings, preserving what worked instead of starting over.
- Role
- Design and build, start to finish
- Scope
- AI authoring workspace · Local image generation · Print production
- Scale
- 11,049 renders → 389 finished pages
The core tension: idealized AI vs. human agency
Generative AI suggests a simple workflow: describe what you want and get the finished result. Building a real product around it exposed a different reality.
I chose local image models for privacy, speed, and fine-tuning control, but they came with three recurring constraints:
- 01The local model bottleneck
- Fine-grained LoRA training and no marginal API cost came with higher variance, character drift, and weak spatial editing.
- 02The “almost-right” trap
- Most failures were small. A page could be 90% right but still fail because of an extra limb, bad hair texture, or stray artifact.
- 03Coupled mutation
- Changing a prompt to fix one detail often regenerated the entire composition, destroying parts that already worked.
3.5%389 of 11,049 renders became finished pages.
Judging candidates, not generating them, became the bottleneck. I designed ColorPages around reducing that attention cost.
Get the character right
Character consistency evolved from words, to references, to trained identity
A book about my nephew fails if the child on page twelve is not recognizably the child on page one.
I did not solve that in one move. Each approach exposed the limitation that led to the next.
Attempt 1 · Prompt-based canon
Describe the character
It helped, but identity still drifted when pose or viewing angle changed.
Read attempt 1Close
Every character started with a written identity canon carried into each prompt. Over time I tightened the rules around which traits belonged to the character versus the illustration style.
The canon says who. The style says how.
It helped, but identity still drifted when pose or viewing angle changed. Negative instructions were unreliable too. Telling the model what not to draw could reinforce the unwanted trait instead.
What I learned: prose was useful reinforcement, but too weak to be the primary source of identity.
Attempt 2 · Pinned reference sheets
Show the character
The reference sheet solved one problem but surfaced another: the character should persist while expression changes scene by scene.
Read attempt 2Close
I moved the stable identity into an approved full-body reference sheet that stayed pinned across scenes.
That worked much better, but introduced a new failure with similar characters. When two toddlers appeared together, traits could bleed between them because the system was treating the cast too generically.
I changed the generation logic to bind traits to each figure individually, and limited that behavior to multi-character pages so it would not override a scene's own instructions.
What changed: identity now had a visual source of truth instead of depending on the prompt to reconstruct it.
Attempt 3 · Expression boards
Separate identity from expression
The reference system worked, but every generation still had to carry and reconcile those images.
Read attempt 3Close
The reference sheet solved one problem but surfaced another: the character should persist while expression changes scene by scene.
Character Studio evolved from a casting portrait, to a full-body sheet, to an expression board built from the locked identity.
I also tried localized face editing. That failed because the mask preserved the very mouth shape I was trying to change. Editing the whole reference proved more reliable.
But even a correct expression on the reference was not enough:
That produced one of the system's lasting rules:
Shape language for identity. Emotion words for expression.
Train the identity
The reference system worked, but every generation still had to carry and reconcile those images.
The next step was putting stable identity into trained weights for Trey and Jasmine. Once that recipe worked, solo pages no longer needed a pinned character sheet.
And importantly, the previous layers did not disappear.
The LoRA recipe still includes the written identity canon. Characters without LoRAs still use pinned sheets. Expression wording still controls emotion. Finished-page repairs still go through the edit model.
Fig. 01Persistent identity, per-scene expression
PersistentThe child, Trey, on every page, drawn by his trained model.
Per sceneThe expression






Per sceneThe pose






Keep what works
Fix only what is actually broken
My first response to a defect was to edit the prompt and generate the page again.
That is still the right move when the prompt is the problem. But often the prompt was already right and the model had simply produced something weird: an extra line, a bad hand, the wrong mouth, or a misplaced object.
Regenerating the whole page meant putting everything that already worked back at risk.
So I built repair tools around the size and type of the defect.
- Erase or ink. Remove an unwanted mark or draw in a missing line without touching the rest of the page.
- Move, cut, copy, and paste. Reposition successful artwork deterministically instead of asking the model to recreate the composition.
- Targeted redraw. Brush over the area that actually needs new drawing, then give the model a new prompt that affects only that region.
- Code-rendered overlays. Handle repeatable elements like lettering outside the image model when precision matters more than generation.
The model still had a role. I just stopped giving it more of the page than the repair required.
43% of finished pages were repaired rather than regenerated: 166 of 389.
Fig. 02Five tools, five real repairs

A second ball the model added. Gone, and nothing else moves.

The paste before this left a white seam through the mouth line. One stroke of black closed it.

I brushed her eyes and asked for open ones. The model redraws only inside the brush; everything outside it comes back as it was.

The mouth, brushed and slid 10 pixels right and 3 down. Real, but subtle at this scale.

I erased a closed-mouth smile and pasted in the open one from a sibling variant.
queued
Encode systemic memory
When a failure repeats, the system should remember it
Some defects were random. Others kept happening.
A fuel gauge made the difference impossible to ignore. The model failed to render it correctly in 45 consecutive attempts. At that point, another prompt tweak was no longer useful experimentation. I had learned something about the model that the system itself did not yet know.
Fig. 03One gauge, asked for six times
The brief asked for“one gauge on Rumble's hood with the needle pointing straight up in the middle”







0
correct gauges in the ~45 Pit Stop Heroes renders I reviewed on Aug 6
Once a failure becomes predictable, the system should stop asking the human to rediscover it.
I started encoding recurring failure modes into validation rules and reusable authoring skills that run before generation.
One recurring example was negative phrasing. Instructions like “no signs” often caused the model to add signs instead. Once that pattern was known, ColorPages could rewrite the brief before spending another render discovering the same failure again.
Fig. 04One brief, caught before it rendered
As I wrote it
…leave the bottom fifth of the page completely empty — no words, no letters, no numbers and no signs anywhere in that band
4 errors
Rewritten
…the whole scene drawn in the top four fifths of the page with the bottom fifth left as one clear open white band
0 errors
To the image model, “no signs” reads as add “signs”.
Prevention introduced its own risk.
One validation rule began rejecting valid basketball scenes because its scope was too broad. I narrowed the rule instead of allowing one known failure mode to constrain unrelated creative work.
The goal was not to eliminate failure.
It was to make sure each failure could teach the system something once.
Turn the method into a workflow
Offloading technical overhead without giving up creative control
As ColorPages accumulated more rules, character logic, and model-specific constraints, planning a book became its own technical job.
I wanted to think in stories, characters, and scenes, not remember which phrasing a model rejected, how many figures a composition could support, or which generation route a character required.
So I built the pack planner around a domain-specific agent harness that applies those skills and rules in the background.
I can start with something rough like “Trey goes to a pit stop” or “add some mythical creatures.” The harness checks that idea against the developing book, known model constraints, and accumulated authoring rules, then turns it into a viable plan.
The first planner proved the method
The first version already worked as a real planning system. It could read the live pack, inspect what was already decided, generate candidate pages, prune weak options, budget the compositions, and stage a proposal for review.
But the planning logic still lived inside a long conversational transcript. The method was working, yet the structure of the decisions was not clear enough.
Fig. 05One session in the first planner

The next step was to give the method a better shape
I evolved that first planner into a gated planning flow: one decision at a time, with the developing book visible throughout.
The gates made the underlying method legible. Instead of burying the logic inside one long session, the interface shows what has already been decided, what question is open now, and what the final Create action will commit.
Fig. 06One planning session, replayed
The harness can stage the plan, but nothing enters production until I press Create.
Decide what not to automate
Testing the boundary between assistance and authority
After building the harness, I tested whether ColorPages should go one step further and predict which generated image I would choose as the final page.
Across 57 scenes, I compared 18 image measurements and seven scoring approaches against selections I had already made.
The best approach improved on random ordering by only 6.1 percentage points. That was not enough evidence to delegate final selection.
The system had earned authority over technical constraints, known failure modes, and planning rules. It had not earned authority over editorial judgment.
So I automated compliance, not taste.
Conclusion
ColorPages started as an experiment with local image-generation models. The harder product problem became designing around what those models could not reliably do.
Character identity became an explicit constraint. Small corrections became deterministic tools. Repeated failures became system memory. That knowledge became an agentic workflow with a deliberate boundary around execution.
Next case study

Home Assistant·Notification systems·IoT
Deciding when a smart home should interrupt.
Read case study