# Simulating the generated contractor site, 2026-09-21

Ten personas, three runs each, on ergo. Version A is the site as
generated (`enquiries: open -> quoted`). Version B adds the stage the
owner described and the generator dropped (`open -> surveyed -> quoted`).

## What held up

- **The simulation found a defect the verifier could not.** Reading
  the first run's records next to the intake showed the generated
  machine had skipped "we go and look at the building". That became
  the `stages` field in needs.yaml, which the lint now enforces, so
  the same omission fails deterministically from here on.
- **Routing and form-filling are stable.** 0 misroutes and 0 abandoned
  forms in 60 cases, once choices were matched by heading (the first
  run's "100% misrouted" was the model answering "Heading: description").
- **One recurring question is a real gap.** Handlers asked about
  arranging a site visit seven times. The enquiry forms collect an
  email address and nothing about access or when someone can come,
  for a business whose first step is a visit.

## What did not

- **Letting a model decide outcomes is worthless.** In the first
  design qwen3:8b sent every case to its success state; after one
  sentence telling it that losing is normal, every enquiry was "lost"
  in one touch (`first-design-qwen3-8b/`). Routes are now scripted and
  who wins is left to the live bucket.
- **The ask-back rate does not discriminate yet.** With
  nemotron-3.5-lightning:30b-a3b as handler, nearly every button drew
  a question (4 of 4), including nonsense (a title deed; an email
  address the form already requires). A measure stuck at its maximum
  cannot rank version A against version B: the A -> B rise in
  ask-backs per case (1.7 -> 2.3) is just B having one more button.

## What it needs before its numbers mean anything

A control for the instrument: run the handler on a record that is
deliberately complete and on one deliberately stripped. A usable judge
asks on the stripped record and stays quiet on the complete one. Until
a prompt and model pass that, read the questions, not the rates.

Speed: nemotron on both sides took about nine minutes per 30 cases;
qwen3:8b about two.