Bendik Aagaard Lynghaug
bedd306854
git-subtree-dir: vendor/trainer git-subtree-mainline:c7c2956e15git-subtree-split:a2a9f4edcf
2.2 KiB
2.2 KiB
Simulating the generated contractor site, 2026-09-21
Ten personas, three runs each, on ergo. Version A is the site as
generated (enquiries: open -> quoted). Version B adds the stage the
owner described and the generator dropped (open -> surveyed -> quoted).
What held up
- The simulation found a defect the verifier could not. Reading
the first run's records next to the intake showed the generated
machine had skipped "we go and look at the building". That became
the
stagesfield in needs.yaml, which the lint now enforces, so the same omission fails deterministically from here on. - Routing and form-filling are stable. 0 misroutes and 0 abandoned forms in 60 cases, once choices were matched by heading (the first run's "100% misrouted" was the model answering "Heading: description").
- One recurring question is a real gap. Handlers asked about arranging a site visit seven times. The enquiry forms collect an email address and nothing about access or when someone can come, for a business whose first step is a visit.
What did not
- Letting a model decide outcomes is worthless. In the first
design qwen3:8b sent every case to its success state; after one
sentence telling it that losing is normal, every enquiry was "lost"
in one touch (
first-design-qwen3-8b/). Routes are now scripted and who wins is left to the live bucket. - The ask-back rate does not discriminate yet. With nemotron-3.5-lightning:30b-a3b as handler, nearly every button drew a question (4 of 4), including nonsense (a title deed; an email address the form already requires). A measure stuck at its maximum cannot rank version A against version B: the A -> B rise in ask-backs per case (1.7 -> 2.3) is just B having one more button.
What it needs before its numbers mean anything
A control for the instrument: run the handler on a record that is deliberately complete and on one deliberately stripped. A usable judge asks on the stripped record and stays quiet on the complete one. Until a prompt and model pass that, read the questions, not the rates.
Speed: nemotron on both sides took about nine minutes per 30 cases; qwen3:8b about two.