Bendik Aagaard LynghaugandClaude Fable 5.1
612745bc66
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4jrCgLiKKHAEFuZUJjckH
45 lines
2.2 KiB
Markdown
45 lines
2.2 KiB
Markdown
# Simulating the generated contractor site, 2026-09-21
|
|
|
|
Ten personas, three runs each, on ergo. Version A is the site as
|
|
generated (`enquiries: open -> quoted`). Version B adds the stage the
|
|
owner described and the generator dropped (`open -> surveyed -> quoted`).
|
|
|
|
## What held up
|
|
|
|
- **The simulation found a defect the verifier could not.** Reading
|
|
the first run's records next to the intake showed the generated
|
|
machine had skipped "we go and look at the building". That became
|
|
the `stages` field in needs.yaml, which the lint now enforces, so
|
|
the same omission fails deterministically from here on.
|
|
- **Routing and form-filling are stable.** 0 misroutes and 0 abandoned
|
|
forms in 60 cases, once choices were matched by heading (the first
|
|
run's "100% misrouted" was the model answering "Heading: description").
|
|
- **One recurring question is a real gap.** Handlers asked about
|
|
arranging a site visit seven times. The enquiry forms collect an
|
|
email address and nothing about access or when someone can come,
|
|
for a business whose first step is a visit.
|
|
|
|
## What did not
|
|
|
|
- **Letting a model decide outcomes is worthless.** In the first
|
|
design qwen3:8b sent every case to its success state; after one
|
|
sentence telling it that losing is normal, every enquiry was "lost"
|
|
in one touch (`first-design-qwen3-8b/`). Routes are now scripted and
|
|
who wins is left to the live bucket.
|
|
- **The ask-back rate does not discriminate yet.** With
|
|
nemotron-3.5-lightning:30b-a3b as handler, nearly every button drew
|
|
a question (4 of 4), including nonsense (a title deed; an email
|
|
address the form already requires). A measure stuck at its maximum
|
|
cannot rank version A against version B: the A -> B rise in
|
|
ask-backs per case (1.7 -> 2.3) is just B having one more button.
|
|
|
|
## What it needs before its numbers mean anything
|
|
|
|
A control for the instrument: run the handler on a record that is
|
|
deliberately complete and on one deliberately stripped. A usable judge
|
|
asks on the stripped record and stays quiet on the complete one. Until
|
|
a prompt and model pass that, read the questions, not the rates.
|
|
|
|
Speed: nemotron on both sides took about nine minutes per 30 cases;
|
|
qwen3:8b about two.
|