Readability is counted (LIX), not judged. Understanding is three
yes-or-no questions per persona, with a gutted twin of every page as
the control: an instrument that still says yes to "Option 1, Option 2"
is not measuring the page. Friendliness is a preference between two
versions shown in both orders; a pick that flips with the order counts
as no preference. Page load (options offered, required fields in the
step) flags a page that should be split into a further question or
have fields moved to a later stage.
First run on the generated contractor site, qwen3:8b as the visitor:
control passed on every need; "what happens next" was 0/2 everywhere
and 2/2 everywhere once the page said who reads it and when (LIX 28 to
30); an official-voice control reads LIX 63 against 27. Friendliness
mostly came back "no preference" at two personas per need: directionally
right (never preferred the cold version), too thin to rely on.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4jrCgLiKKHAEFuZUJjckH
`portal-trainer simulate` reads the model `question_lint --needs-sim`
exports. A visitor model picks an alternative and fills the form in a
manner that varies by run; the case is then walked along the handling
group's own desk buttons to a finished state, a different one each
run, so every state and button is exercised. At each button the
handler model is asked one thing: with only what the form collected,
could this role really do what the button claims? Its question goes
back to the visitor, who may not know.
Who wins is deliberately not simulated. The first version let the
handler decide: qwen3:8b sent every case to "won", and after one
sentence of prompt change, every case to "lost". Outcomes belong to
the live bucket; `success` in needs.yaml is what a report over real
records counts.
`compare` sets two score files side by side. engines/ollama.sh honours
SEED, FORMAT=json and TEMPERATURE so the same case replays the same
way against two versions of a site. generator/reference.md documents
`success` and `stages`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4jrCgLiKKHAEFuZUJjckH
