rate and prefer: understanding, readability, friendliness, page load

Readability is counted (LIX), not judged. Understanding is three
yes-or-no questions per persona, with a gutted twin of every page as
the control: an instrument that still says yes to "Option 1, Option 2"
is not measuring the page. Friendliness is a preference between two
versions shown in both orders; a pick that flips with the order counts
as no preference. Page load (options offered, required fields in the
step) flags a page that should be split into a further question or
have fields moved to a later stage.

First run on the generated contractor site, qwen3:8b as the visitor:
control passed on every need; "what happens next" was 0/2 everywhere
and 2/2 everywhere once the page said who reads it and when (LIX 28 to
30); an official-voice control reads LIX 63 against 27. Friendliness
mostly came back "no preference" at two personas per need: directionally
right (never preferred the cold version), too thin to rely on.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4jrCgLiKKHAEFuZUJjckH
This commit is contained in:
Bendik Aagaard Lynghaug
2026-09-21 19:48:25 +02:00
co-authored by Claude Fable 5.1
parent 612745bc66
commit b9f2acb7b8
7 changed files with 1851 additions and 1 deletions
+1 -1
View File
@@ -327,7 +327,7 @@ fn resolve_choice(names: &[&str], raw: &str) -> String {
/// One JSON-answering call. The backend is asked for JSON where it can
/// be (`FORMAT=json`, honoured by engines/ollama.sh) and the first
/// object in the reply is taken either way.
fn ask(cmd: &str, prompt: &str, seed: &str) -> anyhow::Result<Value> {
pub(crate) fn ask(cmd: &str, prompt: &str, seed: &str) -> anyhow::Result<Value> {
let reply = llm_with(cmd, prompt, &[("FORMAT", "json"), ("SEED", seed), ("TEMPERATURE", "0.7")])?;
Ok(first_object(&reply).unwrap_or(Value::Null))
}