rate: a desk row offers only the buttons that start from its state

The first desk score offered every button of the bucket, where portal
shows a row only the moves from its own state; that scored the live
vel 17 of 26 and sent a rewrite of the neighbour-help desk chasing an
instrument fault. With the row's buttons only: 24 of 26, and the
rewrite loses on the fair test too. Noted: an accent-stripped state id
(ikke_lost) reads as English even to a strong picker.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Bendik Aagaard Lynghaug
2026-09-24 04:39:13 +02:00
co-authored by Claude Fable 5.1
parent 9089503524
commit 8e6a93ebb7
3 changed files with 36 additions and 11 deletions
+10 -7
View File
@@ -328,13 +328,16 @@ its `asked` and `next` columns are the old, pinned ones.
which reads button labels and nothing else: for every desk and bucket
with two or more buttons, a worker is told what has just happened to
a case (in the state's own words, never the label's) and picks the
button among that desk's, in a rotated order. Scored per desk as
right of total, with what each miss was mistaken for. Measured on
2026-09-24: the vel's neighbour-help desk 2 of 4 ("not solved" taken
for "connect a neighbour"), the notice board 1 of 2, the planned
contractor's subcontractor desk 4 of 7 (checking, approved and
rejected all crossed). A desk that scores full marks has labels a
stranger to the business could press right.
button among the ones a row in that state shows, in a rotated order.
Scored per desk as right of total, with what each miss was mistaken
for. Measured on 2026-09-24 on the live vel: 24 of 26, the misses
one button each on the neighbour-help desk and the notice board. Two
limits: the outcome is put from the state id, and a Norwegian id with
its accents stripped (`ikke_lost`) reads as English even to a strong
model; and a first version that offered every button of the bucket,
not the row's, scored the same desks 17 of 26 and sent a rewrite
chasing an instrument fault. A desk that scores full marks has labels
a stranger to the business could press right.
## Measured so far
**Generated: a building contractor** (`runs/contractor-2026-09-21/`),
@@ -47,3 +47,14 @@ reads as volunteering. The live doors stay. What survives every round
is the same four: hearing input read as a report, a committee read as
a hearing, and volunteering read as neighbour help - which says the
doors' seams are the plan's, not the words'.
## Round 4: the neighbour-help desk reworded, and rejected
The first desk score said this desk got 2 of 4 buttons right. Sonnet
reworded its labels (`/tmp` only); the score got worse with both a
small and a strong picker, and the strong one showed why: told a case
"has just become ikke lost", it pressed "Meld ordnet" - it read the
accent-stripped state id as English. The scorer also offered every
button of the bucket where a row shows only its own state's. With the
row's buttons only, the live desk scores 3 of 4 and the vel 24 of 26
over all desks. The labels stay as they were.
+15 -4
View File
@@ -307,19 +307,30 @@ pub fn rate(argv: &[String]) -> anyhow::Result<()> {
if !desks.is_empty() {
println!("{:<20} {:<24} {:>8} {}", "desk", "bucket", "buttons", "mistaken for");
for (page, group, bucket, moves) in &desks {
let labels: Vec<&str> = moves.iter().map(|(_, _, l)| l.as_str()).collect();
// A desk row shows only the buttons that start from the
// row's state, so those are the ones the worker chooses
// among; a state with one button has nothing to confuse.
let mut right = 0;
let mut asked = 0;
let mut mistaken = Vec::new();
for (i, (from, to, label)) in moves.iter().enumerate() {
let labels: Vec<&str> = moves.iter().filter(|(f, _, _)| f == from).map(|(_, _, l)| l.as_str()).collect();
if labels.len() < 2 {
continue;
}
asked += 1;
let picked = button_pick(&cmd, group, bucket, from, to, &labels, i)?;
if picked == *label {
right += 1;
} else {
mistaken.push(format!("{to} as {picked:?}"));
mistaken.push(format!("{from}>{to} as {picked:?}"));
}
}
println!("{:<20} {:<24} {:>3} of {:<2} {}", page, bucket, right, moves.len(), mistaken.join("; "));
report.insert(format!("desk {page} {bucket}"), json!({ "right": right, "of": moves.len(), "mistaken": mistaken }));
if asked == 0 {
continue;
}
println!("{:<20} {:<24} {:>3} of {:<2} {}", page, bucket, right, asked, mistaken.join("; "));
report.insert(format!("desk {page} {bucket}"), json!({ "right": right, "of": asked, "mistaken": mistaken }));
}
println!();
}