rate: a desk row offers only the buttons that start from its state
The first desk score offered every button of the bucket, where portal shows a row only the moves from its own state; that scored the live vel 17 of 26 and sent a rewrite of the neighbour-help desk chasing an instrument fault. With the row's buttons only: 24 of 26, and the rewrite loses on the fair test too. Noted: an accent-stripped state id (ikke_lost) reads as English even to a strong picker. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
9089503524
commit
8e6a93ebb7
@@ -328,13 +328,16 @@ its `asked` and `next` columns are the old, pinned ones.
|
||||
which reads button labels and nothing else: for every desk and bucket
|
||||
with two or more buttons, a worker is told what has just happened to
|
||||
a case (in the state's own words, never the label's) and picks the
|
||||
button among that desk's, in a rotated order. Scored per desk as
|
||||
right of total, with what each miss was mistaken for. Measured on
|
||||
2026-09-24: the vel's neighbour-help desk 2 of 4 ("not solved" taken
|
||||
for "connect a neighbour"), the notice board 1 of 2, the planned
|
||||
contractor's subcontractor desk 4 of 7 (checking, approved and
|
||||
rejected all crossed). A desk that scores full marks has labels a
|
||||
stranger to the business could press right.
|
||||
button among the ones a row in that state shows, in a rotated order.
|
||||
Scored per desk as right of total, with what each miss was mistaken
|
||||
for. Measured on 2026-09-24 on the live vel: 24 of 26, the misses
|
||||
one button each on the neighbour-help desk and the notice board. Two
|
||||
limits: the outcome is put from the state id, and a Norwegian id with
|
||||
its accents stripped (`ikke_lost`) reads as English even to a strong
|
||||
model; and a first version that offered every button of the bucket,
|
||||
not the row's, scored the same desks 17 of 26 and sent a rewrite
|
||||
chasing an instrument fault. A desk that scores full marks has labels
|
||||
a stranger to the business could press right.
|
||||
## Measured so far
|
||||
|
||||
**Generated: a building contractor** (`runs/contractor-2026-09-21/`),
|
||||
|
||||
@@ -47,3 +47,14 @@ reads as volunteering. The live doors stay. What survives every round
|
||||
is the same four: hearing input read as a report, a committee read as
|
||||
a hearing, and volunteering read as neighbour help - which says the
|
||||
doors' seams are the plan's, not the words'.
|
||||
|
||||
## Round 4: the neighbour-help desk reworded, and rejected
|
||||
|
||||
The first desk score said this desk got 2 of 4 buttons right. Sonnet
|
||||
reworded its labels (`/tmp` only); the score got worse with both a
|
||||
small and a strong picker, and the strong one showed why: told a case
|
||||
"has just become ikke lost", it pressed "Meld ordnet" - it read the
|
||||
accent-stripped state id as English. The scorer also offered every
|
||||
button of the bucket where a row shows only its own state's. With the
|
||||
row's buttons only, the live desk scores 3 of 4 and the vel 24 of 26
|
||||
over all desks. The labels stay as they were.
|
||||
|
||||
+15
-4
@@ -307,19 +307,30 @@ pub fn rate(argv: &[String]) -> anyhow::Result<()> {
|
||||
if !desks.is_empty() {
|
||||
println!("{:<20} {:<24} {:>8} {}", "desk", "bucket", "buttons", "mistaken for");
|
||||
for (page, group, bucket, moves) in &desks {
|
||||
let labels: Vec<&str> = moves.iter().map(|(_, _, l)| l.as_str()).collect();
|
||||
// A desk row shows only the buttons that start from the
|
||||
// row's state, so those are the ones the worker chooses
|
||||
// among; a state with one button has nothing to confuse.
|
||||
let mut right = 0;
|
||||
let mut asked = 0;
|
||||
let mut mistaken = Vec::new();
|
||||
for (i, (from, to, label)) in moves.iter().enumerate() {
|
||||
let labels: Vec<&str> = moves.iter().filter(|(f, _, _)| f == from).map(|(_, _, l)| l.as_str()).collect();
|
||||
if labels.len() < 2 {
|
||||
continue;
|
||||
}
|
||||
asked += 1;
|
||||
let picked = button_pick(&cmd, group, bucket, from, to, &labels, i)?;
|
||||
if picked == *label {
|
||||
right += 1;
|
||||
} else {
|
||||
mistaken.push(format!("{to} as {picked:?}"));
|
||||
mistaken.push(format!("{from}>{to} as {picked:?}"));
|
||||
}
|
||||
}
|
||||
println!("{:<20} {:<24} {:>3} of {:<2} {}", page, bucket, right, moves.len(), mistaken.join("; "));
|
||||
report.insert(format!("desk {page} {bucket}"), json!({ "right": right, "of": moves.len(), "mistaken": mistaken }));
|
||||
if asked == 0 {
|
||||
continue;
|
||||
}
|
||||
println!("{:<20} {:<24} {:>3} of {:<2} {}", page, bucket, right, asked, mistaken.join("; "));
|
||||
report.insert(format!("desk {page} {bucket}"), json!({ "right": right, "of": asked, "mistaken": mistaken }));
|
||||
}
|
||||
println!();
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user