First generated site, a held-out test, and the leak it exposed

runs/contractor-2026-09-21: the generator's first end-to-end run, on
intakes/contractor.yaml. Needs parsed first round, structure green
first round, 10/10 on its own personas - and 8/10 headings-only on
ten held-out personas with qwen3:8b as the visitor, because the pages
had been written with the personas in view. Phase 2 now shows the
model the needs without the personas.

`portal-trainer answer` exposes the blind answerer on its own. README
rewritten around what this is: an intake in, a verified content repo
out.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4jrCgLiKKHAEFuZUJjckH
This commit is contained in:
Bendik Aagaard Lynghaug
2026-09-21 18:18:08 +02:00
co-authored by Claude Fable 5.1
parent b6bdac6f62
commit e163c22792
26 changed files with 610 additions and 103 deletions
+11 -2
View File
@@ -83,7 +83,16 @@ fn main() -> anyhow::Result<()> {
}
let needs = std::fs::read_to_string(opts.out.join("needs.yaml")).context("no needs.yaml was produced")?;
// Phase 2: structure.
// Phase 2: structure. The page-writing model sees the needs and
// never the personas: pages written with the test in view route
// their own personas perfectly and strangers badly (measured:
// 10/10 on its own personas, 8/10 headings-only on held-out ones).
let needs_full = needs;
let needs = needs_full
.split("\npersonas:")
.next()
.unwrap_or(&needs_full)
.to_string();
let mut green = false;
let mut report = String::new();
for round in 1..=opts.rounds {
@@ -145,7 +154,7 @@ fn main() -> anyhow::Result<()> {
if defects.is_empty() || round == opts.wording_rounds {
break;
}
let personas = needs.split("\npersonas:").nth(1).unwrap_or_default();
let personas = needs_full.split("\npersonas:").nth(1).unwrap_or_default();
let prompt = format!(
"{REFERENCE}\n\n# Task\n\nThe structure is correct. Visitors are picking the wrong alternative. Each MISROUTE below is a persona who, reading the page, chose an alternative that does not meet their need; HESITANT means they chose right and nearly did not. Rewrite alternative headings first (a heading that needs its description has failed), descriptions second. Keep every `record_as`, `action`, field and state as it is. If you rename an alternative that needs.yaml names in `via`, you may not: keep that one. Return only the pages you change, each complete.\n\n## Defects\n\n```\n{defects}```\n\n## The personas\n\n```yaml\npersonas:{personas}\n```\n\n## Current files\n\n{}\n\n{FORMAT}",
dump(&opts.out)?