build: slots instead of placeholders, and the first worded run

Fallback words read like finished copy, and nemotron handed the front
door straight back. The prompt now shows every string as a slot saying
what belongs there; a slot echoed back counts as no answer and keeps
the fallback.

runs/contractor-build-2026-09-21: nemotron-3.5-lightning:30b-a3b, the
model that produced no loadable site free-form, words a complete
contractor site in 130 s. Verifier green by construction; held-out
personas route 8/8 with descriptions and 8/8 on headings alone
(qwen3:8b as the visitor). "What happens next" is still 0/2 on the
front door: no heading or description says who reads a submission.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4jrCgLiKKHAEFuZUJjckH
This commit is contained in:
Bendik Aagaard Lynghaug
2026-09-21 21:08:08 +02:00
co-authored by Claude Fable 5.1
parent 8c286a8483
commit aef0fe1396
24 changed files with 784 additions and 3 deletions
+32
View File
@@ -8,6 +8,38 @@ series for every role, the state machines behind them, and a desk per
handling role. No software is written for the business; the only thing
produced is content, and every line of it is held to a verifier.
## Build: structure from the plan, words from the model
```
portal-trainer build --plan plans/contractor.yaml --out out/nordvik \
--lint question_lint --llm "MODEL=nemotron-3.5-lightning:30b-a3b engines/ollama.sh"
```
A `plan.yaml` is a needs spec that also says what each form collects.
Everything structural is derived from it with no model involved: the
state machines (read off `stages`: walk the line, be turned down at
any point, succeed only from the end, pass over `skippable` stages),
the front door, a followup page per need, a gated directory per group
with its desk and the pages only that group may open, and every button
the machine allows. The verifier accepts the result by construction.
The model writes words only, one page at a time, into a JSON object
whose shape is fixed. Its strings are merged over built-in fallbacks
key by key, so a bad reply costs wording, never a working site.
| Same contractor, same local model (nemotron 30B-A3B on ergo) | Free-form `generate` | `build` |
|---|---|---|
| A site the verifier accepts | never, in 6 rounds and 16 min | first try, 130 s |
| Held-out personas, with descriptions | - | 8 of 8 |
| Held-out personas, headings only | - | 8 of 8 |
For comparison, a Sonnet-class model writing everything free-form
scored 8 of 10 on headings only and silently dropped a stage the owner
had described. `build` cannot drop a stage: it never decides one.
`generate` (below) remains for turning a plain-words intake into a
first plan; what it should hand over is the plan, not the pages.
## How it works
```
@@ -0,0 +1,8 @@
{"id":"housing-board-damp@/","choice":"Get a professional building survey","confidence":0.95}
{"id":"buyer-before-purchase@/","choice":"Get a professional building survey","confidence":0.95}
{"id":"attic-conversion@/","choice":"Plan your building project with us","confidence":0.95}
{"id":"shop-to-flat@/","choice":"Plan your building project with us","confidence":0.95}
{"id":"electrician-firm@/","choice":"Become a trusted subcontractor","confidence":0.95}
{"id":"scaffolder@/","choice":"Become a trusted subcontractor","confidence":0.95}
{"id":"carpenter-jobseeker@/","choice":"Join our carpentry team","confidence":0.95}
{"id":"apprentice@/","choice":"Join our carpentry team","confidence":0.95}
@@ -0,0 +1,8 @@
{"id":"housing-board-damp@/","choice":"Get a professional building survey","confidence":0.95}
{"id":"buyer-before-purchase@/","choice":"Get a professional building survey","confidence":0.95}
{"id":"attic-conversion@/","choice":"Plan your building project with us","confidence":0.95}
{"id":"shop-to-flat@/","choice":"Plan your building project with us","confidence":0.95}
{"id":"electrician-firm@/","choice":"Become a trusted subcontractor","confidence":0.95}
{"id":"scaffolder@/","choice":"Become a trusted subcontractor","confidence":0.95}
{"id":"carpenter-jobseeker@/","choice":"Join our carpentry team","confidence":0.95}
{"id":"apprentice@/","choice":"Join our carpentry team","confidence":0.95}
@@ -0,0 +1,75 @@
aggregates:
- bucket: applicants
initial: open
states:
declined:
event: declined
hired:
event: hired
interviewing:
event: interviewing
open:
event: received
transitions:
interviewing:
- hired
- declined
open:
- interviewing
- declined
- bucket: enquiries
initial: open
states:
lost:
event: lost
open:
event: received
quoted:
event: quoted
surveyed:
event: surveyed
won:
event: won
transitions:
open:
- surveyed
- lost
quoted:
- won
- lost
surveyed:
- quoted
- lost
- bucket: site_issues
initial: open
states:
closed:
event: closed
in_hand:
event: in_hand
open:
event: received
transitions:
in_hand:
- closed
open:
- in_hand
- bucket: subcontractors
initial: open
states:
approved:
event: approved
awaiting_papers:
event: awaiting_papers
declined:
event: declined
open:
event: received
transitions:
awaiting_papers:
- approved
- declined
open:
- awaiting_papers
- approved
- declined
+149
View File
@@ -0,0 +1,149 @@
needs:
- done_when:
- won
- lost
enters: /
handled_by: project_managers
id: survey-enquiry
lands_in: enquiries
max_fields: 4
max_steps: 1
stages:
- open
- surveyed
- quoted
- won
- lost
success:
- won
via: Get a professional building survey
wants: someone qualified to come and look at it and say what it takes
who: a housing board or private owner with a problem in a building
- done_when:
- won
- lost
enters: /
handled_by: project_managers
id: build-enquiry
lands_in: enquiries
max_fields: 5
max_steps: 1
stages:
- open
- surveyed
- quoted
- won
- lost
success:
- won
via: Plan your building project with us
wants: drawings, a permit application and a price
who: an owner who already knows what they want built or changed
- done_when:
- won
- lost
enters: /project_managers/log-phone-enquiry
enters_as: project_managers
handled_by: project_managers
id: log-phone-enquiry
lands_in: enquiries
max_fields: 4
max_steps: 1
stages:
- open
- surveyed
- quoted
- won
- lost
success:
- won
via: Log a phone enquiry
wants: the enquiry written down before it gets lost
who: a project manager who just took a call or a text about a building
- done_when:
- approved
- declined
enters: /
handled_by: project_managers
id: subcontractor-application
lands_in: subcontractors
max_fields: 5
max_steps: 1
stages:
- open
- awaiting_papers
- approved
- declined
success:
- approved
via: Become a trusted subcontractor
wants: to be approved once, and then be called when a job needs them
who: a trade firm doing electrical, plumbing or scaffolding work
- done_when:
- hired
- declined
enters: /
handled_by: project_managers
id: job-application
lands_in: applicants
max_fields: 3
max_steps: 1
stages:
- open
- interviewing
- hired
- declined
success:
- hired
via: Join our carpentry team
wants: to be considered, without writing a letter
who: a carpenter, bricklayer or apprentice looking for work
- done_when:
- closed
enters: /crew/report-site-issue
enters_as: crew
handled_by: safety_officer
id: report-site-issue
lands_in: site_issues
max_fields: 3
max_steps: 1
stages:
- open
- in_hand
- closed
success:
- closed
via: Report a safety issue on site
wants: to report a hazard, a near miss or wrong materials before they forget
who: anyone on site, ours or a subcontractor's, on a phone with gloves on
personas:
- id: housing-board-damp
need: survey-enquiry
text: I chair the board of a housing cooperative from the seventies. Two ground-floor flats have damp in the walls and we don't know if it's drainage or the facade. We need someone to tell us.
- id: buyer-before-purchase
need: survey-enquiry
text: We are about to buy an old timber house. Before we sign we want a professional to go through it and tell us what is wrong with it.
- id: attic-conversion
need: build-enquiry
text: We want to turn our attic into two bedrooms and a bathroom. We need drawings and someone who handles the application to the municipality.
- id: shop-to-flat
need: build-enquiry
text: I own a former shop on the ground floor and want to make it a flat. I have no idea what is allowed. Can you sort that out and give me a price?
- id: electrician-firm
need: subcontractor-application
text: We are a four-person electrical firm, fully certified. We would like to get on your list so you call us when you need electricians.
- id: scaffolder
need: subcontractor-application
text: I run a scaffolding company. How do I get to do jobs for you?
- id: carpenter-jobseeker
need: job-application
text: I'm a certified carpenter with eight years on rehabilitation jobs. Are you hiring?
- id: apprentice
need: job-application
text: I finish vocational school this spring and need an apprenticeship as a bricklayer.
- id: loose-railing
need: report-site-issue
text: The railing on the third level of the scaffold on the north side is loose. Nobody got hurt but someone will.
- id: wrong-membrane
need: report-site-issue
text: The bathroom membrane delivered today is not the one in the specification. We stopped work on it.
@@ -0,0 +1 @@
qualifies: crew
@@ -0,0 +1,11 @@
alternatives:
- action: /crew/report-site-issue
consequence:
- Report issue
description: Write down a hazard, near miss or wrong materials from the site.
name: Report a site issue
description: A desk for the project managers and safety officer. Move cases through stages, approve subs, and log issues.
name: What do you need to do today?
responsible:
contact: ingrid@nordvikbygg.example
name: Ingrid Nordvik
@@ -0,0 +1,10 @@
alternatives:
- description: A safety officer will assess the issue and assign it to the right person.
name: What happens next
- action: /crew
consequence:
- Back to form
description: ''
name: Back to form
description: Your report has been received. We will review it and get back to you.
name: What type of issue did you report?
@@ -0,0 +1,30 @@
alternatives:
- action: /crew/report-site-issue-sent
consequence:
- Send report
description: Use this form to report a safety issue, near miss, or wrong materials while on site.
features:
- name: Report a site issue
requirements:
- label: Issue type
name: kind
options:
- Hazard
- Near miss
- Wrong materials
- Other
type: select
- label: Where on site
name: where
type: text
- label: What happened
name: what_happened
type: textarea
name: Report a safety issue on site
record_as: site_issues
description: Report hazards, near misses, or wrong materials so they can be fixed.
followup: false
name: What happened on site?
responsible:
contact: ingrid@nordvikbygg.example
name: Ingrid Nordvik
@@ -0,0 +1,123 @@
alternatives:
- action: /received/survey-enquiry
consequence:
- Book a survey
description: We visit, assess and give you a quote for the repair work.
features:
- name: Survey enquiry
requirements:
- label: Your name
name: name
type: text
- label: Phone number
name: phone
type: tel
- label: Building address
name: address
type: text
- label: Describe the problem
name: problem
type: textarea
- label: Site access
name: access
optional: true
type: text
name: Get a professional building survey
record_as: enquiries
- action: /received/build-enquiry
consequence:
- Start project
description: We handle drawings, permits and price for your building plans.
features:
- name: Building project
requirements:
- label: Your name
name: name
type: text
- label: Phone number
name: phone
type: tel
- label: Project address
name: address
type: text
- label: Project description
name: project
type: textarea
- label: When do you want it done
name: timeframe
options:
- This year
- Next year
- No fixed date
type: select
name: Plan your building project with us
record_as: enquiries
- action: /received/subcontractor-application
consequence:
- Apply now
description: Register your trade firm so we can call you for jobs.
features:
- name: Subcontractor sign-up
requirements:
- label: Company name
name: company
type: text
- label: Org number
name: registration_number
type: text
- label: Your trade
name: trade
options:
- Electrical
- Plumbing
- Scaffolding
- Sheet metal
- Other
type: select
- label: Contact email
name: contact_email
type: email
- label: Insurance confirmed
name: insurance_confirmed
options:
- Yes
- Not yet
type: select
- label: Trade approval
name: trade_approval
optional: true
type: text
name: Become a trusted subcontractor
record_as: subcontractors
- action: /received/job-application
consequence:
- Apply for work
description: Share your trade and skills so we can find you a job.
features:
- name: Job application
requirements:
- label: Your name
name: name
type: text
- label: Phone number
name: phone
type: tel
- label: Your trade
name: trade
options:
- Carpenter
- Bricklayer
- Apprentice
- Other
type: select
- label: Your experience
name: experience
optional: true
type: textarea
name: Join our carpentry team
record_as: applicants
description: 'Pick what you need: a building check, a new project, or joining the team.'
name: What brings you to Nordvik Bygg today?
responsible:
contact: ingrid@nordvikbygg.example
name: Ingrid Nordvik
@@ -0,0 +1 @@
qualifies: project_managers
@@ -0,0 +1,98 @@
alternatives:
- action: /project_managers
consequence:
- Continue
description: A carpenter, bricklayer or apprentice looking for their next job can be added here. The team reviews their skills and experience.
features:
- description: ''
name: ''
resource:
requires_group: project_managers
source:
bucket: applicants
kind: kv
transitions:
- from: interviewing
label: Hired
to: hired
- from: interviewing
label: Declined
to: declined
- from: open
label: Start interview
to: interviewing
- from: open
label: Declined
to: declined
name: Job applicants
- action: /project_managers
consequence:
- Continue
description: A housing board or private owner with a building problem, or an owner who wants something built or changed. Track the case from first contact to final result.
features:
- description: ''
name: ''
resource:
requires_group: project_managers
source:
bucket: enquiries
kind: kv
transitions:
- from: open
label: Start survey
to: surveyed
- from: open
label: Mark lost
to: lost
- from: quoted
label: Won
to: won
- from: quoted
label: Mark lost
to: lost
- from: surveyed
label: Make quote
to: quoted
- from: surveyed
label: Mark lost
to: lost
name: Building enquiries
- action: /project_managers
consequence:
- Continue
description: A trade firm doing electrical, plumbing or scaffolding work can be approved once and then called when a job needs them.
features:
- description: ''
name: ''
resource:
requires_group: project_managers
source:
bucket: subcontractors
kind: kv
transitions:
- from: awaiting_papers
label: Approve
to: approved
- from: awaiting_papers
label: Decline
to: declined
- from: open
label: Await papers
to: awaiting_papers
- from: open
label: Approve
to: approved
- from: open
label: Decline
to: declined
name: Subcontractors
- action: /project_managers/log-phone-enquiry
consequence:
- Log call
description: Write down a phone call or text about a building issue before it gets lost.
name: Log phone enquiry
description: Use the buckets below to move cases through their stages. Each bucket is a list of jobs waiting for action.
name: What needs doing today?
responsible:
contact: ingrid@nordvikbygg.example
name: Ingrid Nordvik
@@ -0,0 +1,10 @@
alternatives:
- description: Ingrid and the team will review the details and get back to you within one business day.
name: What happens next
- action: /project_managers
consequence:
- Back to form
description: ''
name: Back to form
description: Your phone call or text has been recorded. It will not be forgotten.
name: Did the message go through?
@@ -0,0 +1,28 @@
alternatives:
- action: /project_managers/log-phone-enquiry-sent
consequence:
- Log phone call
description: Write down the enquiry before it gets lost.
features:
- name: Log phone enquiry
requirements:
- label: Your name
name: caller
type: text
- label: Phone number
name: phone
type: tel
- label: Address
name: address
type: text
- label: What happened?
name: problem
type: textarea
name: Log a phone enquiry
record_as: enquiries
description: Quickly record a building enquiry from a phone call or text.
followup: false
name: What happened on the phone?
responsible:
contact: ingrid@nordvikbygg.example
name: Ingrid Nordvik
@@ -0,0 +1,10 @@
alternatives:
- description: We will contact you within two business days to discuss your project and send a quote.
name: 'Next steps: quote and timeline'
- action: /
consequence:
- Back to form
description: ''
name: Back to form
description: Your build enquiry has been received. A project manager will review it.
name: What happens next?
@@ -0,0 +1,10 @@
alternatives:
- description: Project managers will review your profile and contact you if they want to interview.
name: Next steps
- action: /
consequence:
- Back to form
description: ''
name: Back to form
description: Your job application has been received.
name: What are you looking for?
@@ -0,0 +1,10 @@
alternatives:
- description: Once approved, you will be added to our approved subcontractor list and contacted when relevant projects arise.
name: Approved and added to list
- action: /
consequence:
- Back to form
description: ''
name: Back to form
description: Subcontractor applications are reviewed by our project managers to verify credentials and suitability before approval.
name: What is your trade?
@@ -0,0 +1,10 @@
alternatives:
- description: A project manager will review the details and contact you to schedule a site visit.
name: 'Next: Site survey'
- action: /
consequence:
- Back to form
description: ''
name: Back to form
description: You sent a survey enquiry. This page tells you who will handle it and what happens next.
name: Who will handle your enquiry?
@@ -0,0 +1 @@
qualifies: safety_officer
@@ -0,0 +1,26 @@
alternatives:
- action: /safety_officer
consequence:
- Submit
description: Report hazards, near misses, or wrong materials before you forget them.
features:
- description: ''
name: ''
resource:
requires_group: safety_officer
source:
bucket: site_issues
kind: kv
transitions:
- from: in_hand
label: Mark as closed
to: closed
- from: open
label: Report issue
to: in_hand
name: Report a site issue
description: Log a safety concern or material issue from the site.
name: What issue do you need to report?
responsible:
contact: ingrid@nordvikbygg.example
name: Ingrid Nordvik
@@ -0,0 +1,77 @@
{
"build-enquiry": {
"control_ok": true,
"control_which": 0,
"lix": 27.477272727272727,
"options": 4,
"overloaded": false,
"personas": 2,
"required_fields": 5,
"steps": 1,
"understanding": {
"asked": 2,
"next": 0,
"which": 2
}
},
"job-application": {
"control_ok": true,
"control_which": 0,
"lix": 23.909090909090907,
"options": 4,
"overloaded": false,
"personas": 2,
"required_fields": 3,
"steps": 1,
"understanding": {
"asked": 2,
"next": 0,
"which": 2
}
},
"report-site-issue": {
"control_ok": true,
"control_which": 0,
"lix": 15.618181818181819,
"options": 1,
"overloaded": false,
"personas": 2,
"required_fields": 3,
"steps": 1,
"understanding": {
"asked": 2,
"next": 0,
"which": 1
}
},
"subcontractor-application": {
"control_ok": true,
"control_which": 0,
"lix": 29.01581027667984,
"options": 4,
"overloaded": false,
"personas": 2,
"required_fields": 5,
"steps": 1,
"understanding": {
"asked": 2,
"next": 0,
"which": 2
}
},
"survey-enquiry": {
"control_ok": true,
"control_which": 0,
"lix": 27.92885375494071,
"options": 4,
"overloaded": false,
"personas": 2,
"required_fields": 4,
"steps": 1,
"understanding": {
"asked": 2,
"next": 0,
"which": 2
}
}
}
+22
View File
@@ -0,0 +1,22 @@
content repo written to /tmp/claude-1000/-home-bl-repos-portal/b7802cea-728f-42d2-a751-680a973ba17d/scratchpad/det/nemotron2
OK: 12 question(s), 5 aggregate(s) valid
need "survey-enquiry": / [Get a professional building survey] -> "enquiries"
need "build-enquiry": / [Plan your building project with us] -> "enquiries"
need "log-phone-enquiry": /project_managers/log-phone-enquiry [Log a phone enquiry] -> "enquiries"
need "subcontractor-application": / [Become a trusted subcontractor] -> "subcontractors"
need "job-application": / [Join our carpentry team] -> "applicants"
need "report-site-issue": /crew/report-site-issue [Report a safety issue on site] -> "site_issues"
OK: 6 need(s) structurally met, 0 planned, 10 persona(s)
BUILD_EXIT=0 in 130s
=== held-out personas, qwen3:8b, full ===
wording: 8/8 routed right (100%)
=== held-out personas, qwen3:8b, skim ===
wording: 8/8 routed right (100%)
=== rate ===
need LIX which asked next gutted twin says page load
survey-enquiry 28 very easy 2/2 2/2 0/2 0/2 ok 4 options, 4 required
build-enquiry 27 very easy 2/2 2/2 0/2 0/2 ok 4 options, 5 required
subcontractor-application 29 very easy 2/2 2/2 0/2 0/2 ok 4 options, 5 required
job-application 24 very easy 2/2 2/2 0/2 0/2 ok 4 options, 3 required
report-site-issue 16 very easy 1/2 2/2 0/2 0/2 ok 1 options, 3 required
ALL_DONE
@@ -0,0 +1,2 @@
lang: en
title: Nordvik Bygg
+32 -3
View File
@@ -291,11 +291,40 @@ pub fn merge_words(fallback: &Value, given: &Value) -> Value {
(Value::Object(f), Value::Object(g)) => {
Value::Object(f.iter().map(|(k, v)| (k.clone(), g.get(k).map_or_else(|| v.clone(), |gv| merge_words(v, gv)))).collect())
}
(Value::String(_), Value::String(s)) if !s.trim().is_empty() => Value::String(s.trim().to_string()),
// An unfilled slot echoed back is no answer.
(Value::String(_), Value::String(s)) if !s.trim().is_empty() && !s.contains(SLOT) => {
Value::String(s.trim().to_string())
}
_ => fallback.clone(),
}
}
const SLOT: &str = "<WRITE";
/// The page's wording with every string replaced by a slot that says
/// what belongs there. Shown to the model in place of the fallbacks:
/// fallbacks read like finished copy, and a small model hands finished
/// copy straight back (measured: nemotron returned the front door
/// unchanged). A slot cannot be mistaken for an answer.
fn slots(value: &Value, key: &str) -> Value {
match value {
Value::Object(map) => Value::Object(map.iter().map(|(k, v)| (k.clone(), slots(v, k))).collect()),
Value::String(_) => Value::String(format!(
"{SLOT}: {}>",
match key {
"question" => "the one question this page asks its reader".to_string(),
"heading" | "next_heading" => "a heading, at most eight words, saying what the reader gets".to_string(),
"description" | "page_description" | "next_description" => "one or two short sentences".to_string(),
"button" | "back" | "confirm" => "a button, one to three words".to_string(),
"form_title" => "a title for the form, two to four words".to_string(),
other if other.contains("->") => format!("a button, two to four words, for a case going {other}"),
other => format!("a form label for the field `{other}`"),
}
)),
other => other.clone(),
}
}
fn s(v: &Value) -> &str {
v.as_str().unwrap_or_default()
}
@@ -470,14 +499,14 @@ fn words_prompt(plan: &Plan, path: &str, fallback: &Value) -> String {
.map(|n| format!("- {}: {} - wants: {}. Handled by {}. Stages: {}.", n.id, n.who, n.wants, n.handled_by, n.stages.join(" -> ")))
.collect();
format!(
"You write the words for one page of a website for this business.\n\nBusiness: {}\nAbout: {}\nContact: {} <{}>\nLanguage to write in: {}\n\nEveryone the site serves:\n{}\n\nThe page: {path}\nIt is {purpose}\n\nBelow is the page's wording as JSON, filled with placeholder text. Return the same JSON with every string rewritten. Keep every key exactly as it is; change only the string values.\n\nRules for the words:\n- `question` is a question the reader could ask themselves.\n- A heading says what the reader gets, in the reader's own words, and must work without its description.\n- Lead with what a thing enables. No negations in headings. No trade jargon the reader would not use.\n- Descriptions are one or two short sentences. Labels and buttons are a few words.\n- Write the way people in this trade and country actually speak.\n\n{}\n\nReply with the JSON object only.",
"You write the words for one page of a website for this business.\n\nBusiness: {}\nAbout: {}\nContact: {} <{}>\nLanguage to write in: {}\n\nEveryone the site serves:\n{}\n\nThe page: {path}\nIt is {purpose}\n\nBelow is the page as JSON. Every value is a slot that says what to write there. Return the same JSON with every slot replaced by your words. Keep every key exactly as it is. A slot left as it is counts as no answer.\n\nRules for the words:\n- `question` is a question the reader could ask themselves.\n- A heading says what the reader gets, in the reader's own words, and must work without its description.\n- Lead with what a thing enables. No negations in headings. No trade jargon the reader would not use.\n- Descriptions are one or two short sentences. Labels and buttons are a few words.\n- Write the way people in this trade and country actually speak.\n\n{}\n\nReply with the JSON object only.",
plan.business,
plan.about.trim(),
plan.contact.name,
plan.contact.email,
plan.language,
context.join("\n"),
serde_json::to_string_pretty(fallback).unwrap_or_default(),
serde_json::to_string_pretty(&slots(fallback, "")).unwrap_or_default(),
)
}