uhhm/portal
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 89 Wiki Activity

First generated site, a held-out test, and the leak it exposed

Browse Source 
runs/contractor-2026-09-21: the generator's first end-to-end run, on
intakes/contractor.yaml. Needs parsed first round, structure green
first round, 10/10 on its own personas - and 8/10 headings-only on
ten held-out personas with qwen3:8b as the visitor, because the pages
had been written with the personas in view. Phase 2 now shows the
model the needs without the personas.

`portal-trainer answer` exposes the blind answerer on its own. README
rewritten around what this is: an intake in, a verified content repo
out.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4jrCgLiKKHAEFuZUJjckH
This commit is contained in:
Bendik Aagaard Lynghaug
2026-09-21 18:18:08 +02:00
co-authored by Claude Fable 5.1
parent b6bdac6f62
commit e163c22792
26 changed files with 610 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+114 -101
View File
@@ -1,123 +1,136 @@
# portal-trainer
Onboarding a business onto [portal](https://project.uhhm.no/uhhm/portal)
is mostly a wording problem: the engine is generic, and whether it
works for a given business depends on whether its pages say, in the
visitor's words, the thing the business needs them to answer. This
repo is where that gets tested and, eventually, learned.
A business describes itself in plain words: what it does, who comes to
it, what it keeps track of and through which stages, and who handles
what. `portal-trainer` turns that into a working
[portal](https://project.uhhm.no/uhhm/portal) content repo: a question
series for every role, the state machines behind them, and a desk per
handling role. No software is written for the business; the only thing
produced is content, and every line of it is held to a verifier.
## The loop
## How it works
```
interview ─► needs.yaml ─► pages + aggregates + desks
│ │
│ question_lint (structure: pass/fail)
│ │
└─► personas ─► tasks.jsonl ─► answering engine ─► answers.jsonl
│
question_lint --needs-score ◄──┘
│
misroutes = wording defects ─► revise pages
intake (plain words)
│
▼ phase 1 frozen after this phase: the generator
needs.yaml ─────────────────────── may not weaken the spec to pass
│
▼ phase 2 ◄── question_lint findings, until it exits 0
aggregates.yaml + questions/** + desks
│
▼ phase 3 ◄── misroutes from a blind visitor model
wording
```
- **`needs.yaml`** lives in the business's own content repo. It says
who wants what, which bucket the answer lands in, which group
carries it to a finished state, and what the visitor will tolerate
in submissions and required fields.
- **The verifier is portal's `question_lint`**, not code in this repo.
It reads portal's own content types and ships in every portal
release, so a content repo's CI enforces its needs with no new
dependency. Structure is deterministic and gates CI. Wording is
graded, and gates only if you pass `--min-accuracy`.
- **This repo holds what sits around the verifier**: the trainer
procedure (`skill/SKILL.md`), sector specs to start from
(`sectors/`), and the runs (`runs/`).
1. **Needs.** The intake becomes `needs.yaml`: one need per kind of
person and thing they want, including the roles inside the
business. Each names the bucket the answer lands in, the group that
handles it, the states that count as finished, and the effort the
visitor will tolerate.
2. **Structure.** Pages, state machines and desks are generated, then
revised against portal's `question_lint` until every need is met:
a path within budget, a desk held by the right group, every
finished state reachable, no record stranding. The page-writing
model sees the needs and never the personas.
3. **Wording.** A separate model plays each persona, blind, first with
headings only and then with descriptions. A misroute is a wording
defect and goes back to the generator. A rewrite that breaks
structure is reverted.
Model output is only ever written to `aggregates.yaml`, `site.yaml`
and `questions/**.yaml` inside the output directory.
```
question_lint --path <repo>/questions # structure
question_lint --path <repo>/questions --needs-tasks tasks.jsonl # full wording
question_lint --path <repo>/questions --needs-tasks skim.jsonl --skim # headings only
question_lint --path <repo>/questions --needs-score answers.jsonl [--min-accuracy 0.8]
portal-trainer generate --intake intakes/contractor.yaml --out out/nordvik \
--lint /srv/app/<inst>/current/question_lint --llm engines/ollama.sh
portal-trainer answer --tasks tasks.jsonl --llm engines/ollama.sh > answers.jsonl
```
## The engine contract
The verifier is portal's `question_lint`, not code in this repo: it
reads portal's own content types and ships in every portal release, so
a generated repo's CI keeps enforcing its needs after handover.
A task is one page on one persona's path, as a typed `choice`:
## LLM backends
```json
{"id":"architecture-firm@/","persona":"architecture-firm","page":"/",
"state":{"body":"We run a small architecture firm and ..."},
"questions":{"pick":{"type":"choice",
"instructions":"A visitor reads the page question \"What are you here for?\". Which alternative do they choose?",
"criteria":{"<alternative heading>":"<its description>", "...":"..."}}},
"expect":["<every alternative that still meets the need>"]}
```
A backend is any command that takes a prompt on stdin and prints text.
Strip `expect` (and `need`) before an engine sees it. An answer is one
line: `{"id": "...", "choice": "<heading, verbatim>", "confidence": 0.0-1.0}`.
`state` and `questions` are exactly the arguments of
[laya](https://github.com/NandhaKishorM/laya)'s
`router.predict(state, questions)`, so a laya adapter is a loop over
the file; an LLM adapter is a prompt around the same two fields.
## Engines, and what is known about them
| Engine | Status | What it is good for |
|---|---|---|
| LLM, blind (no `expect`) | used for the first run | Reads like a visitor; slow and costs per task, so use it to find defects, not to gate CI. |
| laya `english` (ModernBERT-large, 421M) | not run yet | 33 ms per question, so it can gate CI and score thousands of generated variants. Its authors measure 0.78 intent accuracy in English. |
| laya `multilingual` | not run yet | Its authors' own sweep puts Norwegian Bokmål at **0.53 accuracy with 0.86 mean confidence**. It is confidently wrong too often to grade Norwegian wording without fine-tuning. Test a Norwegian site in English translation, or with an LLM. |
An engine is a measuring instrument. Before trusting one on a sector,
score it on tasks whose right answer nobody disputes; an engine that
misses those is measuring itself, not the page.
## First run: uhhm.no, 2026-09-21
Nine personas against the live uhhm.no front door, answered by a small
LLM that never saw `expect`. One run each; eight tasks; treat as a
lead, not a measurement.
| What the engine could read | Routed right |
| Backend | Use |
|---|---|
| Headings and descriptions | 7 of 8 |
| Headings only (`--skim`) | 5 of 8 |
| `engines/ollama.sh` | Local Ollama. `MODEL=qwen3:8b` answers a task in about two seconds on ergo's GPU and is the default visitor. Generation wants a larger model (`MODEL=qwen3.6:27b`), which runs mostly on CPU there. |
| `engines/anthropic.sh` | The Anthropic API, given a working `ANTHROPIC_API_KEY`. |
| `engines/mailbox.sh` | Writes each prompt to a directory and waits for a reply file: drive the generator from an agent session, or by hand. |
- A founder who wants a first version built picked **"Work happens
through dialogue, not tickets"** - the collaborator door - in both
modes. With headings only, so did a firm wanting to pay for storage
and backups. The heading reads as how UHHM works with clients.
- With headings only, a student seeking an internship picked
**"Attention is earned"**.
- Structure: four needs met. The site-improvement flow is advertised
and `disabled`, so it is declared `planned`.
The visitor and the generator should be different models. A model
grading its own wording agrees with itself.
## Sectors
## The onboarding site
A sector file is a needs spec with no pages behind it: a failing test
suite for the trainer to make pass, carrying what the sector demands
(why a lead is a pipeline, why a deviation report gets three fields).
`portal-portal/` is the front end of this: a portal content repo whose
front door is the intake interview. A submission lands in
`onboardings` (`open → drafted ⇄ revising → live | declined`) and is
worked from an owners' desk. Today the desk step is a person running
`generate` and sending the link; the intake record already carries
every field the generator reads, so the step that remains is a
listener on `portal.answers.submitted` that runs it and pushes the
draft to a new repo. It passes its own `needs.yaml`.
- `sectors/software-studio/needs.yaml` - UHHM's own, and green against
`uhhm/questions`.
- `sectors/construction/needs.yaml` - a contractor with project
managers, a safety officer, crew and subcontractors: winning survey
and design work, vetting subcontractors, requesting and answering
bids, hiring, deviation reporting. No pages yet.
## Measured so far
**Generated: a building contractor** (`runs/contractor-2026-09-21/`),
from `intakes/contractor.yaml`, a Sonnet-class model generating.
- Phase 1 derived six needs from the owner's prose, including one
nobody asked for: a project manager logging a phone enquiry, taken
from "enquiries arrive by phone, email and text and get lost".
- Phase 2 was green on the first round: 14 files, four state machines
(`enquiries: open → quoted → won | lost`, `subcontractors` with an
`awaiting_papers` loop, `applicants`, `site_issues`), two desks.
- Phase 3, its own personas: 10 of 10 in both modes. That number is
flattering, because in that run the pages were written with the
personas in view. Against ten **held-out** personas, with qwen3:8b
as the visitor:
| Visitor could read | Routed right |
|---|---|
| Headings and descriptions | 10 of 10 |
| Headings only | 8 of 10 |
A scaffolding firm asking "how do I get to do jobs for you?" chose
"Ask about working with us" (the hiring door). An attic conversion
chose "Get someone to look at a building". The generator has since
been changed to hide personas from the page-writing phase.
**Existing site: uhhm.no** (`runs/uhhm-2026-09-21/`), eight tasks.
| Visitor model | With descriptions | Headings only |
|---|---|---|
| small LLM, blind | 7 of 8 | 5 of 8 |
| qwen3:8b (Ollama) | 7 of 8 | 7 of 8 |
| granite4.1:8b (Ollama) | 7 of 8 | 6 of 8 |
All three send a paying client through **"Work happens through
dialogue, not tickets"**, the collaborator door. Different models,
same defect: that is what a wording problem looks like.
Small numbers, one run each. Leads, not measurements.
## What else is here
- `skill/SKILL.md` - the same procedure for an agent or a person
working by hand.
- `sectors/` - needs specs to start from. `construction/` doubles as
the held-out persona set above.
- `generator/reference.md` - everything the generating model is told
about portal content. When the lint rejects something the reference
did not warn about, the fix belongs there.
## Toward a trained model
Nothing here trains weights yet, on purpose. The order that makes
training possible:
1. The verifier has to be trusted first. It is the reward. A generator
trained against a judge that is wrong half the time learns the
judge.
2. Every onboarding leaves a run: needs, the pages at each revision,
the scores. That corpus - what wording routed which persona right,
per sector - is the training data, and it does not exist until the
loop has been run by hand a few dozen times.
3. With a fast engine (laya-class) as the scorer, generating and
scoring many page variants per need is cheap, which is what makes
search, and later fine-tuning a small generator on the winners,
practical.
Nothing here trains weights yet. What would make it possible is
accumulating: every run leaves the intake, the needs, the pages at
each revision and the scores. With a fast local visitor as the scorer,
generating many variants per need and keeping the winners is cheap,
and those winners are the fine-tuning set for a small generator. The
verifier has to be trusted before any of that: a generator trained
against a judge that is often wrong learns the judge.