uhhm/portal
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 89 Wiki Activity

portal-trainer: the loop around portal's needs verifier

Browse Source 
The trainer procedure as an agent skill, the engine contract, two
sector specs (software-studio: UHHM's own, green; construction: a
target with no pages yet), and the first wording run against uhhm.no.
The verifier itself is portal's question_lint (uhhm/portal 188f9bf).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4jrCgLiKKHAEFuZUJjckH
This commit is contained in:
Bendik Aagaard Lynghaug
2026-09-21 16:31:37 +02:00
co-authored by Claude Fable 5.1
commit 7caaa27ba7
7 changed files with 568 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+123
View File
@@ -0,0 +1,123 @@
# portal-trainer
Onboarding a business onto [portal](https://project.uhhm.no/uhhm/portal)
is mostly a wording problem: the engine is generic, and whether it
works for a given business depends on whether its pages say, in the
visitor's words, the thing the business needs them to answer. This
repo is where that gets tested and, eventually, learned.
## The loop
```
interview ─► needs.yaml ─► pages + aggregates + desks
│ │
│ question_lint (structure: pass/fail)
│ │
└─► personas ─► tasks.jsonl ─► answering engine ─► answers.jsonl
│
question_lint --needs-score ◄──┘
│
misroutes = wording defects ─► revise pages
```
- **`needs.yaml`** lives in the business's own content repo. It says
who wants what, which bucket the answer lands in, which group
carries it to a finished state, and what the visitor will tolerate
in submissions and required fields.
- **The verifier is portal's `question_lint`**, not code in this repo.
It reads portal's own content types and ships in every portal
release, so a content repo's CI enforces its needs with no new
dependency. Structure is deterministic and gates CI. Wording is
graded, and gates only if you pass `--min-accuracy`.
- **This repo holds what sits around the verifier**: the trainer
procedure (`skill/SKILL.md`), sector specs to start from
(`sectors/`), and the runs (`runs/`).
```
question_lint --path <repo>/questions # structure
question_lint --path <repo>/questions --needs-tasks tasks.jsonl # full wording
question_lint --path <repo>/questions --needs-tasks skim.jsonl --skim # headings only
question_lint --path <repo>/questions --needs-score answers.jsonl [--min-accuracy 0.8]
```
## The engine contract
A task is one page on one persona's path, as a typed `choice`:
```json
{"id":"architecture-firm@/","persona":"architecture-firm","page":"/",
"state":{"body":"We run a small architecture firm and ..."},
"questions":{"pick":{"type":"choice",
"instructions":"A visitor reads the page question \"What are you here for?\". Which alternative do they choose?",
"criteria":{"<alternative heading>":"<its description>", "...":"..."}}},
"expect":["<every alternative that still meets the need>"]}
```
Strip `expect` (and `need`) before an engine sees it. An answer is one
line: `{"id": "...", "choice": "<heading, verbatim>", "confidence": 0.0-1.0}`.
`state` and `questions` are exactly the arguments of
[laya](https://github.com/NandhaKishorM/laya)'s
`router.predict(state, questions)`, so a laya adapter is a loop over
the file; an LLM adapter is a prompt around the same two fields.
## Engines, and what is known about them
| Engine | Status | What it is good for |
|---|---|---|
| LLM, blind (no `expect`) | used for the first run | Reads like a visitor; slow and costs per task, so use it to find defects, not to gate CI. |
| laya `english` (ModernBERT-large, 421M) | not run yet | 33 ms per question, so it can gate CI and score thousands of generated variants. Its authors measure 0.78 intent accuracy in English. |
| laya `multilingual` | not run yet | Its authors' own sweep puts Norwegian Bokmål at **0.53 accuracy with 0.86 mean confidence**. It is confidently wrong too often to grade Norwegian wording without fine-tuning. Test a Norwegian site in English translation, or with an LLM. |
An engine is a measuring instrument. Before trusting one on a sector,
score it on tasks whose right answer nobody disputes; an engine that
misses those is measuring itself, not the page.
## First run: uhhm.no, 2026-09-21
Nine personas against the live uhhm.no front door, answered by a small
LLM that never saw `expect`. One run each; eight tasks; treat as a
lead, not a measurement.
| What the engine could read | Routed right |
|---|---|
| Headings and descriptions | 7 of 8 |
| Headings only (`--skim`) | 5 of 8 |
- A founder who wants a first version built picked **"Work happens
through dialogue, not tickets"** - the collaborator door - in both
modes. With headings only, so did a firm wanting to pay for storage
and backups. The heading reads as how UHHM works with clients.
- With headings only, a student seeking an internship picked
**"Attention is earned"**.
- Structure: four needs met. The site-improvement flow is advertised
and `disabled`, so it is declared `planned`.
## Sectors
A sector file is a needs spec with no pages behind it: a failing test
suite for the trainer to make pass, carrying what the sector demands
(why a lead is a pipeline, why a deviation report gets three fields).
- `sectors/software-studio/needs.yaml` - UHHM's own, and green against
`uhhm/questions`.
- `sectors/construction/needs.yaml` - a contractor with project
managers, a safety officer, crew and subcontractors: winning survey
and design work, vetting subcontractors, requesting and answering
bids, hiring, deviation reporting. No pages yet.
## Toward a trained model
Nothing here trains weights yet, on purpose. The order that makes
training possible:
1. The verifier has to be trusted first. It is the reward. A generator
trained against a judge that is wrong half the time learns the
judge.
2. Every onboarding leaves a run: needs, the pages at each revision,
the scores. That corpus - what wording routed which persona right,
per sector - is the training data, and it does not exist until the
loop has been run by hand a few dozen times.
3. With a fast engine (laya-class) as the scorer, generating and
scoring many page variants per need is cheap, which is what makes
search, and later fine-tuning a small generator on the winners,
practical.