uhhm/portal
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 89 Wiki Activity

simulate: play every case through every state, judge only what a model can

Browse Source 
`portal-trainer simulate` reads the model `question_lint --needs-sim`
exports. A visitor model picks an alternative and fills the form in a
manner that varies by run; the case is then walked along the handling
group's own desk buttons to a finished state, a different one each
run, so every state and button is exercised. At each button the
handler model is asked one thing: with only what the form collected,
could this role really do what the button claims? Its question goes
back to the visitor, who may not know.

Who wins is deliberately not simulated. The first version let the
handler decide: qwen3:8b sent every case to "won", and after one
sentence of prompt change, every case to "lost". Outcomes belong to
the live bucket; `success` in needs.yaml is what a report over real
records counts.

`compare` sets two score files side by side. engines/ollama.sh honours
SEED, FORMAT=json and TEMPERATURE so the same case replays the same
way against two versions of a site. generator/reference.md documents
`success` and `stages`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4jrCgLiKKHAEFuZUJjckH
This commit is contained in:
Bendik Aagaard Lynghaug
2026-09-21 19:16:28 +02:00
co-authored by Claude Fable 5.1
parent e163c22792
commit 02672547ee
4 changed files with 581 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
engines/ollama.sh
+7 -2
View File
@@ -2,11 +2,16 @@
# LLM backend for portal-trainer on a local Ollama: prompt on stdin,
# text on stdout. MODEL, OLLAMA_HOST, NUM_CTX and TIMEOUT override.
# Thinking is off: the generator wants files, the answerer wants JSONL.
# The simulator sets SEED (so the same case replays the same way
# against two versions of a site), FORMAT=json and TEMPERATURE.
set -eu
jq -Rs --arg model "${MODEL:-qwen3:8b}" --argjson ctx "${NUM_CTX:-32768}" \
--argjson temp "${TEMPERATURE:-0.2}" --arg seed "${SEED:-}" --arg format "${FORMAT:-}" \
'{model: $model, stream: false, think: false,
options: {num_ctx: $ctx, temperature: 0.2},
messages: [{role: "user", content: .}]}' |
options: ({num_ctx: $ctx, temperature: $temp}
+ (if $seed == "" then {} else {seed: ($seed | tonumber)} end)),
messages: [{role: "user", content: .}]}
+ (if $format == "json" then {format: "json"} else {} end)' |
curl -sS --fail-with-body --max-time "${TIMEOUT:-1800}" \
"${OLLAMA_HOST:-http://127.0.0.1:11434}/api/chat" \
-H "content-type: application/json" --data-binary @- |