Bendik Aagaard LynghaugandClaude Fable 5.1
02672547ee
`portal-trainer simulate` reads the model `question_lint --needs-sim` exports. A visitor model picks an alternative and fills the form in a manner that varies by run; the case is then walked along the handling group's own desk buttons to a finished state, a different one each run, so every state and button is exercised. At each button the handler model is asked one thing: with only what the form collected, could this role really do what the button claims? Its question goes back to the visitor, who may not know. Who wins is deliberately not simulated. The first version let the handler decide: qwen3:8b sent every case to "won", and after one sentence of prompt change, every case to "lost". Outcomes belong to the live bucket; `success` in needs.yaml is what a report over real records counts. `compare` sets two score files side by side. engines/ollama.sh honours SEED, FORMAT=json and TEMPERATURE so the same case replays the same way against two versions of a site. generator/reference.md documents `success` and `stages`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4jrCgLiKKHAEFuZUJjckH
19 lines
1004 B
Bash
Executable File
19 lines
1004 B
Bash
Executable File
#!/bin/sh
|
|
# LLM backend for portal-trainer on a local Ollama: prompt on stdin,
|
|
# text on stdout. MODEL, OLLAMA_HOST, NUM_CTX and TIMEOUT override.
|
|
# Thinking is off: the generator wants files, the answerer wants JSONL.
|
|
# The simulator sets SEED (so the same case replays the same way
|
|
# against two versions of a site), FORMAT=json and TEMPERATURE.
|
|
set -eu
|
|
jq -Rs --arg model "${MODEL:-qwen3:8b}" --argjson ctx "${NUM_CTX:-32768}" \
|
|
--argjson temp "${TEMPERATURE:-0.2}" --arg seed "${SEED:-}" --arg format "${FORMAT:-}" \
|
|
'{model: $model, stream: false, think: false,
|
|
options: ({num_ctx: $ctx, temperature: $temp}
|
|
+ (if $seed == "" then {} else {seed: ($seed | tonumber)} end)),
|
|
messages: [{role: "user", content: .}]}
|
|
+ (if $format == "json" then {format: "json"} else {} end)' |
|
|
curl -sS --fail-with-body --max-time "${TIMEOUT:-1800}" \
|
|
"${OLLAMA_HOST:-http://127.0.0.1:11434}/api/chat" \
|
|
-H "content-type: application/json" --data-binary @- |
|
|
jq -er '.message.content'
|