simulate: play every case through every state, judge only what a model can
`portal-trainer simulate` reads the model `question_lint --needs-sim` exports. A visitor model picks an alternative and fills the form in a manner that varies by run; the case is then walked along the handling group's own desk buttons to a finished state, a different one each run, so every state and button is exercised. At each button the handler model is asked one thing: with only what the form collected, could this role really do what the button claims? Its question goes back to the visitor, who may not know. Who wins is deliberately not simulated. The first version let the handler decide: qwen3:8b sent every case to "won", and after one sentence of prompt change, every case to "lost". Outcomes belong to the live bucket; `success` in needs.yaml is what a report over real records counts. `compare` sets two score files side by side. engines/ollama.sh honours SEED, FORMAT=json and TEMPERATURE so the same case replays the same way against two versions of a site. generator/reference.md documents `success` and `stages`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4jrCgLiKKHAEFuZUJjckH
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
e163c22792
commit
02672547ee
+14
@@ -22,6 +22,8 @@ use std::process::{Command, Stdio};
|
||||
|
||||
use anyhow::{anyhow, bail, Context};
|
||||
|
||||
mod simulate;
|
||||
|
||||
const REFERENCE: &str = include_str!("../generator/reference.md");
|
||||
|
||||
struct Opts {
|
||||
@@ -38,6 +40,11 @@ fn main() -> anyhow::Result<()> {
|
||||
// for grading an existing site with any backend. Answers go to
|
||||
// stdout, ready for `question_lint --needs-score`.
|
||||
let argv: Vec<String> = std::env::args().skip(1).collect();
|
||||
match argv.first().map(String::as_str) {
|
||||
Some("simulate") => return simulate::run(&argv),
|
||||
Some("compare") => return simulate::compare(&argv),
|
||||
_ => {}
|
||||
}
|
||||
if argv.first().map(String::as_str) == Some("answer") {
|
||||
let value = |flag: &str| {
|
||||
argv.iter()
|
||||
@@ -209,9 +216,16 @@ fn parse_args() -> anyhow::Result<Opts> {
|
||||
|
||||
/// Prompt on stdin, text on stdout.
|
||||
fn llm(cmd: &str, prompt: &str) -> anyhow::Result<String> {
|
||||
llm_with(cmd, prompt, &[])
|
||||
}
|
||||
|
||||
/// As `llm`, with hints for the backend in its environment (`SEED`,
|
||||
/// `FORMAT=json`, `TEMPERATURE`). A backend is free to ignore them.
|
||||
pub(crate) fn llm_with(cmd: &str, prompt: &str, env: &[(&str, &str)]) -> anyhow::Result<String> {
|
||||
let mut child = Command::new("sh")
|
||||
.arg("-c")
|
||||
.arg(cmd)
|
||||
.envs(env.iter().copied())
|
||||
.stdin(Stdio::piped())
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::inherit())
|
||||
|
||||
Reference in New Issue
Block a user