answer --seed, and runs/spread.sh: a score comes with its spread
Every score so far was one run of a small visitor. `answer --seed n` answers the same tasks with other shuffles and sampling, and runs/spread.sh prints a run's score per seed, so a change can be told from the visitor's mood. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
ec2f07da77
commit
cc50249d7f
Executable
+19
@@ -0,0 +1,19 @@
|
||||
#!/bin/sh
|
||||
# The noise floor: answer one run's tasks with several seeds and print
|
||||
# each seed's score, so a change can be told from the visitor's mood.
|
||||
# runs/spread.sh runs/<run> [seeds...] (default seeds: 1 2 3)
|
||||
set -eu
|
||||
run=${1:?usage: spread.sh runs/<run> [seeds...]}; shift
|
||||
seeds=${*:-1 2 3}
|
||||
IRIS=/var/local/cargo-target/release/iris
|
||||
T=/var/local/cargo-target/release/portal-trainer
|
||||
LLM="MODEL=qwen3:8b $(dirname "$0")/../engines/ollama.sh"
|
||||
cd "$run"
|
||||
for mode in full skim; do
|
||||
skim=""; [ "$mode" = skim ] && skim="--skim"
|
||||
for s in $seeds; do
|
||||
$T answer --tasks tasks-$mode.jsonl --repo site --llm "$LLM" --seed "$s" > answers-$mode-seed$s.jsonl
|
||||
printf "%s seed %s: " "$mode" "$s"
|
||||
$IRIS check --path site/questions --needs-tasks tasks-$mode.jsonl $skim --needs-score answers-$mode-seed$s.jsonl 2>&1 | grep -E "^wording:"
|
||||
done
|
||||
done
|
||||
Reference in New Issue
Block a user