Bendik Aagaard LynghaugandClaude Fable 5.1
cc50249d7f
Every score so far was one run of a small visitor. `answer --seed n` answers the same tasks with other shuffles and sampling, and runs/spread.sh prints a run's score per seed, so a change can be told from the visitor's mood. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
20 lines
854 B
Bash
Executable File
20 lines
854 B
Bash
Executable File
#!/bin/sh
|
|
# The noise floor: answer one run's tasks with several seeds and print
|
|
# each seed's score, so a change can be told from the visitor's mood.
|
|
# runs/spread.sh runs/<run> [seeds...] (default seeds: 1 2 3)
|
|
set -eu
|
|
run=${1:?usage: spread.sh runs/<run> [seeds...]}; shift
|
|
seeds=${*:-1 2 3}
|
|
IRIS=/var/local/cargo-target/release/iris
|
|
T=/var/local/cargo-target/release/portal-trainer
|
|
LLM="MODEL=qwen3:8b $(dirname "$0")/../engines/ollama.sh"
|
|
cd "$run"
|
|
for mode in full skim; do
|
|
skim=""; [ "$mode" = skim ] && skim="--skim"
|
|
for s in $seeds; do
|
|
$T answer --tasks tasks-$mode.jsonl --repo site --llm "$LLM" --seed "$s" > answers-$mode-seed$s.jsonl
|
|
printf "%s seed %s: " "$mode" "$s"
|
|
$IRIS check --path site/questions --needs-tasks tasks-$mode.jsonl $skim --needs-score answers-$mode-seed$s.jsonl 2>&1 | grep -E "^wording:"
|
|
done
|
|
done
|