title: "Experiment management" description: "A prompt run across a dataset, a sweep of hyperparameters and a set of model profiles, scored by evaluators." nodes: 23 progress: 70% generated: ohm spec editor#

Experiment management#

A prompt run across a dataset, a sweep of hyperparameters and a set of model profiles, scored by evaluators.

Contents#

Overview
- Scope
- One matrix: sweeps × profiles × rows
Data model
Execution
Evaluators
- The catalogue
- The LLM judge
Review findings

☑ Overview#

☑ Scope#

note

packages/core/src/experiments.ts, apps/ui/src/views/ExperimentView.tsx, and the experiments/*.experiment.json documents plus their <name>.runs/ folders.

☑ One matrix: sweeps × profiles × rows#

decision

An experiment is the cross product of sweep trials, model profiles and dataset rows, scored by a list of evaluators. That single shape covers prompt A/B tests, model comparisons and hyperparameter searches without three separate features.

☑ Data model#

☑ ExperimentDoc#

data-model

{
  name, description?,
  prompt: { name, version? },
  dataset?: { path, expectedColumn?, limit? },
  rows?: Record<string, unknown>[],
  profiles: string[],
  sweep: { mode: 'grid'|'random'|'schedule', trials?, seed?, variables: VariableSpec[] },
  evaluators: EvaluatorSpec[],
  concurrency?      // default 3
}

Three sweep variable names are reserved and set model parameters instead of prompt variables: temperature, maxTokens, profile.

spec: 07-curves — VariableSpec and the sweep engine

☑ Datasets#

data-model

Loaded from the project's data/ folder, by extension:

.jsonl / .ndjson — one JSON object per non-blank line; a parse failure names the line: "<file>: invalid JSON on line N".
.json — must be an array of objects.
.csv — a hand-rolled parser supporting "" escapes inside quotes, , separators and \n/\r\n/\r row ends, first row as headers, missing cells padded.
anything else — each non-blank line becomes { text: line }.

With no dataset and no inline rows, the experiment runs exactly one empty row — which is what makes a pure sweep over a fixed prompt work.

☑ Runs on disk#

data-model

experiments/<name>.runs/<runId>.json, one file per run holding the whole run. runId is an ISO timestamp with : and . replaced by -, plus four characters of a UUID. Listing projects each file to {id, status, startedAt, endedAt, total, completed, best}, newest first, tolerating unreadable files.

☑ Execution#

☑ Start#

flow

Load the experiment, the prompt doc and the pinned version.
Rows from the dataset or the inline list, sliced to dataset.limit.
Load every curve in the project into a resolver, so a sweep variable can follow a curve by name.
Expand the sweep (limit 500 combinations).
Profiles: the listed ones, else the prompt version's profile, else the active one — each resolved up front so a bad profile name fails before any model call.
total = profiles × trials × rows; zero → "Nothing to run: the dataset is empty"; more than 5000 → refuse with the count, before spending anything.
Loop profiles → trials → rows, rows through a worker pool of concurrency ?? 3.
Cases are collected in completion order then sorted by row index, so a run file reads in dataset order regardless of scheduling.
After each trial: summarise, recompute best as the highest avgScore (ties keep the earliest), save the run file and emit progress.

node: matrix

☑ A single case#

flow

Input is the row merged with the trial's overrides; expected output comes from dataset.expectedColumn. Render → generate → evaluate, with latency measured around generation including the failure path.

When generation fails, every evaluator scores 0/false with detail 'generation failed' rather than the case being dropped. A throwing evaluator is caught per evaluator and recorded in its detail.

Weighted score is Σ(score·weight) / Σweight, forced to 0 on error and to 1 when the total weight is 0. passed requires no error and every evaluator passing.

☑ Summary statistics#

data-model

Per trial: count, passed, errors, passRate, avgScore, avgLatencyMs, p50LatencyMs, p95LatencyMs.

Percentiles are nearest-rank on an ascending-sorted array — i = clamp(ceil(p/100 · n) − 1, 0, n−1) — not interpolated, and 0 for an empty array.

☑ Cancellation is cooperative#

requirement

cancel(runId) sets a flag consulted between rows and between trials. An in-flight generation still completes, because no provider accepts an abort signal. Progress events are throttled to ~300 ms plus one per trial.

spec: 04-models-and-settings — no streaming, no cancellation

☑ Evaluators#

☑ The catalogue#

api

Common shape: { type, weight?, params? } where params may carry value, pattern, threshold, min, max, ignoreCase, profile, rubric. Normalisation is (ignoreCase ? lowercase : identity) then trim, and the target is params.value ?? expected.

Type	Rule
`exact`	normalised equality; 1/0
`contains`	target split on whitespace into terms; score = hits/terms; passes only when every term hits
`regex`	`new RegExp(pattern, ignoreCase ? 'i' : '')` tested against the output
`similarity`	`1 − levenshtein/maxLen`; passes at `threshold ?? 0.8`
`json`	strips a ```json fence if present, then `JSON.parse`
`length`	`output.length` within `[min ?? 0, max ?? ∞]`
`llm_judge`	see below

☑ The LLM judge#

api

System prompt "You are a strict, fair grader."; the user message is the rubric (defaulting to "Is the output correct, relevant and complete for the input?"), the input JSON truncated to 4000 chars, the reference answer when present, the output truncated to 6000 chars, and the instruction "Give a score from 0 to 10. Reply with the number first, then one short reason." with maxTokens: 200.

Parsing takes the first number in the reply, maps n/10 clamped to 0…1, passes at threshold ?? 0.7, and keeps the first 200 characters as the detail. An unparseable reply scores 0 with the reply quoted.

node: risk-judge

☑ Review findings#

☐ An experiment with no evaluators reports 100%#

risk high

With an empty evaluator list the weighted score falls back to 1 and passed is vacuously true, so a misconfigured experiment looks perfect. Either refuse to start without evaluators or report score: null and a distinct "unscored" state.

node: case

☐ Experiment name is not validated for run paths#

risk high #security

listRuns/getRun join the experiment name into a path without the item-name check that every other item route applies (the runId is checked). A name containing ../ escapes the experiments folder — read-only and limited to directories ending in .runs, but it should not be reachable at all.

spec: 12-storage

☐ A `profile` sweep multiplies the run#

risk

The outer loop still iterates exp.profiles while a profile sweep variable overrides it, so N profiles × a profile sweep runs the same effective configuration N times and inflates total. Skip the outer loop when the sweep sets profile.

node: run-loop

☐ The whole run is rewritten after every trial#

risk

O(trials²) bytes written; with thousands of cases and long outputs this gets slow and produces very large files. Append per trial, or write cases to a sidecar.

node: runs

☐ Judge parsing takes the first number anywhere#

risk

A reply beginning "In 2024 the answer…" scores 1.0. There is no format enforcement and no reprompt. Anchor the match to the start, or ask for JSON and parse it.

node: judge

☐ A regex evaluator can hang the run#

risk

User patterns are compiled and executed per case with no timeout, so a catastrophically backtracking pattern stalls everything. Cap the input length or run the match with a deadline.

☐ Smaller observations#

note low

contains matches substrings, not words: expected cat passes on concatenate.
run.completed is incremented outside any lock and progress is throttled, so the counter can lag; total is never recomputed if a trial errors mid-way.
parseCsv drops rows whose fields are all empty, so a legitimately blank record disappears; headers are trimmed but data fields are not.
The seed field is disabled unless the mode is random, yet schedule mode with a random source still consumes the seeded RNG.

Links between nodes#

graph LR
  run_loop["Start"] -.-> matrix["One matrix: sweeps × profiles × rows"]
  judge["The LLM judge"] -.-> risk_judge["Judge parsing takes the first number anywhere"]
  an_experiment_with_no_evaluators_reports_100["An experiment with no evaluators reports 100%"] -.-> case["A single case"]
  a_profile_sweep_multiplies_the_run["A `profile` sweep multiplies the run"] -.-> run_loop["Start"]
  the_whole_run_is_rewritten_after_every_trial["The whole run is rewritten after every trial"] -.-> runs["Runs on disk"]
  risk_judge["Judge parsing takes the first number anywhere"] -.-> judge["The LLM judge"]