Training Metrics And Release Evidence
Know which numbers help tune a run, which numbers prove behavior, and which findings should block routing.
Training metrics are easy to misread because they do not all answer the same question. Loss can tell you whether optimization is happening. Reward can tell you whether a scoring signal moved. pass@k and heldout gates tell you whether the model is actually doing better work.
BashGym treats metrics as evidence layers. The lower layers prove the run was shaped correctly. The upper layers decide whether a model is safe and useful enough to route real work.
North star: executable evidence. A trained student is not better because the loss curve looks nicer. It is better when heldout traces, executable environments, and safety controls support the claim.
Read The Run From The Bottom Up
Start with setup checks. If the dataset is malformed, traces are truncated, replay records are missing fields, or a smoke bundle is not contract-ready, the run is not ready for interpretation. Fix those problems before spending more GPU time.
Next, read training health. Loss, grad norm, learning rate, KL, entropy, and reward curves help tune the run, but they do not prove user-facing improvement. They answer "is the optimizer behaving?" not "should we route traffic?"
After that, inspect signal quality. For terminal RL, reward_std and frac_reward_zero_std matter because group-relative learning needs contrast. If every attempt gets the same reward, the model has no meaningful relative signal. That is usually a curriculum or verifier issue, not a scaling opportunity.
| Layer | Question | Examples |
|---|---|---|
| Setup | Is the run shaped correctly? | Dataset size, truncation, replay schema, smoke readiness. |
| Training health | Is optimization behaving? | Loss, grad norm, KL, entropy, learning rate, throughput. |
| Signal quality | Can the method learn from this signal? | reward_std, zero-std groups, verifier errors, timeout rate. |
| Release evidence | Is the candidate actually better and safe to route? | Heldout traces, pass@k, comparisons, tamper controls, benchmarks. |
Behavior Evidence Comes Later
Behavior evidence is where BashGym should become conservative. Heldout trace score, executable pass@1/pass@k, holdout comparison, and relevant external benchmark evidence are the signals that decide whether the candidate changed the work that matters.
Even then, pass@k is not enough by itself. A model can improve exploration while still failing at reliable first attempts. A candidate can pass easy environments while failing unseen groups. That is why release evidence needs comparisons, holdouts, and safety checks.
Training health
Loss, KL, entropy, grad norm, learning rate, throughput, and memory tell you whether the run is technically healthy.
Signal quality
Reward variation, verifier status, timeout rate, and zero-std groups tell you whether the method has a usable learning signal.
Release evidence
Heldout traces, pass@k, base comparisons, tamper controls, and benchmark evidence decide whether the model claim is credible.
Common Failure Patterns
If loss improves but pass@k is flat, the student may be learning format without outcome-producing behavior. Inspect whether long traces lost the final verifier, whether weak sessions leaked into gold data, and whether heldout tasks require skills missing from the dataset.
If reward_std is zero, the sampled attempts are not giving RL anything to learn from. Make tasks easier, add graded rewards, use active sampling, or warm-start with SFT or distillation. If pass@k is all zero, do not scale RL yet. Prove that at least some attempts can pass.
If pass@k looks good but holdout blocks promotion, assume leakage or shortcut learning until proven otherwise. Check split keys, fixture overlap, spurious-reward controls, and base-vs-candidate comparisons.
If tamper attempts appear, promotion stops. A model that edits tests, verifier files, fixtures, or protected environment state is not merely "noisy." It found a reward shortcut, and the platform should treat that as a release blocker.
World-Model Metrics Are Diagnostic
ECHO/RWML and other JEPA-style terminal-dynamics metrics can be useful for curriculum mining and failure analysis. They can show whether a model is learning to predict useful state transitions. They should not approve a release on their own.
Use world-model quality to explain where rollouts fail, find high-error transitions, or decide what data to collect next. Promote only when those diagnostics line up with heldout behavior and safety evidence.
What A RunCard Should Preserve
A RunCard is how BashGym keeps the story coherent. It should preserve the source manifests, compute target, config snapshot, metrics, release evidence, known limits, and next actions. Without that context, charts become decoration.
Minimum Promotion Evidence
- Heldout trace eval is stable or better than the baseline.
- Executable pass@k improves or meets the declared threshold.
- Holdout comparison passes on unseen groups.
- Spurious-reward and tamper controls stay clear.
- External benchmark evidence supports broad claims when those claims are made.