BashGym Training Methods
Choose the simplest method that has the data, reward signal, and release evidence it needs.
Training method choice should not start with the most exciting acronym. It should start with the evidence you actually have. BashGym's training methods are arranged around that principle: use the simplest method that can teach the missing behavior, then require behavior evidence before treating the student as better.
Core rule: SFT teaches the shape of work. Preference methods teach better choices. Verifier-backed RL only helps when the model can already produce attempts worth scoring.
| Method | Use It When | Evidence That Matters |
|---|---|---|
| SFT | The model needs the format and rhythm of verified coding work. | Heldout trace behavior, truncation checks, and executable pass@k. |
| DPO | You have chosen and rejected answers for the same prompt. | Preference accuracy plus no regression against the SFT baseline. |
| Reward model | You need a scorer for audits, best-of-N, or later RL rewards. | Heldout accuracy, calibration, length bias, variance, and leakage checks. |
| GRPO/RLVR | Executable attempts sometimes pass and sometimes fail. | Reward contrast, pass@1/pass@k, verifier health, and tamper controls. |
| Distillation | A teacher can solve tasks the student cannot yet attempt. | Student pass@k, teacher comparison, and tool-format regression checks. |
Start With SFT When The Student Needs Shape
Supervised fine-tuning is the baseline path because coding-agent behavior has a format. The model needs to learn how a task is framed, how tool calls appear, how terminal output changes the next step, and what a complete verification loop looks like.
A small SFT run is also a systems test. It proves data loading, chat templates, masking, sequence length, metrics logging, and checkpoint output. If an SFT smoke cannot preserve final verifier output or recovery steps, moving to preference tuning or RL just hides the problem under a more expensive method.
Use DPO When The Difference Is Comparable
DPO is useful when BashGym can build chosen and rejected answers for the same prompt. The rejected answer should be a plausible mistake, not a random unrelated failure. Good preference data sharpens decisions the student already understands; bad preference data teaches shortcuts.
The first DPO question is not "did the reward margin go up?" It is "did heldout behavior stay healthy compared with the SFT base?" Preference metrics can improve while task behavior gets worse, so DPO belongs after a baseline and before any claim of improved capability.
Treat Reward Models As Evidence Before Optimization
Reward models, outcome reward models, and process reward models are powerful because they can score or audit candidate behavior. They are also easy to over-trust. A learned scorer should first be used for inspection, best-of-N selection, rejection sampling, or audits.
Before a reward model becomes a training signal, BashGym should validate the reward artifact, require an eval split, check calibration and length bias, and confirm that eval-only sources did not leak into training. The scorer has to prove that it separates useful candidates without rewarding verbosity, shortcuts, or contamination.
Move To GRPO/RLVR When Rewards Have Contrast
Verifier-backed RL becomes interesting when sampled attempts sometimes pass and sometimes fail. That contrast gives group-relative methods a learning signal. If every attempt fails, the environment is probably too hard or the student is not ready. If every attempt passes, the task is too easy to teach much beyond regression stability.
For BashGym, terminal RL is not just an optimizer setting. It depends on executable environments, reliable verifiers, timeout handling, protected-file checks, and reward-hacking controls. pass@k can show exploration gains, but promotion needs heldout comparison and safety gates too.
Practical Method Order
- Use SFT when the model needs format, tool-call rhythm, or basic task shape.
- Use DPO when chosen and rejected answers are tied to the same prompt.
- Use reward models first as scoring and audit evidence, not as an automatic release signal.
- Use GRPO/RLVR when executable environments produce non-trivial reward variation.
- Use distillation when a teacher can solve tasks the student cannot yet attempt.
Distillation And Cascade Training Fill The Gap
Distillation is useful when the student is too weak for RL but a teacher can produce usable traces. It can move frontier behavior into a smaller student or narrow specialist, but it still needs pass@k and heldout checks because the student can copy style without copying competence.
Cascade or domain-staged training helps when one broad run blurs domains together. File operations, shell commands, codebase navigation, and multi-step reasoning can each have different failure modes. Staging them makes regressions easier to see and gives the final generalist a clearer promotion story.
Keep Backend-Dependent Work Honest
DPPO replay, external RL backends, and JEPA-style ECHO/RWML world-model signals are part of the research surface, but they should remain caveated until backend smoke tests and behavior gates exist. Replay readiness is not the same as model improvement. World-model quality is diagnostic until it correlates with heldout pass@k and safety evidence.
What Promotion Requires
A training method is only the path to a candidate. Promotion needs a RunCard with source manifests, compute target, config snapshot, metrics, known limits, and release evidence. The student should route real work only where heldout traces, pass@k, comparisons, tamper checks, and spurious-reward controls support the claim.