RL can only amplify
what’s already there.

Most people fine-tune with SFT. Far fewer have actually run RL post-training, and almost all of the published RLVR work lives on math and code, with Qwen models. I wanted to know when reinforcement learning with verifiable rewards helps a small model reason over structured data, the kind most businesses actually run on, and to answer it with the controls that literature tends to skip. The testbed is NFL box scores: one right answer, legible, and fun to read.

Why verifiable rewards

RLHF needs a learned reward model and preference data, which is the expensive, fragile part. When the answer is checkable, the reward is just is_correct(), and the reward model disappears. GRPO samples a group of answers to each question, scores them, and pushes the policy toward the ones that beat the group average. No value network, no preference pairs. The whole thing runs small: base measurements on a 10GB RTX 3080, training on a single rented A100 at a few dollars a run. The bottleneck is generation, not gradients.

Six tasks, ordered by what they demand

The questions split into six verifiable kinds, ordered by how much of the table the answer touches: a rule-based decision (go for the touchdown or kick the field goal), locate a player and add two of his fields, sum a column, argmax a column and name the player, build a composite scoring line, and threshold-filter the rows to a set. Every result is judged against the task's naive floor — the accuracy of always guessing the majority answer — never against zero, so a task that can be won by guessing can't pose as learning.

Measure what is reachable before spending anything

GRPO can only reinforce what the base model already samples. A task the base never gets right in 64 tries is unlearnable by GRPO, whatever the recipe. So before training anything I measured base pass@k per task. That one cheap measurement says which cells can move and which would just burn GPU time, and it gates everything after it.

The wall is arithmetic, not depth

A depth-2 task (add two fields) is capped just as hard as a depth-4 one, while a depth-3 argmax is highly reachable. The base produces a correct rollout for the select-and-decide tasks often enough for GRPO to grab onto, and almost never for the multi-step numeric ones. Training bears the prediction out: the reachable tasks improve — the decision task went from 0 to 47% correct — and the composite scoring task moves +0.0 and stays there.

Why one model is not a result

The reason the field's Qwen-only numbers aren't trustworthy is the Qwen confound: on Qwen, even random rewards "work," because RL surfaces behavior the model already pretrained. Run the same study across model families and it shows plainly. The two simple-arithmetic tasks are unreachable on SmolLM2 and perfectly reachable on Qwen. A Qwen-only study would watch GRPO light them up and conclude "RLVR taught structured-data arithmetic," when Qwen knew it before training started. The control catches the illusion. The composite-arithmetic wall, on the other hand, holds on both models, which is exactly where a claim about GRPO's limits can honestly be made.

What it showed

SmolLM2 gained 14.2 ± 0.3 points overall, averaged over three seeds — a weak base nearly quadrupling its verifiable accuracy, from 5% to 19%. Qwen barely moved (+0.6, not significant), because its base is already saturated. RLVR amplified the weak base where headroom existed and did nothing where it didn't. The split survives the jump off the synthetic generator: train on synthetic, evaluate on real 2023 NFL box scores, and SmolLM2 gains +21.5 ± 0.1 points (8% to 30%) while Qwen stays flat — the same story on a distribution it never trained against, and the strongest external-validity check I have. Across every seed, family, and recipe — twelve cells in all — the composite scoring task never moved off +0.0. The multi-family control, not a new algorithm, is the contribution.

The figure carries the argument. The base splits cleanly into the tasks it can reach — decide and argmax, on the right — and the multi-step arithmetic it mostly can't, bunched at low reachability; GRPO's gain lands on the reachable side, and the decision task goes 0 → 47%. The composite scoring task sits on the zero line and stays there on every recipe and both base models. That last part matters, because pass@64 alone doesn't explain it: the base lands the composite scoring line about a third of the time in 64 tries, the same as another low-reach task GRPO moves twenty points — yet a GRPO update only reinforces what shows up inside a sampled group, and there it almost never does. Reachable-in-64 is not the same as learnable. The honest read of the wall is its invariance: it holds across both recipes and both model families. The other arithmetic tasks bounce between recipes at six-to-seventeen items each, so I don't read them one by one. Switch the base to Qwen and the cloud collapses top-right — the simple-arithmetic tasks Qwen already pretrained light up, the exact confound a single-model study would miss, while the composite wall holds for it too.

What broke, and how I caught it

The first Qwen run produced a clean +0.0 on every task: a perfect, suspicious null, and a fake. vLLM never emitted Qwen's stop token, so every completion ran to the length cap; masking those truncated completions zeroed the loss, the gradient norm went to zero, KL went NaN, and the optimizer never stepped. The adapter came out byte-identical to the base model. A "no change" result that was really a silent no-op. The per-task table didn't catch it. The training dynamics did: grad_norm = 0 and a NaN KL are not numbers a real run produces. I made the trainer pass each model's EOS ids to vLLM as stop tokens and abort on any no-update run, then re-ran. You only find that one by reading the dials, not the final score.

Where it stands

The headline is no longer a single seed. The clean same-regime matrix ran across three seeds, and the gain holds with tight bars — SmolLM2 +14.2 ± 0.3pp, Qwen flat — so the amplify-weak-not-saturated split is robust to the seed, not an artifact of one lucky draw. What's still open is breadth. The cross-family question wants more than two model families, and OLMo-2 and Llama-3.2 smoke runs already pass with full cells next; and because the data layer is the only file you swap, a second structured domain — synthetic invoices — already rides the same rewards and eval to show the result isn't really about football. The point of the whole exercise is rigor; the seeds are what turned the number into a result.

When you would actually reach for this

The takeaway is a decision rule, not a leaderboard number. Post-training earns its cost when you have a verifiable objective, prompting has plateaued, and the base can already reach the answer some of the time. If it can't, GRPO has nothing to reinforce. Otherwise, prompt.