Show HN: Fine-tune an 8B model on a 4 GB laptop GPU

The interesting engineering move is treating the reference model problem as a streaming problem rather than a memory problem. DPO and KTO both need a reference distribution, and the standard fix, loading a second copy of the base, costs exactly one copy's worth of weights, which on an 8B model in 4-bit is roughly 4 GB, i.e. the entire budget of the target card. Soup sidesteps that by running the same base twice through the layer pipeline: once with the adapter active for the policy logprobs, once with it switched off for the reference logprobs. The material reports this is bit-exact against a non-streamed run, which is the right bar, approximation in preference loss is how you ship a model that passes evals and fails in production.

The trade-off is honestly disclosed in numbers, which is the part of the release worth taking seriously. On the RTX 3050 4 GB test, streamed DPO peaks at 0.914× the SFT peak, and DPO reads the layer stack 1.52× per step. That is the cost of the memory win and it is concrete: an engineer sizing a run can convert those ratios into wall-clock estimates before committing. The exclusion of GRPO and PPO is principled, generation re-reads every layer per token, which destroys the amortization that makes layer streaming pay off. The author is drawing a line around what the technique can and cannot do, rather than marketing a universal solution.

The v0.72.1 bug fix buried in the notes is a useful tell about how the system stores adapters under the hood. Adapters trained with stream_layers on v0.72.0 were saved under keys containing an extra .inner. segment, which meant standard loaders returned the untuned base silently. That is a classic checkpoint-format regression: the training loop was happy because it was reading through the streaming wrapper, but anything that loaded the safetensors file directly, inference, export, evaluation, saw the base model. The diagnostic the release ships (a one-liner that greps for .inner. in the keys) is the kind of operational detail that matters when an adapter that 'trained fine' produces nonsense at serve time.

The earlier releases in the changelog (v0.71.33 through v0.71.40) frame Soup less as a trainer and more as a full post-training loop: reward synthesis from reference outputs, an extraction-based regression gate over seven offline suites (MCQ, arithmetic, tool-calling, JSON validity, safety/refusal), a ship verdict that can be emitted and provenance-bound to the exact recipe, and a speculative-decoding measurement command. Read together, v0.72.4 is not a standalone announcement, it is one piece of a workflow where the same CLI handles data-to-reward, reward-to-adapter, and adapter-to-merge with a gate at the end. The laptop-class training is the attention-grabbing piece, but the operational story is the broader CLI surface.

Only one feed carried this, so corroboration is limited and the framing is the author's own. The claims that are worth trusting from the material are the ones that come with measurements attached: 0.914× SFT throughput, +730 MB for a forced second model, 1.52× layer reads per DPO step, bit-exact against non-streamed. The claims that are not directly substantiated in the extract are the '30-50% of time spent on infrastructure' framing and the overall premise that one command replaces the usual mess, those are positioning, not data. An engineer evaluating this should look at the v0.72.1 checkpoint-format issue first, because it tells you what kind of bugs the project ships and fixes.