Rambarun Komaljeet · measured on one 6 GB laptop GPU
Your GPU probably holds a much larger model than you have been told. This is the stack that does it, the measured numbers, and the honest cost.
gitlab.com/komalbarun/kramba-ai open source · Apache-2.0RTX 4050 laptop, 6 GB — measured 2026-08-14
A 1B-parameter model normally wants roughly 10 GB before it has processed a single token — about 4 GB of weights, 4 GB of gradients, and optimiser state on top. On a 6 GB card the standard advice is to rent an A100.
Five published techniques, one working integration
Nothing here is a new algorithm. Every piece is published and attributed; the contribution is making them work together and measuring what the combination actually costs.
| Technique | What it removes | Saving |
|---|---|---|
| Block-coordinate descent (BAdam) | optimiser state for every block but the active one | ~8× on state |
| CPU weight offload | inactive blocks leave the GPU entirely | scales with depth |
| Ternary weights (BitNet b1.58) | 32-bit weights become −1 / 0 / +1 | 2.33× |
| Tied embeddings | one vocabulary table instead of two | 70M params at 1B |
| Gradient checkpointing | stored activations — recomputed instead | activations to ~50 MB |
Two of these did not previously work together. Tied embeddings share one tensor between the input lookup and the output layer, and the offload system rejected that outright: it assigns every parameter to exactly one block, and this tensor belongs to two. The combination was guarded off as unsupported.
The fix is to pin it — a parameter reachable from two modules is never evicted, because the forward pass uses it at both ends and no single block’s residency window covers both uses. The cost is one vocabulary table held resident, 250 MB at 1B scale. Lifting that guard also re-enabled 13 of the subsystem’s own tests, which had been silently skipped ever since tying became the default, because every one of them builds a tied model and hit the guard at construction.
The same card, a standard training script — measured, both columns
A normal setup — AdamW, bf16 autocast, gradient checkpointing on, nothing else — is not helpless. It is just capped much lower:
| Model | Standard script | With this stack |
|---|---|---|
| 157M | 3.38 GB — fits | fits |
| 243M | 4.80 GB — fits | fits |
| 434M | 9.04 GB — spills | fits |
| 1.07B (recurrent) | far past the card | 4.00 GB |
| 1.07B (plain transformer) | far past the card | 4.04 GB |
| 1.58B | far past the card | 5.92 GB — the ceiling |
~243M on a standard script, ~1.58B with the stack — about 6.5×. Both numbers measured on the same 6 GB card at 4,096 context, stopping at the first spill into system RAM rather than the first crash, because on Windows an over-subscribed GPU does not error — it silently gets four to fifteen times slower.
Worth noting what the table does not show: a transformer and the recurrent hybrid need almost identical memory at these sizes (4.80 GB vs 5.09 GB at 243M), and a plain 1.07B transformer runs under the same stack in 4.04 GB. The 6.5× comes from the training stack, not from the architecture — so it applies to whatever model you are training, including an ordinary transformer.
Fitting is still not finishing. That 1.07B transformer runs at 1,160 tokens/sec, and a properly trained 1B model needs about 20 billion of them — roughly 200 days. The ceiling this stack raises is what you can hold; the wall it does not touch is arithmetic.
Measured on a live 48M training run, 3.27 GB in use
| Item | Size |
|---|---|
| Computing the loss — 4,096 positions × 32,003 vocabulary | ~1,500 MB |
| CUDA context, kernels, workspaces | ~400 MB |
| Weights | 183 MB |
| Optimiser state | 183 MB |
| Layer activations | ~50 MB |
The largest consumer is not the model. It is the cross-entropy over the vocabulary — roughly three times the size of the weights. That cost scales with vocabulary × sequence length, so a small model with a large vocabulary pays nearly as much as a big one.
Chunking and recomputing it returns 1,436 MB — measured, at a cost of 7% step time. The loss is scored a slice of positions at a time and each slice is recomputed during the backward pass rather than kept, which is exactly what gradient checkpointing already does for every other layer and had simply never been applied here.
| 48M, 4,096 context | Allocated | Reserved | s/step |
|---|---|---|---|
| whole-sequence loss | 2,275 MB | 2,584 MB | 3.21 |
| 512 positions at a time | 1,003 MB | 1,148 MB | 3.43 |
It helps small models most, which is the point: at 1B the peak is set by weights and optimiser state instead, and the same change buys nothing there (+33 MB). The smaller your GPU and your model, the more this returns.
The part usually left out
Fitting is not training. Compute is the wall, and no memory technique touches it.
| Model | Fits on 6 GB? | Time to train properly |
|---|---|---|
| 48M | yes | ~1 day |
| 108M | yes | ~5 days |
| 157M | yes | ~10 days |
| 1B | yes — 4.05 GB | ~267 days |
How large a model will fit is a different question from how large one you can usefully train. Measured, same card, one micro-batch at 4,096 context:
| Model | Peak VRAM | Verdict |
|---|---|---|
| 1.07B | 4.00 GB | fits comfortably |
| 1.58B | 5.92 GB | at the edge of 6 GB |
| 2.45B | 9.39 GB | spills to system RAM |
So the practical range on a 6 GB card is up to a few hundred million parameters, trained properly, in days. On a 16–24 GB card the same techniques move that range up several times over. Anyone told their GPU cannot train anything worth having is being sold something.
Measured across three model sizes at a constant learning rate — so every checkpoint is genuinely a model trained to that many tokens, rather than a snapshot part-way through a decaying schedule. Going from 20 to 30 tokens per parameter buys 1.9–3.3%. Going from 30 to 40 buys about 1.1% for 33% more time. Thirty is where the trade turns.
How that number moves with model size is unresolved. Fitting the standard scaling form to all 120 measured points gives a degenerate answer — it implies a 1B model needs less data than it has parameters, contradicting every raw curve. Three model sizes are not enough to separate the terms. That failure is published here rather than quietly dropped.
A code model reached perplexity 4.13 and scored 0% at filling gaps in real code — the task it was built for. The cause: that one skill was 0.02% of the training signal, and never learning it costs 0.0006 on the loss curve. Invisible. After fixing the data recipe, first-token accuracy went from 2.2% to 59.8% on identical compute.
Kept on purpose
| Category | Count |
|---|---|
| Ideas tested and rejected on measurement | 7 |
| Of those, conclusions this project retracted itself | 4 |
| Optimisations that measured slower than doing nothing | 2 |
| Silent bugs caught only by a test designed to fail | 6 |
Including the largest one. This project began as an original architecture. It was eventually benchmarked against the published alternative under matched conditions, lost by 13% at equal data, and had already been replaced in the code months earlier — while the front page went on calling it novel. Both facts are now in the repository.
The rule that came out of all this, applied before any measurement is taken: if the thing being tested were broken, would this test catch it? Answer that before seeing the result. A test that cannot fail is worse than no test, because it turns a bug into a published number.