Rambarun Komaljeet · measured on one 6 GB laptop GPU

A 1.11B model trains in 4.5 GB.

Your GPU holds a much larger model than you have been told. Here is the stack that does it, the numbers, and what it costs.

gitlab.com/komalbarun/kramba-ai open source · Apache-2.0

The headline number

RTX 4050 laptop, 6 GB — measured 2026-09-19

A 1B-parameter model normally wants roughly 10 GB before it has processed a single token — about 4 GB of weights, 4 GB of gradients, and optimiser state on top. On a 6 GB card the standard advice is to rent an A100.

How it is done

Five published techniques, one working integration

No new algorithms. Every piece is published and attributed. The work is making them run together and measuring what that costs.

Two of these did not previously work together. Tied embeddings share one tensor between the input lookup and the output layer, and the offload system rejected that outright: it assigns every parameter to exactly one block, and this tensor belongs to two. The combination was guarded off as unsupported.

The fix is to pin it — a parameter reachable from two modules is never evicted, because the forward pass uses it at both ends and no single block’s residency window covers both uses. The cost is one vocabulary table held resident, 250 MB at 1B scale. Lifting that guard also re-enabled 13 of the subsystem’s own tests, which had been silently skipped ever since tying became the default, because every one of them builds a tied model and hit the guard at construction.

Update 2026-09-26: the previously OOM combination (tied embeddings + CPU offload) now works when combined with chunked cross-entropy (--chunked-loss 512). A 5-step fp32 test at 1.11B parameters ran at 79% of 6 GB = 4.85 GB without OOM. The pinned tied table (~206 MB bf16) plus chunked loss (~1.5 GB saved) makes the previously-untested four-technique stack (BCD + offload + tied + chunked) fully composable.

What you get without any of this

The same card, a standard training script — measured, both columns

A normal setup — AdamW, bf16 autocast, gradient checkpointing — is not helpless. It is capped lower:

~243M on a standard script, ~1.58B with the stack — about 6.5×. Both numbers measured on the same 6 GB card at 4,096 context, stopping at the first spill into system RAM rather than the first crash, because on Windows an over-subscribed GPU does not error — it silently gets four to fifteen times slower.

What the table does not show: a transformer and the recurrent hybrid need almost identical memory at these sizes (4.80 GB vs 5.09 GB at 243M), and a plain 1.07B transformer runs under the same stack in 4.04 GB. The 6.5× comes from the training stack, not from the architecture — so it applies to whatever model you are training, including an ordinary transformer.

Fitting is still not finishing. A 1B model runs at about 1,100 tokens/sec here, and training one properly needs roughly 36 billion of them — about 375 days. The ceiling this stack raises is what you can hold; the wall it does not touch is arithmetic.

Where the memory actually goes

Measured on a live 48M training run, 3.27 GB in use

The largest consumer is not the model. It is the cross-entropy over the vocabulary — roughly three times the size of the weights. That cost scales with vocabulary × sequence length, so a small model with a large vocabulary pays nearly as much as a big one.

Chunking and recomputing it returns 1,436 MB, at a cost of 7% step time. The loss is scored a slice of positions at a time, and each slice is recomputed during the backward pass instead of kept. Gradient checkpointing already does this for every other layer. It had never been applied here.

It helps small models most. At 1B the peak is set by weights and optimiser state instead, and the same change buys nothing there (+33 MB). The smaller your GPU and your model, the more this returns.

What it costs

The part usually left out

Fitting is not training. Compute is the wall, and no memory technique touches it.

Is that just badly written code?

No. The card was first timed on the one thing it does best, to find its real speed — not the number on the box. Then a training step was broken into parts and each part compared against that.

The maths runs at 100.6% of the fastest this GPU was seen to go. Nothing left there. The part that looked promising — the 15% spent waiting instead of calculating — caps out at a 16% gain overall even if made instant, because it is a sixth of the work.

Every remaining improvement added up comes to 20–25%, turning 375 days into about 300. The year is not bad code. It is 36 billion tokens.

How large a model will fit is a different question from how large one you can usefully train. Measured, same card, one micro-batch at 4,096 context:

The practical range on a 6 GB card is a few hundred million parameters, trained properly, in days. A 16–24 GB card moves that up several times over.

What a 6 GB card is good for

Small models, trained properly, in days. Roughly:

The first two are real projects here. The third is not, and no engineering changes that: code models many times larger, trained on trillions of tokens, are still weak at it.

Two results

Twenty tokens per parameter is too low

Measured across three model sizes at a constant learning rate, so every checkpoint is a model trained to that many tokens and not a snapshot part-way through a decaying schedule. Going from 20 to 30 tokens per parameter buys 1.9–3.3%. Going from 30 to 40 buys about 1.1% for 33% more time. Thirty is where the trade turns.

How that number moves with model size is unresolved. Fitting the standard scaling form to all 120 measured points gives a degenerate answer — it implies a 1B model needs less data than it has parameters, contradicting every raw curve. Three model sizes are not enough to separate the terms.

Perplexity can look excellent while the model cannot do its job

A code model reached perplexity 4.13 and scored 0% at filling gaps in real code — the task it was built for. The cause: that one skill was 0.02% of the training signal, and never learning it costs 0.0006 on the loss curve. Invisible. After fixing the data recipe, first-token accuracy went from 2.2% to 59.8% on identical compute.

What failed

11 ideas tested and rejected (measured, not guessed)

1. Original HVN architecture — lost by 13% at equal data vs gated DeltaNet (retracted)
2. torch.compile — 2.4× slower (4 independent measurements)
3. Mixture-of-Experts — loses to simply scaling the dense model
4. Tensor-train factorisation — half params, 22% worse
5. GHOST (weight extrapolation) — loses to doing nothing
6. Looped/weight-shared models — 3 layers × 4 loses to 12 layers by 11× noise floor
7. Chunk-parallel scan — 4× slower (serial triangular solves)
8. Sequence-length curriculum — 2.9% slower (occupancy against 20 SMs)
9. Per-token entanglement rate — no gain at 83M
10. Entropy-gated early exit — does not work (retracted)
11. Adafactor without BCD — spills to host RAM, 10× slower

5 retractions (conclusions this project withdrew)

1. Original HVN "novel architecture" claim — lost to published DeltaNet by 13%
2. #117 cache serving untrained weights — whole epic on aggregate loss that still trended down
3. #122 5-item recall gate — gating scale-up on luck-dominated measurement
4. #132 verdict on train loss — multi-pass regime measures memorisation, not quality
5. #138 CUDA graph "6.8× speedup" — compared one replay against one eager step from same state; both do same thing, passed at max|dlogits|=0 while context silently stopped growing

4 optimisations measured slower than doing nothing

1. torch.compile — 2.4× slower (20.72 vs 8.66 s/step)
2. Chunk-parallel scan (Design A) — 2–4× slower at d=64
3. Sequence-length curriculum — 2.9% slower at 83M
4. Attention layer removal — 6.4% slower (attention is the cheap part during training)

8 silent bugs caught by falsifiable tests

1. #117 — cache served untrained weights; only test watched aggregate loss (trended down)
2. #122 — 5-item recall gate passed by luck, not signal
3. #132 — graded on train loss in multi-pass regime (measures memorisation)
4. #138 — CUDA graph compared 1 step against 1 step from same state; both identical, passed at 0 diff
5. #155 — tokenizer.vocab_size EXCLUDES added tokens; embedding sized 3 rows short, OOB CUDA assert
6. #157 — tokenized .bin cache keyed on string "code", not corpus; silently reused old cache
7. #112 — pre-training audit deepcopied model to GPU, OOM'd 6 GB card at large sizes
8. #159 — cosine LR scheduler compounded decay on every resume (sawtooth LR)

The largest one: this project began as an original architecture. Benchmarked against the published alternative under matched conditions, it lost by 13% at equal data. It had already been replaced in the code months earlier, while the front page still called it novel.

One rule, applied before any measurement: if the thing being tested were broken, would this test catch it? Answer before seeing the result. A test that cannot fail turns a bug into a published number.

Limits

• The architecture is a reimplementation of published work — gated DeltaNet, GLA/Mamba-2 gating, Mamba/RWKV-7 convolution and output gate. The code is original; the design is not.
• Constant-memory inference is a property of that entire model family, not of this work.
• Nothing above 48M has been trained to convergence here. Every scaling statement is measured below that size and should not be extrapolated past it.
• The 100,000-token inference memory figure quoted in the repository is calculated, not measured, and is therefore not quoted on this page.
• Comparisons are single-seed: that removes luck between arms, not luck shared by both. Training the same model three times, changing only the starting random numbers, moved the result by 0.089. Some differences reported in this project are smaller than that and cannot be separated from luck without repeating each across several seeds.
• One measurement was dropped for that reason. An attempt to find how much numerical precision gradients need produced differences of at most 0.016, inside the 0.089 of seed noise. The answer was “cannot tell”, so nothing is claimed.