A 1B-parameter LLM at 20 tokens/second, in a language you've never heard of

I challenged my own programming language to run a real large-language model on a laptop CPU — no C kernels, no BLAS, no llama.cpp. It ended up 4× faster than the reference C implementation.

A few days ago I saw JustVugg/colibri — a 2,400-line, zero-dependency C engine that streams a 744-billion-parameter model from disk. It runs at 0.05–1 tokens/second, and it's glorious. It made me ask a different question: what's the opposite corner of that design space? Not a giant model slowly — a small model fast. Say, a 1B model at 20 tokens/second. On CPU. On the laptop I'm typing on.

And because I build machin — a machine-first programming language where every roadmap idea has to survive dogfooding — the challenge came with a rule that makes it interesting: pure MFL only. The matmul included. No FFI into a C kernel, no OpenBLAS, no hand-written SIMD. If the language can't express a fast LLM engine, that's a compiler problem, and compiler problems are exactly what dogfooding is for.

Honesty first: what "correct" means

LLM engines are easy to demo and easy to fake. Before touching performance I set the correctness bar: greedy decoding must be token-for-token identical to karpathy's llama2.c, the cleanest reference implementation there is. Not "looks coherent" — identical.

M0 was a fp32 forward pass of the stories15M checkpoint: RoPE, multi-head attention with a KV cache, SwiGLU, RMSNorm, ~330 lines of MFL. It matched run.c exactly over 200 greedy tokens on the first build. M1 moved to the real thing — TinyLlama-1.1B, int8 group-quantized (llama2.c's Q8_0 format), grouped-query attention, and a from-scratch sentencepiece BPE encoder — verified against runq.c with a logits-dump mode that diffs argmax at every position. 48 out of 48 positions identical. (Along the way I found that llama2.c's HuggingFace export silently breaks on GQA models — patched.)

First honest number: 2.1 tokens/second, single thread. The reference C on the same box: 1.67. Already ahead, but a long way from 20.

The 2.1 → 21.8 arc

Everything after that was a ratchet, each step re-verified token-identical before it counted:

2.1   naive engine, single thread
2.5   -O3 -march=native on the generated C
6.1   goroutine worker pool (8 workers, jobs over channels)
8.4   peek_i8 builtin (byte loads the vectorizer understands)
9.9   512-bit vector width
15.7  int8 activations (i8 x i8 dot, half the loads)
21.8  dot_i8 builtin + fused qkv / w1-w3 jobs

Two of those steps are the story. The worker pool is machin's Go-flavored concurrency doing real work: matmul rows fan out as packed-int jobs over channels — plain scalars, so nothing is serialized between goroutines — and every row is computed by exactly one worker, so the result is bit-identical at any thread count. Deterministic parallelism isn't a slogan; it's what let me keep the token-identical guarantee while going 3× wider.

The other is dot_i8. Profiling the generated C showed the ceiling: MFL's int is 64-bit, and a 64-bit accumulator forces the autovectorizer into half-width lanes. The identical loop with an int32 accumulator runs 40% faster — and that width is inexpressible in MFL source. So it became a builtin. That's not cheating; that's the design. Machin already ships sha256, keccak256 and aes_gcm as builtins because a language for machines should own its domains' primitives. dot_i8 is to the AI domain what sha256 is to crypto: the signed-byte dot product with a 32-bit accumulator, the group kernel of every quantized matmul on earth. Plain C inside, no intrinsics — the C compiler's autovectorizer does the rest.

The numbers

TinyLlama-1.1B-Chat, Q8_0, greedy decode
---------------------------------------------------------
runq.c -O2 (reference C, 1 thread)              1.67 tok/s
colibri, 1 thread                               6.8  tok/s
colibri, 6 threads, cold      (i5-11320H)      21.8  tok/s
colibri, 6 threads, 200-token sustained        19.0  tok/s
colibri, 12 threads, niced, SHARING the box
with a training job eating 8 cores             17.9  tok/s

Every one of those runs produces output greedy-argmax-identical to the reference. The sustained number is the honest asterisk: my laptop is a 4-core machine that throttles at 91°C, and fully-sustained generation lands at 19.0 — the same binary does 21.8 whenever the package is under 60°C. Interactive use (the way you actually talk to a model — bursts of a few dozen tokens) runs 20+. And the single-thread number is the one I keep staring at: a language that compiles through C, written for AI agents to write, beats the hand-written reference C by 4× — because the compiler emits the shape the vectorizer wants, and the language made it cheap to iterate until it did.

What the challenge left behind

This is the actual point. The engine — machin-colibri, ~600 lines of MFL, now public — is a demo. What it drove into the language is permanent: peek_i8/peek_u8/dot_i8 shipped in machin #435 with self-hosted-compiler mirrors, plus a genuine precision bug filed against machin's inferred race checker (interprocedural happens-before — found because a real engine spawns workers after loading globals in a function, #434). Dogfood drives fixes; the fixes compound for the whole ecosystem.

It also completes a set. Machin now has inferred data-race freedom, a self-hosted compiler, a neural-net framework with six trained-from-scratch demos — and an LLM inference engine that holds its own against C. For a language whose entire thesis is that AI agents are the primary programmers now, running the models themselves felt like the missing room in the house.

The repo has everything to reproduce every number — the engine, the converter, the logits-diff protocol against llama2.c, and the exact hardware disclosed per measurement: github.com/javimosch/machin-colibri.

machin is open source at github.com/javimosch/machin — the ecosystem lives at awesome-machin. Built by Javier Arancibia, the same engineering that goes into intrane.fr.