A 7-billion-parameter LLM, at the speed of a 1B — in a language you've never heard of

A few weeks ago I set myself a challenge: write an LLM inference engine in machin (MFL, a machine-first language I've been building) with zero dependencies — no PyTorch, no llama.cpp, no BLAS, no Python at runtime. Just the language and a static binary. I got a dense 1B model running at 20+ tokens/sec on a laptop CPU, verified token-for-token against the reference C. That felt like the ceiling.

It was — for a dense model. And that ceiling turned out to be the most interesting part.

The wall nobody tells you about

On weak hardware, a dense LLM's decode speed is set by exactly one number: bytes moved per token. Every weight is read for every token. A 1B int8 model moves ~1.1 GB per token, and on my box's memory bus that caps decode at ~20 tok/s — no matter how clever the code.

So I tried all the clever code. Speculative decoding. Contextual sparsity. Adaptive early-exit. int4 weights. Continuous batching. I built and measured every one of them honestly. Every single technique topped out at about 1.35×, because the box is balanced: the moment you save memory bandwidth, you become compute-bound, and vice-versa. There is no free lunch when both resources are equally scarce.

The conclusion was uncomfortable but clear: the disruption isn't in the engine. It's in the model.

Mixture-of-Experts changes the math

The project is named after colibrì, an engine that runs a 744-billion-parameter model on a consumer PC by streaming experts from disk. I finally sat down and read its code, and the insight clicked.

A dense model prices every token at its total parameters. A Mixture-of-Experts model decouples quality from speed: only a few "expert" sub-networks fire per token. OLMoE-1B-7B is 6.9 billion parameters total, but only 1.3 billion active per token — it picks 8 of 64 experts at each of its 16 layers. So you get 7B-class quality at roughly the cost of the 1B I'd already proven.

And because only a handful of experts fire, the cold ones don't need to live in RAM. You mmap the checkpoint and let the OS page cache stream the routed experts in on demand. Total model size becomes bounded by your disk, not your memory.

So I built it. In pure MFL.

Same engine, same integer-matmul kernels, same memory-mapped streaming. I added the MoE routing, the OLMoE attention (with its qk-normalization and rotate-half RoPE), and — the fiddly part — the byte-level BPE tokenizer, all in machin.

Then I did the thing I care about most: I wrote a fp32 reference in numpy, reading the original weights directly, and checked my engine against it token for token.

numpy fp32: The capital of France is Paris. The capital of the United States is Washington
pure MFL (int8): The capital of France is Paris. The capital of the United States is Washington

12 out of 12 tokens identical. Quantization didn't flip a single one.

A 7-billion-parameter Mixture-of-Experts model, running in a language most people have never heard of, producing exactly the same output as the fp32 reference — with the cold experts faulting in from disk as they're needed.

The numbers

Everything below is measured on a laptop CPU (8 cores), warm cache, every config token-identical to fp32:

Two independent levers fell out of it. Quantizing the output projection to int8 buys real speed (9.4 → 14.5), because decode is memory-bound and it's a big read. int4 experts halve the footprint to 4.4 GB — the same balanced-roofline wall makes them slightly slower, but they let the whole thing fit a smaller box. Pick your tradeoff; it's one flag.

It speaks OpenAI

Finally I wrapped it in an OpenAI-compatible server — including the tokenizer, in pure MFL — and pointed the official openai Python client at it:

>>> client.chat.completions.create(model="olmoe-1b-7b",
...   messages=[{"role":"user","content":"Water is made of"}])
"Hydrogen is a gas. Oxygen is a..."

7B-class quality, at ~1B speed, streamed onto hardware you already own, served over a standard API — with no numeric libraries anywhere in the stack.

Why this matters to me

The point was never to beat a datacenter. It was to show that a from-scratch engine, in a language I built, can reach the actual hardware roofline — and then to find the one lever that clears it. That lever wasn't a trick in the code. It was choosing a model whose architecture spends its parameters wisely, and a runtime honest enough to stream the rest from disk.

It's all open source: the engine, the converters, the tokenizer, the honest write-up of every dead end. github.com/javimosch/machin-colibri. And the language underneath it, machin, is where these integer kernels and the mmap streaming primitive all landed as first-class features.

Small engine. Immense model. Hardware you own. That's the whole idea.