I benchmarked my language against Rust and Zig, and deleted my best number

I have been building machin for a while now — a Go-flavored, type-inferred language that compiles through C to a single native binary. It has grown a lot in the last few weeks, and I wanted to answer the obvious question honestly: does it beat Rust and Zig at anything?

It does, at two things, decisively. But the first thing I found was not a win. It was my own benchmark quietly lying to me, and the number it was lying about was the best one I had.

The benchmark was measuring the order I ran things in

machin's repo has had a bench/native-speed suite for months: four compute kernels — recursive fib, a mandelbrot, a sieve, a big integer loop — written in machin, Rust and Zig, producing byte-identical output, so the timing compares the same computation three ways. The published result claimed machin won the integer loop by 20–25%. That claim also shipped inside machin guide, which is what every coding agent reads to learn the language.

When I re-ran it, the margin was gone. Not shrunk — gone. So I read the harness instead of the output, and there it was:

for kernel in kernels:
    for lang in [machin, rust, zig]:
        for _ in range(5):        # all 5 machin runs, THEN all 5 rust, THEN all 5 zig
            time(binary)

It ran every sample of one language before starting the next. On a laptop that heats up and down-clocks during a three-second kernel, that does not measure the languages. It measures who had the misfortune of running last. Zig always went last. Zig always looked slowest.

The fix is four lines — interleave the rounds and rotate who starts each one, so thermal drift is spread evenly across all three. Here is what my headline number did:

intsum 10^9      before (blocked)     after (interleaved)
machin              2832 ms                3079.7 ms
rust                3764 ms                3223.8 ms
zig                 3556 ms                3189.7 ms
                 "machin +20-25%"        machin +3% = a TIE

A 20–25% win became a tie. I deleted the claim from the README and from machin guide. If you had asked me the day before, I would have told you machin beat Rust on integer loops, and I would have been repeating an artifact of the order a Python loop happened to be nested in.

The harness now also refuses to declare a winner inside a 3% band, because the worst run-to-run spread I measured on this machine was 41% of the min sample. Calling winners inside that is how benchmarks start lying in the first place.

So what does machin actually win?

Two things, and neither of them is raw speed. On the four kernels machin wins one clearly (recursion, 26% faster than both), ties two, and loses one. It is in the same tier as Rust and Zig, which is the honest framing — it compiles to C, so it runs about as fast as C, and so do they.

1. It tells you about the bug before you run it.

Here is a program that waits for a value that can never arrive. In machin it is a receive on a channel nothing sends to; in Rust it is rx.recv() with the sender still alive; in Zig it is sem_wait on a semaphore nobody posts. Same program, three languages:

machin   DL001 at COMPILE time: "receive on channel `ch` that is never
         sent to or closed - a guaranteed deadlock"
         ...and at runtime, exit 2 with the wait-cycle:
           fatal: deadlock - all 1 goroutine(s) blocked
             goroutine 0   waiting to receive on channel #0

rust     compiles clean, no diagnostic -> HANGS FOREVER (killed at 5s)
zig      compiles clean, no diagnostic -> HANGS FOREVER (killed at 5s)

This one deserves precision, because it is easy to overclaim. Rust's type system prevents data races. It has never claimed to prevent deadlocks, and that recv() is idiomatic, unsafe-free, perfectly well-typed Rust. Zig does not attempt either. In both, the failure mode is a process that hangs until something outside it notices.

The same holds for an out-of-range index. machin falsify enumerates small concrete inputs and hands back one that breaks the function — before the program runs, on code whose only call site is perfectly in range. rustc and zig say nothing at all.

Neither analysis proves absence, and I do not want to imply otherwise. falsify is unsound-complete: every counterexample it reports is real, but a clean result means "no bug within the bounds", never "correct". DL001 is the opposite trade — sound and false-positive-free, so it only fires when it can prove a channel is never fed, which means a clean result is not a proof of deadlock-freedom either. "machin found nothing" is a much weaker statement than "machin found this bug, here is the input".

2. Binary size.

stripped, both dynamically linked against system libc:

  machin      14 kB
  rust       335 kB

There is no std runtime to link — machin's output is C, and C's runtime is already on the machine. I want to flag the comparison I did not make: unstripped it is 17 kB vs 4291 kB, about 250x. That number is much bigger and much worse, because it mostly measures how much debug info each toolchain leaves in by default.

And here is the correction I owe the 24x itself, which I only caught by going back and asking what it actually measures. It is a comparison of each toolchain's fixed floor, not evidence that machin scales better:

program            machin        rust
hello world       14,544 B    343,568 B
fib(40)           14,544 B    335,472 B
a JSON+HTTP app   26,840 B          -

machin's hello world and its fib are byte-identical in size, and Rust's differ by 2%. Neither number is measuring the program; both measure the baseline each toolchain links in. Real code adds real bytes to both. So the honest form is "Rust starts about 320 kB ahead", not "machin binaries are 24x smaller" — the ratio shrinks as programs grow, the offset is what persists.

And what it loses

A benchmark suite that only lists wins is marketing. Three losses, stated as plainly as the wins:

• Build time: Rust wins. Bare rustc -C opt-level=3 builds these kernels in ~57 ms; machin takes ~95–116 ms, because its number includes the cc -O2 backend run. I deliberately did not use cargo, which would have charged Rust for lockfile resolution and made machin look good for a bad reason.
• Default runtime safety: Rust wins. Given an out-of-range index, Rust traps (exit 101). machin's default build prints a silent wrong 0 and exits successfully — exactly like Zig's ReleaseFast, which read 281479271677952 out of adjacent memory. Both need an opt-in to trap.
• Fully static, Zig wins — by far more than I first reported. I originally wrote "491 kB against machin's 940 kB", about 2x. That 491 kB came from Zig 0.16.0, which inflates it ~32x. On 0.15.2 the fully static stripped binary is 16 kB — about the size of machin's dynamic one, needing nothing on the target — and ~60x smaller than machin's own --static build. The size win in this post is over Rust, not over Zig.

That second one matters enough to repeat, because I caught myself getting it wrong. My first version of that benchmark compared machin --safe against Zig ReleaseFast — machin's checked mode against Zig's unchecked one — and machin looked safe by default. It is not. I rewrote the table to show both modes for both languages.

The sieve: three months of a wrong explanation

machin has always trailed on the sieve kernel by about 1.4x, and the README explained it confidently: "its slice indexing/layout is less optimal than a Rust Vec or a Zig slice." That explanation was wrong, and nobody had checked it because the conclusion sounded plausible.

Timing the phases separately took about ten minutes:

phase                              machin      rust
build the 10M array by append     70-83 ms   27-29 ms
the sieve loop itself              110 ms     111 ms   <- dead tie
the count loop                     5-7 ms       3 ms

Slice indexing ties Rust exactly. The generated C for the hot loop is already what you would write by hand. The entire gap is append growing the array: machin's arenas free nothing mid-life, so growing a slice can only ever allocate a fresh block and memcpy into it, ~21 times, never releasing the old buffers. Vec::push hands the block to realloc, and glibc extends it in place via mremap without copying anything.

So I wrote the obvious fix: when the block being grown is the arena's most recent allocation, hand it straight to realloc. Then I checked one thing before shipping it — whether MFL slices share backing storage:

a := []int{1, 2, 3}
b := a          // b[0] changes when a[0] does -> they SHARE
mutate(c)       // params share too

They do. Today, when append abandons a block, every existing alias keeps pointing at it and keeps reading valid memory, because the arena never frees. With in-place growth, those aliases become use-after-free the moment the block moves. I had written a 45 ms speedup that trades a benchmark number for silent dangling reads — in a language whose entire pitch is catching exactly that class of bug.

I threw it away and filed issue #578 with the three directions that could actually be sound. That fix has since landed — see the update at the end.

What I would tell you the claim is

Not "machin is faster than Rust" — it is in the same tier, and it loses a kernel. Not "machin is safer than Rust" — Rust traps by default and machin does not.

The claim is machin tells you earlier. A deadlock that Rust and Zig discover as a hung process in production, machin reports at compile time with the wait-cycle. An out-of-range index that neither mentions, machin hands you with a concrete failing input before you run. That is a real difference, it is worth real money in debugging time, and it is a different claim from Rust's rather than a bigger one.

Postscript: I ran it all on a second machine

Every number above came from one laptop with 41% worst-case run-to-run spread, and I had told readers that "the ratios are the portable result" without ever testing that. So I wired the suite into CI and ran it on a GitHub runner — AMD EPYC, different compilers, much quieter (15% spread).

The verdicts are portable. machin still wins recursion (26% → 28%), mandelbrot and intsum are still ties, the sieve was still a loss at the time of writing (since fixed — see the update below), and every compile-time result — DL001, FALS001, who hangs, who traps — reproduces exactly.

The precise ratios are not. The sieve gap moved from 1.46x to 1.32x, a ~10% swing — which fits its cause: it is append faulting in fresh pages, the most memory-sensitive thing in the suite, and the EPYC has a better memory subsystem. So read a verdict as portable and a precise ratio as machine-specific.

It also caught the Zig size error above, which is the entire reason for running it.

Every number here is reproducible: the sources for all three languages, the harnesses, and a run.sh per benchmark are in the repo. Start at docs/BENCHMARKS.md, which indexes all seven benchmarks and states the losses next to the wins. The language itself is at github.com/javimosch/machin.

If you re-run them and get different numbers, I would genuinely like to know. That is rather the point of shipping the harness.

Update: the sieve now ties Rust

Everything above describes the sieve as machin's remaining loss, and explains it correctly as append rather than indexing. That gap is now closed, and the interesting part is not the speedup.

The unsound version — hand the arena's newest block to realloc — was the easy 45 ms. Making it safe meant answering one question per slice: does any other live reference observe this backing array?

The design decision that mattered was the failure direction. The tempting implementation is a dynamic "shared" bit on the slice header, set wherever the compiler copies one. Less code — but it fails open: a copy site nobody thought of leaves an alias unmarked, and an unmarked alias is silent memory corruption. So instead: a static whitelist where every operation on a candidate must be a form that provably cannot leak a reference, and anything unrecognised refuses the slice. The cost of a case nobody thought of becomes a missed optimization instead of a dangling pointer.

I shipped that analysis first as a report — machin alias — that changed no generated code at all. Which is how I found out my first version was useless:

module          flow-insensitive   flow-sensitive
bson.src               ~0             2 of 2
xml.src                ~0             5 of 10
reactive.src           ~0             3 of 10

It was sound, and it fired on almost nothing real. "Build it, then use it" is the dominant shape — append in a loop, then hand the finished slice to a helper — and that aliasing use happens after the last append, where it cannot observe a moved array. Had I gone straight to codegen I would have shipped something that optimized my own benchmark and essentially nothing else.

Then the net, before the thing it catches: seven programs that keep a live alias across an append, run under AddressSanitizer in CI. I verified it catches the bug by re-applying the unsound version and watching it go red — rather than assuming a green suite meant anything.

Only then the codegen. Two guards the static proof cannot express: growth is attempted only on the arena's most recent block, and the substr length cache is invalidated on that path, because it keys on pointer identity and realloc hands an address back to the allocator. Missing that second one would have been a silent wrong-length bug in substr() — about as far from append as a bug can land.

Measured on a CI runner rather than my laptop (which was at 58-68% run-to-run noise by then), normalized against Rust because the runner itself drifted ~15% between sessions:

              machin      rust       zig    machin/rust
  before      96.7ms    87.7ms    73.0ms       1.10x
  after      101.9ms   100.6ms    86.0ms       1.01x

The sieve ties Rust. It does not tie Zig, which is still ~1.19x faster on this kernel — down from 1.32x, but a real remaining gap, and I would rather report it than round it into the headline. Shipped in v0.128.0.