How I Made remotecmd 1.7× Faster Than scp Over a 77ms Relay

Two days of reading kernel source, CERN GridFTP papers, and gorilla/websocket internals. The result: parallel TCP streams, atomic relay framing, and gzip writer pooling. 50MB over 77ms RTT — 1.74× faster than single stream, 1.21× faster than scp.

remotecmd is a CLI that lets you run commands and copy files on remote machines — without SSH. It routes everything through a WebSocket relay: your laptop connects to the relay, the relay forwards to the target daemon. No port forwarding, no firewall holes, no SSH key management. It's how I manage 6 servers from anywhere.

The architecture is great for connectivity. It's terrible for throughput. A file transfer from my laptop to rbm21 (a server in Denmark) goes through dk1 (the relay in Germany). That's two hops, 77ms RTT, and a single TCP stream that can't fill the pipe.

A 50MB file took 125 seconds. scp did it in 87 seconds. I was slower than scp. That's unacceptable.

The bottleneck: one stream can't fill a fat pipe

TCP throughput is bounded by the bandwidth-delay product (BDP). On a 77ms RTT link with ~10 Mbps available bandwidth, the BDP is roughly 96 KB. TCP's congestion window needs to grow to that size before it can saturate the link, and a single stream does this slowly — especially through a relay that adds latency.

The solution is well-known in the HPC community: parallel TCP streams. CERN's GridFTP has supported this since the early 2000s. Their research shows linear throughput gains from 2 to 10 streams, with diminishing returns beyond. The idea is simple: open N connections, split the file into interleaved chunks, send them concurrently. Each stream has its own congestion window, so together they fill the pipe faster.

I needed to implement this for remotecmd. But remotecmd isn't raw TCP — it's WebSocket frames through a relay. That made everything harder.

The protocol: interleaved chunks over N WebSocket connections

The client opens N WebSocket connections to the relay. Each stream sends interleaved chunks: stream 0 gets chunks 0, N, 2N...; stream 1 gets chunks 1, N+1, 2N+1... All streams use the same transfer ID, so the daemon knows they belong to the same file.

The daemon pre-allocates the file with Truncate(TotalSize), then each chunk is written at its correct offset using Seek(seq * chunkSize). No reassembly needed — chunks land directly where they belong.

Two new fields in the protocol: ChunkSizeBytes and ParallelStreams. The relay forwards them in the file_transfer message so the daemon knows to expect parallel writes.

The relay bug: interleaved headers and binary data

Here's where it got messy. The relay has an async write queue — it reads a frame from the client, enqueues it, and immediately reads the next frame. This is great for pipelining but catastrophic for parallel streams.

Each file chunk is two WebSocket frames: a JSON header (text) and the binary payload. With a single stream, they arrive in order: header, binary, header, binary. With parallel streams sharing the same relay-to-daemon connection, they can interleave: header from stream 1, header from stream 2, binary from stream 1, binary from stream 2. The daemon sees two headers in a row and breaks.

The fix: when the relay receives a file_chunk header with BinaryChunk=true, it immediately reads the next frame (the binary payload) on the same connection, then forwards both as a single atomic unit through the write queue. A new method sendRawHeaderAndBinary enqueues a length-prefixed pair that the writer goroutine sends back-to-back. No interleaving possible.

The research: what worked and what didn't

I did an intensive web search across four domains. Here's what I found:

What worked

Parallel streams (the big win). CERN's GridFTP research confirmed that 2-10 streams gives linear throughput improvement. I auto-tune based on file size: 5-20 MiB gets 2 streams, 20-100 MiB gets 3, 100+ MiB gets 4. Override with RCMD_PARALLEL_STREAMS=N.

Gzip writer pooling. gzip.Writer is ~40KB per allocation. Creating one per chunk causes GC pressure. Pooled with sync.Pool + Reset(), compressible data transfers got smoother.

Larger write queue. The relay's async write queue went from 64 to 128 frames. With 4 parallel streams × 2MB chunks, that's 8MB in flight — the deeper queue keeps the pipeline full.

What didn't work

sendfile / splice. Go's io.Copy automatically uses sendfile(2) for file-to-socket copies. But our relay uses WebSocket — each chunk needs a frame header, so the data goes through userspace. Can't zero-copy around framing.

TCP_CORK. The Linux man page says TCP_CORK coalesces small writes into full TCP segments. But we can't access the raw socket — gorilla/websocket wraps it. The kernel's tcp_autocorking (enabled by default since 3.14) already does this for us.

Disabling WebSocket masking. RFC 6455 requires client-to-server masking. gorilla/websocket already uses 64-bit XOR (2 GB/s on modern CPUs). At our throughput (~900 KB/s), masking is not the bottleneck.

klauspost/compress. 1.5-2× faster than stdlib gzip. But we already use BestSpeed (level 1), and a sample-based heuristic skips incompressible data before allocating a writer. The marginal gain wasn't worth the dependency.

The numbers

50MB random file, laptop → dk1 relay → rbm21 (Denmark), 77ms RTT:

1.74× faster than single stream. 1.21× faster than scp. All transfers verified with SHA256 checksums — zero corruption.

For compressible data (text, logs, zeros), it's even better. A 10MB file of zeros transfers in 465ms — 22 MB/s — because gzip level 1 compresses it to almost nothing and the sample heuristic kicks in immediately.

Why this matters

remotecmd is designed for agent-first infrastructure. When an AI agent needs to deploy a binary to 6 servers, it doesn't have SSH keys. It doesn't know about port forwarding. It uses remotecmd cp and the relay handles the routing.

But if that transfer takes 2 minutes per server, the agent's workflow stalls. At 72 seconds for 50MB, it's fast enough to be invisible. At 125 seconds, it's a bottleneck.

The parallel streams are automatic. No flags, no configuration. The CLI detects the file size, picks the right stream count, and opens the connections. The user — or the agent — just runs remotecmd cp --target rbm21 --src ./app --dst /opt/app and it's fast.

What's next

The obvious next step is adaptive stream tuning — measure the actual throughput during the first few chunks and adjust the stream count dynamically. If 2 streams already saturate the link, don't open 4. If 4 isn't enough, try 6. The CERN research shows this is feasible with as few as 3 prediction points.

I'm also looking at permessage-deflate — WebSocket's built-in compression extension. It would replace our manual gzip with something the library handles natively, including context takeover for better ratios on similar chunks.

But the core lesson is simpler: read the research, measure everything, and don't assume the obvious optimization is the right one. I spent half a day trying to make TCP_CORK work before realizing the kernel already does it. I almost added a compression library dependency before realizing the sample heuristic already skips incompressible data. The parallel streams — the actual win — came from a 2005 GridFTP paper.

remotecmd is open source. The parallel streams shipped in v2.3.0. Try it: remotecmd-cli update if you already have it, or grab the binary from the releases page.