I Built a Security Auditor for AI Agents. It Has No AI Inside.

I shipped a new open-source tool this week: machin-secure — a KISS, agent-first security auditor. It scans a codebase for secrets, injection, weak crypto, and a dozen other CWE-tagged classes of bug, and streams the findings as JSON Lines. The interesting part isn't the scanner. It's everything I didn't put in it.

The starting point: strix, minus almost everything

Strix is an open-source AI pentesting agent: Docker sandboxes, Playwright browser automation, a Caido proxy, an LLM tool-loop driving the whole scan. It's a serious piece of engineering. I wanted the defensive core of that idea — a tool that finds real vulnerabilities in a codebase — without any of the infrastructure.

My own 15-lines-of-bash-beat-my-3,000-line-search-engine post from a few weeks back is the actual blueprint here: a crude, deterministic pass over the filesystem catches most of what a sophisticated agent catches, with none of the moving parts. So the plan was: build the 15-line version first, and see how far it gets.

It got pretty far. rules.json has 46 CWE-tagged regex detections — hardcoded secrets, eval()/os.system()/pickle.loads, SQL built by string concatenation, TLS verification disabled, cookies without HttpOnly — across Python, JS/TS/Vue, Go, PHP, Ruby, Java, YAML, Dockerfiles. The engine that walks the tree and applies them is about 250 lines of MFL, compiled to one ~50 KB static binary. No index, no database, no run journal — the filesystem gets scanned fresh every time, because freshness beats sophistication.

./secure --target ./some/repo | jq 'select(.severity=="critical")'

Run against a real 10,000-file production monorepo, it found a hardcoded GitHub personal access token and private-key material sitting in an .env.*.bak file. Real findings, zero LLM calls, zero cloud dependency, zero attack surface beyond "reads your files."

The decision that actually matters: no LLM client

The obvious next step for a tool like this is a false-positive filter: run the raw findings through an LLM, have it classify keep/drop. I started designing exactly that — a provider-neutral OpenAI/OpenRouter adapter, a budget, retries, redaction.

Then I asked myself the question I should have asked first: who is running this tool? Not a human clicking a dashboard. An AI coding agent — Devin, Claude Code, whatever's driving the terminal. That agent already is an LLM. Bolting a second model client onto a binary that's only ever invoked by a first model is redundant in the most literal sense.

I'd already solved this exact problem in grepapi, my lead-gen tool: it returns a structured "brief" from /v1/brief, and the operator's own LLM completes the copy. grepapi never runs or bills a model completion itself. Same shape, different domain:

• Every finding gets a stable id: sha256(rule|file|line).
• The calling agent reads the JSONL with its own reasoning and decides what's a false positive.
• It persists that judgment: secure verdict <id> drop --reason "Vue prop binding, not a secret" (or batches many over stdin).
• Future scans auto-suppress dropped findings and report a suppressed count. --show-all brings them back.

secure holds no API key, makes no network call to any model provider, and has zero opinion about which model the agent uses. The verdict store is the agent's own memory, not a cache the tool invented — a plain JSON file the tool doesn't even read except to look itself up.

Reports get the same treatment

I'd also wired up an opt-in --hart flag that generated a deterministic HTML table from the findings and published it to hart, my self-hosted artifact host. It worked. It was also the same mistake in a different shape: authoring a report is judgment — what to highlight, how to group it, what to say about it — and I was baking that judgment into the binary instead of leaving it to the model already in the loop.

So I tore it out. --hart now just prints a hint: the default instance, the publish contract, a pointer to hart.intrane.fr/llms.txt, and the current severity counts. The agent writes its own HTML and publishes it with hart publish. The tool teaches; it doesn't perform.

Proof: a fully non-interactive run

To check this actually holds together end-to-end, I ran the whole thing headless:

devin -p "Run machin-secure against this repo, read the hart hint,
author your own HTML report, and publish it." --permission-mode dangerous

No human in the loop. The sub-agent built the binary, ran the scan, read the guidance line, wrote its own report grouped by file with severity counts, and published it to hart.intrane.fr/a/machin-secure/smoketest — a better-structured report than my original deterministic table, because it came from an agent actually reasoning about the data instead of a template dumping it.

What's in the repo

• 46 rules across secrets, command injection, deserialization, XSS, weak crypto, TLS bypass, SQL injection, path traversal — plain data in rules.json, read fresh every run, extendable without recompiling.
• A verdict loop instead of an LLM filter — the agent's own triage, persisted, and remembered.
• A hint, not a report generator, for publishing to hart.
• One static binary, no Docker, no browser automation, no code execution on the target.

Code's on GitHub, MIT licensed, listed in awesome-machin. If you're building tools that an AI agent will drive, the question I'd ask before adding an LLM API client to anything: is the caller already a model? If so, you probably don't need a second one inside.