Embedded Agent Harness

2026 · 2× nRF52 DK · Zephyr · bare-metal nrfx rig · HIL benchmark for coding agents

Coding agents improve by running code and reading the result. That loop is cheap for web code: run the tests, read the stack trace, retry. For firmware it breaks — a bad register write doesn’t throw a stack trace, it corrupts RAM silently or drops a bus transaction that never happened. No log, no feedback, no improvement.

This is a hardware-in-the-loop bench built to close that loop. Two nRF52 DKs, wired together: one runs the agent’s firmware, the other runs a hand-built emulator that answers as a sensor over real wires at real speed. A daemon owns every instrument on the bench; the agent sandbox has no USB access and reaches hardware only through typed tools. Every flash, bus transaction, and fault gets logged to one timeline.

The sensors are invented, not real parts. Frontier models have seen thousands of real driver implementations during training, so grading against real parts measures memorization as much as skill — contamination. A datasheet I wrote forces the agent to actually read a spec. On the same chip and RTOS, IoT-SkillsBench found human-written context lifted a Zephyr driver task from 28/42 to 41/42, while model-authored context scored 27 — curated, failure-grounded context beats more context, and that’s the design brief below.

Why hardware-in-the-loop, not simulation

The rig doesn’t model a chip in software — it answers on real I2C and SPI lines at real clock speed, so EasyDMA, interrupts, and bus timing all get exercised instead of a mocked register read. A driver that passes against a software model can still fault against a real bus: clock stretching, a chip select held low a microsecond short, an interrupt arriving mid-transaction. Those failure modes only exist on wires.

System architecture

One daemon (hilbenchd) owns every instrument — both J-Links, the rig’s UART link, the logic analyzer, the switchable hub. The agent container has no USB passthrough and no vendor tools (nrfutil, JLinkExe); it reaches a board only through the daemon’s socket, so every guardrail lives in one place instead of scattered across tools an agent could route around.

Three trust zones, one daemon. Only hilbenchd crosses from software into hardware, so every action — a flash, a capture, a power cycle — is checked, serialized and traced in one place, whichever client asked for it.

The rig: a data-driven device emulator

Hand-written, no agent involvement — it’s the actual resume claim under the benchmark. One engine, fed a device description, answers on real I2C and SPI lines and logs every datasheet rule the DUT’s driver breaks.

Nordic’s slave peripherals set the constraints. TWIS clock-stretches automatically while the rig prepares a reply, so repeated-start register reads work naturally. SPIS is stricter: it takes a semaphore on CS-fall, and if the CPU still holds it the frame is ignored and a default byte clocks out — buffers only change between frames, never inside one. So the rig can’t answer “read register X” inside the frame that asked for it; every fictional SPI device is designed around that instead of fought with extra hardware — pipelined reads (frame N returns frame N−1’s answer), or two-frame reads with a minimum CS-high gap.

Each device is one YAML file — register map plus reusable quirk primitives (burst-only auto-increment, shadow-latched registers, power-up NACK windows, conversion delays, mode prerequisites, errata). Three generators turn it into the rig’s C tables, the datasheet, and a Python model used only by hidden tests — checked against each other by a differential test replaying the same transaction script through both.

device: TMPX-200
bus: {type: i2c, address: 0x48, max_hz: 400000}
power_up_ms: 5
registers:
  - {name: WHO_AM_I, addr: 0x0F, reset: 0xA5, access: ro}
  - {name: CTRL, addr: 0x10, reset: 0x00, access: rw, reserved_mask: 0xE0,
     fields: {ODR: [2, 0], BURST: [3], START: [4, self_clearing]}}
  - {name: DATA_H, addr: 0x02, access: ro, latch: DATA_L}
  - {name: DATA_L, addr: 0x03, access: ro}
quirks:
  auto_increment: burst_only
  conversion_ms: 8
  drdy_pin: {active: low, pulse_us: 50}
errata:
  - {id: E1, text: "First read after wake returns 0x00", enabled_by: scenario}

A GPIOTE→PPI→TIMER chain, no CPU in the path, times CS setup, CS-high gaps, and DRDY-to-first-read latency at 62.5 ns resolution — timing bugs get logged as a violation with a section number, same as protocol bugs.

The bench daemon: hermetic runs, leases, and a recovery ladder

Every run flashes, resets, and captures from boot — nothing depends on prior board state, so runs are replayable and safe to share across concurrent sessions. Sessions hold a lease, not a lock; a background task drains RTT continuously so the buffer never overflows between tool calls.

Recovery escalates through a fixed ladder: core reset, pin reset, reopen the probe, erase-all if the debug port is locked, float the rig’s outputs and power-cycle the DUT, full power-cycle plus bench self-test. Automatic steps are logged as events, not interventions — only the terminal “needs human” state records one, which keeps that metric honest rather than inflated by routine recoveries.

The only state a physical intervention gets recorded in. Everything above it is automatic and logged, but doesn't count against the "interventions per task" metric.

Agent tool API

About 15 top-level tools, defined once with a typed schema, generated into a hil CLI (JSON output, works with any shell-based agent) and a thin MCP adapter from the same registry. Every tool returns one envelope: a one-sentence summary, a token-budgeted data block, suggested next commands, and cost split into wall, hardware, and lease-queue time.

run is the workhorse — build, flash, reset, capture, decode, and triage in one call, one round trip instead of five.

{
  "ok": true, "tool": "run", "run_id": "r-0142",
  "summary": "Booted in 212 ms. 14 I2C transactions to 0x48, 1 rig protocol violation, no fault.",
  "bus": {"i2c": {"transactions": 14, "nacks": 0}},
  "rig": {"violations": [{"t_us": 211612, "code": "BURST_REQUIRED",
          "detail": "3-byte read from 0x02 without the BURST bit; device repeated register 0x02"}]},
  "fault": null,
  "triage": {"label": "real_bug", "confidence": 0.9, "by": "rules"},
  "next": ["hil docs register TMPX CTRL", "hil bus r-0142 --full"]
}

rig.violations is the highest-value line the harness produces: it turns “the data looks wrong” into “you broke this datasheet rule, here’s the section” — feedback no compiler warning gives.

Three tiers: reflex, System 1, System 2

Only deterministic checks — compiler, static analysis, rig, analyzer — can fail a build or a test. A small local model (System 1, on a Jetson) sits between that ground truth and the agent: it labels each run into a closed set (pass, real bug, regression, flaky, infrastructure, tool misuse), digests logs to the lines that matter, and matches failure signatures against precedent. It can’t turn a fail into a pass — a wrong label costs a turn, never a false pass — and has to beat a rules-only baseline or it gets cut.

System 1 must beat a rules-only baseline to earn a place in the loop. Most labels are already deterministic — probe errors are infrastructure, rerun disagreement is flaky — the model only handles what rules can't: novel failures, fuzzy signature matches.

Evaluation design

Hidden, hardware-verified grading on tasks no model could have memorized, enough repeats for honest error bars, and every hidden check traced back to something the agent could read.

Tasks, not runs, are the unit of analysis — three runs of one task aren’t independent samples; intervals bootstrap over tasks first, runs within tasks second, conditions compared paired on the same tasks.
Every task ships mutants — broken copies of the reference solution the grader must reject, plus a reference that must pass 10/10. A grader that can’t catch its own known-bad code produces numbers that mean nothing.
Visible tests inform, hidden tests decide. The hidden suite lives in a private repo never mounted into the agent’s container, runs against a clean checkout of the final commit, and is compared against the agent’s own submit claim to catch false success.
Debug tasks are seeded bugs in reference drivers — an ISR/main-loop race with no lock, a dropped volatile that only hangs at -O2, a DMA buffer in flash, a stack overflow from a shrunk thread stack — each exposed by a specific harness component (heartbeat detection, fault decoder, DMA pointer check, rig fault injection).
Interventions are logged, not answered. A per-task hint ladder serves one level of specificity per ask_human call, so help is reproducible and counted by category.

Planned v1 suite: 30 tasks across 8 fictional devices — 12 driver, 12 debug, 6 timing — against Claude Code headless and mini-swe-agent as a fixed scaffold for cross-model runs.