Benchmarks that can hurt you

Duet's headline number is good: 240 ms median response handoff against a cascaded baseline's 1,880 ms. Roughly 8×. If I stopped there, it would look like a launch tweet.

The same table shows Duet's takeover rate is worse — 0.24 against the cascade's 0.00 — and its p95 tail is slower. The full-duplex model that responds beautifully also, sometimes, barges in when it shouldn't. Both facts went in the published results with equal typographic weight.

The mechanism is simple: a benchmark you design after you know your system's strengths is a mirror, not a measurement. The only defense is deciding the metrics before you run anything, and committing to publish whatever comes out. I wrote the metric definitions — including takeover rate, which I suspected Duet would lose — before the first run.

What this buys you:

• Trust that compounds. Anyone who checks the losing numbers once will believe your winning numbers forever.
• A real roadmap. Duet's worse takeover rate isn't an embarrassment; it's the next quarter's work, precisely specified, for free.
• Immunity to your own marketing. The moment your internal numbers and public numbers diverge, you start believing the public ones. That's how products rot.

If your benchmark can't hurt you, it can't help you either.