NPUDure
The six that changed what we believed. Each shows the question asked, the scale it was measured at, the result, and a link to the full report.
Total measurement runs: 421 (418 bench + 3 profile conditions), error rate 0
Every question, scale and result below is imported from
docs/experiments/README.md in the NPUDure repository
and rendered here. The repository is the source of truth; this page follows it.
The ledger holds 20 entries in all, including the ones that returned nothing. Those are worth reading too — a null result is still a bound.
Scale-out
| S2 | Does adding nodes scale linearly 30 runs |
112.9 / 229.0 / 338.4 inf/s, speedup 3.00×, eff 100%, 0 errors S2 |
|---|
The baseline question. Adding nodes multiplied throughput almost exactly, which is the result the rest of the work had to defend.
Transport tuning
| S3.8 | Does the optimization hurt scale-out 36 runs |
135.5 / 263.3 / 387.2, 3N 2.86× (95.3%). Absolute +13.3% but eff 98.9→95.3% S3.8 |
|---|
Faster in absolute terms, worse at scaling. This is why both numbers are always reported together here.
Tail latency
| S3.9a | Where does 3N's 4.5% efficiency loss arise 9 runs |
Server resources all ruled out. The loss shows up in the tail (p50 flat, p99 +36%). TCP retransmits 3.5× — though the micro-mechanism was not isolated S3.9a |
|---|
The missing efficiency was not in the median. Anyone sizing a cluster on p50 would never see this cost.
Scheduler herding
| S0-C 1st | Does a load-aware policy recover the thermal-heterogeneity loss 15 runs |
The policies collapse throughput by 55–58%. Cause: herding on stale heartbeat state — a scheduler bug, found S0-C |
|---|---|---|
| S0-C 2nd | Re-measure after fixing that bug 12 runs |
RR 373.9 / LQ 380.9 / ECT 384.2. Collapse gone, p99 −37%, node latency spread 1.33×→1.00× S0-C §8–11 |
A load-aware policy looked catastrophically bad. It was our own default configuration deciding on stale heartbeat state — the policy was fine once the bug was gone.
Thermal heterogeneity
| S0-A | What happens fanless 30 runs / 32 min |
degradation 11.3%. CPU 2208→816 MHz (king), NPU pinned at 950. The king is 2.4× slower and round-robin still sends it 1/3 S0 |
|---|---|---|
| S0-C 4th | LQ vs ECT under strong heterogeneity (2.4×) 1 run (aborted) |
Gate missed at 1.10×. Thermal conditions were identical (86.8 °C) — what sets heterogeneity is not temperature but the spread in CPU downgrade S0-C §17–19 |
| S0-D calibration | Can heterogeneity be produced deterministically 12 runs |
Yes. Cap 2208→600 moves the spread 1.12×→3.93×. Cap 816 reproduces S0-A (2.4×) to within 6 ms S0-D |
The obvious explanation was temperature, and it was wrong. What tracked the spread was how far clock throttling diverged between boards, which a fixture can now reproduce on demand.
The io_uring decision
| S3.9b | Do node-side syscalls/copies matter in the residual gap 4 conditions |
Not syscalls — ~1% of transport cost (8% being generous). User time exceeds kernel time (9.37 vs 6.99 ms/req). CPU is 48.9% idle = not a constraint. → S4 io_uring cancelled/shelved S3.9b |
|---|
Planned, profiled, then not built. Reducing consumption of a resource that is 48.9% idle does not raise throughput.
Every experiment, what it asked, what was ruled out and under which conditions, plus the raw dataset mapped to each one.