Edge NPU Cluster: Do Three 6-TOPS NPUs Really Make 18 TOPS?

Cheap NPU boards are increasingly easy to find. Buy three, wire them together, and you have 18 TOPS — right?

Nominal TOPS describes one device at its best moment. Nobody publishes what survives when you put three of them behind one scheduler. So we built the thing and measured it.

Three RK3576 boards (NanoPi R76S), 2.5 GbE, standard gRPC. No custom transport, no RDMA, no kernel bypass. YOLOv8n INT8. A scheduler and node agent written in Rust. Two weeks and 421 valid hardware measurements, with zero inference errors in the valid runs.

The headline: it scales. Three nodes reached 3.00× — 112.9 → 229.0 → 338.4 inf/s, tight variance over 30 runs.

That was the easy part. This post is about everything that happened next.

Why Won't It Scale?

What 421 measurements revealed about transport, tail latency, scheduling, and thermal heterogeneity.

1. Optimizing outside the operating point inverted our conclusion

We had to pick a connection count per node. So we fixed the load at concurrency 32 and compared one connection against four. The result was unambiguous: four connections made tail latency 46% worse. One connection it is.

Except something nagged. Was c32 actually a reasonable load?

We redrew the saturation curve. Everything from c24 to c64 was overload — throughput already flat, latency climbing. Overload for all three configurations under test.

We re-measured at the operating point (the lowest concurrency delivering 98% of peak, which turned out to be c12).

c32, fixed load (overload)    4 connections → tail  46% WORSE
c12, operating point          2 connections → tail  18.8% BETTER

The sign flipped. The first measurement wasn't wrong. It was exactly right — it just measured overload behaviour rather than a configuration effect.

We kept the overload data, relabelled it, and stopped using it for operating decisions.

Final operating point: 2 connections per node at concurrency 12 per node. Three nodes: 387.2 inf/s, +13.3% over baseline.

And then a second thing fell out. Scaling efficiency dropped from 98.9% to 95.3%.

Absolute throughput up, scaling efficiency down. They move in opposite directions. Quote either one alone and the trade-off disappears — so this project always reports both, with the measurement conditions attached.

2. The remaining loss appeared in the tail

95.3% efficiency at three nodes. Where did the missing 4.7% go?

We suspected the server first. CPU, memory, the 10G link, scheduler queueing — all ruled out. Scheduler queue wait measured 0.000 ms; routing, 0.004 ms.

Then we plotted latency against node count.

1 node → 3 nodes
  p50    85.95 → 85.65 ms    +0%     dead flat
  p95   118.79 → 153.64      +23%
  p99   136.83 → 197.75      +36%

The median stayed flat. Only p99 grew. Little's law confirmed it: the efficiency loss matches the mean-latency increase exactly. The loss is not in capacity. It is in the tail.

3. Same temperature, different results

Edge devices usually ship without fans, so we ran sustained load for 31 minutes under two cooling conditions.

active cooling   peak 387.7 → steady 380.3    −1.9%
fanless          peak 389.4 → steady 345.4   −11.3%

Expected. Fanless degrades more. What came next wasn't expected.

Under fanless load, one board — king — was running 2.4× slower than the other two. Thermal spread, obviously. Which means load-aware scheduling should be able to absorb it. So we ran a policy A/B.

First attempt: 1.33× spread. The 2.4× didn't reproduce. Probably wasn't hot enough — so we raised preheat to 25 minutes and kept the load continuous.

We got 1.10×. It went down.

At that point we stopped trusting the instrument. The harness was reading temperature after each run finished, and an RK3576 sheds heat in seconds once load stops. We re-aggregated from the 1-second thermal logger instead.

Experiment	Harness said	Actual peak under load	CPU clock p50	Latency spread
S0-A	85.9–86.8 ✓	86.8 / 85.9 / 86.8	1008 / 1800 / 1800	2.40×
2nd	78–79 ✗	86.8 / 85.9 / 86.8	1200 / 1800 / 1800	1.33×
4th	81 / 80 / 80 ✗	86.8 / 85.9 / 86.8	1416 / 1608 / 1608	1.10×

Peak temperature was nearly identical across all three — around 86 °C — while the spread ranged from 1.10× to 2.40×.

So temperature alone did not determine the heterogeneity. What tracked it was how far CPU throttling diverged between boards — and thermal control targets temperature, not divergence. When three boards throttle together, you get no heterogeneity at all.

Thermal conditions are necessary but not sufficient. And thermal control is not a deterministic way to reproduce the divergence you actually need.

"Heat it longer and it will reproduce" was never going to work.

So we grabbed the same lever thermal control uses: scaling_max_freq. Cap king's CPU down a ladder and measure the spread.

cap (MHz)   2208   1608   1200   1008    816    600
spread      1.12   1.18   1.33   1.79   2.26   3.93

A cap of 816 MHz reproduces S0-A to within 6 ms on all three nodes. No 30-minute preheat, no silicon lottery. Heterogeneity became a dial.

4. Our scheduler was herding on stale state

We switched on the load-aware policies (least-queue, ECT) and throughput collapsed by 55–58% — far worse than plain round-robin.

We nearly wrote "load-aware scheduling doesn't suit this workload."

But 55% is not the size of a quality difference. That is the size of a bug. So we read the implementation.

The state the policies scored against was only refreshed by heartbeats. Heartbeat interval: 1 second. Inference: 90 ms. By decision time the state was already stale, so every decision converged on the same "idle" node. Herding.

We replaced it with a locally-tracked in-flight counter.

p99                    232.0 → 146.9 ms   −37%
per-node latency spread  1.33× → 1.00×
CPU utilization spread  10.3pp → 3.1pp

This was a bug in our default configuration. Without the policy A/B we would never have found it.

When performance looks wrong, ask "is the implementation doing what I think?" before "is this approach bad?"

5. We were going to write io_uring. We measured first, and didn't.

The plan was written down: CPU profile → measure syscall and copy cost → buffer pool → io_uring. It was in the spec.

We did the first two.

At the operating point we split per-request node CPU into user and kernel time. Kernel time holds syscall entry, the TCP stack, and copy_to_user; user time holds protobuf serialization, user-space copies, and HTTP/2 framing. io_uring only reaches part of the kernel side.

transport cost          16.35 CPU-ms per request
  user    9.37 (57%)    serialization · user-space copy · HTTP/2
  kernel  6.99 (43%)    syscall entry · TCP stack · copy_to_user

network syscalls        ~165 per request
syscall entry cost      0.165 ms = 1.0% of transport cost
board CPU under load    48.9% idle, no core saturated

Even assuming registered buffers eliminate a 1.2 MB copy in both directions, the reachable slice totals about 8%.

And recovering all 8% would buy nothing, because the board is half idle.

CPU-ms per request is a cost, not a constraint. Reducing consumption of an unsaturated resource does not raise throughput.

We didn't build it. And we recorded that decision in the spec, not the commit log — because §15.3 of that spec already listed the escape conditions, written before any of this was measured: "io_uring may not be applied when: gRPC serialization is the larger bottleneck / the improvement is under 5%…"

Both conditions fired. A pre-registered exit criterion actually triggered.

The six times our instruments misled us

This is probably the part that outlives the numbers. All six are named in the repository's experiment ledger (§4.13), with how each one was caught.

Post-run temperature sampling read 5 °C low. The CSV column was named max_soc_c and was not a maximum. We built an explanation on top of it ("it wasn't hot enough") and the explanation was wrong.
A strace column swap. Our parser read usecs/call where calls was, making call counts look 100× too small. We nearly concluded "strace only attached to one thread, the test is invalid." The measurement was fine; the parser wasn't.
A percentage from the overload region. We had written the residual loss as 13.2% in several documents. That percentage came from a c32 (overload) measurement and had been paired with an operating-point throughput number. The real figure is 16.1%.
Two harnesses fighting. A harness we believed we'd stopped was still alive, hammering the same cluster as a new one at concurrency 36 each. The baseline read 197 inf/s (normal: 391) and the next run returned 82% errors. We came within one step of diagnosing a hardware failure.
An output path overwritten. Reusing a date-stamped results directory replaced a 15-run dataset with 4 rows. Git had it tracked, so we recovered.
A blind spot in our own secret scanner. The script that hunts for plaintext passwords missed one written in prose in a document. It had been built on the assumption that credentials leak through code — and this repository is mostly documentation.

All six have something in common: every one of them looked like success. A number came out. It was plausible. Nothing stopped.

When an instrument's output surprises you, suspect the instrument first. And note that moving a threshold to fit results and fixing an instrument that was measuring the wrong quantity are different acts. When you do the second, write down that the threshold stayed and only the source changed.

What we are not claiming

Better said up front than found in the comments.

Three nodes. Whether any of this holds at four or more is unmeasured.
Small repetition counts. Most configurations are 3–4 runs, which is why no throughput difference under 1% was ever used to rank anything.
Percentiles are run-level averages, not pooled. Each run's worst window gets diluted, so tail figures read low. Fine for comparing conditions; not citable as "the p99 of this system."
The residual 16.1% is unexplained. Local direct inference reaches 161.5 inf/s against 135.5 at the operating point. It looks like path latency rather than CPU cost — the 1.2 MB payload alone is ~8.2 ms round trip — but we did not pin it down.
No perf on these boards. The vendor kernel ships without it, so there is no symbol-level profile; the CPU split comes from /proc/PID/stat.

So, does 6 + 6 + 6 make 18?

On throughput, very nearly — 3.00×.

But nothing that stood in the way was bandwidth or nominal compute. It was the transport operating point, tail latency, clock divergence masquerading as a thermal problem, and a scheduler bug we had shipped ourselves.

And the one thing we were most confident about building — io_uring — turned out, once measured, to have no reason to exist here.

421 measurements, zero errors, all of it published: raw data, harnesses, the list of failures, and the conclusions we had to reverse.

Repository: https://github.com/seongkyounyoo/npudure
Experiment ledger: docs/experiments/README.md — what each experiment asked and what was ruled out under which conditions
Methodology: §4 of the same document

NPUDure is Apache-2.0. The RKNN Runtime and SDK are not included and must be installed from the vendor.