Open-source Edge NPU Cluster
6 TOPS × 3. Does it really become 18 TOPS?
Open-source Edge NPU Cluster runtime for scaling distributed AI inference across low-cost NPUs over standard Ethernet. Written in Rust, speaking plain gRPC — no custom transport, no RDMA, no kernel bypass.
An Edge NPU Cluster is several cheap NPU boards, each holding a full copy of the model, behind one scheduler that hands each incoming request to one of them. It raises how many requests per second you can serve. It does not merge the boards into one larger accelerator, and it does not make any single request faster.
To find out where scale-out performance actually goes, we ran 421 valid hardware measurements with zero inference errors in the valid runs. Short answer: it scales. The interesting part is everything that almost stopped it.
| Hardware | 3 × NanoPi R76S — Rockchip RK3576, 6 TOPS NPU each |
|---|---|
| Network | 2.5 GbE per node; scheduler host on 10 GbE |
| Transport | gRPC over HTTP/2 (tonic), plaintext |
| Runtime | Rust |
| Workload | YOLOv8n INT8 through RKNN, 640×640×3 RGB |
| Measurements | 421 valid benchmark runs, zero inference errors |
| 1 → 3 nodes | 112.9 → 229.0 → 338.4 inf/s — 3.00× |
| Best throughput | 387.2 inf/s with the transport tuned — scaling efficiency falls 98.9% → 95.3% |
| Tested scale | 1, 2 and 3 nodes |
| 4+ nodes | not measured |
Each figure with the conditions it was measured under: Evidence.
Not literally. NPUDure does not combine three NPUs into one 18-TOPS accelerator. It distributes independent inference requests across multiple Edge NPU nodes. In the measured configuration, three nodes delivered 3.00× aggregate inference throughput compared with one node.
Absolute throughput and scaling efficiency move in opposite directions. Tuning the transport bought 13.3% more throughput and cost 3.6 points of scaling efficiency. Quoting either number alone hides the trade-off, so both are always reported here, with the measurement conditions attached.
Six findings almost stopped the scale-out — transport tuning, tail latency, sustained thermal behaviour, a herding bug in our own scheduler defaults, heterogeneity that temperature did not explain, and an io_uring implementation the measurements talked us out of.
These limits are published rather than buried. A result you cannot bound is a result you cannot cite.
Several low-cost edge NPU boards, each running a full copy of the model, fronted by a scheduler that spreads independent inference requests across them over ordinary Ethernet.
For throughput, yes, and it works well. For a single request, no.
No. But on this configuration the throughput scaling was 3.00×, which is closer to the naive sum than we expected.
This site explains why, what and the result. The repository holds how, the evidence and the code — raw CSV, the benchmark harness, the full experiment log and configuration all stay there.
Source code Experiment ledger Raw data Benchmark results Photos of the rig