The problem with queues
Most inference servers are built around a queue. Requests arrive, wait for a batch to fill, run, and leave. That shape is right for a data center, where throughput pays the bills. It is wrong for a camera. A camera does not send requests. It sends a frame every 7.8 ms at 128 fps, and the only answer that matters is the one that arrives before the next frame.
When a frame waits in a queue, the delay compounds. One slow model pushes the next frame back, which pushes the one after it, and within a second the system is reacting to the past. On a robot moving at 2 m/s, a 200 ms lag means 40 cm of travel nobody saw.
Budgets, not turns
The Runtime replaces the queue with a frame budget. Before a graph runs, the compiler measures every node on the target chip and assigns it a slice of the budget. On a Hexagon NPU, Depth-S might get 6.4 ms, Depth-Track 1.2 ms and the policy check 0.2 ms. What is left of the 8 ms is slack, held back for thermal throttling and memory contention.
A frame that cannot finish in time is dropped on purpose, not delayed by accident.
If a node overruns its slice, the scheduler does not wait. It hands the policy check the last good result for that node and marks it stale. Downstream code always gets an answer on time, and it always knows how fresh each part of that answer is.
Compiling the schedule per chip
A schedule that is fast on a Jetson GPU can be slow on a Hexagon NPU, because the two chips disagree about what is cheap. The GPU likes wide kernels and tolerates memory traffic. The NPU wants tensors that fit in on-chip memory and punishes every round trip to DRAM. So we compile one schedule per chip, not one per framework.
- Operators are fused until the working set fits in on-chip memory.
- Pre-processing runs on the image processor where the chip allows a zero-copy hand-off.
- Nodes that do not depend on each other are placed on separate compute units and overlap.
What we measured
On a Qualcomm QCS8550 running Depth-S, Depth-Track and a zone policy, the median frame-to-decision time is 7.8 ms, with a p99 of 9.6 ms over a 24-hour run on the dock-03 camera. The same graph behind a conventional batching server on the same chip had a median of 11.3 ms and a p99 of 41 ms, because bursts of motion filled the queue.
The full method, including the overrun policy and the thermal model, is in the paper. The benchmark harness is open, so the numbers can be reproduced on any supported chip.