Source: Unite.AI
The hardware conversation about edge AI has become a great deal more honest in the last year. A recent piece on this site argued that the old design hierarchy — “maximize throughput, then manage power and thermals around it” — has inverted, and that for industrial deployments power now takes top billing, with raw throughput coming last. It follows an argument this publication has been making for a while: that edge devices are “thermal-limited, not MIPS/compute-limited“, and that smartphones sit at those limits already. Both are real corrections, and they are overdue.
But it still carries one assumption from the world it is correcting. Every item in that hierarchy is budgeted against an assumed workload, and the workload almost everyone is still budgeting against is a single forward pass: a model receives an input, produces an output, and the silicon gets a moment to cool.
That is not what an agent does. An agent decides, calls a tool, reads what comes back, and decides again. How many times it goes around that loop is not a property of your hardware, and it is not really a property of your model either. It is a property of the problem someone handed it that morning. I have agents running on edge hardware, and the thing that took me longest to accept was not that they were slow. It was that the cost of a run was being set somewhere I had no visibility into at design time.
The Loop Is Unbounded Until Someone Types a Number
This is not a rhetorical framing; it is how the frameworks are actually built. In OpenAI’s Agents SDK, the runner “runs a loop”, and when the model produces tool calls, the runtime will “run those tool calls, append the results, and re-run the loop.” The only thing that stops it is a turn limit — exceed max_turns and you get an exception — and the documentation notes that you can pass max_turns=None to disable the limit entirely.
On a server, that number is a billing decision. Somebody notices the invoice.
On a device, that number is a thermal decision, because loop length is duty cycle. And duty cycle is the one variable that passive cooling cannot argue with.
Sustained Load Does Something Different to a Phone Than a Benchmark Does
A March 2026 benchmark put four platforms under exactly this kind of load: a quantized 1.5-billion-parameter model, a fixed 258-token prompt, twenty back-to-back runs, measuring throughput, power, and temperature on each. It is a preprint, and it benchmarks one model on four devices, so treat the specific numbers as a characterization of those platforms rather than a law of nature. The shape of the result is what matters.
An iPhone 16 Pro peaked at 40.35 tokens per second and could not hold it. Degradation showed up within two inferences. It settled at 22.56 tokens per second — a 44 percent reduction — and stayed throttled for 65 percent of the benchmark. Dynamic voltage and frequency scaling, the mechanism that drops clock speeds when junction temperature climbs, did precisely what it exists to do.
The Galaxy S24 Ultra failed differently, and worse. Rather than degrading, the Android thermal governor imposed a hard GPU frequency floor at iteration six, at 78.3°C, and inference stopped. The authors make the point that matters here better than I could: for agent deployments, this is “more disruptive than graceful degradation”, because the system does not get slower; it becomes unusable.
Now hold on to the detail that makes this damning rather than merely interesting. Every one of those twenty runs used the same prompt. That is the friendliest workload an agent’s hardware will ever see, and two flagship phones could not sustain it for twenty repetitions. This is not a new finding either — MELTing point, presented at MobiCom in 2024, concluded that on energy and thermal grounds “the continuous execution of LLMs remains elusive.” Two years and several process nodes apart, the same wall.
Two Curves Move Toward Each Other, and Your Product Breaks Where They Cross
An agent’s loop is worse than a repeated prompt in a specific, mechanical way.
Decoding is memory-bandwidth-bound: throughput is governed by how fast the model can read its key-value cache, not by how many operations the chip can theoretically perform. That cache grows with context. Every loop step appends a tool result, an observation, a partial plan — so step ten is generating tokens against a materially larger cache than step one was.
Meanwhile the device is heating up, and the governor is pulling clocks down.
So the per-step cost rises at exactly the moment the device’s capacity to pay it falls. The two curves converge, and wherever they meet is where your product fails. It is never step one. Step one is where you tested.
There is a scale problem underneath this too. Generative work is simply a different order of expense from what edge silicon spent a decade running: measured across 88 models, text classification cost around 0.002 kWh per thousand inferences against 0.047 kWh for text generation — roughly twenty times more, before any loop multiplies it. Those measurements were taken on a data-center GPU, not a handset, so read them as a ratio between kinds of work rather than as a power figure for your device. For scale, the same study puts a full smartphone charge at 0.022 kWh.
Buy on Joules per Finished Task, Not on Tokens per Second
The most useful result in that 2026 benchmark is the one that looks least impressive.
A Hailo-10H NPU managed 6.9 tokens per second at under 2 watts. Slow — genuinely slow, and the authors say so. But its throughput coefficient of variation was 0.04 percent, two orders of magnitude steadier than anything else tested. The laptop GPU in the same study delivered 131.7 tokens per second at 34.1 watts.
Then compare the two on energy rather than speed: 270.5 millijoules per token on the little NPU against 297.3 on the GPU. Despite a nineteen-fold gap in throughput, the small part did slightly more computation per joule — and it did it with essentially no variance.
If you select hardware on tokens per second, you buy the fast one. If you select on the ability to complete a bounded loop at a predictable cost, which is what an agent actually needs, the ranking changes. The unit that should appear on the spec sheet is joules per completed task, with a variance figure next to it. A benchmark that reports peak throughput is telling you about the first inference of the day.
The Honest Objection, and What It Does Not Solve
The obvious reply is that this is a transient problem: silicon improves, NPUs mature, and anything written about a 2026 phone will look quaint. Or, more practically, offload the expensive steps to a server.
I would take the hardware bet myself. But offloading is the round trip you moved to the edge to avoid, and an agent does not pay it once — it pays it per loop step, and the loop length is the thing you cannot predict. Hybrid designs do not remove the variance; they relocate it onto a network.
The deeper asymmetry does not move with process nodes. A device’s budget is fixed at design time. An agent’s demand is decided at run time, by whatever a user asked for. Better silicon raises the ceiling. It does not tell the agent where the ceiling is.
So tell it. Set the turn limit in the product specification rather than discovering it in a code review, and pick the number from the thermal envelope: decide how many steps fit, then design the agent to produce its best available answer at that bound instead of its ideal answer at an arbitrary one. Treat it as a deadline, not a target.
Then give the agent the budget as an input. Remaining headroom, battery state, whether the platform has already begun throttling — that belongs in the context, the same way the current time does. An agent that knows it is on step eight of ten can summarize and commit. An agent that does not will keep exploring until the operating system decides for it.
And test the tail, not the median, which on a physical device means testing in simulation. The failure case is never the clean run. It is the run that took fourteen steps because a tool returned something ambiguous on step three, and you cannot enumerate those by hand on a phone that needs to cool down between attempts. My own systems train against simulations largely for this reason: the interesting behavior is in the long runs, and long runs are exactly what hardware will not let you sample by hand.
None of this requires a faster chip. It requires admitting that the workload changed shape. Nobody ships a device whose battery is sized for one photograph. We are still shipping devices whose thermal budget is sized for one inference.