Designing a Latency Budget for LLM-Powered Features

The Latency Reality of LLM Inference

An LLM API call is not like a typical microservice call. A REST API call that takes more than 200ms is considered slow. An LLM completion that takes 2 seconds is performing well. A complex generation with a long context window can take 15–30 seconds.

If you are building user-facing features on top of LLM inference without explicitly designing for these latency characteristics, you will build a product that frustrates users, increases infrastructure cost through wasted compute, and fails unpredictably under load.

Latency budget design is the practice of defining the acceptable latency envelope for each operation in your system and designing the architecture around it. For LLM-powered features, this discipline is not optional.

What Goes Into LLM Latency

The latency of an LLM-powered feature is not just the inference time. It is the sum of:

1. Query processing: embedding generation, intent classification, query rewriting — typically 20–100ms
2. Retrieval: vector store query, hybrid search, re-ranking — typically 50–300ms depending on corpus size and re-ranker complexity
3. Prompt assembly: context selection, deduplication, tokenization — typically < 20ms
4. Time to first token (TTFT): the time before the model begins streaming a response — driven by model size, context length, and provider load. Range: 200ms (small models, short context) to 3+ seconds (large models, long context)
5. Token generation time: tokens per second × output token count. A model generating 40 tokens/second producing a 400-token response adds 10 seconds of generation time.
6. Network transmission: streaming response delivery to the client

The sum of these components for a mid-complexity RAG query with streaming output is typically 3–8 seconds for the first substantial response, with the tail of the response completing in 10–20 seconds.

Budget First, Then Architecture

Start by defining the acceptable latency for the user experience you are building:

• Conversational assistant: users accept up to 3 seconds before streaming begins, with streaming output creating a sense of progress
• Search / retrieval feature: users expect results within 1.5–2 seconds — synchronous, not streaming
• Background analysis: latency can be minutes if the user is notified of completion — no streaming required
• Real-time document processing: depends on document size and user context, but typically 5–15 seconds with streaming

Once the budget is defined, design the pipeline to fit within it — not the other way around.

Streaming as a UX Strategy

Streaming is the primary tool for making LLM latency acceptable in interactive features. When users see tokens appearing immediately after submission, the perception of waiting is significantly reduced relative to a loading spinner followed by a full response.

Implementing streaming correctly requires:

• Server-sent events (SSE) or WebSocket for streaming delivery from your backend to the client
• Partial rendering in the frontend — displaying tokens as they arrive, not buffering until complete
• Timeout handling for the stream — if the stream stalls (network issue, provider issue), the client should receive an error rather than waiting indefinitely
• Backpressure management — if the client cannot consume the stream as fast as it is generated, the buffer must be bounded

Streaming is not available in all contexts. For features that require the complete response before any processing can occur (content that must be validated, structured output that is parsed as JSON), streaming provides no benefit and may complicate error handling. In these cases, caching becomes more important.

Caching LLM Responses

LLM inference is expensive. If a significant fraction of queries are semantically identical or near-identical, caching responses can dramatically reduce both latency and cost.

Exact-match caching (cache by full prompt hash) has low hit rates for conversational applications where query phrasing varies. It works well for structured queries with deterministic prompts.

Semantic caching (cache by query embedding similarity) returns cached responses for queries that are semantically equivalent even if phrased differently. Tools like GPTCache and Langchain's semantic cache implement this pattern. Hit rates of 15–40% are achievable in knowledge base applications where users frequently ask the same questions differently.

Cache TTL must reflect the freshness requirements of the underlying data. A cached response to "what is our refund policy?" is valid until the policy changes. A response to "what are today's rates?" may have a TTL measured in minutes.

Model Selection as a Latency Lever

Not all LLM features require the most capable model. A feature that classifies user intent into one of eight categories does not need GPT-4. A feature that routes a support request to the correct department does not need a model that can write code.

Design a model routing layer that applies the minimum model capability required for each task type:

• Intent classification, routing, simple Q&A: smaller, faster models (GPT-3.5-class, Haiku, Llama 3 8B)
• Reasoning-intensive tasks, code generation, complex synthesis: larger models (GPT-4-class, Sonnet, Llama 3 70B)

Routing at the application layer — not at the model provider level — keeps flexibility to swap providers as model capabilities and pricing evolve.

Handling Provider Latency Variability

Hosted model APIs have variable latency. During peak demand periods, provider-side latency can increase significantly — occasionally by 2–3× compared to off-peak performance. This is outside your control, but it must be accounted for in your SLO design.

Mitigation strategies:

• Multi-provider routing: maintain integrations with at least two providers. Route to the faster/cheaper provider by default, with automatic fallback when a provider's P95 latency exceeds a threshold.
• Timeout and fallback: define explicit timeouts per request type. When a request times out, either retry with a different provider or return a degraded response (a simpler model, a cached result, or a graceful error).
• Capacity reservations: hosted providers (OpenAI, Anthropic, Azure OpenAI) offer provisioned throughput options that reduce latency variability in exchange for a committed usage commitment. Evaluate this against your actual latency SLOs.

---

LLM latency is a first-class engineering concern, not a property of the model that you accept as given. The teams that build reliable LLM-powered products are the ones that defined their latency budgets before writing their first API call, designed their pipelines to fit within those budgets, and built observability to know when they are not.