💻

Inference Latency Budget Calculator

Break an LLM response time into time-to-first-token plus generation time, and check it against your target latency budget.

Output tokens

Generation speed

Time to first token

Latency budget

Loading calculator…

How it works

Generation time = output tokens ÷ tokens-per-second · Total = time-to-first-token + generation time

Perceived LLM latency has two parts: time-to-first-token (TTFT), dominated by prompt length and queueing, and the streaming generation that follows at a steady tokens-per-second rate. Because generation is linear in output length, the cheapest way to fit a tight budget is often to cap or shorten the response rather than chase a faster model. TTFT matters most for short answers and for the feel of responsiveness; for long outputs the tokens-per-second throughput is what dominates the wall-clock time.

Worked example

A response of 300 tokens at 60 tokens/second takes 300 ÷ 60 = 5 seconds of generation (5,000 ms). Add 400 ms to first token and the total is 5,400 ms — which blows a 3,000 ms budget by 2,400 ms, so you would trim output length or raise throughput.

Frequently asked questions

What drives time-to-first-token?

Mostly prompt length (the model must read the whole prompt before answering), plus server queueing and network round-trips. Long system prompts and large retrieved contexts push TTFT up; prompt caching can pull it back down.

How do I lower total latency?

Shorten the output (it is linear in token count), pick a faster-generating or smaller model, cap max tokens, and cache or shrink the prompt to cut TTFT. Streaming the response also improves the perceived speed even when total time is unchanged.

Is this per request or concurrent?

This is a single request. Under concurrency, shared GPUs batch requests and per-request throughput can drop, so measure tokens-per-second at your real load rather than the single-stream best case.

Related calculators

Network Latency Budget Calculator

Set a target response time, add up the latency each stage consumes (DNS, connect, server, transfer, render) and see how much of the budget is left.

Batch Size Memory Calculator

Estimate the activation memory a training batch consumes from batch size, sequence length, hidden size and layer count — the term that scales with batch and blows up GPU memory.

Tokens Per Dollar Calculator

See how many LLM tokens a dollar buys at a given per-million-token price, plus how many tokens your monthly budget covers — the fastest way to compare model pricing.

GPU Utilization ROI Calculator

Find the real return on a rented or owned GPU: how idle time inflates your effective cost per productive hour, and whether the revenue it earns beats the bill.

Educational estimate — real costs and memory depend on your provider, framework, model architecture and pricing, which change often. Verify against your own bills and profiling before committing budget.