AI Model Latency Benchmarks

Representative latency and throughput figures for common AI models, to illustrate how providers compare. These are reference numbers, not a live feed.

These figures are representative reference values from standardized prompts, not a live measurement. Real-world latency varies by region, prompt size, and provider load. Run your own numbers with RelayPlane on your machine.

Model	Provider	P50 Latency	P95 Latency	Tokens/sec	Success Rate
Fastestgemini-2.0-flash	google	178ms	312ms	156.3	99.8%
gpt-4o-mini	openai	245ms	412ms	125.5	99.9%
claude-3.5-haiku	anthropic	289ms	478ms	118.2	99.8%
claude-4-sonnet	anthropic	478ms	845ms	82.1	99.8%
gpt-4.1	openai	485ms	856ms	82.7	99.9%
gpt-4o	openai	512ms	892ms	78.4	99.9%
gemini-1.5-pro	google	623ms	1124ms	65.3	99.5%
claude-4-opus	anthropic	892ms	1456ms	48.2	99.7%

Methodology

•Standardized prompts: Short prompts (~50 tokens) with ~100 token responses
•Sample size: 10 requests per model per run
•Reference figures: Illustrative values, not a real-time feed
•Reproduce it yourself: Run the same prompts through RelayPlane with your own keys (BYOK) to measure your real latency

API Access

Access benchmark data programmatically:

GET https://api.relayplane.com/api/v1/benchmarks

Returns JSON with the reference benchmark figures shown above. No authentication required.

Route to the fastest model automatically

RelayPlane can automatically select the fastest provider for your workflows based on real-time benchmarks.

Get Started Learn More