A client rang me at 8 a.m. on a Thursday. Their AI-powered document summariser had been returning 503s since midnight because OpenAI had a partial outage on the gpt-4o endpoint. They were on a £40k/month SaaS contract and their enterprise customer was screaming. I had exactly one question in my head: why had nobody built a retry-to-Anthropic fallback into this thing?
That was eight months ago. Vercel AI Gateway is the closest thing I've seen to an off-the-shelf answer to that exact problem. But "closest thing" is doing a lot of work in that sentence. Let me tell you what it actually does, what the numbers look like, and where you should still be suspicious.
---
What Vercel AI Gateway Actually Is
Most people see the name and assume it's a proxy with some nice logging. It's a bit more than that, but not as much more as the marketing copy implies.
At its core, Vercel AI Gateway sits between your application code and multiple LLM providers: OpenAI, Anthropic, Mistral, Google Gemini, and others. You send a single request to the gateway endpoint. The gateway decides which model/provider to call, handles retries, caches semantic responses, and returns the result. You get one bill line item instead of four API dashboards.
The SDK integration is genuinely tidy. If you're already using the Vercel AI SDK, you swap out your provider import for the gateway client and pass a providers config. Maybe fifteen minutes of work on an existing project.
What it is not is a magic cost reducer on its own. The savings come from three specific behaviours: routing by cost, caching repeated prompts, and avoiding cold restarts after provider failures. If you're not getting value from at least one of those three, the gateway adds overhead without benefit.
---
How the Routing Logic Works
Provider Priority and Weighted Routing
You configure a list of providers with a priority order or a weight distribution. Priority routing is simple: try provider A, and if it returns a 429 or a 5xx, fall through to provider B. Weighted routing splits traffic by percentage, so you can say "70% OpenAI, 30% Mistral" and test cost differences in real traffic without a full migration.
I used weighted routing on a content-generation tool Seahawk built for a media company last quarter. We ran gpt-4o-mini at 60% against mistral-medium at 40% for a month. Mistral came in about 34% cheaper per million tokens at the time, but the output quality for structured JSON extraction was noticeably weaker. We ended up at 80/20 in favour of OpenAI. The point is: the gateway made that A/B test effortless. Without it we'd have been wiring up two separate SDK clients and managing the split in application logic.
Failover Behaviour
Failover is the headline feature and it works as advertised, mostly. When OpenAI returns a 5xx, the gateway retries on the next configured provider within roughly 800ms in my testing. For streaming responses it's a bit messier: the stream can silently restart from the fallback provider, and if you're not handling the stream correctly on the client side you can get a duplicated opening chunk. That tripped us up once.
One thing to know: the gateway does not do semantic failover. It doesn't know that your Anthropic claude-3-5-sonnet response might word things differently than the OpenAI equivalent. You're responsible for prompt compatibility across providers. For most chat-style interfaces that's fine. For structured outputs with strict schemas, test every provider in your failover chain independently before going live.
---
The Caching Layer: Where Real Money Lives
This is the part most people under-appreciate. Vercel AI Gateway includes semantic caching, not just exact-match caching.
Exact-match caching is table stakes: if the same prompt string hits the gateway twice, return the cached response. Semantic caching goes further. Using embedding similarity, it recognises that "summarise this paragraph in three sentences" and "give me a three-sentence summary of this paragraph" are the same request and serves the cached result.
For the document summariser project (the one that nearly gave my client a heart attack), we instrumented the cache hit rate after enabling semantic caching with a 0.92 cosine similarity threshold. Over two weeks of production traffic: 41% cache hit rate. At £0.015 per 1K output tokens on GPT-4o, that's not trivial across 2 million daily output tokens.
Do the napkin maths yourself:
- 2,000,000 output tokens/day
- 41% served from cache = 820,000 tokens not billed
- At £0.015/1K that's £12.30 saved per day
- Over a month: roughly £370
Not life-changing for a big enterprise. Significant for an indie SaaS operator watching margin. And that's one project, one month.
The OpenAI prompt caching feature handles prefix caching natively now, so for long system prompts you're getting some of this for free already. Vercel's semantic cache is additive to that, handling the variation in the user-turn content.
---
Observability: Better Than Nothing, Not Good Enough Alone
Every request through the gateway gets logged: latency, token counts, provider used, cache hit/miss, cost estimate. You see this in the Vercel dashboard. It's clean and readable.
Here's the thing though. If you're running a serious production system, you already have something like Datadog, Grafana, or at minimum LangSmith in your trace pipeline. The Vercel dashboard gives you gateway-level visibility. It doesn't give you span-level traces across your full application. You can't see that a particular user's request took 4.2 seconds because 3.1 of those seconds were spent in your retrieval step before the LLM was even called.
So I treat the gateway's built-in observability as a first filter: is the problem at the LLM call level or somewhere else? For anything deeper, I still export to a proper tracing tool.
One specific number worth knowing: the gateway adds roughly 15-30ms of latency per request on average from what I've measured on EU-region deployments. For real-time voice or sub-100ms UX requirements, that matters. For async document processing, it doesn't.
---
What It Actually Costs to Run
Vercel AI Gateway is included in the Pro plan (£17/month at time of writing) and higher. There's no per-request surcharge from Vercel. You still pay the underlying provider's token costs directly.
The hidden cost is operational: you're now adding Vercel as a dependency in your inference path. If Vercel has an edge network issue, your LLM calls fail regardless of whether OpenAI is perfectly healthy. I've seen this happen once in the past six months, a ~12-minute partial outage on Vercel's edge in the EU-West region. For most apps that's acceptable. For anything with SLA commitments measured in nines, factor it in.
There's also the question of data residency. Your prompts and completions pass through Vercel's infrastructure. For most consumer apps: irrelevant. For healthcare, finance, or anything touching GDPR-sensitive personal data in a meaningful way: read the data processing agreement carefully before you pipe anything through. I've had to tell two fintech clients to skip the gateway entirely for this reason and handle routing in-application instead.
---
When to Use It and When to Skip It
I'll be direct about this because the developer marketing around AI tooling tends to oversell.
Use Vercel AI Gateway if:
- You're already on Vercel and using the AI SDK (zero extra friction)
- You have a multi-provider strategy and want failover without writing retry logic
- Your app has enough repeated or semantically similar prompts for caching to pay off
- You want a cost dashboard without setting up separate provider billing integrations
Skip it or think carefully if:
- You have strict data residency requirements (GDPR, HIPAA)
- You need sub-50ms total inference latency and every hop counts
- You're running on a non-Vercel infrastructure and adding their edge introduces more complexity than it solves
- Your prompt variety is extremely high and semantic cache hit rate will be close to zero anyway
Back in 2022 we built a legal document analysis tool on a self-hosted stack for a City of London firm. Even if Vercel AI Gateway had existed then, it would have been off the table immediately. The prompts contained privileged client information and the firm's IT compliance team would never have signed off on a third-party proxy. We rolled our own provider abstraction layer in about four hours of work. It handled failover across two providers with a simple priority queue. Not glamorous. Totally adequate.
---
The Practical Setup (Short Version)
If you've decided it's right for your project, here's the sequence I follow:
- Enable the gateway in your Vercel project settings under the "AI" tab
- Install or update
aiand@ai-sdk/openai(and whichever other provider packages you need) to the latest versions - Replace direct provider client instantiation with the gateway client from
@vercel/ai-gateway(check the official docs for the exact import path as it's changed once already) - Define your provider list in priority order in the gateway config object
- Set your semantic cache similarity threshold: I start at 0.90 and adjust based on observed hit rates after a week of traffic
- Ship to a staging environment and deliberately trigger provider failures to verify the failover chain works the way you expect
- Monitor cache hit rate and latency p95 for the first two weeks before drawing conclusions
That's it. Genuinely not complicated.
---
FAQ
Does Vercel AI Gateway support streaming responses?
Yes, streaming works through the gateway. The one gotcha is failover mid-stream: if the primary provider drops during a streaming response, the gateway will retry on the next provider, but the stream restarts. Depending on how your client-side UI handles partial content, this can cause a visible flicker or a duplicated first sentence. Test it explicitly in your UI before shipping.
Can I use Vercel AI Gateway without the Vercel AI SDK?
Technically you can hit the gateway endpoint over HTTP directly, but the SDK integration is where it becomes ergonomic. Without the SDK you're writing your own fetch wrappers around the gateway URL, managing streaming yourself, and handling retries manually. At that point you might as well build your own provider abstraction. The gateway's value is tightly coupled to the SDK in practice.
How does semantic caching handle sensitive or personalised data?
It doesn't, automatically. If you're sending prompts that include user-specific data (names, account numbers, session context), the cache will attempt to match semantically similar prompts regardless of whether that data differs between users. This can cause incorrect cache hits in personalised workflows. You should either disable semantic caching for those routes or include a user-scoped cache key if the gateway supports that in your plan tier.
What happens to my requests if Vercel has an outage?
They fail. The gateway is in the critical path. If you need genuine multi-cloud resilience, you should have a circuit-breaker at the application layer that can bypass the gateway entirely and hit providers directly. I keep provider SDK clients initialised as a fallback in any app where uptime really matters.
---
The honest summary: Vercel AI Gateway is a well-built piece of infrastructure that earns its place in a Vercel-native AI stack. It's not going to revolutionise your unit economics, but a 35-40% cache hit rate on the right workload plus automatic failover is a real operational improvement over wiring everything by hand. Just know what it can't do before you commit to it. The 8 a.m. phone call from a panicked client is a miserable way to find out where your assumptions were wrong.