AI & ModelsJuly 14, 202617 min read

AI Costs in Production: How to Optimize OpenAI and Gemini Spending

Per-token prices of closed and open-weight models can differ by more than 25x. Learn the routing, caching, and compression strategies that can cut your bills dramatically.

The Current State: Why AI Costs Are Spiraling Out of Control

The cost difference between flagship and open-weight models has reached a critical point. Take a workload of 1 million support queries a month, averaging 150 input and 200 output tokens. At list prices, GPT-5.2 costs roughly $3,060 a month for that, while DeepSeek V3.2 on DeepInfra costs about $115 – a ~27x difference[7][9]. This isn't a marginal drop, it's a fundamental shift in AI production economics.

Open-weight models have stopped being an experiment. In the a16z and OpenRouter study of more than 100 trillion tokens, they accounted for roughly a third of the traffic routed through OpenRouter, led by DeepSeek, Qwen, Kimi K2, and GLM[6]. Teams aren't just testing open models anymore – they're using them at scale.

Problem #1: Blind Routing

Many teams send everything to their most capable model because "it works." Meanwhile, in many applications a large share of queries are simple classification, data extraction, or basic reasoning – workloads that Gemini Flash-Lite will handle for a fraction of the price. In LMSYS experiments, the RouteLLM router cut costs by over 85% on MT Bench while keeping 95% of GPT-4's quality; on MMLU the saving was 45%[1].

Key lesson: routing isn't about "cheaper models" – it's about matching task complexity to model capabilities. Most tasks don't need GPT-4.

Problem #2: Paying for the Same Context Repeatedly

A typical support chatbot sends the same system prompt and product documentation in every query. That's hundreds of thousands of tokens daily for identical content. Context caching bills repeated input at a fraction of the normal rate – for example, cached input tokens in OpenAI's GPT-5 family cost 90% less than regular ones, and Gemini applies caching automatically for 2.5 and newer models[5]. Google also offers Flex inference – 50% cheaper than the standard API, in exchange for variable latency and best-effort availability[3].

Batch Mode is another lever for non-urgent workloads. If a report can wait 2 hours, there's no point paying for real-time inference.

Problem #3: Redundant API Calls

Semantic caching (GPTCache, Redis) eliminates repetitive queries entirely – before they even reach the API. A user asks "how to reset password" in 50 linguistic variants – the cache returns the same answer without any token cost. This is the last line of defense against runaway costs.

Additional optimization: prompt compression. Microsoft's LLMLingua reports compressing prompts by up to 20x with minimal loss of quality[2]. Fewer tokens = a smaller bill.

Three Key Levers in Practice:

Model routing: the flagship model only where actually needed. Rest of traffic → Flash-Lite or DeepSeek
Context caching: System prompts, documentation, long contexts – cache aggressively
Semantic caching: Eliminate redundant calls before reaching the API

A sensible default is to start with a fast, cheap model such as Gemini Flash and move to a Pro-class model only when you see a concrete quality gap. Flagship models should be reserved for the tasks that need them, and much of the remaining traffic can go to open-weight models on serverless platforms.

Reasoning models (such as Gemini 2.5 Flash with thinking or GPT-5) add a new dimension. A bigger reasoning budget doesn't automatically mean better answers, and long reasoning runs can take minutes rather than seconds. You need to match reasoning effort to your actual latency and quality requirements – and measure it on your own tasks.

Bottom line: AI costs spiral out of control because teams optimize too late – only when the bill exceeds budget. Routing, caching, and compression aren't nice-to-have. They're baseline for every production deployment.

Model Routing: Intelligent Query Direction to Cheaper Models

In production, unit economics matter. When your application handles tens of thousands of queries daily, the difference between $1.75 and $0.26 per million tokens isn't a rounding error in Excel – it's a decision between profitability and burning capital. Model routing lets you exploit this difference without compromising user experience.

How Intelligent Routing Works

The concept is simple: not every query requires a flagship model. Sentiment classification? Gemini Flash-Lite. Invoice data extraction? A small, cheap model. Complex legal reasoning? Only then do you reach for a flagship model. RouteLLM cut costs by over 85% while keeping 95% of GPT-4's quality on MT Bench, and by 45% on MMLU[1].

The key is a classifier before the main model. It analyzes query complexity – length, keyword presence, structure – and routes to the appropriate tier. In practice, a large part of the traffic can land on cheaper models, and you only pay the premium where it actually makes a difference.

Economics of Choice: Production Numbers

Take a concrete case: 1,000,000 support queries a month, averaging 150 input tokens and 200 output tokens – 150 million input and 200 million output tokens in total. At list prices, GPT-5.2 comes to about $3,060 a month. DeepSeek V3.2 through DeepInfra? About $115 – roughly 27x cheaper. Gemini 2.5 Pro sits in the middle at around $2,190[7][8][9].

┌─────────────────┬──────────┬───────────┬──────────────┐
│ Model           │ Input    │ Output    │ Cost/month   │
├─────────────────┼──────────┼───────────┼──────────────┤
│ GPT-5.2         │ $1.75/1M │ $14.00/1M │ ~$3,060      │
│ Gemini 2.5 Pro  │ $1.25/1M │ $10.00/1M │ ~$2,190      │
│ DeepSeek V3.2   │ $0.26/1M │ $0.38/1M  │ ~$115        │
└─────────────────┴──────────┴───────────┴──────────────┘

Open-weight models already handle about a third of the traffic on OpenRouter, with DeepSeek, Qwen, and Kimi K2 among the most used[6]. Per token, the gap between GPT-5.2 and DeepSeek V3.2 in the table above is about 7x on input and about 37x on output.

LiteLLM as Universal Gateway

Theoretically, you can build routing from scratch. Practically? Use LiteLLM. It's a unified API gateway that normalizes interfaces for OpenAI, Anthropic, Google, Azure, and 100+ other providers. You add budget enforcement, routing rules, and fallback logic in one place.

Setup looks like this:

from litellm import completion

# Automatic routing based on complexity
response = completion(
    model="gpt-5.2" if complexity_score > 0.7 else "gemini-2.5-flash-lite",
    messages=[{"role": "user", "content": prompt}],
    max_budget=0.05  # hard cap per request
)

LiteLLM also supports semantic caching (e.g. with Redis), and you can combine it with provider-side context caching – reusing system prompts and reference docs at a steep discount. This removes redundant calls before they even reach the API.

When to Pay Premium

There are scenarios where a flagship model pays for itself immediately. A good rule of thumb is still to start with a Flash-class model and upgrade only when you see a concrete quality gap. But:

Complex reasoning: Claude Opus, GPT-5.x, Gemini Pro – when you need multi-step logic
Mission-critical accuracy: medicine, legal, finance – cost of error > cost of token
Brand experience: customer-facing chatbot for a flagship company can't sound like a bot from 2021

A larger reasoning budget doesn't always improve results, and reasoning-heavy requests can take minutes. Match effort to your actual latency requirements and measure the gain on your own evals.

Routing isn't optimization – it's a core architecture decision. At large scale, the difference between "everything on the flagship model" and intelligent routing can reach tens of thousands of dollars a year – without a noticeable drop in quality for the end user.

Context Caching and Semantic Caching: Eliminating Redundant Costs

The biggest waste of AI budget is repetitive invocations of the same contexts. A typical production system sends the same long system prompt hundreds of times daily, paying for every token. Context caching solves this problem – instead of paying the full rate for the whole context with each invocation, the provider reuses the cached prefix and bills it at a steep discount (90% off for cached input in OpenAI's GPT-5 family)[5].

The mechanism works simply: the first request with a long context (e.g. API documentation, business rules, few-shot examples) is cached on the provider's side, and subsequent requests that start with the same prefix pay a reduced rate. Gemini enables this implicit caching by default for 2.5 and newer models and also lets you create explicit caches[5]. OpenAI applies prompt caching automatically to long, repeated prefixes. Check the current discount for your specific model on the provider's pricing page.

Semantic caching goes a step further – it eliminates redundant API calls entirely. Instead of checking exact string matches, tools like GPTCache use embeddings to detect semantically identical queries. The question "How to reset password?" and "How can I reset my password?" are the same query for semantic cache. Implementation with Redis as a cache layer before the API gives an instant response to repeated questions without any model invocation cost.

The difference between both approaches is crucial: context caching reduces the cost of a single invocation (fewer tokens to process), semantic caching eliminates the invocation altogether (zero tokens, zero latency). In practice, you use both simultaneously – you cache long system contexts and additionally check if a similar user query hasn't already been handled.

The tech stack looks like this: Redis as a semantic cache layer → context caching at the API provider level → prompt compression with LLMLingua (up to 20x fewer tokens)[2] → routing to cheaper models for simple queries. How much you save depends heavily on how repetitive your traffic is, so measure it before and after.

Implementation requires thinking through a cache invalidation strategy. Provider caches expire after a while, so documentation or business rule updates must be reflected in new prompts. A semantic cache needs a similarity threshold – too high and it doesn't catch similar questions, too low and it returns irrelevant answers. Treat values around 0.90–0.95 as a starting point for testing, not a rule, and tune them on real queries.

Cost monitoring after cache deployment usually shows a characteristic pattern: the hit rate is low at first (the cache is building) and then stabilizes at a level that depends on how repetitive your users' questions are. Support chatbots tend to benefit the most; internal tools less so.

Open-Source vs Closed-Source: Cost Analysis in Numbers

The difference between GPT-5.2 and DeepSeek V3.2 isn't theoretical – it follows directly from list prices. For a system handling 1 million support queries a month (150 input + 200 output tokens each), GPT-5.2 generates a bill of about $3,060, Gemini 2.5 Pro about $2,190, and DeepSeek V3.2 through DeepInfra about $115[7][8][9].

Key Cost Metrics:

Model	Input ($/1M tokens)	Output ($/1M tokens)	Monthly cost (1M queries)
GPT-5.2	$1.75	$14.00	~$3,060
Gemini 2.5 Pro	$1.25	$10.00	~$2,190
DeepSeek V3.2	$0.26	$0.38	~$115

List prices at the time of writing; check the providers' pricing pages before you plan a budget.

Quality gap? Often smaller than you think. For many production tasks (ticket classification, data extraction, FAQ answers) a good open-weight model is good enough – but verify it on your own evaluation set rather than on public benchmarks alone.

When does migration pay off? Simple math: if your workload costs more than a few hundred dollars a month on a closed model and doesn't require top quality in 100% of cases, test a cheaper model. Even if a fifth of the queries has to fall back to the flagship model, the blended cost can still be several times lower.

Serverless infrastructure is the second game-changer. Instead of maintaining your own GPUs and paying for idle time, you pay strictly for inference on platforms like DeepInfra, Together.ai, or Replicate. Zero DevOps overhead, automatic scaling.

Hybrid Strategy in Practice: Use a router such as RouteLLM. Simple queries go to DeepSeek ($0.26/1M input tokens), medium ones to Gemini Flash, and only the most demanding to a flagship model. In LMSYS experiments this kind of routing cut costs by over 85% while keeping 95% of GPT-4's quality on MT Bench, and by 45% on MMLU[1].

Open-weight models are no longer an experiment – they handle about a third of OpenRouter's traffic, led by DeepSeek, Qwen, and Kimi K2[6]. Companies have stopped asking "whether" and started asking "how quickly can we migrate".

Red Flags for Migration: If your use case requires frontier-level reasoning (mathematics, complex inference), the quality difference will be larger. If compliance requires certified models (HIPAA, SOC2 with specific vendor) – open-source options are limited. If latency <500ms is critical, some open models may not meet requirements.

But for typical enterprise use cases – customer support, content generation, data extraction – a price gap of an order of magnitude or more is hard to ignore, as long as your own tests show acceptable quality.

Advanced Techniques: Flex Inference, Batch Mode, and Prompt Compression

In production, every cent per token counts, so you need to dig deeper than just model selection. There are three techniques that really reduce costs – provided their trade-offs are understood and implemented consciously.

Flex Inference: Half Price for Higher Latency

Google's Flex inference, introduced in April 2026, is a synchronous tier optimized for cost – you pay 50% of the standard rate in exchange for variable latency and best-effort availability[3]. Sounds like a deal? It is, but only for tasks that don't require real-time responses. If you're processing overnight analytics, generating batch reports, or enriching data offline, Flex makes sense. For chatbots serving live customers – forget it.

Key decision: does your use case tolerate variable latency. If so, routing part of traffic to Flex can reduce monthly bills by several hundred dollars without code changes.

Batch Mode: Discount for Patience

Batch Mode is a flat discount for decoupling execution from an immediate response. OpenAI's Batch API, for example, costs 50% less and returns results within 24 hours[4]; Gemini offers a similar batch mode. Classic examples are embeddings for an entire knowledge base, lead scoring from a CRM, or log summaries. Instead of paying the full rate for each API call, you package jobs and run them asynchronously.

This requires pipeline redesign – you can't just throw a synchronous endpoint into batch and expect miracles. You need a queue, retry handling, and a notification mechanism after completion.

Prompt Compression: 20x Fewer Tokens

LLMLingua can compress prompts by up to 20x, which directly translates to fewer input tokens sent to the API[2]. It works by using a small language model to drop the least informative tokens while preserving the meaning.

Example: instead of sending 10k tokens of product documentation every time, you compress it to 500 tokens. With 100k calls a month, that's 1 billion vs 50 million input tokens – about $1,750 vs $87.50 at GPT-5.2's input rate of $1.75/1M tokens. The catch? You need to host the compression model (it's open source) and you add some latency before each API call.

Reasoning Budget: More != Better

Models with reasoning capabilities can think longer, but it costs – both in tokens and in time, sometimes minutes per request for complex tasks. More reasoning budget doesn't automatically mean better answers.

Practical advice: don't set max reasoning budget "just in case." Start with 16k, measure performance on your benchmark, and increase only if you see real gain. In projects where the team paid 2x more for reasoning that improved accuracy by 1%, the math didn't add up.

The same philosophy applies to models: start with a fast, cheap model such as Gemini Flash and upgrade to Pro only if you genuinely need it. Pay for what you actually use, not for what sounds impressive in a config file.

Enterprise Governance: From AI Studio to Vertex AI

In large organizations, chaos with API keys and lack of spending control is the norm. Teams use Google AI Studio for prototypes, then move to production and suddenly it turns out nobody tracks which model costs how much per team. Migration from AI Studio to Gemini Enterprise Agent Platform gives you central governance, but requires architecture that won't force you to rewrite half your code.

Portkey acts as an API gateway with live spending tracking broken down by model and team. You can see in real time that team A is spending most of its budget on a flagship model, even though many of its requests could be handled by Flash-Lite. Portkey logs every call, so you have an audit trail and can set alerts when a team exceeds a threshold.

Yotta AI Gateway is a unified access layer – you write once against one API, and underneath you can switch between OpenAI, Gemini, and Claude without changing application code. In an enterprise this matters because you're not vendor-locked and can test new models without refactoring. Adding semantic caching at the gateway level eliminates redundant calls entirely.

For large deployments, provisioned throughput is a must-have. OpenAI and Google offer dedicated throughput with guaranteed latency – you pay upfront for capacity, not for tokens. If you're doing 10M+ tokens monthly, provisioned throughput can be cheaper than pay-as-you-go and gives you predictable cost. You set budget controls at project level in Vertex AI: hard limit per team, soft alert at 80%, automatic throttling when you exceed. Without this governance, in an organization of 500+ people it's like driving without brakes – someone will run a batch job on a flagship model and eat your monthly budget over a weekend.

Practical Decision Framework and Optimization Checklist

When you have $5k monthly API costs, you can afford convenience. At $50k it's already a business problem requiring systematic approach. The critical threshold is when AI costs constitute >15% of R&D budget – then the premium for convenience stops being defensible.

Five-Step Optimization Checklist:

Token Audit (week 1): Identify top 20% of endpoints generating 80% of costs. Check prompt length, call frequency, traffic distribution between models.
Quick Wins – Caching (week 2-3): Implement context caching for repeated system prompts. If you send product documentation with every call, caching can cut the cost of those tokens dramatically[5]. Add semantic caching (e.g. Redis) for repeated user queries.
Model Routing (week 4-6): Route simple tasks to Gemini Flash-Lite or DeepSeek V3.2 on DeepInfra. In the example above, moving a workload from GPT-5.2 to DeepSeek V3.2 drops the bill from about $3,060 to about $115 a month. Keep the flagship model for complex edge cases.
Batch Processing (week 7): Move non-urgent jobs (reports, overnight analytics) to Batch Mode or Flex – about 50% off, at the cost of building an asynchronous pipeline[3][4].
Monitoring and Iteration (ongoing): Set up dashboards with three metrics: cost per request (target: <$0.01 for typical tasks), cache hit rate (target: >60% after a month), quality score (user satisfaction or evals – can't drop >5% during optimization).

Migration Roadmap for $5k-50k/month Teams (rough targets – actual savings vary a lot by workload):

Month 1: Audit + context caching → expected 30-40% savings
Month 2: Routing 50% traffic to cheaper models → another 20-30% savings
Month 3: Semantic caching + batch processing → additional 15-20%
Month 4+: Fine-tuning open-source models for specific tasks, self-hosting for highest volumes

Key principle: don't optimize everything at once. Start with biggest cost centers, measure impact, iterate. Teams trying to immediately migrate to self-hosted Llama lose 2-3 months on infra, while they could have saved 60% of costs through intelligent routing in a week.

Re-evaluate your strategy quarterly – prices for a given level of capability keep falling fast, and new models can change the economics overnight. What was cost-effective in Q1 may already be a legacy approach in Q3.

References

Comments

Comments are powered by Cusdis. The widget only loads after you click.