LLM costs aren't what you think: optimization strategies that actually work

Everyone's worried about LLM costs. GPT-4 at $0.03 per 1K tokens adds up fast when you're processing millions of requests. But most cost optimization advice focuses on the wrong things.

We recently helped a SaaS client reduce their monthly LLM spend from $12,000 to $3,200. Here's what actually worked.

The wrong optimizations

Most teams start with these:

• Switching to a cheaper model (GPT-3.5 instead of GPT-4)
• Shortening prompts to reduce token count
• Caching responses aggressively

These work, but they're table stakes. The real savings come from architectural changes.

What actually worked: query routing

Not every query needs GPT-4. In fact, most don't. We built a query classifier that routes requests to the cheapest model that can handle them:

• Simple lookups and formatting: GPT-3.5 ($0.0005/1K tokens)
• Summarization and extraction: Claude Haiku ($0.00025/1K tokens)
• Complex reasoning and code: GPT-4 ($0.03/1K tokens)
• Very simple tasks: fine-tuned open-source models (free after initial training)

The classifier itself is a small, fast model (fine-tuned BERT) that adds 50ms latency but saves 60% on costs. The key insight: most user queries are simple, but teams default to the most powerful model because it's easier than building a router.

Prompt compression that actually works

Prompt compression tools claim 50% token savings but often degrade quality. We found better results with manual optimization:

• Remove hedging language ('you are a helpful assistant that...' → direct instructions)
• Use structured outputs (JSON mode) instead of asking the model to format responses
• Pre-compute what you can (don't ask the model to calculate dates, just pass them in)
• Use few-shot examples sparingly (each example adds tokens—only include if it measurably improves quality)

We cut average prompt length from 800 tokens to 340 tokens without losing quality. The trick was measuring quality rigorously—automated evals caught when compression hurt performance.

Semantic caching

Exact-match caching misses most opportunities. 'What's the weather in NYC?' and 'Weather forecast New York City' are the same query semantically but different strings.

We built semantic caching:

• Embed the query (cheap, ~10ms)
• Check vector DB for similar queries (cosine similarity > 0.92)
• If found, return cached response (skip the LLM entirely)
• If not found, query LLM and cache the result

This gave us a 34% cache hit rate on production traffic. The threshold (0.92) was tuned empirically—too high and you miss valid matches, too low and you return stale/wrong answers.

Batch processing

Real-time inference is expensive. For non-urgent tasks, batch processing is 50% cheaper:

• Summarization of daily reports: batch process at 2am instead of real-time
• Sentiment analysis of support tickets: batch process hourly
• Content generation for scheduled posts: batch process the night before

The API providers (OpenAI, Anthropic) offer batch APIs at half price. The trade-off is latency, but most background tasks don't need instant responses.

The results

After implementing these changes:

• Monthly cost: $12,000 → $3,200 (73% reduction)
• Average latency: unchanged (routing adds 50ms, caching saves 2-5 seconds)
• Quality: unchanged (measured by human evaluation on 500 test queries)
• Implementation time: 3 weeks

What didn't work

We tried several things that didn't move the needle:

• Fine-tuning for every task (only worth it for high-volume, well-defined tasks)
• Local open-source models (inference costs were similar after accounting for GPU time)
• Aggressive quantization (quality degradation wasn't worth the savings)

The takeaway

LLM cost optimization is an architectural problem, not a prompt engineering problem. The biggest wins come from routing, caching, and batching—not from tweaking prompts.

Start with measurement. Know your cost per query, your cache hit rate, your quality metrics. Then optimize systematically.