LLM Costs in Production: What Nobody Tells You Before You Launch

The demo costs nothing. That’s the problem.

Every LLM tutorial runs on a handful of test prompts that cost fractions of a cent. You launch, the users come, and the bill that follows is the first time most founders look seriously at production LLM pricing. By then you’ve already made the expensive architectural choices.

This is post 5 in the AI pillar series. The previous posts covered choosing the right LLM, when RAG is worth building, AI agents for internal tools, and AI automation for business. This one is about what that architecture actually costs when real users hit it.

The cost-per-1M table (the number every team needs to internalize)

Pricing changes frequently — check official docs before committing. These are approximate as of mid-2026:

Model	Input / 1M tokens	Output / 1M tokens	Notes
Gemini 1.5 Flash	~$0.075	~$0.30	Cheapest option for high-volume tasks
GPT-4o mini	~$0.15	~$0.60	Widely-used cheap classification model
Claude Haiku	~$0.25	~$1.25	Fast + nuanced; good for agent sub-steps
Gemini 1.5 Pro	~$1.25	~$5.00	Long docs, GCP-native; wide context window
Claude Sonnet	~$3.00	~$15.00	Complex agents, code, document reasoning
GPT-4o	~$5.00	~$15.00	Deepest ecosystem; most versatile
Claude Opus	~$15.00	~$75.00	Maximum reasoning quality; use selectively

The table reveals a 200× cost spread between the cheapest and most expensive options. Teams that route by task complexity use frontier models for the 10–20% of requests that justify it and cheap models for everything else. Teams that run one flagship model for everything pay frontier prices for work that doesn’t need it.

The worked example: 10,000 users per day

Suppose your AI feature generates a summary from user-submitted documents. Each request: ~2,000 input tokens (the document) and ~500 output tokens (the summary). At 10,000 requests/day:

With GPT-4o:

20M input tokens/day × ($5 / 1M) = $100/day
5M output tokens/day × ($15 / 1M) = $75/day
Total: ~$175/day → ~$5,250/month

With Claude Haiku:

20M input tokens/day × ($0.25 / 1M) = $5/day
5M output tokens/day × ($1.25 / 1M) = $6.25/day
Total: ~$11.25/day → ~$338/month

That is a 15× cost difference on the same task. If GPT-4o and Claude Haiku deliver equivalent quality on document summarization — test it on 50 real examples, they often do — that’s $4,900/month of avoidable cost at 10K users/day, scaling linearly as you grow.

This is also why model selection is an architectural decision, not just a tooling choice. Changing it after launch means auditing every prompt, re-running your evaluations, and possibly rebuilding error handling. Not a quick afternoon.

Prompt caching: the savings most teams miss

Both Anthropic (Claude) and Google (Gemini) offer prompt caching — the ability to cache the first tokens of a prompt and reuse them across requests at a fraction of the normal input cost.

When is this relevant? Whenever you have a large system prompt, a reference document, or a knowledge-base section that appears in every call. For agent workflows, this is almost always true — the agent’s instructions and context are typically the largest part of the input.

Claude’s caching model (approximate):

Cache write: ~25% premium over standard input cost
Cache read: ~90% discount off standard input cost
Cache TTL: 5 minutes, refreshes on each use within the window

The impact at scale: A Claude Sonnet agent with a 5,000-token system prompt running 10K requests/day:

Without caching: 50M system-prompt tokens/day × ($3 / 1M) = $150/day on the system prompt alone
With caching: full price for one write per cache window, then ~$0.30/1M for reads; system-prompt cost drops to ~$16/day

Cache hit rates above 80% can cut total Claude inference costs 40–60% on prompt-heavy workflows. This makes Claude more cost-competitive than raw per-token pricing suggests — especially for agent and document-processing use cases where the same context repeats every call.

Batching for async workloads

Not every LLM call needs to happen in real time. Document processing, report generation, nightly analysis, and background enrichment jobs can all run as batch workloads — and batch APIs are significantly cheaper.

OpenAI Batch API: ~50% discount on standard pricing; asynchronous, results returned within 24 hours.

Anthropic Message Batches: Similar discount and async model.

For any workload that doesn’t require a real-time user response, batching is free cost savings. A report-generation pipeline costing $300/month at real-time GPT-4o rates might cost $150/month on the Batch API with no quality change.

The engineering cost: you handle async callbacks instead of synchronous responses. In most workflow tools (n8n, Make, a simple queue worker), that’s a one-time change.

Three cost mistakes teams make after launch

1. One model for everything. Using GPT-4o for support ticket classification is like hiring a senior architect to answer every support email. The output quality ceiling for that task is lower than the model’s capability ceiling — you’re paying frontier prices for commodity work.

2. Paying for tokens you don’t use. Many LLM calls return far more tokens than necessary because the prompt doesn’t constrain output length. “Summarize this in 2–3 sentences” is both better UX and dramatically cheaper than an open-ended summary prompt that returns 1,000 tokens.

3. No cost observability. If you’re not tracking per-call token usage and mapping it to features and users, you have no signal on where optimizations have the highest impact. Token usage should be a first-class metric in your observability stack — not an afterthought you notice when the bill shocks you.

When self-hosting LLMs makes financial sense

The recurring question at every scale milestone: should we self-host?

The honest answer: almost never before $50K/month in API spend. Self-hosting a frontier-quality open-weight model (Llama 3.1, Mistral Large, Qwen 2.5) requires:

GPU instances: A100 80GB at ~$3–4/hour, or H100 at ~$5–8/hour; reliably serving 70B models typically needs 2–4 GPUs
Engineering setup: 2–4 weeks to configure an inference server (vLLM, TGI), monitoring, autoscaling, and failover
Ongoing ops: the infrastructure cost never disappears — it needs maintenance on every model update

At $10K/month in API cost, cloud API is almost certainly cheaper once you factor in engineering hours for setup and maintenance. At $100K/month, the math often flips — but verify against your actual usage pattern and model requirements.

The middle ground: managed inference providers (Together.ai, Fireworks.ai, Replicate) run open-weight models at rates competitive with the major API providers, without the ops burden. Worth evaluating before running your own.

The three questions to answer before launch

Before your AI feature goes live, three questions that prevent expensive surprises:

What is the realistic request volume at launch vs. 6-month projections? Run the token math at both levels. Make sure the 6-month projection doesn’t require a pricing conversation you haven’t had.
Does your primary use case actually need a frontier model? Test the cheap variant on 50 real examples from your users. The quality gap is smaller than you expect on structured tasks.
Is there a large repeated context in every call? If yes, prompt caching should be in your implementation plan from day one — it’s an architectural addition that’s easier to build in early than retrofit later.

The gap between demo cost and production cost is real, but entirely predictable. The founders who avoid the shock do the math before they build, not after the first bill arrives.

If you’re in the architecture phase and want a second opinion on cost structure before committing to a stack, a fractional CTO can help you scope it correctly. Book a call — no obligation, no sales pitch.

This is post 5 in the AI pillar series. Start with AI automation for business, continue through AI agents for internal tools, RAG, choosing the right LLM, and then this article. Next: Prompt engineering in production — the 6 patterns that improve LLM reliability and output consistency at scale. Then: AI observability in production — the 5 metrics and tooling that make your LLM features visible when something goes wrong at 2am.