What Running LLMs in Production Actually Costs (With Numbers)

Everyone talks about integrating AI into their products. Few talk about the bill that arrives at the end of the month.

At Soamee, we have spent two years building AI features for clients across different industries: from customer support chatbots to RAG systems for legal document processing. In that time, we have learned that the real cost of running LLMs in production has very little to do with what you imagine at first. The surprises almost always go in the wrong direction.

This is the article I wish I had when we started. Real numbers, concrete scenarios, and strategies that actually work.

Token Pricing in 2025-2026: The Table You Need

Before discussing scenarios, we need to be clear on base prices. The relevant models in production today are these:

One important clarification: the price per token is the cost of the tokens themselves. In real production, the system prompt (which repeats on every call), conversation history, and RAG context massively inflate the input token count. This is the most common mistake in cost estimates.

Real Scenarios: What Does It Actually Cost?

Let us run the numbers for the four most common use cases.

Scenario 1: Customer Support Chatbot (1,000 queries/day)

Typical parameters:

• System prompt: 500 tokens (instructions, tone, FAQ base)
• Average conversation history: 800 tokens (4-5 turns)
• User query: 100 tokens
• Model response: 300 tokens

Total per call: ~1,400 input tokens + 300 output tokens

For a simple chatbot at low-to-medium volume, GPT-4o mini, Claude Haiku, or Gemini Flash are perfectly valid options at minimal cost. The key question is whether the response quality is sufficient for your use case.

Scenario 2: Document Processing (500 docs/day, ~5 pages each)

Documents change the equation dramatically. An A4 page has roughly 400-500 words, which translates to around 500-600 tokens.

Parameters:

• 5-page document: ~2,500 input tokens
• Instructions prompt: 300 tokens
• Generated summary/extraction: 500 tokens

Total per document: ~2,800 input tokens + 500 output tokens

For document processing where precision is critical (contracts, medical, financial documents), Claude Sonnet or GPT-4o justify their premium. For structured data extraction from simpler documents, Gemini Flash offers an unbeatable ROI.

Scenario 3: Code Assistant (200 active developers/day)

A code assistant has a very different token profile: code snippets are token-dense (code with its syntax consumes more tokens per character than natural text).

Estimated parameters per session:

• Code context: 3,000 tokens
• Conversation history: 1,500 tokens
• System prompt: 400 tokens
• Response with code: 800 tokens
• Average: 4 interactions/session

Total per session: ~23,600 input tokens + 3,200 output tokens

For code assistants in mid-size teams, costs range from $370 to $714/month depending on the model. Not outrageous, but there is significant optimization headroom with project context caching.

Scenario 4: RAG System (document base + 2,000 queries/day)

RAG systems add a layer: retrieval brings relevant chunks from a vector database that get inserted into every call’s prompt.

Parameters:

• Retrieved chunks: 3 chunks × 400 tokens = 1,200 tokens
• System prompt: 600 tokens
• User query: 150 tokens
• Generated response: 400 tokens

Total per query: ~1,950 input tokens + 400 output tokens

In high-volume RAG, the gap between premium and economy models starts to matter a great deal.

The Hidden Costs Nobody Mentions

Tokens are just the tip of the iceberg. In real production, several cost categories get ignored in initial estimates:

Infrastructure and Observability

• Vector database: Pinecone starts at $70/month for a small index. Weaviate Cloud or Qdrant have cheaper options from $25/month. Self-hosted adds DevOps cost.
• Monitoring and tracing: Tools like LangSmith (from $39/month), Langfuse (open source, self-hosted), or Helicone (from $20/month) are essential in production. Without them, debugging a failing agent is a nightmare.
• API gateway and rate limiting: Managing rate limits, retries, and fallbacks requires either a SaaS solution or engineering time.

Latency and Its Indirect Cost

LLM latency is real and has business cost:

• GPT-4o: 500-1,500 ms time-to-first-token
• Claude Sonnet 4: 400-1,200 ms
• Gemini 2.5 Flash: 200-600 ms
• Llama 3.3 70B (self-hosted, good GPU): 300-800 ms

For conversational user interfaces, latency above 2 seconds has a direct impact on user satisfaction. Streaming mitigates the perception, but not the actual response time.

The indirect cost: if you have workers processing in batch and average latency is 1 second per call, for 100,000 calls/day you need enough concurrency to avoid bottlenecks. More concurrency means more infrastructure.

Error Handling and Retries

LLMs fail. OpenAI and Anthropic APIs have high uptime SLAs (>99.9%), but in production with thousands of daily calls, errors happen. A robust system needs:

• Retry logic with exponential backoff
• Circuit breakers to prevent failure cascades
• Fallback to alternative models when the primary is unavailable

This engineering has a development cost (typically 2-4 weeks to do it properly) and infrastructure cost (if you maintain redundancy across providers).

Cost of Failed Calls

Calls that return an error, timeout, or invalid response also consume tokens (partially) or compute time on self-hosted. In a poorly optimized system, 5-10% of calls can be “waste.” At 10,000 calls/day, that is 500-1,000 wasted calls per day.

Optimization Strategies: How to Cut the Bill 40-70%

After working on several production systems, these are the strategies that actually work:

1. Semantic Caching (potential savings: 30-60%)

Semantic caching goes beyond traditional exact-match response caching. Instead of looking for exact string matches, it uses embeddings to find similar queries and reuse the response if similarity exceeds a threshold.

Typical implementation:

1. On receiving a query, generate its embedding (cost: fraction of a cent)
2. Search the embedding cache for something similar (cosine similarity > 0.92)
3. If hit, return the cached response without calling the LLM
4. If miss, call the LLM and cache the result

In chatbots with repetitive queries (support, FAQs), hit rates can reach 40-60%. In RAG systems with more diverse questions, 15-25%. Tools like GPTCache or custom implementations on Redis are viable options.

2. Model Routing (potential savings: 40-70%)

Not every query needs the same model. A routing system classifies the complexity of each query and sends it to the appropriate model:

• Simple query (“What are your opening hours?”) → Gemini Flash or GPT-4o mini
• Medium query (“Explain the contract terms”) → Claude Haiku or GPT-4o mini
• Complex query (“Analyze these three contracts and give me the key differences”) → Claude Sonnet or GPT-4o

The classifier can be another small LLM (minimal cost) or a traditional classifier trained on examples from your domain. In systems with a typical query distribution (70% simple, 20% medium, 10% complex), the savings are substantial.

3. Prompt Optimization (potential savings: 20-40%)

The system prompt is one of the biggest token inflation culprits. We have seen 2,000-3,000 token prompts that could be reduced to 400-600 without losing effectiveness. Some techniques:

• Eliminate redundancies: “You are a professional and friendly customer service assistant who always responds politely and professionally” → “Support assistant. Tone: professional and direct.”
• Use compact structures: Lists instead of paragraphs, YAML instead of descriptive text
• Move static info to context caching (Anthropic and Google offer reduced prices for cached prompts used across many calls)

Anthropic’s context caching charges the system prompt at a reduced rate (~$0.30/1M tokens vs $3.00/1M on Sonnet) when used across multiple consecutive calls. For a 1,000-token system prompt with 10,000 calls/day, the savings are substantial.

4. Batching and Async Processing

For tasks that do not require real-time responses (document processing, report generation, analysis), batch processing enables:

• Using OpenAI’s Batch API (50% discount in exchange for up to 24-hour latency)
• Optimizing GPU utilization in self-hosted setups
• Better rate limit management without additional queue infrastructure

For 500 documents/day with no immediate result requirement, OpenAI’s Batch API cuts the bill in half.

5. RAG Context Optimization

In RAG systems, how much context you pass to the LLM is critical:

• Precise retrieval: Improve retrieval quality to recover only the most relevant chunks (3 good chunks > 10 mediocre ones)
• Optimized chunking: Smaller, more precise chunks reduce tokens without sacrificing quality
• Reranking: A cheap reranking model (e.g., Cohere Rerank) improves chunk selection before passing them to the LLM

Self-hosted vs API: When Each Makes Sense

The question we get most often: “Wouldn’t it be cheaper to host the model ourselves?”

The honest answer: it depends, but for most companies the commercial API wins until volumes get high.

Self-hosting Costs (Llama 3.3 70B as reference)

To run Llama 3.3 70B with production-grade performance you need:

• Required GPU: Minimum 2× A100 80GB or equivalent for good throughput
• AWS cost: p4d.24xlarge ($32/hour) or g5.48xlarge ($16/hour)
• Dedicated instance 24/7: $32 × 24 × 30 = ~$23,000/month (AWS on-demand)
• With Reserved Instances (1 year): Can drop to ~$11,000-14,000/month
• Spot alternative: Possible, but instability in production requires additional management

More economical self-hosting alternatives:

• RunPod or Lambda Labs: $2-4/hour for A100, versus $8-16 on AWS. For intermittent workloads, much more viable.
• Ollama + own server: For very low volumes or internal use, a server with 2× RTX 4090 (cost ~$3,000) can pay for itself in 6-12 months if the volume justifies it.

Break-even Point

Break-even between API and self-hosting (assuming RunPod at $3/hour for Llama 70B equivalent to Claude Haiku in quality):

• RunPod 24/7: ~$2,160/month
• Claude Haiku equivalent: $0.0023/call × N calls/month

Break-even: 2,160 / 0.0023 ≈ 940,000 calls/month (~31,000 calls/day)

Below that volume, the commercial API is cheaper when you count the real cost of self-hosting (infrastructure, DevOps, model updates, monitoring). Above it, self-hosting starts to make financial sense.

There are cases where self-hosting wins regardless of volume:

• Highly sensitive data that cannot leave your infrastructure (healthcare, regulated finance)
• Ultra-low latency where milliseconds matter and you have the GPU close to the user
• Intensive fine-tuning where you need a model highly specialized for your domain

The Real Cost of a Complete AI Project

For broader perspective, here are typical total cost ranges (not just tokens) for different project types:

Basic support chatbot (1,000 queries/day):

• Tokens: $12-240/month depending on model
• Infrastructure (hosting, vector DB): $50-100/month
• Monitoring: $20-40/month
• Total: $80-380/month

Departmental RAG system (500 users, 10K-document base):

• Tokens: $200-800/month
• Vector DB (Pinecone or similar): $70-200/month
• Additional infrastructure: $100-200/month
• Monitoring and observability: $40-80/month
• Total: $410-1,280/month

AI agent with integrations (200 active users, multiple tools):

• Tokens: $500-2,000/month (agents consume more tokens due to tool calls)
• Infrastructure: $200-500/month
• Advanced monitoring: $80-150/month
• Total: $780-2,650/month

These ranges assume optimized architectures. Without optimization, costs can be 2-3× higher.

Conclusion: AI Is Cost-Effective If Designed Properly

AI in production is not cheap if done poorly, but it is not prohibitive either if designed thoughtfully. The difference between a naive implementation and an optimized one can be 5-10× in costs.

The principles I apply to every project:

1. Start with the cheapest model that solves the problem. Scale up only when there is evidence you need more.
2. Measure everything from day one: tokens per call, error rate, latency, cost per user. Without metrics, you cannot optimize.
3. Implement caching from the start, not as an afterthought. It is the highest-impact optimization.
4. Budget for infrastructure in addition to tokens: the most common mistake in initial estimates.
5. Review the bill monthly and correlate with actual usage. Cost spikes usually reveal bugs or unexpected usage.

If you are evaluating integrating AI into your product or process and want an honest estimate tailored to your use case, that is exactly what we do at Soamee. You can review our AI services or contact us directly.

You might also find our article on how to build a custom AI agent with LLM for customer service useful if you already have a clear use case and want to understand the architecture.