An ungoverned LLM bill in production multiplies by 10 within a quarter. Real support-chatbot case: $800/month became $8k in three months while traffic stayed flat. Every request landed on the priciest model in the catalog, no cache, no routing. Three levers brought the invoice down to $2.4k with flat eval scores: model routing, prompt caching, and batching.

Model routing: match the model to the request

Frontier models cost up to 50x more per token than small ones. A GPT-4-class model charges $3-15 per million output tokens; Claude Haiku or Gemini Flash charges cents. Most production traffic consists of simple lookups, classification, and short extraction, and those tasks run fine on small models.

Keep the router dumb on purpose: regex rules for obvious intents plus a tiny classifier for everything else. Simple requests hit cheap models; hard reasoning escalates to the expensive tier. Across projects I measured, routing alone cut 40-70% of spend.

Prompt caching: stop paying full price twice

Anthropic and OpenAI sell cached prefixes at a deep discount on input. Front-load everything stable: long system prompts, few-shot examples, tool definitions. When 80% of the prompt repeats between calls, full price applies to the variable tail alone.

  • Order matters: the shared prefix must match byte for byte up to the variable part
  • Mark cacheable blocks with cache_control on the Anthropic API
  • Caches expire in minutes (5-60 depending on provider); steady traffic keeps them warm

Semantic caching: reuse whole answers

One level above prompt caching sits semantic caching. Embed the incoming query, compare cosine similarity against past queries, and serve the stored answer above threshold (0.95 tends to be safe). Redis with vector search or pgvector ships this in an afternoon.

Two hard rules apply. Set TTLs per knowledge type and invalidate on every data update. Leave personalized or session-scoped responses out of the cache; leaking one user's context into another's answer turns a saving into a security incident.

Batch endpoints for async work

Summarization, CRM enrichment, embedding backfills: none of these need sub-second responses. Batch APIs (OpenAI Batch API, Anthropic Message Batches) charge about 50% less in exchange for completion windows up to 24 hours. Moving overnight jobs to batch was the cheapest win of the project.

Spend observability

Invisible cost grows unchecked. Track tokens per feature and per cohort, with anomaly alerts firing in hours rather than days. Somebody always finds the endpoint burning money; the open question is whether discovery happens through the dashboard or through the invoice.

The support chatbot ended at $2.4k/month, down from $8k, with stable eval scores. LLM cost governance is ongoing engineering work, with the spreadsheet open beside Grafana.