Every token your AI application sends to a model provider costs money. Every token it receives back costs more. At scale, these costs compound into the single largest line item in your AI infrastructure budget. A 2025 Andreessen Horowitz survey of AI-native companies found that model inference costs consumed an average of 32% of total cloud spending -- up from 12% just two years earlier.
The good news: most applications waste 40-70% of their tokens. Verbose system prompts, uncompressed context, unrestricted output lengths, and missing caches all contribute to massive inefficiency. Optimizing token usage is the highest-leverage cost reduction strategy available to AI engineering teams.
This guide is a technical deep-dive into token optimization for production AI applications. We'll cover prompt engineering, context management, response optimization, and architectural patterns that reduce token consumption without sacrificing output quality.
Understanding Token Economics
How Tokenization Works
Language models don't process text as characters or words -- they process tokens. A token is roughly 3-4 characters in English, or about 0.75 words. However, tokenization is not uniform:
- Common English words are typically one token ("the", "and", "with").
- Less common words may be split into multiple tokens ("cryptocurrency" = 3 tokens).
- Code is generally less token-efficient than prose (more special characters, indentation).
- Non-English languages often require more tokens per word.
- JSON and structured data formats consume more tokens than equivalent plain text due to structural characters (braces, brackets, quotes, colons).
Understanding tokenization helps you estimate costs and find optimization opportunities. Most providers offer tokenizer tools (OpenAI's tiktoken, Anthropic's token counter) that let you calculate exact token counts for any text.
The Pricing Asymmetry
Most providers charge significantly more for output tokens than input tokens:
| Model | Input (per 1M tokens) | Output (per 1M tokens) | Output Premium | |-------|----------------------|------------------------|----------------| | Claude Opus | ~$15 | ~$75 | 5x | | GPT-4o | ~$2.50 | ~$10 | 4x | | Claude Sonnet | ~$3 | ~$15 | 5x | | GPT-4o-mini | ~$0.15 | ~$0.60 | 4x | | Claude Haiku | ~$0.25 | ~$1.25 | 5x |
This asymmetry means that reducing output tokens has 4-5x more cost impact than reducing input tokens. If you can cut output tokens by 50%, you've achieved the equivalent savings of eliminating all input tokens entirely (for similarly sized inputs and outputs).
The Hidden Costs
Token costs extend beyond the per-token price:
- **Latency.** More tokens means slower responses. Each output token is generated sequentially, so halving output tokens roughly halves response time.
- **Rate limits.** Most providers impose tokens-per-minute limits. Wasteful token usage means you hit rate limits sooner, limiting throughput.
- **Context window pressure.** Bloated prompts consume context window space that could be used for more relevant information, degrading output quality.
- **Memory and bandwidth.** Token-heavy requests and responses consume more memory in your application layer and more bandwidth across your infrastructure.
Prompt Engineering for Token Efficiency
System Prompt Optimization
The system prompt is included in every request, making it the highest-leverage optimization target. A system prompt that's 500 tokens too long costs you those 500 tokens on every single API call.
**Audit your system prompts.** Review every instruction in your system prompt and ask: is this actually changing the model's behavior? Remove instructions that the model follows by default.
Before optimization (847 tokens): ``` You are a helpful customer support assistant for Acme Corp. You should always be polite and professional. You should always try to help the customer. If you don't know the answer, you should say so. You should not make up information. You should respond in a clear and concise manner. You should use proper grammar and spelling. You should not use profanity or offensive language. When greeting the customer, you should say hello and ask how you can help... ```
After optimization (312 tokens): ``` You are Acme Corp's support assistant. Answer customer questions using the provided knowledge base. If unsure, say so and offer to escalate. Keep responses under 3 sentences unless the question requires more detail. Format: greeting (first message only), answer, follow-up question. ```
The optimized version is 63% fewer tokens while being more specific and actionable. The removed instructions ("be polite", "use proper grammar", "don't use profanity") describe default model behavior and add cost without value.
**Use structured instruction formats.** Models follow structured instructions more reliably, meaning fewer retries and less need for verbose explanation:
``` ROLE: Customer support for Acme Corp KNOWLEDGE: {knowledge_base_context} CONSTRAINTS:
- Max 3 sentences unless complexity requires more
- Cite knowledge base articles by ID
- Escalate billing issues to human agents
FORMAT: Greeting (first message) > Answer > Follow-up question ```
**Version and test your prompts.** Track token counts for every system prompt version. Set a token budget for system prompts and treat it as seriously as you'd treat a performance budget.
Dynamic Prompt Construction
Don't use a one-size-fits-all system prompt. Construct prompts dynamically based on the specific request:
- **Task-specific prompts.** If your AI handles multiple tasks (support, sales, onboarding), use different system prompts for each task type. A support prompt doesn't need sales instructions and vice versa.
- **Progressive context loading.** Start with minimal context and add more only if the conversation requires it. Don't load your entire product catalog into the system prompt when the user might just need help resetting their password.
- **Conditional instructions.** Only include instructions relevant to the current state. If the user is authenticated, you don't need authentication-related instructions. If it's a follow-up message, you don't need the greeting format.
Few-Shot Example Optimization
Few-shot examples are powerful but expensive. Each example consumes hundreds of tokens. Optimize them:
- **Minimize example count.** For well-defined tasks, 1-2 examples often perform as well as 5-6. Test to find the minimum effective number.
- **Shorten examples.** Use the shortest examples that demonstrate the desired behavior. Long examples don't necessarily teach the model better.
- **Use example selection.** Instead of including all examples in every prompt, select the most relevant examples based on the input query using embedding similarity. This approach (sometimes called dynamic few-shot) dramatically reduces token usage while maintaining or improving quality.
- **Consider fine-tuning.** If you're spending thousands of tokens on few-shot examples per request, fine-tuning the model to internalize those patterns may be more economical. The examples move from per-request input tokens to a one-time training cost.
Context Management Strategies
Conversation History Compression
Multi-turn conversations are one of the biggest token sinks. Each new message includes the entire conversation history, and costs grow quadratically with conversation length.
**Sliding window.** Keep only the most recent N messages in the context. Simple but effective. For most applications, the last 5-10 messages provide sufficient context.
**Summarization.** Periodically summarize older messages into a compact summary that's included in the context instead of the full messages. A 20-message conversation might be 4,000 tokens; a summary of those messages might be 200 tokens.
Implementation pattern: 1. After every 10 messages, generate a summary of the conversation so far. 2. Replace the older messages with the summary in subsequent requests. 3. Keep the most recent 5 messages in full detail. 4. Store the full conversation history separately for audit and retrieval if needed.
**Selective history.** Not all messages are equally relevant. Use embedding similarity to select the most relevant previous messages for the current query, rather than including all messages chronologically.
RAG Context Optimization
Retrieval-Augmented Generation (RAG) systems often over-retrieve, stuffing the context with marginally relevant documents. Optimize retrieval for token efficiency:
- **Retrieve chunks, not documents.** Chunk your knowledge base into 200-500 token segments and retrieve only the most relevant chunks. Retrieving entire documents wastes tokens on irrelevant sections.
- **Re-rank before including.** Use a re-ranking model (e.g., Cohere Rerank, cross-encoder models) to re-order retrieved chunks by relevance, then include only the top chunks.
- **Set a token budget for context.** Allocate a fixed token budget for retrieved context (e.g., 2,000 tokens) and fill it with the highest-ranked chunks until the budget is exhausted.
- **Deduplicate.** If multiple retrieved chunks contain overlapping information, deduplicate before including them in the prompt.
- **Extract, don't include.** For structured data, extract the relevant fields rather than including the entire record. If the user asks about a product's price, include the price field, not the entire product specification document.
For a broader perspective on managing AI costs through intelligent architecture, see our guide to [reducing AI costs with intelligent model routing](/blog/reduce-ai-costs-intelligent-model-routing).
Tool and Function Call Optimization
Tool-use and function-calling features send schema definitions with every request. These schemas can be surprisingly expensive:
- A typical function schema is 100-300 tokens.
- If you define 20 functions, that's 2,000-6,000 tokens per request just for schemas.
- These tokens are charged on every request, even if no function is called.
**Optimize function schemas:**
- Use concise parameter names and descriptions.
- Remove unnecessary parameters.
- Use enums instead of free-text parameters where possible.
- Only include functions that are relevant to the current conversation state.
**Dynamic function loading:**
- Start with a minimal function set.
- Add functions based on the conversation context.
- Remove functions that are no longer relevant.
Response Optimization
Controlling Output Length
Since output tokens cost 4-5x more than input tokens, controlling output length is the highest-impact optimization:
**Set max_tokens.** Always set a `max_tokens` parameter appropriate for the expected response length. Default values are often much higher than necessary.
| Use Case | Recommended max_tokens | |----------|----------------------| | Classification | 10-50 | | Short answer | 100-200 | | Paragraph response | 200-500 | | Detailed explanation | 500-1,000 | | Long-form content | 1,000-2,000 |
**Instruct conciseness in the system prompt.** "Respond in 1-2 sentences" is more effective than "Be concise." Specific length instructions dramatically reduce output tokens.
**Use structured output formats.** JSON output is more predictable in length than free-text output. When you need structured data, use JSON mode or function calling to get precisely the fields you need without narrative wrapping.
**Stop sequences.** Configure stop sequences to terminate generation when the model produces a known end marker. This prevents the model from generating additional content after it's answered the question.
Output Format Optimization
The format you request directly impacts token count:
- **Markdown vs. plain text.** Markdown headers, bullet points, and formatting characters add tokens. If the output doesn't need formatting (e.g., it feeds into another system), request plain text.
- **JSON vs. natural language.** For data extraction tasks, JSON output with specific fields is typically more token-efficient than asking the model to describe the data in natural language.
- **Structured vs. free-form.** Define the exact output structure you need. "Return a JSON object with fields: sentiment (positive/negative/neutral), confidence (0-1), summary (max 20 words)" produces much more predictable and efficient output than "Analyze the sentiment of this text."
Batch Processing
When processing multiple items, batching is significantly more token-efficient than individual requests:
**Single-request batching.** Process multiple items in a single API call:
Instead of 10 requests, each with the system prompt + one item (10x system prompt tokens), send one request with the system prompt + all 10 items (1x system prompt tokens). This alone can reduce total input tokens by 50% or more for batch workloads.
**Batch API endpoints.** Most providers offer batch API endpoints with lower per-token pricing (typically 50% off). If latency is not critical, batch processing can halve your costs on top of the token reduction.
**Parallel structured output.** When analyzing multiple items, request structured output that covers all items in a single response rather than generating separate analyses for each item.
Architectural Patterns for Token Efficiency
Tiered Processing
Not every request needs the same model or the same level of processing. Implement a tiered architecture:
**Tier 1: Pattern matching.** Handle common, predictable requests with traditional code (regex, lookup tables, decision trees). Zero AI tokens consumed.
**Tier 2: Lightweight models.** Route simple AI tasks to the smallest capable model (GPT-4o-mini, Claude Haiku). These models are 10-100x cheaper per token than premium models.
**Tier 3: Premium models.** Reserve expensive models for complex tasks that genuinely require their capabilities.
A well-implemented tier system can reduce total token spending by 60-80%. See our guide on [intelligent model routing](/blog/reduce-ai-costs-intelligent-model-routing) for a detailed implementation guide.
Semantic Caching
Cache AI responses to avoid generating them repeatedly. Unlike traditional caching (exact key match), semantic caching matches requests by meaning:
1. Compute an embedding of the incoming request. 2. Search your cache for requests with embeddings above a similarity threshold. 3. If a match is found, return the cached response (zero generation tokens). 4. If no match, generate a new response and cache it.
Effective semantic caching can reduce AI API calls by 20-40% for applications with repetitive query patterns (customer support, FAQ bots, internal knowledge assistants).
We cover caching strategies in depth in our guide on [AI caching strategies for cost reduction](/blog/ai-caching-strategies-cost-reduction).
Prompt Compilation
For applications that use the same system prompt across many requests, some providers offer prompt caching features:
- **Anthropic's prompt caching** caches the system prompt and reuses it across requests, reducing input token costs by up to 90% for cached content.
- **OpenAI's cached completions** provide similar functionality.
These features are essentially free token savings for applications with stable system prompts. If your provider supports prompt caching, enable it immediately.
Measuring and Monitoring Token Usage
You can't optimize what you don't measure. Implement comprehensive token monitoring:
Key Metrics
- **Tokens per request** (input and output, separately). Track the distribution, not just the average.
- **Tokens per task type.** Different tasks have different token profiles. Monitor each independently.
- **System prompt tokens as percentage of total input.** If system prompt tokens are more than 30% of total input tokens, they're likely over-engineered.
- **Cache hit rate.** What percentage of requests are served from cache?
- **Token waste ratio.** Compare actual output tokens to the useful information in the output. If the model generates 500 tokens but only 100 are used by your application, you're wasting 80% of output tokens.
- **Cost per business action.** Translate token counts into dollars per action (cost per customer support resolution, cost per document analysis, cost per lead qualification).
Optimization Workflow
1. **Baseline.** Measure current token usage across all endpoints and task types. 2. **Identify waste.** Look for the highest-volume, highest-token endpoints. 3. **Implement optimizations.** Apply the techniques in this guide, starting with the highest-impact opportunities. 4. **Measure impact.** Compare post-optimization metrics to the baseline. 5. **Iterate.** Token optimization is ongoing. Models change, usage patterns shift, and new optimization techniques emerge.
For a broader framework on managing AI costs, including token usage as part of total cost of ownership, see our [total cost of ownership guide for AI platforms](/blog/total-cost-ownership-ai-platforms).
Real-World Optimization Results
Here are typical results from applying these techniques to production AI applications:
| Optimization | Token Reduction | Quality Impact | |-------------|----------------|----------------| | System prompt audit | 30-60% of system prompt tokens | None (removes redundant instructions) | | Conversation summarization | 50-70% of history tokens | Minimal (key information preserved) | | RAG context optimization | 40-60% of retrieval tokens | Positive (less noise, more relevant context) | | Output length control | 30-50% of output tokens | Depends on calibration | | Dynamic function loading | 60-80% of schema tokens | None | | Semantic caching | 20-40% of total requests eliminated | None (exact or near-exact matches) | | Model tiering | 60-80% cost reduction | Minimal with proper routing |
Combined, these optimizations typically reduce AI costs by 50-75% with no degradation in end-user experience.
Start Optimizing with Girard AI
Girard AI's platform includes built-in token optimization features: intelligent model routing, semantic caching, prompt management, and detailed token analytics. Our dashboard shows you exactly where your tokens are going and identifies the highest-impact optimization opportunities.
Stop overpaying for AI inference. [Sign up for Girard AI](/sign-up) to see your token optimization opportunities, or [contact our team](/contact-sales) for a personalized cost reduction analysis based on your current usage patterns.