Introduction
Enterprises that embed large language models (LLMs) into SaaS products often see a surprising amount of waste: many requests are exact duplicates or minor paraphrases of earlier queries, driving up token consumption and cloud bills.
Semantic cache proxy is defined as a middleware layer that intercepts LLM calls, searches for semantically similar prior responses, and returns a cached answer when the similarity threshold is met.
Why Duplicate LLM Calls Matter
Every LLM request incurs token‑based pricing. When 30%‑60% of queries are repeats, the cost multiplier becomes significant, especially at scale. Reducing these redundant calls directly lowers monthly spend and improves response latency for end‑users.
Building the Proxy with FastAPI and OpenTelemetry
FastAPI provides an async‑first framework that pairs naturally with the high‑throughput demands of LLM traffic. By instrumenting the service with OpenTelemetry GenAI Semantic Conventions (v1.26+), you gain end‑to‑end traces that capture prompt, response, token counts, and incurred cost.
- Set up a FastAPI app with async endpoints that forward requests to the upstream LLM provider.
- Integrate the OpenTelemetry SDK, defining spans for request receipt, cache lookup, and external API call.
- Export traces to your observability stack (Jaeger, Prometheus, etc.) for real‑time monitoring.
This approach follows the guide “Stop Paying $0.05 Per Trace: Building an OTel‑Native LLM Proxy” which demonstrates a zero‑lock‑in, self‑hosted proxy using Python (FastAPI + AsyncIO) and native OpenTelemetry support.
Implementing Semantic Caching with Embeddings and Redis
The core of cost reduction lies in recognizing when a new prompt is semantically equivalent to a prior one. The typical workflow is:
- Generate an embedding for the incoming prompt using a lightweight model.
- Query a Redis vector store for the nearest neighbor within a similarity threshold.
- If a match is found, return the cached LLM response; otherwise, forward the request upstream and store the new result.
The GitHub repository “ramsinghal7/llm-cost-optimization-proxy” showcases a lightweight FastAPI middleware that applies three independent, toggleable strategies—semantic caching, prompt compression, and model routing—to cut per‑query cost (Aug 14 2026).
Observability and Cost Reporting with OpenTelemetry
With OpenTelemetry instrumentation in place, every request emits attributes such as genai.prompt_tokens, genai.completion_tokens, and genai.cost_usd. Aggregating these metrics lets you:
- Identify high‑frequency query patterns that benefit most from caching.
- Track cost savings over time and report ROI to stakeholders.
- Set alerts for abnormal token spikes that may indicate misuse.
Integrating OpenTelemetry also future‑proofs your architecture, allowing seamless migration to new LLM providers without losing observability continuity.
Frequently Asked Questions
What is the difference between exact caching and semantic caching?
Exact caching returns a stored response only when the incoming prompt matches a previous prompt byte‑for‑byte. Semantic caching uses vector similarity, so paraphrased or re‑ordered prompts can still hit the cache.
Do I need a separate embedding model for the cache layer?
Yes, a lightweight embedding model (e.g., OpenAI’s text‑embedding‑ada‑002) provides fast vector generation without adding noticeable latency.
Can the proxy work with multiple LLM providers?
Absolutely. By abstracting the outbound call behind a provider‑agnostic interface, you can route requests to OpenAI, Anthropic, Mistral, or any compatible API.
How does OpenTelemetry handle sensitive data?
OpenTelemetry lets you redact or hash sensitive attributes before exporting. Follow best practices to mask PII in prompts and responses.
Is Redis the only option for vector storage?
No. Alternatives like Pinecone, Milvus, or Qdrant can be used, but Redis offers low‑latency in‑memory storage that fits well with FastAPI’s async model.
Conclusion
By combining FastAPI, OpenTelemetry, and semantic caching, you can dramatically reduce duplicate LLM calls, lower operational costs, and gain deep visibility into AI‑driven services. Neptune Infotech can help you design and implement such high‑impact solutions tailored to your business needs.
