The Starting Point
The client — a mid-market B2B SaaS company with roughly 15,000 active customer accounts and a support team of 22 agents — came to us with a familiar problem dressed up as a familiar solution. Ticket volume had grown faster than headcount for three straight quarters, average first-response time had crept past four hours, and leadership had already convinced itself that "an AI chatbot" was the fix. Someone on the team had built a weekend prototype using a popular LLM API and a basic retrieval-augmented generation (RAG) setup pointed at the help center. It answered questions. It also, on a bad day, confidently invented a refund policy that didn't exist and quoted API rate limits that were two versions out of date.
That gap — between "it works in a demo" and "it works reliably enough to put in front of paying customers" — is where this project actually started.
The Real Requirements, Once We Dug In
The initial ask was simple: "build us a support chatbot." The requirements that actually mattered took a few weeks of discovery to surface properly, and they shaped almost every architectural decision that followed:
- Answers had to be grounded strictly in current documentation — no exceptions, no "helpful" improvisation when the docs didn't cover a question.
- The system needed to know when it didn't know something, and escalate cleanly to a human agent with full conversation context, rather than guessing.
- It had to handle three tiers of documentation (public help center, tier-2 internal runbooks, and account-specific configuration data) without ever leaking tier-2 or account-specific information across customer accounts.
- Whatever we built had to run within a cost envelope the client could defend to their CFO — the weekend prototype, projected out to full ticket volume, would have cost more per month than two additional support hires.
That last point ended up being the defining constraint of the entire project, which is part of why the eventual cost reduction became as central to the story as the accuracy improvements.
Why the Prototype Broke Down at Scale
The original prototype followed a pattern that's extremely common, and extremely expensive once it hits real volume: every single query, regardless of complexity, went through the same large frontier model, with the same large retrieved context window stuffed into every prompt whether the question needed it or not. A one-line question like "what's your refund window" retrieved and re-processed the same volume of context as a genuinely complex multi-part technical question. At 15,000 accounts and growing ticket volume, that's not a rounding error — it's the entire cost structure of the product.
There was a second, quieter problem: retrieval quality. The prototype's RAG setup used a single flat index over all documentation with no distinction between document types, recency, or specificity. It reliably surfaced the most semantically similar chunk of text — which is not the same thing as the most correct or current chunk of text. Outdated documentation ranked just as highly as current documentation, which is exactly how the system ended up confidently quoting deprecated rate limits.
What We Actually Built
A Tiered Retrieval Architecture, Not a Single Index
Instead of one flat vector index, we restructured retrieval into separate, purpose-specific indexes: public documentation, internal runbooks, and account-specific configuration, each with its own freshness and access-control rules. A metadata layer tagged every document with recency and authority signals, so retrieval could actively down-rank stale content rather than treating a two-year-old changelog entry as equally valid to last week's update. This alone meaningfully reduced the outdated-information problem before we touched anything related to model selection.
Model Routing Instead of One-Size-Fits-All
This was the single biggest lever for both cost and quality. Rather than sending every query to the same large model, we built a lightweight classification step that routes each incoming query based on complexity and risk:
- Simple, high-confidence factual queries (account status, basic policy questions) route to a smaller, faster, considerably cheaper model, with retrieval scoped tightly to the most relevant one or two document chunks.
- Multi-step or ambiguous queries route to a larger model with a wider retrieval window and more careful prompt construction.
- Anything touching billing disputes, account cancellations, or security-sensitive topics routes directly to a human agent, with the system generating a context summary rather than attempting an answer at all.
This routing layer is a genuinely different way of thinking about cost than "pick the cheapest model that mostly works." It's closer to how a well-run support team already operates — junior agents handle routine tickets, complex cases escalate to someone senior — just implemented as an engineering pattern instead of an org chart.
Aggressive Context Trimming and Semantic Caching
We rebuilt the retrieval pipeline to pull only the specific passages relevant to a query rather than whole documents, cutting the average token count per request substantially. On top of that, we added a semantic caching layer: since a large share of support queries are close paraphrases of questions the system has already answered well, cached responses for semantically similar queries could be served without a fresh model call at all, after a lightweight relevance check confirmed the cached answer still applied.
A Verification Layer Before Anything Reaches a Customer
Every generated answer passes through a lightweight verification step that checks the response against the retrieved source passages before it's shown to a user. If the generated answer contains claims that aren't traceable back to the retrieved context, the system either regenerates with tighter constraints or escalates to a human — it does not show an ungrounded answer to a customer, full stop. This single guardrail did more to rebuild the client's confidence in the system than any accuracy metric we could report.
The Results
After a phased rollout — first to a single product line, then expanded to the full customer base over two months — the numbers looked like this:
- ~40% reduction in inference cost per resolved ticket, driven mainly by model routing and semantic caching rather than by using a cheaper model across the board.
- First-response time dropped from over four hours to under two minutes for the roughly 55% of tickets the copilot could resolve without escalation.
- Escalations to human agents arrived with full context attached automatically, cutting the time agents spent on the "let me pull up your account and re-read this whole thread" step that used to eat a meaningful chunk of every handled ticket.
- Hallucination-related complaints dropped to effectively zero post-launch, verified through a sampled weekly audit of flagged conversations, compared to a noticeable handful per week in the original prototype.
- Support agents shifted measurably more of their time toward complex, judgment-heavy tickets — which, anecdotally, also improved team morale, since the most tedious and repetitive tickets were the ones the copilot absorbed first.
What We'd Tell Anyone Building Something Similar
A few lessons from this project apply well beyond this one client:
Cost problems in LLM systems are usually architecture problems, not model problems. The instinct when a system is too expensive is to swap in a cheaper model and accept worse answers. The bigger win almost always comes from routing, caching, and context discipline — using the expensive model only where it's actually earning its cost.
Retrieval quality matters more than most teams expect going in. A weekend RAG prototype and a production RAG system look similar in a demo and behave completely differently in practice. The difference is almost entirely in the retrieval layer — freshness handling, access control, chunk granularity — not in the generation step everyone focuses on first.
"The system doesn't know" needs to be a first-class, designed-for outcome, not a failure state. The single most trust-building feature in this entire build wasn't an accuracy improvement — it was making sure the system escalated cleanly and honestly whenever it wasn't confident, instead of guessing. Business owners evaluating any LLM support system should treat this as a non-negotiable requirement, not a nice-to-have.
The metric that mattered most to the client wasn't accuracy — it was defensibility. Being able to walk into a budget review and show exactly why costs dropped, exactly how often the system escalated appropriately, and exactly what happened when something went wrong turned out to matter more to leadership than any single performance number. Build the reporting and audit trail in from day one; retrofitting it later is much harder.
