Shipping LLM Features That Survive Production

Building an LLM demo takes an afternoon. Building an LLM feature that can sit in front of customers takes considerably longer, and almost all of the difference is engineering that has nothing to do with prompting.

Here is what has to exist before it ships.

1. An evaluation harness, built before the feature

If you cannot measure quality, you cannot tell whether a prompt change helped, whether a model upgrade regressed something, or whether the system is degrading over time. Model behaviour drifts, providers update models underneath you, and every prompt edit is a change with unknown blast radius.

The minimum viable version is not complicated:

• A golden dataset of 50–200 representative inputs with expected characteristics of a good output
• Automated scoring — exact match where possible, an LLM judge with a clear rubric where not
• The whole thing running in CI on every change to a prompt, model or retrieval configuration
• Results tracked over time so drift is visible

Teams skip this because it feels like overhead against a feature that already seems to work. Then a prompt tweak six weeks later degrades a case nobody thought about, and it reaches customers because nothing was watching.

2. Retrieval that has actually been tuned

Most RAG implementations use default chunk sizes, a single embedding model and top-k retrieval, and then people conclude that RAG does not work well for their corpus.

The variables that matter, roughly in order of impact:

Chunking strategy. Fixed-size chunks split mid-sentence and mid-table. Structure-aware chunking that respects headings, list boundaries and table integrity is usually a larger improvement than any model change.

Reranking. Vector similarity is a coarse filter. Retrieving 30 candidates and reranking them with a cross-encoder before passing the top 5 to the model is often the single biggest quality jump available.

Hybrid search. Dense vectors miss exact matches — part numbers, error codes, proper nouns. Combining BM25 keyword search with vector search covers both.

Metadata filtering. Filtering by document type, recency or access permission before semantic search improves both relevance and, critically, security.

That last point deserves emphasis. Retrieval must respect the asker's permissions. An assistant that surfaces a document the user could not otherwise open is a data breach, regardless of how helpful the answer was.

3. Guardrails at both boundaries

On the way in: prompt injection defence, particularly where the model processes untrusted content such as emails, uploaded documents or web pages. Assume anything user-supplied may contain instructions. Structural separation of instructions from data, and constrained tool permissions, do more than any filtering pattern.

On the way out: structured output schemas so responses cannot drift into free-form text your parser was not expecting, PII detection before anything is logged or displayed, and a confidence threshold that routes uncertain cases to a human instead of guessing confidently.

4. Cost controls that exist before the bill does

AI spend has an unusual property: it scales with usage, and usage scales with success. A feature that costs $200 a month during beta can cost $12,000 a month at full rollout, and it usually does so between billing cycles.

What we put in every production LLM system:

• Model routing. Cheap, fast models for classification and extraction; larger models only where reasoning quality justifies the cost. Most systems over-provision here by default.
• Prompt caching. Where the provider supports it, caching a stable system prompt and document context cuts input cost substantially on repeat calls.
• Context discipline. Sending an entire document when three paragraphs would do is the most common source of unnecessary spend.
• Per-tenant budgets with alerting. A runaway loop or an abusive user should trigger an alert and a cap, not a surprise invoice.
• Cost per task as a tracked metric. Not monthly spend — cost per completed unit of work, so you can tell efficiency from growth.

5. Graceful degradation

Model providers have outages. Rate limits get hit. Latency spikes. A production feature needs a defined behaviour for each.

Decide in advance, per feature, what happens when the model is unavailable. Sometimes the answer is a fallback to a second provider. Sometimes it is a cached or simpler deterministic response. Sometimes it is an honest message that the feature is temporarily unavailable — which is a perfectly good answer, and far better than a spinner that never resolves.

Build against an abstraction over providers from day one. It costs very little at the start and it is what lets you switch on price, latency or capability later without a rewrite.

6. Observability of the whole chain

Standard application monitoring does not tell you why a response was poor. You need the full trace: the input, the retrieved context, the resolved prompt, the model and version, the tool calls, the raw output, latency and token counts.

Store it — subject to your retention and privacy policy — because the only way to improve a system like this is to look at the cases where it did badly. Without traces, you are guessing.

Add user feedback capture, even something as minimal as thumbs up and down. It is the cheapest source of labelled failure cases you will ever get, and it feeds directly back into the golden dataset from step one.

The thing that makes the biggest difference

Constrain the problem.

The most reliable production LLM systems we have built are the ones with the narrowest scope. "Extract these seven fields from this document type" is a solvable problem with a measurable success rate. "Answer any question about our business" is not a specification.

Every constraint you can add — a schema, a domain limit, a defined set of tools, a clear escalation path to a human — moves the system from impressive to dependable. Dependable is what ships.

---

We build AI systems that hold up under real load, and we say so when a use case does not justify a model at all. See our approach to AI and machine learning, or tell us what you are working on.