Building AI-Native Products: Lessons from 12 Months of LLM Integration

Evals before everything

The first mistake almost every team makes: writing code before defining what "good" looks like. With traditional software, you know if it works. With LLMs, the output exists on a spectrum — and without a way to measure quality, you're flying blind.

Before you write a single prompt, define 20–30 representative inputs and their expected outputs. These become your eval suite. Every prompt change, model swap, or retrieval tweak gets measured against it. We've had "improvements" that looked great on five manual tests but degraded output across the broader set. Evals catch that. Vibes don't.

RAG is a retrieval problem, not a prompting problem

Most teams spend 80% of their time on prompts and 20% on retrieval. It should be the opposite. If the right context isn't in the window, no prompt will save you.

We learned to treat RAG as a search engineering problem: chunk size matters, overlap matters, embedding model choice matters, and hybrid search (dense + sparse) almost always beats pure vector similarity for production workloads. Reranking with a cross-encoder before final context selection is the single biggest quality improvement we've found that most teams skip.

Don't touch the prompt until the retrieval is working.

Latency is a product problem

A technically correct response that takes 12 seconds is a broken product. LLM latency compounds fast once you add retrieval, tool calls, and guardrails.

Patterns that actually help: stream everything by default, cache aggressively at the retrieval layer, use smaller/faster models for classification and routing tasks (reserve the big models for generation), and batch where user expectations allow. We've cut perceived latency by 40% just by streaming tokens earlier and moving skeleton UI rendering upstream.

Agentic systems need hard limits, not soft guidelines

Multi-step agents are where teams get into trouble. The failure mode isn't dramatic — it's subtle. The agent takes a slightly wrong branch on step 2, and by step 7 it's confidently done something you didn't intend.

Two rules we don't break: every irreversible action requires explicit confirmation (write, send, delete — always a checkpoint), and every agent loop has a hard step ceiling. No infinite retries, no unbounded tool call chains. The ceiling feels arbitrary until the day it saves you from a runaway agent burning API credits or, worse, doing real damage.

Model versioning will break you

You built against GPT-4-turbo or Claude 3 Opus, it worked beautifully, then a model update shipped and your evals dropped 15%. This happens. Model providers push updates, retire versions, change defaults.

Pin your model versions in production. Treat a model upgrade like a dependency upgrade — deliberate, tested against your eval suite, not automatic. We now maintain a model version config per environment (dev can run latest, staging mirrors prod, prod is pinned) and have a one-line rollback.

The boring stuff is still the hard part

Structured outputs, JSON mode, tool call reliability — these feel solved until they're not. At scale, edge cases surface: malformed JSON on 0.3% of requests, tool calls that hallucinate argument names, outputs that are technically valid but semantically wrong.

Schema validation with Zod or Pydantic at every LLM boundary, retry logic with exponential backoff, and fallback paths for generation failures aren't optional in production. They're load-bearing.

Ship small, iterate fast

The teams that get the most out of LLMs aren't the ones with the most sophisticated architectures. They're the ones that put something real in users' hands quickly, instrument everything, and iterate on actual usage data. The best prompt is the one informed by real failure cases from production.

Build the simplest thing that could work. Then measure it. Then make it better.