Large language models have moved from novelty to expectation. Customers now assume your product can summarise, draft and answer — but bolting an LLM onto a live system is where most teams get burned.
Start with a narrow, reversible surface area. Pick one workflow where a wrong answer is cheap, ship it behind a feature flag, and measure before you widen. We treat every model call like an external dependency: timeouts, retries, fallbacks and a hard budget.
Evals are non-negotiable. Before launch we build a golden set of 50–200 real examples and score every prompt change against it in CI. A change that looks better in the playground often regresses on the long tail — the eval catches it.
Guardrails sit on both sides of the model: input validation and PII stripping going in, schema validation and toxicity checks coming out. If the model returns something unparseable, we fail closed to a deterministic path rather than showing the user a broken response.
Roll out by cohort, watch your latency and cost dashboards like a hawk, and keep the kill switch one click away. Done this way, AI features become just another well-instrumented part of the stack — not a production incident waiting to happen.