RAG Beyond the Demo

By Rob Tyrie · Original source: July 14, 2024 · Essay · Standalone edition published October 25, 2026

This edition preserves the current Ironstone text developed from the original source linked below. The full piece also remains in the Field Notes & Essays collection.

It is easy to make a document chat demo. It is much harder to build a retrieval system that deserves a place in a critical workflow.

The demo finds a paragraph and produces a fluent answer. Production has to answer more difficult questions. Was the correct document available? Was it current? Did retrieval miss an exception buried in an appendix? Did compression remove the qualification that changed the meaning? Was the source authoritative, or merely similar?

There is no single RAG architecture. An enterprise may need different stores and retrieval patterns for policies, underwriting guidance, code, customer records and external research. Contextual compression can be useful because it reduces noise, but every compression is also an editorial act. Something is removed. In regulated or consequential work, the system must know what it is permitted to summarize and what must remain intact.

The central requirement is evidence. A team should be able to show that a particular design reduces error on work that matters, not merely that it looks convincing in a demonstration.

Build The Benchmark Before The Chatbot

Collect a set of real questions, including ambiguous ones, stale documents, conflicting authorities and questions whose correct answer is “insufficient evidence.” Label the sources a good practitioner would use. Measure retrieval recall separately from answer quality. Track citation accuracy, unsupported claims and appropriate refusal.

Then compare architectures. Use multiple indexes where the information really has different authority or structure. A production retrieval system is not a clever search box. It is an evidence system with a conversational surface.

Original source

Put the thinking to work: agentic systems, Ironstone Methods and a conversation with Rob.