RAG and agentic RAG
Hire the engineer who grounds your assistant in your own documents.
Teams usually call me when
“Our assistant answers confidently, and wrongly.”
“Compliance needs to know where every answer came from.”
“Everything we know is in SharePoint, and nobody can find it.”
Anatomy of a grounded answer
Are we in compliance with FEMA CPG 101?
Mostly, with three gaps. CPG 101 is guidance, so compliance means your EOP is consistent with it 1, checked here against the base plan checklists in Version 3.1.
Record of changes. The log stops in 2022 and has no summaries 2; CPG 101 expects every change recorded 3.
Maintenance cycle. The plan names no review cycle 4; CPG 101 asks for a regular one 5.
Access and functional needs. The concept of operations doesn’t say how the plan meets those needs 6.
Want a draft of the three missing sections, in your plan’s format?
plan load the CPG 101 v3.1 base plan checklistsretrieve your Basic Plan, section by sectioncompare each checklist item to its sectionverify every gap cites both documents
- 1 FEMA CPG 101 v3.1 · §1.5.2 Review the Plan · pp. 58–59Compliance: the plan is consistent with guidance and doctrine, which provide a baseline
- 2 Your EOP · Basic Plan · Record of ChangesLast entry: Change 4 · March 2022, no summary column
- 3 FEMA CPG 101 v3.1 · Introductory Material · p. 67The record of changes documents each change to the plan
- 4 Your EOP · Plan Development and MaintenanceReviewed as needed by the emergency management coordinator
- 5 FEMA CPG 101 v3.1 · Plan Development and Maintenance checklist · p. 78Establish a regular cycle of training on, evaluating, reviewing and updating the EOP.
- 6 FEMA CPG 101 v3.1 · Concept of Operations checklist · p. 70Describe how plans account for the physical, programmatic and communications needs of people with disabilities and others with access and functional needs
How the weeks go
Assessment and eval setsources, permissions, your reviewersWeeks 1 to 2
Gatethe eval set says it can workGo or no-go, end of week 2
Buildingestion, hybrid search, citationsWeeks 3 to 6
Agentic RAG and handoffplans, checks, evalsWeeks 7 to 8
Two weeks to prove it. You keep the assessment and the eval set either way.
Done before
IEM
Agentic RAG for emergency planning, grounded in each planner’s own documents.
Built as a contractor to Presolved. The eval set tells the team when answer quality drifts, before their users do. Read the story
For your security and procurement review
- Data
- Documents and the index stay in your cloud. Retrieval respects each user’s permissions.
- Audit
- Every answer cites its source passages, so compliance can follow it back.
- Code and IP
- Yours, in your repo.
- Access
- Least privilege, removed at handoff.
- Paperwork
- US LLC. Your MSA and NDA or mine. Professional liability insurance. W-9 on request.
- Billing
- Two weeks to prove it, then fixed scope per phase, invoiced net-30.
Questions
- Which search store?
- The one that fits what you already run. Azure AI Search, OpenSearch, pgvector, and Elasticsearch all work, and the eval set settles the close calls.
- Can it run in our cloud?
- Yes. The index and the agent run in your own Azure, AWS, or Google Cloud account.
- Why not fine-tune?
- Retrieval with evals is cheaper to change and easier to audit. I reach for fine-tuning only when retrieval can’t get there.
Send twenty questions it gets wrong.
Your users’ real questions are the start of the eval set. I’ll reply within a business day with what I think is going wrong.