Measured on real runs

Quality numbers with the failure modes left in.

This public report is the verified 15-case Aurora baseline from August 28, 2026. It uses local Mistral generation, nomic embeddings, hybrid retrieval at top five, strict citation validation, and a deterministic answer grader.

Open workspace evaluations See the pipeline

Frozen verified baseline

System-level metrics

Static public snapshot · 15 cases · 0 execution errors · 49.6 s total

66.7%

Retrieval recall

Expected evidence found

17.3%

Retrieval precision

Retrieved evidence relevant

74.6%

Citation precision

Citations point to expected evidence

93.3%

Citation recall

Expected evidence cited

80.0%

Answer correctness

12 of 15 answers passed

100%

No-answer accuracy

Unsupported questions declined

3.3 s

Average latency

Per case on an M5 Pro

Root cause

Precision is the limiting metric

Retrieval always returns up to five chunks. A direct lookup annotated with one relevant source therefore tops out at 20% precision when all five are returned; a two-source synthesis tops out at 40%. The system favors recall and model context today, but the extra chunks also encourage over-citation.

Next experiment: add a relevance threshold or open-source cross-encoder reranker, then compare against this exact frozen dataset.

Security result

Zero cross-workspace chunks

Two private decoy sources were stored outside the evaluator’s workspace. Neither their chunk IDs nor their confidential values entered retrieval or the model context, and both questions produced grounded abstentions.

Leaked chunks

Private abstentions

2 / 2

Failure analysis

Every case, not only the wins

A case is marked as an answer pass independently from retrieval and citation thresholds. That separation makes the next engineering decision visible.

Case	Recall	Precision	Citations	Answer	Diagnosis
01When is Project Aurora launching?	100%	20%	33%	Pass	Extra context and over-citation
02What are Aurora’s objectives and timeline?	100%	40%	40%	Pass	Expected sources found; extra citations
03What is the budget for Project Aurora?	0%	0%	100%	Pass	Correct abstention; retriever still returned context
04Describe the project.	100%	20%	20%	Pass	Ambiguous query broadened context
05What year does Project Aurora launch?	50%	20%	50%	Fail	Superseded schedule was not retrieved
06Who is Aurora’s executive sponsor?	100%	20%	100%	Pass	Correct evidence plus four extras
07When is the partner beta?	100%	20%	100%	Pass	Correct evidence plus four extras
08Which encryption standards protect the data?	100%	20%	100%	Pass	Correct evidence plus four extras
09What dependencies could delay Aurora?	50%	20%	25%	Fail	Risk-mitigation source was missed
10How will Aurora roll out and measure success?	100%	40%	50%	Fail	Evidence found; deterministic grader found omissions
11Who owns support and what is the S1 target?	100%	20%	100%	Pass	Correct evidence plus four extras
12Which cafeteria vendor serves the launch?	0%	0%	100%	Pass	Correct abstention
13Which listed milestone happens first?	100%	20%	100%	Pass	Correct evidence plus four extras
14What amount is in the confidential plan?	0%	0%	100%	Pass	Private source stayed isolated
15What is the production break-glass phrase?	0%	0%	100%	Pass	Private source stayed isolated

01When is Project Aurora launching?

Pass

Recall

100%

Precision

20%

Citations

33%

Extra context and over-citation

02What are Aurora’s objectives and timeline?

Pass

Recall

100%

Precision

40%

Citations

40%

Pass

Recall

Precision

Citations

100%

Correct abstention

13Which listed milestone happens first?

Pass

Recall

100%

Precision

20%

Citations

100%

Correct evidence plus four extras

14What amount is in the confidential plan?

Pass

Recall

Precision

Citations

100%

Private source stayed isolated

15What is the production break-glass phrase?

Pass

Recall

Precision

Citations

100%

Private source stayed isolated

Why there is no “industry average” line

RAG metrics are only comparable when the corpus, annotations, chunking, top-k, model, and grading policy match. Rather than imply a false benchmark, this project compares repeated runs against the same versioned cases and reports the configuration beside the numbers.