The short version
TL;DR
This is not a general-purpose chatbot. It is a permission-aware search system that turns workspace documents into retrievable evidence, asks a local model to answer only from that evidence, and rejects source references the model did not actually receive.
Different contract
Why it is not a chatbot
The interface looks conversational, but its contract is narrower and more useful for governed knowledge: answer from approved sources or say the evidence is insufficient.
Current boundary: access control is workspace-wide. The system independently verifies each chunk's owning document, but fine-grained per-document ACLs are not implemented yet.
Five stages
Architecture
- 01
Ingest
Validate, hash, chunk, and preserve source offsets.
- 02
Retrieve
Run keyword and 768-dimensional vector search in parallel.
- 03
Authorize
Recheck membership and document ownership before context.
- 04
Generate
Give local Mistral only the authorized top-ranked chunks.
- 05
Evaluate
Persist traces and score a repeatable 15-case benchmark.
Ingestion
Text and Markdown are content-hashed for workspace-level deduplication, split into overlapping chunks, embedded by nomic-embed-text, and stored with source offsets.
Retrieval
PostgreSQL full-text ranking and pgvector distance search produce two ranked lists. Reciprocal Rank Fusion (RRF, k = 60) combines them without hand-tuned weights.
Generation and citations
Mistral sees authorized chunks labeled [source: chunk_id]. The answer parser keeps only citations whose IDs exist in that exact context and logs fabricated references.
Evaluation
Fifteen direct, multi-document, ambiguous, no-answer, conflict, and isolation cases measure retrieval, citations, deterministic answer correctness, and latency.
Open and replaceable
System components
Storage and retrieval
PostgreSQL + pgvector
One database supplies relational workspace isolation, GIN-indexed full-text search, and HNSW vector indexes.
Local inference
Ollama
nomic-embed-text creates 768-dimensional embeddings; Mistral generates grounded answers without a paid model API.
Authorization boundary
Workspace isolation
Every retrieval query is scoped to a workspace, and membership is resolved from the authenticated session.
Defense in depth
Permission checkpoint
Candidate chunks are independently joined through documents and workspace membership before any text reaches the model.
Quality control
Evaluation framework
Stored cases and runs turn quality into reproducible metrics and case-level failure analysis—not vibes.
Measured locally
Evaluation results
This frozen public snapshot comes from a verified 15-case local run. It is a project benchmark, not a universal model score; corpus, annotations, hardware, and retrieval settings all affect it.
- Retrieval recall
- 66.7%
- Retrieval precision
- 17.3%
- Citation precision
- 74.6%
- Citation recall
- 93.3%
- Answer correctness
- 80.0%
- No-answer accuracy
- 100%
- Average latency
- 3.3 s
Expected evidence found
Retrieved evidence expected
Citations on expected evidence
Expected evidence cited
Deterministic benchmark pass rate
Correct refusals without evidence
Per evaluation question
Root-cause finding
Top-5 retrieval favors recall and frequently returns extra chunks. That explains much of the 17.3% precision result and identifies reranking or a relevance threshold as the next experiment—not a metric to hide.
Beyond a demo
Why this matters
Production thinking
Authentication, roles, workspace isolation, timeouts, and authorization checkpoints are part of the main flow.
Measured quality
The system exposes real benchmark results and per-case failures instead of making unsupported accuracy claims.
Honest tradeoffs
Low retrieval precision and local-model latency are visible, explained, and connected to concrete follow-up work.
End-to-end ownership
The implementation covers ingestion, retrieval, generation, citations, traces, evaluation, and a responsive interface.
Deliberate constraints
Technical tradeoffs
Local Mistral
Benefit
No inference bill and document context stays on infrastructure you control.
Cost
The measured run averaged 3.3 seconds per question; hosted APIs are commonly faster, but no external API was benchmarked here.
Recall-first top-5 retrieval
Benefit
More evidence is available to the generator, reducing the risk of omitting a relevant chunk.
Cost
Extra context lowered retrieval precision to 17.3% in the latest benchmark.
PostgreSQL full-text search
Benefit
Search, vectors, transactions, and authorization remain in one operational system.
Cost
It offers fewer specialized ranking controls than a dedicated search cluster, which this project does not yet need.
Deterministic answer grader
Benefit
Results are repeatable, local, fast, and have zero judge-model cost.
Cost
Token-coverage heuristics can miss valid paraphrases and are not a substitute for human labels.
Trust boundary
Security model
Workspace-scoped SQL
Documents and retrieval candidates are selected with a parameterized workspace ID—not a client-provided free-form filter.
Pre-model checkpoint
Candidates are joined through documents and live workspace membership. Unauthorized text is removed before prompt construction.
Hardened sessions
Seven-day sessions use random tokens in httpOnly, SameSite=Lax cookies; production cookies are Secure and only an HMAC-SHA256 token digest is stored.
Password and query safety
Passwords use bcrypt with cost 12. Public production signup is invite-gated, database calls use positional parameters, and mutating routes enforce same-origin requests.
Verified isolation: the authorization scenario returned zero cross-workspace chunks and no protected benchmark facts. Fine-grained document sharing remains future work.
Four steps
How to use it
- 1
Create an account
Sign up with email and password (plus the invite on the public demo); a private workspace is created with you as admin.
- 2
Add a document
Paste plain text or Markdown. The ingestion pipeline chunks and embeds it locally.
- 3
Ask a question
Search returns a grounded answer, validated citations, retrieved chunks, and latency.
- 4
Measure quality
Run private evaluations, inspect failures, and compare live workspace runs with the frozen public baseline.
Honest next steps
Future improvements
Reranking
Add a local cross-encoder after RRF. A 40%+ precision target is a hypothesis to test, not a measured result.
Query expansion
Rewrite or branch ambiguous questions before retrieval while retaining the original query in traces.
Streaming responses
Stream model output and reconcile citations after completion without weakening validation.
Multimodal ingestion
Extract layout, tables, and images from PDFs with source-aware chunk references.
Fine-grained permissions
Add document-level grants when sharing requirements extend beyond workspace membership.
Production packaging
Containerize the application, database migrations, and inference service for an orchestrated deployment.