Origin
The problem
Automotive and EV research is spread across papers, journals, repositories, and technical publications. Finding one specific answer can mean opening dozens of PDFs, searching through them manually, and trying to keep track of where each fact came from.
AutoRAG started from a simple idea: what if a researcher could ask a question in natural language and get a concise answer grounded only in the research papers they had provided?
Designed for
Example question
"What are the thermal management challenges in EV batteries?"
Dozens of PDFs
Ingestion
Retrieval
Generation
Cited Answer
Core experience
From research papers to cited answers
Every document goes through the same sequence before a user can query it. Each step is a deliberate design choice, not a default.
Extract
Clean
Chunk
Embed
Index
Retrieve
Generate
Cite
Offline
The entire system runs locally. No cloud API dependency at inference time.
Grounded
Answers are generated only from retrieved research content, not from model parametric memory.
Cited
Every claim in the response is tagged with its source chunk and paper.
01 — Ingestion
PDF ingestion pipeline
Research papers arrive as PDFs. The ingestion pipeline extracts text using pdfplumber, cleans it, and assigns a quality score before any chunk is accepted into the index.
For problematic or scanned PDFs, the pipeline falls back to OCR automatically, so coverage doesn't depend on PDF quality.
Text extraction
pdfplumber extracts raw text with layout awareness. Tables and structured content are handled where the PDF encoding permits.
OCR fallback
Scanned or image-heavy PDFs that return poor text are automatically routed through an OCR fallback. No manual intervention required.
Quality scoring
Each extracted document is scored before chunking. Low-quality extractions are flagged rather than silently ingested.
Incremental processing
The system tracks which files have already been processed. New papers can be added without reprocessing existing documents.
Engineering detail
Incremental processing means adding 10 new papers to a 40-paper corpus doesn't require re-embedding and re-indexing the original 40. The system tracks processed files by hash and skips them on subsequent runs.
02 — Chunking
Chunking was not a default
Chunking turned out to be one of the highest-leverage decisions in the entire pipeline. Chunks that were too small lost context. Chunks that were too large diluted relevance. The chosen parameters came from empirical testing, not configuration defaults.
750
Tokens per chunk
100
Token overlap between chunks
≥ 80%
Semantic completeness target
The 100-token overlap ensures that concepts straddling chunk boundaries are still retrievable — a sentence that begins near the end of one chunk and concludes at the start of the next isn't lost.
A quality gate enforces that chunks must reach a semantic completeness threshold before being accepted into the index. A chunk that is grammatically incomplete or contextually isolated is rejected.
Key insight
Chunking decisions directly determined retrieval quality. Getting this wrong first meant iterating on the evaluation metrics — which is exactly what the evaluation framework was designed to catch.
03 — Retrieval
Semantic search alone was not enough
Pure semantic retrieval captures conceptual similarity well but can miss exact technical terms, model names, and domain-specific abbreviations. Automotive research is dense with precise terminology. A pure vector search would overlook queries like "BMS fault codes in LFP chemistry."
60%
Semantic Search
Conceptual similarity via embeddings
40%
BM25
Exact keyword relevance
Hybrid Retrieval — initial weighting, treated as a tunable parameter
The hybrid approach combines BAAI/bge-large embeddings for conceptual similarity with BM25 for keyword-level relevance. The 60/40 initial weighting was chosen empirically and is treated as a tunable parameter rather than a permanent setting.
Retrieval quality was measured against a held-out evaluation set. When recall fell below target, the weighting was adjusted and re-evaluated — not assumed.
Embeddings
BAAI/bge-large
1024 dimensions
Local embeddings were chosen over a hosted API to keep the entire pipeline offline and avoid per-query costs. The model runs on device at embedding time and is not called at inference.
Vector Store
Chroma
Local / zero infrastructure
Chroma provides a lightweight, zero-infrastructure vector store appropriate for the project's current scale. No server to run, no cloud dependency.
Storage upgrade path
Now
Chroma
Local, zero-infrastructure vector store. No cloud dependency, no configuration overhead — matched to the project's current scale and learning goals.
Later — upgrade path
Weaviate
A natural migration target once corpus size grows beyond what a local store handles efficiently. Not currently in use.
04 — Generation
From cloud to local inference
The project originally used the Claude API for generation. After the initial prototype, it was migrated to Ollama with Mistral as a deliberate architectural decision — not a compromise.
Original
Claude API
- Faster generation (~2–5s observed)
- Stronger hosted model
- API key dependency
- Per-query cost
- Internet connection required
Current
Ollama + Mistral
- Fully offline
- No API key
- No per-query cost
- Greater data privacy
- Slower on CPU (~5–15s observed)
The latency figures above are development observations on a MacBook Air — not production benchmarks. Local inference speed depends heavily on hardware. The tradeoff was accepted: privacy, portability, and zero running cost outweighed raw generation speed for this use case.
Retrieved chunks
Prompt construction
Ollama
Mistral
Cited answer
Grounding
Answers should show their sources
The model is prompted to cite every factual claim with a source marker. This isn't cosmetic — it is the core of the system's trust model.
Citation format in generated responses
Thermal runaway in lithium-ion cells is most commonly triggered by mechanical deformation or overcharging [Source 2]. BMS fault detection latency below 100ms is considered standard for commercial EV packs [Source 7].
Retrieved chunks and their source papers are surfaced in the Streamlit UI so the user can inspect the original passage behind each citation.
Data Model
Document-and-chunk-centric, not relational
AutoRAG uses a document-and-chunk-centric model rather than a warehouse-style dimensional schema. The unit of retrieval is a chunk; every chunk traces back to its parent paper.
Paper
Chunk
Query Result
Evaluation
Measuring whether retrieval actually works
The system was not judged by whether answers "looked good." A held-out evaluation set of 15–30 questions — tagged by type (factual, comparative) with expected keywords and known relevant papers — was used to measure retrieval against concrete targets.
This meant retrieval quality was a number, not an impression. When a change to chunking or weighting was made, the evaluation ran again.
≥ 80%
Recall@5
The relevant chunk appears in the top 5 results at least 8 times out of 10.
≥ 60%
Precision@5
At least 3 of the top 5 retrieved chunks are genuinely relevant.
≥ 70%
Keyword Coverage
Generated responses include domain-specific expected terminology.
MRR
Mean reciprocal rank — how high in the results the correct chunk typically appears.
Latency
End-to-end query time tracked against a sub-5-second working target.
Source accuracy
Citations in generated output correctly trace to the retrieved source chunk.
Quality dashboard
A custom MetricsTracker records recall, precision, MRR, latency, keyword coverage, and source accuracy to append-only JSONL files under data/monitoring/. A Jupyter notebook visualises retrieval performance and latency over time against their thresholds.
Infrastructure
100% local, zero cloud dependencies
The entire system runs on a single machine. There is no external database, no cloud LLM API call at inference time, and no hosted vector store. Every component is local.
MacBook Air
Development machine
Python environment
Runtime + orchestration
Ollama / Mistral
Local LLM inference
Chroma
Local vector store
FastAPI
Query API layer
Streamlit
Research UI
Results
What worked
End-to-end pipeline
PDF → chunk → embed → index → retrieve → generate → cite
Complete pipeline operational from ingestion to cited answer generation.
Initial indexing
19 chunks
From initial paper set
Early retrieval accuracy
~85%
Against the held-out evaluation set
Query latency target
Sub-5s
End-to-end working target for local inference
Demo — live query against the research corpus
Expansion path
The initial corpus was a handful of papers. The documented path scales to 50+ papers in the near term, with an architecture capable of reaching 500–1,000 papers without fundamental redesign. This is a future target, not a current scale claim.
Iteration
When retrieval failed, I changed the system — not the answer
Evaluation metrics drove architectural changes. When a metric fell below its target, the response was a system change — not an explanation for why the metric didn't matter.
Low Recall
- Increase chunk size
- Try different embedding model
- Adjust hybrid retrieval weighting
Low Precision
- Decrease chunk size
- Apply stricter relevance filtering
- Re-tune BM25 weighting
Poor Answer Coverage
- Inspect retrieved context
- Improve retrieval step
- Re-evaluate against held-out set
The decisions behind the system
Chroma
Local, zero-infrastructure vector store. Right for the current scale and learning objectives.
Instead of
Pinecone
BAAI/bge-large
Local embeddings, offline operation, no per-embedding cost. Runs once at index time.
Instead of
OpenAI text-embedding-ada-002
750/100 token chunks
Empirically balanced context preservation with retrieval relevance. Tested, not assumed.
Instead of
Default chunk sizes
Hybrid retrieval (60/40)
Captures both conceptual similarity and exact keyword relevance for technical domain queries.
Instead of
Pure vector search
Ollama + Mistral
Fully offline, no API key, no per-query cost, data stays local. Privacy and portability were requirements.
Instead of
Claude API
JSONL monitoring logs
Append-only, human-readable, trivial to query in a notebook. Right for the project's current scale.
Instead of
External observability stack
Challenges
What was harder than expected
01
Migrating from Claude API to Ollama/Mistral
The architecture change was straightforward. The observable tradeoff was generation latency: cloud inference ran at approximately 2–5 seconds on development hardware; local CPU inference at 5–15 seconds. The privacy and cost benefits justified the tradeoff for this use case.
02
Chunking strategy that preserved context
Default chunk sizes produced poor retrieval results on technical research text. Reaching the 750/100 configuration required iterating with the evaluation framework — there was no reliable shortcut to the right parameters.
03
Tuning semantic/BM25 retrieval weights
The correct weighting between semantic search and BM25 wasn't obvious. Domain-specific terminology (battery chemistry, EV component codes) responded differently to each retrieval method. Iterating against the evaluation set was the only reliable method.
04
Handling imperfect and scanned PDFs
Research papers vary significantly in PDF quality. The OCR fallback addressed this, but calibrating when to trigger it and how to score the resulting text quality took more iteration than anticipated.
Lessons Learned
What I learned
Chunking can make or break RAG
It was the highest-leverage retrieval decision in the pipeline. Getting it right was the difference between 60% and 85% recall on the evaluation set.
Local LLMs are a real tradeoff
Ollama/Mistral sacrificed some generation speed — and potentially some answer coherence on complex multi-hop queries — in exchange for privacy, offline operation, and zero per-query cost. That tradeoff was correct for this use case.
Quality gates matter early
Metrics, logging, and evaluation set up before the system grew beyond a handful of papers. Without the evaluation framework, it would have been impossible to tell whether changes to chunking or retrieval weighting were improvements.
Document the why
The architectural reasoning — why Chroma not Pinecone, why local embeddings not OpenAI, why 750/100 not default sizes — became as valuable as the implementation itself. Decisions without documented rationale are invisible debt.
Technology Stack
Built with
End-to-end architecture
Research Papers
PDF input corpus
PDF Ingestion
pdfplumber + OCR fallback
Clean + QA
Text cleaning, quality scoring
750-token chunks
100-token overlap
BAAI/bge-large
1024-dimensional embeddings
Chroma
Local vector index
Retrieval fork
Semantic Search
60%
BM25
40%
Hybrid Retrieval
Top ranked chunks
Ollama + Mistral
Local generation
Cited Answer
[Source N] attribution
Serving
FastAPI
Query API
Streamlit
Research UI
Origin
Why I built it
This started with a simple conversation with my husband, a Research Engineer, who wanted a better way to work through the research papers in his folder. What started as an experiment became a working research tool that is now part of our workflow.
It was a useful reminder that good engineering often starts with a very practical problem.
Interested in how it works?