Voice-Enabled RAG over MSMARCO-XI

HH Goa 2026 — Shortlisting Task 2.

Speak a question in Hindi, Bengali or Tamil; get a grounded answer back, with the retrieval path completing inside a 200 ms budget.

🎙  voice  →  Sarvam STT  →  query encode  →  hybrid retrieval  →  rerank  →  guarded generation
                             (local ONNX)     (FAISS + BM25, RRF)             (grounding check)

---

Why this is built the way it is

The 200 ms target dictates the architecture. A conventional RAG stack — hosted embedding API, hosted vector DB, hosted LLM — spends 800–2000 ms, and most of it is TLS handshakes rather than computation. Three consequences run through every design decision here:

1. Nothing in the query path touches the network except generation. The encoder is int8 ONNX running in-process; the vector index is FAISS HNSW held in RAM. Query encode ≈ 6 ms, search ≈ 5 ms.
2. All chunking happens offline. The index is prebuilt. At query time "chunking" costs zero — the work was done at build time, which is what makes the expensive strategies (semantic, sentence-window) affordable at all.
3. Generation is reported as time-to-first-token, streamed. A full answer cannot be generated in 200 ms by any hosted model; TTFT can. There is also an extractive fast path that skips the LLM entirely when the top chunk scores high on a factoid query. Both are reported separately — see Latency.

Sarvam over ElevenLabs because MSMARCO-XI is an AI4Bharat Indic dataset. Sarvam's ASR is trained natively on these languages; the demo asks questions in Hindi against a multilingual index.

---

Chunking: five strategies, benchmarked against each other

The brief asks for a chunking approach that is vast rather than naive. Five strategies are implemented, indexed, and scored on the same query set — the comparison table is the deliverable, not the list.

Strategy	Idea	What it's good at
fixed_256 / fixed_128	Equal token windows, 15 % overlap	Baseline. Uniform batches, fastest build. Structure-blind by design — it's here to be beaten.
recursive_256	Paragraph → sentence → token fallback; overlap in whole sentences	Never cuts mid-sentence, so the overlapped region is always quotable text.
semantic_p85	Embed each sentence, cut where cosine distance to the next spikes past the document's own 85th percentile	Topic-coherent chunks. Relative threshold because absolute cosine cutoffs don't transfer across 13 languages.
sentence_window_2	Embed one sentence, return it ± 2 neighbours	Refuses the precision/context trade: sharp vector, readable context. Expect best hit@1, largest index.
metadata_aware	Recursive splitting + a `[lang	query_type]` header and lead/body position, injected into the embedding text only

Two ideas carry most of the weight:

• embed_text vs retrieval_text are separate fields on every chunk. What the encoder sees and what the LLM is handed need not be the same string. That separation is what makes sentence-window and metadata-aware expressible at all.
• Chunk sizes are measured in real model tokens, with character offsets preserved. A "256-word" chunk in Hindi and in Tamil are wildly different inputs to the same encoder, so every splitter is driven off the tokenizer's offset mapping. Keeping the char spans also gives the guardrail layer exact substrings to point at when checking grounding.

Retrieval

Dense (FAISS HNSW, cosine on normalized vectors) + BM25, fused with Reciprocal Rank Fusion. RRF rather than a weighted score blend because dense and BM25 scores live on incompatible, corpus-dependent scales — RRF reads only ranks, so it needs no per-language tuning.

BM25 indexes character 4-grams alongside words: Tamil, Malayalam and Kannada are agglutinative, there is no usable stemmer for most of these scripts, and word-level matching misses nearly every inflected form. BM25 weights are baked into a sparse term-document matrix at build time, so a query is a row-gather plus a sum — about 1 ms.

---

Evaluation is honest by construction

MS MARCO ships is_selected — a human annotation of which passage actually answered each query. That is used directly as the gold label, so no hand-labelling is involved and every strategy is scored on identical ground truth.

The subtlety that makes the comparison fair: scoring is at the passage level, not the chunk level. Strategies emit different numbers of chunks, so chunk-level scoring would simply reward whoever shreds the corpus finest. A retrieved chunk counts as a hit if the passage it came from is gold, deduped by passage.

Queries with no gold passage are skipped rather than counted as misses — the dataset contains such rows, and scoring them as failures would penalise strategies for a dataset artifact.

---

Latency accounting

Reported at P50 / P70 / P90 / P95 / P100 over 250 queries, using perf_counter_ns (time.time() has ~15 ms granularity on Windows — 8 % of the whole budget). Percentiles use method="lower", so every number published is a latency that was actually observed rather than an interpolation between two runs.

Stages counted against the 200 ms budget:

Stage	Target
query encode (ONNX int8)	~6 ms
dense search (HNSW, ef=64)	~5 ms
BM25 search	~1 ms
RRF fusion	<1 ms
cross-encoder rerank (top 20 → 4)	~15 ms
input guardrail	~10 ms
generation, first token	~120 ms

STT is timed and reported separately: it is network-bound and inherently proportional to utterance length, and the brief's budget starts at "chunking + vector DB retrieval". This is called out rather than hidden — the full voice-to-answer wall clock is in the report too.

---

Repo layout

src/vrag/
  chunking/      tokenization.py  multilingual sentence splitting + token offsets
                 base.py          Chunk/Document/Chunker + strategy registry
                 fixed.py recursive.py semantic.py sentence_window.py metadata_aware.py
  embed/         encoder.py       local ONNX int8 encoder, e5 prefix handling
  index/         dense.py         FAISS HNSW + payload store
                 sparse.py        BM25 as a precomputed sparse matrix
                 hybrid.py        RRF fusion, the retrieval entry point
  eval/          metrics.py       passage-level hit/recall/nDCG/MRR
  telemetry/     timing.py        per-stage spans, percentile reporting
  data/          loader.py        MSMARCO-XI flattening + gold labels
scripts/         prepare_data.py  eval_chunking.py
tests/           test_chunking.py test_retrieval.py

Running it

pip install -r requirements.txt
cp .env.example .env          # add SARVAM_API_KEY, ANTHROPIC_API_KEY

python scripts/prepare_data.py          # streams the MSMARCO-XI subset to data/
python scripts/eval_chunking.py         # the bake-off -> results/chunking_comparison.md

python tests/test_chunking.py           # no downloads needed
python tests/test_retrieval.py

Both test files run standalone (python tests/…) with no pytest and no model download — the chunking tests use a whitespace tokenizer and the retrieval tests use deterministic synthetic embeddings, so wiring bugs surface without waiting on the dataset.

---

Status

• Chunking layer — 5 strategies, multilingual sentence splitting, 8 invariant tests green
• Retrieval — FAISS HNSW + BM25 + RRF, 9 integration tests green
• Evaluation — passage-level metrics, latency spans, comparison harness
• Encoder ONNX export + prepare_data run against the live dataset
• Cross-encoder reranking
• Sarvam STT + WebSocket streaming
• Harness: typed stages, retries, circuit breaker, fallback chain
• Guardrails: input filter, retrieval-confidence gate, grounding check
• Deploy + live link