System Design RAG

• 100 Documents 

1. Ingestion 

• Parsing
• OCR
• Chunking 
• Storing

1. Vector database 
2. Index creation 
3. Metadata store Relational 

• BM25
• HNSW
• IVF

Selecting vector store

2. Generation 

• API Gateway
• Authentication 
• Metadata Filtering
• Keyword Lookup using BM25 Index
• HNSW search

• Reciprocal Rank Fusion 
• Optional Cross Encoder
• Intent Classification
• Routing Rules
• Model Selection
• Guardrails
• Observability
• Output Generation

3. Evaluation

• Recall@K 

• 100K documents

With 100K documents, the architecture changes significantly to optimize scalability and retrieval quality. The ingestion pipeline becomes distributed (Kafka + multiple workers), chunk count grows to millions, and I would introduce reranking (cross-encoder) because hybrid search alone is no longer sufficient. The vector index would use HNSW if memory permits or IVF-PQ/DiskANN for lower memory footprint, with the index sharded across multiple OpenSearch nodes. Metadata filtering becomes critical to reduce the search space before vector search. I would also introduce query rewriting, document deduplication, hierarchical retrieval (document → section → chunk), and context compressionto avoid sending too many chunks to the LLM. The vector database and BM25 indexes would be independently scalable, caching would be added at both retrieval and LLM layers, and offline evaluation would become mandatory to continuously tune chunk size, K values, reranking thresholds, and retrieval quality. The overall flow remains the same, but retrieval becomes a multi-stage pipeline rather than a single search step.

Ingestion

• Kafka
• IVF-PQ
• Disk ANN
• Multi Node Opensearch
• Metadata filtering

Retrieval

• Query Rewriting
• Document Deduplication
• Hierarchical Retrieval
• Context Compression
• Auto Scaling BM25
• Catching
• Multi stage Retrieval

Recall@K measures whether the retrieval system returns the correct document or chunk within the top K results. 

For example, suppose you have 100 test questions, and for each question you already know the correct policy chunk. 

If you retrieve the top 5 chunks (K=5) and the correct chunk appears in the top 5 for 92 questions, then Recall@5 = 92/100 = 92%. 

In a RAG system, Recall@K is critical because if the correct chunk is not retrieved, the LLM cannot generate the correct answer, regardless of how powerful it is. Typically, we aim for a high Recall@5 or Recall@10 before optimizing the generation model.

• LLM as a Judge for Unlabelled data 

Dimensions 

• correctness
•  groundedness
•  completeness
•  relevance
•  hallucination 

 Rubric  

Score 1 (Severe Hallucination): The response is entirely fabricated, completely contradicts the reference context, or introduces significant external facts that directly change the meaning of the source text.  

Score 2 (Major Hallucination): The response gets the general topic right but includes major unverified claims, blends facts from outside the context, or subtly changes key metrics, dates, names, or outcomes.  

Score 3 (Minor Hallucination): The response is mostly accurate but contains minor embellishments, logical leaps, or unmentioned details that do not drastically change the overall meaning but are still strictly unverified.

Score 4 (No Hallucination, But Missing Context): The response contains zero fabricated facts or external claims, but it omits critical context from the reference document, making the statement slightly misleading.

Score 5 (Perfectly Factual): Every single claim, metric, and detail in the response is explicitly and fully supported by the provided reference context. No assumptions or external facts are introduced.

Prompt Design 

You are an expert quality assurance judge. Your task is to evaluate a Model Response for hallucinations based strictly on the provided Reference Context.

[Reference Context]

{{insert_source_text}}

[Model Response]

{{insert_llm_output}}

Evaluation Process:

1. Break down the Model Response into individual factual claims.

2. For each claim, check if it is explicitly supported by the Reference Context.

3. Note any contradictions, external facts, or unverified assumptions.

4. Assign a final score from 1 to 5 based on the rubric below.

[Rubric]

- 5: Perfectly Factual. Every claim is fully supported by the context.

- 4: No Hallucination, But Missing Context. Completely accurate but leaves out critical background.

- 3: Minor Hallucination. Mostly accurate but contains minor unmentioned details or embellishments.

- 2: Major Hallucination. Contains major unverified claims or changes key metrics/names.

- 1: Severe Hallucination. Completely fabricated or directly contradicts the context.

Output Format:

Reasoning: [Your step-by-step analysis of each claim]

Score: [Single digit choice: 1, 2, 3, 4, or 5]

Glossary 

• Recall

Recall measures how many of the relevant (positive) items were successfully retrieved.

Formula:
Recall = TP / (TP + FN)

Where:

• TP (True Positive): Relevant chunk was retrieved.
• FN (False Negative): Relevant chunk was not retrieved.
• TN (True Negative): Irrelevant chunk was not retrieved (not used in recall).
• FP (False Positive): Irrelevant chunk was retrieved (used in precision, not recall).

In RAG, the priority is to minimize False Negatives, because if the correct policy chunk is missing from the retrieved results, the LLM cannot answer correctly. That is why Recall@K is one of the most important retrieval metrics.

• HNSW

When you search for similar vectors (e.g., finding text embeddings with a similar meaning), an exact search requires calculating the distance between your query vector and every single vector in the database. This approach (O(N) brute-force complexity) is too slow for production apps.  

HNSW scales this down to logarithmic complexity (\(O(\log N)\)) by connecting vectors in a multi-layered graph, allowing the search algorithm to "skip" large chunks of the database.

Working : 

Searching an HNSW index mimics a highway system: you take the expressway to get close to your destination city, and then use local streets to find the exact house. 

1. Enter at the Top: The search begins at a predefined entry point on the absolute highest layer of the graph. 
2. Greedy Search (Top-Down): The algorithm evaluates the distance from the query vector to the current node's neighbors. It jumps to whichever neighbor is closest to the query.
    
3. Local Minima: When the algorithm can no longer find a neighbor closer to the query than its current node, it drops straight down to the exact same node on the next layer below. 
4. Repeat and Refine: It repeats this greedy search on the lower layer, leveraging the denser connections to fine-tune the match .
5. Final Stop: This process continues until it reaches Layer 0. On Layer 0, it uses a priority queue to keep track of the top efSearch closest neighbors, returning your final result

Parameters

When building or querying an HNSW index in databases like Pinecone, Milvus, Qdrant, or pgvector, you must balance accuracy (recall) against speed and memory by tweaking three parameters:

• M (Max Connections per Node): The maximum number of bidirectional links created for each new node at Layer 0.
  • Low M (e.g., 8-16): Less memory usage, faster builds, but lower search accuracy on complex datasets.
  • High M (e.g., 32-64): Better recall for high-dimensional data, but uses significantly more RAM. 
• efConstruction (Build Depth): Controls how many entry points are evaluated when adding a new vector to the index.
  • Higher values: Increases index build time but yields highly optimized graphs with better accuracy.
• efSearch (Search Depth): Controls the size of the dynamic candidate list evaluated during a query.
• Higher values: Increases search time but drastically improves accuracy (recall). This can be adjusted on the fly per query without rebuilding the index
• IVF

An Inverted File (IVF) index is an indexing algorithm used in vector databases to speed up similarity searches. By dividing a massive dataset into smaller, localized clusters (called "cells" or "partitions"), it allows the system to search only the most relevant data rather than calculating distances across every single vector.  

The IVF process relies on a two-step approach: 

1. Clustering (Index Creation): The database uses an algorithm like k-means to group all data vectors into distinct clusters, each represented by an anchor point called a centroid. Every vector is then assigned to its nearest centroid to create an inverted list of vectors per cluster.
   
2. Probing (Search Query): When you run a query, the system first finds the closest centroids and then searches only within the vectors of those specific clusters. 

Concepts

• nlist: The total number of clusters/centroids the index is divided into when it is built. A higher nlistcreates smaller clusters, which means fewer vectors to search per cluster. 
• nprobe: The number of clusters the algorithm is allowed to check during a search. A higher nprobeincreases accuracy but takes more time, while a lower nprobe increases speed. 

Variations

IVF is rarely used alone; it is usually combined with other methods to save memory or boost speed: 

• IVF_FLAT: Searches exactly within the selected clusters using brute force. It provides a great balance of speed and high accuracy.
• IVF_PQ (Product Quantization): Compresses the vectors within the clusters to heavily save on memory. It trades a tiny amount of accuracy for major storage efficiency.
• IVF_SQ (Scalar Quantization): Reduces the precision of the numbers in the vectors to save space.

When to use 

IVF is highly effective for massive datasets where a standard brute-force search is too slow. It trades a small percentage of absolute accuracy for a massive reduction in search time (often performing 10x to 100x faster). 

Training

To train an IVF index, you follow a three-step process: train the cluster centroids, add your vector data, and query the index using the nprobe parameter.

Here is how to implement and train it using Faiss (the industry-standard library for IVF).

Pipeline

• Train: The algorithm runs k-means clustering on a representative dataset to find the coordinates of the centroids (nlist).
• Add: The database assigns every vector in your dataset to its nearest centroid, building the inverted file lists.
• Search: The query looks at the closest nprobe centroids and searches only the vectors inside them.

import numpy as np

import faiss

# 1. Setup dimensions and dummy data

dimension = 128 # Size of your vectors

num_vectors = 50000 # Total dataset size

# Generate random vector data (float32 is required by Faiss)

np.random.seed(42)

data = np.random.random((num_vectors, dimension)).astype('float32')

# 2. Define the Quantizer and Initialize IVF

# The quantizer computes the distance between vectors and centroids

quantizer = faiss.IndexFlatL2(dimension)

nlist = 100 # Number of clusters to create

index = faiss.IndexIVFFlat(quantizer, dimension, nlist, faiss.METRIC_L2)

# 3. Train the Index

# This step runs k-means to find the 100 centroids

print(f"Is index trained? {index.is_trained}") # Output: False

index.train(data)

print(f"Is index trained? {index.is_trained}") # Output: True

# 4. Add data to the trained clusters

index.add(data)

print(f"Total vectors indexed: {index.ntotal}")

# 5. Query the Index

index.nprobe = 10 # Search the 10 closest clusters out of 100

query_vector = np.random.random((1, dimension)).astype('float32')

distances, indices = index.search(query_vector, k=5) # Find top 5 matches

print("Closest vector IDs:", indices)

 

Performance

• Training Data Size: You do not need your entire dataset to train the index, but you need enough. Faiss recommends between 30 * nlist and 256 * nlist vectors for the train() step.
• Choosing nlist: A good baseline is \(nlist = 4 \times \sqrt{N}\), where N is your total number of vectors.
• The Accuracy Trade-off: If your search results are inaccurate during testing, increase your index.nprobe value. If it is too slow, decrease index.nprobe.

 

• IVF-PQ 

• DiskANN

• Hierarchical Retrieval

Hierarchical retrieval (or Small-to-Big retrieval) is an advanced RAG technique that organizes data into multiple structural levels, such as summaries, sections, and detailed paragraphs. It mimics how humans scan a table of contents before diving into specific details, increasing search precision while preventing the loss of broader context.  

Instead of slicing documents into random, isolated chunks, the system links them in a parent-child or tree-like structure:  

Coarse Retrieval (High-level): The system searches across document titles or broad summaries to quickly pinpoint the most relevant document or major section.  
Fine-Grained Retrieval (Low-level): Once the right section is found, it zooms in to retrieve smaller, specific text chunks that directly answer the user's query.  
Context Condensation: The system selectively merges the detailed child-chunks into their larger parent-context before sending it to the LLM.  

Hierarchical retrieval greatly outperforms standard flat-retrieval in domains with dense, long, and highly structured information. Key applications include: 

Legal and Regulatory Compliance: Quickly referencing specific clauses while preserving the full paragraph or contract section for context.
Academic & Financial Research: Cross-referencing findings across multiple SEC filings or scientific papers.
Enterprise Knowledge Bases: Navigating complex policy documents across different departments.