Hybrid Retrieval Pipeline & Reciprocal Rank Fusion

Overview

Traditional Retrieval-Augmented Generation (RAG) pipelines rely solely on vector distance, frequently returning conceptually similar text that is operationally irrelevant or factually superseded. The Hybrid GraphRAG retrieval engine (src/retrieval/) combines:

1. Dense semantic vector search (Qdrant)
2. Sparse exact-keyword search (SPLADE / BM25)
3. Structural graph traversal (Memgraph)
4. Zero-trust PostgreSQL validation (EvidenceGate)

Pipeline Architecture & Execution Flow

                           [ User Query: "CDS audit trail requirement" ]
                                                 |
                   +-----------------------------+-----------------------------+
                   |                                                           |
                   v                                                           v
       [ Dense Vector Search ]                                     [ Sparse Vector Search ]
       Embedding: 4096-dim dense                                   Vocabulary: SPLADE weights
                   |                                                           |
                   +-----------------------------+-----------------------------+
                                                 |
                                                 v
                                  [ Reciprocal Rank Fusion (RRF) ]
                                  Computes fused rank score RRF(d)
                                                 |
                                                 v
                                  [ Memgraph Graph Context ]
                                  Enriches with multi-hop neighbors
                                  (:Requirement)-[:VERIFIES]->(:Test)
                                                 |
                                                 v
                                  [ EvidenceGate Zero-Trust Filter ]
                                  Live PG Hash Check & Supersession Filter
                                                 |
                                                 v
                                  [ LLM Prompt Context Assembly ]

Reciprocal Rank Fusion (RRF) Formulation

When combining candidates from dense vector search ($D$) and sparse keyword search ($S$), the retriever computes the Reciprocal Rank Fusion score for each document chunk $d$:

$$RRF(d) = \sum_{m \in {dense, sparse}} \frac{1}{k + \text{rank}_m(d)}$$

Where:

• $k$ is a smoothing constant (configured to $60$ by default).
• $\text{rank}_m(d)$ is the 1-based ordinal position of document $d$ in the result list from model $m$.

Chunks appearing near the top of both dense and sparse retrieval lists receive significantly higher composite scores than chunks that only match in one modality.

Graph Expansion via Memgraph

Once candidate chunks are identified, the retriever extracts the referenced entity codes (such as URS-CDS-004 or SOP-QA-012) and queries Memgraph:

MATCH (req:Requirement {code: $req_code})
OPTIONAL MATCH (req)<-[:VERIFIES]-(test:TestProtocol)
OPTIONAL MATCH (req)-[:SPECIFIED_BY]->(spec:Specification)
OPTIONAL MATCH (req)<-[:IMPACTED_BY]-(dev:Deviation)
RETURN req, collect(DISTINCT test) as tests, collect(DISTINCT dev) as deviations

This multi-hop context is appended directly to the retrieved chunk payload, enabling the downstream Pydantic AI agent to evaluate validation coverage and open quality incidents simultaneously.