Why Deterministic Retrieval is Critical For Enterprise Grade AI Systems

Developers building GenAI Apps for Enterprises are quickly realising that it’s all about “The R” of RAG. Retrieval is the key and rightly so. GenAI use cases in Enterprise usually involve Document RAG. Documents like product documentation, SOPs, SOWs, Contracts, Operating procedures and guidelines and so on. The information is set in stone with 0 room for generation. For instance - In a manufacturing plant, detailed product documentation of a machinery is already available.If a manufacturing process requires a 15 point checklist, that’s the fact. Nothing can change that. The use case revolves around retrieving select information about the machine’s operation. Nothing needs to be generated. In fact, there is no room for generation. 

Enterprises need RAR - Retrieval Augmented Responses. Deterministic Retrieval is all you need to get going with GenAI in Enterprises. Developers have been bombarded with tons of RAG approaches. There are as many XX RAG methods/ approaches, and dozens of new papers on Arxiv being published aggressively and this where developers end up missing the trees for the woods. In this article, we will cover what exactly is deterministic retrieval, and why does it matter so much in the context of modern AI and also briefly touch upon how to get started.

Understanding What is Deterministic Retrieval

At its core, deterministic retrieval is a method of information retrieval that guarantees consistent and predictable results. AI systems using deterministic retrieval, would return the same output for a given input query and knowledge base. Always. This consistency is not just a matter of convenience—it's a fundamental requirement for building AI systems that can be relied upon. This is precisely what makes deterministic retrieval a key fundamental for mission critical systems like Legal, Finance, Pharma to name a few. Here are few examples to understand this better - 

• Running SQL queries on RDBMS - Classic example of deterministic retrieval.

 SELECT * FROM Customers WHERE CustomerID = 'ALFKI'; 

This query will always return the same result set, assuming the database hasn't been modified between executions.

• Knowledge Graph Navigation - Another classic example where graph traversal gets you to the same information for the same query, always

What Makes Deterministic Retrieval appealing to Enterprise Use Cases

Enterprise AI systems have 5 key prerequisites and this short list can become a sort of guiding checklist as developers choose their GenAI stack. 

1. Reliability and Reproducibility
2. No Black Boxes i.e Explainability
3. Precision 
4. Consistency Across Scaling
5. Controls, Compliance & Auditing 

Let’s now understand how deterministic retrieval approach fares on these key principles:

• Reliability and Reproducibility: Deterministic retrieval ensures that given the same input query and knowledge base, the system will always return the same results. This consistency is mission critical in high-stakes domains like healthcare or finance.

2. No Black Boxes ie. Explainability: With Deterministic retrieval, developers always have a clear audit trail making it's easier to trace back and explain the original source of specific information.This also makes it's easier to identify and fix issues in the system, as well as to measure improvements over time.

3. Precision: Language models can often "hallucinate" or generate false information. Deterministic retrieval helps ground the AI's responses in the enterprise knowledge system, eliminating the risk of fabricated or incorrect information. For businesses dealing with proprietary or sensitive information, deterministic retrieval ensures that responses are based on approved, factual content rather than potentially inaccurate or outdated information.

4. Controls, Compliance and Auditing: Enterprise deal with sensitive data with specific access levels - these can be baked into the knowledge graph like structure, ensuring only the right user can retrieve the right information. Structures that enable deterministic retrieval ensure these controls are baked in.

There are more challenges specially around scaling and maintaining consistency. Enterprise data sources are dynamic. Product information, operating procedures are updated, modified, deleted with time. As knowledge bases grow, deterministic retrieval maintains consistency, which is crucial for large-scale enterprise applications. In essence, conventional RAG techniques leveraging only vectors and embeddings fall short on all of these.

Purely Vector based Retrieval falls terribly short on all of these.

Understanding Reasons Why Hallucinations Occur in Vector RAG

1. Vector Similarity ≠ Semantic Relevance: Similar vectors don't always correspond to semantically relevant information, leading to retrieval of irrelevant data more than often.
2. Context Sensitivity: LLMs can misinterpret the context of retrieved chunks, leading to responses that don't accurately reflect the original information. This problem compounds when the retrieved chunks were only half relevant.
3. LLMs are creative: LLMs are excellent at generate plausible text aka predicting the next toekn, which leads to "filling in gaps" with information that seems correct but isn't present in the retrieved data.
4. Lack of Grounding: Unlike deterministic systems like a Knowledge Graph, Vector RAG doesn't have a mechanism to verify generated information against a known, factual database. Knowledge Graph based systems act as the ultimate knowledge database.
5. Training Data Influence: LLMs may introduce information from their training data, which could be outdated, biased, or irrelevant to the specific query.

By contrast, deterministic retrieval bypass these issues by working only with explicitly defined, pre-validated information in a structured knowledge graph. This ensures that every piece of information in the response can be traced back to a verified source, eliminating the risk of hallucinations.

‍

Why Changing Algorithms or Adding Reranking Doesn't Fully Solve Hallucinations

Changing search algorithms or adding reranking, only gets you so far in terms of absolute reliability.

The core challenge lies in the fundamental nature of how LLMs generate text based on probabilistic patterns rather than explicit, verifiable facts.

‍

• Using Different Search Algorithms - While this definitely improves the quality and relevance of retrieval but it doesn't guarantee factual accuracy or exhaustiveness of the retrieved information.
• Impact of Reranking : Reranking helps prioritize more relevant passages among the initially retrieved set. This can and does lead to better context being provided to the LLM for answer generation. However, reranking still operates on the initially retrieved set and doesn't introduce new information. If the retreived information had noise, the TopK can still have factually irrelevant information.
• Persistent LLM related issues - The core problem is about pushing in large chunks of text through LLM and their core nature. LLMs ability to generate plausible text and combine information in a manner that seems logical but is factually incorrect, exposes the application to a ton of reliability issues.

In contrast, deterministic retrieval systems:

• Work with explicitly defined, pre-validated information.
• Don't rely on generating new text, only on combining known facts.
• Can trace every piece of information in the response back to a verified source.

‍

Impact of Document Scale on Retrieval and Hallucinations

Large enterprises deal with 100s of documents each with 1000s of pages. Unlike most demo RAG applications which are built on small PDFs with a couple of pages, the level of complexity is exponentially different when you take the same RAG application to an enterprise context.

While larger document sets provide more comprehensive information, they also significantly increase the challenges in accurate retrieval and the potential for hallucinations. This underscores the importance of sophisticated retrieval algorithms, effective reranking strategies, and possibly hybrid approaches that combine the strengths of both vector-based and deterministic retrieval methods.

1. Precision vs. Recall Trade-off: In large document sets, there's often a trade-off between precision (getting exactly the right information) and recall (getting all relevant information). This trade-off directly impacts risk of hallucination:
   • High Precision, Low Recall: May miss important context, leading to gaps, which can trip the LLM.
   • Low Precision, High Recall: May introduce too much irrelevant information, increasing the chance of the LLM making incorrect assumptions.
2. Contextual Understanding: As the scale increases, it becomes more challenging for the retrieval system to understand the full context of the query and the retrieved information. This can lead to mismatches that the LLM might try to reconcile in incorrect ways.
3. Managing Conflicting, Correlated and Causal Information: Large document sets often contain concepts that are wide ranging and occasioanally conflicting information. Think about 1000s of research papers on immno-oncology. LLMs struggle to reconcile these differences, potentially leading to inconsistent or hallucinated responses.
4. Retrieval Depth vs. Speed: With large document sets, there's often a practical limit to how many documents can be retrieved and processed in a reasonable time. This limitation might lead to missing crucial information, increasing the risk of hallucinations. This also directly impacts the agent performance and subsequently the end user experience.
5. Query Ambiguity: In a small document set, query ambiguity is less of an issue as the scope is limited. In large sets, ambiguous queries can lead to retrieving a wide range of tangentially related information, increasing hallucination risks.

This further highlights why deterministic approaches, which rely on structured knowledge rather than large, unstructured document sets, can be particularly valuable in scenarios requiring high accuracy and reliability.

‍