Karthik Soman invented KG-RAG — knowledge-graph-based retrieval augmented generation — while building biomedical knowledge graphs at UCSF, and now leads enterprise-scale agentic AI at SAP America. One question carries from a PhD in computational neuroscience to the enterprise: how do you build intelligent systems that actually work in the real world? His answer isn't a better accuracy number but a different kind of answer — one a human can trace, question, and learn from. He walks through the case that convinced him: enriching patient records with a 40-million-node biomedical graph surfaced an olfactory-receptor gene that flagged Parkinson's five years early, catching a prodromal case a clinician had missed — not because the model was more accurate, but because it pointed at a mechanism. Then he moves to the enterprise, where the curated ontologies of biomedicine don't exist: why you lean on the topology already inside your documents, why graph-based reasoning is a 60-year-old idea LLMs merely made usable on the fly, exactly when a graph earns its value over vector RAG (multi-hop questions, smaller models, private data the model never saw), and the least glamorous advice in AI — data hygiene first, then AI hygiene.
Karthik Soman is a Data Science Expert at SAP America, where he leads the development of enterprise-scale agentic AI systems. He is the inventor of KG-RAG (Knowledge Graph-based Retrieval Augmented Generation), a framework that fuses knowledge graphs with large language models for context-aware, token-efficient knowledge retrieval. His research has been published in several high-impact journals, and he serves on the editorial boards of multiple peer-reviewed AI and bioinformatics journals. He holds a PhD in Computational Neuroscience from IIT Madras.
Four clear reads — who should act, and how urgently.
Direct, single-hop factual lookup? A strong model with vector RAG gets you most of the way. The graph earns its keep the moment a question requires chaining two or three relationships that never sit in the same chunk — or when you're on a smaller, cheaper model that can't lean on parametric knowledge. That's where you buy both accuracy and a traceable path to the answer.
You won't inherit UCSF's decades of curated vocabularies inside an enterprise. Instead of forcing freeform text into a clean ontology and stalling for years, build a loose graph from the structure already there — cross-references, document topology, the relationships encoded in your tables. Lower resolution, but a real knowledge landscape and a pragmatic path in.
A vector database retrieves by similarity but stores chunks disconnected — so it can return the right answer with no chain of provenance, and on private data the model never trained on, a confident wrong one. If your stakeholders need to see how an answer was reached, that requirement points past vector search toward a graph.
The least celebrated layer is the one everything rides on: a model is only as good as its data. Fix the foundation before the strategy deck. And as vibe-coding democratizes AI, the test of responsible use isn't whether you can make something work — it's whether you know how to fix it when it breaks. A thousand solutions on unhygienic data is a thousand problems.
VivekYou've done groundbreaking work on SPOKE at UCSF, you developed KG-RAG, you've spoken at conferences — but that's the LinkedIn resume. I'd love the real story: how did you end up doing what you do now? Walk us through the journey.
KarthikEach step felt like a natural progression rather than a leap. It started with a curiosity about the brain — my PhD at IIT Madras was in computational neuroscience, using mathematical models to understand how neurons communicate. What that really taught me was how to think about complex networks and extract meaning from highly complex data. The next step was biomedical knowledge graphs at UCSF, and there's a beautiful connection: the brain is a network, and it's that interconnected structure that gives it its power. Knowledge networks give connected data the same kind of power. That's where I invented KG-RAG — a framework that combines knowledge graphs with LLMs to make AI more accurate, more trustworthy, and grounded in real-world knowledge. Instead of the model relying only on what it learned in pre-training, KG-RAG extracts a relevant subgraph and hands it to the model as precise, structured context. The number I'm proudest of: KG-RAG got a campus-wide deployment at UCSF. That work at the intersection of graphs and generative AI is exactly what brought me to SAP. As Andrew Ng put it, AI is the new electricity — it's domain-agnostic. I changed domains completely, but the same challenge follows me: building AI that's powerful and reliable, explainable, and grounded. I never really switched fields — I just kept following one question: how do you build intelligent systems that actually work in the real world?
VivekPeople casually say LLMs are an excellent representation of the human brain. There's clear evidence they miss a lot — the System 1 / System 2 split, for one. You studied how the brain processes information and you've worked extensively on LLMs. What's your read — are we there, or what's the missing piece?
KarthikWe're comparing apples and oranges. It's artificial intelligence, not the human intelligence we see in the brain. The connection is real at a high level — early perceptron work was trying to emulate what the brain does — and an LLM is a neural network: it works because of the weight connections between artificial neurons, trained by backpropagation. But the brain is a dynamical system. Its basic unit is far more nuanced; there are multiple cognitive centers, each itself a dynamical system. You can study the brain at the single-neuron level — Hodgkin-Huxley — at the level of oscillations tied to states of mind, or at the functional level with MRI. Across all of it the consensus is that the brain is dynamic. An LLM's basic unit, by contrast, is a static neuron. What's amazing is that with enough representation capacity it extracts patterns from trillions of tokens and generalizes — the universal approximation theorem says a big enough network can approximate almost any function. These are stacked mathematical functions doing an amazing job. But equating them with the brain is a huge oversimplification; we're not giving the brain enough credit. Karpathy was asked the same thing on a podcast and landed in the same place.
VivekYou were using graphs long before LLMs were popular. What changed with LLMs — and where have you found a knowledge graph more powerful than a conventional ML model?
KarthikLet me take that with a specific project, pre-ChatGPT, at UCSF: early detection of Parkinson's. The neurons degenerate long before diagnosis; a clinician diagnoses only once motor symptoms like tremor show up, but the degeneration starts years earlier. So we asked: can we diagnose Parkinson's before a clinician can? The standard ML approach takes a patient's history and returns a risk score — "this patient has a 77% probability." Useful, but a black box. The clinician is left asking, why? That transparency is exactly what typical ML models lack. What the knowledge graph gave us was different in kind.
KarthikWe enriched each patient's electronic health record with SPOKE, a biomedical knowledge graph we built at UCSF — close to 40 million nodes, over 70 million relationships, integrated from more than 40 public databases: gene ontology, disease ontology, UniProt, NCBI taxonomy. You can traverse it: start at a gene, find the protein it encodes, the disease that protein is associated with, the compound that treats it, whether that compound regulates the gene. A patient reports maybe nine clinical variables — symptoms, medications, lab tests. We connect those to the graph, run a graph algorithm to pull in the surrounding biology, and build an enriched patient signature — driven by clinical data but carrying gene and protein context that clinical data alone never has. Train any ML model on that and the accuracy jumps versus raw clinical features. But accuracy isn't the point I hinge on — it's explainability. The model can say: I think this patient has Parkinson's because I see these genes and proteins reported as biomarkers, even though they weren't in the clinical record.
KarthikOne striking result: a gene encoding an olfactory receptor turned out to be highly relevant for identifying patients five years before their actual diagnosis. That connects to real biology — loss of smell is one of the earliest prodromal symptoms of Parkinson's, often years before any motor sign. The graph didn't know the patient reported smell issues; it inferred a biological pathway that pointed at the same mechanism. And a movement-disorder specialist independently reviewed charts — in one case our model correctly flagged a patient as prodromal that the clinician had missed. Not because the clinician wasn't skilled, but because the signal was scattered across biological connections no human can trace across a chart in real time. That's not a better number — it's a different kind of answer. One a clinician can question, investigate, and learn from. That's when I realized knowledge graphs aren't a nice-to-have for accuracy; they're essential for building AI clinicians can reason with, not just take orders from.
JoshuaWhat struck me is you're using two modes at once. One is strict typification — entities, classes, relationships, the RDF world, which you can do in a property graph too. But you're also running data science on those connections, because some things aren't codified as rules yet — you don't know A causes B — and you vectorize that. So you've got a symbolic side and a neural side fused together.
KarthikExactly — that culmination of symbolic AI and the machine-learning approach is what makes the difference, especially in an accuracy-sensitive space like biomedicine.
VivekIn healthcare you had beautifully curated ontologies you could use almost out of the box. Now you're at a much larger enterprise, with messier data. What's the ground reality — their readiness to have ontologies in-house, or even the awareness of what an ontology is?
KarthikSPOKE stood on decades of community-curated ontologies — gene ontology, disease ontology, UniProt, NCBI taxonomy — vocabularies thousands of researchers refined for years. Every gene has a stable identifier, every disease a consistent name across sources. Integrating them still took real work, but the underlying vocabulary already existed and was trustworthy. In the enterprise, data harmonization is a fundamentally different challenge — and not a governance failing. Companies invest heavily in standardizing structured data, where there's a schema and defined fields. But freeform text — documents, wikis, tickets — lives in a different world, because natural language doesn't carry a schema's built-in discipline. The same concept gets phrased ten ways across ten documents, with no stable identifier to anchor it. The instinct is to do named-entity recognition, entity resolution, relation extraction — harmonize everything into a clean ontology. That can hit a pragmatic ceiling, not because the data is disorganized, but because freeform text lacks the terminological consistency biomedical curation gives you.
KarthikSo a different approach worth considering: lean on the topology already in the documents. Look at Wikidata — the knowledge base itself has a landscape: how documents cross-reference each other, the structure within a document. Rather than imposing an ontology on top of unstructured text, build a graph out of the documents' structure. And for structured data — tables, databases — build the graph directly from the relationships already encoded in the data. Ontologies are incredibly powerful when they exist. But where that curation infrastructure isn't available, the structure of the data itself can be the starting point for graph-based reasoning. It's not a replacement for a well-curated ontology — it's a pragmatic path in, when the alternative is waiting years to build one from scratch.
JoshuaEven within structured data, the same column or table name can mean something completely different elsewhere in the enterprise. How should enterprises control meaning reliably, when the same term carries a different context in another system?
KarthikThat's entity resolution — an old problem. "Apple" is a company or a fruit depending on context. At enterprise scale the key word is scalability: those ambiguous terms are scattered across thousands of tables. I'd draw on Proto-OKN — NSF's Open Knowledge Network — a multi-institution project to federate disparate knowledge bases into one platform with a shared schema. And NIH's Biomedical Data Translator, from NCATS, did the same for biomedicine with a common language called BioLink: define your own schema, but the moment your database enters the ecosystem it has to speak BioLink. Standardization is the very first thing, especially at enterprise scale. Whatever consensus prolific researchers reached to make a country-wide collaboration work carries weight you can translate into the enterprise — many departments, many tables, ambiguity in the terminology, unified once they share a common language.
KarthikI see AI in three layers: data is the most foundational, then application, then the model. You can build all the nifty models you want, but if your foundational data layer is crappy, your model is crappy — a model is only as good as its data. Investing time, funding, and energy into the data-standardization pipeline and seamless integration across modalities has a ripple effect that reinforces every downstream model.
JoshuaSo partly this is a data mesh across functions. What about hierarchically, by domain — finance has FIBO, legal has LKIF. Would you recommend starting from the domain ontology and layering the company's specifics on top?
KarthikStick to first principles — don't reinvent from scratch. Take SPOKE: think of gene, disease, and organism as different departments. Just as departments in an organization synergize, these biomedical departments have dynamic connections between them. But we didn't build one giant ontology for the whole ecosystem — there's a disease-specific ontology, a gene-specific one, a symptom-specific one. Data modeling and data integration are two separate pillars. Each domain should invest in its own vocabulary, because that's where the nuanced semantics live — the way "Apple" means different things by context. Once each domain has a defined ontology, the federated integration — the data mesh — becomes seamless.
KarthikFor domains without a curated ontology, I've personally adopted what I'd call an ontology-free, or loose-KG, approach — capturing the document structure itself. A single node isn't a highly granular entity; it can be a paragraph, or a list of cross-references. You don't get super-high resolution, but you get the high-level knowledge landscape — and that's proven invaluable for RAG.
KarthikPeople build a vector-database RAG and it answers the question — but they come back and say it's not giving me the chain of provenance. I need to know how this document relates to the others we use day to day. A vector database can't give you that; in its essence the data is disconnected — you store semantic embeddings, and yes you can bolt on parent-child metadata, but that hits a ceiling. A knowledge graph gives you those topological connections for free. Walking that graph — the chain of provenance — is what made stakeholders happy: not just the accuracy, but being able to see how the answer was reached.
JoshuaEverywhere you look in computing, there's a graph.
KarthikExactly. People treat graph-based reasoning as a brand-new idea, but it's decades old. Ross Quillian, around 1967, modeled human knowledge as semantic networks — concepts as nodes, relationships as edges, inference as literally walking the graph. That's 60 years before LLMs. In the 70s and 80s, Minsky and Brachman formalized it into frame systems and conceptual graphs — the RDF flavor. The 90s and 2000s gave us the semantic-web stack — RDF, OWL, SPARQL — and large-scale graphs like DBpedia and Freebase, used for reasoning and question answering at scale. The 2010s brought representation learning — TransE, TransR, RotatE, node2vec — and graph neural networks like GCN and GraphSAGE. So graphs for reasoning are an established research tradition. What LLMs changed isn't the idea of using graphs to reason — that's 60 years old. What's new is that models can now read and use graph structure as context, in natural language, on the fly.
KarthikThat 71% came from benchmarking on BioMixQA — a dataset domain experts curated for this, which we released on Hugging Face. The 71% relative improvement in question answering was on Llama-2 13B, a smaller open-source model. We saw improvement on GPT-3.5 and GPT-4 too, but a smaller margin — because GPT-4, in 2023, already had a lot of biomedical knowledge baked into its parameters. So for straightforward, single-hop factual questions, a strong model with standard vector RAG is often good enough. But for complex questions that need multi-hop reasoning — chaining two or three relationships that don't sit next to each other in any single document — a knowledge graph helps significantly. In a vector database each chunk is disconnected; you retrieve by similarity, with no explicit relationship from one chunk to the next. A knowledge graph encodes those relationships directly — this gene interacts with this protein, implicated in this pathway, associated with this disease. When a question requires walking that chain, the graph already has the connective tissue. So: mostly direct lookup, vector RAG plus a strong model gets you there. The moment you need multi-hop reasoning, or you're on a smaller, cheaper model, that's where a graph earns its value — in accuracy and in traceability.
JoshuaAnd your biomedical QA was public — probably already in the model's training. A company's private knowledge isn't. So the chance it's in the model's parametric memory is low, the hallucination risk is higher, and you won't get real provenance.
KarthikA hundred percent. And there's a difference between making data public and getting insight out of it. There's the informatics pyramid: raw data at the bottom — you can publish it, it doesn't matter. Label it and it becomes information. Draw connections between those labels and it becomes knowledge. Traverse those connections for mechanistic insight and that's wisdom. Data to information to knowledge to wisdom — that's exactly what a knowledge graph provides, whether or not your data is public. A standard vector-RAG approach mostly leaves you at the information level. The connections — knowledge — is where it hits a ceiling: how many top-K do you retrieve, and how do you know it's the right K? For a complex question you might need a thousand; you can't know a priori. A knowledge graph gives you that organically.
VivekThe question we ask every guest: if you could change one thing about how enterprises approach this, and it happened tomorrow — what would it be?
KarthikTwo things. First, data hygiene — spend more on the foundational data layer. It's the least celebrated but the most important; everything is built on it, and it matters for any enterprise, Fortune 500 or startup. The earlier the better. Second, AI hygiene — governance. With AI democratization, people without much formal AI knowledge can now build things, which is genuinely good. But building something that works doesn't make it a good solution. The real distinction between raw usage and responsible usage isn't whether you can make something work — it's whether, when it breaks, you know how to fix it. No coding agent or vibe-coding tool will do that for you. Picture a thousand people building a thousand AI solutions on unhygienic data — the problem compounds. Start from first principles: good data hygiene, and democratize AI responsibly with AI hygiene.
VivekData hygiene and AI hygiene — simple, grossly underrated, massively ignored, and gloriously boring advice. Thank you, Karthik.
Open this episode in your assistant with the summary, key points, and a link to the full transcript — then put the insights to work in your own context.
Opens in a new tab · shares only this episode's public transcript link
Melli Annamalai has spent 27 years at Oracle watching technology waves crest and break — multimedia retrieval, the semantic web, big data, property-graph analytics, and now knowledge graphs for AI. As the Distinguished Product Manager who leads graph technologies there, she makes an argument you rarely hear from a database vendor: a knowledge graph is one tool in the kit, not the destination. She traces why semantic-web tech stalled for two decades — a steep learning curve, a custom RDF/OWL/SPARQL ecosystem, and the year-and-a-half payback that senior management wouldn't fund — why AI finally gave enterprises a reason to invest, and where these projects still quietly fail: over-engineering everything into a graph, tuning and tooling gaps, and the security officer who shuts it all down. Along the way, a working definition of "ontology" for non-technical buyers, natural language as the new query language, and Oracle's converged-database bet to collapse the graph, vector, and agent layers into the place the data already lives.
Alan Morrison has spent two decades watching technical vocabularies get coined, hyped by vendors, and hollowed out — knowledge graphs, grounding, and now "context." Drawing on 20 years running PwC's Technology Forecast think tank, he argues the vocabulary problem in enterprise AI isn't a communication failure but an architectural one: when words stop meaning anything, architecture follows marketing and teams hold the repair bill for years. He makes the case for data-centric over application-centric, why incumbents stay stuck, and the smallest credible first step toward a knowledge-graph layer.
Tony Seale spent two decades stitching together siloed data inside investment banks — Lehman, Deutsche Bank, UBS — until he concluded the hard part was never the model, it was the data. Now widely followed as "the knowledge graph guy," he makes a precise argument: a large language model is rented intelligence you don't own, while the ontology and knowledge graph you build are the durable IP that stays inside your organization. He walks through the neuro-symbolic loop — why LLMs are continuous and knowledge graphs discrete, and why you cycle between them rather than pick one — coins Seale's Law on how marketing hollows out a word's meaning, and warns that companies that don't consolidate their "ontological core" will watch their advantage quietly leak out. Along the way: semantic data products and the DPROD standard, why true open standards (schema.org, W3C) beat vendor "open," and why the agentic web arriving next makes getting your data estate in order the only move that counts.