Every AI vendor shows up to a demo with a pre-cooked setup. None of them generate a document live and run it cold. They won't even change a single word in the prompt.
BoP is a weekly LinkedIn Live where CogniSwitch co-founders Vivek and Josh put their own product on the spot — live, unscripted, on data we generate in front of you. We generate a synthetic enterprise document on the spot, run questions through CogniSwitch, and show you whether we get the same answer every time. Including when we don't.
CONSISTENCY
Context graph retrieval is deterministic — the same ontology entities and triples come back regardless of how the question is phrased. LLMs will still generate and omit from that data. We tested both. We proved both.
Q1 asked 3× returned the same 21 entities and 16 triples every time. Retrieval is deterministic under identical conditions.
Q2 and Q3 used different phrasing for the same clinical intent. Equivalent concept sets retrieved. The graph handles paraphrasing.
"children" ≠ "pediatric" in SNOMED CT. Q4 returned 9 of 18 expected entities. A retrieval failure, not a generation failure.
Even from identical, deterministic retrieval, LLM verbosity and emphasis shifted across runs. Open-ended questions got longer answers. Pointed ones got shorter ones. This is the generation layer — a separate problem from retrieval.
Pediatric Antibiotic Prescribing Protocol
Generated live using Claude during the session · Format: Markdown / PDF
Same question, three runs. Does retrieval return the same result every time?
✓ ConsistentSame intent, different phrasing. Does retrieval generalize across paraphrases?
✓ ✓ ✗ — Gap at Q4Antibiotics should be prescribed only when there is clinical or laboratory evidence of bacterial infection, or when empiric treatment is warranted due to high clinical probability of bacterial origin.
Antibiotic prescription is indicated when bacterial infection is confirmed or strongly suspected clinically. Timing is critical — early in severe cases, watchful waiting in mild viral presentations.
Pediatric antibiotic dosing is weight-based (mg/kg). Age-band adjustments apply across neonates, infants, toddlers, school-age, and adolescents. Renal and hepatic function further modify dosing. Maximum daily doses cap weight-based calculations.
Antibiotic doses are calculated based on patient weight in kilograms, with specific mg/kg ratios varying by antibiotic. Maximum daily doses apply.
Incomplete retrieval → incomplete generation. The LLM had no age-band data to work with.
Semantic Discontinuity
The term pediatric is encoded in SNOMED CT within the source document. The term children is a natural language synonym not mapped as an equivalent concept in the ontology. When Q4 used children instead of pediatric, the system traversed a different graph path and returned only 9 of the 18 expected entities — the age-band breakdowns were not retrieved.
Define the synonym relationship once in the ontology. It resolves consistently thereafter. This is a graph problem with a graph solution — not a prompt engineering workaround.
Define the synonym relationship once in the ontology.
Gap types like this are categorized and scored in the CogniSwitch Context Quality Index.
Read the CQI paper →Single synthetic document. Real enterprise data is messier, multi-source, and contradictory. Cross-document retrieval consistency is untested here.
The ontology gap was surfaced, not fixed. We did not re-run Q4 after adding the synonym to confirm the gap closes. That is the next experiment.
One domain, one session. We tested in healthcare because SNOMED CT is a well-structured ontology. Other domains may have different gap profiles.