For Data Labs

Source-backed verification for alignment workflows.

Domain tasks often lack a ready-made correctness check. CogniSwitch turns domain knowledge into explicit criteria for training data, reward functions, and quality assurance.

SOPBench / healthcarePublished trace

What makes this
action permissible?

The agent wants to appeal a claim.
The procedure requires an appeal-window check.

Policy window
30days
Elapsed time
45.5days

Limit exceeded. Elapsed time 45.5 days exceeds the 30-day appeal window. Verdict: not permitted.

Verdict

Not permitted

Appeal window exceeded
Evidence behind the judgment

Turn 6get_claim_details2024-12-01

Turn 10interaction_time2025-01-15

Two published traces. The same domain requirement.
A missing check and a failed check produce distinct evidence.

Domain knowledge supplies the verification criteria that the task does not provide on its own.

Every headline SOP breach identified.
Without an LLM judge.

In SOPBench’s healthcare and bank evaluations, the CogniSwitch Knowledge System identified every breach in the headline comparison, checking agent behaviour against procedure requirements.

Healthcare

01

Insurance and provider administration

5 agents · 733 judged conversations

Oracle-identified breaches caught
242 / 242
Agreement on oracle-compliant conversations
487 / 487

No finding on the 487 oracle-compliant conversations.

Healthcare results

Banking

02

Agent procedure compliance in banking

9 agents · 1,474 headline conversations

Oracle-identified breaches caught
749 / 749
Agreement on oracle-compliant conversations
723 / 725

Two flags on oracle-compliant conversations.

Banking results

Headline figures follow the exclusions documented in each report. These evaluations measure agreement with SOPBench’s code oracle; they do not establish comparative accuracy against an LLM judge.

Scope and exclusions

Healthcare accounting

Four replay-shape cases are held out of the headline result. Fifteen conversations were not judged because their task pairing was ambiguous or unresolved. The evaluation covers healthcare insurance and provider administration, rather than clinical diagnosis or treatment.

Banking accounting

Thirty-two judged conversations are excluded from the comparison because an independent test of corrected benchmark code identifies oracle errors; they include all 21 oracle breaches the verifier did not flag. Thirty-one would otherwise be in the headline. Thirty-six mismatched conversations were not judged. The headline counts each agent’s first try at each task.

A verdict you can inspect.

Each judgment connects the agent’s recorded actions to the applicable SOP requirements. Graph retrieval and audit inference produce the verdict and supporting trace without an LLM judge.

Explore the healthcare verification records

One domain foundation.
Three offering packages.

Use explicit domain knowledge to prepare training data, provide reward signals, and check quality across your alignment workflow.

01

Data preparation

Verified SFT datasets

Domain-grounded training examples with reasoning traces and edge cases.

Rubrics

Structured scoring criteria linked to domain rules.

What the lab gets

Training examples and evaluation criteria with an inspectable basis for correctness.

02

Training

Graders / reward functions

Score outputs and actions against domain rules.

RL environments

Simulated workflows with tools, policies, and explicit checks.

What the lab gets

Verifiable reward signals and environments for practicing domain-specific tasks.

03

Quality assurance

Label QA / audit

Check labeled data against domain knowledge and flag contradictions.

Benchmarks / evals

Held-out, rule-grounded test sets.

What the lab gets

Checks on dataset quality and evidence of model or agent performance.

Shared foundation

Domain-ontology engine

Explicit domain knowledge
across all three packages

Start with the package your workflow needs. Extend the same domain foundation as your requirements grow.

Bring your
benchmark dataset.

Bring your benchmark dataset, domain policies, scoring criteria, and reference judgments. Measure verification coverage, judgment quality, latency, consistency, and cost on the tasks your team already evaluates.

Evaluate your benchmark

What we’ll measure

Coverage
Which tasks and criteria can be verified.
Judgment quality
Agreement, disagreements, and supporting evidence.
Latency
Time per evaluation, with knowledge preparation measured separately.
Consistency
Whether repeated evaluations return the same verdict for the same inputs and knowledge version.
Cost and effort
Preparation, evaluation, review, and maintenance.