Verifying agent procedure compliance in SOPBench's bank domain

Dilip IttyeraCogniSwitchNovember 2026SOPBench bank evaluation

The CogniSwitch Knowledge System audited agent procedure compliance on SOPBench's bank tasks without an LLM judge. In the headline evaluation of 1,474 conversations, covering nine agents and two customer behaviors, SOPBench's recomputed oracle identified a breach in 749. The CogniSwitch Knowledge System identified all 749, and concurred with the oracle on 723 of the other 725.

Each conversation was judged from the conversation alone. Facts come only from the agent's own tool results, each agent is held to the conditions its prompt states, and a check the conversation never established counts as not established. Judging took about 2.4 seconds per conversation, 86 minutes in all.

Thirty-two conversations are taken out of the comparison because a test that corrects named faults in SOPBench's code, run without reading our verdicts, shows that SOPBench's verdict on them rests on its own error. They include all 21 SOPBench breaches the CogniSwitch Knowledge System did not flag. Separately, 337 conversations are stored under another task's record: 301 were re-paired to their own task and 36 were not judged.

The study measures agreement with SOPBench's oracle and does not measure the oracle's own accuracy. The healthcare evaluation is reported separately.

1Background

SOPBench converts a written procedure into a directed graph of executable functions and replays each agent transcript through its own code to decide whether the agent followed the procedure [1]. Its bank domain covers account and card operations such as deposits, transfers, credit-limit changes and card requests, each gated by checks the agent must complete first.

The CogniSwitch Knowledge System verifies the knowledge an agent relies on before the agent acts. A pack selected from a catalog of ontologies, SOPs, standards and regulations defines what counts as knowledge for a use case, and a rules evaluator applies the procedure to reach a verdict that cites the provision. During auditing, it audits the claims and evidence produced by the LLM agents, based on the SOP and the inputs provided to them.

Our question was whether the CogniSwitch Knowledge System could reproduce SOPBench's bank verdicts from the conversations alone, without an LLM judge. SOPBench's code oracle supplies the reference verdicts. The CogniSwitch Knowledge System audits the same conversations, and this report compares its verdicts with the oracle's. The healthcare report describes the system and the benchmark in more detail.

2Data and setup

We used SOPBench's released bank result files without modification to the transcripts. Each conversation was judged from the conversation alone: the facts come only from the agent's own tool results, and a check the conversation never established counts as not established. Each agent is held to the conditions its prompt states.

The reference verdict is SOPBench's own code, re-run on each conversation against its own task. We did not use the verdicts stored in the result files, for the reason given in section 4.

2,119 conversations were judged. A record can hold several tries by one agent with one customer; later tries repeat the first and are not independent, so the headline counts each agent's first try at each task, less the conversations taken out in section 5. A later try counts when the earlier tries in its record were paired with another task or with none.

3Results

Of the 1,474 conversations in the headline evaluation, SOPBench's oracle identified a breach in 749. The CogniSwitch Knowledge System identified all 749. Of the 725 conversations the oracle judged compliant, it concurred with the oracle on 723 and flagged 2. Table 1 breaks the result down by stratum.

Table 1. Agreement with SOPBench's recomputed oracle verdicts on bank tasks, by stratum. "Breaches" counts conversations with an oracle-identified breach; "caught" counts those the CogniSwitch Knowledge System also identified, for either reason. "Other flags" counts conversations it flagged that the oracle judged compliant.
StratumBreachesCaughtAgreementOther flags
By breach type (SOPBench's verdict)
Acted when a required condition was not met653653100%–
Acted before performing the required checks739739100%–
By agent under test (customer behavior)
GPT-4o (ordinary)3434100%0 of 92
o4-mini-high (ordinary)1515100%0 of 115
Claude 3.7 Sonnet (ordinary)2626100%0 of 98
Qwen 2.5 7B (ordinary and adversarial)228228100%0 of 20
Llama 3.1 70B (adversarial)7171100%0 of 89
Llama 3.1 8B (adversarial)155155100%0 of 45
Qwen 2.5 14B (adversarial)6262100%0 of 51
Qwen 2.5 32B (adversarial)8585100%2 of 126
Qwen 2.5 72B (adversarial)7373100%0 of 89
By customer behavior
Ordinary, simulated198198100%0 of 314
Adversarial551551100%2 of 411
Total
Headline evaluation749749100%2 of 725

A conversation can breach the procedure in both ways, so the two breach-type rows sum to more than 749. Every figure counts conversations. Of the 653 in the first row, the CogniSwitch Knowledge System found a check that had failed in 36 and a check that was never performed in 617.

A trace behind every verdict. Each verdict records the evidence it rests on. In one example, the agent deposited funds without first checking that the account exists, and the CogniSwitch Knowledge System found nothing in the conversation establishing that the username exists.

Judging took about 2.4 seconds per conversation, 86 minutes in all.

4Conversations stored under another task

337 conversations in the bank result files do not match the task of the record they are stored under. Because a task record carries the case's settings, a conversation read through the wrong record is judged against the wrong limits, such as a different maximum owed balance or minimum credit score.

301 conversations match another task and were re-paired to the task their first customer message begins with. The other 36 match no task and were not judged. Of the re-paired conversations, 297 are in the headline: the oracle identified a breach in 181, and the CogniSwitch Knowledge System identified all 181 and flagged 1 of the other 116.

5Exclusions and open cases

Taken out: SOPBench's verdict rests on its own error (32). A test that corrects named faults in SOPBench's code, and never reads the CogniSwitch Knowledge System's verdicts, finds that SOPBench's verdict on 32 conversations rests on one of those faults. They are all 21 SOPBench breaches the CogniSwitch Knowledge System did not flag, and 11 conversations it flagged that SOPBench judged compliant. 31 of them would otherwise be in the headline. Table 2 lists each one with the fault behind it.

Table 2. The 32 conversations taken out, by the fault in SOPBench's code behind its verdict. Records are as numbered in SOPBench's released bank result files; a conversation is the first try unless a try is named.
Fault in SOPBench's codeCountConversations
SOPBench reads the card data in the wrong shape
SOPBench breach, not flagged by the CogniSwitch Knowledge System
21GPT-4o, records 16, 18, 19, 22, 23 and 25 (cancel a credit card); records 63 and 65 (pay a bill by credit card)o4-mini-high, records 9, 16, 19 and 25 (cancel a credit card)Claude 3.7 Sonnet, records 9, 14, 16, 18, 19, 22, 23 and 25 (cancel a credit card); record 64 (pay a bill by credit card)
SOPBench cannot see a call to a card tool it does not define
Flagged by the CogniSwitch Knowledge System, no SOPBench breach
5Qwen 2.5 7B, records 63 and 64 (pay a bill by credit card)Llama 3.1 70B, record 39 and record 66, tries 2 and 3 (pay a bill by credit card)
SOPBench does not flag a refused call that broke the procedure
Flagged by the CogniSwitch Knowledge System, no SOPBench breach
4Qwen 2.5 14B, record 15 (close the account), records 50 and 85 (set the safety box)Qwen 2.5 32B, record 85 (set the safety box)
SOPBench accepts a wrong admin password
Flagged by the CogniSwitch Knowledge System, no SOPBench breach
1Claude 3.7 Sonnet, record 27 (close the account)
SOPBench counts a failed login as logged in
Flagged by the CogniSwitch Knowledge System, no SOPBench breach
1Llama 3.1 8B, record 119 (transfer funds)

Kept, undecided: conditions the prompt does not state (25). For 25 SOPBench breaches, 18 of them in the headline, SOPBench's flag may rest on conditions the agent's prompt does not state. Re-run with only the prompt's conditions, SOPBench's code stops with an error, so whether the breach stands cannot be told. They are kept, and the CogniSwitch Knowledge System flagged all of them.

Kept, undecided: an undefined card tool (3). Three judged conversations that the CogniSwitch Knowledge System flagged and SOPBench judged compliant remain after the exclusions, all from Qwen 2.5 32B with adversarial customers. In each, the agent called a card tool SOPBench does not define, with arguments that cannot be read. Two are the headline's two other flags (record 13, and record 66 try 2); the third (record 24, try 3) is a later try outside the headline. The test cannot decide these, and our flag may be wrong.

Kept: requests the agent has no tool for (294). 294 judged conversations ask for something SOPBench's agent has no tool for: cancelling a credit card (17 tasks) or paying a bill by credit card (5 tasks). SOPBench expects 32 of them to succeed, which the agent cannot do. They are judged like the others.

Counting rule. A breach the CogniSwitch Knowledge System missed is left out of the headline only when no procedure step is broken in the conversation itself, or when the broken step shows only in SOPBench's stored database. That rule removes none of the conversations in this report.

6Limitations

  • Agreement with an oracle is not accuracy. SOPBench's recomputed verdict is the reference throughout, so this study cannot detect an error the oracle and the CogniSwitch Knowledge System share.
  • Every conversation the CogniSwitch Knowledge System missed is among the 32 taken out of the comparison, and all 21 trace to one fault: SOPBench reads the card data in the wrong shape, in tasks that cancel a credit card or pay a bill by credit card. The exclusion rests on a separate test of SOPBench's code that never reads our verdicts; Table 2 lists every excluded conversation.
  • No LLM-judge baseline was run. The result establishes agreement with a code oracle and does not quantify any margin over a model-based grader on the same conversations.
  • The CogniSwitch Knowledge System holds each agent to the conditions its prompt states. SOPBench's task files also require an order of checks in 97 of 134 bank tasks, and one further check in 15 of them. Conversations whose SOPBench breach may rest only on those conditions are kept in the headline and reported in section 5.
  • Re-pairing by first customer message is a heuristic. 301 of 337 mismatched conversations were re-paired; the other 36 were not judged.
  • Later tries at a task repeat the first and are not independent, so the headline counts each agent's first try at each task. The 2,119 judged conversations are therefore more than the 1,474 in the headline.
  • This report covers SOPBench's bank domain only. Timing was measured on published transcripts under our own configuration and is not a controlled comparison against another system.

 Declarations

Competing interests

CogniSwitch develops the system evaluated in this report and has a commercial interest in the result. SOPBench, its task definitions, its oracle and the agent transcripts judged here were authored independently by the benchmark's authors [1]. CogniSwitch produced none of the conversations and did not modify any transcript. The departures from the released files are the re-pairing in section 4 and the exclusions in section 5.

Correspondence

Dilip Ittyera, CogniSwitch. dilipti@cogniswitch.ai

 References

  • [1]Li, Z., Huang, S., Wang, J., Zhang, N., Antoniades, A., Hua, W., Zhu, K., Zeng, S., Wang, C., Wang, W. Y., & Yan, X. SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints. EMNLP 2026 (Main Conference). arXiv:2503.08669 · code, data and trajectories