Verifying agent procedure compliance on SOPBench

1Background

Most agent benchmarks score an outcome: the task either completed or it did not. An agent can reach a correct outcome while skipping an identity check, ignoring an appeal window, or acting on a policy it never retrieved. In regulated work the procedure is the requirement, so an outcome-only score cannot distinguish a compliant run from a lucky one.

SOPBench converts a written procedure into a directed graph of executable functions. Helper functions retrieve the data a precondition depends on; service functions perform the action. An agent is permitted to call a service function only once prior helper calls have established what the procedure requires. The benchmark's own execution code replays each transcript and returns a verdict, which makes grading deterministic and removes the need for a human annotator or an LLM judge.

The published benchmark provides 167 tools across seven customer-service domains and over 900 verified test cases, together with more than 24,000 agent trajectories [1].

Naming. Two benchmarks share this name. This report concerns SOPBench [1], from UC Santa Barbara and Google DeepMind, which derives code-oracle verifiers from SOP source and reports constraint and ordering verdicts. Amazon publishes a separate dataset named SOP-Bench [2] spanning twelve industrial domains. Results here do not transfer to it.

2Data and setup

We used SOPBench's released result files without modification to the transcripts. Each conversation was judged from the transcript alone. The procedure settings for each case, meaning its numeric limits and which checks apply, were read from that case's task file and were not inferred from the conversation.

The reference verdict for every conversation is SOPBench's own, recomputed by running SOPBench's code against the conversation's own task. We did not use the verdicts as stored in the result files, for the reason given in section 4.

3Results

The verifier agreed with the oracle on every breach, in every stratum. Agreement held equally for the two breach types the oracle distinguishes, across agents of widely differing capability, and under adversarial customers attempting to induce procedure violations.

Determinism was measured separately. Re-running the full set returned an identical verdict on each of the 1,138 individual actions judged. Throughput was 733 conversations in 35 minutes, a median of 2.3 seconds per conversation.

4A pairing error in the released files

Two of SOPBench's result files store 144 conversations against another task's record. Because a task record carries the case's settings, a conversation read through the wrong record is evaluated against the wrong numeric limits, for example a different income threshold or enrolment period.

We re-paired each conversation to its own task by matching its first customer message, which is unique to a case in most instances. 129 conversations matched a single task and were re-paired. Evaluated against their own settings, those 129 contained 53 breaches, of which 5 were not detectable under the settings they had been stored with.

This is a property of the released artefacts, not of the benchmark's design. We report it because it affects any re-analysis of the same files, and because our headline figures depend on the re-pairing.

5Structure of a verdict

Each verdict records the constraint evaluated, the values read, and the turn at which each value entered the conversation. Two examples follow, both resolving to the same outcome by different routes: in the first the required condition was never established, and in the second it was established and failed.

call 5  appeal_claim                    → Not permitted
  constraint   within_appeal_period     (never established)
  claim_date   not established before this call
  appeal_period          = 30 days
  clock.interaction_time = 2025-01-15T12:56:39Z, turn 8

constraint within_appeal_period         → failed
  claim.claim_date       = 2024-12-01, turn 6,  get_claim_details
  clock.interaction_time = 2025-01-15, turn 10
  elapsed                = 45.5 days
  appeal_period          = 30 days      → outside the window

6Method

Counts reconcile as follows. 733 conversations were judged. SOPBench's recomputed verdicts identify 246 breaches among them. 4 are held out of the headline figure: SOPBench's replay flags them because a claim lookup returned in a different shape from its own system's, while no action was refused and no state changed. The headline therefore reports 242 breaches and 487 compliant conversations.

15 conversations are not judged at all: the 11 that match several tasks giving different verdicts, and the 4 that match no task. Each is identified by file and index in the released data. Leaving them out is a reported exclusion, not a filter applied after seeing results.

7Limitations

• Agreement with an oracle is not accuracy. The oracle defines ground truth throughout, so this study cannot detect an error the oracle and the verifier share.
• No LLM-judge baseline was run. The result establishes agreement with a code oracle and does not quantify any margin over a model-based grader on the same conversations.
• One domain of seven. SOPBench spans seven customer-service domains [1]; the released result files judged here cover a single insurance and policy-service domain across ten goals. Generalisation to the other six is untested.
• Re-pairing by first customer message is a heuristic. It resolved 129 of 144 cases unambiguously, and the remaining 15 were excluded rather than assigned a best guess.
• The verifier reads each case's settings from the task file. It is not tested here on recovering those settings from a procedure document alone.
• Timing was measured on published transcripts under our own configuration and is not a controlled comparison against another system.

8The judged conversations

All 733 judged conversations are published in full, each with SOPBench's recomputed verdict, ours, and the constraint trace behind every call that was refused. Nothing is summarised away: the four conversations where the two verdicts differ are reachable by filter.

Opening messages are truncated as published. "Database" records whether the resulting data state matched SOPBench's own replay. A conversation with no refused call shows the calls the agent made and the permission decision for each.

 Declarations

 References

• [1]Li, Z., Huang, S., Wang, J., Zhang, N., Antoniades, A., Hua, W., Zhu, K., Zeng, S., Wang, C., Wang, W. Y., & Yan, X. SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints. EMNLP 2026 (Main Conference). arXiv:2503.08669 · ACL Anthology · code, data and trajectoriesThe benchmark evaluated in this report. 167 tools across seven customer-service domains, over 900 verified test cases, and more than 24,000 released agent trajectories.
• [2]Nandi, S., Datta, A., Nama, R., Patel, U., Vichare, N., Bhattacharya, I., et al. SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agents. arXiv:2506.08119 · doi:10.1145/3770855.3817538 · codeA different benchmark with a near-identical name, from Amazon: 2,000+ tasks across twelve industrial domains. Not evaluated here, listed to prevent confusion.