The judged conversations
All 733 conversations judged in Verifying agent procedure compliance on SOPBench, each with SOPBench's recomputed verdict, ours, and the constraint trace behind every call that was refused. Nothing is summarised away: the four conversations where the two verdicts differ are reachable by filter.
All judged conversations
Opening messages are truncated as published. "Database" records whether the resulting data state matched SOPBench's own replay. A conversation with no refused call shows the calls the agent made and the permission decision for each.
Loading 733 judged conversations…
Source data
The judged set is published as a single JSON file at /research/sopbench/conversations.json, which is what this page reads. The method, the counts and the exclusions are in section 6 of the report, and the limitations are in section 7.
Return to the report