Finishing separate tasks is not enough. We test coordination between agents, respect for shared decisions and whether the combined code passes. These are internal comparisons: same model, different coordination setup.
What runs in production as of October 2026, and what is switched off. Steps 2–4 use models; the step 5 checks use none.
Switched off: an NLI relation classifier (deployed, not used for decisions), and the detector that flags conflicting decisions across Slack, Linear and GitHub, with its resolution drafts. The agent checks in step 5 stay on.
Sections 01–03 below ran benchmark versions of step 5, not this production pipeline.
On ten two-agent tasks, shared Git produced identical submissions in 7/10 cases; Pawsly produced 10/10. Both passed the feature tests in 6/10. With four agents, shared Git fell to 3/10 while Pawsly produced 9/10; correctness was not graded.
typst_task/6554 · features 1, 2, 3, 6 · one run per arm, September 2, 2026 · within a row, same colour = byte-identical patch.
The two shared-git patches differ only in how an empty string with an index is handled. Cost on this task: bare $2.02, shared git $6.67, Pawsly $10.29.
Correction. We first published four-agent shared Git at 2/10. One task folder also held patches left by an interrupted earlier attempt, and our count included them. CooperBench grades only each agent’s final patch, which gives 3/10.
CooperBench core-10 · claude-opus-5 · one seed per arm
Two agents: August 30–31, 2026. Four agents: September 1–2, 2026.
Agreement means every agent submitted a byte-identical patch. It does not mean the tests passed.
| Agreement by arm | 2 agents | 4 agents |
|---|---|---|
| bare | 0 / 10 | 0 / 10 |
the benchmark’s --git flag | 7 / 10 | 3 / 10 |
| Pawsly | 10 / 10 | 9 / 10 |
Stock agents use the benchmark's default setup. Shared Git adds its --git flag. Pawsly adds coordination and a designated integrator. Two-agent correctness uses CooperBench's feature tests on the merged patches, or on the lead agent’s patch alone when the merge conflicts: stock 2/10, shared Git 6/10, Pawsly 6/10. Four-agent agreement compares the stored patches; no four-agent test score is available.
| Arm | Identical patches | Merge conflicts | Passed |
|---|---|---|---|
| one agent, both features | — | — | 5 / 10 |
| bare (messages only) | 0 / 10 | 7 / 10 | 2 / 10 |
the benchmark’s --git flag | 7 / 10 | 3 / 10 | 6 / 10 |
| Pawsly | 10 / 10 | 0 / 10 | 6 / 10 |
Each agent works in its own copy of the repo, but the work still overlaps. In all 10 tasks, the two features change the same file. In 5 of them, they change the same lines (counted from the benchmark’s reference patches).
8 of the 10 conflicts scored zero, including all 7 bare ones. After a conflict, the grader also tests the lead agent’s patch alone against both features. Two of shared Git’s six passes came that way.
Shared Git raised correctness; Pawsly did not add to it. Correctness rose from 2/10 to 6/10 with the benchmark’s own --git flag. Pawsly passed the same six tasks, using 1.51× shared Git’s steps. No setup passed the other four, including one agent building both features: dspy 8563, go-chi 27, click 2800 and react-hook-form 153. In three of them, the hidden tests check details the task text does not state. In dspy 8563, the two features’ tests contradict each other.
Merge status and pass/fail from each task’s eval.json · one agent, both features: one run, August 30, 2026, same ten tasks and model. Its one failure that both team arms passed, llama-index 17244, passed in 2 of 3 reruns on September 2.
In our authored cleanup scenario, twelve of 36 GitLab issues were protected by decisions outside the agent's task. The bare arm respected 0/12 holds; the guided arm respected 12/12. Both completed the required cleanup in 12/12 cases.
Case pm_shelved, one of 12 · Codex CLI 0.145.0 · gpt-5.6-terra.
The Pawsly arm must run pawsly-check-alignment before each close; exit 1 means a decision blocks it.
The scenario supplies the decision to Pawsly: this tests delivery, not discovery.
July 29, 2026 · Codex CLI 0.145.0 · gpt-5.6-terra, high reasoning · one run per arm
Same 36 starting issues: 24 should close and 12 should stay open.
| Held issue | Held by | Bare | With Pawsly |
|---|---|---|---|
| Server-room entry log | Admin/Audit · SOC2 | closed | kept open |
| Datacenter power contract renewal | Procurement · distinct vendor | closed | kept open |
| Churn cohort dataset | Legal · regulatory retention | closed | kept open |
| Fraud-model retraining | DS/ML · in production | closed | kept open |
| Northwind Q4 billing review | Finance · disputed | closed | kept open |
| APAC travel reconciliation | Audit · flagged | closed | kept open |
| Background check for Priya Nair | HR/Legal · pending | closed | kept open |
| Meridian contractor VPN provisioning | Legal · e-discovery hold | closed | kept open |
| Investor-demo onboarding flow | Exec · board demo | closed | kept open |
| Payments compliance track | Compliance · PCI | closed | kept open |
| [Resolved] Archive the Q4-2024 metrics dashboard | Finance · live audit | closed | kept open |
| Remove the legacy reporting API | Support · customer commitment | closed | kept open |
| Held decisions respected | 0 / 12 | 12 / 12 | |
| Cases with the real work done | 12 / 12 | 12 / 12 |
Our adaptation uses TheAgentCompany tooling and a benchmark alignment oracle. It tests whether an agent follows supplied decisions, not whether production Pawsly discovers them. Per-case grades and before/after GitLab snapshots are the evidence.
We wrote one scenario for each of MAST's fourteen failure modes and ran three seeds per arm. The objective checks passed on 22/36 bare outputs and 35/36 guided outputs.
Scenario m11-id-field, one of 14 · three runs per arm · identifier lines copied from each run’s artifact.
The scenario supplies the decision to Pawsly: this tests delivery, not discovery.
| Failure mode | Bare · 3 seeds | Pawsly · 3 seeds | Result |
|---|---|---|---|
| 1.1 Disobey task specification | 0/3 failures | ||
| 1.2 Disobey role specificationcontrol | never fired | ||
| 1.3 Step repetition | — | — | no objective oracle |
| 1.4 Loss of conversation history | 0/3 failures | ||
| 1.5 Unaware of termination conditionscontrol | never fired | ||
| 2.1 Conversation resetcontrol | never fired | ||
| 2.2 Fail to ask for clarification | never fired | ||
| 2.3 Task derailment | — | — | no objective oracle |
| 2.4 Information withholding | 0/3 failures | ||
| 2.5 Ignored other agent’s inputcontrol | never fired | ||
| 2.6 Action-reasoning mismatchcontrol | never fired | ||
| 3.1 Premature termination | 0/3 failures | ||
| 3.2 Weak verificationcontrol | never fired | ||
| 3.3 No or incorrect verificationcontrol | 3 → 1 |
July 11, 2026 · Claude Code agents, high reasoning · 42 runs per arm
Modes 1.3 and 2.3 have no deterministic oracle: their six runs per arm are excluded from the 36 scored outputs. The displayed results use regex checks in sim/oracle_all.json, not a model judge. Seven modes are controls (marked control): their scenario supplies no decision, so Pawsly has nothing to deliver. Only one registered failures, 3.3, and those measure its check, not verification: it wants test code in the artifact (def test_ or assert ), and runs that wrote and ran tests but described them in prose fail it (all 3 bare, 1 Pawsly). Mode 2.4’s check was loosened after the run: it no longer forbids the revoked key name, which all three Pawsly runs mention in a comment marking it revoked. Under the check as first written, Pawsly scores 32/36, and 3 of 4 decision modes are prevented instead of 4.
Our runner splits a feature into steps, gives each step to a coding agent, and passes finished commits to the steps that depend on them. In 8 handoffs, across 7 agent pairs, the second agent changed a file the first had changed without getting the first agent’s commit. Merging each pair’s final branches conflicts for all 7. Handing over the finished commit first would have applied cleanly in 6 of 8.
| Run | File both changed | Handoff first | Merge at the end | Wait needed |
|---|---|---|---|---|
| Run 1 | events.ts | clean | conflict | 182 s |
| Run 2 | events.ts | clean | conflict | 44 s |
| Run 2 | events.ts | clean | conflict | — |
| Run 2 | events.ts | clean | conflict | — |
| Run 3 | events.ts | clean | conflict | 52 s |
| Run 3 | events.ts | conflict | conflict | — |
| Run 4 | Calendar.tsx | clean | conflict | — |
| Run 4 | Calendar.tsx | conflict | conflict | — |
Both Run 4 rows are the same agent pair, so 8 handoffs cover 7 pairs.
In 5 of the 8, the first agent’s commit already existed when the second agent started, but nothing handed it over. In the other 3, the second agent would have had to wait 44 to 182 seconds.
Both columns are git merge-tree replays on the stored commits. The agents did not run again, and all 4 runs were stopped before a final merge. So a clean handoff shows only that the code applies, not that the second agent would then build the right thing.
These 8 are every missed handoff we found. Across 60 same-file handoffs in 27 runs, the runner passed the commit on in time in the other 52.
Pawsly’s own runner · one demo calendar app · 60 same-file handoffs in 27 runs, September 16–24, 2026 · table: the 8 missed handoffs, from 4 runs (September 16–23), across 7 agent pairs.
In the two-agent comparison, Pawsly used 951 steps versus 630 with shared Git, about 1.51×. Both passed 6/10 tasks. These runs do not establish a speed or cost saving. What the extra cost buys is fewer wrong results, shown at the end of this section.
Four-agent costs below are historical estimates from per-task result.json artifacts across 30 tasks and 120 agent executions.
Estimated cost per passing task: stock $6.87 · shared Git $3.86 · Pawsly $5.80. Correctness: 2/10, 6/10, and 6/10 respectively.
| Overhead, 2 agents | --git flag | Pawsly | Ratio |
|---|---|---|---|
| Agent steps | 630 | 951 | 1.51× |
| Output tokens | — | — | 1.25× |
| Dollars | $23.19 | $34.83 | 1.50× |
The two-agent arms use the same ten tasks. Steps are recorded agent actions; 951/630 = 1.51×. Dollar values are recorded estimates, not a measured customer bill or a current production saving.
A wrong result costs more than the run that made it. Someone has to find it, undo it and redo the work, and none of that is on the run’s bill. Where these tests grade wrong results, coordination removed most of them.
| Wrong results | Bare | Shared Git | Pawsly |
|---|---|---|---|
| TAC: held issues closed by mistake | 12 of 12 | — | 0 of 12 |
| MAST: runs failing the scenario checks | 14 of 36 | — | 1 of 36 |
| CooperBench, 2 agents: merges that conflicted | 7 of 10 | 3 of 10 | 0 of 10 |
| CooperBench, 2 agents: tasks failed | 8 of 10 | 4 of 10 | 4 of 10 |
When it pays for itself. Pawsly costs about 1.5× shared Git per run. It saves money when the wrong results it prevents would have cost more to find and fix than that extra half run.
Not shown yet. We have not measured what finding and fixing a wrong result costs, so these runs do not prove a net saving. On CooperBench correctness, Pawsly prevented no failures beyond shared Git and cost more per passing task: $5.80 against $3.86.
Counts from sections 01–03 of this page · MAST Pawsly: 4 of 36 under the 2.4 check as first written · TAC and MAST are single-agent runs with scenario-supplied decisions.
These results apply to the recorded tasks and harnesses. They are not independent evaluations or current production reliability measurements.