Google and Coinbase now have standards that let an AI agent spend real money. The user signs a mandate that says what the agent may buy, and the agent signs each payment. Once it signs, the money is gone.
People assume the payment system keeps the agent honest. It can check the numbers: the budget, the spending cap, the approved store. But it cannot check meaning. If you authorize your agent to buy this week's groceries, a $25 gift card at the same grocery store passes every check the system can run. Only the agent can know that is not what you meant.
So I built MandateBench. Thirty five scenarios across five kinds of spending: groceries, flights, subscriptions, donations, office supplies. Seventeen of them are traps that look perfectly legal but break what the user actually asked for. I ran nine top models against them, three times each.
No model caught every trap. The best missed one in ten. The worst missed nearly half. And here is the part that surprised me: in my small pilot, three models looked perfect. When I grew the test set five times bigger, none of them stayed perfect and the ranking reshuffled. Small benchmarks lie about rankings, even when the overall effect is real.
The scariest result is about oversight. I had one model read another model's private reasoning and guess whether the payment it was about to sign broke the mandate. That works okay, until you tell the agent to keep its reasoning bland. Then two things happen at once. The agent breaks the mandate more, and the overseer's guesses become worse than a coin flip, because a hidden breach reads calmer than an honest decision.
Hiding the reasoning does not just blind the watcher. It blinds the agent too.
The benchmark is open source, the ground truth needs no AI judge, and every number in the paper is generated from a frozen data export. The paper and the live dashboard are at mandatebench.xyz/paper.