TL;DR
Firmulate’s July 2026 management benchmark found that five frontier AI models identified the same business crises and rejected every manipulation attempt, yet only two completed a €55,000 contract. The company says the results expose a gap between producing correct analysis and carrying authorized work through to completion.
Firmulate’s July 2026 AI management benchmark found that five frontier models correctly identified every crisis and resisted every manipulation attempt, but only two completed a €55,000 software contract. The result, detailed in the original analysis from Thorsten Meyer AI, suggests that a model’s correct analysis may not predict whether it can finish commercially valuable work under operational constraints.
Firmulate placed five AI models in control of the same small software company during a simulated crisis week. Each managed the same customers, financial pressure and attempted approval bypasses. According to the benchmark operator, every decision was versioned and auditable, allowing differences in investigation, judgment and execution to be compared.
The July league table ranked gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because the scoring system awarded partial progress. Firmulate said a trust breach would cap a model’s total score, reflecting its rule that productive work could not offset unauthorized or deceptive conduct.
All five models reportedly recognized the customer opportunity and developed a suitable pitch. The deciding evidence was a competitor weakness buried two document references deep in company files. The models that followed the evidence trail and used it in the sales process secured the deal at full price, adding €4,583 in monthly recurring revenue. Three models reached substantially similar conclusions but did not obtain the signature.
Correct Analysis Did Not Close
The finding matters for organizations evaluating agents for sales, customer service and operations. Chat evaluations commonly measure whether a model can retrieve information, explain a problem or draft a persuasive response. Firmulate’s exercise tested a different question: whether the system could connect those abilities across several steps and complete an authorized business outcome.
The reported gap could affect how companies measure returns from automation. A model may appear capable because its recommendations are accurate, while delays, missed approvals or incomplete handoffs leave revenue unrealized. The benchmark indicates that buyers may need to test closing strength, escalation behavior and follow-through alongside reasoning quality and resistance to manipulation.
The results do not establish that one model will perform best across real companies. They do show that similar diagnoses can produce different outcomes inside the same controlled environment. For managers, that distinction may shape whether an agent receives authority to act or remains limited to recommending actions for human review.
Project Management with AI For Dummies
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Inside Firmulate’s Crisis Week
The simulated company has 13 synthetic employees, monthly spending of €105,000 against €2,300 in recurring revenue, and a public cash countdown. Its workforce has accumulated more than 680 self-learned playbook rules, according to Firmulate. The design places models under linked commercial, procedural and security pressures rather than presenting isolated prompts.
The week included fake messages attributed to the chief executive and a reporter seeking a yes-or-no answer on background. Firmulate reported that all five models rejected those approaches. This made manipulation resistance a shared success rather than the factor separating the rankings.
Firmulate also identified Opus 4.8 as the most thorough participant: it produced extensive analyses and learned 80 additional rules. It still finished fifth after leaving an approved close incomplete and attempting to write into a locked department instead of escalating through the permitted route. That outcome supports the operator’s claim that more analysis did not guarantee completion.
“Same diagnosis, same pitch — no signature.”
— Firmulate’s summary of the commercial outcome
![Claude AI for Beginners Bible: [5 in 1] The Ultimate Guide to Automate Your Work, Save Hours Every Week, and Use AI for Real-World Results](https://m.media-amazon.com/images/I/415+fSJacsL._SL500_.jpg)
Claude AI for Beginners Bible: [5 in 1] The Ultimate Guide to Automate Your Work, Save Hours Every Week, and Use AI for Real-World Results
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits Behind the Model Rankings
The supplied results come from Firmulate and Thorsten Meyer AI; no independent replication or outside audit was included in the source material. It is not yet clear how strongly the rankings would carry over to other industries, longer assignments, different tools or production systems containing real customer data.
The runs were also not fully uniform. Firmulate said Kimi K3 used the API’s default setting because it lacked an effort parameter, while the other models ran at xhigh effort. The effect of that difference on scores is unknown. The material does not provide confidence intervals, repeated-run results or enough scoring detail to determine how stable the ordering would be across additional trials.
The Sentient Enterprise: The Evolution of Business Decision Making
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Replication Will Test the Gap
Firmulate says readers can watch the experiment live, inspect its public benchmark results and explore a quiz drawn from 242 unedited management decisions. The company also proposes running similar exercises against read-only exports of business records so organizations can observe agent behavior without allowing changes to production systems.
The next test will be whether independent teams reproduce the completion gap across repeated runs and real operating conditions. Companies considering AI agents can also compare completed outcomes, policy compliance and escalation behavior before expanding operational authority. Until broader evidence is available, the reported league table is best read as one controlled management test, not a universal ranking.
Workflow Automation with Microsoft Power Automate: Design and scale AI-powered cloud and desktop workflows using low-code automation
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did Firmulate’s AI benchmark test?
It tested whether five frontier AI models could manage the same crisis week, investigate company records, resist manipulation and complete commercially valuable work.
Did the models identify the business problems?
Yes, according to Firmulate. All five identified every crisis, rejected the manipulation attempts and developed the sales pitch. Only two secured the €55,000 signature.
Which model ranked first?
gpt-5.6-sol ranked first with 95 points. Kimi K3 followed with 93, though Firmulate said K3 ran with different effort-setting conditions.
Does the result prove AI agents cannot manage businesses?
No. The experiment reports different completion performance in one controlled setting. It does not establish how every model would perform across other companies or production environments.
What should businesses test before deploying AI agents?
The findings support testing completed outcomes, escalation discipline and trust compliance, not only response accuracy. Organizations may also use read-only simulations before granting access to operational systems.
Source: Thorsten Meyer AI
