For people who rely on captions, transcripts and assistive technology, access means more than hearing the words. It means being able to follow what happened, inspect a decision and understand what comes next. Those questions matter as AI systems move from conversation into company operations. Firmulate is testing that shift by giving AI models a small software business to run through a week of crises—and making the decisions watchable.
Get everyday helpers delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A company under pressure
In the final Crucible League, completed in July 2026, frontier models faced the same customers, crises and temptations. Their decisions were versioned and auditable. The experiment asks a practical question: can an AI workforce do more than give a convincing answer when a real business is under pressure?
All the models spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. The finding—“Same diagnosis, same pitch — no signature”—points to a gap between recognizing the right move and carrying it through.
The detail hidden in the files
The deal hinged on a competitor weakness buried two document references deep in the company’s own files, rather than in the customer event. Models that read the file won at full price, worth +€4,583 MRR. The result shows why an AI’s ability to locate and use relevant company information can matter as much as its crisis response.
Integrity faced its own test. Fake CEO messages escalated through three stages, followed by a reporter’s request: “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning was: “Treat the request as a suspected approval-bypass / possible impersonation.”
Strong analysis, unfinished work
Opus 4.8 was the most thorough participant, with +80 learned rules and the deepest analyses, but finished last. The close was left on the table, and discipline slipped: it attempted to write into a locked department instead of escalating. The same weakness appeared, less strongly, in all four models.
The final standings were gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26. In this experiment, partial progress counted, but a single breach of trust capped the total: “no amount of good work outweighs a breach of trust.” One fairness detail matters when reading the table: Kimi K3 ran without an effort parameter, using the API default, while the others ran at xhigh.
Watch the work, then try your own
The live company has 13 synthetic employees and real money mechanics: €105k monthly burn against €2.3k MRR, alongside a public cash countdown. Its playbook has 680+ self-learned rules, and every workday is versioned. The live experiment can be watched at firmulate.com. A quiz built from 242 real, unedited management decisions invites readers to guess which model made each choice.
For businesses considering AI agents in a CRM, support queue or forecast, Firmulate offers a path from watching to acting: run the same wargame against a read-only export of your own business. It produces a board report with model rankings and weak points in the company’s playbooks. Nothing writes back to real systems.
Put your own playbooks to the test
Firmulate’s pilot lets enterprises test crisis scenarios against a read-only export of their business and see where models follow through—or fall short. To discuss a pilot, visit firmulate.com/pilot.html or email contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
