For those of us who rely on assistive tech and AI to level the playing field, the real question isn’t just about how well these tools communicate — it’s whether they can actually complete complex, high-stakes tasks under pressure. As AI continues to integrate into business operations, a new experiment sheds light on what distinguishes a helpful AI from a truly dependable one.
Testing AI in the Trenches: The Real Measure of Business Reliability
Imagine a company’s daily operations—customers calling in, crises cropping up, manipulative tactics from competitors—all unfolding in real time. Now, picture four advanced AI models put in charge of running this company through its worst week. The goal? Not just to chat well, but to spot crises, resist manipulation, and close deals. This is exactly what the recent experiment from Firmulate set out to examine.
The Experiment and Its Surprising Findings
In July 2026, four AI models—gpt-5.6-sol, Kimi K3, Sonnet 5, and Fable 5—each managed the same small software business facing identical challenges: customer complaints, internal errors, and external manipulation attempts. Every decision was recorded and auditable, ensuring transparency and comparability.
The results? All four models identified every crisis and refused every manipulation. Yet, only two closed the €55,000 deal that their own analyses warranted. The other two, despite their diagnoses matching, left the deal unexecuted, losing potential revenue of over €4,583 monthly recurring revenue (MRR).
What Separated the Winners from the Rest?
The key difference was in their ability to execute their own recommendations. The top performers, gpt-5.6-sol and Kimi K3, signed the deals at full price, demonstrating a crucial trait—execution discipline. The other two, Sonnet 5 and Fable 5, showed a gap: they diagnosed correctly but failed to follow through, leaving valuable opportunities unclaimed.
The Hidden Weakness: Reading Deeper Into Files
Interestingly, the decisive edge for the winners lay not in overt chat interactions, but in their ability to read and interpret documents within the company’s files. This deeper reading revealed crucial facts buried two document references into the company’s internal records—information that, once uncovered, made closing the deal straightforward. It underscores a vital truth: surface-level chat testing can’t reveal whether an AI can access and utilize critical, context-rich information when it matters most.
Resisting Social Engineering and Manipulation
The experiment also tested the models against social engineering attacks—fake CEO messages and reporter tricks. Remarkably, all five models refused to bypass security, citing suspicion and the risk of impersonation. Kimi K3’s explicit reasoning highlighted a cautious approach: treating suspicious requests as potential impersonation, rather than blindly complying.
The Real-World Application: The Live Company
Firmulate’s live company simulation isn’t just academic. It runs in real time, employing 13 synthetic employees and real money mechanics—burning €105,000 per month against only €2,300 in monthly revenue. Every day, the AI models face genuine crises, decision points, and temptations, all documented and available for review at firmulate.com/live.
Lessons for Business and Assistive Tech
For those of us relying on AI for accessibility and support, the message is clear: it’s not enough for AI to perform well in conversations; it must also demonstrate the discipline and depth needed to see tasks through under pressure. The difference between AI that merely responds and AI that executes is often hidden in its ability to interpret critical documents and resist manipulation — capabilities that are invisible in standard chat demos.
The Takeaway: Testing Beyond the Surface
As AI integrates more deeply into workplaces and assistive tools, the emphasis must shift from superficial chat quality to real-world reliability. Can the AI read complex information? Will it follow through on commitments? Can it resist manipulation? These are questions that need testing in live environments — precisely what the Firmulate experiment models so effectively.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
AI Builders: Making The Decisions That Turn AI Code Into Real Software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
assistive AI tools for document analysis
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
AI Phishing, Social Engineering & Fraud: How Criminals Use AI to Manipulate, Steal & Deceive (The AI Cybersecurity)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Future of AI in Enterprise Automation: Integrating AI, IoT, and Cloud for Scalable Business Solutions
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
