One AI Chatbot. One Fake Policy. Hundreds of Lost Customers. “The Bot Said What?” Last week, Cursor—a code editor powered by AI—ran into a problem. Its chatbot, “Sam,” confidently told a user: “You can only use one device per subscription. That’s our policy.” The user canceled. Others followed. Reddit lit up. One problem: That policy didn’t exist. The AI made it up. Cursor had a bug. The real problem? It took hours to notice. This is a great example of how not to roll out AI in customer-facing roles. Here are 7 hard lessons every business owner should take away:(Number 4 is most important) 1. AI Needs a Human Safety Net Never let it run fully unsupervised. Build in review, alerts, and escalation. Hallucinations aren’t bugs. They’re features. 2. Always Disclose the Bot If your AI has a name and no label, that’s not clever. That’s deceptive. 3. Audit Logs or Die If you don’t know what it said, you can’t fix it. Cursor didn’t know until customers were already gone. 4. Model Drift Is Real (and Dangerous) AI models change over time. That’s called model drift. They get new updates, new weights, new behavior—sometimes overnight. You didn’t change your script. But your AI did. 5. One Fake Answer Can Cost You Everything This wasn’t a pricing bug. It was a trust nuke. You don’t get many of those. 6. Your Customers Don’t Blame the Bot To them, it’s your company talking. Saying “the AI did it” is like blaming the intern. 7. AI Is a Copilot, Not the Pilot Use it to draft. To sort. To assist. But always keep your hands on the wheel. I’m all in on AI. We use it in our work. I believe in its future. But let’s not pretend it’s fully baked. You wonder how many other customer bots are quietly making up policies right now…