The AI Red Team Skeptic: How I Double-Check My AI Work A little public service announcement for anyone using AI and wondering: Is my AI making mistakes? Yes. Constantly. But often the mistakes are subtle. Sometimes it fabricates sources—citations look real, links to nothing. Sometimes the math is just… wrong. The problem is: it sounds very authoritative while being confidently wrong. That’s why I built what I call my Red Team Skeptic Prompt. (Modeled after how the U.S. military uses Red Teams to attack and stress-test their own defenses.) Here’s how it works: After I run a core AI prompt (let’s say underwriting or due diligence), I run a second AI prompt designed to reverse-engineer the first answer. This Red Team prompt looks for: Logical gaps Math errors Fabricated sources Unsupported assumptions It doesn’t just accept the output — it tries to break it. The best part? Most of the bigger LLM models can actually do this pretty well—if you give them the right instructions. But even with Red Team Skeptic running, I still manually spot-check. AI catches more when you have it audit itself, but it’s not human yet. Example: Just this morning, GPT-4o did a water loss calculation. Two completely different leaks. Two completely different rates. Same final loss numbers. Why? It lazily reused the prior math without recalculating. Without my eye, I would’ve missed it. Always Red Team your work. Always double-check. I guess you have to do that with humans too.