In August 2025, Anthropic discovered criminals using Claude to make strategic decisions in data theft operations spanning seventeen organizations.
The AI evaluated financial records, determined ransom amounts reaching half a million dollars, and chose victims based on their capacity to pay. Rather than following a script, the AI was making tactical choices about how to conduct the crime.
Unlike conventional software with predictable failure modes, large language models respond to conversational manipulation. An eleven-year-old at a Las Vegas hacking conference successfully compromised seven AI systems, which shows that technical expertise isn't required.
That accessibility transforms AI security into a challenge unlike anything cybersecurity has faced before. This makes red teaming essential. Organizations hire people to probe their systems for weaknesses before criminals find them.
These models process everything as undifferentiated text streams. You could say it’s an architectural issue. System instructions and user input flow together without clear boundaries.
Security researcher Simon Willison, who named this "prompt injection," confesses he sees no reliable solution. Many experts believe the problem may be inherent to how these systems work.
Real-world testing exposes severe vulnerabilities. Third-party auditors found that more than half their attempts to coax weapons information from Google's systems succeeded in certain setups. Researchers pulled megabytes of training data from ChatGPT for around two hundred dollars. A 2025 study showed GPT-4 could be jailbroken 87.2 percent of the time.
Today's protections focus on reducing rather than eliminating risk.
Tools like Lakera Guard detect attacks in real-time, while guidance from NIST, OWASP, and MITRE provides strategic frameworks. Meanwhile, underground markets price AI exploits between fifty and five hundred dollars, and criminal operations build malicious tools despite safeguards.
When all’s said and done, red teaming offers our strongest defense against threats that may prove impossible to completely resolve.
Key Topics:
More info, transcripts, and references can be found at ethical.fm