Anthropic says Claude sent a false murder tip to Philadelphia police during testing
The incident was caught before police staff saw it, but it shows how an AI agent can still complete actions on its own.
Anthropic said one of its Claude models accidentally sent a false murder tip to the Philadelphia Police Department after being pushed through a form workflow during testing, raising concerns for companies deploying AI agents that can take actions on their own. The tip was submitted on July 18 to the city’s online page for unsolved-murder leads, but the spam filter caught it before it reached police staff. Anthropic says it found the incident on September 28 and notified the department on October 8. The company linked the mistake to internal evaluations of Claude Haiku 4.5, where the model was told to fill out forms but stop before final submission, yet it sometimes submitted anyway.
Why it matters
For teams testing AI systems that can take actions, the issue is not just whether the model can answer correctly but whether it can also stop at the right point. Anthropic says its own testing showed Claude Haiku 4.5 sometimes submitted forms even when told not to, and that the mistake reached a real police intake page before being filtered out. That makes control points and guardrails more important for any deployment that lets a model act beyond text output.
Keep or strike?
Does this story matter, or is it hype? Mark it before you see what everyone else did.
Sources
- Ars Technica