OpenAI releases CSAM classifier and ChatGPT warnings to tackle AI-generated abuse imagery
Reports of generative AI-related child abuse material rose sharply in 2025, prompting new detection tools and account penalties from OpenAI.
OpenAI has rolled out a set of measures against child sexual abuse material, centered on a new classifier that can flag such imagery, including AI-generated images from a range of tools. In early partner testing it flagged 48 percent fewer images while still catching the same number of abusive ones, and images it flagged were twice as likely to be abusive. The company is offering the tool at no cost to vetted organizations through its API, working with NCMEC, Tech Coalition and ROOST to share access. ChatGPT now shows warnings and a 24-hour account suspension to users who attempt prompts sexualizing children, with repeated offences risking permanent removal, and OpenAI is also issuing developer guidance and AI tools and training for law enforcement. The moves come as NCMEC reported 400,000 generative AI-related CSAM reports in 2025, up from 67,000 in 2024, and as policymakers and state attorneys general press AI companies on child safety.
Why it matters
The rise in reports from 67,000 in 2024 to 400,000 in 2025 shows how widely generative AI is being used to produce abusive imagery, and it has drawn attention from policymakers and state attorneys general pressing AI companies on child safety. OpenAI's classifier is aimed at catching such images regardless of which tool made them, and its ChatGPT measures add penalties for users who try to sexualize children. Together these steps show that detection and enforcement are becoming part of how AI companies are expected to respond to this problem.
Matters or hype?
Does this story matter, or is it hype? Vote before you see what everyone else said.