Trusence Every claim has a source
Last updated 17 November 2026 Search Türkçe
Security Developing

OpenAI releases CSAM classifier and ChatGPT warnings to tackle AI-generated abuse imagery

Reports of generative AI-related child abuse material rose sharply in 2025, prompting new detection tools and account penalties from OpenAI.

Updated with 1 new report since it was first published · latest 16 November at 23:07

OpenAI has rolled out a set of measures against child sexual abuse material, centered on a new classifier that can flag such imagery, including AI-generated images from a range of tools. In early partner testing it flagged 48 percent fewer images while still catching the same number of abusive ones, and images it flagged were twice as likely to be abusive. The company is offering the tool at no cost to vetted organizations through its API, working with NCMEC, Tech Coalition and ROOST to share access. ChatGPT now shows warnings and a 24-hour account suspension to users who attempt prompts sexualizing children, with repeated offences risking permanent removal, and OpenAI is also issuing developer guidance and AI tools and training for law enforcement. The moves come as NCMEC reported 400,000 generative AI-related CSAM reports in 2025, up from 67,000 in 2024, and as policymakers and state attorneys general press AI companies on child safety.

Why it matters

The rise in reports from 67,000 in 2024 to 400,000 in 2025 shows how widely generative AI is being used to produce abusive imagery, and it has drawn attention from policymakers and state attorneys general pressing AI companies on child safety. OpenAI's classifier is aimed at catching such images regardless of which tool made them, and its ChatGPT measures add penalties for users who try to sexualize children. Together these steps show that detection and enforcement are becoming part of how AI companies are expected to respond to this problem.

Sources · 2