A $27M plan to clean up the internet
Matt V
February 12, 2025
AI SAFETY
Google, OpenAI, Roblox, and Discord have launched ROOST (Robust Open Online Safety Tools), a non-profit focused on child safety online.
It aims to make AI-driven safety tools free, open-source, and widely accessible, helping companies detect and report child sexual abuse material (CSAM) more effectively.
The initiative was partly driven by the rise of generative AI, which has changed online risks.
Former Google CEO Eric Schmidt, a ROOST founding partner, says the project is meant to speed up innovation in child safety.
While details are limited, the tools will use large language AI models to improve CSAM detection and streamline reporting.
ROOST comes at a time when lawmakers are pushing for stricter child safety rules on social media.
The National Center for Missing and Exploited Children (NCMEC) reported a 12% increase in child exploitation cases from 2022 to 2023.
Roblox and Discord—both ROOST members—have faced criticism for failing to protect young users, with lawsuits highlighting unsupervised adult-child interactions on their platforms.
The initiative is backed by $27 million in funding from philanthropic groups and will bring together AI experts, child safety advocates, and open-source developers.
Key features include:
AI-powered moderation tools, which companies can integrate via APIs.
A "community of practice", offering vetted AI training datasets and identifying safety gaps.
A unified approach, combining existing detection and reporting tools from ROOST members.
Big tech’s safety era
ROOST’s approach builds on past efforts, like Discord’s Lantern project with Meta and Google, and Roblox’s AI model for detecting harmful content.
However, it’s still unclear how ROOST’s tools will work with existing CSAM detection systems, like Microsoft’s PhotoDNA.
Alongside its ROOST involvement, Discord has launched an “Ignore” feature, allowing users to mute messages without notifying the sender.
“Safety is a common good,” said Discord’s Chief Legal Officer Clint Smith, emphasising the goal of making the entire internet safer for young users.
I think we can all agree this is a good thing.