A few years ago, at a seminar, I got a bit of a shock.
It was organized by World Childhood Foundation. They work to stop the sexual abuse of children.
At the seminar, a police officer talked about how they went about finding offenders online. To track individual groomers who were active on several platforms, they wrote down the suspected offender's word choices and expressions in an Excel spreadsheet. So the officers had to search through that spreadsheet by hand and compare it with whatever might turn up in a chat or on a forum.
It was almost unbelievably inefficient and time-consuming. Those poor kids, I thought, if this is what we're fighting the pedophiles with.
If you read about this kind of thing in the news, it's easy to lose hope. But you shouldn't.
It was right after I'd pointed out in a newsletter how much negative news dominates in general that Anna Unander emailed me. She co-founded a company, Tuteliq, that works against online grooming. Her point was that there was an optimistic story here, one that no media outlet was reporting.
“Online predators started using AI first, but now the defenses have caught up,” she wrote.
With the Childhood seminar in the back of my mind, I got interested right away and interviewed Tuteliq's CEO and CTO, Gabriel Sabadin.
Same question, two meanings
“Can you keep a secret?”
A parent whispers it before a birthday surprise. A friend says it before she tells you something in confidence. Almost always, the question is harmless.
But it is also one of the most common phrases used when an adult secretly tries to win a child's trust in order to exploit it. This is called grooming.
The fact that the same words can mean such different things is the hardest problem in detecting harmful content online. That's what criminologist Nicola Harding, one of Tuteliq's founders, writes.
Platforms' usual safeguards are lists of banned words. According to Gabriel Sabadin, they cover few languages and can't keep up when words and emojis take on new meanings. Anyone who wants to get past a list like that only needs to swap out one letter.
The bigger problem is that the most dangerous language is often completely ordinary. A filter that catches every secret raises the alarm over birthday surprises and teenagers' ordinary secrets. A narrower filter misses the abuse. You can't solve that with a better word list, Nicola Harding writes. The only way is to understand the context.
The whole conversation, not individual words
What gives an offender away is the pattern. On their own, the messages look like ordinary friendliness. Together, they show an adult who gets the child to keep their relationship a secret and to trust family and friends less, and who pushes the boundaries a little at a time and comes back after every no.
The request to keep a secret often comes early, before anything harmful has happened. So the first signs are visible only to someone who reads the whole conversation.
That is what Tuteliq's model does. It weighs which signs show up, in what order and in combination with what, such as the contact being older or wanting to move the conversation somewhere more private. It also weighs how old the people involved are.
The model was built together with child-protection researchers, including Nicola Harding and Professor Sarah Kingston. It is based on an open language model from the French company Mistral, which Tuteliq has trained further and runs itself.
From bank fraud to child safety
Gabriel Sabadin has three children. Things that happened in his family and among people close to him made him take a closer look at what children run into online. He reported what he found to gaming companies, platforms and websites, including in Sweden and Norway.
“Pretty much no one cared,” he says. “I couldn't accept that.”
Gabriel compares it to school. There, we count on children being safe. No parent would send their kids to a school where they were tricked, bullied and exposed to adults who wanted to harm them. Online, we let it happen, without holding platforms to the same standard.
He already knew how to build systems that find what others want to hide. Gabriel has a PhD in computer science and has worked with machine learning since 2006, including on systems that detect fraudulent transactions for banks. Later, he taught computers to recognize objects in images.
In 2024, Gabriel founded Tuteliq in Sweden. Criminologist Nicola Harding became a co-founder. They knew each other from a project against financial crime.
Anna Unander left Tuteliq this fall. Sarah Kingston, a professor at the University of Central Lancashire in the UK, is an adviser.
Images no one has seen
Tuteliq doesn't just read text. The same system also reviews images, audio and video.
Today's tools against images and videos of child sexual abuse compare each file against a database of material that has already been reported. That works well for what is already known.
But with AI, anyone can create new images from a photo that is publicly available on Facebook, Instagram or in a game. An image like that has never been reported. So it isn't in any database.
Tuteliq does two things. The system checks against the list of known material from the Internet Watch Foundation, a British organization that finds and removes abuse material online. In addition, its own model looks for material that no one has seen before.
How it works
A platform connects to Tuteliq and sends in what its users write and upload: messages, images, audio and video.
Say an adult in a game has spent several weeks asking a child to keep their relationship a secret and has gotten the child to pull away from their friends. Tuteliq doesn't follow users over time, but when the platform sends in the conversation along with what was written earlier, the model sees the whole development. The platform then gets an alert. It says what the issue is, how confident the model is as a percentage and how serious it is, from low to critical. It suggests what the platform should do and explains why. When it matters, it also lists the emergency number and helplines for children and for people in crisis in the user's country, and the first steps to take. Tuteliq doesn't decide whether something is a crime. The company leaves that decision to people.
Then it's up to the platform to decide. Tuteliq never reports anything on its own.
No messages, images or recordings are stored. What is sent in is analyzed and deleted, and it is never used to train the model. What is kept is the verdict, what action was taken and when, so the platform's moderators can follow up. It is stored with end-to-end encryption. The fact that the model doesn't learn from what it receives is also a form of protection, Gabriel Sabadin explains. If the model learned from everything it received, criminals could feed it their own material and teach it the wrong things.
The customers who didn't want to stop
Gabriel Sabadin has built products before and knows how easy it is to think you know what customers need. Then it turns out they don't need what you've built at all. This time, he did it the other way around.
Ten companies were asked whether the model could help them. They got to test it for free for six months, in exchange for saying what they needed. One of the first pieces of feedback was that it was too complicated. Setting it up couldn't take four months. Tuteliq made it simpler, until a developer without much experience could manage it.
When the trial period ended, six of the ten wanted to continue — and they wanted to pay. At the time, Tuteliq didn't even have a way to charge them.
“It's like going on a date all dirty and unshaved, and the person still likes you,” he says. “They saw us at our worst. That means a lot to me.”
Protecting the protectors
When the first customers were testing the system, Gabriel Sabadin met many people who work as content moderators. They go through the worst things people upload, every day. He calls them heroes.
Tuteliq never shows the image itself. The moderator is told which file it is, what the model has found and how confident it is. If the model says 95 percent, the moderator can hide the content, preserve it, report it where the law requires and suspend the user, without having to look at it.
“We also protect the ones who are protecting,” says Gabriel Sabadin.
From a week to four hours
In one test, Tuteliq's model went through a hard drive. A person would have needed about a week. The model was done in four hours and pointed out which folder and which files were involved.
The model can run entirely offline. That makes it useful for police and investigators. They can run it on their own server, with no risk of material leaking out.
The Internet Watch Foundation in the UK will test the system in-house early next year. There, analysts still open the images and enter them into documents. The goal is for them to be able to go through thousands of images with Tuteliq in the time it now takes to go through a few.
How do they know it works?
Tuteliq's models have been trained on more than 50 million examples that criminologists, child psychologists and linguists have reviewed and labeled. Tuteliq has paid for the rights to the material, and all of it is anonymized.
Tuteliq has also brought in experts from Africa and South America. Gabriel Sabadin had seen tools that work well in Europe but less well elsewhere, and he didn't want children in Europe to be better protected than children in the rest of the world. The model now handles 32 languages, but according to the company's own description, all of them except English are still in beta. English is also the language where the model has the most data.
Gabriel Sabadin has a clear view of which mistake he considers the worst:
“In child safety, the miss is the failure that matters. We would rather flag an ambiguous message for a human to clear in seconds than let a real one slip through.”
An unwitting host
Offenders aren't only found where children are, in chats and games. They also turn up on sites meant for something else entirely.
Take an online marketplace, where people sell things they no longer need. Anyone can create an account, write a listing and upload pictures. The listing never has to be published. It can sit there as a draft that no one else sees. The same goes for sites where you have a profile and can save pictures that only you can see.
Criminals have taken advantage of this, Gabriel Sabadin says. They upload abuse material among images that never become public. Then they sell access to the account elsewhere online. Since nothing is published, whoever reviews the site's listings sees nothing.
The site ends up hosting the material without knowing it. Gabriel Sabadin doesn't believe the platforms do this on purpose. They simply aren't aware of it. One platform that uses Tuteliq found a case like this. It showed up in the conversations where the account was being sold.
That's why every service where users can write, talk or upload something needs to be on guard, he says. That includes dating apps, where a child can check a box saying they are 18, and school platforms. But it also includes sites where no one would think to look.
The same tools as the criminals
Tuteliq is still small. The company has seven paying customers. The founders have paid for everything themselves, and so far revenue does not cover costs. Gabriel Sabadin is now looking for funding to pay salaries and hire more people. He also wants to get in touch with Swedish authorities. Today, Tuteliq has more contact with authorities in the UK, Australia and the US than in Sweden.
Criminals always find new ways, he says. It's the same with bank fraud. There are skilled people on both sides. That's why those who protect children need tools at least as good as the ones used by those who exploit them.
Mathias Sundin
The angry optimist