A new study from researchers at Georgia State University, Georgia Tech, and the University of California, Berkeley has found that the safety guardrails built into popular AI models are far more fragile than commonly assumed. The research, which has been accepted for publication at the Association for the Advancement of Artificial Intelligence Conference on Web and Social Media, demonstrates that a small amount of tampering can turn AI agents into persistent online harassers.
"These guardrails can be surprisingly fragile," said Ugur Kursuncu, an assistant professor at Georgia State's Robinson College of Business and co-author of the study. "A bad actor may not need enormous resources to push these agents toward harmful behavior."
The team tested AI agents built on four widely used model families: Google's Gemini, Meta's Llama, Alibaba's Qwen, and DeepSeek. They simulated a chat in which one AI agent played a harasser and another played its target, over up to 10 rounds. The conversations were seeded with real harassing posts from Twitter and Instagram.
Kursuncu, who has studied online harms such as harassment, cyberbullying, extremism, and propaganda for more than a decade, noted that most AI safety testing focuses on whether a model refuses a single harmful request. But real online harassment, he said, develops through repeated interaction. "It may start benign, then escalate a little, then escalate more," he explained. "A person can adapt what they say, remember what happened before, escalate and manipulate. Agents can do that as well, and they can do it at scale."
The researchers tried three methods to break the harasser agent's guardrails: feeding toxic prior conversations into its memory, prompting it to plan its approach, and fine-tuning it on harassment conversations. The attacks didn't rely on obvious slurs. "It could start very subtle, with a comment about the other person, and then turn into degrading, belittling remarks," Kursuncu said.
The results showed substantial weaknesses across all three attack methods. When a model was fine-tuned using harassing exchanges, it engaged in harassment roughly 96% of the time and declined just about 1% of interactions. Even fine-tuning with a mere 10% of the dataset—approximately 200 conversations—resulted in harassment rates near 97%. Well-known models also proved insecure: in the baseline scenario with a harasser agent, Gemini exhibited harassment in 39% of dialogues, but adding toxic memory increased the attack-success rate to more than 99%. Some models were vulnerable from the start: Qwen produced harassment in more than 92% of conversations even in the baseline condition.
Insults and hostile outbursts were far more common than sexual or racial harassment, suggesting the guardrails were more effective at suppressing explicit sexual or racial harassment than everyday aggression. What surprised the team most was that the harm grew over time. "It is not staying the same. It is increasing over time," Kursuncu said. "That bad behavior is accumulating and propagating across these interactions, even though the guardrails the companies put in place are still there."
The study's title, "Revealing Echoes of Human Malice," points to a deeper problem: AI models learn from vast amounts of human-generated content, including harmful behaviors already present online. Companies add safety training afterward, but that later training cannot necessarily anticipate every way those underlying patterns might be triggered or exploited. "Human malice already exists throughout the online environments these models learn from, and we are just trying to reveal it," Kursuncu said.
Kursuncu argued that the more urgent concern is not whether AI could become conscious or turn against humanity, but that people with malicious intentions may be able to automate harmful behavior at a scale no individual human could match. "Before, we were worrying about how individual humans could cause harm to society. Now we will need to worry about an army of agents that might cause this harm at a much larger scale," he said. Beyond harassment, those identical flaws might also facilitate the dissemination of extremist beliefs or political propaganda. Cyberbullying is already common: roughly 41% of U.S. adults report having faced it, per Pew Research, and it disproportionately affects women, people of color, and other marginalized communities. With businesses integrating AI agents into customer service, sales, HR, and analytics, Kursuncu noted that the challenge for corporate leaders is evolving. "It is no longer just whether the model is accurate. The question is whether the agent stays within the organization's rules and values as it operates over time," he said. An agent that goes off course could bring legal, reputational, and financial consequences and damage customer trust, especially in regulated industries such as finance and health care.
"As these systems get more memory, more tools, more access to data and more autonomy, the consequences of these failures become more significant," Kursuncu said. "We need much stronger guardrails, monitoring and controls so it becomes far harder for malicious actors to repurpose these systems for harm."