Data poisoning attacks: Sounding the alarm on GenAI’s silent killer

A computer being guarded by cybersecurity.
(Image credit: iStock)

When researchers at software management company, JFrog, routinely scanned AI/ML models uploaded to Hugging Face earlier this year, the discovery of a hundred malicious models put the spotlight on an underrated category of cybersecurity woes: data poisoning and manipulation.

The problem with data poisoning, which targets the training data used to build Artificial Intelligence(AI)/Machine Learning(ML) models, is that it's unorthodox as far as cyberattacks go, and in some cases, can be impossible to detect or stop. Attacking AI this way is relatively easy and no hacking in the traditional sense is even required to poison or manipulate training data that popular large language models (LLMs) like ChatGPT rely on.

Data poisoning can be used to make the AI model do your bidding. Or AI models can be convinced to give erroneous output by modifying the data sent into a trained model. These are two different types of attack–one that is done before the AI model deploys, the other done post-deployment. Both are incredibly difficult to ferret out and guard against.

In its analysis, JFrog noted the “intriguing“ payload embedded within the model looked like something researchers would upload to demonstrate vulnerabilities or showcase proofs-of-concept. That was not the case with the nefarious models uploaded to Hugging Face’s AI collaboration repository. Researchers may have been behind it because the payloads had links to IP addresses from KREOnet, or the Korea Research Environment Open Network.

Sam Curry

Global VP and CISO in Residence at Zscaler.

Built-in AI problems exacerbate detection while fertilizing exploits

Examples of training data manipulation can be traced to the origins of machine learning, with researchers demonstrating subtle adversarial attacks on input results in a model outputting an incorrect answer with high confidence a decade ago.

It’s even possible that generative AI models scrapping the internet could eventually “poison” themselves as their outputs become inputs for future training sets, in a process known as “degenerative model collapse.”

What muddles the waters further is that AI model reproducibility is in itself a challenge as there are vast pools of data used to train models, and researchers and data scientists may not even understand exactly what went into a model and what is coming out, exacerbating the detection and traceability of malicious code.

Inconvenient as all of this sounds in the AI gold rush, turning a blind eye to data poisoning and data manipulation can embolden attackers to focus on stealth backdoor exploits of AI software. The results can be malicious code execution, as in the case of Hugging Face, new vectors to successfully carry out phishing attacks, and misclassified model outputs that lead to unexpected behaviors, depending on the goals of the attacker.

In a world increasingly blanketed with an ecosystem of interconnected AI, GenAI, LLMs, and APIs, the global cybersecurity industry should release a collective shudder and take action to protect against the rise of attacks on AI models.

Protecting against the “indefensible”

Experts advise several techniques to protect AI-driven systems from data poisoning or manipulation campaigns. Most focus on the data training stage and the algorithms themselves.

In its “Top 10 for LLM Applications” list, the Open Source Foundation for Application Security (OWASP) recommends steps to prevent training data poisoning, starting with paying attention to the supply chain of internally and externally sourced training data, with continuous verification of data sources across pre-training, fine-tuning, and embedding stages and flagging of any biases or anomalies.

OWASP also recommends “sanitizing” the data with statistical outlier and anomaly detection methods to hunt down any adversarial data from potentially being fed into the fine-tuning process.

If training data is corrupted, alternate AI algorithms can be used to deploy the impacted model. More than one algorithm can be used to compare results, and fallback to pre-defined or averaged outputs when all else fails. Developers should closely examine AI/ML algorithms that interact or feed into others, as it can lead to a cascade of unexpected predictions.

Industry experts also suggest that cybersecurity teams check the robustness and resilience of their AI systems by pentesting and simulating a data poisoning attack.

A 100% cybersecure AI model can be built and poisoned using training data. There is no defense other than validating all the predictive output, which is very expensive computationally.

Building a resilient future for AI

Without trust and dependability, the greatest innovation in tech may hit the brakes.

Organizations need to prevent backdoor threats in AI code generation by treating the entire ecosystem and supply chains that underpin GenAI, LLMs, etc. as part of the overall threat universe.

By monitoring the inputs and outputs of these systems and detecting anomalies with threat intelligence, findings and data from these efforts can help developers promote and use controls and protections in the AI software development lifecycle.

Overall, examining the risks of AI systems within the broader business processes, including checking the entire data governance lifecycle, and monitoring how AI behaves in specific applications, you can stay one step ahead of one of the most challenging issues facing cybersecurity.

We've featured the best database software.

This article was produced as part of TechRadarPro's Expert Insights channel where we feature the best and brightest minds in the technology industry today. The views expressed here are those of the author and are not necessarily those of TechRadarPro or Future plc. If you are interested in contributing find out more here: https://www.techradar.com/news/submit-your-story-to-techradar-pro

TOPICS
Sam Curry

Global VP & CISO in Residence at Zscaler.

Read more
An abstract image of digital security.
Identifying the evolving security threats to AI models
Abstract image of cyber security in action.
Protectors of the modern world: defending against Shadow ML and Agentic AI
Half man, half AI.
Generative AI has a long way to go as siloed data and abuse of its capacity remain a downside – but it does change the game for security teams
Closing the cybersecurity skills gap
AI security: establishing the first and last layer of defense
A stylized depiction of a padlocked WiFi symbol sitting in the centre of an interlocking vault.
Sounding the alarm on AI-powered cybersecurity threats in 2025
A hand reaching out to touch a futuristic rendering of an AI processor.
Balancing innovation and security in an era of intensifying global competition
Latest in Pro
Epson EcoTank ET-4850 next to a TechRadar badge that reads Big Savings
I found the best printer deal you won't see in the Amazon Spring Sale and it's got a massive $150 saving
Microsoft Copiot Studio deep reasoning and agent flows
Microsoft reveals OpenAI-powered Copilot AI agents to bosot your work research and data analysis
Group of people meeting
Inflexible work policies are pushing tech workers to quit
Data leak
Top home hardware firm data leak could see millions of customers affected
Representational image depecting cybersecurity protection
Third-party security issues could be the biggest threat facing your business
An image of network security icons for a network encircling a digital blue earth.
Why multi-CDNs are going to shake up 2025
Latest in News
Nintendo Music teaser art
Nintendo Music expands its library with songs from Kirby and the Forgotten Land and Tetris
An image of Pro-Ject's Flatten it closed and opened
Pro-Ject’s new vinyl flattener will fix any warped LPs you inadvertently buy on Record Store Day
The iPhone 16 Pro on a grey background
iPhone 17 Pro tipped to get 8K video recording – but I want these 3 video features instead
EA Sports F1 25 promotional image featuring drivers Oscar Piastri, Carlos Sainz and Oliver Bearman.
F1 25 has been officially announced, with this year's entry marking a return for Braking Point and a 'significant overhaul' for My Team mode
Garmin clippd integration
Garmin's golf watches just got a big software integration upgrade to help you improve your game
Robert Downey Jr reveals himself as Doctor Doom to a delighted crowd at San Diego Comic-Con 2024
Marvel is currently revealing the full cast for Avengers: Doomsday, and I think it's going to be a long-winded announcement