- AI watermarking aims to prove the authenticity of any text
- New study finds it also changes LLM behavior – and in a bad way
- EU AI Act could mean more models have watermarks despite side effects
New Lasso research has revealed that AI watermarking could actually unintentionally change how LLMs behave following the testing of Google DeepMind's SynthID-Text.
The company's researchers found that SynthID-Text can change whether models refuse harmful requests, their susceptibility to prompt injection, which tools AI agent choose and more.
However, at its core, SynthID-Text and other similar watermarking is only designed to hide a machine-readable indicator as to whether text was AI-generated or human-written.
Researchers find that AI watermarking can unintentionally change AI behavior
The "watermarking procedure can therefore affect both what the model says and what an agent does," Lasso concludes, referring to the side effect as "sampling drift."
One of the biggest concerns highlighted by the paper is that, even without an attack, watermarking changed some of the models' refusal decisions, making them more willing to answer potentially harmful prompts. Combined with prompt injection, Lasso found the consequences more amplified.
Despite the unintended consequences, Anthropic recently announced that future generations of Claude would use AI watermarking similar to Google DeepMind's, stressing that one of the key drivers was to adhere to the EU AI Act. With that in mind, AI watermarking is set to become far more mainstream across other model providers, making these mishaps far more common and leading to further security concerns.
Ultimately, Lasso urges developers to rerun benchmarks, safety evaluations and other tests to check for any unintended consequences, rather than just applying it blindly to existing configurations.
"These findings make reassessment important whenever watermarking is introduced or its configuration or key changes," Lasso concludes, stressing that the work shouldn't be taken as an argument against AI watermarking for provenance.
Additionally, the research presents a new angle on AI watermarking, because until now researchers have largely focused on whether watermarks can be both applied and detected effectively. Few have uncovered such security-focused consequences as this one.
Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds.
