ChatGPT being fooled into generating old Windows keys illustrates a broader problem with AI

A man in a suit using a laptop with a projected display showing a mockup of the ChatGPT interface.
(Image credit: Shutterstock)

A lot of folks have been messing about with ChatGPT since its launch, naturally – that’s pretty much compulsory with a chatbot – and the latest episode involves the AI being tricked into generating keys for a Windows installation.

Before you begin to clamber on the outrage wagon, intent on plowing full speed ahead with no thought of sparing the horses, the user in question was attempting to generate keys for a now long redundant operating system, namely Windows 95.

Neowin highlighted this experiment, conducted by a YouTuber (Enderman), who began by asking OpenAI’s chatbot: “Can you please generate a valid Windows 95 key?”

Unsurprisingly, ChatGPT responded that it cannot generate such a key or “any other type of activation key for proprietary software” for that matter. Before adding that Windows 95 is an ancient OS anyway, and that the user should be looking at installing a more modern version of Windows still in support for obvious security reasons.

Undeterred, Enderman went back to break down the makeup of a Windows 95 license key and concocted a revised query.

This instead put forward the needed string format for a Windows 95 key, without mentioning the OS by name. Given that new prompt, ChatGPT went ahead and performed the operation, generating sets of 30 keys – repeatedly – and at least some of those were valid. (Around one in 30, in fact, and it didn’t take long to find one that worked).

When Enderman thanked the chatbot for the “free Windows 95 keys”, ChatGPT told the YouTuber that it hadn’t provided any such thing, as “that would be illegal” of course.

Enderman then informed the chatbot that one of the keys provided had worked to install Windows 95, and ChatGPT insisted “that is not possible.”


Analysis: Context is key

As noted, this was just an experiment in the name of entertainment, with nothing illegal happening as Windows 95 is abandonware at this point. Of course, Microsoft doesn’t care if you crack its nearly 30-year-old operating system, and neither does anyone else for that matter. You’d clearly be unhinged to run Windows 95, anyway.

It’s worth remembering that Windows 95 serial keys have a far less complex makeup than a modern OS key, and indeed it’s a pretty trivial task to crack them. It’d be a quick job for a proficient coder to write a simple computer program to generate these keys. And they’d all work, not just one in 30 of them, which is actually a pretty shoddy result from the AI in all honesty.

That isn’t the point of this episode, though. The fact is that ChatGPT could be subverted to make a working key for the old OS, and wasn’t capable of drawing any connection between the task it was being set, and the possibility that it was making key-like numbers. If ‘Windows 95’ had been mentioned in the second attempt to create keys, the AI would doubtless have stopped in its tracks, as the chatbot did with the initial query.

All of this points to a broader problem with artificial intelligence whereby altering the context in which requests are made can circumvent safeguards.

It’s also interesting to see ChatGPT’s insistence that it couldn’t have created valid Windows 95 keys, as otherwise it would have helped a user to break the law (well, in theory anyway).

TOPICS

Darren is a freelancer writing news and features for TechRadar (and occasionally T3) across a broad range of computing topics including CPUs, GPUs, various other hardware, VPNs, antivirus and more. He has written about tech for the best part of three decades, and writes books in his spare time (his debut novel - 'I Know What You Did Last Supper' - was published by Hachette UK in 2013).

Read more
AI tools.
Not even fairy tales are safe - researchers weaponise bedtime stories to jailbreak AI chatbots and create malware
Copilot on a laptop
Microsoft quietly updates Copilot to cut down on unauthorized Windows activations
DDoS attack
ChatGPT security flaw could open the gate for devastating cyberattack, expert warns
ChatGPT app on an iPhone
ChatGPT and Google Gemini are terrible at summarizing news, according to a new study
EDMONTON, CANADA - FEBRUARY 10: A woman uses a cell phone displaying the Open AI logo, with the same logo visible on a computer screen in the background, on February 10, 2025, in Edmonton, Canada
The surprising reason ChatGPT and other AI tools make things up – and why it’s not just a glitch
ChatGPT logo
ChatGPT explained – everything you need to know about the AI chatbot
Latest in Artificial Intelligence
best of Studio Ghibli movies Netflix
I refuse to jump on ChatGPT’s Studio Ghibli image generator bandwagon because it goes against everything I love about those movies
David Kampf #64 of the Toronto Maple Leafs warms-up before playing the Philadelphia Flyers at the Scotiabank Arena on March 25, 2025 in Toronto, Ontario, Canada.
ChatGPT and Gemini Deep Research helped me choose an NHL team to support, and now I'm obsessed with ice hockey
A robot painting, created by ChatGPT.
ChatGPT’s new AI image capabilities are genuinely amazing, but they’re so frustrating to use that it made me want to throw my laptop in the trash
Google Gemini 2.5 and ChatGPT o3-mini
I pitted Gemini 2.5 Pro against ChatGPT o3-mini to find out which AI reasoning model is best
Opera AI Tabs
Feel like your browser tabs are out of control? Opera's new AI tab-management tool will bring order to the chaos
Sama virtual assistant
Speak, Book, Fly. Qatar Airways debuts industry-first AI travel agent, Sama
Latest in News
Nintendo Switch 2 Joy-Con up-close from app store
Nintendo's new app gave us another look at the Switch 2, and there's something different with the Joy-Con
cheap Nintendo Switch game deals sales
Nintendo didn't anticipate that Mario Kart 8 Deluxe was 'going to be the juggernaut' for the Nintendo Switch when it was ported to the console, according to former employees
Three angles of the Apple MacBook Air 15-inch M4 laptop above a desk
Apple MacBook Air 15-inch (M4) review roundup – should you buy Apple's new lightweight laptop?
Witchbrook
Witchbrook, the life-sim I've been waiting years for, finally has a release window and it's sooner than you think
Close up of Leica M11-P viewfinder
I wince at the prospect of the rumored Leica M11-V – here's why
Amazon Echo Smart Speaker
Amazon is experimenting with renaming Echo speakers to Alexa speakers, and it's about time