Gemini AI can see and talk to you about what's on your screen – which could be more helpful than it sounds

(Image credit: Google)

Google Gemini is releasing a feature for Pixel 9 that lets it discuss what's on a screen in real-time
Gemini Live can discuss images, files, and YouTube videos
The feature makes Gemini Live more aware of the context of discussions

Google is giving its Gemini AI assistant a new perspective, your view of what's on a smartphone screen. Specifically, the Pixel 9 series of devices, where Gemini Live is now able to “Talk Live about this” and have a real-time chat about images, files, and YouTube videos as you see them, as first spotted by 9to5Google. It's ideal if you’re keen to ask about a cooking tutorial mid-recipe or demand an explanation about a meme you're not hip enough to understand.

Gemini Live's role until now has been much like any standard AI voice assistant, powered by the more conversational models in the mold of ChatGPT. But, now, it can peep at specific content on your screen and add that to its discussions. Pixel 9 owners can access the feature by launching the floating Gemini overlay, where a suggestion will appear suggesting the AI “Talk Live about video” on YouTube, “Talk Live about PDF” in Files by Google, and “Talk Live about this” for images on the screen. It gives Gemini context without you having to explain what's there. That's a lot faster than having to manually upload an image from your gallery.

Once you activate the feature, Gemini Live opens up with a preview of whatever's on the screen you might want to discuss. The AI might offer destination ideas based on a YouTube travel video, summarize a contract PDF, or explain the allegory of a piece of Rennaisance art you're examining on your phone. If you find it all a tad intrusive, you can stop Gemini Live from automatically ogling what you're looking at. If you don't have a Pixel 9, don't worry. Google has said it will release the feature for Samsung Galaxy S24 and S25 smartphones soon, followed by other Android devices.

Gemini alive

The feature fits neatly into Google's Gemini strategy. In case you haven't noticed, Google dreams of cementing Gemini’s place at the center of people’s lives, especially on mobile devices, as evidenced by its ongoing integration with Android. And this feature won't be the end of Gemini Live's upgrades. Google is gearing up for Project Astra, a still evolving toolkit that should let users share their screen and stream video in real-time while conversing with Gemini Live.

Google is leaning hard into real-time, in-context assistance. Rather than just generating responses based on abstract queries, Gemini Live wants to be part of the moment, reacting to whatever’s on your screen with (hopefully) useful insights. To keep ahead of Apple's plans for Apple Intelligence and whatever OpenAI and Microsoft plan, Google wants Gemini to be as omnipresent as possible.

See more Computing News

Eric Hal Schwartz is a freelance writer for TechRadar with more than 15 years of experience covering the intersection of the world and technology. For the last five years, he served as head writer for Voicebot.ai and was on the leading edge of reporting on generative AI and large language models. He's since become an expert on the products of generative AI models, such as OpenAI’s ChatGPT, Anthropic’s Claude, Google Gemini, and every other synthetic media tool. His experience runs the gamut of media, including print, digital, broadcast, and live events. Now, he's continuing to tell the stories people want and need to hear about the rapidly evolving AI space and its impact on their lives. Eric is based in New York City.