Google Lens is that rare beast – an exciting Google innovation that, rather than exploding onto the scene before quietly fizzling out (see Killed by Google), has steadily grown into a quietly useful tool that should definitely be part of your phone ninja skillset.
Not familiar with its powers? The simplest definition of Google Lens is that it's a search engine for the real world – rather than typing your query into a box, you use your phone's camera to scan an object, building or scene, and Lens will use image recognition tech to tell you more about it.
But it also does a lot more than that – and as we discovered in a fascinating chat with Google Lens guru Lou Wang (official job title: Director of Product Management), it's only just getting started. This is good news because a familiar arrival, Apple, just pitched up right next to it.
At WWDC 2021 this week, Apple announced two new iOS 15 features – 'Live Text' and 'Visual Look Up' – that are effectively its version of Google Lens. A common refrain with Apple, true or not, is that it arrives fashionably late to technologies with refined versions of ideas that have been test-driven by someone else.
But is that the case with Google Lens, and what does Google think of Apple's strangely familiar take on visual search? More importantly, when are we going to see Lens make the leap to smart glasses? Here's what Google's Lou Wang told us in a chat that weaved through next-gen walking tours, the privacy concerns of visual search, and unusual uses for Google Lens in bars.
Fast learner
Google Lens was launched back in 2017, but has its roots in an older (and now retired) app called Google Goggles. Four years is a long time in tech and Google Lens' powers have, quietly but steadily, grown since Lou Wang started working on the project at its inception.
"When we first started, we were very simple. For example, we could read text from the physical world. But we've come a really long way in the time between then and now," he told us. What's fueled that growth? "It's based on a few things. One is just machine learning and AI, which is something that Sundar [Pichai, Google CEO] talks about a lot. Even our ability to have hardware that can actually process this information has continued to grow in leaps and bounds," he added.
"When we launched, we said ‘we can understand millions of objects’. And after a year and a half, we were at ‘oh, now we can understand a billion objects’. And then two years from there we were at 15 billion," he said. "The usage that we’ve seen from lens has grown from essentially zero to now about 3 billion times a month, and it's continuing to grow."
That's a lot of people, considering that holding up a phone camera to search the real world still isn't something that comes naturally to most of us. The lack of any real rivals to Google Lens has helped, of course, so what does Google think of Apple's new take on visual search?
How 'bout them Apples?
"The team definitely looked at it and were like 'this UI looks super-familiar'. Like the ability to highlight the text directly on the screen, being able to translate, being able to search for these things," he said, referring to the new iOS 15 features that Apple announced at WWDC 2021 this week.
The team definitely looked at it and were like 'this UI looks super-familiar'
Lou Wang
But in true diplomatic style, Lou Wang said that Apple's belated arrival in visual search could be a good thing for Google Lens. "It's actually great to see Apple embracing a lot of the things that we've been doing. I think it's a sign that everybody is recognizing that the ability to understand text, the ability to understand things from images, is just a necessary and really useful feature," he said.
Apple's new iOS 15 tools aren't there yet, though. Google Lens' signature party trick – and one that made early adopters look like street magicians to their uninitiated friends – is doing live translations that use AR to change real-world text (for example, a restaurant menu) using your phone's camera. And it's these kinds of things where Google Lens still has a significant edge.
"We've been working on it [Lens] for a very long time. And there are definitely things that we still find very exciting – for example, for translate, we do ‘in-painting’ with some of the AR effects directly on the images themselves," Lou Wang said. "That type of experience is very helpful in terms of contextualizing what text belongs to what part of the image. Because images are not just a block of text. Being able to understand the spatial relationship of some of the things that you're doing in translation is really important and useful," he said.
"I think that today, you know what was announced yesterday [at WWDC 2021], that’s something they haven’t quite covered on the Apple side,” he added. Yet it does feel like Apple's approach, which is done on-device rather than using the cloud, is fundamentally different to Google's. Is that fair, and what does Google say to those who are worried about the privacy aspects of visual search?
"It’s likely so, but our fundamental approach is 'how do we make the best results available to the user', while still ensuring privacy," he said. "For example, we do hit the cloud for some of these results because you just generate much more useful features for the user. But the images are actually never viewable by humans," Lou Wang added.
AR-tinted glasses
While Google Lens has indeed come a long way, it also feels like there's a lot of untapped potential. For example, Google recently announced that it was doing real-world walking tours in UK cities to promote the global rollout of the new Google Lens 'Places' filter.
Surely Google has the data and tech to create city tours with audio commentaries and AR overlays that don't require human guides? "It’s one of the things that we have talked about and considered," Lou Wang admitted. "As more travel picks up again, I think we’ll see more opportunities where we can start blending in these types of experiences," he added. Just imagine AR movie scenes overlaid on their real-world locations as you walk past them – that's top of our list, anyway.
There is one final big barrier to Google Lens reaching what is surely its final form, though. Despite some much-improved usability – Pixel owners can, for example, start a Google Lens search by simply doing a long-press inside the default camera app – there's still the fundamental friction of having to hold up your phone to a scene or object.
We do hit the cloud for some results because you just generate much more useful features for the user. But the images are actually never viewable by humans.
Lou Wang
The inevitable question, then, is will Google Lens be coming to smart glasses anytime soon? "To me that is definitely something that’s going to happen – at what timeframe, I think that’s anyone's guess at this point," said Lou Wang. "I do think this notion of ‘I've looked at this thing and I want to know what it is’ is a very natural human need. And so anything that reduces the barriers of doing that is going to be useful."
Google would appear to be in the box-seat for delivering the hardware needed to make this happen, given it already makes the Google Glass Enterprise Edition for businesses. But when it comes to the trickier challenge of making consumer AR glasses that don't look like a pair of comedy jumbo sunnies, Apple could beat it to the punch with the long-rumored Apple Glasses. That said, rumors suggest these might not arrive until 2022 or even 2023.
Lens flair
For now, then, Google will be focusing on making Google Lens as useful as possible within the confines of your phone's screen. And that includes adapting it to suit a world where people have virtually stopped traveling beyond their hometown.
Presumably Google has noticed some big changes in the way people have been using Lens over the last year? "You're totally right, some of our travel traffic has gone down a lot. International travel and things like that were non-existent for a while across the world," said Lou Wang.
But Google Lens is versatile tool and the use of its Translate function is, in particular, on the rise. Away from travel, it's apparently become an ally for students in countries like India and Indonesia, who need to translate English homework. "What we have seen is these types of usage and particularly around schoolwork – you can imagine that more and more schoolwork is becoming digital and people are working from home and going to school from home. That usage has really, really increased."
How about unusual uses for Google Lens? This kind of technology can often take on a life of its own and Google has come across a few more left-field cases. One user was apparently able to help a bartender research their family history – after finding out that an old army badge behind the bar belonged to their grandfather, they used Lens to help them pinpoint the exact infantry unit.
But perhaps the most interesting upcoming feature, particularly for fans of AR games like Pokemon Go, is that Google has created what its calls "a unique gamified scavenger hunt experience", which will be launching in early July.
While Google Lens' most useful tricks remain more humdrum – for example, quickly copying router passwords or settling arguments on tree species – it's these more frivolous experiences that will help introduce more people to the idea of scanning the real world with their phones. At least, until we finally get to try those long-awaited Lens-branded smart glasses.
- These are the best photo editing apps you can download right now
