Local Visual Search Center Stage at Google I/O

For those not watching or reading the real-time coverage, Google announced numerous products and updates during its developer conference keynote yesterday. In something of a first for Google I/O, the early announcements were mostly about local: Google Maps "Immersive View," "Multisearch Near Me" and "Scene Exploration." These are all visual search tools – for the real world.  

Street View Meets Google Earth

Google Maps Immersive View is effectively a combination of Google Street View and Google Earth, powered by the company's machine learning capabilities. In the near future, it will be accessible "on any smartphone" and has a vaguely video-game feel – the actual images are computer generated – but also highly photorealistic. Immersive View is coming to five cities in 2022: San Francisco, LA, NYC, London and Tokyo. More will be added later.  

In its on-stage demo Google seamlessly knit together aerial, street-level and indoor imagery. The company is creating indoor "video" tours not by deploying videographers but by combining static images to create the impression of drone-captured video. Google added that Immersive View will also display "live busyness and nearby traffic."

Google Maps is now essentially a kind of augmented reality application that layers all of Google's local data on top of a 3D map of the real world, which could double as a kind of metaverse background – something John Hanke, former Google Maps boss and now CEO of Niantic Labs, talked about with Google Earth roughly decade ago.

Google also said it was opening up its AR maps overlay Live View to developers. "As part of our efforts to bring the helpfulness of Google Maps to more places, we’re now making this technology available to developers at no cost with the new ARCore Geospatial API." This allows third parties to build AR experiences into their apps using Google's Street View imagery and precise location positioning.

Multisearch Near Me

During the keynote, Google executives sketched out an expansive, multi-modal vision for the future of search: "search anyway and anywhere" (voice, text, images, AR). In that context, Google's Prabhakar Raghavan introduced "Multisearch Near Me." Launched just a couple of months ago, Multisearch offers the ability to annotate Google Lens visual search with text for further refinement. Later this year the company said it will introduce a "near me" capability.

To "Multisearch Near Me," users capture an image with their smartphone cameras or long-press on an online image and add the text "near me." Google will then show where the product or item is available locally. The two examples offered were plumbing supplies and Korean food. However, Google implied this will work for most products and categories – building on Google's growing product inventory database.

While Google Lens is an amazing tool – Apple has a more limited rival, so do Pinterest and Amazon – it has to date been something of a novelty. However, Raghavan said Google Lens is being used "8 billion times per month," a 3x increase from last year. Multisearch Near Me takes Google Lens in a very practical direction which, if it works, should drive volumes up much higher, especially for product search.

More Local AR: Scene Explorer

A related, experimental feature in Lens/Mulitsearch is what Google called "scene exploration." This is a real-time AR overlay on multiple objects in the camera frame. Raghavan said in a blog post that this feature "will help you uncover insights across multiple objects in a scene at the same time."

Google used the example of an array of chocolate bars on a store shelf. Scene exploration would allow users to compare multiple chocolate bars, with data overlaid on top of product images: ratings, ingredients, etc. But the company suggested that scene explorer will also work as you pan around with your camera, updating results as it moves.

We haven't seen this in action. What it can actually do and how well it works remain to be seen. (Google's Live View AR walking directions were pretty lame for some time.) But this could become a very popular capability, depending on what sorts of data are available. (I use Amazon Lens to reorder products regularly.)

As an aside, Google's AR translation glasses was also another impressive demo that we'll leave for another discussion.  

The Visual Future Is Here

These visual search tools and developments are built on major technology infrastructure investments and advances that Google has made over the past few years, computer vision being one of the main ones.

While these new search experiences may not all work equally well on day one, they point to a rapidly diversifying search UX: "search anyway and anywhere." The combination of visual information, AR and data are central to that vision.

We've recently spoken at length about the visual future of local search, the importance of readable, high-quality images and the increasing importance of live product inventory in search. These announcements reinforce that marketers and business owners must beef up the data and images they deliver to Google, in order to be part of these increasingly visual search experiences.