Summary
- Google will soon allow users to upload videos along with text queries to enhance search accuracy.
- Gemini AI will facilitate the upload of videos depicting problems, allowing it to search through various online resources for solutions.
- The feature will initially be available for testing in Search Labs for US users in English.
At the 2021 Google I/O developer conference, the company unveiled an AI breakthrough called Multitask Unified Model, or MUM, which can process and understand various formats like text, images, and videos all at once. It features a multisearch option that combines pictures with text to refine results. Soon, you’ll be able to ask questions through videos in Google Lens too.
The goal is to integrate video searching more deeply into Google Search, powered by Gemini AI. Announced at today’s Google I/O conference, the “Ask with video” feature will soon allow you to upload a video along with a text query. This combination will help the AI provide a comprehensive summary of the answers you’re looking for.
Google I/O 2024: Everything announced so far
It’s all about AI this year — here’s everything Google’s talked about so far
Gemini AI will allow you to upload a video showing a problem you’re facing and then search through user forums and other online resources to help you out. For instance, Google’s Rose Yao shared on stage her experience with a second-hand turntable she purchased. She couldn’t get the needle to properly contact the record. After uploading a video of the issue, Gemini swiftly located a guide that explained how to adjust the arm balance for her specific turntable model.
According to Yao, Search will comprehend questions you might ask out loud and analyze the video frame-by-frame, using Google’s speech models, deep visual understanding, and the custom Gemini model. Each frame went into Gemini’s long context window, allowing Search to accurately identify the make and model of the record player and understand the motion across frames, pinpointing that the tonearm was drifting, Yao explained. Google Search then scoured the web, pulling insights from articles, forums, videos, and more, and compiled everything into an AI overview.
Google could really boost its search relevance
This feature could really shine for searches where it’s tough to describe what you’re looking for with just words—situations where there’s a visual element that might be interpreted in various ways. By merging video with text in a single query, Google could greatly improve its ability to deliver relevant search results.
The feature will first roll out as a test in Search Labs for US users in English. Google plans to leverage the phone’s camera in Search to stay competitive as users increasingly turn to platforms like TikTok and Instagram for short-form videos, which are cutting into its main Search product.
Gemini comes to Google Photos to supercharge search and curation
Gemini will make search in Google Photos a lot smarter
