The Photos app is getting a new feature in iOS 15 called Visual Look Up, and it leverages the powerful on-device machine learning that Apple devices are known for.
Note: This is a beta so the location of these features within Settings may change.
iOS 15 Visual Search
So what is Visual Look Up? As far as I can tell in these betas, this is only found within the Photos app. It means that occasionally the machine learning can detect the contents of the photo and give you information about it.
So far, it only works on some photos but not others. If you swipe up on a photo you’ll see information like the time and date when the photo was taken/saved, the app in which it was edited (if any), and camera data like ISO, camera model, file size, and more.
But sometimes when you swipe up you’ll see an icon on the photo, like a leaf. Tap that icon and the algorithm will try to identify the photo content. Example: I have a photo of a plant in my camera roll and the machine learning identified the species along with links to Wikipedia.
Apple’s page says: “With Visual Look Up, users can learn more about popular art and landmarks around the world, plants and flowers found in nature, breeds of pets, and even find books.”
Overall it’s a great way for smartphones to interact with the physical world, and could have some applications for augmented reality in the future.