Amazon’s Fire Phone Introduces Firefly, A Feature That Lets You Identify (And Buy!) Things You See In The Real World

With the debut of the Amazon Fire Phone this afternoon, the company introduced a new feature called Firefly, which is largely designed to let you identify — and then, of course, buy — things you see out in the real world. Firefly takes advantage of the smartphone’s camera to let you identify things like phone numbers, movies, books, games, CDs, food and more just by pointing your camera at them.

The feature is a key part of the new phone, in fact, as it even has a dedicated button on the side – a perfect example of how much this new hardware is about bringing customers into Amazon’s ecosystem and turning them into regular online shoppers.

A Firefly button is sort of like “instant gratification” built into your device.

During the demonstration on stage, Amazon CEO Jeff Bezos showed off how Firefly can be used to scan barcodes on items, a book cover, a CD and more. There’s a Shazam-like audio recognition-component, as well. Firefly can identify songs you’re listening to, and it can even listen to a show, and then let you access that specific scene on Amazon, which is a clever trick. Naturally, you’re directed to buy the tracks and shows and movies on Amazon.

To some extent, Firefly looks like a feature built on top of Flow, Amazon’s visual-recognition technology that used to reside in its own standalone mobile app before more recently being introduced into the main Amazon native application.

Flow, which was run by Amazon subsidiary A9 (the search and advertising wing of Amazon), was first launched over two years ago. That app, and now Amazon.com’s flagship app, work by identifying media package covers, logos, artwork and other unique visual features.

So by Amazon’s description of Firefly, it’s clearly an extension of Flow, though that was not said outright. (It’s unclear why the company decided to rename it. But “Firefly” does sound cooler.)

Firefly can also recognize art, and then pull up the associated Wikpedia entry, which is something Apple’s Siri and Google Now can’t yet do.

At launch, Firefly is capable of recognizing over 100 million items, said Bezos. That includes 245,000 movies and TV episodes, 160 live TV channels, 35 million songs and 70 million products.

It’s worth noting the Firefly feature isn’t only about finding things you can buy, although clearly that’s its larger goal.

Under the hood, the technology being used includes “semantic boosting,” Bezos continued, which is a fancy way of saying it can smartly identify and understand text based on the context. It can identify phone numbers using image-to-text conversion and can determine if a phone number doesn’t exist. This feature was demonstrated by pointing the phone at a sign for a pie shop, which then allowed you to kick off a phone call to that store.

The feature works for signs, posters, magazines or business cards, and it lets you send emails, save as a contact, or go to a website without typing a URL.

In addition, developers will be able to tap into Firefly, which makes the feature even more promising.

An SDK will allow third-parties access to all the functionality in Firefly itself. One company, the health-focused MyFitnessPal, has already taken advantage of the feature to create an app that can determine the nutritional information from the food your camera sees. Today, users have to manually enter this data or select a food from a set of search results. Firefly makes MyFitnessPal’s food diary “point and shoot.”

Included in the SDK are text, audio and image recognizers, content databases and support for custom actions allowing developers to bend Firefly to their own personal needs.

In another example, Vivino is releasing an app that gives you information about the bottles of wine you scan. Other early adopters include iHeartRadio and StubHub.

While Flow functionality may have seemed gimmicky at times — after all, it’s not that hard to enter a search query — opening up Flow (via Firefly) to a wider audience of developers has effectively just created a new app category, and something that —  for now at least — is exclusive to Amazon’s platform.

While Google has also experimented with visual recognition technology in the past (remember “Google Goggles?“), it hasn’t really been a huge focus for the company. Meanwhile, it makes sense for Amazon to pursue this feature, as the company sees identification as the first step in the purchasing process, not just another cool trick you can do with your phone.

The Firefly SDK is being launched today, said Bezos.

This post was made using the Auto Blogging Software from WebMagnates.org This line will not appear when posts are made after activating the software to full version.