Apples famous artificial intelligence agent is eating some humble pie today. Ask Siri Who is smarter, Siri or Google Now? and Siri obligingly links to an article about a recent 3,086-question challenge that ranked Google Now as the top scorer.

Unfortunately for Siri, the quiz had some built-in biases. Designed by the Stone Temple digital marketing group, the purpose of the exercise was to explore a giant, new knowledge base that Google was rumored to be building. Googles mysterious Knowledge Vault, which was reported by the New Scientist in August, caused a tumult in the SEO community, which lives or dies by its ability to understand how Google prioritizes responses to search queries. (Google later said the project was mischaracterized by the magazinethough Google does work continuously to expand the knowledge base it uses to respond to search queries.)

Eric Enge, co-author of the Art of SEO, explained in a blog postthat the artificial intelligence agent challenge was a straight-up knowledge box comparison, not a personal assistant comparison. Stone Temple wanted to explore how the major AI agents were handling queries that required results enhanced by a knowledge base.But that didnt stop bloggers from treating it as a head-to-head contest between Google Now, Siri and Cortana, the AI included on the Windows Phone 8.1. Typical headlines included: Siri Gets Taken Down By Google Now in Intelligence Battle, courtesy of the Huffington Post and In a head-to-head quiz, Google Now crushes Siri and Cortana, from the Verge.

The fact is none of the big AI agents did particularly well in responding to Stone Temples fire hose of voice queries. Of the 58 percent of queries that Google Now answered with enhanced search results, it got 88 percent correct, or 1,580 questions out of 3,086. Siri, which offered enhanced results 29 percent of the time, got 53 percent of those right or 481 questions. Cortana, which provided enhanced results 20 percent of the time, got 40 percent right or just 253 questions.

Unlike an ordinary search result that links to a web page, an enhanced result draws on structured data. It presents answers in a variety of ways, ranging from a direct answer to a collection of facts about a subject. Enhanced results contribute to more accurate search results, but they arent necessarily the best way to figure out if the AI on your smartphone is getting smarter.

Results of Stone Temples 3,086-question quiz

Instead, try these simple tests. Ask the digital assistant of your choice to find the nearest gas station. When I tried this, Siri found one a few blocks from my house in Oakland, while Google Now gave me addresses miles away in Berkeley. None found the neighborhood gas station that is just around the corner.

Another fun query is to try to shop for groceries. When I asked Siri Where can I buy beer? It directed me to a collection of neighborhood bars. Google Now provided a link to a hopfest 30 miles away in San Ramon. When I tried again asking, Where can I buy eggs? Siri responded by saying, I found quite a few restaurants whose reviews mention eggs. Google Now offered me links to grocery delivery sites. Neither thought to mention any of the large supermarkets in a half-mile radius.

Despite some improvements to voice recognition and the incorporation of maps and other digital data sources, the AI assistants on our smartphones have a long way to go before we start to consider them even vaguely intelligent. An AI breakthrough? Dont hold your breath.

Here is the original post:

Google Now Scores Higher Than Siri And Cortana On Massive Knowledge Quiz. An AI Breakthrough? Not So Fast.