The Lynx R1 MR headset.
XR typically refers to the spectrum of spatial technologies that cover the gamut between virtual (VR) reality and augmented reality (AR). The world is largely familiar with the concept of VR, the complete immersion in a digital space. It helps that VR is already available on consumer platforms such as Oculus Quest and PlayStation VR. Less understood is AR, which seeks to enhance the real world by overlaying the digital world on top it. Even within these two categories of XR, there are additional levels of immersionsome smart glasses feature a single monochrome display. Others, such as Hololens or Magic Leap, provide more advanced iterations of AR.
While many are familiar with the concept of VR, the market and the world are still becoming acquainted with AR. While VR is already available on consumer platforms such as Oculus Quest and PlayStation VR, consumer AR that everyone wants is still years away. People want AR glasses to have the graphical capabilities of todays Oculus Quest, inside the fashionable form factor of todays Ray Ban Stories. However, thats simply not possible with todays optical, wireless, battery and processing capabilities. Qualcomm is at the forefront of enabling headworn AR and VR solutions, including the Oculus Quest, Ray Ban Stories, and many other headsets.
I recently spoke with Qualcomms GM of XR, Hugo Swart about the companys vision and foundational work on Mixed Reality. The company has been building chipsets and platforms for XR since the early days and had one of the earliest AR platforms called Vuforia in 2012 and enabled pioneers in headworn AR like ODG. Qualcomm Technologies were also pivotal in enabling early smartphone VR solutions as well as the first Standalone VR headsets including the wildly popular Oculus Quest 2.
Why we need Mixed Reality
In the meantime, technologies like RGB video pass-through AR or Mixed Reality are bridging the gap on the way to AR. You may have heard Microsoft use the term Mixed Reality to describe its XR platform. While this is similar, in the context of this column, Mixed Reality refers to enabling both AR and VR within a single VR device. Microsofts definition refers to different devices sharing the same spatial computing platform and potentially space. In addition to possessing VR capabilities, Qualcomms vision for a Mixed Reality device must be able to use passthrough color cameras to recreate a feed as close to the real world as possible for the purpose of AR.
Mixed Reality capable VR devices must also be lighter and more compact than current VR devicesthey need to feel almost invisible to the user much like a pair of AR glasses would be. Were already seeing this in devices targeted towards Mixed Reality, such as the Lynx R-1 which runs on Qualcomm Snapdragon XR2 chipset. I expect that well see similar devices from Facebook, Apple and other tech giants pursuing AR very soon
Right now, Qualcomm believes there are four paths within AR that are evolving in parallel, on different parts of the XR spectrum:
Eventually, these capabilities will merge to the point where it becomes difficult to tell the difference between each category. The lines will become very blurry, like what were seeing with VR today.
Mixed Reality features
In speaking with Hugo I learned that there are a common series of challenges that must be overcome in order to fully enable this new form of spatial computing. First, Mixed Reality headsets require the lowest latency possible along the entire image processing pipeline, since they must capture whats happening in the real world, process it, render it and then display it inside of the headset, in almost real-time. If things are too out of sync, the user will feel like they are drunk or have motion sickness. There cannot be any weak links in the chain, not even in the display. Every millisecond matters in Mixed Reality, even more than in VR or AR. This is a problem that Qualcomm recognizes and is working to mitigate. According to Hugo Swart, the companys GM of XR, Qualcomm has managed to bring photon-to-photon latency down to under 10 milliseconds from the previously standard 30ms+.
Second, Mixed Reality needs to have a deep understanding of the environment and be able to utilize AI algorithms to recognize objects and map the users surroundings. This is what allows the headset to overlay digital information over the real-world environment for AR functions. Additionally, even if the user is using the headset in VR mode, it needs to be able to help with boundaries and object avoidance. Snapdragon XR2 chipset, which is the standard for todays XR headsets, has these capabilities built-in.
In addition to low-latency and environmental awareness, a Mixed Reality device needs to have enhanced interactivity. In other words, users should be able to interact in AR, VR and MR experiences using different user interfaces and control schemesfor example, hand tracking for base-level interactions and controllers for more intensive applications like gaming and creative work.
A Mixed Reality device must also consider how the user interacts with both virtual and physical objects while inside of the headset and what that might mean for the users immersion. For example, it will need to understand the differences between different types of objects, applying the proper level of opacity to them depending on their degree of relevance. When you look at the previous generation of XR chipsets, the Snapdragon XR2 increased AI performance by 11 times, which is crucial in enabling such features.
Last but certainly not least is the fact that Mixed Reality headsets require a fundamentally better field of view (FoV) than most AR headsets, which typically feature between 30 and 50 degrees. This is because bending light is difficult, and these devices light engines are limited by what can fit inside their necessarily lightweight and compact form factors. Todays technology simply cannot produce anything beyond 50 degrees, for the most part. There are a few exceptions, but these solutions tend to be more appropriate for engineering environments versus consumer, due to their size and weight.
One workaround to this bottleneck is RGB pass-through technology, which delivers the FoV that users expect, on a much more accessible timeline. Snapdragon XR2 chipset was specifically designed to support up to 7 different simultaneous camera streams including ultra-low latency processing to enable a lag-free RGB passthrough experience. Qualcomms powerful Adreno is designed to enable the higher resolutions of Mixed Reality displays to mitigate the loss of visual fidelity that results from an increase in FoV. People want access to AR and MR today, not in three to five years when lightweight AR headsets finally support an acceptable field of view and levels of performance. For AR to be accessible, Mixed Reality headsets must flourish. This will allow developers to build AR experiences on todays available hardware for tomorrows AR headsets.
Mixed Reality: a consumer or enterprise technology?
With how much Mixed Reality borrows from both VR and AR, it would not be surprising for it to inherit many of the same use cases. In fact, consumer and enterprise use cases for AR and VR translate almost perfectly to Mixed Reality. However, there are some use cases that are even better in Mixed Reality than VR or AR by themselves.
One of those is enterprise training. Imagine being able to take advantage of Mixed Realitys wide FoV, overlaying graphics onto, but still very much seeing the real world. This makes Mixed Reality a powerful proposition for job training.
There are additional Mixed Reality applications already in use in the enterprise, such as remote assistance and guided work instructions. However, these stand to become much more powerful and capable than the simple 2D head-mounted solutions we have today. There are also consumer use cases aplenty, since a user can turn virtually any surface in their home into a gigantic display without losing their real-world spatial and situational awareness.
Final thoughts
Many in the industry agree the convergence of AR and VR is inevitable. Mixed Reality is still in its early days, but much of the XR industry appears to be moving in that direction in the near term. Mixed Reality will not only benefit from the technological developments of AR and VR, but also facilitate them at the same time. There will be many users who will want a single, Mixed Reality device to cover the gamut of XR. Qualcomms technological innovations inside of the XR2 are at the core of making that possible today. However, not everyone will want this unified device and may prefer to only have AR or only have VR. Users want choice and hopefully the market will give them that with a diversity of XR devices and approaches. Much like the XR headsets and Android smartphones of today, many will likely be powered by Qualcomm technologies.
Disclosure:My firm, Moor Insights & Strategy, like all research and analyst firms, provides or has provided research, analysis, advising, and/or consulting to many high-tech companies in the industry, including Qualcomm. I do not hold any equity positions with any companies cited in this column
Read more:
Qualcomm And Mixed Reality: The Next Step In The Evolution Of XR? - Forbes
- EvolutionM.net - Mitsubishi Lancer Evolution | Reviews, News ... [Last Updated On: June 12th, 2016] [Originally Added On: June 12th, 2016]
- Evolution - Wikipedia, the free encyclopedia [Last Updated On: June 19th, 2016] [Originally Added On: June 19th, 2016]
- Darwin's Theory Of Evolution [Last Updated On: June 22nd, 2016] [Originally Added On: June 22nd, 2016]
- Evolution - Conservapedia [Last Updated On: June 28th, 2016] [Originally Added On: June 28th, 2016]
- History of Evolution | Internet Encyclopedia of Philosophy [Last Updated On: June 28th, 2016] [Originally Added On: June 28th, 2016]
- Darwin's Theory Of Evolution [Last Updated On: June 28th, 2016] [Originally Added On: June 28th, 2016]
- Evolution - Simple English Wikipedia, the free encyclopedia [Last Updated On: June 29th, 2016] [Originally Added On: June 29th, 2016]
- Evolution (2001) - IMDb [Last Updated On: July 5th, 2016] [Originally Added On: July 5th, 2016]
- Introduction to Human Evolution | The Smithsonian Institution ... [Last Updated On: July 7th, 2016] [Originally Added On: July 7th, 2016]
- EvolutionM.net - Mitsubishi Lancer Evolution | Reviews, News ... [Last Updated On: July 12th, 2016] [Originally Added On: July 12th, 2016]
- Evolution - Biology-Online Dictionary [Last Updated On: July 12th, 2016] [Originally Added On: July 12th, 2016]
- Introduction to Human Evolution | The Smithsonian Institution ... [Last Updated On: July 12th, 2016] [Originally Added On: July 12th, 2016]
- Evolution - Bulbapedia, the community-driven Pokmon encyclopedia [Last Updated On: July 12th, 2016] [Originally Added On: July 12th, 2016]
- What is Evolution - explanation and definitions [Last Updated On: July 12th, 2016] [Originally Added On: July 12th, 2016]
- Apps/Evolution - GNOME Wiki! [Last Updated On: July 12th, 2016] [Originally Added On: July 12th, 2016]
- History of Evolution | Internet Encyclopedia of Philosophy [Last Updated On: July 12th, 2016] [Originally Added On: July 12th, 2016]
- Recent Articles | Evolution | The Scientist Magazine [Last Updated On: July 16th, 2016] [Originally Added On: July 16th, 2016]
- Evolution - The New York Times [Last Updated On: July 25th, 2016] [Originally Added On: July 25th, 2016]
- Evolution : Pictures , Videos, Breaking News [Last Updated On: August 2nd, 2016] [Originally Added On: August 2nd, 2016]
- Faculty & Staff - Biology | Biology | High Point University ... [Last Updated On: August 8th, 2016] [Originally Added On: August 8th, 2016]
- Evolution (software) - Wikipedia, the free encyclopedia [Last Updated On: September 2nd, 2016] [Originally Added On: September 2nd, 2016]
- Evolution | Answers in Genesis [Last Updated On: September 2nd, 2016] [Originally Added On: September 2nd, 2016]
- Evolution (2001) - Rotten Tomatoes [Last Updated On: September 18th, 2016] [Originally Added On: September 18th, 2016]
- Human evolution - Wikipedia, the free encyclopedia [Last Updated On: September 22nd, 2016] [Originally Added On: September 22nd, 2016]
- Evolution - Wikipedia [Last Updated On: October 20th, 2016] [Originally Added On: October 20th, 2016]
- MyEvolution // About Evolution [Last Updated On: December 9th, 2016] [Originally Added On: December 9th, 2016]
- Evolution of the Web [Last Updated On: December 10th, 2016] [Originally Added On: December 10th, 2016]
- Evolution | Pokmon Wiki | Fandom powered by Wikia [Last Updated On: January 14th, 2017] [Originally Added On: January 14th, 2017]
- Evolution - RationalWiki [Last Updated On: January 14th, 2017] [Originally Added On: January 14th, 2017]
- Tracking the Evolution of Student Success - Inside Higher Ed [Last Updated On: February 6th, 2017] [Originally Added On: February 6th, 2017]
- Ivanka Trump's Beauty Evolution, From 1998 to Today Watch - Us Weekly [Last Updated On: February 6th, 2017] [Originally Added On: February 6th, 2017]
- Lumpy, hairy, toe-like fossil could reveal the evolution of molluscs - The Guardian [Last Updated On: February 6th, 2017] [Originally Added On: February 6th, 2017]
- USM Darwin Day: 'Genesis' a parallel to evolution - The Student Printz [Last Updated On: February 6th, 2017] [Originally Added On: February 6th, 2017]
- Cultural evolution and the mutilation of women - The Economist [Last Updated On: February 6th, 2017] [Originally Added On: February 6th, 2017]
- How Evolution Alters Biological Invasions - ScienceBlog.com (blog) [Last Updated On: February 6th, 2017] [Originally Added On: February 6th, 2017]
- Psychonauts in the Rhombus of Ruin Feels Like an Evolution of Double Fine's Adventure Game Roots - UploadVR [Last Updated On: February 7th, 2017] [Originally Added On: February 7th, 2017]
- The Evolution of Accessible Travel: 5 Podcast Takeaways - Skift [Last Updated On: February 7th, 2017] [Originally Added On: February 7th, 2017]
- Convergent Evolution: Why Some Plants Became Carnivorous - Science 2.0 [Last Updated On: February 7th, 2017] [Originally Added On: February 7th, 2017]
- The Queer Evolution of Kristen Stewart - Advocate.com [Last Updated On: February 7th, 2017] [Originally Added On: February 7th, 2017]
- Gold's Gym Regina rebrands to become Evolution Fitness - Regina Leader-Post [Last Updated On: February 7th, 2017] [Originally Added On: February 7th, 2017]
- Late-night hosts on the evolution of Trump: 'Dickish to dictatorish' - The Guardian [Last Updated On: February 7th, 2017] [Originally Added On: February 7th, 2017]
- Exhibition charts 500 years of evolution of robots - Phys.Org [Last Updated On: February 7th, 2017] [Originally Added On: February 7th, 2017]
- Blockchain: Investment (R)Evolution For Developing Markets - Forbes [Last Updated On: February 7th, 2017] [Originally Added On: February 7th, 2017]
- See the Evolution of the Famed Porsche 911 in 7 Photos - WIRED [Last Updated On: February 7th, 2017] [Originally Added On: February 7th, 2017]
- How evolution turned ordinary plants into ravenous meat-eaters - Wired.co.uk [Last Updated On: February 7th, 2017] [Originally Added On: February 7th, 2017]
- Are Evolution Fresh Drinks 'Poison'? - snopes.com [Last Updated On: February 7th, 2017] [Originally Added On: February 7th, 2017]
- The Evolution and Maturation of HPC in the Enterprise - CIO [Last Updated On: February 7th, 2017] [Originally Added On: February 7th, 2017]
- From Whoa to 'Wick:' The Evolution of Keanu Reeves - Film School Rejects [Last Updated On: February 9th, 2017] [Originally Added On: February 9th, 2017]
- 'Goldilocks' genes that tell the tale of human evolution hold clues to variety of diseases - Science Daily [Last Updated On: February 9th, 2017] [Originally Added On: February 9th, 2017]
- London exhibition charts 500 years of evolution of robots - Chicago Sun-Times [Last Updated On: February 9th, 2017] [Originally Added On: February 9th, 2017]
- Chimpanzee feet allow scientists a new grasp on human foot ... - Science Daily [Last Updated On: February 9th, 2017] [Originally Added On: February 9th, 2017]
- 'Evolution To Revolution' As New York Fashion Week Gets Political - NPR [Last Updated On: February 9th, 2017] [Originally Added On: February 9th, 2017]
- Orangutan squeaks reveal language evolution, says study - BBC ... - BBC News [Last Updated On: February 9th, 2017] [Originally Added On: February 9th, 2017]
- Deeper origin of gill evolution suggests 'active lifestyle' link in early ... - Science Daily [Last Updated On: February 10th, 2017] [Originally Added On: February 10th, 2017]
- Horse evolution bucks evolutionary theory - Science News [Last Updated On: February 10th, 2017] [Originally Added On: February 10th, 2017]
- From Tara Palmer-Tomkinson to Cara Delevingne: the evolution of the It girl - The Guardian [Last Updated On: February 11th, 2017] [Originally Added On: February 11th, 2017]
- Evolution gives rhyme its reason - Aurora News Register [Last Updated On: February 11th, 2017] [Originally Added On: February 11th, 2017]
- Evolution of in-car audio tech moving at 'speed of sound' - Times of India [Last Updated On: February 11th, 2017] [Originally Added On: February 11th, 2017]
- Scientists solve fish evolution mystery - Phys.Org [Last Updated On: February 11th, 2017] [Originally Added On: February 11th, 2017]
- The Difference Between Healthy Love & Unhealthy Love - Collective Evolution [Last Updated On: February 12th, 2017] [Originally Added On: February 12th, 2017]
- This Woman Was Raped & Forgave Him, So They Did A Ted Talk Together - Collective Evolution [Last Updated On: February 12th, 2017] [Originally Added On: February 12th, 2017]
- A primer on Darwin Day: Some religious groups embrace 'Theistic evolution' - LancasterOnline [Last Updated On: February 12th, 2017] [Originally Added On: February 12th, 2017]
- Pokmon Go Eevee evolution: How to evolve Eevee into Vaporeon, Jolteon and Flareon with new names - Eurogamer.net [Last Updated On: February 12th, 2017] [Originally Added On: February 12th, 2017]
- Evolution of baseball from power to speed has left SBs behind - Chicago Sun-Times [Last Updated On: February 12th, 2017] [Originally Added On: February 12th, 2017]
- More order with less judgment: An optimal theory of the evolution of cooperation - Science Daily [Last Updated On: February 12th, 2017] [Originally Added On: February 12th, 2017]
- How the horse can help us answer one of evolution's biggest questions - Raw Story [Last Updated On: February 12th, 2017] [Originally Added On: February 12th, 2017]
- China Is Now The World's Largest Producer of Solar Power ... - Collective Evolution [Last Updated On: February 13th, 2017] [Originally Added On: February 13th, 2017]
- Community Viewpoint: Evolution, like gravity, is much more than theory it is a fact - Kdminer [Last Updated On: February 13th, 2017] [Originally Added On: February 13th, 2017]
- See the Evolution of Movie Magic With Every Oscar Winner for ... - Gizmodo [Last Updated On: February 13th, 2017] [Originally Added On: February 13th, 2017]
- How the horse can help us answer one of evolution's biggest questions - Phys.Org [Last Updated On: February 13th, 2017] [Originally Added On: February 13th, 2017]
- How evolution alters biological invasions - Phys.Org [Last Updated On: February 13th, 2017] [Originally Added On: February 13th, 2017]
- The Evolution of Valentine's Day - Inside Science News Service [Last Updated On: February 14th, 2017] [Originally Added On: February 14th, 2017]
- Why evolution may be tech billionaires' biggest enemy - The Week Magazine [Last Updated On: February 14th, 2017] [Originally Added On: February 14th, 2017]
- Russell Westbrook is leading an evolution in NBA rebounding - Washington Post [Last Updated On: February 14th, 2017] [Originally Added On: February 14th, 2017]
- Eye Evolution: A Closer Look - Discovery Institute [Last Updated On: February 14th, 2017] [Originally Added On: February 14th, 2017]
- How evolution alters biological invasions -- ScienceDaily - Science Daily [Last Updated On: February 14th, 2017] [Originally Added On: February 14th, 2017]
- Evolution always wins: University of Idaho video game uses mutating aliens to teach science concepts - The Spokesman-Review [Last Updated On: February 14th, 2017] [Originally Added On: February 14th, 2017]
- Geneticists track the evolution of parenting - Phys.Org [Last Updated On: February 14th, 2017] [Originally Added On: February 14th, 2017]
- How this cockeyed squid shines a light on deep sea evolution - Christian Science Monitor [Last Updated On: February 14th, 2017] [Originally Added On: February 14th, 2017]
- Cockeyed squid shines light on deep sea evolution - Christian Science Monitor [Last Updated On: February 15th, 2017] [Originally Added On: February 15th, 2017]