Which New Areas of Virtual Reality Will Use Eye-Tracking? – Slate Magazine (blog)

Chinese twins wear VR headsets as they ride in a roller coaster simulator at the Wantong VR Park on Nov. 27 in Beijing.

Kevin Frayer/Getty Images

This question originally appeared on Quora, the place to gain and share knowledge, empowering people to learn from others and better understand the world. You can follow Quora on Twitter, Facebook, and Google Plus.

Answer by Kynan Eng, research group leader, neuroscience of VR and applications to rehabilitation:

I think that most of the first uses of augmented realityand virtual realityeye-tracking will be to improve general AR/VR headset comfort and usability. Why? As a direct input device, eye-tracking is actually fairly frustrating and useless. However, as a general contextual signal indicating possible user intent or attentional focus, it is quite useful. Many use cases for eye-tracking will work in the background, and will probably include the following:

Graphics rendering resource allocation: If a person is looking somewhere, more graphics rendering resources can be allocated in that general direction. This can provide better quality output for a given amount of rendering power.

Data prefetch: Some VR data operations require time to complete, e.g. looking up something in an online database. If a person glances in a particular direction, data fetching can begin in the background even before the person selects an item to interact with. This improves the perceived responsiveness of the VR environment, which can be especially useful e.g. over mobile data networks.

Multimodal smart 3-D object selection: In VR, pointing at a small object in a cluttered environment can be quite difficult. Eye-tracking can be used to help disambiguate the object that the user intends to select by combining the information with the controller input.

Automatic headset calibration: A headset that knows where the users eyes are can better adjust its own image output parameters for optimal user comfort.

Balance manipulation: The vestibulo-ocular reflex is a well-known automatic effect linking eye movements to changes in the vestibular system. Knowing eye movements as well as those of the headset (via accelerometers) allow deductions about the likely state of the users vestibular system, and thus enables systematic manipulations to heighten changes in balance or possibly to reduce the effects of motion sickness during VR use.

What all of these use cases have in common is that, when working well, you dont notice that they are doing anything. In fact, I would go as far as to guess that some of these eye tracking use cases are essential for enabling truly useable VR/AR for mass-market applications.