Evaluating the Impact of Augmented and Virtual Reality – aam-us.org

The dazzling effects of augmented and virtual reality are obvious, but how can museums know if they actually facilitate learning? Photo credit: Courtesy of Chicago History Museum

I am perfectly astonished at the effect.

The thrilling scenes therein depicted are beyond description.

It is true and really wonderful.

These are reactions from visitors to the Chicago Fire Cyclorama, who, in 1892, were captivated by the sensation of standing in the center of a rotunda, surrounded on all sides by a detailed painting of a burning city. Perched atop slowly spinning platforms, audiences got a 360-degree view and a narrated tour of the already historic 1871 Chicago Fire. Two tons of oil paint, six tons of canvas, and twenty person-years of labor resulted in scenes so massive and so realistically rendered that crowds delighted at the feeling of being transported back in time. Cycloramas were as close to virtual reality as you could get in 1892, and the novelty left an impression.

Apart from the quaint turns of phrase, the sentiments from the Chicago Fire Cyclorama visitors are strikingly similar to those offered by recent users of the Chicago History Museums virtual and augmented reality project, Chicago00 (Chicago zero zero), which we developed as a new kind of museum exhibition: mobile, immersive, and driven by the choices of the user. The project elevates our mission to share Chicagos stories by activating them on the digital landscape.

It felt like I could reach out and touch everything around me.

The experience took me back in time.

I loved it, magnificent view into what it was like back then.

Like the creators of cycloramas, the team behind CHMs experimental digital history initiative aims to astonish. Only instead of tons of paint and twenty thousand square feet of canvas, we leverage new technology, images from CHMs extensive Prints and Photographs Collection, and other media archives to connect users with pivotal events in Chicagos past. Through augmented reality apps and virtual reality videos, the project superimposes historical imagery onto the contemporary landscape. Stunning visuals accompany ambient sound, music, and narration to accomplish immersive storytelling that grounds users in the history all around them. At the edge of the Chicago River between Clark and LaSalle Streets, for example, smartphones become portals to the past, resurrecting ghostly scenes from when the S.S. Eastland capsized on that very spot over a century ago. The AR is a moving experience for users, inducing gasps and exclamations that satisfy both the historians and the creative team behind the experience.

But Chicago00 was always supposed to be more than a spectacle. It was meant to take the simulated reality that has been delighting gamers and amusement park goers for years and apply it to learning and connecting with history in new ways. As a history museum, our goal from the outset was to explore digital storytelling as a means to deliver on our mission. Our partnership with filmmaker and professor Geoffrey Alan Rhodes, which began in 2013, has pushed us to take the stories at our fingertipsthe 1929 St. Valentines Day Massacre, the 1933 Century of Progress Worlds Fair, the 1968 Democratic National Convention protestsand revitalize them with new interpretation and innovative presentation for an expanded audience. Designed as a series of experiments, each 00 initiative has been an evolution, allowing us to test, iterate, and refine. When the project was honored with a MUSE award from AAM in 2018, CHM and our partners were thrilled with how far we had come and how much we had learned.

The only thing was, even as we watched the people engaging with our VR and AR experiences gasp, we could not tell you for certain if they were learning anything about history. Museum AR and VR is relatively new, and very little has been developed and shared on evaluating the effectiveness of these formats in achieving experience goals for audiences. Analytics tell a nebulous story of views, clicks, and likes, but its difficult to draw meaningful conclusions from these numbers. According to museum evaluator Kate Haley Goldman, Museums lack tools, scales, and even a vocabulary for measuring immersion, presence, and empathy, and how those elements might contribute to learning outcomes. With the support of a National Endowment for the Humanities Digital Projects for the Public grant, CHM set out to change that. We used the development of our latest VR experience, Chicago00: 1893 Worlds Columbian Exposition, to create and test an evaluation tool to measure the cognitive and affective impact of these emerging digital experiences.

Working alongside Haley Goldman and her team at HG&Co, we embarked on a process to help us understand how users engage with digital history experiences, what impressions they had of our experience, and what was sinking in, with the goal of integrating this user testing into future design development.

As a test case, HG&Co evaluators observed about fifty participants trying out a previous Chicago00 VR project, Century of Progress, a vivid portrayal of the 1933 Worlds Fair that culminates with a simulated trip up the 628-foot Sky Ride to get a birds eye view of the city. Afterwards, they interviewed the participantswho included students on a field trip, museum educators, and general museum visitorsabout the experience.

While participants were inside the headset, the evaluators observed them pointing, exclaiming, and even grimacing, as they were brought high up in the Sky Ride, then cajoling friends and family members to try it out next. The wow factor was in full effect. The ensuing interviews confirmed the experience achieved powerful sensory immersion for the participantsso much so that, in some cases, they had trouble retaining or reflecting on the ideas being communicated. At times, they even actively tuned out the audio content in order to focus on being there. We learned that multiple viewings were warranted to fully grasp the historical information, which supported our approach to make the experiences free, easily accessible, and appealing to return to.

The disconnect between the sensory elements and the narration was a key finding for us. We also discovered that:

These findingssome surprising and some expectedbecame a formative study for our new Worlds Columbian Exposition VR. We considered the complexity and pacing of the narration, the then-and-now images working together to say history happened here, and the drama of the moving Ferris wheel. Realizing that the experience would prompt curiosity about the 1893 fair, we developed a web portal with mapped imagery and interpretive content. Available now, Chicago 00: 1893 Worlds Columbian Exposition is also the first of our digital history projects to be thoroughly evaluated with our new Experience Evaluation Rubric.

Based on the user testing for Century of Progress, HG&Co developed a framework for impact measurement for museum VR/AR called the Experience Evaluation Rubric (EER). The EER was designed to build a shared vocabulary and specificity around mixed reality experiences. At CHM, we use it to assess and inform the episodes we create of historical events in specific physical locations, but the tool is easily applied to other types of museum-based content and a range of experience goals.

The EER has two sections. The first is for the designer or evaluator to complete, and it captures details about the physical and digital environment of the experience and the content presented. The second is a participant-facing assessment that asks users to score twenty or so statements related to presence factors as well as cognitive and affective impacts. Using a scale of 1 (not at all) to 7 (completely), audiences rate qualities ranging from vividness and captivation to time-travel and physicality of place. These latter two elements were of particular interest to our history-driven, location-based project, but other museums might place more emphasis on traits related to agency or social presence, for example, depending on their intended effects.

Over time, data from the two sections of the EER can be analyzed for emergent patterns, such as what type and context of experience provides what type of impact. Museum digital producers can then make informed decisions about the kinds of content, technology, and settings that best support a desired outcome. For example, after viewing our Worlds Columbian Exposition episode, 81 percent of test subjects indicated they wanted to visit the Museums Worlds Fair exhibition content, which demonstrates VRs potential to support our on-site audience development goal. We also discovered that 87 percent of test subjects were very interested in the details within the virtual world, which indicates that the design of the experience succeeds at piquing curiosity about historical visuals. Finally, over 80 percent reported learning new facts, affirming the educational effectiveness we sought as we designed the experience.

As the Chicago History Museum works toward its commitment to becoming a digital-first museum, projects like Chicago00 are at the top of our strategic agenda. With support from the National Endowment for the Humanities, which encourages new, technology-driven interpretive methods for museums and widespread sharing of resulting products and tools, we have been able to meaningfully advance this goal. We now have evidence of the efficacy and appeal of the mixed-reality work we are developing through the Chicago00 project and a pathway for designing future experiences with our desired outcomes in mind. Our Experience Evaluation Rubric offers clear guidance for how all museums experimenting with augmented and virtual reality can understand user engagement and deliver meaningful content.

Immersive storytelling has come a long way since the peak days of cyclorama exhibitions. Worlds displayed on building-size paintings have shrunk to fit inside screens and headsets, but the potential is bigger and the reach is farther than ever before. We see AR and VR as tools that will greatly influence the future of our museumand probably yours, too.

Colleen McGaughey is the senior manager of institutional advancement at the Chicago History Museum, where she oversees government, foundation, and corporate grants. During her more than 20-year career in nonprofit development, she has played a key role in diverse grant-funded initiatives ranging from HIV prevention and hunger relief to adult literacy instruction and museum exhibitions. She has authored hundreds of proposals grounded in research and driven by mission. Her practice involves program planning and evaluation strategies as essential phases of fundraising. She has a BA in philosophy from Loyola University Chicago and an MA in liberal studies from Northwestern University and insists both prove useful in her work.

John Russick is the senior vice president of the Chicago History Museum where he oversees the departments of collections and curatorial affairs, exhibitions, communications, and the Museums Research Center. His latest digital initiative, the Chicago 00 Project, is a collection of interactive augmented and virtual reality experiences that showcase the Museums film, photo, and sound archive, winning a MUSE award in 2018 and a Chicago Innovation award in 2019. His most recent publication, "The Museum Inside Out" was published in the January/February 2020 issue of the American Alliance of Museum's magazine. Other publications include, "A Place for Everything: Museum Collections, Technology, and the Power of Place,(Medium, 2014) and "Connecting Kids to History with Museum Exhibitions" (Left Coast Press, 2010). He served as a program advisor on the 2011 Florentine Films documentary, Prohibition, directed by Ken Burns and Lynn Novick. Since 2008, he has organized the American Alliance of Museums annual Excellence in Exhibition Label Writing Competition.

Read the original:

Evaluating the Impact of Augmented and Virtual Reality - aam-us.org