If youve played a recent Forza or Gran Turismo video game, you already know: These virtual cars are almost indistinguishable from their real life counterparts, with all the curves and light reflections that make them look like theyve driven straight out of a car commercial and into the game. But unlike films made by companies like Pixar, these game cars arent rendered over the course of months. They render 60 times every second.

Now Epic Gamesa company known for making the Unreal Engine which powers many big budget video game on the marketand VFX studio The Mill are showing just how far the realtime rendering of photorealistic graphics can go. In their new short called The Human Race, you can actually choose the car you want to star in a short Chevy commercial, then watch as it instantly appears within the video.

The end product is basically a choose-your-own-adventure The Fast & The Furious short. And technology like this is about to change the way movies, games, and everything in between are made forever.

"We created a virtual production toolkit to visualize what you see in the filma virtual car," says Boo Wong, global director of emerging technology at The Mill. "But that can be extended to any character, prop, etc. From a visual effects point of view, thats super exciting."

To film the demo, the Mill used a pseudo-car called the Blackbird. The Blackbird is basically a physical placeholder for a CGI car that will be added in post-production. (Thats right! Many cars you see in car commercials are fake!) This vehicle is like a dune buggy, fit with visually trackable markers and filled with 4K RED cameras that shoot outward. Usually, the Mill's team shoots a commercial with the Blackbird because they need to film the spot before a cars final design is readyor because the car is so secretive they dont want to publicize it. But the ensuing post-production, in which the CGI car is added to real footage, takes months. Single frames can take hours, even days, to render.

Enter Epic, on the software end. Its new technology is called Project Raven. An extension of its Unreal Engine 4 used in video games, Raven has been customized to support augmented reality applicationsin fact, the platform is built to support systems like Googles Project Tango.

With Project Raven connected to the Blackbird, visual insanity ensues. The Unreal Engine gets all this real-time information from the real environment simultaneously. Footage from those 4K cameras onboard the Blackbird is mapped onto the curves of the CGI car, rendering a super-realistic reflective shell on its surface. Software analyzes this footage, too, spots the sun, and infers where its position must be in the sky, creating a realistic lighting system.

On set, the director can look through a preview monitor to see the dune buggy prop car re-skinned in real time as the CGI car from any conceivable angle. Of course, preview systems like this exist in the special effects world already, but heres the twist: The director sees the final, photorealistic pixels that will be in the actual commercial, rendered at 24 frames per second. (Sure, thats a bit slower than a high-end video game, but it meets the traditional benchmark of Hollywood films just fine.)

So where does this technology go next? As weve pointed out here on Co.Design, the Unreal Engine is evolving, making its way into more mainstream entertainment. Nickelodeon shows now use Unreal to produce TV programming with a quicker turnaround. And the latest Star Wars movie, Rogue One, actually used the Unreal Engine to render some shots of the movie's K-2SO droid.

For agencies like the Mill, this technology means it can shoot something for a client once, but easily repurpose it for multiple platforms and campaigns. In a world in which any asset within a film can be swapped out, it adds the immediate capability of customization and interactivity. Unreal Engine has also pledged to support Pixars Universal Scene Descriptions, meaning its verging toward compatibility with the Hollywood 3D machine.

"Were a games company. What we see happening is the gamification of everything in our lives," says Kim Libreri, CTO at Epic Games. "If youre watching an animated TV show, why shouldn't you be able to change the costume on princess, or change the location, have a personalized experienceand share your version with friends?"

Or, as I like to imagine: One day soon, you'll pause the NBA game youre watching, reach for the Xbox controller, and take over where the real players left off.

Read the original post:

This Crazy New Technology Transforms Movies Into Video Games - Co.Design (blog)