Presentation Takeaways – Immersive Media in Entertainment

This year, SCAD Film Fest hosted a panel of experts in immersive media: Ravindra (Ravi) Velhal, global content strategist at Intel, Ethan Shaftel, VR producer/director, Ted Schilowitz, futurist at Paramount Pictures, Frank Radice, expert in residence at Defintion6, and Nancy Bennett, chief content officer at Two Bit Circus. Their talk is paraphrased below.

Frank: (To Ted) How did VR come to be, and where is it going?

Ted: It’s important to recognize that “virtual reality” as a format is not new. It’s been around roughly as long as traditional film and tv, it’s just been called other things like “creation of simulation”. It’s typically lived in areas like theme parks, where experiences can actually be built in scale and scope, so the tech surrounds all of your senses, including your physical body. What makes it different from traditional media is its spatiality. Over last decade, we’ve gained the ability to carry a theme park around with you and strap it to your face, which opens up new possibilities.

Frank: (To Nancy) You made music videos at MTV for a large part of your career. How is your process different between making videos for 2D media versus immersive reality?

Nancy: “The medium is the message” – the content commands what technology or process the creator should use to achieve it. When I think about making videos for VR, because it mostly depends on the funding and assignment I’ve been given, I start by asking what’s appropriate, what communicates the artistic statement. I look at the song structure, lyrics, motifs, themes, and emotions, and find a way to capture that which feels true to life, or to how we imagine or feel.

As for my pre-visualization thought process, I imagine having eyes in the back of my head, since VR completely surrounds you. I try to picture my place in the room around me – where am I, what am I seeing, what is my attention drawn to? I try to figure out the direction of the action around me.

Frank: (To Ethan) What is your process like as a director for VR?

Ethan: The term “director”, at least how we use it for traditional cinema, starts to lose its meaning in the immersive space. For film, directors use a massive, pre-existing visual language, where the use of certain equipment and techniques creates certain meanings and ideas. But in VR, we’re still building that technology and visual toolset – so a lot of being a director for VR is about experimenting to establish new ideas and techniques.

For projects like Kaiju Confidential, I look for insights in the themes and characters, the emotional ideas of the content. In Kaiju Confidential specifically, since the characters are two kaiju monsters having petty fights, I realized I could do a lot with scale; they’ve very big physically, but very small emotionally. So the visual choices I made have a lot to do with perspective and height.

Nancy: You also have to think from almost a “reverse” point of view. VR directors have to put themselves in the audience seat, since the end result is literally from the viewer’s perspective.

Ravi: Cinema has always communicated an emotional experience. New technology just expands the boundaries of how it can do that. So immersive media isn’t just VR, it can also involve surround-sound audio or even smell (like Le Musk); it just has to start with a good story. We’re also using some exciting new technology – volumetric capture means that whatever a person can see, a creator can capture since now they aren’t limited to a camera recording from a fixed, limited perspective.

Ted: Volumetric capture means that the action can live in actual depth. Cinema has always used magic tricks to make audience believe something is happening even though they know it’s just actors or code, but immersive reality means we can take that to next level.

Ravi: The Grease 40th anniversary project (video above) takes volumetric captured actors, and uses a game engine to translate the footage into a 3D AR experience that can be viewed from any perspective. The same technique is also being used for projects like Shark Tank, CNN, and the Super Bowl.

Frank: What’s the role of sound in immersive media?

Ethan: The more realistic the sound is in an experience, the more your brain fills in the gaps of an imperfect image. Even if the visuals are stylized, good audio gives a subconscious sense of reality. Audio is also a great tool for giving the viewer spatial direction; the director can use audio cues to direct the viewer’s attention, so the action moves through the frame in an elegant, narratively-justified way.

Frank: Can VR make money? Specifically, how does location-based immersive media fit into entertainment industry?

Nancy: 2 Bit Circus is a micro-amusement park with 40 thousand square feet of “immersive play”, which includes everything from VR to immersive escape rooms. Its mission is to bring people to bring people together to play elbow-to-elbow, and a lot of other companies have been doing similar things, like Dreamscape and The VOID. So in terms of the difference between location-based immersive reality and home, headset-based experiences, it’s up to the user’s choice of what they feel like doing. It also comes down to the content that’s available for those platforms.

Ravi: Location-based content can also be used to grab audience attention, so they can be drawn to related home-based content. There’s a VR experience for Spiderman: Far From Home that people love to do on-location, and they often end up downloading the full experience to play at home.

Audience: We’ve seen VR and AR integrated into theme parks more recently. Is this a symbiotic relationship, or is one trying to overtake the other?

Ted: Theme parks are always looking for new tools to improve their experiences. VR is a new tool that a lot of them are getting into, but it has obvious issues of scale and throughput since each headset can only accommodate one guest. My personal vision of the future of theme parks (about ten years out) is for everyone’s phone to double as an immersive media device, which can be activated in certain areas to play out curated, theatrical content. This could also make the whole world a theme park; imagine Savannah ghost tours with high quality AR!

Ethan: Theme parks are already participatory worlds which we experience in first person. We can think of VR as “cinema plus architecture”, capturing attention at specific parts, managing sightlines, etc. a lot like theme parks.

Ravi: There are lots of immersive reality-focused amusement parks in Asia, due to space and real estate limitations.

Audience: When we see movies in a theater, it’s a participatory experience because we view them as part of an audience. Will this experience be lost if we shift towards VR, which is more individual?

Nancy: There’s a lot of interest in VR at the enterprise (organizational/corporate) level, where the technology is being used for virtual collaboration. So pairing between multiple headsets is something that’s being worked on. Again though, the question comes down to what the user wants out of an experience.

Ethan: Live social platforms are being developed for VR; Michael Abrash at Facebook Reality Labs presented earlier this year (see video) on the possibility for a headset capturing the user’s facial and body data in real time, and recreating them as a moving, speaking avatar. VR doesn’t have to be an individual experience.

Audience: Where will immersive tech be in 5-10 years, since we’re starting to hit (theoretical) limits of computational power?

Nancy: Developments like 5G and quantum computing might mitigate some of the computational requirements of VR such as rendering.

Ravi: The displays we (Intel) are building now are capable of 8k resolution, so we don’t pay a lot of attention to theoretical limits. The best way to figure out the actual limits is to just go and build stuff.

Ethan: That being said, as an individual creator there’s practical issues of making a good, affordable thing within a short time frame. If you have limited resources, it’s important to find what’s cool and working now, then build an approach that fits that tech.

Ted: The question we should ask ourselves as creators is “What’s it going to take to make an immersive device as ubiquitous and usable in ten years as traditional screen devices are today?” Students especially should start building into the future, now.

Nancy: Study wins the race; more important than anything is just building and testing and experimenting, because what you make now will effect everything made from now on.