Virtual reality is not filmmaking

Let’s agree on something: 360-degree video presented inside a virtual reality headset isn’t really virtual reality. It’s a novel, sometimes powerful experience, but it’s not what makes VR arguably the most exciting breakthrough in storytelling since moving images.

While much has been made of immersion as a defining trait of virtual reality, a key component of normal (i.e. non-virtual) reality is painfully absent in the live-action and pre-rendered experiences I see being confused with true VR: interactivity.

In real life, you make decisions. Those decisions change the world around you — or your experience of it — sometimes in profound, unexpected ways. Real life decisions aren’t presented to you as tidy forks in the road. Real life decisions are messy. You often don’t realize you’ve even made a decision until you’re reeling from its consequences.

Real life decisions emerge from continuous behavior within an environment — not discrete, binary choices.

The Oculus Rift announced Oculus Touch controllers at E3 2015

The Oculus Rift announced Oculus Touch controllers at E3 2015

Virtual reality, then, should strive to bring that same level of interaction to a user. Doing so implies (at least) the following:

1. Virtual reality is not filmmaking.

Whenever a new medium comes along, we spend years trying to cram old media into it. Jesse Schell, who’s been working with virtual reality since the first wave of VR in the 1990s, puts it nicely in his Gamasutra post:

Throughout the history of entertainment, the first impulse of those who create in a new medium is to imitate what came before. Early movies were just filmed stage plays, with no cuts, close ups, or camera motion. Early internet videos were attempts to imitate the format of television (30 and 60 minute shows).

In all cases the pattern is the same — the new medium is derided for not being as good as the old one, but then, gradually, after many experiments by pioneers, the unique strengths of each medium come to light, showing it to be powerful in a way that was not possible before.

An LA Times article covering the release of Oculus Story Studio’s latest VR project, Henry, voices the concerns of some traditional filmmakers:

As VR cinema skeptics have noted, [interactivity] can undermine big story moments. No filmmaker wants to have Michael Corleone shoot Sollozzo and McCluskey in a restaurant while an audience member is in the kitchen ogling the appetizers.

This “argument” is simply a failure to understand that VR is not a new vehicle for old experiences (i.e. films) — it is a new medium entirely. Will you have the same emotional reaction to a VR retelling of The Godfather that the original film delivered? Probably not. Can you have a different — possibly even more profound — emotional reaction to some other VR experience? Absolutely.

As Ramiro Lopez Dau, the director of Henry, says later in the same LA Times article:

“What I’ve learned with ‘Henry’ is that it’s not really a movie anymore. It’s storytelling in a new medium, and we’re trying to figure out what to call it,” he said. “A movie is the closest name because it’s the most familiar. But if we keep adding layers of complexity, we’ll come up with a new one.”

2. Virtual reality is real-time.

In order to have a truly VR experience, the virtual world must be rendered in real-time.

360° pre-rendered experiences — whether live action or CG — presented in virtual reality headsets are enjoying a spike of novelty now, but once VR becomes more widespread, that novelty will quickly fade. Rapidly improving real-time technologies will make “live action VR” feel comparatively dull, even lifeless.

Don’t buy it?

Imagine this simple scenario:

You’re standing in a concrete cell lit only by feeble light from cracks in the cast iron door before you. You tug on the door’s handle, but it doesn’t budge. Locked tight.

You hear a chirping sound coming from the corner of the room. It seems to be emanating from under a small table there — really just an overturned wooden crate.

You kneel down and peer into the crate. Something glints in the shadows. On your hands and knees, you crawl closer, peering into the darkness. You can just make out the twitching whiskers of a mangy rat. As it turns to face you, you see a golden key clamped firmly in its mouth.

Slowly, you reach out and lift the crate, exposing the rodent. To your surprise, it doesn’t scurry away. Instead, it crawls eagerly toward you, gingerly droping the key in your other hand.

Freedom!

The mix of fear, wonder and ultimately exhilaration that such an experience could create would be a direct result of being able to explore your surroundings in real time. In a passive version of the experience, you would enjoy the same level of sensory immersion, but without being able to crouch down, move toward an object and interact with it, you would be a catatonic observer, locked in a paralytic trance.

You wouldn’t feel a sense of ownership over the outcome of the situation because your actions would be limited to simply rotating in space and “gazing” at things. While this gazing activity could conceivably trigger a branching narrative, the experience would be “on rails,” a finite set of vignettes that shatters any sense of agency in the virtual world and instead forces a narrative upon you.

Here’s where game developers have a massive advantage over traditional filmmakers. They’ve been crafting experiences in real-time engines like Unreal and Unity (and many others) for decades. Real-time gaming platforms provide all the graphical and interactive scaffolding you need to create interactive worlds — and immersive storytelling experiences.

That’s not to say that VR storytelling and gaming are the same. They’re not. But from a purely technical perspective, game-making is far ahead of filmmaking in understanding the challenges — and potential solutions — that face VR storytelling.

“So pre-rendered VR is a waste of time?”

No, not at all.

But live action and pre-rendered storytelling are inherently limited to a set of experiences that I would call “passive VR.” If you’ve enjoyed any of Chris Milk’s VR works, you know that these experiences can be quite powerful. (My personal favorite is “Evolution of Verse,” created in partnership with VFX powerhouse Digital Domain. Download the VRSE app to watch the film.)

"Evolution of Verse," a stunning passive VR film directed by Chris Milk with Digital Domain

“Evolution of Verse,” a stunning passive VR film directed by Chris Milk with Digital Domain

Passive VR is pushing the boundaries of what can be achieved without interactivity — and it appears that these boundaries reach farther than most imagined. It’s important, however, to distinguish between passive VR and true VR.

Why? Because from both technical and creative standpoints, true VR is much, much more difficult than passive VR. Equating passive VR and true VR shortchanges the exciting challenges ahead for the latter — and the attendant discoveries that will be made along the way.

3. Virtual reality means giving up control.

When it comes to VR as a storytelling medium, one of the emerging masters is former Pixar animator and director Saschka Unseld.

Saschka Unseld presenting lessons learned while making Lost at the Tribeca Film Festival 2015.

Saschka Unseld presenting lessons learned while making Lost at the Tribeca Film Festival 2015.

Unseld now works with Oculus Story Studio, where he directs short-form interactive experiences like Lost, which debuted at this year’s Tribeca Film Festival to rave reviews. In a recent blog post (required reading for anyone interested in VR storytelling), Unseld explains his team’s gradual acceptance that trying to control the user’s experience was holding them back:

One of the most powerful aspects of film is the total control over the shot. We talked endlessly about how to regain that control. How could we make sure the viewer always looks in the correct direction?

We tried guiding the audience’s view through audio cues. We had a bird fly by the viewer to capture their attention and guide their gaze towards a point in the scene. We also tried to design the set in a way that guides the viewer’s gaze to the right areas.

However, each time we implemented one of these dictatorial tools too heavy handedly, the storytelling started to feel forced, staged, and artificial.

Over time we stepped away from this sort of thinking. To embrace VR as its own unique medium, we have to let go of our almighty control of what the audience sees. Instead of instantly pushing the story onto the viewer, we take a step back for a while and let the viewer take part in discovering the story themselves.

The impulse to “tell” a story is loaded with authorial intent to control the user’s experience. Telling, after all, is a mostly one-way experience. But good storytellers respond to their audiences, skipping details or embellishing others in order to achieve the greatest emotional impact. They read the room and making adjustments to their material on the fly.

Screenshot from The Stanley Parable, a labyrinthine game that challenges what a story can be.

Screenshot from The Stanley Parable, a labyrinthine game that challenges what a story can be.

Perhaps “telling” a story is the wrong starting point altogether.

Recent games like The Stanley Parable and Gone Home present their narratives through a series of discoveries. They aren’t telling stories so much as presenting all the pieces of a story and letting the user assemble them however they wish. While there is a master narrative, the specific experience of it is unique for each user.

Tags: , , ,

About the author

Justin Cone

/ justincone.com
Together with Carlos El Asmar, Justin co-founded Motionographer, F5 and The Motion Awards. He currently lives in Austin, Texas with is wife, son and fluffball of a dog. Before taking on Motionographer full-time, Justin worked in various capacities at Psyop, NBC-Universal, Apple, Adobe and SCAD.

10 Comments

Jamil

“Perhaps “telling” a story is the wrong starting point altogether.” AMEN!

layne braunstein (@laynardo)

Thank you thank you for this article. I am getting really tired of big post studios starting their “VR” department, and having zero understanding of what true VR is. People always ask me if VR is another fad, and I answer, “your passive VR is a fad, real VR is only just beginning”. We have been silently working on a true VR experience at fake love for along time now, within Unreal; and I can promise you, it’s super hard work.

IMO, if someone really wants to be “in” VR, you have to understand gaming and be a gamer on some level. If you aren’t, you are going to be steamrolled over by the true VR storytellers emerging.

M Fahd Chishty

Yup! VR has quite a ways to go yet…even though i haven’t worked directly in VR (only Augmented Reality) but i agree with your statement that you do need to understand gaming in order to pull off the best experience for user

Joash Berkeley

Excellent point on distinguishing VR based on how we don’t realize we’ve made a decision until the consequences are reaped.

Clyde DeSouza (@cly3d)

>>Real life decisions are messy. You often don’t realize you’ve even made a decision until you’re reeling from its consequences. Real life decisions emerge from continuous behavior within an environment — not discrete, binary choices.

>>Virtual reality, then, should strive to bring that same level of interaction to a user.

I find this essay interesting yet contradictory.. (the author contradicts the arguments made in the first half with this: ““So pre-rendered VR is a waste of time?”
No, not at all.
But live action and pre-rendered storytelling are inherently limited to a set of experiences that I would call “passive VR.” If you’ve enjoyed any of Chris Milk’s VR works, you know that these experiences can be quite powerful.”…)
——

So which is it? There is no such thing as “passive VR” Virtual Reality is what it literally means. A Virtual Reality world.

Going by the “passive” argument… would we say that any period of inactivity (interactivity) by a person or brain in the real world would mean that person has slipped into “passive” reality in the real world?

(Imho) VR is VR… in some VR experiences you will ‘lean forward’ and make decisions in the VR world. In other VR experiences you’ll just sit back and let a story unfold.

For example. Would the movie Avatar be considered a passive VR experience? Forget for a moment that the film is even being viewed on an HMD.
In a cinema, were not the audiences immersed in the highly detailed world of Pandora? Did they not feel emotionally connected with the people of that world or the message of the film?

Were the audiences making active decisions in the movie? Imagine how much more powerful the suspension-of-disbelief would be if this movie with it’s digital assets were converted for a VR HMD.

Would we demote the experience as being passive knowing that our senses (and reflexes) were fully engaged?

From the days of gathering around a bon-fire…to cinema and now VR… humans always like to be ‘passive’ when being told a story, allowing themselves be drawn into the story’s world. They don’t want to *have* to physically interact or make decisions in a virtual world, every time.

…therein, perhaps, lies the key to what Virtual Reality film-making is about.

aaronmspence

This is a great article, but for my money the True Virtual Reality you describe sounds a lot more like gaming than anything else. It reminds me of the beautifully rendered scenes of games like Myst, brought into the modern VR age.

When I want to watch something, I don’t want to be crawling around on the floor looking for rats with keys in their mouth. I want to be passively entertained, with curated content. ie show me the best bits, don’t dump it all in front of me and make me use time/energy trying to figure it out cause I’ll just turn it off. When I want that experience, I’ll play games.

I think the market for the active 3D generated real time stories is a little wider than the gamer market, but I don’t see it stretching into the TV/Movie going market place. There’s a massive difference between active and passive entertainment, and an equivalent different in the users of either media.

The other item that’s being taken for granted is that live action = 3D graphics and no one cares one way or the other. I think this is wrong, as this is to say that 3D animated movies are the same as live action movies. They’re not the same, and the audience knows it.

I have little interest in looking at a CG rendering of a real place in VR, it’s not real. I much prefer to see the actual place in live action. Maybe one day real time CG will be photo realistic to the point that no one can tell the difference, but even in that day I think people will still care to see REAL content of real places rather than CG only content.

Randall Rudd

After nine grueling months, my group has created a ‘proof of concept’ movie-like experience in what we call “Immersion” (VR movies). The concept we proved is that one can successfully tell narrative stories in immersion. We learned this by experimenting with film techniques in VR, (zooms, dissolves, etc.) and discovered first-hand which ones work — or don’t work, along with discovering an entire new phenomenon we call “Empathic Immersion,” which changes everything. There exists a massive difference between theorists who suppose and the guys working on the front lines actually forging this new medium of immersion. Don’t believe what you read, unless the author has a reel to back his theories. BTW, StoryStudios makes VR cartoons. Do not confuse this with narrative VR movies or immersions. Whole different experience.

Clyde DeSouza (@cly3d)

>Don’t believe what you read, unless the author has a reel to back his theories. BTW, StoryStudios makes VR cartoons. Do not confuse this with narrative VR movies or immersions. Whole different experience.

Well put.
I’ve been working with film techniques in VR. Very exciting space to be in.
teaser: https://www.youtube.com/watch?v=D-9VtiDi1Vk (in 2D until Youtube upgrades to Stereo360)

M Fahd Chishty

Nice work there CLye. I think you have grasped the essence of VR and i think that essence is the only thing that you need to get while trying to understand and then utilize some technology.
Wish to see an even improved version of ur work in the future.
Most probably in an Oculus Rift :P

Sean

Intriguing article Justin! I love gaming and storytelling. It is my opinion that interactivity is the antithesis of storytelling.

I have yet to experience a game in which I felt the interactivity improved the impact of the storyline. I just finished playing The Stanley Parable and I believe it further proves this point. Story within games generally works by tricking the user into following certain paths (or stories). However, I am interested in experiencing Oculus’ Henry to decide if it can enhance the empathy you feel for a character. Since character and story are so closely linked.

Comments are closed.