Specular studio is searching for the holy grail of virtual reality: combining live action footage within a real-time, interactive context. And they’re getting close.
Up to this point, most VR has been either “passive” (360-degree pre-rendered videos that the viewer watches without interaction) or “active” (fully interactive, game-like experiences). Like many things they do, Specular is attempting to blur those lines with their newest project, Blackout.
In order to pull off this Herculean feat, Specular is seeking support via Kickstarter as they continue production and refine their hardware-software system, DepthKit, which has made the entire project possible.
Talking to Specular is deeply inspiring. They are artists, technologists and filmmakers in equal measure. Our interview with them has something for everyone.
Q&A with Specular: Making Blackout
Before we get into the technical bits (and I definitely want to get into the technical bits), let’s talk big picture. Although the Kickstarter video explains it, can you tell us the story/premise for Blackout?
Blackout is a virtual reality film placing you on the New York City subway. When the train stalls and the power goes out, you are gifted with telepathic power to explore the inner thoughts of your fellow passengers.
Activated by your physical movement in the train, the story unfolds as a series of interconnected mental monologues. Blackout focuses on personal stories of times in the spotlight — setbacks and breakthroughs — giving a glimpse into the experiences that shaped the strangers’ self identities as performers on the public stage.
Where did that idea come from?
We were inspired by contact we have with crowds of people during daily urban life. Although we share so much time and space with them, many of these strangers we will never meet.
It seemed like the perfect premise for exploring through virtual reality. We wondered what it would feel like to interact with realistic representations of people without them directly responding to your presence.
What was the writing process like for this?
We best address this question by describing Blackout as a documentary in a fictional setting. Everyone of our passengers ‘plays’ themselves, telling unscripted stories from their memory in the first person.
It required a tricky and deliberate interview process that our collaborator Charlotte Kaufman pulled off beautifully. She describes this process in a conversation we had with her here. After their interview, each member entered our capture stage for a 3D scanned portrait to be placed on the train — as well as a few other surprises to be revealed upon the release of the film!
So with that approach, it’s less about writing and more about editing, right?
Blackout’s editor Hannah Jayanti took a documentarian approach to the interview material, sussing out emergent themes between all the strangers and composing them into emotional arcs.
During the experience of Blackout, each moment you are immersed in one of the themes. You may freely explore the streams of consciousness of the passengers around you through your gaze and movement. If you spend long enough with one person, the train will fade away, and you’ll find yourself transported into their memory.
Our goal is to allow you to discover a narrative rather than being forced to see a particular sequence.
Nice. It’s less about story “telling” and more about story discovery.
Blackout is a way for us to explore how virtual reality is distinct from linear video. We intend to find what is possible when creating content specifically for this new interface. With virtual reality, interactivity and first person perspective are inherent and Blackout embraces these aspects as the primary way to deliver the story.
We’ve chosen to collaborate with traditional documentary filmmakers for their sensibilities to take the experience beyond technological novelties. We’ve been drawing inspiration from Wim Wenders, Abbas Kiarostami and Frederick Wiseman in the effort to create a quiet, yet poetic experience with our strangers.
What role has life in the city played in general for Specular? How important is it to your creative practice?
New York is loud and expensive and there are so, so many people. Yet it’s vibrant, exciting and filled with millions of people speaking an estimated 800 different languages.
There’s a line that sums this up for us very well from author Sarah Schulman on Benjamin Walker’s Theory of Everything in his excellent New York after Rent Series. She says:
One of the things that I have learned about life being a New Yorker is that people have contradictions and people make mistakes and there’s nuances in behavior. […] Riding that subway gives you knowledge about the variation of emotionality in human beings. Urbanity teaches us every single second of every single day that other people are different and that knowledge is crucial for creating new ideas for the future.
It seems like every few years, we make a personal project about this process — pay tribute to the city that inspires and challenges our creative work. From past projects like Depth Editor Debug, to the tongue-and-cheek slyPhone commercial, we are often inspired to comment on the social and political dynamics in the shared public spaces of the city.
MTA train details
Is there a larger comment about cities you’re trying to make with Blackout?
Beyond our personal creative process, the story of Blackout has a global significance. Today, more than half of the world’s population lives in cities – a number that is estimated to grow to two out of three by 2050.
With global humanitarian crises unfolding daily, the realities of a continuous stream of strangers in cities is a critical one to understand. We hope that Blackout inspires us to acknowledge the social barriers that restrict us and to not be strangers in real life.
There’s a kind of humanitarian undercurrent to all this, then.
Mei-Ling spent the summer teaching yoga and meditation to incarcerated women on Rikers Island and was given an opportunity to better understand how some of those women ended up there.
As a society, it’s too easy for us to judge people without ever fully understanding them. That prejudice deeply affects one’s own self image and how we act.
The sense of compassion Mei-Ling felt in those moments was overwhelming and really made her question how she thinks about the city and the people who call it home. Our intention for Blackout is that it will be an experience that rewards people for their curiosity about others.
Field recording with Antfood’s VR audio rig
Specular is three people, and you each have a different focus or strength. Can you briefly tell us about each person’s perspective?
Specular is composed of three founding members, Mei-Ling Wong, Alexander Porter and James George. Our diverse backgrounds from film production, photography, and computer science give us unique strengths for creating documentary work that engages with new technology.
Mei-Ling Wong My background is in production for commercials. I’ve produced from pretty much every angle: agency, live action, post-producing and live interactive installations — and my role traditionally is as the main point of contact between our clients and internal team.
How is producing for commercials different than producing for VR?
Usually, the commercial production process is very linear and predictable. There’s a very clear path, an established visual language that allows for a clear set of checks and balances. One of the biggest challenges with a project like Blackout is that the entire creative process is nonlinear and experimental. There’s no clear or established path or process.
Additionally, with projects like Blackout, we’re only answerable to ourselves for how long things take and how the experience itself develops. A lot of the time, we’re creating the tools and the process as we go, which can be challenging for someone in my role.
What have you carried over from the commercial world?
I try to bring an eye for creating efficiencies in the process and an understanding of the kinds of challenges that artists in a production environment might be faced with — with the goal of enabling creative expression through the exploration of emerging technologies.
What about you, James? What’s your story?
James George Before founding Specular, I was working as an independent freelance creative coder and media artist. I come from a traditional computer science background but have dedicated that expertise to media art and film where the focus has been on the design or experience rather than the tech.
I see code as having the power to unlock new modes of expression in emerging technology and work to define projects that illustrate and excavate more empathetic aspects of recent capabilities. Since my focus has largely been on 3D graphics and computer vision technologies, the emergence of virtual reality has been an exciting community for me to explore and discover what’s possible.
And last but not least, Alexander?
Alexander Porter Far before Specular, my first love was the documentary photo essay. But since then I’ve studied media theory, worked in photography and as a DP for documentary films.
Alexander on set
After a media theory-fueled crisis of faith, I’ve expanded my view of what constitutes photography, and I’ve been working in this area known as ‘computational photography.’
What is “computational photography”?
Computational photography is the area of research that brought us things like the panorama, HDR, hyperlapse, etc. but increasingly includes 3D scanning and rendering — I consider myself a practitioner of it as a creative discipline.
Increasingly, I started working in this unusual role doing VFX for documentaries, such as an HBO documentary called Love Child, a role I really enjoy. James and I have been collaborating on various computer and camera projects since around 2011.
In my (mildly divisive) op-ed, “Virtual Reality is Not Filmmaking,” I argued that “true” VR is real-time. What I like about Blackout is that it basically short-circuits the arguments I put forth in that article by combining pre-recorded performances and real-time exploration. Is that how you see it?
We loved your article and the comments thread! It touches on one of the key aspects we are exploring with Blackout: How can we create a compelling live-action film in a video game environment that fully embraces the freedoms of real time 3D while retaining the qualities of live action?
A lot of 360 spherical video work is great for transporting us to real world places, allowing us to look around as things unfold. But when it comes to crafting surrealities, it’s a very inflexible medium. For this, we turn to game engines, but can’t abandon our past sensibilities in film and photography. Our DepthKit capture system fuses these two worlds, letting us put real performances into our virtual worlds.
!["[We] can’t abandon our past sensibilities in film and photography. Our DepthKit capture system fuses these two worlds, letting us put real performances into our virtual worlds."](https://s35726.pcdn.co/wp-content/uploads/2015/12/productshot_2.jpg)
“[We] can’t abandon our past sensibilities in film and photography. Our DepthKit capture system fuses these two worlds, letting us put real performances into our virtual worlds.”
Also, to trot out a tired McLuhanism, it’s not uncommon for a new medium to contain an older medium as its content, especially in the early days. So to us it makes sense that we’d be seeing video in virtual reality headsets right now.
One thing you’ve identified is that we are definitely into hybrids. Often the most interesting answer for us isn’t binary — spherical video vs video games, linear vs non-linear narratives — but an unlikely combination of the techniques available.
In the Kickstarter video, you use the term “volumetric filmmaking,” which is new to me. Can you explain what’s meant by that?
Volumetric filmmaking encompasses any photographic capture technique that creates a 3D representation of the scene as opposed to a 2D one.
Most people will have first encountered this type of imagery with the emergence of Microsoft Kinect hacking, and the resulting glut of 3D scanned point cloud themed music videos that ensued after its release. But there are also interesting ways of creating volumetric video using multiple video cameras to create a 3D scene with a technique called photogrammetry or ‘videogrammetry.’
Is volumetric filmmaking something being widely pursued by others — or is Specular alone in the wilderness?
We may be in the wilderness, but we’re not alone out here!
Volumetric video is a big topic in computational photography, and growing quickly because it is very relevant to augmented and virtual reality media production.
Photogrammetry software like Agisoft Photoscan and Autodesk’s 123D Catch are some of the most accessible ways of creating 3D models from photographs or videos. Microsoft demonstrated an elaborate volumetric video system this year along with several tech startups popping up with impressive demonstrations: 8i, Uncorporeal, and Mimesys to name a few.
But for now, public release of these seems far away, so we are excited to be building and sharing our own approachable tools as we create our projects.
Tell us a little more about DepthKit and how it was used in Blackout.
DepthKit is a hardware and software system for capturing, processing, and publishing 3D cinematic volume data.
The workflow has three stages:
First, you use DepthKit software and hardware on set to record data streams from multiple cinema cameras and depth cameras simultaneously. For Blackout, we used four Canon 1DCs and four Microsoft Xbox Kinect One sensors encircling a capture stage.
The recorded data is then processed by the DepthKit application and exported as self-contained sequences that can be easily added to a game engine like Unity for realtime playback on desktop, iOS/Android or web.
For the post-production peeps out there, we support export as 3D model formats compatible with Maya, After Effects, and the like as well!
We should say that Blackout is very much a work in progress, and we have some interesting plans on improving the ‘fidelity’ of our passengers.
By the way, if you back the Kickstarter at the $290 DepthKit DIY level, you will be among the first insiders to receive the beta version of this workflow for yourself.
How did you guys develop DepthKit?
DepthKit has an interesting history: the tool used to be called RGBDToolkit, and was originally prototyped when Alexander and James made their photo series in the subway.
In 2012, we released a free version for post-production effects which has been used in some great music videos, like Fitz and Tantrums’ Out of My League, Eminem’s Rap God and even Young Rival’s Magic Eye Music Video, plus countless other experimental videos.
While DepthKit creates nice effects for pre-rendered images and videos, we knew the future of this technique is part of a new medium of interactive. In 2013, James’ co-directed an interactive documentary called CLOUDS with filmmaker Jonathan Minard, which premiered at Sundance Film Festival using the Oculus Rift as the interface.
Still from CLOUDS
The documentary profiles the creative coding community behind the tools OpenFrameworks and Processing. Built entirely with code, the documentary was the first time DepthKit capture was used in an interactive story.
Since then, we’ve have been focused on creating DepthKit as a system for real time and usability. There are big updates planned in the future.
Lytro recently announced its “Immerge” system, which uses a fixed-point 360 camera to capture the entire light-field available. In a way, it seems like the perfect complement to the system you guys have developed, which encircles actors to capture their performance.
We’re incredibly excited about what Lytro is pioneering, and believe that down the line, lightfields could drive one of the biggest shift in photography since the move from film to digital imaging.
Lytro’s system may in fact be complementary to DepthKit, and can be viewed as an extreme version of what we are approximating using depth cameras today. However, without breaking Moore’s law and a few laws of physics, the sheer amount of data and physical size of Lytro’s system will keep it out of the hands of indies for the next several years.
What’s next for Specular? Are you guys pointing yourselves more in the direction of technological development or creative development? (Or both simultaneously?)
Mei-Ling I would say both. I don’t think any of us sees a future for Specular where our practice is not equally grounded in both technical and creative development.
Creative expression drives our technical invention, this approach has played a huge role in our process thus far. It’s a constant juggling act, but it’s key to how we see ourselves growing as a company.
We’re interested in working collaboratively with other artists in an effort to satisfy our desire to create and do more with less. Engaging with a range of creative communities really inspires us, and we want to continue to develop tools for enabling creative expression.
James We’re really looking forward to showing the world Blackout, and continuing to tame this patch of wilderness for the next generation of artists and filmmakers.