Lights, camera, virtual action!
Filmmaking is an art that constantly evolves, and the rise of Virtual Production (VP) is not only changing the industry but also our approach to workflows and production pipelines.
New Virtual Production (VP) techniques completely reshape the dynamic relationship between pre-production and post-production.
As innovative pre-production processes emerge, it’s crucial for filmmakers to stay informed and adapt to changes that will elevate their films to new heights.
These new processes also bleed into post-production, having a profound overall creative impact. This article will equip you with the knowledge and insights to empower you and take your filmmaking endeavors to the next level.
Written by Rene Amador and Ryan Arms from ARwall.
The importance of pre-production
Pre-production has always been a critical phase in filmmaking, where ideas take shape and projects are visualized under the director and dept head’s visions for the first time.
Traditionally, visual effects (VFX) were added in post-production as an afterthought, but with VP & CGI, virtual elements are now seamlessly integrated into the pre-production process.
This can happen in various ways: the writer and director consulting with the VP team on sequences, pitch visualization to get more buy-in from stakeholders for key sequences, previsualization, technical visualization to plan equipment and cast blocking, and, of course, the actual real-time environment creation. More recently, this process may also include the development of XR and AI toolsets and models.
Early testing of in-camera XR effects using a TV and art mannequin, featured at VRLA 2017.
This shift is having a profound impact on the entire filmmaking journey. If there was any doubt before, there is none now: pre-production plays a critical and pivotal role in conceptualizing the show, as well as laying the groundwork for immersive and visually stunning worlds.
Integrating digital effects enables the seamless blending of practical and virtual elements, creating a rich and cohesive visual experience for the audience.
From an artist’s perspective, designing digital sets and characters for real-time visualization and virtual scouting brings opportunities for innovation and experimentation.
Getting buy-in early for off-the-wall concepts is a simple achievement when the idea lives and breathes in the context of an entire world’s story. It sets the stage for a smoother production process and reduces the reliance on extensive design occurring in post-production.
Our recommendation to filmmakers at this stage will always be to get comfortable with the tech at home, so we built home versions of our ARFX software for that purpose, replicating the same way we learned on a 4KTV.
Early testing of in-camera XR effects using a TV and art mannequin, featured at VRLA 2017.
ARwall’s first explainer video released in April 2018, showing the first tests of in-camera effects, which resulted in ARwall winning the patent for the invention of in-camera XR filmmaking.
Picture this: Real-time visualization
In Virtual Production (VP), real-time rendering stands as one of the most revolutionary advancements, delivering 24, 30, 60, or more frames per second.
It bestows upon filmmakers a magic wand, enabling them to visualize virtual environments, characters, and props in real time, granting immediate adjustments and iterations in moments instead of hours or days.
However, part of the enchantment lies in the fact that, like filmmaking of the golden era, all of this building and design must be prepared before the cameras start rolling.
By utilizing the power of game engines like Unreal Engine, harnessing the capabilities of 3D modeling software, and employing tracked camera movements, artists can weave in captivating digital imagery with their live-action talent, breathing life into props and worlds that will captivate audiences on the screen.
Setting lighting, explosive effects and a rotation all in scene allowed for a complete shot during filming of the Crossfire Sierra Squad trailer. A simple in-engine menu allows for real-time color and lighting adjustments; in this case, it’s ARFX Pro Plugin.
Setting lighting, explosive effects and a rotation all in-scene allowed for a complete shot during filming of the Crossfire Sierra Squad trailer. A simple in-engine menu allows for real-time color and lighting adjustments; in this case, it’s ARFX Pro Plugin.
Enter the realm of virtual scouting
Within the visualization process, virtual scouting emerges as a game-changer. Filmmakers can harness the power of virtual cameras and digital props to create immersive digital locations based on proposed sets.
This unlocks a world of boundless possibilities, allowing for the exploration and evaluation of diverse visual options, even before physical production commences.
Imagine the freedom of making informed decisions about set designs, camera angles, and overall aesthetics without being confined by physical limitations.
Virtual scouting makes that possible, enabling you to push the boundaries of creativity and shape your vision to create captivating visual experiences from the project’s inception.
Checking the effects systems and angle prior to shooting.
The Virtual Art Department (VAD): The unsung heroes
When embarking on VP filmmaking, pre-production introduces a new dynamic — the Virtual Art Department (VAD), led by the Virtual Art Director, in turn led by the department head, the Virtual Production Supervisor. The VAD is pivotal in shaping the visual elements of the project during the early development phases.
The VAD is the creative powerhouse, shouldering the responsibility of designing, creating, and managing all visual elements of the project.
From crafting intricate 3D prototypes to developing camera-ready props and environments, the VAD will collaborate closely with the pre-vis team, ensuring a seamless transition from initial idea to iterative realization.
Together, they form a harmonious partnership, breathing life into the project and laying the foundation for a cohesive and visually captivating film experience.
Collaboration takes center stage here — the Virtual Production Supervisor, VAD, pre-vis team, and physical production design crew seamlessly blend their expertise to refine and iterate on visual elements.
This chemistry lays the foundation for a successful and visually mesmerizing film. Through open communication, the exchange of ideas, and a shared vision, dreams come to life on the virtual canvas.
Like a synchronized symphony, their harmonious efforts will elevate your film to new levels and deliver an awe-inspiring visual experience. Successful VADs and VAD teams can work and speak comfortably across three disciplines: 3D graphics, cinematography, and traditional art and design.
Good collaboration can lead to visually striking shots like these from Discovery Channel’s Tales from the Explorer’s Club Series.
Synergy between pre- and post-production
By incorporating real-time graphics and color design during the pre-production phases, the efficiency of post-production is significantly enhanced as well since nearly all that work can be utilized in an “Unreal for Post VFX” workflow.
Integrating In-Camera Visual Effects (ICVFX) or Real-Time Chromakey elements during production also streamlines the process further down the line, minimizing any need for extensive move-matching work in post.
By letting their VP teams know ahead of time this color, scene, and tracking data will need to be delivered, post supervisors will not only save time but also empower filmmakers to fine-tune their creative vision through the VFX process, resulting in a more cohesive and immersive final product.
You’ll most commonly use the saved color data either to create patches that are composited into the shot or to generate fully CGI plates that match the live-action footage.
The most popular question is generally about the camera tracking data and what it can do for post-compositing. The answer is you can do quite a bit.
Once camera tracking data is saved in Unreal Editor, most likely as a recorded Sequencer event, it can be exported into a variety of formats for import into Nuke, After Effects, etc.
From on set of Discovery Channel Tales from the Explorer’s Club Series shot on Soapbox Films and ARwall’s Virtual Production stage in Burbank, showing just how complete an in-studio shot can be of an exterior environment, using in-camera effects.
At that point, you’ll need to ensure the world scale and orientation of the camera are correct; UE uses an unpopular orientation standard and a 1-meter unit scale, so be sure to look out for that.
As we venture into this exciting frontier, it’s essential for filmmakers to recognize the intrinsic link between pre-production and post-production in the VP process.
By embracing the role of the Virtual Art Department and ensuring effective planning of your virtual environments, you can vastly enhance and speed up the post-production phase to the point where much of the work is preconfigured for the post-team
The seamless integration of these workflows paves the way for a more immersive, visually captivating, and ultimately groundbreaking film experience.
So, now is the time to equip yourself with the knowledge and tools to take full advantage of the boundless potential of Virtual Production.
From on set of Discovery Channel Tales from the Explorer’s Club Series shot on Soapbox Films and ARwall’s Virtual Production stage in Burbank, showing just how complete an in-studio shot can be of an exterior environment, using in-camera effects.
About the Authors
Rene Amador is a Virtual Production Supervisor, and Ryan Arms is a Virtual Production Producer and Virtual Art Director; together, they’ve run virtual production depts for major studio projects, commercials, fan films, and everything in between on over 100 deployments.
Both work at ARwall, a pioneer in XR in-camera effects, and most recently a major contributor to an Emmy win for the Art, Sets, and Scenic Design of Muppets Haunted Mansion, which used virtual sets for about 70% of the shots, available to stream now on Disney+.