Lighting for Virtual Production

Virtual Production with XR technology is revolutionizing the film industry. Rendering environments in real-time has given a creative advantage in film and advertising that has not been seen before. It brings the post-production workflow into pre-production, making the term “fix it in post” less relevant. Rendering in real-time means that what you see is what you get, which gives the creative directors more liberty to make art changes on-set instead of during post-production. In this article, I will share some lighting concepts regarding real-time rendering, which anyone that wants to dive deep into this AR/XR field can explore.

When I first think of Virtual Production in Real Time, the main aspect that comes to mind is environmental art that would be projected on the background of a subject. However, even though background projection is the crucial aspect of Virtual Production, this technology can be split into two workflows depending on the client’s needs: Augmented Reality (AR) and Extended Reality (XR set extensions). Both workflows use real-time graphics and involve motion capture with the subject or camera motion. However, even though both are slightly different, it all comes down to one primary aspect: lighting.

In Augmented Reality, the central aspect is the interaction of digital elements with real life. Therefore, these digital elements need believable shadows and reflections to feel like they are part of the same scene, elevating realism. To gain accurate reflections from real life when creating an AR subject, lighting artists use HDRIs. These High Dynamic Range Images collect extra lighting data enhancing realism better than just regular lighting with Point lights, Spotlights, etc. Virtual Lighting should always interact with the real world regarding Augmented Reality. The Digital Multiplex system for AR (DMX) controls lighting features from the ceiling of an actual set. This system allows recreating any real-world lighting virtually to light XR elements. This system also controls blueprints within the Unreal Engine Software to change, for example, the time of day, the subject’s color, and special FX. These values are sent to a “patch,” which is one or more exposed parameters in the DMX system that allows control of the blueprint.

Lighting for Extended Reality (XR) takes into consideration lighting an interior or exterior environment more than just an interactive subject like Augmented Reality. The Light Function Material in Unreal can be used to create a sharp internal light pattern or unique light projections that just a simple texture or a decal can’t make. It allows the user to create custom alphas, movements, and shapes with lights provided by the Engine. These functions are also suitable for creating fake god rays or intense concert spotlights.

For both Augmented and Extended realities workflows, the user needs to have control of post-processing settings. One of the main aspects of post-processing is exposure control. When moving around a virtual environment, sometimes different perspectives have exposure shifts. Therefore, it is essential to establish exposure limits. By adjusting the bloom, the artist can control the intensity to match the environment’s needs on set. Apart from exposure control, it is essential to color-correct the background to match the LED wall. Sometimes, pre-production screens do not have the same color range as the final output recorded from the LED volume. Any adjustment needed on set should be established during pre-production with a blueprint parameter, so it can easily be controlled or changed on-set.

Emissive materials, light temperature, and color are other creative approaches that can help real-time lighting. The lighting lumen system within Unreal Engine has helped enhance reflections, cast light, and emissive materials. The subtle glow these lights emit can enhance the beautiful visuals, particularly in dark environments. Bright emissive hues can also guide the viewer’s eye in a bright or busy background. These lights, in combination with the purpose of the color, push forward the visuals of a real-time environment. The artist should pick each color’s task wisely by asking himself: Which color will direct the viewer’s eye?, Which color will decorate the environment? And which color will rest in the background? Intentionally picking each color’s purpose will make a big difference in the environment’s design. Less is more; sometimes, three lights are just enough  (Fill, Rim, and Key light).

The industry is starting to scratch the surface with Real-Time Rendering technology, and the results have been amazing so far. I look forward to seeing where this promising technology will take creatives within the film industry and beyond.

References for this article are from Unreal Technical Artist at Silver Spoon Bianca Lopez and professor of Visual Effects Bridget Gaynor.