LED-based extended reality production has been accelerating ever since the pandemic hit.
Now, it’s a widely deployed technology spanning everything from broadcast to live events. We immerse ourselves in the details
Extended reality (XR) – a virtual production technique making use of an LED volume to let talent see the environment they are supposed to be interacting with – continues to gain traction. It’s delivering immersive, innovative experiences in all kinds of scenarios – from reality TV shows and music performances to live sport.
“The whole ecosystem involved is just larger, as are the requirements for the technical crew,” says Marina Prak, marketing manager at Roe Visual Europe. “It touches on almost every discipline and product involved. Suddenly, all the stand-alone equipment needs to be interconnected, synced and working together flawlessly. That requires an awful lot of technical backbone to make it all happen. Workflows for all the staff involved need to be integrated and every aspect of the production must be thought through in pre-production.”
In live entertainment, XR allows artists to connect with their audiences – even if they may not be in the room with them. In 2020, one of the first major applications of XR was Katy Perry’s performance of Daisies at the American Idol finale. Instead of singing for a live audience, Perry sang her single on an LED stage with real-time graphics rendered on the LED walls and appearing in-camera.
“She was immersed in a colourful virtual environment – allowing a virtual audience to experience an incredible performance,” says Tom Rockhill, chief commercial officer for Disguise, whose technology drove the event. “Since then, we’ve seen widespread adoption of XR.”
In fact, more than 600 productions and 350 stages in over 50 countries are using Disguise’s XR solution. Last year, DJ Alan Walker used XR to power his virtual concert World of Walker for electric vehicle manufacturer Nio’s annual product launch day. In August, singer-songwriter Camila Cabello made a virtual album launch concert on TikTok using XR. Karate Combat 35 also used XR to create a hybrid event where in-person attendees and a virtual audience could both see the fighters immersed in a CGI environment.
The technology is being used by broadcasters, too, immersing viewers – and particularly guests – into the virtual environment. Plazamedia is at the vanguard, having commissioned Mo-Sys to deliver a turnkey LED studio in Germany ahead of the 2022 World Cup. Using Mo-Sys tech, the new studio blends the real set with the virtual world, creating the illusion of a 360° environment, and adding the creative freedom to deliver real-time 2D keyed and rich 3D on-air graphics into the scene – together with multicamera switching at UHD 4K resolution.
Initially, XR was expensive. There was a perception that you needed a huge volume similar to the enormous one used on Disney+ show The Mandalorian. Now, productions of all sizes are using XR. All you really need is a tracked camera, an LED backdrop, content and sometimes just a basic Disguise set-up.
“Since the technology and markets have matured and there are more studios around the world, companies no longer need to build a stage in order to do an XR production,” says Rockhill. “There are many available for hire. Disguise has a powerful community of studios based around the world that can help you.”
There are still challenges, notably around training. “There is definitely a knowledge and skills gap,” says Stephen Gallagher, marketing director at Mo-Sys. Content producers who are considering making the transition to an LED virtual studio need to find a partner who understands the complete workflow, from camera tracking to lighting, and the nuances of LED screen technology. There aren’t many companies with that level of knowledge, capability and experience. Prak adds: “Not only do we suffer from a more general post-pandemic skills gap, but the skills required to make this all happen are scarce. We need qualified and trained operators and highly technical people capable of making the interconnected ecosystem from camera to LED screen work. The technical set-up and complexity of an XR stage is still underestimated. Where a stable-performing LED screen capable of handling the required technical specs is helping in the process, it’s not the most complicated part of the whole system.”
All the while, innovation is advancing. Ghostframe, which is developed by AGS, Megapixel VR and Roe Visual, offers live multisource display and capture.
“It’s beneficial in virtual production or XR studios, for either film or broadcast, as it allows you to shoot scenes with multiple cameras in one take,” notes Prak. “Each camera can simultaneously shoot a different background. You can add your XR worlds with the option for hidden tracking or even have director cues only visible in the studio on the volume.”
Mo-Sys points to its BMR (broadcast mixed reality) solution that combines an LED content server with an MOS-controlled on-air graphics system powered by Unreal. Designed for broadcasters who want to transition to an LED-based virtual studio, with MOS-controlled on-air graphics based
on Unreal, this is targeted at sport, news and current affairs programming.
Disguise recently teamed up with Pomfort to reduce technical barriers between Livegrade Studio’s established on-set colour grading workflow and the Disguise virtual production workflow. Its cloud platform enables users to collaborate, store and access new versions of products for review. It has also evolved 2.5D workflows, meaning you can use flat 2D images in a 3D view to simulate a full 3D environment.
“This gives anyone access to the look of complex and intricate 3D scenes in just a few clicks,” explains Rockhill. “On top of this, we’ve launched Disguise Metaverse Labs, offering R&D, consultation and creative services.”
That looks forward to a day when the metaverse ‘is going to alter human experience as we know it’, according to Rockhill. “XR provides the intersection and interplay between the physical and virtual worlds, to enhance performance, communication and production. This lets us build a gateway to the metaverse so that live acts can perform in metaverse platforms.”
Artists like Ariana Grande, DJ Marshmello, Justin Bieber and Kaskade are already doing it, and a best metaverse performance category has been added to the VMAs.
“By 2025, we can guarantee there will be a virtual or metaverse layer to almost all the experiences of today,” Rockhill reckons – all powered by the cornerstone technologies of XR at this moment.
Comments