Voyaging through the experiences and spaces catalysing the immersive audio-visual movement
Words Katie Kasperson
Immersive experiences are becoming the new norm, quenching our thirst for realism in everything we do. Why watch a movie on a TV when you could watch it via a virtual reality (VR) headset? Why listen to music on a single speaker when you could install a surround sound system? These are the rhetorical questions the industry is currently addressing.
To achieve true immersion, the technology must be advanced enough to convincingly simulate reality. This often involves tricking the senses – namely sight and sound – with spatial audio, holographic humans, larger-than-life LEDs or headsets and haptic suits. Over the following pages, we dive into the companies, technologies and case studies at the forefront of today’s immersive experiences.
Immersive audio has technically existed since the emergence of surround sound, first used in the forties at a screening of Disney’s Fantasia. The film included various audio channels and speakers to give the audience a sense of sonic movement, making it seem as though Flight of the Bumblebee was flitting around the room. After that, surround sound took off, eventually making its way into home movie theatres and stereo set-ups. It wasn’t until 2012 – when Dolby debuted its Atmos technology – that surround sound unofficially rebranded to ‘spatial audio’.
Now, everyone wants spatial audio – in their headphones, in their homes, even in their auditoriums and arenas. While a sense of immersion is relatively easy to achieve on a small scale, larger spaces are a slightly different story. “Big venues bring different issues than a studio,” begins Sjoerd Wierda, sales engineer at Audio Acoustics. “Live sound is always more hands-on than studio work. In movie theatres it’s about playback, but in a live environment – where you move from space to space, from venue to venue and have different room sizes – that’s a different beast.”
Audio Acoustics recently renovated the Theaterhotel De Oranjerie – a live concert venue in Roermond, Netherlands. “It’s a standard theatre; roughly 500 to 600 seats,” says Wierda. “They were looking for a partner to do a complete remodel of the whole theatre – front of house, back of house and so on. They wanted to be able to cater to the orchestras without doing a complete rebuild.
“We had a discussion about variable acoustics,” Wierda continues, which are essentially adaptable environments that can vary their acoustic properties. “You can choose to have a simple loudspeaker for variable acoustics, which works, but it doesn’t have the range and the output needed for immersive audio – what we think the future will bring. In the next ten years, you will see a big jump into immersive audio.”
Together, Audio Acoustics and industry partner Adamson created a multi-purpose system that blended Adamson’s immersive engine (which ‘has a great size, good channel count and really nice user layout’, according to Wierda) and an audio server ‘where we can matrix all the electro-acoustic signals’. “It was powerful enough to do all the back-of-house – so all the paging, background music, recording, hearing aid assistance – on one big platform. It’s quite unique what we did over there,” Wierda explains.
In the age of immersive audio, Wierda notes that while it might be tempting to simply increase the number of speakers, this won’t achieve the desired result. Rather, it’s about the placement and amplification. “You can benefit from having good placement of a sound source,” he notes. “You can sit down and think: ‘I see a singer or a guitarist here, but the sound is coming from there’. That’s quite a big thing,” Wierda explains, attributing it to decreasing latencies. He adds: “We are also pushing towards the boundary of amplification. You can hear somebody talking without noticing they’re being amplified.”
Immersive audio is ‘where we’re going’, believes Wierda. “That’s the future. The transition from analogue to digital was a big step for many people. This is throwing everything overboard.”
Located in central London, the Twist Museum (The Way I See Things) is an interactive and immersive attraction inviting visitors to question their realities through sensory illusions. Blending art and science, Twist is an audio-visual playland, with the tech commissioned and installed by Crossover.
The museum’s various exhibits – including a distortive Digital Mirror and a Digital Kaleidoscope, both designed by multimedia artist Aristotle Roufanis – use an array of technology to give visitors a sense of complete immersion. The mirror, for instance, is really a camera and large format display, manipulating reflections via processing hardware. The kaleidoscope loops audio in real time using Ableton Live software.
Elsewhere, Martin Audio loudspeakers enhance the Audio Interactive Space – an octagonal room which combines soundscapes with synchronised LED lighting. The speaker – an SX110 ten-inch slimline subwoofer – provides LF rumble, chosen for its ‘super-compact form factor, and the fact it could be easily flown without taking up valuable floor space’, according to Crossover senior systems designer Juan Garcia.
Martin Audio speakers also inhabit the Mind Hub – at the centre of which hangs a cube made of 1000 LED spheres. In this room, as well as Twist’s events area, eight Adorn A55Ts pump out background music powered by a VIA2004 four-channel, 2000W Class D amp.
Twist’s zones are all interconnected, guiding each visitor through a curated, contemplative journey designed to challenge their perspective and ignite curiosity. Without immersive audio, the museum wouldn’t be as engaging nor able to create convincing illusions that keep Londoners (and out-of-towners) coming back for more.
Though not physically present in the arena, these concerts have been carefully crafted by all four members of ABBA
Founded in 1975 by Star Wars director George Lucas, Industrial Light & Magic (ILM) is considered an industry leader in VFX, visual development, feature animation and immersive experiences. The mind behind ABBA Voyage, as well as Las Vegas Sphere’s Postcard from Earth and U2:UV Achtung Baby Live, ILM is continuously expanding and innovating, responsible for the VFX in more than 500 films and TV series, having recently released VR experiences Avengers: Damage Control and Star Wars: Secrets of the Empire.
Turning towards live music and transforming the modern concert experience, ILM’s latest endeavour is KISS, the legendary rock band which recently wrapped its farewell tour – but not before adding a surprise ending displaying its members as digitally immortalised avatars and hinting at an everlasting encore.
“ILM has created an industry around turning the wildest ideas into reality, using cutting-edge techniques to help visionary artists tell their stories,” begins Grady Cofer, visual effects supervisor at ILM. “It was in this way that the wildly successful ABBA Voyage was born. The show was built on ILM’s latest technology in visual effects, digital humans, virtual production and LEDs.”
KISS’s concerts will operate using the same cocktail of modern machinery, bringing the de-aged band members to life with advanced performance-capture technology. “First, Gene, Paul, Tommy and Eric came up to ILM where they were captured in 3D,” Cofer explains of the step-by-step process. “We have an Academy Award-winning system called Medusa, where you sit in a chair while surrounded by high-resolution cameras. All those images allow us to reconstruct their faces with incredible precision. Once we have their exact likeness, proprietary tools at ILM allow us to capture their faces while they are singing and use that to drive the face performance of the digital avatar.” He concludes: “Even though the avatars are virtual, the performance is all KISS – every move, gesture and expression comes from them.”
ILM’s technology is making waves in live music, bridging the gap between the practical world and digital one, letting audiences travel to any place imaginable. The company is constantly coming up with novel ideas for an even more immersive experience: “You could take the stage and bring it out to the audience, wrapping it around them,” suggests Cofer. “This can revolutionise live music entertainment. In terms of creativity, the possibilities are endless.”
It’d be wrong to talk about immersive technology without mentioning VR. Having come a long way since its popularisation in 20th-century science fiction, VR rests entirely on a sense of immersion, with the participant fully entrenched in a digital simulation and leaving their physical world behind.
While VR is primarily used in gaming, it has cross-disciplinary applications in education (such as military training), healthcare (virtual rehabilitation) and entertainment (escape rooms, for instance). It’s a format that knows no bounds, which is part of its appeal; users can visit fictional locations, interact with digital avatars and do things they normally wouldn’t dare – like skydiving.
Virtual skydiving is an up-and-coming activity, becoming more commonplace and less nausea-inducing (VR has historically invoked motion sickness, also known as cyber sickness in some) as the technology improves. The experience normally combines indoor simulated skydiving with VR goggles to create the physical feeling of falling while sending audio-visual input to the brain.
In the UK, iFLY – with locations in London, Manchester, Basingstoke and Milton Keynes – offers a 360° VR experience that lets the diver choose between nine scenic locations, including Hawaii and the Swiss Alps. Guided by a trained instructor, iFLY provides a safe yet adrenaline-heavy alternative to a potentially hazardous activity. Who says you need to fall from the sky to have a good time?
ILM’s Stagecraft captured every detail of KISS’s final show – from facial expressions to fingertips
While technologically impressive, VR generally falls short of true immersion because it targets certain senses (sight and sound) and not others (smell, taste or touch). As with iFLY’s virtual skydiving – which importantly introduces a bodily sensation to an otherwise digitally led experience – VR seems to be heading towards haptics.
In Ready Player One – which mainly takes place in a futuristic metaverse – the characters wear full-body haptic suits and walk on treadmills (in other words, they use immersion rigs) while engaged in VR. The bodysuits transmit a signal so a digital touch becomes a physical sensation, taking VR tech to new heights and – in the novel – making the characters even more likely to prefer the metaverse over real life. But the treadmills also play another key role: players can walk or run and mimic a sense of movement.
Recently unveiled by Lanny Smoot – a long-time Disney employee who’s just been inducted into the National Inventors Hall of Fame – the Holotile Floor is a treadmill-like surface which does everything in its power to keep its user centred. Modular, omnidirectional and large enough to fit multiple people, the floor promises to be a game changer in VR by supporting shared experiences and keeping users engaged (they won’t be disrupted by falling off the floor’s edge, for instance).
The Holotile Floor isn’t merely a VR accessory; it can also have applications in live performance. For example, theatre groups could use the floor as a set piece, allowing actors to simulate motion during a scene change or create a practical effect by moving a prop seemingly by magic.
And magic is Smoot’s whole shtick. He’s responsible for the floating head and changing portraits in Disney’s Haunted Mansion ride and has left his mark on many other theme park attractions. For Smoot, the Holotile is a thing of versatility, and though it has yet to find a designated use within Disney, its possibilities are wide open.
An obvious choice for entertainment, immersive technology is also a useful tool in education. “Shown to have a number of benefits for students, an immersive space is always a shared experience,” introduces Henry Brown, marketing executive at Igloo Vision. Unlike in a virtual learning environment, ‘the teacher can retain full autonomy over the session and offer the instant feedback essential in education’.
Igloo Vision creates immersive spaces, primarily through LED walls, for a global network of clients. “Broadly speaking, we see four main use cases: simulation, visualisation, collaboration as well as engagement,” adds Brown. “With an Igloo space, you get a sense of presence and scale otherwise not possible unless you are in that environment. It tricks the mind into thinking it’s somewhere it’s not.” It’s like stepping into another world.
The company developed its own in-house software platform – Igloo Core Engine – to “work with the widest amount of content possible,” explains Brown. Igloo’s immersive spaces can integrate with YouTube 360°, Google Earth and Metaverse Learning, for instance, as well as Unreal Engine, ‘allowing for a theoretically unlimited number of environments and scenarios to be created within an Igloo’.
Commonly used for training purposes across multiple industries, Igloo spaces are ideal for staff onboarding, particularly in cases which require real-world validity. Training can be costly, time-consuming and involve safety risks; Igloo minimises these variables while still providing a true-to-life simulation. When Med Learning Group used one of Igloo’s 360° cylinders in an educational programme, for example, 98% of its participants said that the VR content enhanced their learning experience.
Igloo’s immersive spaces couple the Core Engine – which has a ‘stripped-down interface’, according to Brown – and Control Panel app, simplifying operation, while Igloo’s True Perspective software removes distortion and automatically formats content to a given space. “True Perspective and Igloo Core Engine combine to make immersive spaces easy to use and highly functional, making the widest range of content compatible with the space,” points out Brown. This seamless integration maximises user efficiency.
Besides the traditional LED wall, Igloo adds floor and ceiling projection – ‘built in collaboration with Panasonic and Falkbuilt’ – to its immersive spaces. “It’s particularly useful for viewing any content at height, like a building site, as you really do feel like you’re up in the air. This is great for immersive training to give potential workers the feeling of what it’s like to be on-site before travelling there,” describes Brown.
Floor and ceiling projections add realism to learning spaces
Immersive technology is clearly here to stay, with cross-disciplinary applications throughout the audio-visual industry. Whether it’s an activity or live attraction, in the education or entertainment field, engagement is the name of the game. Immersive environments not only offer a better user experience, but also drive up commercial revenue thanks to their inherent appeal.
As technology continues to improve, we can only expect further immersion, with the metaverse serving as a solid example of where VR is headed. We will likely see more wearables – such as headsets, haptic gloves and suits – hitting the market, while Disney’s Holotile Floor is primed to make waves in gaming, live performance and theme park displays. As hybrid education and employment are seemingly the future, we can anticipate more immersive methods of instruction and assessment.
The era of the immersive experience is only beginning, and there’s so much left in store that we have yet to imagine. Leave it to science-fiction writers and technology experts to make any firm predictions; a treat for the senses is all we’re searching for.
This feature was first published in the Autumn 2024 issue of LIVE.
When Wheel of Fortune and Jeopardy! needed to upgrade their wireless communications system, they turned to Clear-Com FreeSpeak wireless for their iconic television productions.
The Red Clay Strays toured America last summer and fall with dozens of sold-out amphitheaters cheering on their irresistible blend of Southern rock, country, and blues.
New release expands connectivity, simplifies management, and unlocks creative power
What happens when one of Britain’s sharpest lyricists meets real-time rendering and tracking AV tech?
Célestin Soum trusts SMODE to operate and live direct the full video system for Theodora’s first large scale tour across France and Belgium.