George Lucas famously said that sound and music are 50% of a movie – but when it comes to virtual reality that percentage is far higher.
Audio is the guiding force in VR. It's the tether that keeps a viewer from floating aimlessly in an unreal world, a signpost that can steer the direction of a movie or game's narrative. When you have complete freedom to look wherever you want, audio is there to offer a semblance of control.
One of the figureheads in creating 3D audio for these new virtual worlds is Fraunhofer. In its vast research facility, Fraunhofer was instrumental in the creation of the MP3 audio standard in the 1990s, and was part of the team that developed AAC.
It's now using its vast audio knowledge to better the VR experience, and in 2015 authored the MPEG H codec, which is being used by the majority of home entertainment companies to deliver 3D soundscapes, not just in virtual reality but in devices such as soundbars as well.
"We look at VR audio from an end to end perspective – from content capture to playback on devices," explains Jan Nordmann, senior director of business development at Fraunhofer.
"One of our first customers was Samsung with the Gear VR. Samsung used our Cingo headphone technology from day one in the Gear VR product. And now the LG 360 is using the same technology."
Cingo is a technology that is being used more in more in virtual reality - very recently, it was announced that Hulu's VR experience. This means that users of Hulu's VR app will benefit from movies and TV shows streamed using the HE-AAC surround sound audio codec and rendered for headphones through Fraunhofer Cingo.
Getting 3D audio right is a complicated business but Fraunhofer has created a production plug-in that allows sound editors to move around objects in a mic, much like you would different audio tracks. Bring them closer to the listener and they get louder, move them from left to right and they will know that an object is moving around them.
Couple this with VR's freedom to let you move and focus anywhere in a vast space and this could cause some audio headaches but there's a factor that helps with sound awareness and that's head tracking.
Audio tracking
"One major technical difference between watching a movie on a tablet and watching it in VR is head tracking," explains Nordmann.
"In virtual reality, we know where you are looking - on a tablet we basically don't. If you don't have headtracking in VR, then the sound doesn't move. So we use the head-tracking data supplied by the VR device and track the audio accordingly."
Fraunhofer's codecs are powering pretty much all mobile-based VR units audio-wise. Its Cingo technology offers a virtual reality experience that is as, well, realistic as possible.
"VR done right creates the illusion of full immersion, a full presence. If you do it wrong, then it completely breaks the illusion," says Nordmann.
"It can make you sick. If there is a disconnect between audio and video in VR then it is much more traumatic than out of sync audio with TV.
"Once you have that disconnect, your brain starts thinking about the VR process and you get thrown out of the illusion."
To make sure this doesn't happen, Fraunhofer has created a state of the art laboratory to test 3D audio in the best possible environment. The studio has a vast array of speakers to create a 3D sound, with the most important being the adjustable height speakers that offer up the spatial awareness needed for VR.
Fraunhofer doesn't create any VR hardware but because it licenses the codecs out, it is key in helping other companies understand how best to capture 3D audio - including developing its own prototype mics to show off the technology.
"While you can use a 'microphone tree' with an experienced operator, the recent 360 camera models have a different form factor and their audio capabilities are limited," explains Nordmann.
"We want to make it easy for consumers and professionals, so we have on-going research with prototype mics. We developed processing capabilities to get the 3D audio for a number of microphones that we would then encode with our codecs."
This facility is key to making sure VR audio is as good as it can be. Get it wrong and, like Nordmann says, the illusion breaks. Get it right, though, and you can not only aid the story but tell it, too.
"With 360 video and VR, the viewer can wander where they want, while the action is elsewhere. A director can use audio cues, however, to guide someone through the story. An explosion somewhere can make you turn your head and the story continues in that space," Nordmann
"Going from surround sound to 3D audio is a bit like going through stereo to surround decades ago. You have to experiment, learn when things go wrong. We have developed the tools to make this process easier."
