When we asked if he thought setting VR and AR up in opposition to one another was a reductive approach and whether we’d be better combining the technologies to develop something that incorporated elements of both Sweeney agreed:
“Yes, I think that exactly. I think the best AR developers of the future are going to be the developers who are building VR games right now and learning those lessons early on.
“AR is a superset of VR and a superset of everything else. Once you have AR glasses you can put on your head and have a very high quality display seamlessly combining computer images with the real world, then you don’t need a TV or a monitor because the glasses can project one wherever you like.
If you black out the background in AR you could make an immersive VR experience and if you make the view translucent so you can see through it you just have an augmented view of the real world. I think that's the ultimate and best form of display tech we’ll have.”
On VR's next steps
Until we get to that next stage, however, Sweeney listed a few things that VR is going to need to improve upon in terms of hardware in the meantime:
“First of all we need improvements to consumer convenience. It’s a fairly lengthy process to set up a VR hardware installation and fiddle with drivers and all these different things. It’s definitely not an experience for everyone yet. I think once we improve ease of use and set up it’d be awesome to have wireless so you don’t have to have a cable to keep track of. It’d be great to have inside out tracking so you don’t have to have outside cameras to track your motion and the headset can do it in a self-contained way.
"It’s definitely not an experience for everyone yet"
Tim Sweeney
It’d be great to have a lighter form factor so it’s much more comfortable to wear for longer periods of time. It’d be great to have increases in resolution so we can create higher quality graphics and have experiences that have more text and other user interface interaction.”
As far as software, Sweeney was happy to see that AAA titles such as Fallout and Doom were beginning to appear on headsets.
“I think it’s very good,” he enthused, “When you look at the way the game industry develops you have to look at the economic side of it too. Games are expensive to build, we need sources of funding and publishers can be a great means of allowing developers to create games that they wouldn’t be able to create on their own. VR needs that right now. I think we’ll see a lot of innovation in game design and business models in the future.”
On games as services
A business model that’s grown exponentially over the past few years is the idea of games as a service. Massive multiplayer online worlds such as The Division and Destiny that are constantly evolving, changing and selling are becoming increasingly popular. Phil Spencer of Xbox said these kinds of games are the future and financially viable.
As far as Sweeney is concerned these kinds of games are also massively beneficial from a developer perspective as “there are major benefits to building a game once and improving it over a long period of time based on user feedback and behavior. It’s kind of depressing to have to build a game once, take all the user feedback and then spend the next 3 years building another game. Every 3 years there’s improvements - much better to improve it every day right?
Multiplayer games are especially perfect for that model because they don’t rely on a fixed base of single player episodic content.”
For those who prefer their games single-player and story-driven he was certain that “we’ll continue to see story-driven single player content continue with its current business model. Different games can adapt in different ways.”
On AI
In the future, though, Sweeney believes the kinds of online multiplayer games we play could change significantly both in terms of the breadth of experiences they offer and the way we play them.This, he says, will be the result of developments in motion capture technology and AI.
“[Real-time motion capture has] been very rapidly adopted and it’s improving the productivity of game developers but I think its next step is going to be in AR and VR hardware with inward facing cameras that capture your face in real-time. We’d love to see that technology in the hands of all users so that while you’re playing a game, the game can scan your face and your emotions and you can be in a multiplayer experience and see other players.
Right now multiplayer is so low fidelity that you can mainly just type to each other and shoot. Sometimes you can have voice exchange, but the avatar faces just have blank stares.
"[Multiplayer games are] so brute force oriented, but they could become much more social in the future if you had the nuances of human interaction incorporated into them”
Tim Sweeney
It’d be awesome if you could see an actual players faces projected onto an in-game character in a multiplayer game. Imagine how realistic that would feel. I think that could also really change the dynamic of multiplayer games right now. I think they’re so brute force oriented, but they could become much more social in the future if you had the nuances of human interaction incorporated into them.”
Even without real-time motion capture, Sweeney thinks advancements in AI have the potential to greatly improve the way game characters move and express themselves and create something that’s much more realistic and natural.
“I think there’s a lot that can be gained from incorporating more advanced and modern forms of AI into our game processes. Especially in the areas where we can use a huge amount of data to have algorithms figure out how humans act in the real world and extrapolate that in the game world.
For example, the way we do facial animation for humans in games, especially AI controlled humans, is very basic right now. I think if we had thousands of hours of footage of people interacting where we could scan their faces and then take that and map it onto game characters would be really interesting.
We could see more human emotion. Most games have a character animations driven by several hundred canned animations that were motion captured and they use a very old school algorithm to piece these chunks together. What you’d really like to do is scan thousands of hours of people walking around in the real world and use that to figure out how to get a game character from one location to another. Because it’s not always turn walk in a straight line and stop, right? There’s a lot of interesting possibilities to it that can be discovered through deep learning applied to a huge amount of data.
That will really make games improve hugely in terms of visual quality over the next 10 years in some unexpected ways.”
On the metaverse
With developments in AI, VR, AR all progressing neatly together just as massive multiplayer worlds take a more prominent place in gaming, it certainly seems like we’re hurtling towards the creation of a metaverse, where people exist and interact in a reality that’s a simultaneous combination of the real and digital worlds.
Sweeney has said before that he thinks is what we’re heading towards and although it’s easy to imagine it taking a very dystopian turn (Ready Player One anyone?) Sweeney thinks it’s going to be much more positive than that:
“It’s funny that so much of virtual world metaverse fiction is dystopian but I don’t think that’s necessarily going to be a part of it. I think that’s the great things about games becoming more social in nature."
“It’s funny that so much of virtual world metaverse fiction is dystopian but I don’t think that’s necessarily going to be a part of it"
Tim Sweeney
Sweeney explained that while parents see their kids at home playing games as a bad thing right now, "in a few years these kids will be wearing AR glasses and running around outdoors and playing these video games in the real world together. It’ll be a social experience. So you won’t be able to complain at that point that video games are isolating."
"They’ll be social and empowering," he said, "and the fact that there’ll be some computer generated pixels on the thing as you’re doing it isn’t going to mean you’re playing a video game. It’s you in real life and there’s computers helping make real life better.”
Unfortunately, we have a tendency not to be particularly kind to one another when we go online but there’s a good chance an improved sense of each other as people in the digital space could change that.
“Ultimately, I’m an optimist. I do think that once you can see other people's’ faces and you can see them as real human beings you’re going to be much much more polite and much more of your real world instincts and understanding of social conventions will kick in in a way that they simply don’t when you’re reading a bunch of text online and typing a bunch of angry comments into a comment section. I think that’ll greatly improve.”
With dystopic visions of our automated digital future being so commonplace it’s somewhat refreshing to hear such a prominent industry voice speaking so confidently about the positives that are coming. To Tim Sweeney, the gaming industry needs to be open and accessible to everyone to flourish and that’s something we’re more than happy to get on board with.
