Because we in Silicon Valley are newness junkies, it can feel like an act of sabotage to have memories, but, for better or worse, I have them. It’s been more than forty years since I co-founded the first company to make headsets and software for simulated experiences, and came up with familiar terms like virtual and mixed reality. Since then, virtual reality has flooded the public imagination in waves; back in the nineteen-eighties, for instance, it had quite a presence in movies, cartoons, TV shows, the occasional arcade game, and a few early consumer products, like the Nintendo Power Glove. I still love V.R. But, these days, I sense that what I experience of it, what I enjoy in it, is different from what it has come to mean to many enthusiasts.
Back then, at the beginning, did I talk about V.R. like the people I disagree with now? Sometimes I did! I did occasionally promote V.R. as an alternate cosmos that might swallow us all to good effect. I don’t agree with that sort of talk now, but at the time the joy of being edgy and extreme was too great to resist. Every young technical person wants to see around the corners that others cannot, to be the harbinger of great change. To explain to someone in the early eighties what V.R. was—to give them a demo—was an ego rush, because they often couldn’t grasp what was being said or what had just happened to them. It was a primal validation, a power trip, and I wish I had done it with more humility. But here we are, after almost a half century of products, movies, and startups, and V.R. people still seek that rush. Apple’s promotional videos for its new headset—the Vision Pro, a metal-and-glass, ski-goggle-like apparatus that costs around thirty-five hundred dollars—show people experiencing V.R. for the first time. The producers either had to search hard for those people or ask actors to pretend.
Apple’s entrance into V.R. has symbolic weight, because the company has had so much influence on computers and phones. Back in 1984, when the Macintosh was introduced, a few members of the Mac team left Apple to join my startup, V.P.L., and help us create the first generation of commercial V.R. products. At the time, we guessed that Apple itself would enter the market in 2010. We knew that the consumer adoption of the technology was a long way off. We were selling tools for millions of dollars to customers like NASA. But despite the conservative clients, our early V.R. software was radically strange. You could program while in our virtual world, and see all the variables not as textual symbols but as virtual objects; there was no source code. I used to wax on about how virtual reality would lead to a new style of “post-symbolic” communication, in which we would make experiences for one another, sharing them directly instead of just describing them.
The hope was to make V.R. a place for spontaneous invention. Virtual reality would allow groups of people to “play the world into existence” on virtual devices that resembled musical instruments. (Instead, we now talk about “speaking the world into existence” using A.I.) It would be like a social lucid dream.
In the intervening decades, V.R. has thrived at two extremes in the quest for “killer apps.” It has long been an established industrial technology: if you’ve flown, ridden, or sailed in a factory-built vehicle in the last thirty years, virtual reality may have played a central role. It’s been used to design surgical procedures and train surgeons ever since our first simulated gallbladder, at Stanford Med, some three decades ago; Boeing, Ford, and many other companies started using VR for design in the early days as well. And then there are the visionary, mystical, and philosophical applications. V.R. can be a way of exploring the nature of consciousness, relationships, bodies, and, perception. In other words, it can be art. V.R. is most fun when approached that way.
In between the two extremes lies a mystery: What role might V.R. play in everyday life? The question has lingered for generations, and is still open. Gaming seems likely—but, for most gamers, not so much. There are many reasons why V.R. and gaming don’t quite work, and I suspect that one is that gamers like to be bigger than the game, not engulfed by it. You want to feel big, not small, when you play. (“Star Wars” might have got this right with holographic chess.) Apple’s initial round of Vision Pro apps, like those from its competitors, aren’t entirely compelling, either, and can even have a lonely, dystopian flavor. (Watching a simulated big-screen movie, by yourself?) But my belief is that the quotidian killer apps will come. Maybe you’ll use V.R. to learn quickly about the Airbnb at which you’ve just arrived. Maybe V.R. will help you assemble IKEA furniture. Maybe!
Virtual-reality headsets come in various forms. A major divide has to do with how they acknowledge the real world. Some headsets obscure the surrounding environment completely; this is typical in gaming headsets. But there is another option, which I used to call “mixed” reality, and which came to be known as “augmented” reality in the nineteen-nineties. Some mixed or augmented headsets, such as the Microsoft HoloLens or the system created by Magic Leap, allow you to see the real world through the headset glass so that it can be combined with virtual content using challenging optical techniques. Others, like Apple’s Vision Pro and the recent offerings from Meta, capture the real world with cameras, then render it as part of the virtual environment so that it can be combined with fabulated content.
Camera-based mixed reality is vastly easier to accomplish than the optical version, but it is concerning. Early research by a Stanford-led team has found evidence of cognitive challenges. Your hands are never quite in the right relationship with your eyes, for instance. Given what is going on with deepfakes out on the 2-D Internet, we also need to start worrying about deception and abuse, because reality can be so easily altered as it’s virtualized.
The attention-maximizing business model that drives current social media naturally gravitates toward alarming content that activates primal fight-or-flight responses. Fortunately, Apple is one of the companies that doesn’t rely on that model. Nonetheless, experiences will come from all kinds of companies, and V.R. could agitate and depress people even more than the little screens on smartphones.
Today, in much of the online world, algorithms calculate “feeds”—a term suggesting that people are cattle, and not free-range ones. This is a good reason for wanting virtual reality to be an environment in which users actively create their own content, rather than passively experiencing what big companies have built for them. For most of the technology’s history, however, virtual experiences have been hard to build and maintain. This has been one of V.R.’s biggest problems. I saw the first V.R. teaching demonstration of general relativity at least as early as 1992, and have seen dozens more since then; they’re often wonderful, and help users grasp the concept in new ways. But they only run for a year or so because there are too many variables in a V.R. system for creators to keep experiences available. Graphics chips change, and with them the layers of mediating software. That’s true for other programs, too, but with V.R., when the properties of a headset (like field of view) or an input device shift, the whole experience and interaction method must often be rejiggered. It’s too much ongoing effort, so it usually doesn’t happen; developers move on to other projects. The exceptions have been locked-down V.R. experiences that assume a minimal level of interaction, which limits the magic.
Still, the first generation of V.R. entrepreneurs thought of it as a fundamentally active experience. In fact, the illusion at the heart of virtual reality requires activity on your part: the virtual world only starts to feel real when you move your head; it’s when your surroundings compensate for that motion that they appear to exist outside of you. And yet we often see V.R. depicted as a funnel that dumps spectacular experiences into the brain of the user.
Today, the V.R. community seems to want to make headsets smaller and smaller, until they disappear. But in the old days I’d build one-of-a-kind V.R. headsets into big masks from different cultures, sometimes adding lightning bolts and feathers. I wanted the headsets to be vibrant, exciting objects that enriched the real world, too. Apple’s device shows the eyes of the wearer, by means of screens on the front, and this creates the illusion that the headset is transparent. But in past decades we used similar screens to show those in the real world the face of the virtual avatar that the wearer was inhabiting: a viking, an octopus, an abstract geometric construction. You wouldn’t want to disguise a motorcycle as a bicycle. If you’re going to wear a headset, you should be proud of that weird thing on your head!
Apple is marketing the Vision Pro as a device you might wear for everyday purposes—to write e-mails or code, to make video calls, to watch football games. But I’ve always thought that V.R. sessions make the most sense either when they accomplish something specific and practical that doesn’t take very long, or when they are as weird as possible.
The practical side of V.R. is a scattering of wonderful niches: in addition to surgical simulation and vehicle design, the technology is used by oil companies to simulate geological structures, by drug companies to envision molecules, and by planners working on city centers. The new frontier, which might apply more to everyday life, is the spontaneous creation of practical apps that you might not even bother to save. My research group, for instance, has presented a prototype system—the “mixed-reality copilot”—that allowed us to recreate, with a single voice request, a program that allows you to use your hands to paint and sculpt with virtual stuff. A decade ago, it took months to make that kind of program. Hopefully, in the near future, one will be able to ask for a V.R. relativity simulation tailored for a student who has color blindness and A.D.H.D., and it will simply appear. More prosaically, you might walk through a facility in augmented reality, asking an A.I. for instant advice about potential safety hazards and fixes. These ideas might even work already: one of the curious features of this accelerated period of A.I. development is that there aren’t enough minutes in the day to try everything.
On the weird edge, it turns out you can change your body plan in V.R. You can become different animals. You can map your body to that of a lobster or an octopus, and experience, to a significant extent, the control of that other body. The brain has had to adapt to many body plans over the course of its evolution, and it’s pre-adapted to work with more. When you change your body, you can also play with the flow of time. By shifting the rhythm of the natural sway of your limbs, and also how the objects around you move and change in response, you alter the reference points that your brain uses to mark the flow of time. You can speed it up or slow it down. In V.R., you can change the rules of the world. You can exist in strange geometries that are too hard to describe in words. You can become an archipelago of parts instead of a continuous animal. You can blend and share bodies with others, to a surprising degree.
All this can happen, and yet there’s a little nub that remains unchanged. There’s still a you floating in the middle of it all. This makes virtual reality a consciousness-noticing machine. Lately, people in A.I. have colonized the word “consciousness”; they sometimes suggest that their models have it. This can make us wonder if it’s even real. But V.R. reminds us that experience really exists. It’s at the core of V.R. It exists even when the world around it is an illusion.
There are fresh, urgent reasons to reaffirm the value of experience. It is impossible to judge technology without a sense of its purpose—and its only plausible purpose is to benefit people, or perhaps animals, or the over-all ecosystem of the planet. In any case, if we pursue technologies that make it hard to delineate the beneficiaries—for instance, by blending brains into robotics not to cure a disease but just because it seems cool—then we make the very idea of technology absurd. The central question of the technological future is how to identify the people who are supposed to benefit from technology, especially if they seem to have melted into it.If people aren’t special, how can we act in a way that benefits people? We can’t. The principles of ethics, design, and even technology itself become nonsense. What can that specialness be? It must be something that is not technologically accessible, since technology expands unpredictably. It’s a little mystical. The definition of people must be one of apartness. We must now put people on pedestals, or they will drown.
When I put on a V.R. headset, I still notice that I am floating there, that I exist independently of the information I experience. But then there’s the moment I take off the headset, which is the very best. In the nineteen-eighties, we used to try to sneak flowers or pretty crystals in front of people before they would take off their headsets; it was a great joy to see their expressions as they experienced awe. In a sense, this was like the awe someone might experience when appreciating a flower while on a psychedelic drug. But it was actually the opposite of that. They were perceiving the authentic ecstasy of the ordinary, anew.
This is the kind of experience you can have only if you use V.R. fleetingly, not constantly. Here we come to one of the greatest differences between what I love about virtual reality and how it is often promoted today. Venture capitalists and company-runners talk about how people will spend most of their time in V.R., the same way they spend lots of time on their phones. The motivation for imagining this future is clear; who wouldn’t want to own the next iPhone-like platform? If people live their lives with headsets on, then whoever runs the V.R. platforms will control a gigantic, hyper-profitable empire.
But I don’t think customers want that future. People can sense the looming absurdity of it, and see how it will lead them to lose their groundedness and meaning. In the early days of V.R., I actually tried living in it all the time; on a number of occasions, I stayed in virtual reality for several days. In fact, back at the start, some of my young colleagues and I made a vow: if we ever became parents—a possibility as remote for us at that time as a visit to Alpha Centauri—we would slip baby V.R. googles onto our kids at birth, then swap in bigger ones as the kids grew. That way, our kids could grow up in four dimensions, and become the greatest mathematicians ever. When I told my daughter about this, at around age eight, she was angry that I’d denied her this chance.