Since the original decade-old concept video for the ill-fated Google Glass, there has been a clear line in the sand for the holy grail of augmented reality perfection. Many headsets have tried and failed. Here’s how the Apple Vision Pro is setting up itself for success by learning from the lessons of its predecessors.
MVP Requirements for AR
When I think of AR headsets, I think of the minimum viable product. I define the MVP to be a hypothetical headset that some small group of early adopters find viable and useful to wear consistently throughout their daily waking lives for extended periods of time.
This may be a very small group of a few thousand people, but crossing this MVP threshold creates a dedicated community of developers who have the basic tools and the direct feedback they need to design and prototype AR applications for real-world scenarios. This definition was shaped by my experiences with the XR community in Tokyo, many of whom pushed the Meta Quest as a makeshift AR device far beyond its reasonable limits.
If you build it, they will come.
– Field of Dreams
A true minimum viable product for AR has five basic requirements. These are the benchmarks that Vision Pro and its peers should be evaluated on.
- Wearable everywhere all the time 🚶 — For an AR app ecosystem to succeed, you need a device that users can wear on any occasion without feeling tethered to their room or chair. All the awesome potential of AR in providing you real-time visual overlays on the real world is wasted if the only thing you can look at is your living room. If you are stuck in your chair, than VR is almost always better than AR. The ultimate goal here is a device that every human wants to use 24/7, while the MVP is a device that 1% of the people are enthusiastic enough to use while they are out and about.
- Understands the physical surroundings 🧠 — Once you have the device glued to the front of your users’ eyeballs, you want a powerful sensor and software suite that can make sense of the scene in front of the user. This includes semantic understanding (is it a cat?), spatial understanding (is the cat in front of a chair?) and persistent understanding (is this the same Siamese cat from yesterday?). The ultimate goal is a human-scale 3D spatial and semantic map of the entire world, while the MVP is sufficient spatial understanding to create geometrically immersive visual overlays.
- Preserves real-world interactions 🤝 — The ultimate AR device is invisible to the eyes. A user viewing the world through the device should have the ability to see the world exactly as it is with no obstruction or visual imperfections. Another person in the same room should be able to interact with a user wearing the AR device without noticeable barriers. An MVP here would be something that obviously intrudes on the experience, but not enough to make the real-world interactions infeasible. For example, we don’t consider a person wearing sunglasses incapable of social interactions, but we would if they were wearing blindfolds and noise-cancelling headphones.
- Maximizes realism 💯 — The ultimate AR device produces visual and audio experiences that can be selectively indistinguishable from the reality that it is augmenting. This includes a large field of view (FOV) that earlier AR/VR devices struggled with. A virtual cat should appear to navigate real-world furniture exactly like a real cat, if that is what the intended experience calls for. The MVP here probably needs to produce spatially-aware visuals that at least obey some of the geometrical constraints of the real world without necessarily achieving photorealism.
- Network effects 🌎 — Yes, fully immersive VR chat can be great and over the long term can produce its own network effect, but what’s the value proposition for the early adopters if none of their friends can hop into VR? The minimum viable product needs to be viable even before direct network effects kick in. Typically, this means working well with an ecosystem of existing products and services that do not require other participants to also own headsets.
How Vision Pro (attempts to) checks the boxes
Given the above framing, here’s how every design decision of the Vision Pro pushes the boundaries of current technology to achieve the needs of an MVP for AR by checking one or more of the five boxes. 🚶🧠🤝💯🌎
4k resolution per eye — This is 2.4 times the number of pixels of Meta’s high end Quest Pro headset. Higher pixel resolution is not just about producing more photorealistic renderings and 3D overlays but, perhaps more crucially, enabling text to be comfortably readable. This is why the Vision Pro keynote presentation emphasized this point multiple times.
Readable text at reasonable sizes means a true replacement for flatscreen monitors for both work and play, which means fewer reasons to ever take off the headset. Text-based apps being useable also taps into Apple’s network effects by making iOS apps useful.
🚶✔️🧠◻️🤝◻️💯✔️🌎✔️
3D spatial understanding — When Apple built ARKit for iOS devices, it quickly became clear that it was testing ground for something that would eventually become the Vision Pro. This spatial understanding framework now underpins almost every major feature in the Vision Pro, from the real-world lighting effect on the digital 3D UI elements, to the ability for your friends to break through the VR environment.
Meta Quest experimented with and implemented many of these features, but lacks an existing AR developer ecosystem to refine its ideas and the custom silicon to push them to their limits. Some AR and computationally intensive spatial interactions feel handicapped on the Meta Quest.
🚶◻️🧠✔️🤝✔️💯✔️🌎◻️
Spatial audio — It was a cool gimmick on the AirPod Pro, but Apple was clearly playing the long game there. Not only does it clearly help with immersion and realism for AR content, but with the right pass-through audio processing, it can also keep the user connected to sounds from the physical world. That takes away another reason for the user to feel the need to remove their headset.
🚶✔️🧠◻️🤝✔️💯✔️🌎◻️
Eye-tracking, gesture and voice controls — All major VR systems converged on using two spatial controllers with joysticks because they are fundamentally video gaming devices. But while the controllers offer precise motion controls great for gaming, they keep your hands occupied and prevent you from interacting with the real world. You are less likely to wear the headset for prolonged periods of time if you have to bring controllers with you everywhere you go.
🚶✔️🧠◻️🤝✔️💯◻️🌎◻️
External battery — This makes the headset itself more ergonomic, incentivizing longer user sessions. The battery itself can also be higher capacity as a result of less stringent weight restrictions, further enabling prolonged use. It’s not as aesthetically slick as embedded batteries, but the Vision Pro makes choices here that signals its intend to stay on your face.
Even the MagSafe-esque connector suggests that the battery is designed to be swapped out quickly when you are not at your desk.
🚶✔️🧠◻️🤝◻️💯◻️🌎◻️
EyeSight — What at first glance appears gimmicky is actually a critical tool in preserving real-world interactions and lowering the social cost for users who choose to continuously wear the Vision Pro.
The Meta Quest never attempted a solution for this problem and therefore can never cross the mental barrier of being a single-user device that you should rip off your face the moment your friend steps into the room.
🚶◻️🧠◻️🤝✔️💯◻️🌎◻️
Digital Persona — Another dividend from Apple’s investment in ARKit and the iPhone’s front-facing camera and depth sensor suite. Digital avatars have been tried in every XR platform, but this is the first one backed by some serious 3D scanning hardware that does not take up a whole room.
If properly executed, a digital twin can not just make for a cool pure VR chat experience, but serve as a suitable interface for a headset user to interact across platforms with iOS and macOS users. This leverages existing Apple network effects to bootstrap the ecosystem. Facebook tried doing the same thing in 2017 in a pre-Horizon product called Facebook Spaces, but the compromise of using cartoon avatars make them entirely unusable when the person on the other end of the call is on their smartphone.
🚶✔️🧠✔️🤝◻️💯✔️🌎✔️
12ms sensor fusion latency — This is probably the single most impressive achievement that makes the entire endeavor possible.
Whereas earlier headsets like Microsoft HoloLens and Magic Leap attempt to project rendered images on an optical lens through which the user can see the real world directly, Valve and Facebook pioneered the technique of streaming live footage from front-facing cameras to a traditional VR screen setup to create a rendered AR experience. Vision Pro takes this latter technique to a whole new level.
This approach has the benefit of enabling better color representations of AR content and more flexibility in selectively showing mixtures of reality and virtual reality. The problem is that processing the camera images to look right for 3D depth perception takes up computing resource and can potentially result in imaging latency that lead to jitteriness and motion sickness.
Apple claims its specialized R1 chip can combine the inputs from 12 cameras, five sensors, and six microphones, with a latency of just 12ms, which is a remarkable feat.
This has many downstream benefits. Compared to the HoloLens approach, it results in much wider field of view, image fidelity and the optionality of offering true VR. Compared to Meta Quest, it has much better real-time performance and can actually serve as the primary view of the real world for the user without impeding their ability to perform tasks.
Low latency sensing also enables almost everything else on this list.
🚶✔️🧠✔️🤝✔️💯✔️🌎◻️
Incrementalism over perfectionism
Like the first iPhone, the Vision Pro is far from perfect. It is clearly an amalgamation of compromises rooted in present day limitations. But it also appears to be the best combination of compromises made with the best consumer electronic design and manufacturing capabilities that we have seen in this space to date.
When compared to its contemporary peers, it seems more evolutionary than revolutionary. But if this incremental step crosses the elusive viability threshold, then its impacts with the hindsight of history may yet prove to be a paradigm shift.
Just as PDAs and early smartphones meandered through numerous design combinations before the birth of the iPhone (an MVP for keyboard-less multi-touch interfaces) sparked a rapid evolution towards creating ever more perfect rectangular glass slabs, the Vision Pro is attempting to establish a standard for what it means for an AR device to augment reality, rather than obscure it.
Now this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning.
– Winston Churchill
Final Note
There is the obvious dystopic potential in everything I have described above, and I make no strong claim on the societal desirability of what I refer to as “the ultimate AR device.” But I do believe in the inevitable march of human inventiveness and problem solving. We are powerful optimization engines, and the Vision Pro, if it delivers on its simple set of promises, casts a bright spotlight on the clear path ahead for the industry.
If in three years time, there are tens of thousands of augmented reality enthusiasts across the world who spend the majority of their waking hours wearing the Vision Pro, and just a small fraction of these hardcore early adopters choose to spend their time pioneering new UX and design paradigms for this inconceivable way of interacting with the world, then the Vision Pro would forever be the MVP and inspiration from which every future augmented reality experience is conceived and prototyped. Everything else beyond that is just pure upside for Apple.