Passthrough camera access has become a buzzword in the XR community. While Meta, Apple, and Pico have already made their positions clear, there’s a lot of anticipation surrounding Google’s approach with Android XR. I had a chat with Google, and they confirmed they plan to implement a feature similar to what we see on phones. Stick around to learn more!
The Camera Access Dilemma
Let’s unpack this for those who might be new to the conversation. Today’s advanced standalone VR headsets also qualify as MR headsets. These devices display an RGB passthrough view captured by their front cameras, enabling cool mixed reality apps like Cubism, Starship Home, and Pencil.
Developers like me are eager to access these camera feeds to deploy AI and computer vision to enhance user experiences. I believe that true mixed reality needs camera access; our apps must be aware of their surroundings to function effectively. For instance, using a workaround on the Quest, I created an AI+MR prototype for interior design. Without camera access, such innovation wouldn’t have been feasible.
But wait! Privacy is a major concern here. If a rogue developer gains access, they could secretly capture and analyze images, potentially sniffing out sensitive details like ID documents or credit cards left on a table. And let’s not forget the risk of unauthorized facial or body captures.
It’s a complex issue: balancing user privacy with the potential of advanced mixed reality features.
How XR Companies are Handling This
Initially, camera access was unrestricted, which allowed for exciting experiments. I recall experimenting with camera textures on the Vive Focus, leading to projects like diminished reality and marker tracking.
However, as mixed reality gained traction, companies started restricting camera access due to privacy fears. Meta, Pico, HTC, and even Apple were quick to block developers from accessing camera feeds.
Yet the developer community began pushing for access, arguing that if smartphone apps can access cameras with user consent, why not XR devices? This pressure yielded results: Meta announced a “Passthrough API” for early this year. But what’s Google’s game plan for Android XR?
Treating Android XR Like a Phone
Globally, Android dominates as the phone operating system, allowing apps to request camera access with user permission. Google aims to make Android XR compatible with these apps, ensuring a seamless start for Android XR users.
In a series of emails, Google confirmed this approach for XR devices. Users will still grant permissions for developers to access specific camera feeds, like the world-facing or selfie cameras. This setup mirrors traditional smartphone operations.
Developers can use familiar Android tools like CameraX on XR headsets to manage these streams, making way for innovations as they can analyze or save these feeds. However, while you can access the user-facing camera, the world-facing camera offers an avatar stream, aligning with Apple’s Vision Pro strategy, ensuring a consistent user experience across devices.
A Word of Caution with Android XR
It’s crucial to note that Android XR is still in the preview phase, and no official headsets are out yet. So, these plans could potentially change, though it’s unlikely.
The Road to Expanded Camera Access
With Google and Meta opening the doors to camera access, it seems other companies might follow suit. The coming years could be a golden era for mixed reality, unlocking vast new opportunities for developers. It’s an exciting time, and I’m eager to see what innovations will emerge from our community!
(Header image courtesy of Samsung)
Note: This blog contains ads and affiliate links to help sustain it. Clicking on an affiliate link will earn me a small commission. Check out my full disclosure for more details.
Feel free to share this with fellow innovators!
