We recently got a Samsung Galaxy XR headset. We wanted to understand the Android XR ecosystem firsthand, not just read about it, but actually build something for it.

Then the Gemini Live Agent Challenge showed up at exactly the right time.

We made Pack Pro, a hands-free AR packing assistant. Before you start packing, you tell it where you're going, how long you'll be travelling, and what activities you have planned. It uses that context to suggest a smart packing list tailored to your trip. Then when you're actually packing, you point your headset camera at your luggage, talk to the assistant, and it watches what goes into the bag in real time, tracking what you've packed and reminding you what's still missing. All voice-driven, overlaid on the real world.

Packing is one of those tasks that's surprisingly stressful for how mundane it is. You forget things, second-guess yourself, and either overpack or arrive somewhere missing something obvious. Having an assistant that can both help you plan what to bring and actually watch you pack felt like a genuinely useful thing to build.

We spent a good chunk of time just ideating. Musical instrument learning in AR was one idea we liked, where the headset could overlay guidance as you play. We also explored an AI meditation guide and a gardening assistant that could identify your plants and tell you what they need. All of them were interesting in their own way, but none felt quite right for this particular challenge.

We kept coming back to packing because it has immediate, practical utility and it's a perfect excuse to push vision models in a real context. You're already looking at your bag while you pack, so having the AI look with you through the headset makes total sense.

Since we were new to Android XR, we weren't sure what platform to use. We looked into Android Studio but found it cumbersome for a short hackathon. Then we came across xrblocks, Google's WebXR component library, which had a Gemini Live template that let us get something running quickly. We built Pack Pro on top of that.

The core stack ended up being xrblocks for the WebXR layer, the Gemini Live API for real-time multimodal AI (it handles audio, vision, and text simultaneously), and Firebase for deployment.

We usually work in Unity, where CLI coding tools don't fit as well since so much of the work is in the editor rather than in code. WebXR is just a codebase, so we decided to use Claude Code, especially given the time pressure.

The first attempt was a mistake we probably should have anticipated: one big prompt asking it to generate the whole app. It produced something, but figuring out what was broken and why was painful. There were too many moving parts with no clear starting point for debugging.

So we scrapped it and started fresh, one component at a time. Get a piece working, verify it, move on to the next. This approach worked much better. Claude Code was good at thinking through the architecture upfront, figuring out how the pieces should fit together, and flagging potential problems before they became real ones. When we hit confusing behaviour from the xrblocks library, it would dig into the source and come back with an explanation rather than just guessing. That saved us a lot of time on a codebase we'd never touched before.

One thing worth knowing if you're using Claude Code: turn off thinking mode for straightforward tasks. We were letting it run deep reasoning on things that didn't need it, and it was just slow. Reserving thinking mode for genuinely complex problems made the whole workflow much faster.

The trickiest part technically was getting function calling to work well with the Gemini Live API. Pack Pro relies on function calls heavily since the assistant needs to build and update the packing list, mark items as packed, and move between different phases of the session. Two problems came up repeatedly: the model would sometimes narrate what it had just done after a function call, which got repetitive and annoying, and function calls were occasionally making responses noticeably slow. Both required a lot of iteration on the system prompt to sort out.

There were also some Gemini Live API configuration quirks that took time to figure out, mostly around how parameters need to be structured in ways that aren't obvious from the documentation. Not blockers, but the kind of thing that eats an hour when you're trying to move fast.

A working demo app. You put on the headset, have a short conversation about your trip, and the assistant builds a packing list based on your destination, the weather, how long you're staying, and what you'll be doing. Then you switch to packing mode, point the camera at your bag, and it watches. It confirms items as you pack them and reminds you what's left.

It's a hackathon demo, not a finished product. But it works end to end, and it's a pretty clear demonstration of what this kind of experience could be.

XR Blocks is genuinely worth considering for XR prototyping. If you want to build something for a headset quickly and you're comfortable with web development, it gets you there much faster than going native. And it comes with a lot of useful examples and templates.

One-shot prompting doesn't work for complex apps. Build incrementally. It's less exciting but you'll actually ship something.

AI coding tools are especially useful when you're in unfamiliar territory. The value isn't just writing code faster, it's being able to explore a new framework without getting stuck every time you hit something undocumented.

The Gemini Live API is impressive but needs careful tuning. Real-time multimodal AI is genuinely powerful. Getting it to behave consistently, especially with function calling, takes time and experimentation.

The packing list right now is a floating panel, which works but doesn't take full advantage of what XR can do. We want to add spatial indicators, visual overlays showing where items should go in your bag, and use eye tracking so you can glance at an item to mark it. That's where it starts feeling less like an app and more like an actual spatial experience.

Pack Pro is our first project on Android XR. We're planning more.

Try it out in an WebXR supported browser here: https://pack-pro-firebase.web.app/?key=YOUR_GEMINI_API_KEY