Making Magic Leap One Great

Disclaimer: I used to be a Director of Product at Magic Leap, I haven’t worked there in ~2 years, and I did buy my options/have a solid desire for/interest in the company’s success.

I’ve been using a Magic Leap One daily for over a week. I’ve been using it as a customer and not a developer — trying to make it part of my life — and I wanted to share my thoughts, especially focused on where I want to see it improve. I know this is a Creator release, but as an early adopter/AR advocate/tech product person, I want to use it. While I know some of how the sausage is made, I’ll try hard to ignore that given my knowledge is out of date, and at the end of the day, customers care about the result, not how you get there. Finally, apologies in advance for the bad screenshots. Screenshots don’t accurately reflect where you’re looking and there are no visual guides to help you line them up.

Mixed reality is a really amazing experience. When you put on a headset and see holograms — for lack of a better word — it is just magical. A 40-some year old British friend of mine was actually giggling playing Angry Birds in my living room. ML1’s boot up experience where you have a very cute, small world floating in front of you with a jumping astronaut, hummingbird, and more is one of my favorite parts of the experience. It’s a delightful scene that makes me smile each time I boot the device. I’ve had a lot of fun playing with ILMxLAB’s Project Porg experience each day: it’s really neat sitting on my floor and seeing these two virtual porgs walk up to me, make eye contact, and more. It’s easy to see the potential and where mixed reality is going.

But let’s talk more about where it is now, starting with the hardware. ML1 looks and feels quite premium. The unboxing was beautiful. The headset is quite comfortable. Magic Leap did a great job distributing the weight on the headset. The longest I ever wore HoloLens 1 was 40 minutes, and my nose hurt so much that I still cringe to think of wearing it. I wore ML1 for hours one day, and while my nose felt a little pinched at the end and a bit of dry eye, it was quite comfy. (Aside: I’m curious to see how the upcoming nreal headset is for long-term use, weighing 85g like very heavy glasses but with all the weight on your nose.)

People often focus on the field of view of all Mixed Reality devices. Magic Leap One is about 50° diagonally. Sure, it’d be nice if it were bigger, but it’s quite good, especially when coupled with thoughtful design that works with that limitation. Full-scale things that appear in front of me are immediately clipped, making me aware of that limit; e.g., the elephant demo that starts with a near-full-size elephant in front of me. But when content is smaller and designed to fit within the bounds, you don’t notice the clipping plane, such as when you cup your hands in the elephant demo and see it appear in your palm.

An elephant in my hands doesn’t make me notice the clipping plane :) The full-sized one does. I tried twice to get a more centered screenshot also looking down more so you could see my hands. I failed.

Now the bad, starting with image quality. The angular resolution is too low. While this doesn’t affect games, it hurts daily use cases, like web browsing. Text is only legible when it’s close. Similarly, photos from Gallery looked splotchy and bad unless they were near to me.

Wow, the rat’s nest of cables is just awful. I had everything on the floor charging, and it looked awful. Packing ML1 up to carry is painful because of all the cables. The stiff/big cable on the Lightpack is really annoying to deal with, and especially when I’m using the strap to carry the Lightpack — which I switched to after the Lightpack fell out of my pocket while playing with porgs — the cable just always seems in the way.

The rat’s nest of cables involved in charging ML1.

Finally, the lean-back experience was physically unusable. I tried leaning back on my couch, and some mix of the stiff cable and the pressure on the back of the headset just didn’t work: the headset slid all around and nothing was comfy. This kills the whole category of “get comfy and watch video” use cases for me because I can’t get comfy.

What would I want to see in ML2? In priority order:

• Higher resolution. Make text more legible and images look better.
• Solve the cable hassle. Whether the Lightpack/headset go wireless or just have smaller/more flexible cables, make this better. Especially if you let me lean back in a chair while wearing everything. While less important, an integrated control/Lightpack charger would free up a plug.
• Enable a lean-back design so I can chill on my couch or sit in an airplane seat
• Remove the soap-bubble effect. This a funky effect where everything has weird “bubbles” around it that shift color and change slightly as you move your head. It’s especially noticeable with text and fields of color.
• Increased field of view. But not at the expense of resolution.
• Brighter display. The tint on ML1 is fairly dark — not unusably so but not ideal — but even so, when I was looking towards a window, the image became fairly transparent.
• Make it smaller and lighter. I’d like it to be easier to take ML2+ with me.

Notice what’s not on this list? 5G. 5G works well outdoors, but it doesn’t work well within buildings (or so a person who runs a 5G lab told me — the Verizon 5G router has to be by a window). The first 3 items on my wish-list have to be solved before I’ll want to take/use ML* regularly. And even then, I would be taking it to places where I have a Wi-Fi signal available. I think it’ll be a while before this will be a use-everywhere device.

The Software

Let’s switch gears and talk software. I’ve downloaded/used a lot of the apps available on Magic Leap World but haven’t purchased any of the paid apps yet, and I can’t use things I don’t subscribe to, like Direct TV. Let me focus on the 3 topics that affected my experience the most, all of which I think are solvable via software updates.

Clipping plane

It’s really cool that ML1 has depth planes and creates less of a vergence–accommodation issue than other devices. Maybe it’s part of the reason why I could use ML1 for a couple hours comfortably with no eye strain. However, the software is artificially limiting the clipping plane to about 15” away. This makes it so when the control becomes a ray gun or a slingshot and you hold it at a natural distance, you see clipping. Or when you walk up close to something, it clips. Please turn this limit off. I get that it creates eye strain. But it’s a more pleasant experience to have a second of eye strain when I hold up a slingshot than to see it clip and break the experience.

The clipping on near objects breaks the experience. (Also, sorry for the awkward angles on screenshots — they don’t precisely match what I see through the headset, including the field of view.)

SLAM/Mesh Generation/World Awareness

Mixed reality relies on a device’s ability to quickly and accurately make a digital copy of the real world’s geometry — a 3D “mesh” — and then to know where you are within it. This is why many experiences start by having you look/move around, often showing a grid to indicate what you’re scanning and what the mesh looks like. (The Game of Thrones experience has a really cool UI to hide the mesh generation processes.) It’s impressive what ARKit/Core, 6d.ai, and others are achieving in this space with a singular smartphone camera. Given how many sensors ML1 has, I would’ve expected this mesh generation process to work really well. Especially since Magic Leap’s released a beta of their Shared World (ARCloud) system to let users share meshes, that would imply a lot of confidence in the meshes they’re making.

But in my experience, the mesh generation algorithm needs work. It had a lot of issues with walls, intersection of furniture and the floor, and even certain areas on my matte hardwood floor. My dull metal, flat coffee table didn’t work well, the books and other stuff on my coffee table created weird artifacts, my TV didn’t at all turn into a flat object, and my shiny wood piano’s flat top completely confused it.

One of my most frustrating experiences related to mesh generation was trying to start Dr. Gordbort’s Invaders. I went into my home office, which is the simplest room in my apartment (decent size, open space, 4 distinct walls with texture). I started scanning, pushing the virtual buttons Invaders creates to get you to move around the room, and everything seemed fine. Then the “start game” button appeared. As I went to push it, it went away and I got a message saying I had to keep scanning. This happened 9 times: I started counting after 3. I actually ran to press it when it appeared the 9th time. My door was open the first couple times, so I closed it, but it kept happening. After that it worked pretty well, but I think Invaders still included the hallway beyond the closed door as part of the mesh, creating gameplay issues.

I brought ML1 into work to show people and tried setting up Invaders in a small room. It had a lot of trouble creating a mesh of the frosted glass wall, and whenever I looked at the open, white wall corner area in the room, it’d lose head pose, which is how the device determines where in space it is.

The most annoying thing, though, was that apps never seemed to share meshes. I’d mesh a room in Landscape, and when I launched Porgs, I’d have to re-scan. If I switch to Angry Birds in the same room, I’d have to re-scan. I’d really rather apps share the same mesh for the spaces I’m in and continuously update in the background. And I’d like to see that before any mesh sharing, as no one around me has an ML1, so mesh sharing is a non-issue.

The User Interface

Let me define 3 key terms before we discuss the UI. Magic Leap has two types of applications, “immersive” and “landscape/Lumin.” Immersive applications are built in Unity or Unreal and take over the entire environment (hence immersive). Landscape apps are built within Magic Leap’s native Lumin runtime and, while more limited from a 3D capability sense, they support persistence and multitasking. Within Landscape, content lives within “prisms,” a fancy term for a 3D window/content volume.

While some immersive apps especially have well-done interfaces and interactions, such as Invaders’ using the Control for the ray gun and holding up your hand to create a shield, ML1’s overall user interface is an inconsistent and frustrating mess. Some general criticisms:

• Lack of feedback, especially on boot. When I first booted ML1, I heard the boot sound…and then nothing happened. No sound, no image, no nothing. I wasn’t sure if it was broken. After a long while, an image showed up. Whenever there’s a long process, there has to be clear feedback something is happening.
• Quitting immersive apps is inconsistent. In some, the home button pulls up a menu and there’s a Quit option. Others don’t have this and you have to force-quit the app (press and hold home for a few seconds). But that seems to suspend the app, not quit it, as sometimes the state resumes when you reopen it rather than restarting? Magic Leap’s support page says this minimizes an app, but there’s no minimization visual indicator, like the Genie effect when something minimizes into the dock on macOS. Minimizing an immersive app looks like you’ve exited the app. And typically in products, a long press is used to force-quit/reboot something, not minimize it.
• The Home button is inconsistent. Sometimes it pulls up a launcher. Sometimes in Landscape apps, it’s a back button. Sometimes in immersive apps it pulls up a menu. It’s confusing. An important button like that should have a clear purpose — do whatever is needed to take you home and pull up the launcher. Think about how on the iPhone, from day 1, the home button always took you home.
• Selection is inconsistent. Once you’ve picked an action/button/object, sometimes you tap to confirm selection. Sometimes you use the trigger. In one MagicKit experiment, you use the bumper to select.
• Inconsistent multi-modal integration. ML1’s inputs include eye gaze, head pose, voice input, gesture support, and control (6 DoF plus trackpad). But really you don’t want to use them one by one, you want to combine these input methods to create a more “natural” interface where things just work. ML1 almost never does this, as far as I can tell. The only place I could find voice input was in a keyboard. I found I awkwardly had to be quiet after entering the text because I then had to use the control to select a done button before voice input stopped listening. (My Apple TV is so much nicer where I push to speak and let go when done.) The only example of gesture support I found was in Invaders where you hold your hand up to be a shield. Prism placement in landscape is technically multi-modal with head pose and using the trackpad, but it was so awful and hard to use (more shortly), it’s a steaming pile of garbage, not a multi-modal interface.
• Direct vs. Indirect Manipulation. Some interfaces, like pushing a button in Invaders, are a direct manipulation where you walk up and push a button. This is incredibly satisfying and easy to do. Most of the interface is indirect, where you use the control like a laser pointer. I get the need for indirect manipulation and the convenience if you’re sitting down, but the controls to create that indirect manipulation are often incredibly unintuitive. For example, when moving prisms in landscape, doing different actions on different parts of the trackpad do different things (rotation, scaling, moving in/out). I can never remember what does what, and that small surface area makes my thumb hurt. There needs to be better guidelines and consistency around when to do direct vs indirect manipulation, and if it’s indirect, how do you make it obvious? Heck how about combining the two — if I’m interacting with a remote object, put controls in front of me, possibly projected onto a physical surface so that I get real-world feedback when I press them?
• The on-screen keyboard is barely usable. I could never type anything successfully and only used this to get to the voice input button. Otherwise I used the Mobile App for text entry. Heck I tried setting a pin code with the limited on-screen numbers keypad and gave up when I couldn’t successfully enter the same 6-digit number twice.

Ironically given all the ways the UX is inconsistent, one of the top 3 best practices in Magic Leap’s User Experience/Interaction Model Guidelines is to “be reliable,” aka consistent. However given the third most important tip about building for spatial computing is “Be mindful. Empathetic design is socially conscientious,” I suspect many devs — including those inside Magic Leap — dismiss this doc as fluff.

This was unfortunately the best screenshot I could get :/ I had a CNN screen on the far wall. I wanted to move it. I pressed the bumper button on the Control, which the UX guide says is supposed to bring up the placement controls and does on other content within Screens. CNN spawned a new window right in front of me. I tried to point at the X button to close it and failed to select that button. I looked off the window and pressed the bumper again, seeing if focus was the issue. Bam, another window appeared. Ugh. This is a mess. I can’t figure out how to close these windows (or make them actually load/play video).

Let’s talk about Lumin/Landscape and the UX within it. Landscape is an incredibly smart strategic play. What we want with AR is when you put on the headset, you see a new, persistent world. You see your content, whether it’s a photo, browser window, or more at the same time. You want persistent multitasking, and that’s what Landscape provides.

I had a true magic moment because of Landscape. Without thinking about it, I’d put a photo on my wall in my living room and had another prism open by my couch. I’d then walked to my home office to open Invaders. When I was done, I quit Invaders and started to walk down my hallway back to my living room, and those prims were right where I’d left them. That moment convinced me Landscape is the foundation of the future and a unique competitive advantage.

However, I have two big criticisms of how it’s used in ML1. First, most of the Landscape apps feel like 2D screens that just float in 3D space. Gallery, for example, just floats there. Same with Helio (the web browser). I really wish ML had taken the time for their shipping, showcase apps to focus on how Mixed Reality, spatial computing, and 3D makes something better for the customer and build that experience.

This doesn’t just mean making the entire UX 3D, which is also not a good thing, but rather really thinking about how to leverage having 3D space available — the premise of spatial computing — to make an interaction better. A simple example of really using space would be if rather than a flat grid, what if Gallery took advantage of depth and had stacks to provide an extra layer of organization? Madefire is a nice example of how depth can really plus an experience with 2D content. Sure, some of the apps have 3D icons and a tiny bit of depth, but it doesn’t make the UX better; it just makes it shinier.

Second, prism management — a fundamental task — is painful. To engage it, you look at a prism, press the bumper, and choose move. Then, your head pose sort of moves the volume around, but it lags a ton and feels imprecise and unintentional. As I tried to place an extracted photo from Gallery onto my wall, which seems like a key task you’d do with Gallery, the prism moved as if it were drunk while I stared at the wall, trying to coax it there.

To further place the prism, you press different sides of the control’s trackpad to scale, rotate, and move the prism closer/farther. Those interactions didn’t feel linear; it often felt like there was acceleration/deceleration when you press. This made it hard to get precise adjustment.

The lack of snapping to surfaces — as far as I could find — made it even harder to get a prism where I wanted it. It took me a few minutes to position a photo on my wall, and even then it wasn’t flush nor totally centered. It was bad. It was so bad, I didn’t want to do it again. So I ended up with just one photo sort of placed on my wall.

It took far too much effort to put this virtual photo on my wall. Yes, it’s intersecting the light switch. It was so hard to place it, I’ll live with that overlap. Taking this screen capture was slightly awkward, too, as the counter didn’t show properly. Sorry.

Another example of when prism management becomes frustrating is in Helio, when I want to move a window a little closer to make it more legible (there’s a limited range where text is legible). After pressing the bumper, targeting move, and selecting it, I then have to press the bottom of the trackpad to move the prism closer. It accelerates quickly and comes too close towards me. So I press the other way. Now it accelerates too far away. It took a lot of yo-yoing before I got it near where I wanted. Geesh. If window management had been this bad in Mac OS 1.0, I don’t think a GUI would’ve caught on.

Looking more at Helio, I’ll say that my experience was pretty bad, but it might have been mitigated by pairing a Bluetooth keyboard/mouse. But I only had the included Control and the Mobile App.

When you’re within a Helio prism, you get (essentially) a mouse cursor that you move by swiping on the trackpad surface. Within a web page, this cursor is really hard to use as it just feels completely inaccurate. It feels both very slow and imprecise at the same time, which is odd because usually a slower mouse makes it easier to be accurate.

I loaded CNN.com and tried to pick an article. I missed it and it opened a different one. When you mouse to the top of the web page to pick Back, the cursor disappears and now you have more traditional targeting — tracking at a different speed. The back button was hard to hit. I finally got it and moused back onto the homepage. I picked the right article, but then a video started auto-playing. It was really hard to target/hit the pause button. I gave up reading the news after that and tried Twitter, since there’s less clicking and mostly scrolling.

To scroll within a page, you squeeze the top/bottom of the control’s trackpad. I also found you can’t just start/resume scrolling. You have to move the cursor slightly and then scroll. If you let up, you have to repeat this. (That feels like a bug.) But I really don’t think anyone thought about how much you scroll browsing the web when picking squeezing as the scrolling trigger. Using Helio made my thumb hurt. Constantly squeezing something is no fun. Twitter was practically unusable both because of the pain of scrolling but also because the page seemed pretty sensitive about where the cursor was/when I could scroll. If it was over a tweet, it wouldn’t always scroll. If it was off the timeline, it wouldn’t scroll. And so on.

At some distances, the text isn’t too bad. The inverted/dark mode in Helio helps with legibility but is inconsistent, inverting some images but not others. My profile photo isn’t inverted, for example.

It was really satisfying to successfully manage to get to YouTube and play the new SpiderMan trailer! The video looked surprisingly good. But when I tried to use Netflix (ugh logging in was a complex process involving the Mobile App and my password manager), while the previews played, the video itself wouldn’t play.

Finally, let’s talk about Avatar Chat/Social. I couldn’t actually use this because I didn’t have anyone to talk to. No one I know outside some ML employees has ML1. (And frankly if I really wanted to have an engaged, synchronous conversation with my friends who still work there, I’d rather use FaceTime so I could see all their nonverbal expression, but usually we just talk on the phone or text.) The app seems to have more features for “troll control” to stop people from talking to me than ways to find people to talk with.

So I tried playing with the avatar editor. This editor feels like someone went “editing avatars is fun, so let’s give you every possible control.” The first time I tried making my avatar, I saw the controls for eyebrow tilt and said “this is too much to do right now.” The second time, I had the same reaction when I saw chin width. I have no clue what my eyebrow tilt and chin width look like. The third time I just chose to ignore those fine controls and try to get my avatar in broad strokes, and yet in the places I wanted to use to do so — like hair style — there were so few choices that I ended up with an avatar that doesn’t really resemble me.

This editor is a bad mix of choice fidelity. If you want to give me detail control, you need a quick way to set a good default. Have me take the headset off, point it at me, take a photo, and pre-define my avatar from that. If you don’t want a fast default, give me fewer controls with a limited set of choices that give me something close enough. Right now, this variable level of fidelity made me just feel like I failed to make an avatar.

I’ve watched videos of the final version of this, and it feels a lot like a copy of Facebook Spaces. Except applying the same space management from VR to AR seems wrong. Specifically, within the bounds of a chat, Avatar Chat tries to define the same spatial relationships between our spaces — if my avatar is to your left, I’m actually standing to the left of your avatar. This works in VR because we both go to a shared, virtual space. But in AR, our physical environments are different, and this leads to funky results. As the one review of Avatar Chat I found noted, there’s a lot of asking the other person to move to match your space, and avatars disappear behind walls. It’ll be interesting to see how this scales when you cast apps (I couldn’t find real-world video showing it).

OK, that’s a lot of stuff. It’s more than I intended :) I know some of this criticism is harsh, but there’s also a lot Magic Leap should be proud about. There’s a lot in ML1 that makes me smile. That feeling reminds me of the original iPhone. But when you try to use ML1 as more than a demo — unlike the original iPhone — ML1’s software and especially its UX falls flat at best and is frequently maddening. However, the nice thing about software is it’s a lot easier to update than hardware! With a software overhaul, ML1 could really be a satisfying product and set the stage for the future.

📝 Read this story later in Journal.

👩‍💻 Wake up every Sunday morning to the week’s most noteworthy stories in Tech waiting in your inbox. Read the Noteworthy in Tech newsletter.