Unearthing Microsoft’s HoloLens feels a little like walking the decks of the Titanic. Four years ago, Microsoft’s augmented-reality headset ignited the imaginations of consumers and developers alike with its promise of lifelike animated sprites that could perch on real-world objects. Then… it sank.
It’s almost criminal that Microsoft’s original HoloLens demos never saw the light of day. Bending down to peer “inside” a coffee table into the Minecraft underworld was an utterly transformative experience. But at least Microsoft’s vision of using the HoloLens as a business tool apparently is alive and well.
As is the HoloLens. Because Microsoft never sold its device to consumers, PCWorld never formally reviewed it. But we have the original on hand. As Microsoft is expected to bring forth the next generation of HoloLens at Mobile World Congress (MWC) in Barcelona this weekend, we gave it another run-through, focusing on what made it best—and what could be better.
Mark Hachman / IDG
In this shot of the Microsoft HoloLens, you can see what Microsoft calls the “environment understanding cameras” that map the world around you, as well as the front-facing 2MP camera.
A consumer failure with corporate potential
As a consumer device, the HoloLens is dead. Only one Microsoft Store on the West Coast even has a HoloLens in stock for demonstration purposes. Microsoft’s remaining HoloLens supplies are sold through Insight, a solutions provider, Microsoft support reps said. On Microsoft’s site, the “Commercial Suite” version of the HoloLens costs a cool $5,000.
Mark Hachman / IDG
The HoloLens shipped in a hard plastic case.
But commercial it was not, back in 2015. Removing it from its cloth-covered plastic shell three years later reveals a lovingly engineered piece of consumer hardware. The HoloLens perches fairly comfortably on my head, supported by a removeable nose piece and my ears. Clicking a dial adjusts the tightness of the fit. There’s an optional head strap, too, if you’d prefer taking its 1.28-pound weight on the crown of your head. I wore the HoloLens pretty solidly for about three hours in one stretch. While I was conscious of the weight, I never felt uncomfortable.
Mark Hachman / IDG
The HoloLens audio is controlled by a pair of buttons on one side; you can control the brightness of the display with the others.
The HoloLens includes built-in speakers, which can be adjusted with a pair of buttons on one side. On the other, a similar pair of buttons controls the brightness of the display. An optional headphone jack is available too. Underneath the power button is a row of pinprick-sized holes, which light to indicate the battery level. (The HoloLens drains fast, and our unit provided about three to four hours of charge in fairly constant use.)
All that pales in comparison to the focal point of the HoloLens: the display. While virtual-reality devices like the first HTC Vive tethered you to the PC by way of a display and power cord, the augmented-reality HoloLens superimposes the display over the real world—understanding where physical surfaces are, and placing objects accordingly. You’re free to roam wherever you’d like. It’s magical, when it works.
Mark Hachman / IDG
A dial ratchets the HoloLens tightly around your head. I preferred the weight (1.2) pounds to rest on my nose and the brim, and from there my ears. An optional head strap can take some of the weight, too.
By now, the hardware underlying the HoloLens is fairly well understood, at least in broad strokes. In terms of resolution, though, the lack of a PC-powered display means that the HoloLens essentially provides 720p of resolution per eye. Combine that with the relatively limited field of view, and that’s where the experience gets complicated.
Want a brief look at what it’s like to actually use the HoloLens? Here’s a video using the HoloLens itself, taken with its built-in mixed-reality camera.
The first fix: Field of View (FOV)
HoloLens works its magic by suspending holograms, or what appears to be a three-dimensional object, virtually in front of you. There’s not that much difference between the image that a monitor paints on the glass in front of your eyes and the image that the HoloLens paints on each lens in front of you.
Mark Hachman / IDG
Standing about five feet away, the HoloLens field of view is about the size of this window. (The HoloLens front-facing camera is offset, with a slightly wider angle of view, so it captures more than you see.) The photos you’ll see here are shot with the HoloLens mixed-reality camera, overlaying holograms onto the real world.
The difference is what HoloLens does next. It uses a front-facing camera to scan the room in front of you, interpreting what it sees as a three-dimensional mesh, then attaches the hologram to that mesh itself. Using the HoloLens usually means anchoring various windows (such as Settings, for example) to a nearby surface, then using any three-dimensional open space near you as your HoloLens playspace. The illusion of physically walking around the hologram, as well as being able to anchor it or have it interact with real-world surfaces, fools your eyes into thinking it’s real.
Mark Hachman / IDG
The farther back you are, the more you can see. This is about eight feet away, against a different door.
That illusion is especially effective if the hologram remains in the field of view—the rectangle that serves as the HoloLens screen. The FOV covers much of the width of a door at a few feet, or about 50 degrees. Anything outside of it, though, can’t be entirely seen, and this instantly reminds you of the limits the HoloLens FOV imposes.
Mark Hachman / IDG
Back up far enough, and even large holograms like this Tyrannosaurus Rex can fit into your field of view (FOV).
Because the HoloLens “knows” where in virtual space a hologram exists, it’s possible to have one directly in front of you and another to the side or rear. It’s trying to turn and find the second hologram when frustration can kick in. (A setting allows you to trash all holograms loaded into your scene, even the ones you can’t see.) But the first-generation HoloLens could never convincingly offer up the illusion of, say, a forest, as the HoloLens lacks the capability to swamp your peripheral vision.
The FOV problem isn’t just a visual issue. Interacting with holograms and apps typically requires “air clicking” a hologram or window, in much the same way you’d click a mouse. Triggering the Start menu requires a “bloom” hand gesture, like a flower opening. In both cases, though, the HoloLens must be able to detect your gesture. (There are two alternatives—voice control and a small Bluetooth “clicker”—but gestures are the most common way of interacting.)
Mark Hachman / IDG
Holograms can either be “two-dimensional” Windows apps, or full-fledged three-dimensional objects. Using a two-handed gesture, you can place these windowed apps along surfaces, where HoloLens will remember them, even if you’re facing the other way. (The front-facing camera is only 2MP, so the resolution here is lousy. They look better in real life.)
Anyone who’s used the HoloLens, then, would rate the field-of-view as the primary area for improvement. If I were designing the next-generation HoloLens, however, I’d strike a compromise, offering users the option of dialing down the peripheral detail.
In the VR space, this is known as foveated rendering, concentrating the processing power on rendering detail where your eyes are looking. That’s probably a necessity for a new HoloLens, too—more on that in a minute. But I also appreciated the blank space at the edge of my vision, either to glance at my PC’s screen, a person, or something else. If Microsoft were to improve the HoloLens, I’d leave the option to “fade out” the periphery, providing a more grounded sense of place to the user.
The second fix: Resolution
It’s no secret that early VR users suffered from vertigo, caused by the latency between what the displays rendered and what the user “expected” to see. Though there’s still some noticeable render lag within the HoloLens, the fact that your vision tracks real-world objects helps prevent any nausea.
Mark Hachman / IDG
It’s a little easier to tell that’s actor George Takei when using the HoloLens, and an app called Actiongram. But if Microsoft is striving for realism, more resolution is a must.
I still experienced discomfort. Dialing up the brightness of the display is a must. Even then, however, holograms can be low-resolution, and sometimes laggy when interacting with you. After an hour or two, my eyes ached from the strain. A reported processor spec bump must happen to ensure a next-gen HoloLens will be comfortable to use.
The resolution isn’t merely a visual issue, either. HoloLens maps your environment, applying a very coarse surface mesh over walls, desks, tables, and more. (Objects can be “anchored” to a surface like a desk, and certain games will actually use these surfaces as part of the experience.) A spartan workspace is easily accommodated. But as your environment moves further away from minimalism, HoloLens struggles to keep up.
My basement office, full of monitors and boxes and peripherals and lamps, would probably give organizational guru Marie Kondo the cold sweats. To compensate, HoloLens would occasionally draw a big black surface that would interrupt the scene. A next-gen HoloLens would be well served by increasing the number and density of the three-dimensional mesh of points it uses to creates a virtual space.
The third fix: UI
Certain elements of the HoloLens UI haven’t held up over time, including the “nose first” method of navigating. Too much navigation involves steering a cursor from point to point by slightly moving your head. It’s especially painful when you have to enter a Windows or Wi-Fi password: Manually pushing the cursor from letter to letter is a major pain.
Mark Hachman / IDG
Navigating the Start menu through the bloom gesture is fine. But when you have to navigate, letter by letter, to spell out a password—ugh. (Again, the HoloLens camera is awful, but there’s no way to take a photo without it.)
Microsoft’s first-generation HoloLens works around this somewhat through voice recognition, where many simple tasks, such as opening the Settings menu, can be performed by merely saying a word, like “settings.” But modern VR equipment is beginning to include eye tracking, which can save time navigating from element to element. If Microsoft is developing a next-generation HoloLens, it would be a shame if eye tracking were excluded.
The HoloLens voice recognition works well, but even it can be unnecessarily confusing. Some commands can be performed simply by saying the command; others require you to invoke Cortana beforehand, which makes little sense. Still, I found I relied on voice more and more to avoid tediously steering the cursor.
Mark Hachman / IDG
This little gizmo is the HoloLens “clicker,” which you can click to input choices, or “twist” to scroll. The important aspect of the clicker is that you can click without the HoloLens camera seeing your movements, which is far more comfortable than simply air-clicking all of the time.
Lifting your hand to constantly “air click” gets old, too. But scrolling is the worst, requiring you to air-click and drag through the minimal field of view. Microsoft’s optional “clicker” basically acknowledged that air navigation failed. But couldn’t Microsoft have tried something a bit different, such as “clicking” by audibly snapping your fingers?
The fourth fix: More apps
If you think the Windows app store looks empty, you should see the HoloLens aisle. At one point, UWP apps like Twitter were considered to be the future, traversing PC, phone, and HoloLens alike.
Mark Hachman / IDG
It’s just another hologram, but opening the Rome tour on Microsoft Studios’ own HoloTour via this painting is a great start, especially as it seamlessly works its way into…
You can exhaust most of the best HoloLens apps in an afternoon. HoloTour offers genuinely interesting 360-degree recorded video tours of Machu Picchu and Rome, complete with a recorded tour guide. Step back, though, and they aren’t much different from a HoloLens-specific version of Myst, as you virtually explore hotspot after hotspot. Holo Galaxy isn’t much better, with a cursory 3D view of the solar system and other stellar points of interest.
Mark Hachman / IDG
…A 360-degree video tour of Rome. Yes, you’ve probably seen 360-degree video before. But it’s all nicely presented, with a friendly, informative narrator.
RoboRaid, jaw-dropping in its time, creates virtual bugs and drones that literally crawl out from your HoloLens-mapped walls. You’ll need to use oral commands, your air-click skills, and the ability to hop nimbly about the room to avoid laser blasts. It’s still a lot of fun, but not too different from the “shooting gallery” games that tend to be bundled with VR gear.
Mark Hachman / IDG
In RoboRaid, robotic bugs and drones crawl out of the walls, firing laser blasts, which you must physically avoid either by ducking or moving out of the way. Air clicking returns fire.
Little Conker may be the best showcase for the HoloLens’ talents. After scanning the area around you, the HoloLens and Conker uses your environment as a level for this platformer. It’s not especially hard, but it’s still pretty neat to turn your desk, chair, couch, and other furniture into a virtual playground where the demented squirrel can hop up and run around.. (You can save what it calls your “playspace” for future games, so that you don’t need to re-scan it again.)
Mark Hachman / IDG
Asobo Studio made both Fragments and Young Conker, both of which show off the surface mapping capabilities to best effect. Here, Young Conker literally makes the stairs part of the platforming level.
Login issues meant that I didn’t have a chance to spend as much time with Fragments (video walkthrough, with spoilers), another mystery game that uses the HoloLens environment mapping to full effect. But Fragments actually has a plot, as the user navigates though “memories” to solve crimes. Some have called it a “hidden pixel” game, and that’s probably accurate. But it’s still one of the best experiences on the HoloLens.
Mark Hachman / IDG
While other HoloLens games are probably fine for kids, Fragments takes a darker tone by asking you to solve crimes.
These apps—RoboRaid, Little Conker, and Fragments– showed off the HoloLens augmented-reality and surface-mapping capabilities to best effect. Still, HoloLens never lived up to its entertainment possibilities. Microsoft showed off Minecraft several times as a public demonstration of HoloLens’ potential, but never delivered. (That’s not surprising for a device that ended up as a business tool.) I’ve always quietly hoped for a Dream Park-like experience mixing the real world and augmented reality, but that probably will never be. In all, though, you can’t help feeling that there was untapped potential.
A HoloLens 2 wish list
Is/was the HoloLens a failure? From a consumer perspective, yes. Even from its new perspective as a business device, Microsoft reportedly has sold only about 50,000 devices, according to Bloomberg. But that same story trumpets a $480 million contract Microsoft signed with the U.S. Army to develop HoloLens further for the battlefield, and that’s not chump change. Microsoft’s also hedging its bets by developing its Dynamics 365 Remote Assist tool—a fancy name for the original remote-assistance demo it showed off at the HoloLens launch in Redmond—as an Android app.
If I were creating a wish list for the HoloLens 2, it would be fairly straightforward:
A wider, higher-resolution field of view. Lag isn’t as critical here, but Microsoft needs more detailed, immersive holograms—especially if it plans to serve a business audience working with CAD projects, detailed wiring, and more. Microsoft at least had to consider a high-speed wireless link, such as the WiGig-powered connection HTC began talking about in 2017 and may have included in the Vive Cosmos. Whether such a link is built into a new HoloLens for additional computational horsepower is an intriguing possibility.
A lighter headset. A pound of headgear wasn’t that fatiguing for me, but lightening the load would accommodate more users for a longer length of time.
A UI that minimizes text input. You’ll never write a paper using the HoloLens, or probably put together a sales presentation, either. But steering the cursor across a HoloLens virtual keyboard is a miserable experience. Logging in using biometrics, or a secondary phone, and supporting automatic pairing technologies is a must. Dictation works somewhat acceptably.
Better integration with the cloud. Normally, photos synced on a Windows PC can be automatically uploaded to OneDrive. But to pull photos from the HoloLens, I had to manually download the OneDrive app, then copy photos from the device to a folder. the Photos app should have automatically synced every image I captured. It didn’t.
More apps. Yeah, it’s a pipe dream. If Microsoft is aiming for the business market, expect to see a lot of PowerBI tools for visualizing data, rather than RoboRaid 2. But we can dream, right?
We know Microsoft has something up its sleeve for Barcelona. The fact that HoloLens inventor and Microsoft technical fellow Alex Kipman is scheduled to appear is a good sign that a next-gen HoloLens is in the offing. Here’s Microsoft’s teaser for its event, scheduled for Sunday, February 24:
Though PCWorld will be attending, we’re not sure how much time (if any!) Microsoft will allow us with any new devices it shows off. HoloLens has generally remained behind Microsoft’s curtains, and it looks like it might stay that way.
There’s no shame in designing business-facing hardware. Products like the Surface tablets and laptops, after all, can be used by consumers and business users alike. Here’s hoping that Microsoft learns its lessons from the past, though, and delivers a device that will wow us all over again.
Correction: Microsoft publicly debuted the HoloLens four years ago, not three. Updated at 11:25 AM on Feb. 22 with video.