Enabling the hardware for the metaverse

Article By : Nitin Dahad

The metaverse is much talked about, especially with companies like Facebook and Nvidia presenting their visions of the digital future. But what exactly is the metaverse, what are the hardware challenges in enabling it, and what elements are already in place? Analyst firm IDTechEx explored the progress in areas such as displays, image sensors, optics, haptics, augmented reality / virtual reality in some depth recently; we present some of that discussion here.

The endgame for the metaverse is for virtual worlds to exist seamlessly alongside the real, with immersive interactions between the two changing our perception of physical presence. The software for this is nearly there, but the hardware still has many hurdles to overcome.

The metaverse promises a step-change in how society communicates but, without the hardware technology, will ultimately remain a pipe dream. We will still be able to interact with the metaverse through our phones and laptops, but these will become legacy devices. In order to immerse ourselves, this requires a virtual reality (VR) headset; for true integration between the created and the physical, augmented reality (AR) devices are required.

AR/VR设备行业中的试金石是社会的可接受性，并且没有传感，显示和光学技术的进步，AR和VR耳机将不足以实现这一目标。尽管Idtechex表示，尽管在外观或显示具有IMAX质量的图像方面，但IDTECHEX表示，真正的AR眼镜仍然与射线篮板坐在一起。Meta（以前是Facebook）在更名的同时宣布了其AR眼镜项目，承认它们距离可行性是多年的。VR耳机有时会邀请与Robocop进行比较 - 只有最高端设备才开始模糊真实与没有的事物之间的界限。

Long term, the goal is a light and comfortable device that you can wear all day, switching between AR and VR whilst enabling natural interactions between yourself and other metaverse users. The technological journey towards the hardware needed to create this hypothetical device presents even more compelling developments and complex challenges than the software.

MicroLED displays are part of the solution

Putting a screen right in front of your eyes reveals things we do not notice whilst looking at a phone or TV. We see the gaps between pixels in a phenomenon called the screen door effect and the rule of thumb is that 60 pixels per degree (ppd) of field of view are required for VR or AR to start looking like reality, leading to big demands on resolution. On top of this, optics are required to focus and size these images correctly for our vision. In the case of AR, these optics are very inefficient, leading to brightness demands in the millions of nits – for reference, the iPhone 13 Pro Max screen maxes out at 1200 nits.

MicroLED displays are a promising solution for AR and VR. They do not suffer from burn-in like OLED displays, can have crazy brightness levels (JadeBird makes one for AR applications with a maximum brightness level of 3 million nits), and enable tiny pixel pitches, with Mojo Vision producing what they call a nanoLED display that is small enough to fit into a contact lens and has a subpixel pitch of 900nm.

However, there is one major issue: microOLED microdisplays are not good at producing full-color images, with blue microLEDs being significantly more efficient than other colors. Quantum dot color conversion is the favored solution here, enabling conversion of blue light to red and green, and these quantum dots can be inkjet printed or lithographically patterned. There are still concerns with longevity, especially in very bright microdisplays, as well as reliance on heavy metals in many formulations. The development timeline for microLED displays is outlined in detail inthis reporton technology, commercialization, opportunity, market and players.

基于“增大化现实”技术的最大战场,尤其是combiner optics. These devices overlay projected images on a transparent lens. Here, companies fight for the best color rendition, the widest field of view, and the largest eye box to enable a convincing display experience that works for every set of eyes.

Considering the big news this year in this area, surface relief waveguides seem to be the solution the industry is betting on. In May, WaveOptics, was acquired by Snap (another social media giant looking towards the metaverse) and in November, investment by such giants as Samsung Electronics in Digilens led to a valuation of $500 million. Both are fabless waveguide firms. An exciting development from Digilens is its TREX waveguide, which can double effective display resolution, providing one weapon in the arsenal of reaching the 60ppd.Another reporton optics and displays in AR, VR, and MR covers the technologies, players and Markets, and delves into optical combiner and display tech in these applications in detail.

Eye-tracking technology

If you tried to build a headset that covered all 135° of each eye’s horizontal field of view at 60ppd resolution, things would quickly get unmanageable – fortunately, only the center of our vision is of this high quality, with the outer edges not being so demanding. By tracking our eyes, resolution can be maximized in the center of where the user is looking whilst lowering demands elsewhere.

In the future, this eye-tracking tech may even be used to project AR/VR images directly onto the retina via laser beam scanning, getting around the need for combiner optics and correction for glasses wearers: that is, if consumers can get comfortable with the idea.

Companies in this space are usingemerging image sensor technologiesto track the eye more efficiently. Event-based vision can help keep processing demands down by natively recording movement instead of a stream of conventional image frames. Using printed image sensors, eye-tracking technology can be squeezed into a more svelte package. Meta Materials Inc. (no relation to Meta) is already embedding microcameras directly into glasses lenses and this approach will be integrated into combiner or magnifier optics for AR and VR respectively as the technology matures.

触觉和感官体验

到目前为止，所有讨论的内容都只不过是一台高端电视，如果您不能以类似于现实世界的方式与它互动。AR和VR设备不仅需要感知我们的运动，而且为了完全沉浸，还需要触觉（触摸反馈）设备。2021年11月，Meta的真人秀实验室（RL）部门展示了一种原型触觉手套产品，其中包括Mark Zuckerberg的视频进行了各种演示。该手套使用微流体系统向手的不同区域传递局部触觉反馈，从而向每只手指提供了进一步的触摸反馈。尽管该原型与HAPEX的产品相似，但Meta的IP位置在这里很强，它代表了以Metaversecopo的公司的努力来确保能够提供超过音频的感官体验。

这位分析师说，最近耳机的感应方面的关键获胜者是飞行摄像头进行手工跟踪的时间，消除了与VR和AR设备进行游戏控制器样式互动的需求。众所周知，苹果已在VR/AR空间投资多年，在2021年10月，行业消息来源报告说，LG Innotek已开始向公司提供飞行摄像头的时间，以在2022年发行的VR头戴式耳机。一种技术，通常是一个有力的陈述，即它将变得无处不在，代表了AR和VR的强大未来中的另一个数据点。

IDTechEx concludes to say that, in a limited way, the metaverse is already here. The hardware development mountain that prevents its full realization is slowly being climbed and light, good-looking AR glasses replacing our phones and laptops in the future feels like a near certainty.

This article was originally published onEmbedded.

Nitin Dahadis a correspondent for EE Times, EE Times Europe and also Editor-in-Chief of embedded.com. With 35 years in the electronics industry, he’s had many different roles: from engineer to journalist, and from entrepreneur to startup mentor and government advisor. He was part of the startup team that launched 32-bit microprocessor company ARC International in the US in the late 1990s and took it public, and co-founder of The Chilli, which influenced much of the tech startup scene in the early 2000s. He’s also worked with many of the big names—including National Semiconductor, GEC Plessey Semiconductors, Dialog Semiconductor and Marconi Instruments.