October 25, 2019

The Real Driving Force Behind the Success of the Notch and the Card and How to Use It

What is the first rule of the tech industry? Be ready to break the rules. As ubiquitous as the word “innovation” is in marketing materials and articles, the reality is innovation has always been essential to remain relevant, despite its risks.­­­

The best example that comes to mind is the controversial move the tech world witnessed two years ago. After several years of design tweaks and software upgrades, Apple relinquished its smartphones’ signature look composed of thick bezels and a home button in favor of an almost “edge-to-edge” display with no chin but featuring the Notch.

That deceptively rash design call had many Apple users questioning their allegiance to the brand, while Apple’s peers looked closely at what the notch kept hidden in plain view: a face recognition system. By dismissing Touch ID, the company’s face recognition solution had been pushed at the forefront of consumers’ minds.

There it stayed for the following two years, during which competitors’ devices running on Android adopted a very similar housing for their front cameras, with the more ambitious trying to give their own spin to the notch.

Increasing the screen-to-body ratio by opting for a notch became the sole goal… or did it? Could Apple’s plan for their Notch go beyond unlocking your device more securely?

After two years of building trust around their 3D face recognition system to the point where today, the notch is seen as a brand staple more than an inconvenience, I believe the company has revealed its long term plan. It all makes sense once we think of the ecosystem Apple has always sought to strengthen and that now includes a new financial component.

Where does that leave their peers who have also adopted the notch to offer face recognition? They will have to chase their own promising venues. In that sense, Xperi can not only provide a cutting-edge 3D face recognition solution but help them find use cases for it beyond face unlock.

Notch 2

How the notch broke the Internet

By the end of 2016, Xiaomi had established itself as a smartphone powerhouse that was not afraid to innovate. The Mi Mix phone, created in collaboration with a French artist, caused quite a stir with its 91.3% screen-to-body ratio, effectively abandoning top and side bezels, leaving just a bottom one to house the front camera.

Mobile manufacturers’ primary goal quickly became offering users more screen real estate. The next to push the envelope in that direction was Essential, a startup led by Andy Rubin.

Rubin’s phone was inspired by the Mi Mix but it brought a unique take on the bezel-less trend. The smartphone’s display wrapped around the front camera, showcasing the first unofficial notch.

Meanwhile, in Asia, Sharp had unveiled the Aquos S2, a handset with a Free Form Display that, similarly to the Essential phone, covered the front almost entirely, swirling around the selfie camera and the underlying earpiece.

Neither got the public’s attention for long, resulting in disappointing sales and ultimately, in Rubin’s decision to give up on a sequel. Unknowingly, however, these phones cast a glimpse into the future that was going to be shaped by Apple later on.

During their fall event, Tim Cook and his team unveiled the iPhone X, a phone that eliminated the bottom and side bezels, yet kept a portion of the top one called the Notch. While it was significantly different than their previous iPhones, suggesting they had considered their users’ wishes for change, Apple received quite the backlash.

The notch was disruptive, even if it was necessary. There stood the phone’s front camera with its 3D face recognition solution, sensors for proximity and ambient light, a flood illuminator and IR camera, besides the microphone and speakers.

From an engineering point of view, the design made sense. But consumers are not engineers. With its peculiar design and steep price point, the highest set by a smartphone maker at that time, iPhone X received mixed reviews.

Mixed, but plenty. In 2017, iPhone X was in top 3 Google searches globally spurring a tech media storm. Even so, for a while, it seemed that the iPhone X would be Apple’s first losing bet.

However, the company’s earnings call next spring revealed the contrary: Apple users had chosen iPhone X more than any other iPhone. By May, it was believed that around 50 million iPhone X units had been sold.

Breaking the rules paid off.

Apple’s brand identifier starts a revolution

The fuse was lit. The Notch, whether defended or condemned, became Apple’s brand identifier and the talk of the town.

2018 saw top mobile manufacturers take a leaf from Apple’s tree and released smartphones with similar notches.

Some borrowed the shape but narrowed the area surrounding the sensors. Others minimized the notch as much as possible, until it was given a completely new form. The teardrop and punch-hole notches appeared.

Quite a few hid the notch by taking advantage of a software feature that covered the top edge of the device with a black band.

The presence of notches on a myriad of smartphone models led to the rise of face recognition as a way to unlock the device.

Apple chose a structured light system for their Face Unlock. This system was dependent on the presence of a dot projector and the number of dots projected. By forming a pattern that was then projected on the user’s face and calculating the subsequent distortions in the pattern, iPhone X received a depth map that helped it recognize and authenticate its owner.

It was a solid solution with a big caveat: the high bill of materials. Outfitting each iPhone X with a large form factor sensor and a VCSEL and LED illuminator did not come cheap. Hence, the high price point established by the company for the iPhone X.

However, the price did not deter consumers in the slightest, so OEMs took a leap of faith. Like Apple, they gave their notches a dual purpose: to provide a bigger screen-to-body ratio and to house 3D face recognition solutions for security purposes.

Unlike Apple, some of them took the chance to lower the bill of materials by opting for a less costly technology: time of flight.

Notch 3

When one need leads to another…

… the supply chain shifts.

Consumers’ expectations and competitors’ strategies heavily influenced the consumer electronics market and they led to a new supply chain.

3D face recognition providers became more sought out than ever. Dot projectors, time of flight and stereoscopic sensors were weighed in to find the ones best suited for every mobile manufacturer’s needs.

Stereoscopic solutions had the lowest cost but the accuracy of the depth map did not come close to the one provided by time of flight and structured light ones. Therefore, many companies redirected their attention towards time of flight systems, capable of creating a depth map by measuring the time it takes a beam of light to reach a subject and come back to the source.

Unlike structured light, time of flight uses all the available points in the image so it is largely dependent on the sensor used. Moreover, time of flight is very compact – a big plus – and uses a very small amount of processing power. It is more cost-effective than structured light and behaves better in bright light.

After careful consideration, companies like Google, Huawei and Samsung opted for a time of flight camera. However, they implemented it on the rear of the device, as a means to provide depth for portraits and improve subject and object detection.

LG was one of the few manufacturers that installed it on the front camera for face recognition, with Huawei and Oppo choosing to go with structured light. Meanwhile, Sony boosted their production of 3D sensors.

The wave of phones released with face recognition has contributed to a market estimated at $3.2 billion this year and $7 billion by 2024.  

The technology is registering an unprecedented success which prompts many to question how much longer is the unlocking capability of face recognition going to be a big enough differentiator to drive sales.

Apple is not one of them. The mobile manufacturer guessed the answer a while ago, maybe even since the release of Face ID. That is why they have made sure to show the role 3D face recognition systems can play not only in unlocking a phone, but enabling payments through a proprietary financial product.

The final puzzle piece falls into place: Apple Card

Advertised as the smartest credit card available, Apple Card was introduced this spring. From the beginning, its virtual nature implied that every purchase has to be authorized with Face ID, the face recognition solution Apple introduced two years ago, or Touch ID.

Can you see the whole picture now? The bet Apple took with iPhone X and Face ID was not just a way to appease their customers’ need for a fresh smartphone design or a way to ride the wave of edge-to-edge displays.

The plan was to build trust around face recognition until the audience was comfortable enough to use it to authorize payments from a credit card they can’t see, with no visible numbers and stored in a place they can’t touch.

Moreover, it was about building trust to let a consumer electronics company handle their finances securely and maybe even better than a banking system could.

As surprising as it first seems, Apple Card fits perfectly into Apple’s ecosystem. The Apple Card lives in the Wallet app, where it can also be frozen if lost. The Maps app ties in with a user’s payment history by showing exactly where they spent a certain amount of money. Finally, different rewards motivate users to make purchases from Apple Store.

Will it be enough to secure the Apple Card’s success? That is debatable, but it is clear that after two years of touting the importance of 3D face recognition to unlock a device, Apple agrees with what Xperi has known all along: face recognition can do so much more.

Notch 1

Where can Android go – the untapped potential of Face Recognition 

Lacking a long term plan with multiple use cases for face recognition, mobile manufacturers are now considering eliminating the notch. Time of flight sensors are still included in the smartphone manufacturing process, just not as part of the front camera.

Instead, the front of the device is being stripped of bezels as much as possible. Selfie cameras find their place in mechanical pop-up, sliding or even under-glass solutions; on some devices, fingerprint recognition solutions are already fitted under the display.

This certainly helps with the design of the smartphone – the device starts resembling the futuristic, thin block of glass seen in sci-fi series.

It does not help with the evolution of the mobile user interface, however. By avoiding breaking the rules once more, companies are limiting the use of face recognition in the detriment of consumers. Simultaneously, they are missing out on new and enhanced functionality that can increase their ROI.

As creators of FaceSafe, a depth-agnostic, neural-based 3D face recognition solutions, we can’t sit idly by. We believe a cutting-edge solution such as FaceSafe with liveness detection and the potential to work always-on can be at the core of new user experiences.

FaceSafe will not only keep personal data private thanks to its very low false recognition rate and low false rejection rate; after enrolling multiple users, it could give way to further personalization.

By opting for a device with 3D module and XPERI’s FaceSafe, users will not only unlock their devices safely but will be able to customize their user experience.

Imagine having two phones in one device; each of them with its own theme, passwords, apps and usage patterns.

Moreover, our 3D face recognition solution could take smartphone photography to another level. It could use machine learning to understand the user’s photo editing preferences, for instance.

By learning those preferences, after each authentication, the device could prompt automatic editing in the style most used by the person in question. Alternatively, the handset could edit them to fit the current mood of the user by adding emotion recognition to the mix.

We also envision FaceSafe as a first step towards taking custom stills in real time. By recognizing who is opening the rear camera and their photography preferences, the mobile device fitted with our solution could suggest effects and filters before the editing phase and could apply them in real time.

Current times call for innovative measures. With the hardware already here and 3D face recognition solutions at hand, it would be a shame to not pursue every user-empowering scenario this technology allows for. Unlocking the device was the first step – enriching the user experience will be the next.

Sumat Mehra has been part of the digital imaging revolution since 1988 and witnessed the growth of imaging technology from inside of large corporations as well as start-ups. Sumat has held positions in core research in image quality, product development for mature productization, and led strategic marketing, product marketing, as well as sales and business development teams. He has worked in imaging science, medical image processing, astronomical imaging, pure image quality R&D, and imaging standardization. Today, Sumat is focused on building winning customer relationships with companies focused on imaging and audio devices in the mobile markets.

Back to Stories