Four sets of faces stared out from a television screen, each split down the middle by a black line. On each face, one half was “real,” or at least a video feed of the person talking and moving their lips.
The other side was a computerized avatar.
When their creator, Yaser Sheikh, removed the dividing lines, it was nearly impossible to remember which side was the video and which side was the avatar. And that’s the point.
Mr. Sheikh, director of Facebook’s Reality Lab in Oakland — where these hyperrealistic avatars are born — has a personal stake in the project.
He’s not originally from Pittsburgh, so he wants a better way to communicate with his friends and family all over the world.
“I feel like there is potential for this tech to completely change the way we connect with other people,” he told a group of about 60 people at the Extended Reality Executive Conference on Carlow University’s campus Wednesday evening.
Read More
The Pittsburgh lab is one of three such hubs across the country, laying the groundwork for virtual reality and augmented reality technologies for Facebook.
The other two offices are in Redmond, Wash. and Sausalito, Calif.
Mr. Sheikh hopes the avatars will not only foster personal relationships with loved ones, but also improve business communications.
Conference calls with more than one or two people can be seriously burdensome, he pointed out. Video conferencing tools like Skype, Facetime and Zoom are decent options.
“Nonetheless there is still something missing,” he said. “We like to hear voices, we like to see people we’re talking to, how they move, how they express themselves.”
At its core, virtual reality is the ability to place a person in a totally new environment, often generated by a computer or captured by video. At the moment, this most often materializes through a VR headset, like the Oculus Go, which costs about $200.
Augmented reality, on the other hand, places a virtual layer on top of the real world — think of the game Pokemon Go and all of those little Charmander and Squirtle characters that look like they’re running around your local grocery store. To experience AR, all that’s required is a smartphone.
Mr. Sheikh said the Oakland lab wants to go beyond VR and AR to create these avatars. That’s done through what he calls “photorealistic telepresence.”
There are already some competitors chipping away at that idea.
In 2017, Pixar, Disney Research and Epic Games (known for creating the viral video game sensation Fortnite) unveiled a video demo called “Meet Mike,” which was the height of photorealistic avatars at the time.
“Mike” has pores that add dimension to his face, his eyes sparkle like they’re watery, his smile lines wrinkle when he moves his mouth. But he’s not perfect — his teeth appear a bit wooden or unnatural at times.
Hollywood has been able to accomplish this feat for years, Mr. Sheikh notes, but it takes artists hours upon hours to do so.
Mr. Sheikh’s mission is to create these lifelike renderings in near real-time to foster better communication between people who aren’t physically together.
He and his team do this by creating “codec avatars,” or accurate depictions of what people really look like.
That requires more than just a VR headset. The Facebook Reality team adds additional sensors to the headsets to record what the face is doing or body is doing.
That’s all done in a special space in the Pittsburgh lab called “Mugsy,” a globular room aptly named for the unique mug shots that are taken.

Inside Mugsy, a system of cameras and microphones on the headset capture what the person inside is doing. It can capture 180 gigabytes of data per second.
There are two key factors in creating a realistic avatar from this data: shape and texture.
When someone moves their eyebrows, for example, blood rushes to that part of the forehead, changing the color of the skin, Mr. Sheikh noted.
Lighting is also key, he said. Glimmers in eyes and shadows on the face are constantly changing based on lighting position.
“And if these things are missing, it makes me feel like in some way you are a mannequin or there’s something fake about it,” Mr. Sheikh said.
All of that data generates an avatar through neural networks — a set of algorithms, or computer instructions, that are modeled after the human brain.
Neural networks can interpret sensory data, like those wiggling eyebrows or red cheeks.
Yaser Sheikh, director of Facebook's Reality Lab in Oakland, explains how the office creates hyper-realistic avatars with the goal of making virtual communication as human as possible. pic.twitter.com/3G4aKoG5Qv
— Courtney Linder 🤖 (@LinderPG) April 10, 2019
After data is collected on subjects in the Mugsy space, it’s all about sending that facial information to its final destination — maybe the person on the other end of your Facetime call in the future.
Mr. Sheikh’s hope is that one day, the team can build models that will allow everyday people to quickly and easily create their own avatars from just a few images.
In the meantime, keep using Facetime or sending those funky Apple Memoji avatars of yourself to friends and family.
It’s the closest thing to reality — outside of reality — that exists for now.
Courtney Linder: clinder@post-gazette.com or 412-263-1707. Twitter: @LinderPG.




Comments
Welcome to Post-Gazette subscriber-only commenting community. Please review our community guidelines.