Face.DJ is an app that clones you into virtual reality. All that is needed is a photo of yourself.
Based on technology for 3d face reconstruction developed by VisionLabs, this app makes a photorealistic 3d model of your face and renders you in a virtual scene. Then you can try a new hairstyle on, or grow a virtual beard. Those capabilities are provided by the app. The entire pipeline is quite simple: you take a photo of yourself with your cell phone camera, wait for the app to build the model, then you can add decorations of your choice to the scene.
3D face reconstruction from single photo is made possible by the use of face landmarking technology and a deformable virtual face. Your individual features such as lip contours, eyes, eyebrows and overall outline of the face get detected in the photo, then the virtual face gets fit to those. This face can change its geometry based on your individual appearance. Corresponding parts of the virtual face tailor their mutual positions to the make-up of your features. Resulting geometry brings third dimension to your photo. Also, the technology analyzes your skin color in order to extend the texture from the face region to the entire head.
Face landmarking algorithm proceeds in the two following steps. First of all, the algorithm detects your face in the photo, then it narrows down to the face crop and annotates it with a set of predefined landmarks. Both steps employ machine learning. The algorithms learn from a large corpus of data annotated by hand. The data is comprised of thousands of face images with landmarks positioned on them.
Deformable face is based on 3d face scans obtained with specialized equipment. The face scans vary in appearance as people do. Those are the scans of actual people. Differences come in the position of the eyes, size of the nose, shape of the mouth. Some of the scanned faces are oval-shaped while the others are square-shaped. These variations allow to build new virtual faces as the blends of the actual scans.