Face recognition and facial tracking technologies are proving to be the need of the hour. Right from using it to unlock your device instead of a code or visualizing how a pair of glasses would appear over their face, facial feature tracking technologies are gaining a huge traction from users globally. Given that, we at Net Solutions recently developed a virtual jewellery try-on app using Apple’s vision framework. This is an iPhone app.
In the current post, we will exemplify this via a virtual nose ring to help you understand how you should go about building it. Before we look into it, let us understand what vision framework is.
About Apple’s Vision Framework
Vision framework, provided by Apple in iOS 11+ versions, enables applications to perform a number of powerful functions such as identifying faces, their features and expressions such as a smile, frown, anger, left eyebrow, eyes, etc. It also allows barcode detection. All this and more, solely done with the purpose of ensuring users’ privacy.
Our Virtual Jewellery Try-on app uses Apple’s vision framework for face detection capabilities. It can recognize several points or features across a human face such as lips, eyes, eyebrows, pupils, nose, median line, and nose crest.
How to Build a Virtual Jewellery Try-on App
Building this app using Apple’s vision framework is quite simple. It combines the functions of the following three components -
- Vision Framework: This helps with detection of face, facial features, and classification of scenes in images and video.
- AVFoundation: This helps in creating a live stream video.
- SceneKit: This helps in putting across a 3D virtual jewellery on a specific (detected) facial point.
Let us now look into building the app by following the below steps:
1. Set up a live streaming session using AVFoundation framework.
Follow the steps below:
2. Convert each frame from live streaming to image data.
Below is a demo of the same:
3. Insert image data into the vision framework’s request to detect the facial landmarks as shown in the below image:
You are now aware of how the app is built. Let us look at how we can build this app using a nose ring.
We started with placing a virtual nose pin around a human face. We used Vision and SceneKit for performing this function. Here, you need to identify the points of the Nose where you wish to augment the video by adding a nose ring.
What was the Challenge faced?
The challenge lay in picking a single point from among an array of points over the nose to fit the nose ring.
How did we overcome the Challenge?
We identified the left side of the nose and specified the index of array at 6.This helped us fix up a point on the live stream layer. Next, we set up the ideal position for the virtual nose ring on SceneKit’s layer. This was done in accordance with the detected point on the live stream layer; however, as there was no in-built function for setting up the index and position on SceneKit, the conversion from one scale to another was difficult.
To ensure precision, we created a formula. This helped us with the conversion of a multiplier value and an offset value for placing the Nose Ring accurately on SceneKit’s layer. We provided Z coordinate of object’s position a static value of -1 (-ve because camera’s direction in sceneKit is towards the -ve Z direction) and the Nose ring was oriented in a profile view towards the viewer.
The code shown below will help ensure that the nose ring moves in accordance with the movement of the face:
Unlike ARKit, vision framework does not detect the anchor points, which have all the details about the position of the object in 3D plane or orientation of the object or in rotation angles etc. However, the popularity of such apps that help recognize facial features and expressions is on the rise. Therefore, the scope for this jewellery try-on app is not limited. You can use this framework to build several other app for the purpose of entertainment too. One of the greatest advantages of this app is that it does not depend on third-party libraries to function. The libraries are in-built, and hence, high performing.
If you are looking for any assistance on Augmented Reality applications on mobile and customize it to your needs or build any digital solutions for a better customer or employee experience, you can reach out to us at: [email protected].