Vision Framework for Face Landmarks detection using Xamarin.iOS

Mobile devices are getting better and better at solving sophisticated tasks. Not only because of better hardware, but also due to modern trends towards AI – such tasks as face detection, barcode recognition, rectangle detection, text recognition, etc. are now supported on the operating system level making it really simple to solve them in your app. Here I am going to show how to detect face landmarks in real time using the Vision framework. The demo app that we’re going to build here is also available on GitHub.

AVCaptureSession

The first thing to do is to configure an instance of AVCaptureSession to capture the video stream from the front camera. We’re going to direct the stream to

  1. AVCaptureVideoPreviewLayer to preview it on the screen
  2. AVCaptureVideoDataOutput to perform the face landmarks detection

Let’s start with a small helper property to get the front camera AVCaptureDevice. We’re using the AVCaptureDeviceDiscoverySession specifying that we’re interested in the front camera.

public AVCaptureDevice GetDevice()
{
    var videoDeviceDiscoverySession = AVCaptureDeviceDiscoverySession.Create(new AVCaptureDeviceType[] { AVCaptureDeviceType.BuiltInWideAngleCamera }, AVMediaType.Video, AVCaptureDevicePosition.Front);
    return videoDeviceDiscoverySession.Devices.FirstOrDefault();
}

Now the AVCaptureSession itself.

public void ConfigureDeviceAndStart()
{
    var device = GetDevice();
    if (device.LockForConfiguration(out var error))
    {
        if (device.IsFocusModeSupported(AVCaptureFocusMode.ContinuousAutoFocus))
        {
            device.FocusMode = AVCaptureFocusMode.ContinuousAutoFocus;
        }
        device.UnlockForConfiguration();
    }
    // Configure Input
    var input = AVCaptureDeviceInput.FromDevice(device, out var error2);
    _captureSession.AddInput(input);
        // Configure Output
    var settings = new AVVideoSettingsUncompressed()
    {
        PixelFormatType = CoreVideo.CVPixelFormatType.CV32BGRA
    };
    var videoOutput = new AVCaptureVideoDataOutput
    {
        WeakVideoSettings = settings.Dictionary,
        AlwaysDiscardsLateVideoFrames = true
    };
    var videoCaptureQueue = new DispatchQueue("Video Queue");
    videoOutput.SetSampleBufferDelegateQueue(new OutputRecorder(View, _shapeLayer), videoCaptureQueue);
    if (_captureSession.CanAddOutput(videoOutput))
    {
        _captureSession.AddOutput(videoOutput);
    }
    // Start session
    _captureSession.StartRunning();
}

Here we’re initiating the capture session by adding instances of the AVCaptureDeviceInput and AVCaptureVideoDataOutput classesWe’re setting AlwaysDiscardsLateVideoFrames to true to save some memory (well, it’s true by default, but let’s make it explicit). And also what’s important here is the OutputRecorder – our implementation of IAVCaptureVideoDataOutputSampleBufferDelegate which will do the face landmarks detection.

VNSequenceRequestHandler and VNDetectFaceLandmarksRequest

At this point, we have the configured AVCaptureSession and we’re ready to process the output to detect face landmarks. To do this let’s override the DidOutputSampleBuffer method.

public class OutputRecorder : AVCaptureVideoDataOutputSampleBufferDelegate
{
    public override void DidOutputSampleBuffer(AVCaptureOutput captureOutput, CMSampleBuffer sampleBuffer, AVCaptureConnection connection)
    {
        using (var pixelBuffer = sampleBuffer.GetImageBuffer())
        using (var ciImage = new CIImage(pixelBuffer))
        using (var imageWithOrientation = ciImage.CreateByApplyingOrientation(ImageIO.CGImagePropertyOrientation.LeftMirrored))
        {
            DetectFaceLandmarks(imageWithOrientation);
        }
        sampleBuffer.Dispose();
    }
    ...
}

The method is called every time there are new frames captured. We’re creating a CIImage and passing it to the DetectFaceLandmarks method which will use the Vision framework to detect face landmarks and draw on the overlay layer. Note that we need to properly dispose all objects, otherwise the app becomes unresponsive very quickly.

VNSequenceRequestHandler _sequenceRequestHandler = new VNSequenceRequestHandler();
VNDetectFaceLandmarksRequest _detectFaceLandmarksRequest;
void DetectFaceLandmarks(CIImage imageWithOrientation)
{
    if (_detectFaceLandmarksRequest == null)
    {
        _detectFaceLandmarksRequest = new VNDetectFaceLandmarksRequest((request, error) =>
        {
            RemoveSublayers(_shapeLayer);
            if (error != null)
            {
                throw new Exception(error.LocalizedDescription);
            }
            var results = request.GetResults<VNFaceObservation>();
            foreach (var result in results)
            {
                if (result.Landmarks == null)
                {
                    continue;
                }
                var boundingBox = result.BoundingBox;
                var scaledBoundingBox = Scale(boundingBox, _view.Bounds.Size);
                InvokeOnMainThread(() =>
                {
                    DrawLandmark(result.Landmarks.FaceContour, scaledBoundingBox, false, UIColor.White);
                    DrawLandmark(result.Landmarks.LeftEye, scaledBoundingBox, true, UIColor.Green);
                    DrawLandmark(result.Landmarks.RightEye, scaledBoundingBox, true, UIColor.Green);
                    DrawLandmark(result.Landmarks.Nose, scaledBoundingBox, true, UIColor.Blue);
                    DrawLandmark(result.Landmarks.NoseCrest, scaledBoundingBox, false, UIColor.Blue);
                    DrawLandmark(result.Landmarks.InnerLips, scaledBoundingBox, true, UIColor.Yellow);
                    DrawLandmark(result.Landmarks.OuterLips, scaledBoundingBox, true, UIColor.Yellow);
                    DrawLandmark(result.Landmarks.LeftEyebrow, scaledBoundingBox, false, UIColor.Blue);
                    DrawLandmark(result.Landmarks.RightEyebrow, scaledBoundingBox, false, UIColor.Blue);
                });
            }
        });
    }
    _sequenceRequestHandler.Perform(new[] { _detectFaceLandmarksRequest }, imageWithOrientation, out var requestHandlerError);
    if (requestHandlerError != null)
    {
        throw new Exception(requestHandlerError.LocalizedDescription);
    }
}

The method is quite simple. First, we initiate a new VNDetectFaceLandmarksRequest by specifying a handler which will iterate through all results and draw them (note that we’re doing the drawing on the UI thread). And second, we’re using the VNDetectFaceLandmarksRequest to perform the detection on the CIImage from the previous step.
And lastly, the DrawLandmark method:

void DrawLandmark(VNFaceLandmarkRegion2D feature, CGRect scaledBoundingBox, bool closed, UIColor color)
{
    var mappedPoints = feature.NormalizedPoints.Select(o => new CGPoint(x: o.X * scaledBoundingBox.Width + scaledBoundingBox.X, y: o.Y * scaledBoundingBox.Height + scaledBoundingBox.Y));
    using (var newLayer = new CAShapeLayer())
    {
        newLayer.Frame = _view.Frame;
        newLayer.StrokeColor = color.CGColor;
        newLayer.LineWidth = 2;
        newLayer.FillColor = UIColor.Clear.CGColor;
        using (UIBezierPath path = new UIBezierPath())
        {
            path.MoveTo(mappedPoints.First());
            foreach (var point in mappedPoints.Skip(1))
            {
                path.AddLineTo(point);
            }
            if (closed)
            {
                path.AddLineTo(mappedPoints.First());
            }
            newLayer.Path = path.CGPath;
        }
        _shapeLayer.AddSublayer(newLayer);
    }
}

Since the Vision framework returns normalized points of landmarks we’re transforming them to the screen coordinates before drawing. The rest code is just about adding a new CAShapeLayer with the drawn line.

Conclusion

Here I showed you how simple it is to perform such a complex task as the detection of facial landmarks. If you’re creating your own app that uses this feature, don’t forget to add an NSCameraUsageDescription to your info.plist. Also, keep in mind that the Vision framework is available on iOS 11+. Happy coding!

Related Blog Posts

We hope you’ve found this to be helpful and are walking away with some new, useful insights. If you want to learn more, here are a couple of related articles that others also usually find to be interesting:

Our Gear Is Packed and We're Excited to Explore with You

Ready to come with us? 

Together, we can map your company’s tech journey and start down the trails. If you’re set to take the first step, simply fill out the form below. We’ll be in touch – and you’ll have a partner who cares about you and your company. 

We can’t wait to hear from you! 

This field is for validation purposes and should be left unchanged.

Together, we can map your company’s tech journey and start down the trails. If you’re set to take the first step, simply fill out the form below. We’ll be in touch – and you’ll have a partner who cares about you and your company. 

We can’t wait to hear from you! 

Montage Portal

Montage Furniture Services provides furniture protection plans and claims processing services to a wide selection of furniture retailers and consumers.

Project Background

Montage was looking to build a new web portal for both Retailers and Consumers, which would integrate with Dynamics CRM and other legacy systems. The portal needed to be multi tenant and support branding and configuration for different Retailers. Trailhead architected the new Montage Platform, including the Portal and all of it’s back end integrations, did the UI/UX and then delivered the new system, along with enhancements to DevOps and processes.

Logistics

We’ve logged countless miles exploring the tech world. In doing so, we gained the experience that enables us to deliver your unique software and systems architecture needs. Our team of seasoned tech vets can provide you with:

Custom App and Software Development

We collaborate with you throughout the entire process because your customized tech should fit your needs, not just those of other clients.

Cloud and Mobile Applications

The modern world demands versatile technology, and this is exactly what your mobile and cloud-based apps will give you.

User Experience and Interface (UX/UI) Design

We want your end users to have optimal experiences with tech that is highly intuitive and responsive.

DevOps

This combination of Agile software development and IT operations provides you with high-quality software at reduced cost, time, and risk.

Trailhead stepped into a challenging project – building our new web architecture and redeveloping our portals at the same time the business was migrating from a legacy system to our new CRM solution. They were able to not only significantly improve our web development architecture but our development and deployment processes as well as the functionality and performance of our portals. The feedback from customers has been overwhelmingly positive. Trailhead has proven themselves to be a valuable partner.

– BOB DOERKSEN, Vice President of Technology Services
at Montage Furniture Services

Technologies Used

When you hit the trails, it is essential to bring appropriate gear. The same holds true for your digital technology needs. That’s why Trailhead builds custom solutions on trusted platforms like .NET, Angular, React, and Xamarin.

Expertise

We partner with businesses who need intuitive custom software, responsive mobile applications, and advanced cloud technologies. And our extensive experience in the tech field allows us to help you map out the right path for all your digital technology needs.

  • Project Management
  • Architecture
  • Web App Development
  • Cloud Development
  • DevOps
  • Process Improvements
  • Legacy System Integration
  • UI Design
  • Manual QA
  • Back end/API/Database development

We partner with businesses who need intuitive custom software, responsive mobile applications, and advanced cloud technologies. And our extensive experience in the tech field allows us to help you map out the right path for all your digital technology needs.

Our Gear Is Packed and We're Excited to Explore with You

Ready to come with us? 

Together, we can map your company’s tech journey and start down the trails. If you’re set to take the first step, simply fill out the contact form. We’ll be in touch – and you’ll have a partner who cares about you and your company. 

We can’t wait to hear from you! 

This field is for validation purposes and should be left unchanged.

Thank you message.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.