Using Apple’s Machine Learning for License Plate Recognition

180227_machine-learning-for-lp

Apple’s Machine Learning frameworks, released a few months ago is inspiring leading businesses to build more intelligent applications that can perform complex functions with ease. License Plate Recognition is one such application that can be built using this framework. In the current post, we will guide you on building the application that can effectively recognize license plates and prove significant for use across industries such as security, transport, and insurance.

Before we delve into the specifics of this application, let us look into understanding the concept of License Plate Recognition and Apple’s Machine Learning framework.

Understanding License Plate Recognition

It is a process of recognizing number plates using Optical Character Recognition (or OCR) on images. Here, the OCR undertakes recognition of the printed text or images using a three-step process called DCR – Detect, Capture, and Recognize.

Machine Learning (ML) essentially implies a phenomenon where the machine has the capability to learn and deliver relevant yet better experiences. It is a branch of Artificial Intelligence involving data analysis and automatic analytical model building.  Core ML is a framework built by Apple and was launched a couple of months ago. It supports features such as face tracking, face detection, landmarks, text detection, rectangle detection, bar code detection, object tracking, and image recognition.

CORE ML MODEL

Core ML supports a variety of machine learning models and this includes neural networks, tree ensembles, support vector machines, and generalized linear models. Core ML requires the Core ML model format (models with a .mlmodel file extension). Apple and Google provide many Core ML models that can be downloaded easily and be used in the application. Additionally, various research groups and universities publish their models and training data, which may not be in the Core ML model format. To use these models, you need to convert them into Core ML format (through Keras and Caffe).

Let us now look into how the application is built.

Building the Application for LPR

Now that you are aware of the key concepts used for bringing about this application, let us now delve into how this all began. We started by looking at various frameworks that use neutral networks for image recognition in iOS. The prominent ones we found were  Tesseract and SwiftOCR.

We used both the frameworks one by one. While the Tesseract framework consumed considerable time for the setup, the SwiftOCR was relatively faster; however, the latter lacked in terms of accuracy. We then realized the need for a framework which could offer better accuracy without consuming too much memory. We then identified Apple’s recently launched Vision framework for face recognition, text, barcodes, QR codes, etc.

About Vision Framework

It gives you easy access to Apple’s models for detecting faces, face landmarks, text, rectangles, barcodes, and objects –

image-1

Its key features include the following:

  1. Detects face rectangle and face landmarks
  2. Finds projected rectangular regions surface.
  3. Finds and recognizes barcodes.
  4. Finds regions of visible text.
  5. Determines the horizon angle in an image.
  6. Detects transforms needed to align the content of two images.
  7. Processes images with Core ML model.
  8. Track movements of a previously identified arbitrary object across multiple images or video frames.

Using Vision Framework

We performed two functions using the vision framework –  finding regions of visible text to split the word image into characters and process images with core ML model. So, let us learn how to cut the images with vision “VNDetectTextRectanglesRequest”.

Here through vision, we detected the characters but could not recognize them. For recognition, we need the core ML trained model as shown below:

image-2

image-3

Creating the Core ML Model

We first looked for the Core ML model, which could recognize the license plate. However, we ended up with a model which could only recognize the text and not only the entire license plate. We then decided to train the model through keras, which is an open source neural network library written in Python. It is designed to enable fast experimentation with deep neural networks.

We decided to build the model in Python and added the following frameworks:

image-4

  1. Keras to create the model from a dataset
  2. Image to fetch the pixels of the character image
  3. Numpy for a high-performance multidimensional array object.
  4. Core Ml tools to convert into ml model
  5. sklearn for splitting the dataset for training and testing

The Process

Now, we needed the data (images of characters 28*28) which had to be trained. We collected the data from the web and clicked 40-50 images of number plates. We sliced the characters and then converted it to 28*28 resolution from one number plate image. Hence, we decided to make this automated by creating an app in iOS and generated thousands of character images with the help of vision framework.

Some takeaways for the conversion flow:

  1. Train model in your favorite framework.
  2. Convert the model to .mlmodel using coremltools python package.
  3. Use the model in your app.

image-5

Some Critical Use Cases for License Plate Recognition

Check on the Authenticity of Abandoned Vehicles: This feature can be leveraged by the police forces for law enforcement purposes. Here, the application can enable them to quickly check the authenticity of number plates installed on abandoned vehicles via mobile devices. Hence, this can curb illegal acts/anti-social elements. An additional camera can focus on the driver’s face and save the image for future security tests. Additionally, this technology does not need separate installations for each vehicle, unlike other technologies that install a transmitter for each vehicle.

Automation of Electronic Toll Collection: LPR can be used for automating the regulated entry of vehicles through toll barriers. It couples with an access control system to recognize number plates listed in the toll collection database. The barrier would open only for those vehicles that are recognized at the electronic toll collection point as shown in the below image:

The blue rays read the license number plate and match it with the database of registered number plates.

image-6

LPR Devices that are Available as Hardware 

There are different types of devices available in the market to recognize the number plate. For instance, CARMEN Parking is a cost-effective solution used for vehicle entry; CARMEN FreeFlow is a device used for recognition of fast moving vehicles are effective LPR solutions. Some other examples include:

  • Dongles for various SW versions – USB, PCIe, FXVD4 card, single or multi-license, the FXVD4mc_s video capture card.
  • Connecting any analog camera to the system, the FXCAMd 102 digital ANPR camera with a motorized optic for easy installation and ideal image quality, the FXCAM IBW_2000 ANPR camera.
  • A “plug-and-play” pre-set solution for the optimal image and the smart ANPR camera, the all-in-one unit, integrated ANPR in the camera.

On the downside, the above hardware devices come with a hefty price as they require a setup. Another limitation is the size and portability of the devices as the setup consumes a considerable amount of time.

Mobile Can Help Overcome the Limitation of LPR Devices

Machine Learning brings a set of advantages that surpasses all its limitations. This solution is not only cost-effective but also quick and needs no installation. Since this application runs through your mobile phone, it comes in handy too. Our first account with ML and mobile yielded tremendous responses and we are in a process to add some more use cases of Machine Learning to our list. Stay tuned for more on this.

If you are looking for any assistance on Machine Learning applications on mobile and customize it to your needs or build any digital solutions for a better customer or employee experience, you can reach out to us at [email protected].

Amit Manchanda

About the Author

Amit Manchanda is working at Net Solutions as Project Lead and has over 9 years of experience in technologies like ASP, Adobe Flex, and Android. He has been part of SME (Subject Matter Expert) Group for RIA applications. He possesses a sound understanding of technical requirement/problem analysis and resolution for providing the best solutions to clients. He is passionate about his work and enjoys interacting with his team. In his leisure time, he loves to listen to music, watch cricket, and play with his daughter.

Leave a Comment

Ravi

8:39 PM, Jun 11, 2018

Hello!
Is there a way for "isolating the characters from number plate" and feeding those characters to the model so that it can be used to train a model that identifies those characters?

Monika Saini

1:02 PM, Mar 12, 2018

Very clear cut explanation of ML. Can we detect only specific objects instead of all, using this technology?