• Bringing Machine Learning to Mobile Apps with TensorFlow

    02.featured_image-1

    In the new omni channel paradigm, where multiple digital touchpoints are connected to offer one of many customer perspectives, the need for intelligent apps and systems, becomes essential. Thanks to machine learning (ML) and deep learning techniques, this is now possible.

    In this article, we will discuss how we could build intelligent mobile apps using Tensor Flow, Google’s machine learning framework.

    What is TensorFlow?

    03.tensorflow-1

    Conceptualized by the Google Brain team, DistBelief, what TensorFlow was called before it’s upgrade, was built in 2011, as a proprietary system based on deep learning neural networks. The source code of DistBelief, was modified and made into a much better application-based library, and in 2015, became known as TensorFlow.

    It combines machine learning with deep learning (deep structured learning or hierarchical learning) models and algorithms, and utilizes them via a common symbol. It uses the high-level programming language, Python, for building mobile and web apps to provide a front-end API, and further execute the apps in C++, a high performing object-oriented programming (OOP) language.

    04.-TensorFlow-Cube-1

    Its latest version, TensorFlow 1.7.0 has been designed keeping deep learning in mind, but may also be used in solving other cases.

    For easy understanding, debugging, and optimization of TensorFlow programs, its creators came up with TensorBoard, which is a suite of visualization tools. Allowing you to visualize graphs to plot quantitative metrics on executing the graph and display additional data, such as images to pass through it. The table below depicts the same:

    05.-TensorBoard-1

    TensorFlow apps

    There are several types of apps that support machine learning, and TensorFlow allows you to leverage most of them. They include:

    • Sentiment analysis;
    • Google translate;
    • Text summarization; and
    • Image recognition

    Of the above four, image recognition is widely used by leading organizations such as Airbnb, eBay, Dropbox, Twitter, Uber, SAP, Qualcomm, IBM, Intel, Google, Facebook, Instagram, and Amazon, and more, for several purposes.

    TensorFlow: pros and cons

    TensorFlow can be viewed as a low-level toolkit for running complex math. Its target group includes researchers, who are solely focused on building experiential learning architectures, and eventually converting them into working software. Listed below are its pros and cons:

    Pros

    • Its responsive build allows easy visualization of every aspect of a graph.
    • Owing to its platform flexibility, parts of it can exist standalone and others merged.
    • When it comes to distributed computing, it is easily identifiable on both CPU and GPU.
    • Its auto differentiation capabilities benefit gradient-based machine learning algorithms. This allows you to compute several derivatives of values in relation to other values, thus leading to a graph extension.
    • It offers advanced support for queues, asynchronous computation, and threads.
    • It can be customized and is an open source.

    Cons

    • To operate, proper understanding of machine learning, as well as advanced calculus and linear algebra, is necessary.
    • GPU memory conflicts with Theano, (a Python library that permits defining, optimizing, and evaluating mathematical expressions, involving multi-dimensional arrays effectively), when imported in the same scope.
    • Does not support OpenCL (Open Computing Language)

    What TensorFlow brings to the table: Use Cases

    Deep learning’s primary software learning tool, TensorFlow, is an open source artificial intelligence (AI) library, that utilizes data flow graphs for building models. It is primarily used for classification, perception, understanding, discovering, prediction, and creation.

    Following are the use cases of TensorFlow:

    1. Sound or voice recognition

    time-to-market

    Sound-based apps constitute one of the most common uses of TensorFlow. Through proper data feed, it is capable of capturing the following audio signals:

    • Voice recognition – largely used in IoT, automotive, security, and UX/UI
    • Voice search – utilized in telecom mobile manufacturers
    • Sentiment analysis – largely utilized in CRM
    • Flaw detection (engine noise) – largely utilized in automotive and aviation industry

    Apple’s Siri, Google Now for Android, and Microsoft Cortana for Windows Phones, are the most common use cases. Identifying languages is another use of voice recognition. Speech-to-text apps may be utilized to identify snippets of sound in bigger audio files, which can be further transcribed as text.

    Sound-based apps are largely used in CRM. For example, TensorFlow algorithms assuming the role of customer service representatives, can direct customers to the required information, quicker than actual personnel.

    2. Text-based applications

    TensorFlow use cases also include text-based apps, including sentimental analysis (CRM, Social Media), threat detection (social media, government) and fraud detection (insurance, finance)

    Identifying languages is a popular use for text-based apps. Below are a few examples:

    • Google Translate supports 100+ language translations, and works on any app. It includes translating jargon into plain language.
    • Text Summarization. Google identified that through sequence-to-sequence learning, shorter texts can be easily summarized. For example, producing news article headlines.
    • SmartReply. In this case, automatic e-mail responses are generated.

    3. Image recognition

    sound_or_voice_recognition

    Image recognition recognizes as well as identifies people and objects in images, to understand both the content and context better.

    It is widely used across social media channels, in the telecom and mobile manufacturing industries, for face recognition or detection, image search, motion detection, machine vision, and photo clustering. It may also be used in automotive, aviation, and healthcare industries.

    TensorFlow image recognition algorithms are capable of classifying and identify objects within bigger images. It is largely used in engineering apps, in identifying shapes for modeling purposes (for example, 3D space construction from 2D images), and by social media for tagging photos (for example, Facebook’s Deep Face). Herein, by analyzing about 600+ images of trees, TensorFlow would be able to identify an unknown tree.

    This technology is popular in the healthcare industry too where its algorithms have the capability of processing more information by identifying more patterns than humans. Furthermore, computers are now able to review scans and detect more illnesses than humans.

    4. Time series

    TensorFlow time series algorithms enable forecasting non-specific time period, and aids in generating their alternative versions.

    A common use case here is recommendation. Leading companies such as Amazon, Google, Facebook and Netflix, are able to analyze customer interactivities through this algorithm, and then compare it with other users to gauge a customer’s buying mindset.

    These recommendations evolve with time. For example: offering customers gifts or recommending TV shows they may enjoy.

    Other uses are in the field of accounting, finance, security and IoT, government, etc.

    5. Video detection

    TensorFlow, is also used in motion detection, security, airports, real-time thread detection in gaming, and in UX/UI fields.  Further, universities are working on large-scale video classification datasets, such as YouTube-8M. This helps accelerate research into understanding large-scale video, transfer learning, representation learning, and domain adaptation approaches, for video.

    Other uses

    Being an open source library, there are also other use cases which largely contribute to machine learning.

    Using TensorFlow, you can build other machine learning algorithms on it, such as decision trees or k-Nearest Neighbors. Below is an ecosystem of TensorFlow:

    efficient_developers

    As shown above, TensorFlow, is well integrated, and is dependent on GPU processing, python, and C++. It can also be integrated with container software such as docker, etc.

    How we integrate TensorFlow in mobile apps: Training on Flowers

    daisyalgorithan

    Using TensorFlow image recognition algorithm, we created an Android app to identify (training) the following flower categories:

    1. Daisy
    2. Dandelion
    3. Roses
    4. Sunflowers
    5. Tulips
    6. Not a flower

    Below are the step-by-step instructions:

    1. The system must contain either of the following Python versions:

    • Python 2.7
    • Python 3.3+

    2. Install TensorFlow by invoking one of the following commands:

    $pip install tensorflow

    3. Next, retrain an image classifier

    Modern image recognition models have huge parameters; therefore, training (identifying) them from scratch requires a lot of labeled training data, and a considerable amount of computing power (hundreds of GPU-hours or more).

    We then used the transfer learning technique, which is a shortcut method.  We took a piece of a model, which was already trained on a related task, and reused it in a new model.

    Next, we will learn how to run TensorFlow on a single machine, and will train a simple classifier to classify images of flowers.

    4. To begin, we required images for different categories to teach the model about new categories. Next, we placed the images in different folders under the main folder: flower_images

    5. Next, we clone TensorFlow repo via:

    git clone https://github.com/googlecodelabs/tensorflow-for-poets-2

    6. We will now navigate to tensorflow-for-poets-2 using the following command on the terminal:

    $cd tensorflow-for-poets-2

    7. Next, we copy the flower_images folder in tf_files folder in tensorflow-for-poets-2

    8. We ran the training using the following command:

    python -m scripts.retrain \
    –bottleneck_dir=tf_files/bottlenecks \
    –how_many_training_steps=500 \
    –model_dir=tf_files/models/ \
    –summaries_dir=tf_files/training_summaries/”${ARCHITECTURE}” \
    –output_graph=tf_files/retrained_graph.pb \
    –output_labels=tf_files/retrained_labels.txt \
    –architecture=”${ARCHITECTURE}” \
    –image_dir=tf_files/flower_photos

    9. The retraining script will now write data to the following two files:

    • tf_files/retrained_graph.pb, which contains a version of the selected network with a final layer retrained on your categories
    • tf_files/retrained_labels.txt, which is a text file containing labels

    10. Next, convert the model to TFLite format for deploying in the mobile devices:

    toco \
    –input_file=tf_files/retrained_graph.pb \
    –output_file=tf_files/graph.lite \
    –input_format=TENSORFLOW_GRAPHDEF \
    –output_format=TFLITE \
    –input_shape=1,224,224,3 \
    –input_array=input \
    –output_array=final_result \
    –inference_type=FLOAT \
    –input_data_type=FLOAT

    11. This should output a “optimized_graph.lite” in your “tf_files” directory.

    12. Use this generated optimized_graph.lite model in your project.

    The integration is now complete.

    Conclusion

    Machine learning is on the rise, and has influenced the way we utilize our mobile devices. Leveraging machine learning, building apps relating to healthcare, recommendation engines for media, and social media sentiment mining, etc. has been the way of the future. Google’s TensorFlow, an open source library that caters to large-scale machine learning and numerical calculations, is here to stay!

    Being the most widely used library for apps like Google Search, Google Translate, Google Photos, and many more, TensorFlow is the tool that makes machine learning and artificial intelligence extend beyond their reaches!

    call_to_action
    Amandeep Singh

    About the Author

    Amandeep Singh is a self-directed and motivated technical developer with 8 years of experience in iOS technologies. Excellent at developing and maintaining iOS applications, he is an avid learner of technologies, given his strong development background. Outside of work, Amandeep likes surfing the net and listening to music.

    Comments

     
     


    get in touch

    Ready to discuss your requirements?

    Request Free Consultation