This repo was moved from @motlabs group. Thanks for @jwkanggist who is a leader of motlabs community.

Awesome Machine Learning DEMOs with iOS

We tackle the challenge of using machine learning models on iOS via Core ML and ML Kit (TensorFlow Lite).

한국어 README

Contents

• Machine Learning Framework for iOS
  • Flow of Model When Using Core ML
  • Flow of Model When Using Create ML
• Baseline Projects
  • Image Classification
  • Object Detection & Recognition
  • Image Estimation
  • Semantic Segmentation
• Application Projects
  • Annotation Tool
• Create ML Projects
• Performance
  • Measure module
  • Implements
• See also

Machine Learning Framework for iOS

• Core ML
• TensorFlow Lite
• Pytorch Mobile
• fritz
• etc. Tensorflow MobileDEPRECATED)

Flow of Model When Using Core ML

The overall flow is very similar for most ML frameworks. Each framework has its own compatible model format. We need to take the model created in TensorFlow and convert it into the appropriate format, for each mobile ML framework.

Once the compatible model is prepared, you can run the inference using the ML framework. Note that you must perform pre/postprocessing manually.

If you want more explanation, check this slide(Korean).

Flow of Model When Using Create ML

Baseline Projects

DONE

• Using a built-in model with Core ML

• Using a built-in on-device model with ML Kit

• Using the custom model for Vision with Core ML and ML Kit

• Object Detection with Core ML

TODO

• Object Detection with ML Kit
• Using built-in cloud model on ML Kit
  • Landmark recognition
• Using the custom model for NLP with Core ML and ML Kit
• Using the custom model for Audio with Core ML and ML Kit
  • Audio recognition
  • Speech recognition
  • TTS

Image Classification

Name	DEMO	Note
ImageClassification-CoreML		–
MobileNet-MLKit		–

Object Detection & Recognition

Name	DEMO	Note
ObjectDetection-CoreML		–
TextDetection-CoreML		–
TextRecognition-MLKit		–
FaceDetection-MLKit		–

Pose Estimation

Name	DEMO	Note
PoseEstimation-CoreML		–
PoseEstimation-TFLiteSwift		–
PoseEstimation-MLKit		–
FingertipEstimation-CoreML		–

Depth Prediction

DepthPrediction-CoreML		–

Semantic Segmentation

Name	DEMO	Note
SemanticSegmentation-CoreML		–

Application Projects

Name	DEMO	Note
dont-be-turtle-ios		–
WordRecognition-CoreML-MLKit(preparing…)		Detect character, find a word what I point and then recognize the word using Core ML and ML Kit.

Annotation Tool

Name	DEMO	Note
KeypointAnnotation		Annotation tool for own custom estimation dataset

Create ML Projects

Name	Create ML DEMO	Core ML DEMO	Note
SimpleClassification-CreateML-CoreML			A Simple Classification Using Create ML and Core ML

Performance

Execution Time: Inference Time + Postprocessing Time

(with iPhone X)	Inference Time(ms)	Execution Time(ms)	FPS
ImageClassification-CoreML	40	40	23
MobileNet-MLKit	120	130	6
ObjectDetection-CoreML	100 ~ 120	110 ~ 130	5
TextDetection-CoreML	12	13	30(max)
TextRecognition-MLKit	35~200	40~200	5~20
PoseEstimation-CoreML	51	65	14
PoseEstimation-MLKit	200	217	3
DepthPrediction-CoreML	624	640	1
SemanticSegmentation-CoreML	178	509	1
WordRecognition-CoreML-MLKit	23	30	14
FaceDetection-MLKit	–	–	–

Measure module

You can see the measured latency time for inference or execution and FPS on the top of the screen.

If you have more elegant method for measuring the performance, suggest on issue!

Implements

	Measure	Unit Test	Bunch Test
ImageClassification-CoreML	O	X	X
MobileNet-MLKit	O	X	X
ObjectDetection-CoreML	O	O	X
TextDetection-CoreML	O	X	X
TextRecognition-MLKit	O	X	X
PoseEstimation-CoreML	O	O	X
PoseEstimation-MLKit	O	X	X
DepthPrediction-CoreML	O	X	X
SemanticSegmentation-CoreML	O	X	X

See also

• Core ML | Apple Developer Documentation
• Machine Learning – Apple Developer
• ML Kit – Firebase
• Apple’s Core ML 2 vs. Google’s ML Kit: What’s the difference?
• iOS에서 머신러닝 슬라이드 자료
• MoT Labs Blog

WWDC

Core ML

• WWDC2020

  • WWDC2020 10152 Session – Use model deployment and security with Core ML
  • WWDC2020 10153 Session – Get models on device using Core ML Converters
  • Vision
    • WWDC2020 10673 Session – Explore Computer Vision APIs
    • WWDC2020 10099 Session – Explore the Action & Vision app
    • WWDC2020 10653 Session – Detect Body and Hand Pose with Vision
    • TECH-TALKS 206 Session – QR Code Recognition on iOS 11
  • NLP
    • WWDC2020 10657 Session – Make apps smarter with Natural Language

• WWDC2019

  • WWDC2019 256 Session – Advances in Speech Recognition
  • WWDC2019 704 Session – Core ML 3 Framework
  • WWDC2019 228 Session – Creating Great Apps Using Core ML and ARKit
  • WWDC2019 232 Session – Advances in Natural Language Framework
  • WWDC2019 222 Session – Understanding Images in Vision Framework
  • WWDC2019 234 Session – Text Recognition in Vision Framework

• WWDC2018

  • WWDC2018 708 Session – What’s New in Core ML, Part 1
  • WWDC2018 716 Session – Object Tracking in Vision
  • WWDC2018 717 Session – Vision with Core ML
  • WWDC2018 709 Session – What’s New in Core ML, Part 2
  • WWDC2018 713 Session – Introducing Natural Language Framework

• WWDC2017

  • WWDC2017 710 Session – Core ML in depth
  • WWDC2017 208 Session – Natural Language Processing and your Apps
  • WWDC2017 510 Session – Advances in Core Image: Filters, Metal, Vision, and More
  • WWDC2017 506 Session – Vision Framework: Building on Core ML
  • WWDC2017 703 Session – Introducing Core ML

Create ML and Turi Create

• WWDC2020
  • WWDC2020 10642 Session – Build Image and Video Style Transfer models in Create ML
  • WWDC2020 10156 Session – Control training in Create ML with Swift
  • WWDC2020 10043 Session – Build an Action Classifier with Create ML
• WWDC2019
  • WWDC2019 424 Session – Training Object Detection Models in Create ML
  • WWDC2019 426 Session – Building Activity Classification Models in Create ML
  • WWDC2019 420 Session – Drawing Classification and One-Shot Object Detection in Turi Create
  • WWDC2019 425 Session – Training Sound Classification Models in Create ML
  • WWDC2019 428 Session – Training Text Classifiers in Create ML
  • WWDC2019 427 Session – Training Recommendation Models in Create ML
  • WWDC2019 430 Session – Introducing the Create ML App
• WWDC2018
  • WWDC2018 712 Session – A Guide to Turi Create
  • WWDC2018 703 Session – Introducing Create ML

Common ML

• WWDC2020
  • WWDC2020 10677 Session – Build customized ML models with the Metal Performance Shaders Graph
• WWDC2019
  • WWDC2019 803 Session – Designing Great ML Experiences
  • WWDC2019 614 Session – Metal for Machine Learning
  • WWDC2019 209 Session – What’s New in Machine Learning
• WWDC2018
  • WWDC2018 609 Session – Metal for Accelerating Machine Learning
• WWDC2016
  • WWDC2016 715 Session – Neural Networks and Accelerate
  • WWDC2016 605 Session – What’s New in Metal, Part 2

Metal

• WWDC2020
  • WWDC2020 10632 Session – Optimize Metal Performance for Apple Silicon Macs
  • WWDC2020 10603 Session – Optimize Metal apps and games with GPU counters
  • TECH-TALKS 606 Session – Metal 2 on A11 – Imageblock Sample Coverage Control
  • TECH-TALKS 603 Session – Metal 2 on A11 – Imageblocks
  • TECH-TALKS 602 Session – Metal 2 on A11 – Overview
  • TECH-TALKS 605 Session – Metal 2 on A11 – Raster Order Groups
  • TECH-TALKS 604 Session – Metal 2 on A11 – Tile Shading
  • TECH-TALKS 608 Session – Metal Enhancements for A13 Bionic
  • WWDC2020 10631 Session – Bring your Metal app to Apple Silicon Macs
  • WWDC2020 10197 Session – Broaden your reach with Siri Event Suggestions
  • WWDC2020 10615 Session – Build GPU binaries with Metal
  • WWDC2020 10021 Session – Build Metal-based Core Image kernels with Xcode
  • WWDC2020 10616 Session – Debug GPU-side errors in Metal
  • WWDC2020 10012 Session – Discover ray tracing with Metal
  • WWDC2020 10013 Session – Get to know Metal function pointers
  • WWDC2020 10605 Session – Gain insights into your Metal app with Xcode 12
  • WWDC2020 10602 Session – Harness Apple GPUs with Metal

AR

• WWDC2020
  • TECH-TALKS 609 Session – Advanced Scene Understanding in AR
  • TECH-TALKS 601 Session – Face Tracking with ARKit
  • WWDC2020 10611 Session – Explore ARKit 4
  • WWDC2020 10604 Session – Shop online with AR Quick Look
  • WWDC2020 10601 Session – The artist’s AR toolkit
  • WWDC2020 10613 Session – What’s new in USD

Examples

• Training
  • Keras examples: https://keras.io/examples/
  • Pytorch examples: https://github.com/pytorch/examples
• Inference
  • TFLite examples: https://github.com/tensorflow/examples/tree/master/lite
  • Pytorch Mobile iOS example: https://github.com/pytorch/ios-demo-app
  • FritzLabs examples: https://github.com/fritzlabs/fritz-examples
• Models
  • TensorFlow & TFLite models: https://tfhub.dev/
  • Pytorch models: https://pytorch.org/hub/
  • CoreML official models: https://developer.apple.com/machine-learning/models/