Discovery project Find swarmlets for devices around you Victor

Discovery project Find swarmlets for devices around you Victor Nouvellet Terraswarm - April 2017 Visiting Student at University of California Berkeley Berkeley 04/05/2017 1 / 31

[Dad] Which app should I use to turn on the light? [Mum] Can you explain me how to close the garage from my phone? [Sister] I forgot that we had a connected padlock! [Brother] I don’t have time to setup all this things! [Grandma] Way too complicated for me [Me] Explaining. Configuring. Repeat. 2 / 31

Who am I? Victor Nouvellet Visiting Student Researcher, University of California Berkeley & Senior MSc Student, Computer Science & Information Technology Department at INSA Lyon 3 / 31

Outline 1.The Discovery Problem 2.Description & challenges 3.A solution with accessors 4.Work in progress 4 / 31

THE DISCOVERY PROBLEM 5 / 31

28.1 billion # of IoT devices in 2017 But much less people know how to use them. Source: Statistica 6 / 31

IoT: Number of connected devices worldwide (2012 to 2020) 7 / 31

A simple example : the Kubi Implementation time: 1 week Number of implemented accessors: 2 accessors Number of implemented mobile app: 1 iOS app Number of reused accessors: 1 accessor Skills needed: Mobile, Bluetooth & JS development skills What if we could improve that? 8 / 31

DESCRIPTION & CHALLENGES 9 / 31

What are we talking about? Analogy: (almost) a piece of cake Recipe Swarmlet Ingredients Accessors / Actors Chef Software Engineer Not everyone is a chef but .many want to use well-designed recipes to make the best cake 10 / 31

Where to find them? Contextually (in your environment) mixing location systems Location systems: - Global Positioning System (GPS) - outdoor - Networks signal strength: Wifi, Bluetooth, Zigbee - indoor & outdoor - Beacons - indoor & outdoor - Sounds - indoor & outdoor - Light - indoor 11 / 31

When to find them? Devices have to be up to date for user usability - On user needs - On fixed periods - Using deterministic temporal semantics to synchronize devices actions 12 / 31

Why is this needed? Cyber-Physical Systems have to find a way to reach people needs. It has to be done - Easily - Without code - With less configuration 13 / 31

Challenges The global solution to tackle the problem should be: - Decentralized - Scalable - Adjustable - Modular - Simple (and it is hard) 14 / 31

A SOLUTION WITH ACCESSORS 15 / 31

Missing parts & connections - Actors & accessors are the main components A lot of Web standard components can already be used Only a few components are missing 16 / 31

Discover reachable devices Gather devices metadata you can reach on multiple kind of networks Type Personal area network (PAN) Local area network (LAN) Metropolitan area network (MAN) Wide area network (WAN) Range Within reach of a person Applications Cable replacement for peripherals Within a building or Wireless extension of campus wired network Within a city Worldwide Wireless inter-network connectivity Wireless network access Standards Bluetooth, ZigBee, NFC IEEE 802.11 (WiFi) IEEE 802.15 (WiMAX) Cellular (UMTS, LTE, etc.) 17 / 31

Discover reachable devices Gather devices metadata you can reach on multiple kind of networks Considering PAN & LAN only, discovery accessors for: - Wifi: arp protocol and management frames - Bluetooth/BLE: basic advertising information and eddystone - Zigbee: manufacturer-defined application objects analysis - NFC: tags protocols (libnfc) - Other: infrared, audio, etc. 118 / 31

Monitor connectivity and events Devices can be in or out of reach, connected, disconnected, in idle mode. Also, this information need to be constantly available along with other information like battery or signal strength. 19 / 31

Discovery project in a nutshell Top-down approach: Discovery Accessors: Discover devices around Raw Context Accessor: Monitor connectivity and events Context Aggregator Accessor: Find and store accessors Contextual Broker: Classify accessors in user context Mutable Accessors: Give general definitions of 20 / 31

WORK IN PROGRESS (Bottom-up approach) 21 / 31

Classify and filter available accessors Contextual Broker (100% done) Use of mutable accessors to test for reification Classify available accessors Provide the list of available classes Make a filter available to only detect some classes Trigger a new discovery to get more information about a device and find a better 22 / 31

Find accessors for device metadata Accessor4Device (10% done) - One webservice : https://accessor4device.herokuapp.com/device - [Soon] Multiple supported devices - [Soon]Aware Multiple(100% accessors Context done) This accessor generalizes the capability of REST.js by providing a simple interface for a user to select a concrete REST service and provides the context (set of inputs needed) to invoke that concrete REST service and set of outputs needed from the service. 23 / 31

Keep raw device data & monitor connectivity Raw Context Accessor (not started yet) - Store devices data, metadata & states - Trigger lost & found devices - Trigger on connect & disconnect - Communicate with Discovery Accessors to start/end scans 24 / 31

Discovery Accessors Should use capabilities of the host (Wifi, Bluetooth, Zigbee, NFC, .). The Discovery Accessor (100% done, made by Elizabeth Latronico) Use Nmap or ARP to get MAC Address from IP Other Discovery Accessors (not started yet): - A Bluetooth Discovery Accessor (BDA) - A Wifi Discovery Accessor (WDA) 25 / 31

Discovery Accessors Bluetooth Discovery Accessor (BDA) Steps that BDA should follow: - Scan for devices - Connect to devices - Read services and characteristics - Detect Eddystone protocol 26 / 31

Discovery Accessors Wifi Discovery Accessor (WDA) Use Nmap with -A and/or -sV options to detect OS and standard services. A trade off decision between detection time and services information to be gathered has to be made. 27 / 31

Swarmlets & iOS devices relationship iOS app as swarmlet host (100% done) - demo JXCore host let iOS apps instantiate accessors and run swarmlets. iOS app as accessor enabled device (80% done) AccessorsKit is an iOS framework to drop in your app project and use in conjunction with AccessorModule iOS frameworks. For now, two AccessorModules are being implemented: - KubiAccessorModule - CameraAccessorModule 28 / 31

Summary Democratize cyber physical systems by enabling users to reuse existing swarmlets in their own context. 29 / 31

What is next? - Create a marketplace for accessors and swarmlets - Use machine learning to find the appropriate swarmlet in the context - Let Smart Assistants (Siri, Cortana, Alexa, Google Assistant) handle the swarmlet selection and configuration process with user - Locally (or not) store configurations for already known accessors - Allow bots to create swarmlets themselves based on existing swarmlets, mutable accessors and I/O descriptions 30 / 31

Thank you! Any questions? 31 / 31