Controlling a Robot with Gestures on the Air with Leap Motion Controller

Introduction

For decades, motion controls have held a persistent place in our visions of the future. We’ve watched the super heroes, mad scientists, and space cowboys of popular media control digital experiences with just a wave of their hands. We’ve been captivated by these powerful, natural, and intuitive interactions; imagining what it would be like to have that power at our own fingertips. Tony Stark’s workshop, Star Trek’s ‘Holodeck’, Firefly’s holographic brain scanner, and Minority Report’s pre-crime visioning computers all exude a sense of power and mastery, along with paradoxical senses of simplicity, ease, intuitiveness, and humanity. Simply, these experiences feel — “Arthur C. Clarke” style—magical.(1)

gestures1The Leap Motion Controller senses how you naturally move your hands and lets you use your computer in a whole new way. Point, wave, swipe, reach, grab. Pick something up and move it or just tap. The Leap Motion Controller doesn’t replace your keyboard, mouse, stylus, or trackpad. It works with them, and without special adapters. With the Leap Motion software running, just plug it into the USB on your Mac or PC, and you’re off.

Controlling a robot with swipe gestures on the air seems to be an interesting proposition. In fact it is quite easy to implement with any of the development environments such as JavaScript, Unity/C#, C++, Java, Python and Objective-C, and others. The Leap Motion controller provides application programming interfaces (APIs) for most of the established programming languages like the ones referred before.

Objectives

Control a robot with gestures on the air with Leap Motion controller, using bluetooth to comunicate between the PC and the robot.

Requirements

A 8051 robot “Robo-51” (could be a robot with any other micrcontroller) with serial bluetooth ZX-Bluetooth module, running a firmware for bluetooth control, and a Leap Motion controller:

Robo-51

Robo-51

ZX-Bluetooth

ZX-Bluetooth module

Leap Motion controller

Leap Motion controller

Installing Leap Motion Controller

You need first of to install Python in your PC.

To help you get started programing with the Leap Motion API for Python, the following articles provide an overview of the Leap Motion tracking data, how the Leap Motion hardware and software works, and how to setup a project that uses the SDK:

And for the fastest possible introduction to the Leap Motion API, here’s the essential:

Once you have a good idea of what you CAN do, take a moment and consider what you SHOULD do. These articles can help:

For an in-depth discussion of the Leap Motion API, refer to the following articles:

Installing Modules in Python

You will need to install PySerial and Win32com.Client modules. Once duly installed you are ready to write the Python code.

Python code

Write the following Python code:

################################################################################
# Copyright (C) 2012-2013 Leap Motion, Inc. All rights reserved.               #
# Leap Motion proprietary and confidential. Not for distribution.              #
# Use subject to the terms of the Leap Motion SDK Agreement available at       #
# https://developer.leapmotion.com/sdk_agreement, or another agreement         #
# between Leap Motion and you, your company or other organization.             #
################################################################################

# Adapted by: Tayeb Habib tayeb.habib@gmail.com http://redacacia.me
# from: https://github.com/michieldwitte/ code for LeapMotion VLC controller

# imports
import Leap, sys, time, serial, win32com.client
from Leap import CircleGesture, KeyTapGesture, ScreenTapGesture, SwipeGesture

# Serial port parameters
serial_speed = 9600
# Your serial port instead of ttyUSB0
serial_port = 'COM6'
ser = serial.Serial(serial_port, serial_speed, timeout=1)

# Initializing voice output
speaker = win32com.client.Dispatch("SAPI.SpVoice")
speaker.Speak("I'm Robo-51.I'm at your command!!")

class SampleListener(Leap.Listener):
    alarm = 0
    def on_init(self, controller):
        pass
        #print ""

    def on_connect(self, controller):
        print "Connected"

        # Enable gestures
        controller.enable_gesture(Leap.Gesture.TYPE_CIRCLE);
        controller.enable_gesture(Leap.Gesture.TYPE_KEY_TAP);
        controller.enable_gesture(Leap.Gesture.TYPE_SCREEN_TAP);
        controller.enable_gesture(Leap.Gesture.TYPE_SWIPE);

    def on_disconnect(self, controller):
        # Note: not dispatched when running in a debugger.
        print "Disconnected"

    def on_exit(self, controller):
        print "Exited"

    def on_frame(self, controller):
        # Get the most recent frame and report some basic information
        frame = controller.frame()
        if not frame.hands.is_empty: 
            for gesture in frame.gestures():
                if gesture.type == Leap.Gesture.TYPE_SWIPE:
                    swipe = SwipeGesture(gesture)

                    if swipe.direction[0] > 0.5 and self.alarm < time.time():
                        self.alarm = time.time() + 1.00
                        print "Right"
                        #voice output
                        speaker.Speak("Right!")
                        #write data to robot
                        ser.write('b')
                        
                    if swipe.direction[0] < -0.5 and self.alarm < time.time():
                        self.alarm = time.time() + 1.00
                        print "Left"
                        #voice output
                        speaker.Speak("Left!)
                        #write data to robot
                        ser.write('c')
                        
                    if swipe.direction[1] > 0.5 and self.alarm < time.time():
                        self.alarm = time.time() + 1.00
                        print "Forward"
                        #voice output
                        speaker.Speak("Forward!")
                        #write data to robot
                        ser.write('d')
                        
                    if swipe.direction[1] < -0.5 and self.alarm < time.time():
                        self.alarm = time.time() + 1.00
                        print "Backward"
                        #voice output
                        speaker.Speak("Backward!")
                        #write data to robot
                        ser.write('a')
                        
                if gesture.type == Leap.Gesture.TYPE_SCREEN_TAP and self.alarm < time.time():
                        self.alarm = time.time() + 1.00
                        print "Stop"
                        #voice output
                        speaker.Speak("Stop!")
                        #write data to robot
                        ser.write('s')
                        
                if gesture.type == Leap.Gesture.TYPE_KEY_TAP and self.alarm < time.time():
                        self.alarm = time.time() + 1.00
                        print "Stop"
                        #voice output
                        speaker.Speak("Stop!")
                        #write data to robot
                        ser.write('s')

    def state_string(self, state):
        if state == Leap.Gesture.STATE_START:
            return "STATE_START"

        if state == Leap.Gesture.STATE_UPDATE:
            return "STATE_UPDATE"

        if state == Leap.Gesture.STATE_STOP:
            return "STATE_STOP"

        if state == Leap.Gesture.STATE_INVALID:
            return "STATE_INVALID"

def main():
    print "swipe right for turning right"
    print "swipe left for turning left"
    print "swipe up for going forward"
    print "swipe down for going backward"
    print "key/screen tap for stopping"
    print ""

    # Create a sample listener and controller
    listener = SampleListener()
    controller = Leap.Controller()

    # Have the sample listener receive events from the controller
    controller.add_listener(listener)

    # Keep this process running until Enter is pressed
    #print "Press Enter to quit..."
    sys.stdin.readline()

    # Remove the sample listener when done
    controller.remove_listener(listener)


if __name__ == "__main__":
    main()
[/code]

Running the code

Run the Python code. Note that you will have to pair your PC with bluetooth with the bluetooth module on your robot and know the correct COM port (in my case it is COM6). If all is going well you will see the robot to be controller with swipe gestures and tap.  The results on my PC are:

controlo_leapThe gestures used here are swipe left, swipe right, swipe up, swipe down and tap.

A video showing the robot being controlled by Leap Motion controller with swipe gestures and tap will be made soon and published here.

Conclusions

Controlling a device (in our case a robot) through gestures on the air, is quite an interesting endeavor. In fact it is quite easy to control a device with Leap Motion controller. With little more coding and refining our strategy, we can ensure that right commands are sent every time to the device, through gestures on the air, and those commands are not misinterpreted by the Leap Motion controller.

 

Advertisements

About Tayeb

Electronics engineer, part-time webmaster and owner of "Aliatron", a tech-oriented company registered in Portugal and Mozambique.
This entry was posted in Robotic Projects and tagged , , , , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s