Introduction
For decades, motion controls have held a persistent place in our visions of the future. We’ve watched the super heroes, mad scientists, and space cowboys of popular media control digital experiences with just a wave of their hands. We’ve been captivated by these powerful, natural, and intuitive interactions; imagining what it would be like to have that power at our own fingertips. Tony Stark’s workshop, Star Trek’s ‘Holodeck’, Firefly’s holographic brain scanner, and Minority Report’s pre-crime visioning computers all exude a sense of power and mastery, along with paradoxical senses of simplicity, ease, intuitiveness, and humanity. Simply, these experiences feel — “Arthur C. Clarke” style—magical.(1)
The Leap Motion Controller senses how you naturally move your hands and lets you use your computer in a whole new way. Point, wave, swipe, reach, grab. Pick something up and move it or just tap. The Leap Motion Controller doesn’t replace your keyboard, mouse, stylus, or trackpad. It works with them, and without special adapters. With the Leap Motion software running, just plug it into the USB on your Mac or PC, and you’re off.
Controlling a robot with swipe gestures on the air seems to be an interesting proposition. In fact it is quite easy to implement with any of the development environments such as JavaScript, Unity/C#, C++, Java, Python and Objective-C, and others. The Leap Motion controller provides application programming interfaces (APIs) for most of the established programming languages like the ones referred before.
Objectives
Control a robot with gestures on the air with Leap Motion controller, using bluetooth to comunicate between the PC and the robot.
Requirements
A 8051 robot “Robo-51” (could be a robot with any other micrcontroller) with serial bluetooth ZX-Bluetooth module, running a firmware for bluetooth control, and a Leap Motion controller:
Robo-51
ZX-Bluetooth module
Leap Motion controller
Installing Leap Motion Controller
You need first of to install Python in your PC.
To help you get started programing with the Leap Motion API for Python, the following articles provide an overview of the Leap Motion tracking data, how the Leap Motion hardware and software works, and how to setup a project that uses the SDK:
And for the fastest possible introduction to the Leap Motion API, here’s the essential:
Once you have a good idea of what you CAN do, take a moment and consider what you SHOULD do. These articles can help:
- Menu Design Guidelines
- User Orientation and Tutorial Guidelines
- Application Asset and Marketing Guidelines
- User Experience Guidelines
For an in-depth discussion of the Leap Motion API, refer to the following articles:
- Connecting to the Leap Motion
- Tracking Data
- Frames
- Hands
- Fingers and Tools
- Gestures
- Touch Emulation
- Motions
Installing Modules in Python
You will need to install PySerial and Win32com.Client modules. Once duly installed you are ready to write the Python code.
Python code
Write the following Python code:
################################################################################
# Copyright (C) 2012-2013 Leap Motion, Inc. All rights reserved. #
# Leap Motion proprietary and confidential. Not for distribution. #
# Use subject to the terms of the Leap Motion SDK Agreement available at #
# https://developer.leapmotion.com/sdk_agreement, or another agreement #
# between Leap Motion and you, your company or other organization. #
################################################################################
# Adapted by: Tayeb Habib tayeb.habib@gmail.com http://redacacia.me
# from: https://github.com/michieldwitte/ code for LeapMotion VLC controller
# imports
import Leap, sys, time, serial, win32com.client
from Leap import CircleGesture, KeyTapGesture, ScreenTapGesture, SwipeGesture
# Serial port parameters
serial_speed = 9600
# Your serial port instead of ttyUSB0
serial_port = 'COM6'
ser = serial.Serial(serial_port, serial_speed, timeout=1)
# Initializing voice output
speaker = win32com.client.Dispatch("SAPI.SpVoice")
speaker.Speak("I'm Robo-51.I'm at your command!!")
class SampleListener(Leap.Listener):
alarm = 0
def on_init(self, controller):
pass
#print ""
def on_connect(self, controller):
print "Connected"
# Enable gestures
controller.enable_gesture(Leap.Gesture.TYPE_CIRCLE);
controller.enable_gesture(Leap.Gesture.TYPE_KEY_TAP);
controller.enable_gesture(Leap.Gesture.TYPE_SCREEN_TAP);
controller.enable_gesture(Leap.Gesture.TYPE_SWIPE);
def on_disconnect(self, controller):
# Note: not dispatched when running in a debugger.
print "Disconnected"
def on_exit(self, controller):
print "Exited"
def on_frame(self, controller):
# Get the most recent frame and report some basic information
frame = controller.frame()
if not frame.hands.is_empty:
for gesture in frame.gestures():
if gesture.type == Leap.Gesture.TYPE_SWIPE:
swipe = SwipeGesture(gesture)
if swipe.direction[0] > 0.5 and self.alarm < time.time():
self.alarm = time.time() + 1.00
print "Right"
#voice output
speaker.Speak("Right!")
#write data to robot
ser.write('b')
if swipe.direction[0] < -0.5 and self.alarm < time.time():
self.alarm = time.time() + 1.00
print "Left"
#voice output
speaker.Speak("Left!)
#write data to robot
ser.write('c')
if swipe.direction[1] > 0.5 and self.alarm < time.time():
self.alarm = time.time() + 1.00
print "Forward"
#voice output
speaker.Speak("Forward!")
#write data to robot
ser.write('d')
if swipe.direction[1] < -0.5 and self.alarm < time.time():
self.alarm = time.time() + 1.00
print "Backward"
#voice output
speaker.Speak("Backward!")
#write data to robot
ser.write('a')
if gesture.type == Leap.Gesture.TYPE_SCREEN_TAP and self.alarm < time.time():
self.alarm = time.time() + 1.00
print "Stop"
#voice output
speaker.Speak("Stop!")
#write data to robot
ser.write('s')
if gesture.type == Leap.Gesture.TYPE_KEY_TAP and self.alarm < time.time():
self.alarm = time.time() + 1.00
print "Stop"
#voice output
speaker.Speak("Stop!")
#write data to robot
ser.write('s')
def state_string(self, state):
if state == Leap.Gesture.STATE_START:
return "STATE_START"
if state == Leap.Gesture.STATE_UPDATE:
return "STATE_UPDATE"
if state == Leap.Gesture.STATE_STOP:
return "STATE_STOP"
if state == Leap.Gesture.STATE_INVALID:
return "STATE_INVALID"
def main():
print "swipe right for turning right"
print "swipe left for turning left"
print "swipe up for going forward"
print "swipe down for going backward"
print "key/screen tap for stopping"
print ""
# Create a sample listener and controller
listener = SampleListener()
controller = Leap.Controller()
# Have the sample listener receive events from the controller
controller.add_listener(listener)
# Keep this process running until Enter is pressed
#print "Press Enter to quit..."
sys.stdin.readline()
# Remove the sample listener when done
controller.remove_listener(listener)
if __name__ == "__main__":
main()
[/code]
Running the code
Run the Python code. Note that you will have to pair your PC with bluetooth with the bluetooth module on your robot and know the correct COM port (in my case it is COM6). If all is going well you will see the robot to be controller with swipe gestures and tap. The results on my PC are:
The gestures used here are swipe left, swipe right, swipe up, swipe down and tap.
A video showing the robot being controlled by Leap Motion controller with swipe gestures and tap will be made soon and published here.
Conclusions
Controlling a device (in our case a robot) through gestures on the air, is quite an interesting endeavor. In fact it is quite easy to control a device with Leap Motion controller. With little more coding and refining our strategy, we can ensure that right commands are sent every time to the device, through gestures on the air, and those commands are not misinterpreted by the Leap Motion controller.
RedAcacia 