Hackszine.com: Hand gesture multitouch using only a webcam

Hand gesture multitouch using only a webcam

Andy Wilson of MS Research—a name you may recognize from yesterday's $1 gesture recognition post—is responsible for a number of pretty unbelievable projects involving image processing and human computer interfaces. It's the sort of stuff that really blurs the boundaries between real and digital environments.

I was blown away by the video above, in which Andy demonstrates a multitouch-like hand gesture interface. Get this. It uses only a standard webcam.

The webcam is positioned to watch your keyboard and by simply making a pinching gesture with your thumb and index finger, you can grab and move objects on the screen, or rotate them by twisting your hand. Pinching with two hands, you can control two separate points on the screen, allowing you to easily perform more complex zoom and rotation actions by pulling your hands apart or moving them relative to each other.

I haven't seen source for this anywhere, but he does describe the technique, which is quite clever. By subtracting the background and examining the topology of the remaining image (just the solid background and your hands), you can easily determine how many shapes are made by the background.

With fingers unpinched, the background is a single shape, albeit with a hand shaped isthmus pushing into it. When you pinch and form a circle with your thumb and forefinger, things change. A little island is created in the middle of your fingers and the background becomes two distinct shapes. The position and rotation of the inner shape provides you enough information to control objects on the screen.

Hand Gesture Multitouch [via Procrastineering]
Andy Wilson

Previously:
Gesture recognition for Javascript and Flash

Posted by Jason Striegel | Nov 7, 2008 07:05 PM
Design, Software Engineering, User Interface | Permalink | Comments (0)