Work in Progress

I have been working (on and off for the past year+) on interfacing with 3D objects (files like *.stl, *.obj, etc.) through hand gestures. I realize the name for the project is a bit of a misnomer! The purpose is to use your hands in “real-life” rather than through the mouse and keyboard to manipulate 3D objects in a more “natural” way.

I believe this project was subconsciously inspired by a video released some time ago by SpaceX. However, their demo involves the use of some extra add-on motion sensors. I want to enable the same interactivity with the bare bones: a simple built-in laptop camera. This makes the problem more difficult, because with just a camera we can only see a 2D representation of the scene (our hands). (This can be slightly bypassed by more advanced pose estimation and with infrared sensors, which are included on some laptops, but I want to keep it as basic as possible so that there are no hardware limitations. More advanced software for handtracking could be useful, i.e. using the LeapMotion controller. But this costs about $100 and one purpose of this project is to not require (m)any add-ons!)

There are several simple features:

  • Zooming: think pinch to zoom
  • Rotating: not always intuitive due to 2D representation
  • Panning: think of dragging hands across the screen

There should (update: I have now implemented these!) also be other simple poses which allow for:

  • Pausing: so the image is locked in place between functions
  • Resetting: so the image goes back to its original position

Currently, one’s forefinger is used to “drag” and rotate the object in the direction of motion, about the center of mass (assuming uniform density). This gives the impression of applying a torque on the object to rotate it. In the future, I could instead explicitly calculate the resulting rotations (from \(\tau = I \alpha\)), but for now the current solution suffices.

I highly request suggestions on alternate ways of defining rotation (which are intuitive to use)!

Check out a simple demo here: Alt Text As you can see, we have the capability of controlling a 3D object in a “hands-free” way (free of keyboard/mouse, that is). As you can also see, there are still some issues. For instance, the pausing function (keep hands still) is a bit noisy. There are many parameters in the handtracking.py file that can be fine-tuned to help with some of these issues (params like: hand tracking minimum confidence levels, sigmoid smoothing params, and many others).

BTW - I am also working on a GUI interface for the same project (see qt_app.py in the repo). Hopefully this can eventually be turned into a standalone executable file. If I were more comfortable with PyQt and C++, I would build (and maybe will re-build) the project there.

Please file all bugs/feature requests at this repository.