Depth Piano

The Problem: detecting key pressing and the amount of pressure the player apply to each key

Why not 2D? to capture accurately the pressure the user apply to each key in 2D, we need to place the camera in the place of the player body – not feasible. Our Approach: the depth-camera is placed above the keys (in this way the camera doesn’t interfere the player). at the beginning we set for each key a virtual plane (Using PCA for estimating the normal vector to the plane from the 3D points) and for every frame we estimate the current plane that defined by the keys and check the angle defined between the ground plane (from the beginning) to the current plane, if we passed certain threshold – it is a key pressing. notice that we can tell that amount of pressure applied to the keys by taking the derivatives of the angle (along the frames).

Implementation: our algorithm was implemented using C++ supported by the OpenCV library and the RealSense SDK for Intel Depth Cameras (a.k.a RSSDK).

Yarin Didi, Inbar Pinsly, Sean Man