Japanese

Tips: Using Hand Tracking

The hand tracking system in VMagicMirror runs with low CPU cost, but there are some limitations.

There also exists some preferred environments for the stability.

Limitations

This system does not aim to capture the precise hand position.

The finger state check is rough and limited to detect “rock, paper, scissors” pattern.

The orientation to move hand is up-down or left-right. Forward and backward motion is not detected.

Light up the room so your face and hands are captured clearly from the web camera.

Please put away the objects behind you whose colors are similar to your skin color.

For the clothing, please avoid to expose your shoulders and elbows.

Also the clothing with different color from your skin color is preferred.

Cf: How the Hand Tracking Works

In VMagicMirror, the hand tracking system is realized with following steps, on the base of face tracking technology. You do not have to understand the process perfectly, but it would help to understand the background of limitations.

Step1: Detect face area with deep neural network system.

Step2: Get the mean color value at the center of face area, to estimate user’s skin color.

Step3: In the left and right area of the face, try to pick up the similar color area (Blob).

Step4: If the blob area is enough large, then treat them as hand area.

Step5: Check the shape of the blob to count stretched fingers.

Step6: From the Step4 and Step5, the system get the hand area size and finger stretching count, so it can estimate the hand shape.

The following tweet shows the image.

The black painted area is detected face area. This black paint is a pre-process to prevents the incorrect hand detection from neck or other skin colored areas around the face.

Pink circles on the hand means the area between fingers detected. This part shows that, the detection is done though it is not so highly precised.