Japanese

Face Tracking

In v4.0.0 and later, VMagicMirror provides face tracking option from Face Tracker tab in control panel.

image

Content

Supported Tracking Options

Following tracking options are available.

  • Web Camera (Lite): Low PC loaded face tracking with web camera.
  • Web Camera (High Power): High PC loaded face tracking with web camera.
  • Ex. Tracker: Face tracking with iOS app (iFacialMocap). If you have supported device, the option will perform best while keeping low PC load.

You can also disable web camera with web camera tracking option selected. In this case, you will get very low PC load.

NOTE

There are also available tracking way to use VMC Protocol data receive, though the setup steps might be more complicated. Please see detail at VMC Protocol.

Setup for each Tracking

This section shows how each face tracking will work and what tracking settings are available

Each tracking mode supports Calibrate Face Pose to adjust avatar’s pose.

Note that, avatar’s facial settings is also available in Face tab in setting window.

Web Camera (Lite)

Web Camera (Lite) uses web camera for face tracking, while avoiding high PC load. The option is useful if your PC environment has another performance-critical content like 3D games.

image

Web Camera (Lite) does not use blink / lipsync data by web camera input. Instead, head motion and microphone inputs generate blink and lipsync behaviors.

This mode supports following settings.

  • Use Web Camera: Turn on to use web camera for tracking. Turn off to reduce PC load.
  • Disable horizontal flip: Turn on to disable horizontal flip when applying tracking result.
  • Eye Look Target: Select where the avatar look to. Available in Streaming tab. Select Mouse to makes the avatar look at the orientation mouse pointer exists. Select Fixed to fix head motion except face tracking. User is similar to Fixed, but different when the avatar body does not face straight to the monitor, by Free Camera Mode. In this case User makes the avatar looks head to straight to monitor.

Web Camera (High Power)

Web Camera (High Power) mode performs high quality face tracking, while using more PC load than Web Camera (Lite) mode. This mode is recommended if your PC resources can run high loaded multiple apps, or VMagicMirror is the only performance-critical app to run on the PC.

image

This tracking gets blink and mouth facial data based on web camera input.

Following options are available.

  • Use Web Camera: Turn on to use web camera for tracking. Turn off to reduce PC load.
  • Disable horizontal flip: Turn on to disable horizontal flip when applying tracking result.
  • Enable Forward/Backward Move: Turn on to allow move avatar forward and backward(*).
  • Apply Lip Sync based on Web Camera: Turn on to apply mouth motion based on web camera input. When turned off, microphone input based lipsync value is applied instead.
  • Use Perfect Sync: Apply the facial data as Perfect Sync blendshape values. Please see detail at Perfect Sync.
  • Eye Look Target: Same option in Web Camera (Lite). Note that, this option is only available when camera is disabled in Web Camera (High Power) mode.

*Enable Forward/Backward Move option does almost nothing if the avatar touches virtual keyboard, gamepad etc. To use this option, confirm that Standing Only option is selected at Streaming tab > Motion > Body Motion Style.

Also, this tracking mode support Blink Tracking Settings.

image

The settings handle how to control Blink Value, which is a part of face tracking result in Web Camera (High Power).

Blink Value is a value which get small when your eyes are open, and get large when eyes are closed. The value range is between 0 and 100, and the actual range varies depending on indivisual differences, webcam placement, etc.

  • Blink Value when Eye Opened (%): Specify Blink Value when eyes are open. Typically 0~30 is appropriate value.
  • Blink Value when Eye Closed (%): Specify Blink Value when eyes are closed. Typically 40~70 is appropriate value.
  • Use Mean Blink value: Turn on to treat mean Blink Value, (Left + Right) / 2, for both eye. Use this option especially when you want to move avatar’s both eye same. Note that the option disables tracking wink facial correctly.
  • Use Adjusted Value when Perfect Sync is Enabled: Turn on to apply value adjust calculation when Perfect Sync option is active. Enable the option when the avatar’s blinking does not move enough.
  • Show Blink Value Preview: Turn on to see actual Blink Value for both eyes.
    • Reset Preview buttons resets min/max value result.

By using Show Blink Value Preview option, you can adjust other options based on blink value’s range.

  • Blink Value when Eye Opened (%): Specify Blink Value larger than min value in preview.
  • Blink Value when Eye Closed (%): Specify Blink Value smaller than max value in preview.
  • If both values get close, then half-open state become rare and blink open/close motion will be clear.

For example, following screenshot shows Min=5~10, and Max=70~80.

image

In this case, there are some setups to make good result.

  • To apply several tracking result including half-opened eyes:
    • Blink Value when Eye Opened (%): 20
    • Blink Value when Eye Closed (%): 60
  • To apply blink motion clearly:
    • Blink Value when Eye Opened (%): 35
    • Blink Value when Eye Closed (%): 40

Ex.Tracker

External Tracker is a feature to collaborate with iOS app iFacialMocap.

If you have supported device, the option will perform very good tracking result while keeping low PC load.

See detail at Exteral Tracker App and Connect to iFacialMocap.

Face Switch

Face Switch is a feature to switch avatar’s face by user expression.

image

Face switch has parameters to setup.

  • Threshold: Select from 10% to 90%, to specify when the face switch is triggered. Higher value means you have to more clear expression.
  • BlendShape: Choose the BlendShape to apply, or select (Do Nothing)(*) as empty selection.
  • Keep LipSync: You can check it for the BlendShape with only eye motion, so the LipSync still work.

*(Do Nothing) indication might incorrect appearance as Japanese expression “(何もしない)”. In this case you can choose (何もしない).

Note that some face expressions are difficult to be recognized.

Also note that, this feature is not an extension of face tracking, but also considerable as shortcut key assignment via your face.

This means irrelevant assignment will be still useful.

For example, you can assign a special face expression for tongue out condition, even if the expression does not include tongue out motion at all.

NOTE

Word to Motion feature has higher priority. When face switch and Word to Motion input runs simultaneously, then Word to Motion output is applied.

Also v2.0.2 and later version support accessory visibility control during Face Switch is applied.

Check Show Accessory Option to show accessory visibility selection.

See Accessory page for the detail of accessory feature.