Japanese

Tips: Perfect Sync

Perfect Sync is supported from v1.3.0.

What is Perfect Sync?

“Perfect Sync” is an advanced feature of External Tracking, to achieve rich facial expressions. The feature is based on External Tracking, so it requires Face ID supported iPhone or iPad.

Please see External Tracking beforehand, to understand how to use the system.

From the technical view, Perfect Sync maps the all blendshapes obtained by iOS ARKit FaceTracking to VRM’s BlendShapeClips.

Try Perfect Sync

*If you already know how perfect sync works with other applications (like Vear), you can skip this section.

There are 2 ways to try Perfect Sync right now.

The first way is to load ready-to-use model.

Download 千駄ヶ谷 渋（iPhone用BlendShapeあり）, and load on VMagicMirror. Then, open Ex.Tracker tab and connect to the iOS app. Then turn on Use Perfect Sync check.

That’s it! Try move your eyebrow, mouth, or tongue as you want.

Also, you can try other models which work on Vear Perfect Sync. The model working on Vear Perfect Sync will also work on VMagicMirror. (*Please see the bottom section about compatibility.)

The second way is to use VRoid Studio model. This way is available if the model is created by VRoid Studio, and you do not add any Blendshape to model mesh.

Load the model, and open Ex.Tracker tab and connect to the iOS app. Then turn on Use Perfect Sync, Use VRoid Default Setting check. now it is ready!

The second way is handy but the expression is not so rich. For example, this way does not support tongue out, cheek puff etc.

Setup Step1. Prepare model BlendShapes

This section is about how to get ready your VRM to perfect sync.

The first step is to create 52 blendshapes corresponding to iOS ARKit FaceTracking. This requires some CCD tool like Blender.

Please see the variation of the blendshape and how the mesh should move on each blendshape in the following blog post. (In Japanese, but you will be able to read with some translation tools.)

iPhoneトラッキング向けBlendShapeリスト (The BlendShape list for iPhone Face Tracking)

If you need an example, 千駄ヶ谷 渋（iPhone用BlendShapeあり） model will be the help.

Note that, the blog above shows all of 52 blendshapes, but you can skip some of them if you feel they are too subtle.

Setup Step2. Create BlendShapeClip on Unity

Import the model, and add following 52 BlendShapeClips. Please set the names of the BlendShapeClip as following.

Basically each BlendShapeClip should use only one Blendshape created in previous step, and its weight will be 100.

If you do not create blendshape in Step 1 for some BlendShapeClip, then leave those BlendShapeClip content empty, which means the clip actually does nothing.

After the setup, export the VRM.

References

See the following blog posts (in Japanese), to check the techniques for perfect sync model setup.

iPhoneトラッキング向けBlendShapeリスト (The BlendShape list for iPhone Face Tracking)

VRoidでかんたん！パーフェクトシンク（1/3）VRoidモデルのFBXエクスポート (VRoid Perfect Sync (1/3) Export FBX)

パーフェクトシンクであそぼう！ (Enjoy Perfect Sync!)

パーフェクトシンクの顔をお着換えモデルに移植しよう (Copy perfect sync data to cloth changed model)

　

Note. The background of Vear compatibility

The requirement for Perfect Sync in VMagicMirror is almost same as Vear. This is based on 2 background,

First, Perfect Sync’s specification comes from reference models and blog posts by Hinzka. Those resources also make direction for Vear, so the specifications are similar.

Second background is that, VMagicMirror aims to reduce the user task of model re-setup, by make model specification almost same.

However, note that VMagicMirror does not ensure perfect compatibility to Vear, about Perfect Sync.