# Lip-sync Engine Evaluation

Last updated: 2026-05-20

## Goal

Pick the next local open-source talking-face/lip-sync path for the Godfortune Shorts pipeline.

Target style:

- 2D cute Korean boy `dosa` mascot
- absurd comedy song / TTS shorts
- stable character identity across scenes
- controllable mouth motion for Korean narration and songs
- local-first execution on RTX 3060 12GB

## Current Finding

Do not rely on manual mouth/eyebrow overlay as the final visual style.

The overlay proof is useful because it showed the control problem clearly, but final visuals should be produced by a proper talking-face or portrait-animation engine. The safe direction is:

1. ComfyUI generates character, scene, and expression keyframes.
2. A talking-face/lip-sync adapter animates face/mouth from audio.
3. Render stage composites background, character layer, subtitles, music, and effects.

## Candidate Summary

| Candidate | Role | Fit for our 2D mascot | Install risk | License/commercial concern | Recommendation |
| --- | --- | --- | --- | --- | --- |
| LivePortrait + ComfyUI-LivePortraitKJ | portrait animation / retargeting | high enough to test first | medium | better if using MediaPipe path | first proof |
| SadTalker | classic talking head from image + audio | medium | medium-high, old Python/Torch stack | Apache-2.0 upstream, but dependency stack is older | second comparison |
| Wav2Lip | lip-sync existing video | low-medium technically, but style fit is weak | medium | open repo states commercial use is prohibited | avoid for production |
| AniPortrait | audio-driven portrait diffusion | medium visually, high compute | high | check full license before use | later only |
| Hallo/Hallo2 | high-quality audio-driven portrait | potentially good, but heavy | high | research/high GPU path | defer |
| EchoMimic | audio + landmark driven portrait | potentially good, singing relevant | high | Apache-2.0 | defer until baseline works |

## Candidate Notes

### LivePortrait / ComfyUI-LivePortraitKJ

Why it fits:

- Has ComfyUI node integration, so it matches our existing ComfyUI-first direction.
- Supports image-to-video and video-to-video style workflows.
- The ComfyUI node offers MediaPipe as an alternative detector and notes that this avoids non-commercial InsightFace dependency concerns.
- More likely to work as a lightweight proof on RTX 3060 than diffusion-heavy audio portrait systems.

Risk:

- It is not an audio-first singing model by itself. We may need driving video, retargeting, or a separate audio-to-mouth/landmark step.
- Face detection/cropping may struggle with the large Korean `gat`; test must use the face-focused crop first, then full character.

Decision:

- First install/test candidate.
- Test through isolated custom node gate, not by merging into production worker directly.

### SadTalker

Why it fits:

- It is explicitly built for single portrait image plus audio to talking-head video.
- It has CLI inference and `--still --preprocess full` options that may help with stylized/full-body-ish images.
- Apache-2.0 is acceptable at the upstream project level.

Risk:

- Official install path uses Python 3.8 and old Torch/CUDA versions, which conflicts with our current Python 3.12 / Torch 2.11 / CUDA 12.8 ComfyUI environment.
- More realistic talking-head bias. Our 2D mascot may look warped.
- Should not be installed into the main ComfyUI venv.

Decision:

- Test second in an isolated venv/container only if LivePortrait is insufficient.

### Wav2Lip

Why it fits:

- Strong lip-sync accuracy for existing video.
- Works with CGI/synthetic faces according to the upstream README.

Risk:

- The upstream open-source repo states outputs are for research/academic/personal use only and commercial use is strictly prohibited.
- It modifies mouth regions in an existing video, not a full stylized animation system.
- It may recreate the same two-mouth/artifact problem if face detection or padding is not tuned.

Decision:

- Do not use in production path.
- At most use as a private benchmark/reference if licensing is acceptable for internal tests.

### AniPortrait

Why it fits:

- Audio-driven portrait animation from a reference portrait.
- Could generate more natural motion than simple overlays.

Risk:

- Diffusion-style pipeline is heavier.
- Photorealistic portrait bias is not ideal for our 2D mascot.
- Needs separate installation and model management.

Decision:

- Defer until lightweight LivePortrait/SadTalker proof is done.

### Hallo / Hallo2

Why it fits:

- Audio-driven portrait animation with long-duration/high-resolution direction.

Risk:

- Official docs list A100 as tested GPU.
- Hallo requires CUDA 12.1; Hallo2 lists CUDA 11.8/Torch 2.2.2. Both are separate heavy environments.
- Not a good first experiment on RTX 3060 12GB.

Decision:

- Defer.

### EchoMimic

Why it fits:

- Audio-driven and singing demos are relevant to `fortune_song_short`.
- Apache-2.0 license.
- Landmark-driven mode may become useful if we later control expressions/pose.

Risk:

- Heavier research stack.
- More moving parts than needed for the first proof.

Decision:

- Keep as later candidate after we establish a lightweight baseline.

## Recommended Next Test

### Gate LP-1: LivePortrait Node Import

Install only `ComfyUI-LivePortraitKJ` into a reversible branch/folder and verify:

- ComfyUI starts.
- Custom nodes import.
- No conflict with existing VideoHelperSuite/IPAdapter.
- Models are stored under `/data/studio-core/engines/ComfyUI/models/liveportrait`.
- Use MediaPipe detector path first to avoid InsightFace non-commercial dependency.

### Gate LP-2: Static Face Crop Motion

Input:

```text
/data/studio-core/artifacts/character_assets/char_godfortune_boy_dosa/reference/face_focus/main_face_reference.png
```

Expected:

- short 2-3 second motion preview
- no severe face drift
- no broken `gat`
- mouth/face movement is more natural than SVG overlay

### Gate LP-3: Audio/TTS Integration Feasibility

If LivePortrait needs a driving video instead of raw audio:

- use a generated or simple driving face motion clip
- keep audio sync as a later adapter responsibility
- compare against SadTalker only if raw audio-driven sync is required immediately

## Working Decision

Use this priority:

1. `LivePortrait + ComfyUI-LivePortraitKJ` proof
2. `SadTalker` isolated comparison
3. `EchoMimic` or `AniPortrait` only after the first two fail or are not good enough
4. Avoid `Wav2Lip` for production because of commercial-use restriction in the original open-source repo

## Gate Results

### Gate LP-1: Node Import

Result: pass.

- Node repository: `https://github.com/kijai/ComfyUI-LivePortraitKJ`
- Local path: `/data/studio-core/engines/ComfyUI/custom_nodes/ComfyUI-LivePortraitKJ`
- Commit: `4d9dc6205b793ffd0fb319816136d9b8c0dbfdff`
- ComfyUI startup imported the node in about `0.1s`.
- `/object_info` exposed 8 LivePortrait nodes:
  - `DownloadAndLoadLivePortraitModels`
  - `LivePortraitLoadMediaPipeCropper`
  - `LivePortraitLoadFaceAlignmentCropper`
  - `LivePortraitCropper`
  - `LivePortraitProcess`
  - `LivePortraitComposite`
  - `LivePortraitRetargeting`
  - `LivePortraitLoadCropper`
- Existing IPAdapter smoke workflow still passed after dependency install.

Installed dependencies directly instead of running the full requirements file, because the requirements file pins `numpy<=1.26.4` while the current ComfyUI venv uses `numpy 2.4.4`.

Freeze snapshots:

```text
/data/studio-core/logs/comfyui-venv-freeze-before-liveportrait-20260520T082834Z.txt
/data/studio-core/logs/comfyui-venv-freeze-after-liveportrait-20260520T082834Z.txt
```

### Gate LP-2: Godfortune Face Motion Preview

Result: pass with caveats.

Output:

```text
/data/studio-core/artifacts/character_assets/char_godfortune_boy_dosa/liveportrait/face_motion_v1/liveportrait_face_motion_v1.mp4
/data/studio-core/artifacts/character_assets/char_godfortune_boy_dosa/liveportrait/face_motion_v1/liveportrait_face_motion_v1_contact_sheet.png
/data/studio-core/artifacts/character_assets/char_godfortune_boy_dosa/liveportrait/liveportrait_manifest.json
```

Performance log:

```text
/data/studio-core/logs/perf/20260520T084033Z-godfortune-liveportrait-face-motion-v1-models-ready
```

Observed:

- Identity and `gat` stayed stable.
- Motion was subtle because the driving source was a temporary mouth-shape still sequence.
- Peak sampled VRAM was about `7073 MiB`.
- Successful prompt elapsed time was about `24s`.
- A follow-up lip-retargeting test with artificial mouth-shape stills was rejected:
  - extreme `O/U` mouth frames failed face detection in FaceAlignment
  - detectable-only frames completed but produced an eye artifact and still weak mouth movement
  - conclusion: LivePortrait is not enough as the lip-sync engine unless we have a natural driving video or a separate audio-to-landmark driver

Compatibility patches applied to the local custom node:

- `liveportrait/utils/landmark_runner.py`: `torch.load(..., weights_only=False)` for PyTorch 2.6+ compatibility with `landmark_model.pth`.
- `face_alignment/api.py`: relative detector import fixed to use `__package__` because the custom node directory contains hyphens.

Blocked path:

- `LivePortraitLoadMediaPipeCropper` is currently blocked by `mediapipe 0.10.35` package API mismatch: `No module named 'mediapipe.framework'`.

Working path:

- `LivePortraitLoadFaceAlignmentCropper`
- `face_detector=blazeface_back_camera`
- `landmarkrunner_device=torch_gpu`
- `face_detector_device=cuda`
- `face_detector_dtype=fp16`

Next comparison:

- Run SadTalker in an isolated environment as a raw audio-driven talking-face baseline.
- Keep EchoMimic as a later candidate if SadTalker quality is too realistic or unstable for the 2D mascot.

## Sources

- SadTalker GitHub: https://github.com/OpenTalker/SadTalker
- LivePortrait GitHub: https://github.com/KlingAIResearch/LivePortrait
- ComfyUI-LivePortraitKJ GitHub: https://github.com/kijai/ComfyUI-LivePortraitKJ
- Wav2Lip GitHub: https://github.com/Rudrabha/Wav2Lip
- AniPortrait GitHub: https://github.com/Zejun-Yang/AniPortrait
- Hallo GitHub: https://github.com/fudan-generative-vision/hallo
- Hallo2 GitHub: https://github.com/fudan-generative-vision/hallo2
- EchoMimic GitHub: https://github.com/antgroup/echomimic