# ComfyUI 2D Character Video Direction

Last updated: 2026-05-21

## Correction

The `v2_base_candidate_01` proof showed that primitive vector mouth and eyebrow overlays are not production-quality.

That test only validated:

- baked-in mouth/eyebrow conflicts can be avoided
- a mouthless/browless base can receive external face parts
- a lightweight renderer can assemble a review video

It did not validate final visual quality.

The visible issue is correct:

- vector mouth and brows do not match the generated character's line weight, lighting, perspective, or softness
- the character looks like a sticker with unrelated facial parts
- this is not acceptable for the Godfortune production look

Rejected artifacts were moved out of the active rig folder:

```text
/data/studio-core/artifacts/rejected/godfortune/20260521-v2-overlay-mismatch
```

## Revised Direction

Do not use hand-drawn primitive overlays as the final face system.

Use one of these two paths:

### Path A: ComfyUI Generates Expression/Mouth Sheets

ComfyUI creates the character-matched facial assets:

- neutral base
- polite expression
- excited singing expression
- shock expression
- verdict expression
- mouth shapes or full lower-face variants

The renderer only sequences matched images and adds:

- beat bounce
- captions
- camera punches
- verdict stamps
- transitions

This keeps deterministic editing while avoiding the visual mismatch.

### Path B: ComfyUI Generates Short Full-Face/Full-Body Clips

ComfyUI receives:

- character reference image
- pose or motion guide
- audio or timing guide
- prompt for performance mode

Then it generates short clips where the whole face changes together.

Candidate node families:

- IPAdapter for character/style reference
- ControlNet or pose/depth/lineart control for structure
- AnimateDiff for short consistent animation
- Wan/InfiniteTalk/S2V style workflows for audio-driven lip sync

This can look more natural, but it costs more VRAM/time and can drift or flicker.

## Recommended Hybrid For Godfortune

Use a hybrid:

1. ComfyUI generates matched expression sheets and short reaction clips.
2. Studio-core renderer arranges those assets into Shorts with captions and music timing.
3. For important song hooks, test ComfyUI audio-driven workflows separately.
4. Keep primitive overlays only as debug visualization, not as final output.

## ComfyUI Test Matrix

Test in this order:

### Test 1: Matched Expression Sheet

Input:

- current cute boy dosa reference
- prompt for `overly_polite`, `overly_excited`, `shock`, `verdict`

Goal:

- same character
- same hat/outfit
- no identity drift
- expressions match line style

Pass condition:

- still images look coherent before any video generation

Current result:

```text
/data/studio-core/artifacts/character_assets/char_godfortune_boy_dosa/comfyui_tests/expression_sheet_v1
```

Assessment:

- Better than primitive overlays because facial parts are generated in the same image domain.
- Not yet production-ready.
- Identity drifts between cells.
- Some cells generate unwanted paper text/props.
- SD1.5 + light IPAdapter is only a baseline, not enough for final character lock.

Next requirement:

- stronger character consistency strategy, likely ControlNet/mask/inpaint, LoRA, better checkpoint, or higher quality image-edit model
- do not move to video until still expression consistency is acceptable

### Test 2: Matched Mouth Sheet

Input:

- same reference
- prompt for mouth shapes only or lower-face variants

Goal:

- mouth shapes match the character's style
- no double mouth
- no mismatched sticker feel

Pass condition:

- mouth sheet can be placed on the base without obvious visual mismatch

### Test 3: AnimateDiff / IPAdapter / ControlNet Short Clip

Input:

- character reference
- simple motion prompt
- optional pose or lineart guide

Goal:

- 3-5 second motion proof
- body and face move together
- character remains recognizable

Pass condition:

- less awkward than manual overlay proof
- no severe identity drift

### Test 4: Wan/InfiniteTalk/S2V Audio Lip Sync

Input:

- character reference
- clean vocal or TTS
- optional pose guide

Goal:

- audio-driven mouth movement
- expression and mouth are regenerated together

Pass condition:

- mouth feels integrated
- identity stays acceptable
- runtime fits RTX 3060 constraints

## Practical Rule

If the generated mouth/eyebrow layer looks pasted on, do not continue that path.

For this concept, face parts need to be generated in the same visual domain as the character, or the full face needs to be regenerated per expression.

## Sources

- ComfyUI Wan2.1 InfiniteTalk audio-driven lip-sync template: https://comfy.org/tr/workflows/templates-wan2_1_infinitetalk_music-1eab7aa23f6a/
- Wan2.2 S2V pose-controlled lip-sync workflow overview: https://www.runcomfy.com/comfyui-workflows/pose-control-lipsync-with-wan2-2-s2v-in-comfyui-audio2video
- AnimateDiff + ControlNet + IPAdapter cartoon-style workflow overview: https://www.runcomfy.com/comfyui-workflows/comfyui-animatediff-controlnet-and-ipadapter-workflow-video2video
- Consistent character workflow using ControlNet/IPAdapter: https://learn.runcomfy.com/create-consistent-characters-with-controlnet-ipadapter
- ComfyUI AnimateDiff Evolved: https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved
- ComfyUI IPAdapter Plus: https://github.com/cubiq/ComfyUI_IPAdapter_plus