# 2D Character Automation Research

Last updated: 2026-05-21

## Goal

Find a repeatable automation path for Godfortune Shorts that can produce:

- 2D character performance
- varied expressions
- song/voice mouth timing
- fast scene direction
- caption-heavy vertical Shorts
- deterministic review artifacts
- server-side rendering without manual editing

The current target is not a single AI-generated video. The target is a controllable production rig that can generate many Shorts from content-ops payloads.

## Working Conclusion

The best first production path is:

```text
layered 2D character asset
-> expression/pose/viseme manifest
-> audio/lyrics timing
-> code-driven timeline renderer
-> FFmpeg final encode
```

This is more suitable than using a full AI video model for every shot because:

- character identity stays controllable
- mouth shapes can be timed to lyrics
- expressions can be exaggerated intentionally
- rendering cost is predictable
- failures are easier to debug
- scene templates can be reused

AI image/video generation can still be used for reference images, backgrounds, reaction cut stills, and occasional non-deterministic B-roll.

## Recommended V1 Stack

### 1. Rhubarb Lip Sync

Use for audio-to-mouth cue extraction.

Role:

- input: WAV/OGG voice or vocal stem
- optional input: transcript/lyrics text
- output: JSON/TSV/dat mouth timing

Why it fits:

- command-line tool
- Linux support
- purpose-built for 2D mouth animation
- supports JSON export
- supports non-English phonetic recognizer

Limit:

- Korean singing may need tuning
- if vocals are mixed with loud music, a vocal stem or clean guide vocal is better
- first version can use lyrics/beat-grid timing before Rhubarb is integrated

Source:

- https://github.com/DanielSWolf/rhubarb-lip-sync

### 2. Revideo or Motion Canvas

Use for code-driven 2D timeline rendering.

Role:

- compose character layers
- switch mouth sprites by viseme timing
- animate body bounce, squash/stretch, eyes, brows, props, camera, captions
- render MP4 or image sequence through a server-side pipeline

Why it fits:

- TypeScript timeline is good for reusable templates
- audio/caption sync is easier than in traditional animation tools
- content payload can become render input JSON
- Node is already available in this VM

Preference:

- Revideo is more directly aligned with headless/API rendering and dynamic inputs.
- Motion Canvas is a strong authoring model but is more editor-oriented.

Sources:

- https://motioncanvas.io/
- https://github.com/midrender/revideo

### 3. FFmpeg

Use as final mux/encode layer.

Role:

- combine rendered video + final audio
- normalize resolution/FPS/codec
- create preview/contact sheet
- generate final MP4 artifact

Status:

- already installed by user.

## Recommended V2/V3 Options

### Godot

Use if we want a reusable 2D stage engine instead of a pure web/video renderer.

Good for:

- puppet-like character stage
- 2D scene templates
- particles, shaders, camera moves
- reusable "mini game engine" style scene logic

Concern:

- more engine/project management overhead
- movie output may need conversion workflow
- less direct than TypeScript for payload-driven templates

Source:

- https://docs.godotengine.org/en/4.0/tutorials/animation/creating_movies.html

### Blender Grease Pencil

Use for high-quality handcrafted 2D/2.5D asset authoring or special shots.

Good for:

- 2D/3D hybrid visuals
- camera depth, lights, props
- Python automation
- background rendering

Concern:

- heavier than needed for fast 병맛 song templates
- mouth/pose automation is more work than sprite-layer rendering

Source:

- https://docs.blender.org/manual/en/latest/advanced/command_line/render.html

### Synfig

Use for vector tweening and traditional 2D animation experiments.

Good for:

- open-source vector animation
- CLI rendering
- parameterized vector layers

Concern:

- integration friction with our API/manifest pipeline
- less natural for fast generated Shorts than code-driven templates

Source:

- https://synfig.readthedocs.io/en/stable/quick_start/animation_basics.html

### OpenToonz

Use as an artist tool, not primary automation renderer.

Good for:

- professional 2D animation workflow
- Xsheet/batch rendering
- plastic/cutout style experiments

Concern:

- Linux support/install path is less clean
- automation around payload-driven generation is heavier
- better for manual production than automatic per-job server rendering

Sources:

- https://opentoonz.readthedocs.io/en/latest/installing_opentoonz.html
- https://opentoonz.readthedocs.io/ja/latest/rendering_the_animation.html

### Lottie

Use for reusable overlays and effects, not the main character rig.

Good for:

- title bursts
- zodiac icons
- particles
- CTA animations
- verdict stamps

Concern:

- not ideal for raster character mouth animation unless the asset is designed as vector layers

Source:

- https://github.com/airbnb/lottie-web

### PixiJS

Use as a lower-level renderer if Revideo/Motion Canvas is too restrictive.

Good for:

- high-performance sprite/particle rendering
- WebGL effects
- custom renderer control

Concern:

- video export pipeline must be built around it
- less batteries-included than Revideo for video composition

Source:

- https://pixijs.com/7.x/guides/basics/what-pixijs-is

## Tools To Avoid As Primary Path For Now

### Remotion

Technically strong, but licensing needs care.

It is excellent for React-to-video and server-side generation, but official pricing/licensing states company usage needs a license. It can be reconsidered later if the license is acceptable.

Source:

- https://www.remotion.dev/

### Rive

Useful runtime ecosystem, but the editor/workflow is not open-source-first in the same way. Good for interactive animation, not the first server-rendered Shorts pipeline.

Source:

- https://rive.app/docs/runtimes/getting-started

### AnimeEffects

Interesting open-source 2D deformation tool, but not the best primary automation foundation.

Reason:

- more artist-tool oriented
- uncertain fit for headless per-job rendering
- better as an experiment if mesh deformation becomes necessary

Source:

- https://animeeffectsdevs.github.io/

## Godfortune Asset Model

Use an explicit asset manifest:

```json
{
  "character_profile_id": "char_godfortune_boy_dosa",
  "rig_id": "singing_mascot_rig_v2",
  "base_layers": {
    "body": "body.png",
    "head": "head_no_mouth.png",
    "hat": "gat.png",
    "hands": "hands.png"
  },
  "mouth_shapes": {
    "closed": "mouth_closed.png",
    "a": "mouth_a.png",
    "e": "mouth_e.png",
    "i": "mouth_i.png",
    "o": "mouth_o.png",
    "u": "mouth_u.png",
    "smile": "mouth_smile.png",
    "shout": "mouth_shout.png"
  },
  "expression_layers": {
    "normal": "eyes_normal.png",
    "excited": "eyes_excited.png",
    "shock": "eyes_shock.png",
    "blank": "eyes_blank.png",
    "verdict": "eyes_verdict.png"
  },
  "anchors": {
    "mouth": [540, 860],
    "eyes": [540, 730],
    "body_center": [540, 1060]
  }
}
```

The current rough prototype should be treated as proof-of-concept only. It is not a production-quality mouth rig because the base image still has baked mouth/face details.

## Character Consistency Direction

The character identity is a cute boy `dosa` mascot.

Consistency should preserve:

- cute round face
- boy `dosa` silhouette
- black `gat`
- pale blue traditional outfit
- warm but mischievous Godfortune personality
- large readable eyes
- simple mouth region designed for viseme swaps

The acting range should be intentionally extreme:

- gentle scenes are not merely calm; they are overly polite, overly refined, and theatrical
- excited scenes are not random chaos; they are overly energetic, musical, absurd, and beat-driven
- verdict scenes should feel like the same mascot delivering a strong stamp-like final judgment

This means the rig needs multiple expression sets, not multiple unrelated character designs.

## Scene Template Model

For `fortune_song_short`, use repeatable scene templates:

```text
hook_title
singing_dosa_front
reaction_cut
fortune_icon_burst
chorus_bounce
verdict_stamp
cta_card
```

Each template should accept:

- duration
- caption lines
- beat markers
- character expression
- mouth timing
- background style
- effect intensity
- camera motion preset

## Automation Pipeline

Recommended first implementation:

```text
content-ops payload
-> scene manifest builder
-> music/audio adapter stub or real generator
-> lyrics timing adapter
-> viseme cue generator
-> 2D renderer
-> FFmpeg finalizer
-> artifact manifest
-> review_pending
```

Initial timing can be simple:

- lyrics line durations from payload target duration
- syllable-based mouth switching
- beat-grid bounce
- expression cuts on hook/verdict words

Then upgrade:

- Rhubarb for spoken/sung guide vocal
- vocal separation if needed
- beat detection
- stronger expression scheduler

## Immediate Recommendation

Do not use primitive vector mouth/eyebrow overlays as the final face system.
The first `v2_base_candidate_01` proof showed visible style mismatch between the generated character and the drawn face parts.

ComfyUI should be used to generate character-matched expression sheets, mouth sheets, or full short clips before the renderer sequences them.

Next practical step:

1. Create a ComfyUI matched expression-sheet test.
2. Create a ComfyUI matched mouth/lower-face-sheet test.
3. Test a short AnimateDiff/IPAdapter/ControlNet clip.
4. Test Wan/InfiniteTalk/S2V only after still-expression consistency is acceptable.
5. Use the code renderer for sequencing, captions, beat motion, and final assembly.

Detailed ComfyUI direction:

```text
docs/comfyui-2d-character-video-direction.md
```

## Acceptance Criteria For V1

- 1080x1920 MP4 output
- 10-15 second hook render
- body bounce on beat
- mouth shape switches from timing data
- at least 5 expressions
- Korean captions
- final verdict stamp
- artifact manifest references the rig, cues, audio, and final video
- render can run from CLI without manual UI