# ComfyUI Node Compatibility Plan

Last updated: 2026-05-20

## Purpose

This document defines the pre-install compatibility plan for the Godfortune video pipeline.

The goal is not to install every attractive ComfyUI node. The goal is to build a stable, reproducible worker stack for:

- character and background generation
- character consistency from reference assets
- short animated clips
- exaggerated facial expressions and mouth movement
- music-backed Shorts rendering
- final artifact handoff to the existing `studio-core` job pipeline

## Current Runtime Baseline

- Host role: `studio-core` Ubuntu VM
- GPU: NVIDIA GeForce RTX 3060
- VRAM: 12288 MiB
- NVIDIA driver: `595.58.03`
- Python: `3.12.3`
- Torch smoke-test version: `2.11.0+cu128`
- ComfyUI source path: `/data/studio-core/engines/ComfyUI`
- ComfyUI commit: `6887165a9d657ced4f0122c0ca5368dc74125d80`
- Model root: `/data/studio-core/models`
- Cache root: `/data/studio-core/cache`
- Baseline check script: `scripts/check_comfyui_baseline.sh`

Do not install large models until the node import checks and minimal workflows are passing.

## Latest Baseline Check

Checked at: 2026-05-20T03:49:10Z

Passing:

- `/data/studio-core` is mounted.
- Available artifact/model disk space is about 460 GiB.
- Python 3.12 venv support is available.
- `nvidia-smi` sees the RTX 3060.
- `git` and `curl` are available.
- ComfyUI checkout is present at the pinned commit.

Known gap:

- `ffmpeg` is missing from PATH.

Impact:

- Gate 1 base ComfyUI startup can proceed without FFmpeg.
- Gate 2 video helper and any `RenderEngine` MP4 composition require FFmpeg installation first.

## Gate 1 Result: Base ComfyUI

Checked at: 2026-05-20T03:56:43Z

Status: passed

Installed:

- Dedicated venv: `/data/studio-core/engines/comfyui-venv`
- ComfyUI source: `/data/studio-core/engines/ComfyUI`
- PyTorch: `2.11.0+cu128`
- Torchvision: `0.26.0+cu128`
- Torchaudio: `2.11.0+cu128`
- Transformers: `5.8.1`
- ComfyUI runtime version reported by API: `0.21.1`

Verification:

- CUDA is available inside the ComfyUI venv.
- Device detected: `NVIDIA GeForce RTX 3060`.
- ComfyUI started on `127.0.0.1:8188`.
- `GET /system_stats` returned HTTP data and reported the GPU.
- `GET /` returned HTTP 200.
- ComfyUI was stopped after the smoke test.

Artifacts:

- Startup log: `/data/studio-core/logs/comfyui/gate1-base-start.log`
- Freeze file: `/data/studio-core/logs/comfyui/gate1-freeze.txt`
- Run script: `scripts/run_comfyui_base.sh`

Warning observed:

- ComfyUI printed: `You need pytorch with cu130 or higher to use optimized CUDA operations.`

Decision:

- Keep `torch 2.11.0+cu128` for now because CUDA works and the server starts.
- Do not switch to cu130 until a real workflow shows a performance or compatibility reason.
- Record this warning as a performance optimization gap, not a Gate 1 blocker.

Performance log:

- Perf wrapper: `scripts/run_with_perf_log.sh`
- Log dir: `/data/studio-core/logs/perf/20260520T040356Z-gate1-comfyui-startup`
- Startup wall time to `/system_stats`: about `4.10s`
- CPU used by startup command: `123%`
- Max resident set size: about `994 MiB`
- GPU memory during startup: about `1 MiB` to `113 MiB`
- GPU process was stopped after the check.

## Performance Measurement Policy

For any medium or high-load step, record performance before deciding that a node or model is acceptable.

Track:

- wall-clock time
- max resident memory
- CPU utilization
- GPU utilization
- VRAM usage
- GPU temperature and power
- disk I/O
- stdout and stderr

Use:

```bash
scripts/perf_snapshot.sh
scripts/run_with_perf_log.sh <label> -- <command>
```

Default sample interval is 2 seconds. For short tests, use:

```bash
STUDIO_PERF_SAMPLE_SECONDS=1 scripts/run_with_perf_log.sh <label> -- <command>
```

Do not run high-end video, music, lip-sync, or render tests without a perf log.

## Gate 2 Result: Video Helper

Checked at: 2026-05-20T04:09:44Z

Status: passed

Installed:

- Node: `ComfyUI-VideoHelperSuite`
- Path: `/data/studio-core/engines/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite`
- Commit: `4ee72c065db22c9d96c2427954dc69e7b908444b`
- Commit summary: `4ee72c0 2026-05-13 fix(metadata): stop double-stringifying prompt in MP4 metadata`

Additional Python packages:

- `opencv-python==4.13.0.92`
- `imageio-ffmpeg==0.6.0`

FFmpeg:

- System FFmpeg: `/usr/bin/ffmpeg`
- Version: `6.1.1-3ubuntu5`
- Encoders available: `libx264`, `aac`, `h264_nvenc`, `hevc_nvenc`

ComfyUI startup with VideoHelperSuite:

- Status: passed
- Log dir: `/data/studio-core/logs/perf/20260520T040852Z-gate2-vhs-comfyui-startup`
- Startup wall time to `/system_stats`: about `4.10s`
- CPU used by startup command: `138%`
- Max resident set size: about `1021 MiB`
- VideoHelperSuite import time: about `0.1s`
- GPU memory during startup: about `1 MiB` to `113 MiB`

Image sequence to MP4 smoke tests:

- Test frames: 60 PNG frames, `540x960`, 30 fps, 2 seconds
- Frame generation log: `/data/studio-core/logs/perf/20260520T040926Z-gate2-generate-test-frames`
- CPU H.264 output: `/data/studio-core/tmp/gate2-output/test_libx264.mp4`
- CPU H.264 log: `/data/studio-core/logs/perf/20260520T040938Z-gate2-ffmpeg-libx264-test`
- NVENC H.264 output: `/data/studio-core/tmp/gate2-output/test_h264_nvenc.mp4`
- NVENC H.264 log: `/data/studio-core/logs/perf/20260520T040944Z-gate2-ffmpeg-nvenc-test`

Observed performance:

- `libx264`: about `0.18s`, CPU `211%`, max RSS about `228 MiB`, output `17 KiB`
- `h264_nvenc`: about `0.41s`, CPU `126%`, max RSS about `227 MiB`, output `39 KiB`

Decision:

- Keep VideoHelperSuite enabled.
- Use `libx264` as the default reliable encoder for now.
- Keep `h264_nvenc` available for longer render tests where GPU encoding overhead is worthwhile.
- Gate 3 can proceed to character consistency, but only after selecting the first base image model and IPAdapter model set.

Artifacts:

- Freeze file: `/data/studio-core/logs/comfyui/gate2-freeze.txt`

## Gate 3A Result: IPAdapter Node Import

Checked at: 2026-05-20T04:23:28Z

Status: passed

Installed:

- Node: `comfyui-ipadapter`
- Path: `/data/studio-core/engines/ComfyUI/custom_nodes/comfyui-ipadapter`
- Commit: `b188a6cb39b512a9c6da7235b880af42c78ccd0d`
- Commit summary: `b188a6c 2024-09-13 fix middle block patching`

ComfyUI startup with VideoHelperSuite and IPAdapter:

- Status: passed
- Log dir: `/data/studio-core/logs/perf/20260520T042328Z-gate3a-ipadapter-startup`
- Startup wall time to `/system_stats`: about `5.10s`
- CPU used by startup command: `114%`
- Max resident set size: about `1025 MiB`
- IPAdapter import time: about `0.1s`
- GPU memory during startup: about `1 MiB` to `113 MiB`

Artifacts:

- Freeze file: `/data/studio-core/logs/comfyui/gate3a-freeze.txt`

## Gate 3B Result: Lightweight SD1.5 IPAdapter Smoke Workflow

Checked at: 2026-05-20T04:28:15Z

Status: passed

Downloaded model files:

- Checkpoint: `/data/studio-core/engines/ComfyUI/models/checkpoints/v1-5-pruned-emaonly.safetensors`
- Checkpoint size: `4265146304` bytes
- CLIP Vision: `/data/studio-core/engines/ComfyUI/models/clip_vision/CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors`
- CLIP Vision size: `2528373448` bytes
- IPAdapter light: `/data/studio-core/engines/ComfyUI/models/ipadapter/ip-adapter_sd15_light_v11.bin`
- IPAdapter light size: `44642977` bytes

Download performance:

- Log dir: `/data/studio-core/logs/perf/20260520T042433Z-gate3b-download-sd15-ipadapter-models`
- Wall time: about `1m 21s`
- Disk output: about `6.4 GiB`

Smoke workflow:

- Workflow file: `/data/studio-core/tmp/comfyui-workflows/gate3-ipadapter-smoke.json`
- Reference image: `/data/studio-core/engines/ComfyUI/input/gate3_reference.png`
- Output image: `/data/studio-core/engines/ComfyUI/output/gate3_ipadapter_smoke_00001_.png`
- Output resolution: `384x512`
- Steps: `8`
- Sampler: `euler`
- Scheduler: `normal`

Workflow performance:

- Log dir: `/data/studio-core/logs/perf/20260520T042806Z-gate3b-ipadapter-smoke-workflow`
- End-to-end wall time including server startup: about `9.26s`
- Prompt execution time reported by ComfyUI: about `2.86s`
- CPU used by wrapper command: `109%`
- Max resident set size: about `8.43 GiB`
- GPU memory sampled during workflow: up to about `4721 MiB`
- GPU utilization sampled during workflow: up to about `52%`
- GPU temperature sampled during workflow: up to about `55C`

Decision:

- Keep the SD1.5 IPAdapter stack as the lightweight compatibility baseline.
- This is not the final production style stack.
- Use this baseline to build the first Godfortune character reference sheet and expression preset workflow.
- Do not move to SDXL, FLUX, AnimateDiff, or LivePortrait until this SD1.5 reference workflow is repeatable and connected to `studio-core` artifacts.

## Production Style Target

The `fortune_song_short` visual target is closer to absurd Korean comedy song animation than cinematic video generation.

Required traits:

- vertical 9:16 Shorts
- fixed recurring characters
- simple but expressive poses
- exaggerated mouth shapes and facial reactions
- fast subtitle rhythm
- repeated hook/chorus visual motifs
- quick scene cuts
- fortune/saju visual symbols: zodiac, five elements, gold particles, talismans, clouds

Avoid copying specific characters, songs, lyrics, melodies, or unique branded visual identity from reference YouTube channels. Use the reference only as a production grammar: absurd comedy song, repeated hook, expressive face cuts, and simple animated staging.

## Boundary: What Belongs In ComfyUI

Use ComfyUI for:

- character reference sheet generation
- background stills
- expression and pose stills
- image-to-image variation
- short image-to-video experiments
- optional face/expression animation if the selected node is stable

Do not force these into ComfyUI first:

- final timeline assembly
- subtitle timing
- YouTube metadata
- music generation
- TTS generation
- upload

Those belong behind existing `studio-core` ports:

- `MusicEngine`
- `TTSEngine`
- `RenderEngine`
- `PublishEngine`

Reason: ComfyUI custom nodes frequently add Python and binary dependencies. Keeping audio, render, and publishing outside ComfyUI reduces dependency blast radius.

## Candidate Node Matrix

Install and test one group at a time. Record commit hashes before moving to the next group.

| Priority | Node or Package | Purpose | Initial Risk | Gate |
| --- | --- | --- | --- | --- |
| 1 | Base ComfyUI | image workflows, API backend | low | server starts, queue prompt works |
| 2 | VideoHelperSuite | load/combine image sequences, audio/video helper nodes | low-medium | import passes, image sequence to mp4 test |
| 3 | IPAdapter | character reference consistency | medium | reference image affects output without import errors |
| 4 | ControlNet Aux | pose/depth/canny preprocessors | medium | one preprocessor workflow passes |
| 5 | AnimateDiff-Evolved | short looping motion clips | medium-high | 2-4 second low-res clip passes |
| 6 | AdvancedLivePortrait or LivePortrait KJ | face expression and mouth movement experiments | medium-high | still face to expression/mouth movement passes |
| 7 | Wan native workflow or WanVideoWrapper | higher-end I2V/T2V video generation | high | 1-2 second fp8/offload low-res test fits 12GB VRAM |

## First Implementation Strategy

The safest first production path is not full AI video generation.

Use this path first:

1. Generate character sheet stills.
2. Generate expression presets: smile, shocked, blank stare, angry, crying, final verdict.
3. Generate pose presets: front, side, pointing, holding talisman, singing, verdict.
4. Generate background stills by scene type.
5. Generate or synthesize simple mouth-shape variants.
6. Render movement with the `RenderEngine`: scale, pan, shake, bounce, frame cut, subtitle pop, particle overlay.
7. Use generated song audio from `MusicEngine`.
8. Compose final MP4 with FFmpeg through `RenderEngine`.

This matches the target style better than a single long T2V render and gives deterministic control over captions and beat timing.

## Music Generation Direction

Primary candidate:

- ACE-Step or ACE-Step 1.5 as a separate `MusicEngine` adapter.

Why:

- It fits the existing payload fields: `lyrics`, `music_style`, `vocal_style`, `target_duration_seconds`.
- It is better aligned with song generation than normal TTS.
- It can be isolated in its own virtual environment.

Do not install an ACE-Step ComfyUI node first. Build a separate adapter first, then consider a ComfyUI node only if it is clearly more stable.

Fallback candidates:

- MusicGen for BGM or instrumental loops.
- Stable Audio Open for sound effects or simple background audio.

## TTS Direction

Use TTS only for `fortune_tts_short` and narration-oriented formats.

Candidate families to evaluate later:

- MeloTTS for Korean-capable multilingual TTS
- StyleTTS2 for higher-quality experiments
- Piper or Kokoro-derived stacks for lightweight local narration if Korean voice quality is acceptable

TTS should remain outside ComfyUI unless a specific node is proven stable and useful.

## Mouth Movement Direction

Do not begin with full lip-sync.

Recommended order:

1. Sprite-style mouth shape switching controlled by audio amplitude and lyric timing.
2. Expression preset switching by beat and phrase.
3. LivePortrait-based mouth/face movement for close-up shots.
4. Wav2Lip, MuseTalk, SadTalker, or LatentSync only after dependency and quality checks.

For the target style, exaggerated expression timing is more important than perfect phoneme-level lip-sync.

## Version and Dependency Rules

- Use a dedicated ComfyUI virtual environment.
- Do not reuse the API server `.venv`.
- Do not install ComfyUI custom nodes into the system Python.
- Do not install all nodes through ComfyUI Manager in one batch.
- Pin ComfyUI commit before node installation.
- Pin each custom node commit.
- Save `pip freeze` after every successful group.
- Keep model files under `/data/studio-core/models`.
- Keep Hugging Face and torch caches under `/data/studio-core/cache`.
- If a node requires downgrading torch globally, reject it or isolate it into a separate engine.

## Proposed Paths

```text
/data/studio-core/engines/ComfyUI
/data/studio-core/engines/comfyui-venv
/data/studio-core/models/comfyui
/data/studio-core/models/music
/data/studio-core/models/tts
/data/studio-core/cache/huggingface
/data/studio-core/cache/torch
/data/studio-core/logs/comfyui
```

## Compatibility Gates

Each gate must pass before moving to the next group.

For high-load gates, run commands through:

```bash
scripts/run_with_perf_log.sh <label> -- <command>
```

This records wall time, CPU and memory snapshots, GPU samples, and disk I/O samples under `/data/studio-core/logs/perf`.

### Gate 0: Baseline

- `nvidia-smi` works.
- torch CUDA smoke test passes.
- ComfyUI source checkout is clean enough to reproduce.
- storage root is mounted at `/data/studio-core`.
- repeatable check script passes: `scripts/check_comfyui_baseline.sh`.

### Gate 1: Base ComfyUI

- create dedicated venv
- install torch and ComfyUI requirements
- start ComfyUI bound to localhost first
- verify process starts without custom nodes
- run one tiny image workflow or API prompt test

### Gate 2: Video Helper

- install VideoHelperSuite only
- verify import
- combine a tiny image sequence to MP4
- verify FFmpeg path and codec behavior

### Gate 3: Character Consistency

- install IPAdapter only
- download only the smallest required models for a first test
- generate a character variation from a reference image
- record exact model filenames and locations

### Gate 4: Pose and Preprocessors

- install ControlNet Aux
- run one canny/depth/pose preprocessor test
- verify it does not break IPAdapter workflows

### Gate 5: Lightweight Animation

- install AnimateDiff-Evolved
- test 2-4 seconds at low resolution
- reject or defer if VRAM behavior is unstable

### Gate 6: Face and Mouth

- first candidate: `kijai/ComfyUI-LivePortraitKJ`
- use MediaPipe detector path first to avoid InsightFace non-commercial dependency risk
- test expression/motion change from the Godfortune face-focused reference
- store models under `/data/studio-core/engines/ComfyUI/models/liveportrait`
- keep SadTalker as an isolated second comparison, not inside the main ComfyUI venv
- see `docs/lipsync-engine-evaluation.md`

### Gate 7: High-End Video

- test native Wan workflow or WanVideoWrapper only after the first production path works
- use fp8/offload/low-res settings first
- do not block production on Wan if 12GB VRAM is insufficient

## Failure Policy

If a node fails:

- record the failing command and traceback
- record Python, torch, ComfyUI commit, and node commit
- remove only that node group
- do not downgrade the shared ComfyUI environment unless the whole stack decision changes
- prefer adapter isolation over forcing incompatible dependencies into the main ComfyUI venv

## External References

- ComfyUI system requirements: `https://docs.comfy.org/installation/system_requirements/`
- ComfyUI manual install: `https://docs.comfy.org/installation/manual_install`
- ComfyUI Wan workflow docs: `https://docs.comfy.org/tutorials/video/wan/wan2_2`
- AnimateDiff-Evolved: `https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved`
- VideoHelperSuite: `https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite`
- ComfyUI IPAdapter: `https://github.com/comfyorg/comfyui-ipadapter`
- ComfyUI-LivePortraitKJ: `https://github.com/kijai/ComfyUI-LivePortraitKJ`
- AdvancedLivePortrait: `https://github.com/PowerHouseMan/ComfyUI-AdvancedLivePortrait`
- ACE-Step: `https://github.com/ace-step/ACE-Step`

## Next Action

Next work item:

1. Create the dedicated ComfyUI venv.
2. Install only base dependencies.
3. Start ComfyUI without custom nodes.
4. Save the first baseline log and freeze file.
5. Stop before downloading large image/video/music models.

Before that, run:

```bash
scripts/check_comfyui_baseline.sh
```