# Godfortune Character Consistency Diagnosis

Last updated: 2026-05-21

## Problem

`expression_sheet_v1` generated face-integrated expressions, but the result still failed the main requirement:

- expression changes were weak
- character identity drifted between cells
- outfit/hat/face proportions changed
- unwanted paper text/props appeared

The primitive overlay proof failed for a different reason:

- mouth and brows looked pasted on
- line weight, lighting, softness, and perspective did not match the base character

So the problem is not only "ComfyUI failed".
The current setup is not constrained enough for this character task.

## Current Local Setup

Installed:

- ComfyUI `0.21.1`
- SD1.5 checkpoint: `v1-5-pruned-emaonly.safetensors`
- CLIP-Vision H model
- IPAdapter SD1.5 light: `ip-adapter_sd15_light_v11.bin`
- VideoHelperSuite
- LivePortraitKJ

Missing for strong character lock:

- SD1.5/SDXL anime/webtoon checkpoint tuned for 2D characters
- IPAdapter Plus / Plus Face model
- ControlNet lineart / openpose / depth models
- inpainting checkpoint
- character LoRA
- mask-based expression inpaint workflow
- AnimateDiff/Wan motion models

## Likely Causes

### 1. Model mismatch

The active base model is vanilla SD1.5.

Vanilla SD1.5 is a compatibility baseline, not a strong webtoon character-production model. It can generate images, but it does not naturally preserve a custom cute Korean boy dosa mascot across expression variants.

This is a model problem.

### 2. Weak reference adapter

The installed IPAdapter model is `ip-adapter_sd15_light_v11.bin`.

The IPAdapter Plus README describes the light model as a light-impact model, while `ip-adapter-plus_sd15` is the stronger model and `ip-adapter-plus-face_sd15` is portrait-focused.

This means our current adapter is intentionally weak for hard identity lock.

This is an adapter/model problem.

### 3. Whole-image img2img changes too much

The current expression workflows encode the whole image and denoise the whole latent.

Even if the prompt only asks for facial expression, the model is free to alter:

- hat
- hair
- outfit
- hand
- paper prop
- face shape

This is a workflow problem.

### 4. Denoise tradeoff

Higher denoise creates more expression variation but changes identity.
Lower denoise preserves identity but barely changes expression.

The current test used denoise around `0.48-0.64`, which is high enough to drift the character.

This is a parameter/workflow problem.

### 5. No mask/inpaint constraint

Expression changes should not regenerate the whole character.

ComfyUI's own inpainting docs describe using masks and the `VAE Encoder (for Inpainting)` to modify only specific regions. For our case, the edit area should be:

- eyes
- eyebrows
- mouth
- possibly cheeks

Not:

- hat
- robe
- hands
- background

This is a missing workflow constraint.

### 6. No structural control

The current workflow does not use ControlNet lineart/openpose/depth.

ControlNet's purpose is to add conditional control to diffusion models. For our use case, it should lock:

- silhouette
- head shape
- hat outline
- body pose
- hand position

This is a missing node/model problem.

### 7. Reference image was not born inside the local model family

The selected Godfortune reference was drafted through OpenRouter image generation, then copied into the local ComfyUI pipeline.

That is acceptable for visual direction, but SD1.5 is now trying to imitate a character/style it did not originally generate and does not know as a token.

If the production character is generated with the same local checkpoint, or if a LoRA is trained, consistency becomes easier.

This is a source-of-truth problem.

## Diagnosis

The current failure is mainly:

```text
workflow constraint problem
+ weak adapter problem
+ base model mismatch problem
```

It is not primarily a "ComfyUI cannot do this" problem.

It is also not only because the character was first drafted outside ComfyUI, although that makes the task harder.

## Correct Next Tests

Do not jump directly to video.

Test consistency in this order:

### Test A: Same workflow, lower denoise sweep

Purpose:

- measure the preserve/change tradeoff

Run:

- denoise `0.20`
- denoise `0.30`
- denoise `0.40`

Pass condition:

- identity is stable and expression visibly changes

Expected:

- likely preserves identity but expression change remains weak

### Test B: Face-region inpaint

Purpose:

- prevent hat/outfit/hand drift

Required:

- facial expression mask
- inpainting workflow
- ideally inpainting checkpoint

Pass condition:

- only eyes/mouth/brows change
- hat/outfit/body stay fixed

### Test C: Stronger IPAdapter model

Purpose:

- determine whether light IPAdapter is the bottleneck

Candidate models:

- `ip-adapter-plus_sd15.safetensors`
- `ip-adapter-plus-face_sd15.safetensors`

Pass condition:

- stronger character lock without overcopying

### Test D: ControlNet lineart / pose

Purpose:

- lock silhouette and composition

Candidate controls:

- lineart
- softedge
- openpose
- depth

Pass condition:

- same hat/face/body layout across expression images

### Test E: Better 2D checkpoint

Purpose:

- determine whether vanilla SD1.5 is too weak for this style

Candidate category:

- SD1.5 or SDXL anime/webtoon checkpoint compatible with IPAdapter/ControlNet

Pass condition:

- expressions are stronger while character style remains coherent

### Test F: Character LoRA

Purpose:

- make Godfortune an actual local character identity

Needed:

- curated character sheet
- 10-30 clean variations, or generate/correct a small training set

Pass condition:

- prompt token reliably recalls the same cute boy dosa

## Recommended Immediate Move

Run Test A first because it installs nothing.

If Test A fails as expected, run Test B/C before installing video nodes:

1. Inpaint only the face region.
2. Add stronger IPAdapter Plus/Face.
3. Add ControlNet lineart if identity still drifts.
4. Only then test AnimateDiff/Wan.

## Test A Result: Denoise Sweep V1

Date: 2026-05-21

Output root:

```text
/data/studio-core/artifacts/character_assets/char_godfortune_boy_dosa/comfyui_tests/denoise_sweep_v1
```

Variants generated:

- `denoise 0.20`
- `denoise 0.30`
- `denoise 0.40`

Observed result:

- identity preservation improved compared with the earlier high-denoise expression sheet
- expression difference remained weak
- mouth shape stayed in the same friendly smile family
- eyebrow/eye acting did not become musical, absurd, shocked, or theatrical enough
- props/outfit still changed slightly, though less than before

Conclusion:

The current workflow cannot satisfy both requirements at the same time:

```text
same cute Godfortune identity
+ visibly extreme expression acting
```

Lower denoise preserves identity but does not create enough expression change.
Higher denoise changes expression slightly more but starts changing the character.

This confirms that the next useful test is not another whole-image img2img pass.
The next useful test is a face-region inpaint or stronger identity/control workflow.

## Runtime Note

During Test A, ComfyUI used CUDA successfully:

- Torch: `2.11.0+cu128`
- Device: `NVIDIA GeForce RTX 3060`
- Approximate generation speed: `22 steps in about 6.5-8.3 seconds` per image after model load

But `nvidia-smi` currently reports:

```text
Failed to initialize NVML: Driver/library version mismatch
```

So CUDA generation is working, but GPU monitoring through NVML is not reliable until the driver/library mismatch is resolved, likely by rebooting the VM after the recent NVIDIA driver package update.

## Sources

- ComfyUI official inpainting guide: https://docs.comfy.org/tutorials/basic/inpaint
- ComfyUI IPAdapter Plus: https://github.com/cubiq/ComfyUI_IPAdapter_plus
- ControlNet official repository: https://github.com/lllyasviel/ControlNet
- AnimateDiff Evolved for ComfyUI: https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved