# ComfyUI Character Consistency Deep Research Last updated: 2026-07-20 ## Conclusion The user's premise is partly correct: prompt dialect, negative conditioning, sampler, CFG, resolution, LoRA trigger, and node settings materially change the result. The previous Studio experiments did not control these variables well enough. The remaining correction is that a diffusion random seed does not identify a character. In ComfyUI's KSampler it initializes the noise tensor. Positive and negative conditioning, model, resolution, sampler, scheduler, CFG, steps, denoise, LoRA, and reference/control inputs then determine the denoising path. A fixed seed is useful for exact reproduction and controlled comparison, but it is not a production identity lock. Production direction: ```text model-specific prompt recipe + approved master reference + masked IPAdapter identity conditioning + lineart/canny silhouette control + short state prompt + fixed recipe seed for reproducibility -> body state -> registered face/mouth state -> separate prop composite ``` ## Verified Source Facts 1. ComfyUI documents `seed` as the random-noise seed. The KSampler separately receives model, positive/negative conditioning, steps, CFG, sampler, scheduler, latent, and denoise. Therefore seed equality does not mean conditioning equality. 2. ComfyUI states that high CFG can reduce quality and that sampler and scheduler alter the denoising path. 3. Animagine XL 4.0 was trained with ordered tag captions. Its official recipe puts subject and rating first, quality tags last, and recommends CFG 5, 28 steps, Euler Ancestral, and native SDXL portrait buckets such as 896x1152. 4. ComfyUI's LoRA node modifies both model weights and, independently, CLIP text embeddings. A LoRA strength is not just a visual-style slider. 5. IPAdapter Advanced center-crops non-square references. Its early timesteps have the strongest compositional influence, and its attention mask can restrict the affected region. 6. ControlNet adds an explicit image condition with controllable strength and timing. OpenPose controls joints; it does not specify a large-head silhouette. Sources: - [ComfyUI KSampler documentation](https://docs.comfy.org/built-in-nodes/sampling/ksampler) - [ComfyUI text-to-image parameters](https://docs.comfy.org/tutorials/basic/text-to-image) - [ComfyUI CLIP prompt weighting](https://docs.comfy.org/built-in-nodes/ClipTextEncode) - [ComfyUI LoRA node](https://docs.comfy.org/tutorials/basic/lora) - [ComfyUI ControlNet node](https://docs.comfy.org/tutorials/controlnet/controlnet) - [Animagine XL 4.0 official model card](https://huggingface.co/cagliostrolab/animagine-xl-4.0) - [Illustrious official text-to-image API example](https://illustrious-xl.ai/docs/text-to-image) - [Illustrious official tag-booster API](https://dev.illustrious-xl.ai/docs/text-enhance) - [ComfyUI IPAdapter Plus node reference](https://github.com/cubiq/ComfyUI_IPAdapter_plus/blob/main/NODES.md) - [IP-Adapter paper](https://arxiv.org/abs/2308.06721) ## Current Installation Audit Confirmed local stack: - ComfyUI commit `b22d0fb9`, dated 2026-06-25 - IPAdapter Plus commit `a0f451a` - Advanced ControlNet commit `ba0795a` - SDXL IPAdapter Plus ViT-H is installed - SDXL OpenPose, Canny, and anime lineart ControlNets are installed The installed stack is sufficient for the next controlled still-image test. The problem is primarily the graph recipe, not a missing basic ComfyUI capability. ## Previous Recipe Defects | Defect | Direct evidence | Effect | | --- | --- | --- | | Overlong conditioning | Adult positive prompts were 172-195 words; negatives were 98-100 words | Important identity and geometry terms competed with style and failure terms | | Semantic conflict | Negative contained `adult realistic anatomy` while positive required an adult | Negative conditioning could also push away the adult concept | | Foreign trigger leakage | `simple_cartoon` was included in common prompts used with unrelated model/LoRA recipes | Model comparison was contaminated | | Wrong Animagine dialect | Natural-language paragraphs and quality tags at the front | Did not follow the checkpoint's tag-order training recipe | | Non-native resolution | Direct tests used 768x1024 | Did not use the documented 896x1152 portrait bucket | | Seed not actually fixed | State scripts added `+10` and `+20` to neutral seed | Previous state comparison did not test fixed-seed consistency | | IPAdapter over-conditioning | Full portrait, weight 0.86, start 0.0, end 0.9, no attention mask | Reference body geometry and composition were copied with identity cues | | Insufficient geometry control | OpenPose supplied joint points only | Could not enforce a 2.6-head silhouette or head diameter | | Prop in base pass | Phone and neck strap were requested with body, age, outfit, and style | Phone was placed on the forehead in several outputs | ## Controlled Experiments ### 1. LoRA Ablation Path: ```text /data/studio-core/artifacts/media_workbench_research/ round_chibi_pilot_v1/lora_ablation_v1/ ``` Prompt, seed, sampler, scheduler, steps, CFG, and resolution were fixed within each checkpoint pair. Only LoRA loading changed. - Animagine OFF and ON retained nearly the same geometry. - Illustrious generated black circular eyes and the misplaced phone with LoRA OFF. - Illustrious LoRA ON shortened and stabilized the body in that seed. Result: LoRA was not the sole or primary cause of the observed failures. ### 2. Model-Specific Prompt Protocol Path: ```text /data/studio-core/artifacts/media_workbench_research/ round_chibi_pilot_v1/prompt_protocol_v1/ ``` Each row compares legacy prompt, corrected prompt at legacy settings, and corrected prompt at model-oriented settings while retaining the identity seed. - The misplaced phone disappeared when it was removed from the base character pass. - Adult role, haircut, apron, cardigan, and palette adherence improved substantially. - Correct settings improved finish and anatomy, but did not guarantee 2.6 heads. - Changing resolution and CFG with the same seed visibly changed identity details. Result: prompt and inference settings were major failure sources. Text alone still did not provide the hard geometry lock. ### 3. Fixed-Seed State Changes Path: ```text /data/studio-core/artifacts/media_workbench_research/ round_chibi_pilot_v1/fixed_seed_states_v1/ ``` Checkpoint, LoRA, identity prompt, negative, seed, resolution, sampler, scheduler, steps, and CFG were fixed. Only neutral/speaking/extreme state suffixes changed. - The barista retained coarse hair and apron identity, but body proportions, arm length, face, and apron construction changed. - The middle-aged man duplicated in neutral and lost receding hair and mustache cues in later states, reading as a younger character. Result: a fixed seed can preserve coarse correlations for some prompts, but it fails the neutral/action/extreme production identity gate. ## Correct Prompt Rules Prompt construction is enforced by: ```text src/studio_core/youtube_pipeline/character_prompts.py ``` The caller supplies short atomic tags in four bounded groups: immutable identity, state, costume, and render style. The compiler owns quality suffixes, LoRA triggers, model negative baselines, ordering, CFG validation, deduplication, and the final recipe fingerprint. It rejects commas inside a tag, registered props in a base character prompt, foreign LoRA triggers, and protected identity terms repeated in negative conditioning. ### Animagine XL 4.0 Opt ```text 1girl/1boy, solo, original character, safe, immutable identity tags, pose and expression tags, costume and palette tags, flat render tags, masterpiece, high score, great score, absurdres ``` Start with the official concise negative. Add at most a small identity-specific set. Do not insert another LoRA's trigger. Start at 896x1152, 28 steps, CFG 5, Euler Ancestral, normal scheduler. ### Illustrious XL v2 Use concise tags or a tag-booster result. Keep the style-LoRA trigger only when that LoRA is loaded. Start with the official concise negative family. The current research grid is CFG 5.5/6.5/7.0, 28 steps, Euler Ancestral, normal scheduler. The Illustrious recipe is approved only for the current three-state research gate. It is not production-approved until the complete 28-asset pack passes. ## Seed Policy Store a seed only with an exact recipe fingerprint containing all model, LoRA, prompt, control, reference, sampler, resolution, and software-version inputs. - Use fixed seed for exact reruns and one-variable experiments. - Use a small seed search to select a neutral candidate. - Keep the chosen seed across a state experiment, but never count this as an identity gate pass. - Any checkpoint, prompt, resolution, LoRA, ControlNet, or reference change creates a new recipe revision even if the numeric seed is unchanged. ## Production Conditioning Roles | Input | Owns | | --- | --- | | Checkpoint | base anatomy, prompt vocabulary, render prior | | Style LoRA | line/color/style tendency | | Prompt | semantic identity, role, outfit, requested state | | Seed | reproducible starting noise for the exact recipe | | Low-weight masked IPAdapter | approved visual identity cues without copying the whole composition | | Face-excluded Lineart ControlNet | head size, compact silhouette, costume blocks while leaving expression free | | OpenPose | joint action only | | Explicit expression tags | neutral, speaking, and extreme facial roles | | Compositor | phone, microphone, cards, captions, effects | ## Three-State Gate Outcome The accepted comparison is stored at: ```text /data/studio-core/artifacts/media_workbench_research/ round_chibi_pilot_v1/masked_structure_expression_gate_v2/ ``` Verified outcome: - Seed-only neutral/speaking/extreme generation failed because head count and facial geometry changed. - Full-image lineart plus IPAdapter kept the body stable but also suppressed the extreme expression. - Face-only VAE inpaint preserved the body but introduced unacceptable face color and patch boundaries. It is rejected for this character style. - Lineart applied outside a face exclusion mask, combined with IPAdapter weight 0.35 from 15% to 65%, kept the broad silhouette, costume, and composition stable. - Explicit `small open mouth` plus `large eyes` negative separated speaking from `large white eyes`, `wide open mouth`, `comedic shock`. This passes the three-state research gate. Seed remains a reproducibility input, not the identity lock. ## Next Gate Do not generate all 28 assets in one unattended batch. Follow the state-pack order defined in `config/godfortune_character_state_pack_v1.json`: 1. Generate and approve the five body-pose silhouettes. 2. Expand the 12 facial expressions with the accepted face-exclusion recipe. 3. Produce the six mouth states for deterministic viseme timing. 4. Generate the five extreme reactions last because they carry the highest drift risk. 5. Extract transparent assets and run palette, edge-halo, extra-limb, and mobile-readability gates. Props remain separate compositor layers and must not enter the character generation prompt. ## Five-Pose Silhouette Outcome The selected pose pack is stored at: ```text /data/studio-core/artifacts/media_workbench_research/ round_chibi_pilot_v1/five_pose_silhouette_gate_v3/ ``` The first OpenPose attempt failed at 3.5-4 heads because the nose and neck joints were placed too high, leaving an adult-length torso. Moving the face and neck down and compressing hip-to-ankle distance restored the 2-3-head range. - `neutral_front`: approved reference reused; regenerating it introduced a green shadow. - `speaking_one_hand`: silhouette pass. - `presenting_or_pointing`: silhouette pass. - `hands_on_hips_or_folded`: silhouette pass. - `recoil_or_lean`: conditional silhouette pass; pair it with a shock expression. The first recoil draft produced loop-shaped extra limbs. Replacing the two-raised-arm recoil with a simpler lean/spread pose, removing IPAdapter from the draft stage, and adding `extra arms` and `extra hands` negatives removed the defect. This is a silhouette-only pass. Hair fringe, eye geometry, apron construction, and palette still vary across pose drafts. The next gate must refine identity and costume on these fixed silhouettes before generating the 12 facial-expression assets. ## Multi-Character Transfer Outcome The reusable recipe was also tested on three non-barista identities, including one dataset holdout. The accepted comparison is stored at: ```text /data/studio-core/artifacts/media_workbench_research/ round_chibi_pilot_v1/multi_character_transfer_gate_v3/ ``` The source portraits are tall and do not match the delivery geometry. Each identity must therefore pass through a compact neutral normalization stage first. After that neutral is approved, the tall source must not be reused by IPAdapter or structure conditioning; it provides identity and costume evidence only. - `glasses_man`: pass after separating the accessory. Prompt-only and local-lineart attempts made the lenses look like solid black eyes, introduced turquoise irises, produced a sleepy gaze, or removed the frames. The accepted path generates a front-facing character base without eyewear and lets one registered facial layer own the small pupils and transparent glasses frames over neutral, speaking, and shock states. Side-parted black hair, orange cardigan, and khaki trousers remain recognizable. - `elder_bun`: pass after repair. Reusing the tall reference created a detached duplicate head. Using the normalized draft as the identity reference and requiring one silver top bun fixed the composition. Wrinkles and other age cues remain weak. - `curly_raincoat`: conditional holdout pass. The first prompt interpreted the hood as a lantern-like head object and produced T-pose sleeves. `hood down`, `arms at sides`, and the two observed negatives `lantern` and `headwear` removed those defects. Curl volume and the extreme face remain modest. This result supports conditional generalization, not production approval. Character- specific negatives are capped at two and may be added only after a visible failure. The barista and these three identities must pass identity, costume, transparent- background, and palette gates before any full 28-state expansion. ## Four-Character Identity Refinement The current four-character review is stored at: ```text /data/studio-core/artifacts/media_workbench_research/ round_chibi_pilot_v1/character_identity_gate_v1/ ``` The user approved the separated eye-and-glasses layer on 2026-07-21. Two remaining identity defects were then repaired without changing approved dialogue-state logic: - `elder_bun`: registered crow's-feet and smile-line strokes restore readable age cues while the silver bun, tracksuit, body, and mouth states remain untouched. - `curly_raincoat`: a face-erased local hair lineart guide restores shoulder-wide ringlet volume. The ambiguous word `halo` was rejected after it created a literal yellow halo. A bounded HSV correction inside the registered hair region restores brown hair without recoloring the yellow raincoat. The user approved both corrected identities on 2026-07-21. They no longer block the barista pose gate. ## Barista Five-Pose Identity Refinement The current review candidate is stored at: ```text /data/studio-core/artifacts/media_workbench_research/ round_chibi_pilot_v1/barista_five_pose_palette_refinement_v3/ ``` Replacing the complete head failed even when the accepted neutral head was reused. It created neck seams, duplicate hair around the old silhouette, and an oversized head-to-body relationship. Higher IPAdapter weight also changed the eyes, body height, apron, and background glow. The five approved pose rasters are therefore kept as the structural source. The accepted candidate changes only bounded palette regions. Brown-red hair pixels and mint apron pixels are normalized to samples from the approved neutral master; hands, feet, pose lineart, and expression pixels are untouched. This restores a coherent hair and costume palette while keeping deliberate deadpan and recoil eyes. Apron construction remains a separate blocker. A bounded VAE inpaint removed the bow but changed the apron to dark olive and introduced a fold. Direct polygon fill removed the detail but broke the upper and side outlines. Both outputs were rejected. The next costume step is one authored plain-apron layer registered per body pose, not another full-character diffusion pass. After visual approval of the palette and face direction, extract transparent foregrounds, validate edge halos and palette metrics, then begin the twelve ordinary expressions. The recoil pose must receive a registered shock expression in that step.