# Local Hybrid Music Pipeline Audit v1 Date: 2026-07-23 ## Decision Adopt the staged resource-control pattern, but do not adopt the proposed pipeline verbatim. The verified design is an evidence-gated pipeline in which each optional stage must improve the same locked-lyrics candidate before it is retained. The current host has one RTX 3060 with 11.63 GiB usable VRAM, 15 GiB system RAM, 4 GiB swap, and about 296 GiB free on `/data/studio-core`. It cannot use the two-GPU HeartMuLa placement example. Every large audio engine must run under one exclusive GPU lease, and resident ACE-Step or ComfyUI workers must release GPU and host memory before another engine starts. ## YouTube Commercial Boundary Godfortune is treated as commercial production from the first public upload. This is true even when Studio is not sold as SaaS: monetization, sponsorship, channel growth, and indirect business benefit can all make the use commercial. A model blocked from commercial or production use may be measured internally, but none of its audio, embeddings, rewrites, or derivatives may enter a published artifact. The license gate checks five independent layers: repository code, exact official model weights and revision, generated-output terms, every third-party checkpoint or LoRA, and the rights to training/reference performances and voices. A permissive code license does not clear an unlicensed community checkpoint or an identifiable performer's voice. | Component | Official terms checked | Godfortune use | | --- | --- | --- | | ACE-Step 1.5 code and official XL weights | MIT, not Apache 2.0 | Production-eligible; Korean diction still unqualified. | | HeartMuLa repository and related official weights | Apache 2.0 | Production-eligible; quality-first local candidate. | | DiffRhythm 2 code and weights | Apache 2.0 | Production-eligible; second local candidate. | | YuE current official repository and weights | Apache 2.0; monetization encouraged with `YuE by HKUST/M-A-P` credit requested | Production-eligible with attribution recorded; accelerated runtime still unqualified. | | Stable Audio 3 | Stability AI Community License | Conditional accompaniment use only: register commercial use, remain below USD 1M aggregate annual revenue or obtain Enterprise terms, and retain the exact model terms. | | LeVo 2 / SongGeneration 2 | Research/education only; commercial or production use prohibited | Internal research only; no output may enter YouTube artifacts. | | EXAONE 3.5 model and output | EXAONE NC; commercial use of model, derivatives, and output prohibited | Excluded from production and unnecessary for locked lyrics. | | RVC code | MIT | Engine is eligible, but only with an owned or contractually cleared voice dataset and checkpoint. | | BSRoformer.cpp code | MIT | Code is eligible; each converted community separator weight needs its own explicit commercial license. | Stable Audio 3 is not a Korean singer candidate. Its official prompting guide says the models do not produce intelligible vocals, so it is evaluated only for instrumental beds or stems. Open-source or open-weight status alone is never used as a production approval. ## Claim Audit | Claim | Result | Engineering consequence | | --- | --- | --- | | HeartMuLa supports `--lazy_load` | Verified | Required on this single-GPU host. The implementation deletes HeartMuLa after token generation, runs garbage collection and `torch.cuda.empty_cache()`, then loads HeartCodec. | | HeartMuLa supports separate LM and codec devices | Verified | Useful only on a multi-GPU host. This server exposes only `cuda:0`. | | HeartCodec BF16 is a safe memory optimization | Rejected | The official guide warns that BF16 codec inference may degrade audio quality. Keep HeartMuLa BF16 and HeartCodec FP32 for the quality test. | | Long RVC files always cause an indexing burst failure | Rejected | Official RVC preprocessing already silence-splits and then produces approximately 3.7-second chunks with 0.3-second overlap. Failure is not inevitable and an extra pydub slicer is not required. | | `-40 dB`, 500 ms, and 3-5 seconds are mandatory RVC settings | Partly true | Short clean clips are appropriate, but current official defaults are approximately -42 dB, 400 ms minimum interval, 500 ms retained silence, 3.7-second chunks, and 0.3-second overlap. Dataset inspection matters more than copying one threshold. | | RVC is generally optimal at 140-160 epochs | Unsupported | The official trainer runs to `total_epoch` and does not implement validation-based early stopping. Select checkpoints by held-out song tests; do not encode a universal epoch range. | | RMVPE corrects singing technique and high notes | Rejected | RMVPE estimates F0 robustly. It does not invent correct notes, diction, vibrato, phrasing, or breath control. RVC mainly transfers timbre while following the source timing and pitch. | | BS-RoFormer gives perfect or lossless separation | Rejected | It estimates masks in the STFT domain and reconstructs audio. WAV output can be losslessly encoded, but bleed, phase, transient, and reassembly artifacts remain possible. | | `BSRoformer.cpp` is a mature ByteDance C++ product | Rejected | The architecture originates from the SAMI-ByteDance paper, but the GGML C++ port is a small community project. CUDA/Vulkan and quantized models exist, but runtime, weight licenses, and output parity require qualification. | | EXAONE 3.5 can be the production lyric planner | Rejected | EXAONE 3.5 uses the EXAONE 1.1 NC model license. More importantly, Studio is not allowed to rewrite content-server-approved lyrics. A planner may only fill Studio-owned metadata. | | This chain guarantees Suno-grade output | Rejected | Separation, conversion, and mastering cannot restore lyrics or musical information absent from the guide generation. Suno v4.5 remains a human-reviewed reference normalized to 100, not a guaranteed outcome. | ## Corrected Pipeline ### 0. Immutable intake and deterministic planning - Preserve the approved dialogue, lyrics, order, checksum, and musical interval. - Compile only Studio-owned section tags, tempo candidates, key candidates, engine settings, and evaluation plans. - Do not add EXAONE 3.5 to the production path. An LLM is optional for research metadata suggestions and must never change locked content. ### 1. Whole-song qualification - First production candidate: HeartMuLa 3B under its official Apache 2.0 license, `--lazy_load true`, HeartMuLa BF16, HeartCodec FP32, batch one. It moves first because the objective is maximum vocal quality rather than shortest setup time. - Second production candidate: DiffRhythm 2 with exact approved lyrics. - Third production candidate completed: ACE-Step 1.5 XL under MIT. INT8 plus CPU/DiT offload fit at 9.73 GiB peak CUDA, but the exact package recovered only 25.27% of Korean characters and 12.5% of ordered words. A 72-second duration-only control reached 50.55% and 47.5%, proving that the approved lyric density is the primary local bottleneck. A dry acoustic style-only control improved separated-vocal accuracy from 25.27% to 35.16%, confirming arrangement and vocal-effect masking as a secondary cause. Dry a cappella reached only 32.97%. Direct whole-song vocal use is blocked. - Fourth production candidate: one corrected YuE-exllamav2 JP/KR run. It keeps its Apache 2.0 provenance and requested attribution, but stops after one more lyric gate failure because the prior local profile produced no approved words. - Research-only control: LeVo 2 medium because its current license prohibits commercial and production use. - Stable Audio 3 Medium and ACE-Step XL remain separate accompaniment candidates even when their direct Korean vocal path is not qualified. No model advances because it rendered successfully. It must pass lyric order, omission, repetition, Korean diction, musical continuity, and human listening gates. ### 2. Native stems before estimated stems Prefer a model that generates native vocal and accompaniment tracks. If only a mix exists, compare the current HTDemucs path against BSRoformer.cpp FP16 and Q8 on the same lossless mix. Measure vocal bleed, instrumental bleed, transient loss, high-frequency loss, and audible boundary artifacts. Verify the license of each converted weight separately from the MIT C++ code license. The separator is rejected for that song when the separated vocal or instrumental scores below the original mix or when ghost vocals remain in the accompaniment. ### 3. Optional character-timbre conversion RVC is optional and may only use a rights-cleared character/performance dataset. Do not train against an identifiable artist without explicit rights. Feed RVC the cleanest available dry guide vocal; avoid stacking separator artifacts when a native vocal track exists. Use official preprocessing as the baseline. Split training, validation, and holdout by source song before slicing so adjacent chunks from one recording cannot leak across splits. Save checkpoints on a fixed step schedule and rank them using: - approved-lyrics CER/WER and ordered line coverage; - F0 correlation, voiced/unvoiced error, and octave errors; - speaker or character similarity; - clipping, spectral discontinuity, metallic-noise, and breath artifacts; - blind Korean listener preference. Stop when validation quality plateaus or worsens. Epoch 140-160 may be included in the checkpoint sweep, but it is not a hard-coded optimum. ### 4. Mix and master - Keep lossless intermediate stems and their native sample rate. - Align replacement vocals before mixing and fail on drift or phoneme truncation. - Apply subtractive EQ, conservative compression, de-essing, automation, limiting, loudness, and true-peak gates. - Export the approved master at 48 kHz WAV. Resampling a lower-quality source to 48 kHz does not create missing fidelity. ## Resource Lifecycle Each large model stage runs in its own subprocess: 1. Acquire the exclusive GPU lease and stop or unload resident GPU engines. 2. Verify output/cache paths are under `/data/studio-core`, not the nearly full root filesystem. 3. Record free VRAM, available RAM, swap use, model revision, dtype, and command. 4. Load one model family and checkpoint stage output to disk. 5. Exit the subprocess to guarantee CUDA-context reclamation. 6. Verify VRAM and RAM returned below the post-stage threshold before the next load. 7. Resume from the checkpoint after OOM or worker failure instead of restarting the complete song. In-process `del`, garbage collection, and `empty_cache()` remain useful, but process isolation is the stronger service boundary. A watchdog must mark the stage `resource_blocked` rather than allowing the Linux OOM killer to terminate Studio. ## Qualification Order 1. HeartMuLa 3B lazy-load hardware fit and exact-lyrics listening test. 2. DiffRhythm 2 exact-lyrics generation and hardware fit. 3. ACE-Step 1.5 XL exact-lyrics test, then one corrected YuE-exllamav2 run. 4. Stable Audio 3 Medium versus ACE-Step instrumental accompaniment test. 5. HTDemucs versus only license-cleared BSRoformer.cpp weights. 6. RVC no-conversion versus conversion A/B using a rights-cleared character voice. 7. Final mix/master A/B against the Suno v4.5 reference normalized to 100. Do not install RVC or train a voice model until a guide vocal passes the lyric and performance gates. Voice conversion cannot repair a failed singer. ## HeartMuLa Qualification Result The first local candidate completed on 2026-07-23 using seed `20260723`, the official 3B and HeartCodec revisions, HeartMuLa BF16, HeartCodec FP32, CFG 1.5, and lazy loading. The upstream wrapper initially allocated an 8,192-token KV cache regardless of requested audio length and exhausted the 11.63 GiB device. The local runtime now sizes the cache to prompt length plus requested audio frames. This changes cache capacity only; weights, dtype, CFG, sampling, and generated token positions remain unchanged. - Output: 26.72-second, 48 kHz stereo PCM WAV. - Wall time: 49.44 seconds; maximum host RSS: 5,837,464 KiB. - HeartMuLa GPU allocation before unload: 7.68 GiB. - HeartCodec GPU allocation before unload: 6.20 GiB. - Separated-vocal automatic character accuracy: 82.42%. - Ordered approved-word coverage: 70%; omitted words: 12 of 40. - Raw mix: -12.62 LUFS integrated and +0.22 dBTP. The hardware gate passes. The locked-lyrics and signal gates fail, and human Korean and music-quality review are still pending. This candidate is reviewable but not production-approved. Do not start RVC or mastering as a way to hide the missing lyrics; compare the result directly, then continue to DiffRhythm 2 as the next independent engine qualification. ## DiffRhythm 2 Qualification Result The second local candidate completed on 2026-07-23 using seed `20260723`, the official model and MuQ revisions, FP16, 16 steps, CFG 2.0, and text-only style conditioning. The official inference entry point classified Hangul as `other` despite including a Korean phonemizer, so the local runtime routes input through the repository's configured multilingual tokenizer. Approved words and order were unchanged. - Requested duration: 26.67 seconds; decoded output: 28.03 seconds. - Wall time: 30.81 seconds; maximum host RSS: 9,822,312 KiB. - Peak CUDA allocation: 4.86 GiB. - Separated-vocal automatic character accuracy: 4.40%. - Ordered approved-word coverage: 0%; omitted words: 40 of 40. - Full-mix ASR recovered no speech; the separated-vocal transcript was unrelated to the approved lyrics. - Raw mix: -16.99 LUFS integrated and -3.28 dBTP. The hardware, loudness, and peak gates pass, while duration and locked-lyrics gates fail. This is a clear Korean production failure rather than an accompaniment-masking issue, so additional DiffRhythm 2 seed search is stopped. The next independent engine qualification is ACE-Step 1.5 XL. A duration and supported-language control isolated the failure. The released inference entry point is explicitly Chinese-English only and rejects Hangul as `other`. A local multilingual-phonemizer connector can make Hangul execute, but a 152.23-second same-lyrics control still generated English-classified vocal-like audio with no approved Korean words. The repository's official Chinese example, run through the same model, sampler, FP16 path, and decoder, was classified as Chinese with 99.23% probability and recovered many input phrases. This rules out a generally broken installation. The public Korean comparison audio also has an empty Korean ASR transcript, so it must not be treated as proof of exact Korean conditioning. ## Sources - https://github.com/ace-step/ACE-Step-1.5/blob/main/LICENSE - https://github.com/ace-step/ACE-Step-1.5/blob/main/docs/en/ace_step_musicians_guide.md - https://github.com/HeartMuLa/heartlib - https://github.com/ASLP-lab/DiffRhythm2 - https://github.com/multimodal-art-projection/YuE - https://github.com/levo-demo/LeVo/blob/main/LICENSE - https://github.com/LG-AI-EXAONE/EXAONE-3.5/blob/main/LICENSE - https://stability.ai/license - https://github.com/Stability-AI/stable-audio-3/blob/main/docs/guides/prompting.md - https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI - https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/wiki/Instructions-and-tips-for-RVC-training - https://arxiv.org/abs/2306.15412 - https://arxiv.org/abs/2309.02612 - https://github.com/chenmozhijin/BSRoformer.cpp - https://huggingface.co/chenmozhijin/BSRoformer-GGUF - https://github.com/LG-AI-EXAONE/EXAONE-3.5