Add CampPlus speaker-embedding conditioning to Ming Omni TTS (bailingmm) - #837
Open
MukaiTensho wants to merge 1 commit into
Open
Add CampPlus speaker-embedding conditioning to Ming Omni TTS (bailingmm)#837MukaiTensho wants to merge 1 commit into
MukaiTensho wants to merge 1 commit into
Conversation
The official Ming-omni-tts voice-cloning path conditions the LLM on a 192-d CampPlus speaker embedding injected via spk_head into a " speaker_1:<spk><audioPatch></spk>\n" block (see the official modeling_bailingmm.prepare_input_embed and spkemb_extractor.py). The port defined spk_head and converted the campplus weights but never used either, so voice identity relied solely on the acoustic prompt latents and drifted on expressive text (wrong-speaker output and early stop-head truncations). - extract the CampPlus embedding (16 kHz kaldi fbank, 80 bins, dither=0, per-dim mean subtraction -> campplus.onnx via onnxruntime) when ref_audio is a path and campplus.onnx exists in the model dir - build the official <spk> block and replace the placeholder embedding with spk_head(embedding), mirroring the reference implementation - honor the existing --use_zero_spk_emb flag (previously silently ignored by this model) - optional deps (onnxruntime, kaldi-native-fbank) are imported lazily; extraction failure falls back to the previous behaviour Validated on Ming-omni-tts-16.8B-A3B-bf16 (M1 Ultra): cloned speech F0 locks to the reference speaker (114 Hz ref -> 112-126 Hz out) where it previously drifted, e.g. to 267-286 Hz on emotional sentences. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AAZHRZo8bhYk7nLeJDuUS
lucasnewman
reviewed
Jul 20, 2026
| def _campplus_session(self): | ||
| sess = getattr(self, "_campplus_sess", None) | ||
| if sess is None: | ||
| import onnxruntime |
Collaborator
There was a problem hiding this comment.
This shouldn't use ONNX, we should just port it to MLX. CAM++ is independent of this model (see https://arxiv.org/abs/2303.00332) and we already have at least one implementation in MLX -- we should just convert the weights to load it in-graph.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Voice cloning with Ming Omni TTS (
bailingmm) loses the speaker identity on expressive text: sentences with strong prosody (exclamations, questions, emotional lines) frequently come out as a completely different speaker (e.g. a male reference cloned into a female voice) or get truncated to a ~1.6 s stub by the stop head.Root cause: the official implementation conditions generation on a CampPlus speaker embedding (
spkemb_extractor.py, used viause_spk_emb=Truein every official cloning example) which is injected into the sequence as aspeaker_1:<spk><audioPatch></spk>\nblock whose placeholder embedding is replaced byspk_head(embedding)(modeling_bailingmm.prepare_input_embed). The port ships all the pieces —spk_headis defined and its weights are loaded,campplus.onnxis even converted to safetensors inpost_load_hook— but none of it is ever used:from_pretrainedskips campplus weights,spk_headhas no call sites, and the input sequence has no<spk>block. Without the identity anchor the model relies purely on prompt-latent continuation, which holds for plain declarative text but drifts on expressive text. The--use_zero_spk_embCLI flag was also silently ignored for this model.Change
dither=0, per-dim mean subtraction →campplus.onnxvia onnxruntime) whenref_audiois a path andcampplus.onnxexists in the model directory.<spk>block and replace the placeholder embedding withspk_head(embedding)atspk_idx + 1, mirroring the reference implementation (MoE variant).--use_zero_spk_emb(zeroed identity vector after projection, per the official cookbook's voice-design/IP examples).onnxruntime,kaldi-native-fbank) are imported lazily and extraction failures fall back to the previous behaviour, so nothing breaks for users withoutcampplus.onnx.Validation
Ming-omni-tts-16.8B-A3B-bf16 on M1 Ultra, cloning a male reference (median F0 114 Hz), official decode params (
temperature=0.0,cfg 2.0,sigma 0.25):A separate instability remains (occasional 1.6 s silent stubs from the flow decode even at
temperature=0.0, much more frequent with instruction captions) — that looks like an independent issue in the decode loop and is out of scope here; happy to file details separately.🤖 Generated with Claude Code
https://claude.ai/code/session_018AAZHRZo8bhYk7nLeJDuUS