VoxCPM2: fine-tune a custom voice in 10 minutes on one GPU

Hosted voice APIs are convenient until a client asks for a custom brand voice that never leaves their VPC. That is when open-weights TTS stops being a hobby project and becomes infrastructure.

VoxCPM2 from OpenBMB is the August 2026 answer gaining traction: a 2 billion parameter tokenizer-free TTS model, 30 languages, 48kHz output, and LoRA fine-tuning from 5-10 minutes of audio. The repo crossed 36,000 GitHub stars the week it hit my digest.

I deploy voice agents for clinics and service businesses. This is the class of tool I evaluate when "we cannot send audio to ElevenLabs" is a hard requirement.

What makes VoxCPM2 different

Most TTS stacks convert audio to discrete tokens, generate tokens, then decode back to waveform. VoxCPM2 skips the tokenizer entirely. It generates continuous speech representations end-to-end on a MiniCPM-4 backbone.

That matters for voice cloning because tokenization bottlenecks often smooth away the micro-details that make a voice sound like a specific person.

Three cloning modes

VoxCPM2 ships three distinct voice workflows:

1. Voice Design — describe a voice in natural language (gender, age, tone, pace). No reference audio needed.
2. Controllable Cloning — upload a short reference clip, then steer emotion, speed, and style while keeping the timbre.
3. Ultimate Cloning — provide reference audio and its transcript. The model continues from the reference, preserving rhythm and vocal nuance.

Mode 3 is what you want when a client gives you 30 seconds of their founder's podcast intro and expects the agent to sound like the same person on a sales call.

Fine-tuning in 10 minutes of audio

The fine-tuning docs are practical, not aspirational.

OpenBMB tested 5 minutes and 10 minutes of single-speaker audio. Both converged after ~2,000 training steps with default LoRA config. Internal benchmarks put LoRA at roughly 98% speaker similarity versus full SFT, at half the VRAM and orders-of-magnitude smaller checkpoints.

Basic LoRA launch:

python scripts/train_voxcpm_finetune.py \
  --config_path conf/voxcpm_v2/voxcpm_finetune_lora.yaml

Inference with a fine-tuned checkpoint:

python scripts/test_voxcpm_lora_infer.py \
  --lora_ckpt /path/to/checkpoints/finetune_lora/step_0002000 \
  --text "This is the cloned voice on a support call." \
  --prompt_audio reference.wav \
  --prompt_text "Exact transcript of reference.wav" \
  --output cloned_output.wav

The WebUI covers training and inference without writing scripts. Hugging Face hosts weights and a demo space.

Performance numbers that matter for voice agents

Latency drives conversational UX. Reported real-time factors on an RTX 4090:

RTF under 1.0 means faster than real-time speech. At 0.13, you have headroom for STT, LLM, and network hops in a voice agent loop.

For comparison, I covered Fish Audio S2.1 Pro earlier: hosted API, ~70ms TTFB, closed production stack. Fish wins on managed latency and scale. VoxCPM2 wins on data sovereignty, fine-tuning control, and zero per-character fees.

When I pick VoxCPM2 over a hosted API

Choose self-hosted VoxCPM2 when:

• Audio cannot leave the client's network (HIPAA-adjacent clinics, legal, finance)
• You need a persistent custom voice trained on proprietary recordings
• Per-minute API costs would exceed one GPU rental at your call volume
• You want Apache 2.0 freedom for a white-label voice product

Stay on hosted APIs when:

• You need 50+ languages day one without training data
• Your team cannot operate GPU inference
• Sub-100ms TTFB is non-negotiable and you will not optimize locally
• Compliance is simpler with a vendor DPA than self-hosted audit

Integration pattern for voice agents

The stack I wire for on-prem voice:

1. STT layer — on-device model like Audio8 ASR 0.1B or cloud fallback
2. LLM — whatever the client approves (local or API)
3. TTS — VoxCPM2 with LoRA checkpoint for the brand voice
4. Orchestration — LiveKit, Pipecat, or custom WebSocket bridge

VoxCPM2's vLLM-Omni deployment path gives you an OpenAI-compatible TTS endpoint, which drops into many agent frameworks without custom adapters.

Risks and honest limits

• 30 languages sounds broad, but quality varies. Test your target locale before committing.
• 8GB VRAM is manageable, but production needs redundancy, health checks, and model versioning.
• Voice cloning ethics are your problem. Get consent for reference audio. Disclose AI voices to end users.
• The repo is moving fast (VoxCPM1.5 to VoxCPM2 in months). Pin versions in production.

VoxCPM2 is the most practical open voice cloning stack I have seen for teams that need custom timbre without custom ML research. Ten minutes of audio and a single GPU is a real threshold, not marketing.

Building an on-prem voice agent with a custom brand voice? Book a free discovery call.