OmniVoice Real-Time TTS for Polish

Benchmarking OmniVoice against Chatterbox on RTX 5090 — latency, streaming, voice cloning, and audio quality compared head-to-head.

The Models

Architecture Overview

Key Architectural Difference

Chatterbox is real-time because it streams token-by-token – the autoregressive decoder emits tokens sequentially, each decoded to audio and sent immediately. TTFA = time to generate the first few tokens.

OmniVoice uses masked diffusion – all audio tokens start as [MASK] and are iteratively unmasked over N steps. Each step runs a full forward pass over the entire sequence. Tokens are revealed by confidence score, not position. Partial audio cannot be decoded mid-generation.

However, OmniVoice’s raw inference speed is so fast (RTF 0.01–0.04) that text-level chunking achieves comparable or better TTFA than Chatterbox’s token streaming.

Test 1: Baseline Latency

Full generation, voice cloned, no chunking. Total wall time from generate() to returned tensor.

• RTF improves with longer text (fixed overhead amortized over more audio).
• 16 steps is the sweet spot: ~2x faster than 32, quality degradation is minimal.
• At 8 steps, RTF drops to 0.009–0.054 – 18–110x faster than real-time.

Test 2: Chunked Streaming (TTFA Measurement)

Simulates streaming: split text at sentence boundaries, generate each chunk independently, measure time to first completed chunk.

Medium text (196 chars, 2 chunks)

Long text (488 chars, 5 chunks)

Per-chunk breakdown (long text, 16 steps):

• At 16 steps, TTFA is 169ms – 32% faster than Chatterbox’s typical 250ms.
• Each chunk generates ~4–7s of audio in ~150–170ms. The playback buffer is enormous.
• Chunking overhead is minimal: total gen time ~30% higher than single-shot, but the streaming benefit far outweighs it.

Test 3: Voice Clone Prompt Caching

create_voice_clone_prompt() pre-encodes reference audio into reusable tokens.

Prompt encoding is already fast (37ms). Caching is still worthwhile for a server to avoid redundant re-encoding.

Test 4: Concurrent Inference

Same text, 3 requests

Mixed text lengths, 3 concurrent

• GIL + shared model weights limit true parallelism to ~1.66x speedup for 3 threads.
• Individual request latency increases ~1.8x under contention.
• Chatterbox’s InferenceSlot + SlotPool pattern would improve this further.

Test 5: Pipeline Breakdown

Detailed instrumentation of every pipeline stage, averaged over 5 runs per configuration. RTX 5090, float16, voice-cloned.

Where the milliseconds go

The OmniVoice pipeline has 7 stages. Stages 1–4 (preprocessing) are negligible. Stage 5 (iterative decode) dominates at 87–97% of total time.

Long text (488 chars, 28.5s audio, 950 tokens)

Per-step cost

Each diffusion step runs a full Qwen3-0.6B forward pass over the entire sequence. Cost scales with sequence length:

All steps are uniform in cost – no step is significantly more expensive than others. Halving the step count halves the LLM time almost exactly.

Key findings

• LLM forward passes = 87–91% of total time. Everything else is rounding error.
• Text tokenization: <1ms. Negligible even for long text.
• Duration estimation: <0.1ms. Rule-based, no neural network.
• CFG + token selection: 5–8%. Classifier-free guidance math, Gumbel sampling, top-k.
• Audio decode (vocoder): 4–21ms. HiggsAudioV2 on GPU. Scales with audio duration.
• Post-process: 3–24ms. Silence removal via pydub (CPU). Linear with audio length.
• Optimization target: Reducing per-step LLM cost (quantization, flash attention, KV caching) would yield near-linear total speedup.

Test 6: First-Chunk-Optimized Streaming

Best strategy: split at first sentence, generate short first chunk for minimum TTFA, generate rest while first chunk plays.

Even for the longest text (488 chars, 29s audio), there is zero playback gap at any step count. The first chunk produces 7s of audio, providing a massive buffer window. Margin is 6–7 seconds – enough to absorb network jitter, encoding, and client buffering.

Test 7: GPU Memory Profile

Incremental cost per concurrent inference is ~150 MB. Substantial headroom for additional model instances or concurrent requests.

OmniVoice vs Chatterbox

Chatterbox vs OmniVoice – Side-by-Side (weronika voice)

Same text, same voice (weronika), same GPU. Chatterbox generated via production server. OmniVoice at 16 steps (recommended) and 32 steps (best quality).

Tiny – “Dzien dobry.”

Short – News sentence (66 chars)

Medium – PLL LOT incident (196 chars)

Long – Full news story (488 chars)

OmniVoice Internal Comparisons

Step count and chunking tradeoffs within OmniVoice.

32 vs 16 steps (medium text)

Baseline vs Chunked (medium text, 16 steps)

8-step quality floor (medium text)

Optimized streaming vs Baseline (long text)

Paralinguistics: Non-Verbal Sound Tags

OmniVoice supports expressive non-verbal tags embedded directly in text. All samples: Polish, voice-cloned (weronika), 16 steps.

Supported tags: [laughter], [sigh], [confirmation-en], [question-ah], [question-oh], [question-ei], [question-yi], [surprise-ah], [surprise-oh], [surprise-wa], [surprise-yo], [dissatisfaction-hnn]

All tags produce distinct non-verbal sounds at the marked positions. The mixed sample (4 tags, 10.44s) demonstrates natural flow between speech and emotions. Generation time stays consistent (~150ms) regardless of tag count; the mixed sample is longer (247ms) only because the text itself is longer.

Conclusion

OmniVoice is viable for real-time streaming TTS and outperforms Chatterbox on raw speed metrics. The masked-diffusion architecture prevents true token-level streaming, but sentence-level chunking achieves 169ms TTFA at 16 steps with zero playback gaps. Combined with excellent voice quality, 600+ language support, and low VRAM footprint, OmniVoice is a strong candidate for production TTS deployment.

For consulting on real-time TTS integration, streaming architecture, or AI/ML engineering, contact Folx.