Pocket TTS: 100M-Parameter TTS and Voice Cloning

In recent years, the Text-to-Speech ( TTS ) technology has developed significantly.The voices have become very natural, realistic pauses, good pronunciations and expressive prosody.

Text-to-Speech (TTS) powers voice bots, virtual assistants, voice search, IVR systems, accessibility tools, audiobooks, and real-time voice-enabled applications.

Nonetheless, such profits are usually at the expense. Most of the models are big with a requirement of heavy computing power and GPUs. Pocket TTS does the reverse.

This model is an open-source one with only 100 million parameters and can even run faster than real time on an ordinary CPU. It does not need a GPU.

It is amazing how it manages to do so without sacrificing sound quality. The secret is in a new architectural re-consideration known as Continuous Audio Language Models (CALM).

The weaknesses of the traditional TTS strategies

The current state of the art TTS systems are usually based on either of the two following approaches:

Neural audio codecs turn raw audio into discrete tokens, which a language model then predicts, similar to how it predicts text.

Diffusion-based techniques that, but not in the same way, do not use tokens but instead do dozens of iterative denoising steps. The two cause grave bottlenecks:

Discrete tokens are lost in the case of compressed data. They require additional bits, tokens and a lot more computing power to achieve a higher quality.

Predicting audio one token at a time is slow because each frame needs many tokens. Diffusion models are effective yet slow.

They demand numerous denoising steps, and cannot be usable in real-time on a CPU. There is always a tradeoff between quality, inference speed and model size. Pocket TTS avoids this completely.

The Breakthrough Continuous Audio Representations

Pocket TTS works with continuous audio instead of fixed symbols. It processes sound from start to finish.

There is no tokenization, no quantization errors and no tokenization explosion in long sequences.

This fundamental decision eliminates whole levels of complexity and inefficiency.

Introduction to High-Level Architecture

The system is composed of the three pieces which combine beautifully:

1) Continuous Latent Audio VAE

A causal Variational Autoencoder turns raw audio into smooth latent vectors and then back into sound. The lack of discrete codebook implies there is no problem of collapse or trade-offs in bit rate.

The quality of the sound is equal or superior to the token-based techniques of the same scale. It is also fully causal, and hence supports streaming and real-time usage.

2) Causal Backbone of Transformers

At the heart of it lies a Transformer which interprets text tokens and audio already produced by it.

An intentional, limited delay gives the model an opportunity to peep at future words. This assists it in making better choices in rhythm and tone as well as pronunciation before speaking.

This brings about more stable alignment, improved pronunciation and smooth natural flow.

3) Head of one-step consistency model

The model adopts the method of consistency instead of slow diffusion to clean noisy data in a single step.

It does not have any repetitive denoising processes or convoluted loops. One fast prediction is made in the creation of each audio frame. This is the first cause why Pocket TTS has blazing CPU speeds.

Trick training of Stable One-Step Inference

Single-step generation may add errors even when not done with care. Pocket TTS is having smart tricks to this:

During training, model purposefully adds noise to the previous audio data and feeds it to the Transformer. This assists model to remain stable even in cases where the last outputs are not ideal.

The amount of noise added determines the style of output. The low noise produces more stable speech whereas high noise produces more expressiveness and variety.

Classifier-Free Guidance is used in the latent space to make speech follow the text better and stay clear, without slowing down the model.

How Pocket TTS remains so small (Just 100M Parameters)

This entire teacher model begins bigger (~300M).

Latent distillation is applied to make pocket TTS smaller. A small model is trained to imitate the internal audio patterns in the teacher with the same high prediction head.

The result is fewer layers, much lower memory and computing needs, and almost no loss in quality.

Performance Highlights of Pocket TTS

• Competitive or optimal Word Error Rate (WER) and Character Error Rate (CER) vs considerably larger models
• Good mean opinion score (MOS) and human preference ratings.
• Real time generation on the normal CPUs.
• Actual one frame per second efficiency.
• Zero-shot voice cloning (using short reference clips) of high quality.

This is an unusual combination of practical quality + practical speed that makes it special.

Why This Matters

Pocket TTS proves that:

• The use of discrete audio tokens is not a necessity
• The only path to great synthesis is diffusion
• With continuous modeling, smaller, cleaner and much faster results are possible

If this gradual approach keeps working, discrete token methods may end up being a temporary solution rather than the final one.

Pocket TTS is completely open source. It works well with local applications, privacy-centric projects, edge computers, or just with people who do not wish to use GPUs or the cloud.