11 min read

Arxiv Dives - Generating Speech from Text with Fast Speech-2

Arxiv Dives - Generating Speech from Text with Fast Speech-2

Every Friday at Oxen.ai we host a public paper club called "Arxiv Dives" to make us smarter Oxen 🐂 🧠. We believe diving into the details of research papers is the best way to build fundamental knowledge and keep up with the bleeding edge. If you would like to join the discussion live, sign up here.

The following are the notes from the live session. Feel free to follow along with the video for the full context.

Fast Speech-2 Resources

Paper: https://arxiv.org/abs/2006.04558

Examples: https://speechresearch.github.io/fastspeech2/

Why Text to Speech? (TTS)

Audio generation is fascinating, but doesn’t get as much attention as LLMs or Computer Vision.

Speech is the oldest form of communication between people. There are millions of hours of podcasts, and videos, and audio out there on the internet that capture nuanced interactions and tons of knowledge.

If you think about our sensory system as humans, sight and sound are two of the most important senses. (Smell, taste, and touch are hard to encode into a computer, until we have a smell-o-scope).

Jokes aside, I think the trifecta that currently are encoded into computers, and are everywhere, are vision (image & video), audio (podcasts, music, video), and text (which is just kind of a lossy encoding of human thought).

Generating audio could be a key component to intelligence. A great book on Generative Deep Learning States:

Generative Deep Learning, 2nd Edition
Generative AI is the hottest topic in tech. This practical book teaches machine learning engineers and data scientists how to use TensorFlow and Keras to create impressive generative deep learning … - Selection from Generative Deep Learning, 2nd Edition [Book]

In order to generate something as complex as audio that has meaning, you have to have an underlying representation of that meaning.

If neural networks can generate speech or audio, we might be able to use the same network of knowledge for general intelligence.

Background Knowledge

If you have never done work in speech, we are going to try to get us from zero to hero in audio land, assuming very little prior knowledge (besides having ears 👂) to make the rest of the paper make sense.

Waveform = Way to represent raw audio. You may have 44,100 samples per second, for a total of 5,000,000+ samples for a 2 minute audio clip. Or a 5 second audio clip would be 44,100*5=220,500 samples.

How to Visualize Sound in Python
There’s an abundance of music and voice data out there and interesting applications to go with them. Here, we show you how to visualize sound in Python.

Mel-spectrograms = is an image with frequency on one axis, time on the other, and amplitude (decibels) as the color.

Getting to Know the Mel Spectrogram
Read this short post if you want to be like Neo and know all about the Mel Spectrogram!

Humans can only hear in certain frequencies.

Dogs can hear in much higher frequencies.

Mel Scale is constructed such that sounds of equal distance from each other on the Mel Scale, also “sound” to humans as they are equal in distance from one another.

Where the difference between 500 and 1000 Hz is obvious, whereas the difference between 7500 and 8000 Hz is barely noticeable.

Phoneme = smallest unit of speech distinguishing one word (or word element) from another. Think of it like a token of text, but for speech. An example of two phonemes might be “ta” and “p” for “tap”, which can distinguish it from “tab” or “tag” or “tan”.


Autoregressive = Predict the next thing given the previous thing. Work really well, but are very computationally expensive if the sequence is long.

Non-Autoregressive = Any other strategy. In this case a strategy similar to WaveNet from Deepmind

Distillation = Taking knowledge and information from larger models and distilling it into smaller ones, by getting rid of “unnecessary knowledge” or dead neurons.

Neurons in Large Language Models: Dead, N-gram, Positional
We analyze a family of large language models in such a lightweight manner that can be done on a single GPU. Specifically, we focus on the OPT family of models ranging from 125m to 66b parameters and rely only on whether an FFN neuron is activated or not. First, we find that the early part of the net…

Conditional Inputs = just extra information you can feel into the model to help it make its prediction. In this case they feed “pitch”, “duration”, and “energy” into the model to help it create the sound better.

Why Non-Autoregressive?

Autoregressive can be slow, this is why we pick a non-Autoregressive for the waveform generation.

Training FastSpeech (the original) relies on an Autoregressive teacher model and knowledge distillation.

FastSpeech: Fast, Robust and Controllable Text to Speech
Neural network based end-to-end text to speech (TTS) has significantly improved the quality of synthesized speech. Prominent methods (e.g., Tacotron 2) usually first generate mel-spectrogram from text, and then synthesize speech from the mel-spectrogram using vocoder such as WaveNet. Compared with t…

FastSpeech had some disadvantages

  1. Teacher student distillation pipeline was complicated and time consuming
  2. During the transfer from student to teach, the teacher model was not accurate enough, so the process suffered from too much information loss.

Fast Speech 2 aims to solve these problems.

  1. Get rid of the student - teacher, and just learn from the ground truth targets
  2. Introduce more variation information of speech (pitch, energy, duration) as conditional inputs. They take the pitch, energy and duration direction from the waveforms.
  3. Directly generate speech waveform from text in parallel, which has the benefits of end-to-end inference

Fast Speech 2 is 3x faster to train than FastSpeech. Is faster for inference. And “can even surpass Autoregressive models”


Text to speech (TTS) has made a lot of progress in the recent years with the use of neural networks.

Previous neural TTS models first generate Mel-spectrograms autoregressively from text, then synthesize speech from the generated spectrograms using a separately trained vocoder.

Fast Speech 2 Text → Intermediate Representation → Predict Pitch, Duration, Energy → Spectrogram → Waveform

Fast Speech 2s Text → Intermediate Representation → Predict Pitch, Duration, Energy → Waveform

These autoregressive models suffer from slow inference speeds and robustness (word skipping, repeating).

Non-Autoregressive TTS models have been designed to address these issues.

Models like WaveNet have been shown to be able to reverse the spectrogram back to waveform.

They directly generate a speech waveform from text in inference, which speeds the whole process up, and simplifies the whole pipeline.

Speech synthesis and voice cloning - Marvik
Speech synthesis literally means producing artificial human speech. It presents a lot of practical applications, such as music generation, text-to-speech conversion or navigation systems guidance. Traditional approaches for speech synthesis involved searching for speech units that match an input tex…

Why Spectrogram vs Waveform?

[D] Waveforms vs. spectrograms as inputs to a Convolutional Neural Network: why do ML researchers tend to prefer the latter?
by u/Probono_Bonobo in MachineLearning

Model Overview

Text is a very low information input to capture all of this variation.

Everyone has a different voice, accent, pitch, volume, etc. All this information gets lost when generating audio just from text alone.

To address this issue, they add variance adaptors in the middle of the architecture.

There is an Encoder that converts a “phoneme embedding sequence” into a latent space.

Then a variance adaptor that adds in information about pitch and energy and duration into the latent space. They do this by having the model try to predict these variables from the latent space.

And finally there is a Mel-spectrogram decoder that converts the latent space into a Mel-spectrogram sequence in parallel.

In Fast Speech 2 they use a feed-forward transformer block, which is a stack of self-attention layers and a 1D-convolution.

For Fast Speech 2s they simplify even further to directly generate the waveform from text, getting rid of the Mel-spectrogram generation (acoustic model) and waveform decoder (vocoder).

Variance Adapter

This is how they capture all the nuance.

The duration predictor takes in the latent space sequence and predicts the duration of each phoneme. They use a tool called Montreal forced alignment (MFA) to extract the phoneme duration. Duration - specifically phoneme duration, represents how long the speech voice sounds

The pitch predictor tries to predict a pitch spectrogram for each frame. This is a key feature to convey emotions and greatly affects the speech prosody (pattern, rhythm and sound, think poetry reading), mathematically it can be measured as different values of frequency

The energy predictor takes a short-time Fourier transform of the frames as the energy and tries to predict the original values of the energy. From the hidden spaces. Energy is frame level magnitude of the Mel-spectrograms which affects the volume and prosody.

Fast Speech 2s

Get rid of the cascaded Mel-spectrogram generation and vocoder.


Waveforms contain way more data than Mel-spectrograms. Which means it is difficult to get one component to translate all the information from text to waveform.

Waveforms are also a very long sequence, and we are limited by GPU memory. As a result you can only train on short audio clips that correspond to partial text sequences, so it makes it hard to capture the relationship between generated audio clips from the short text snippets.

They use adversarial training in the waveform decoder to force it to recover phase information, which has traditionally been hard to do.


They use the structure of WaveNet (seminal 2016 paper) for the architecture.

The waveform decoder tries to predict multiple resolutions of the Mel-spectrograms and the Short Time Fourier Transform (STFT) values and they use a Parallel WaveGAN architecture as well. They throw away the Mel-spectrogram decoder and only use the waveform decoder to synthesize audio.


I wish they had a better picture of this full decoder, but we would have to reference the other papers.

Experiments and Results

They use the LJSpeech Dataset which has 13,100 English audio clips (about 24 hours of audio) and the text transcripts.

They use 12,228 samples for training, 349 samples for validation and 523 samples for testing.

They randomly chose 100 samples for subjective evaluation.

All the audio files are of a sample rate of 22050 and they chop them up into 1024 sized frames and skip 256 values as they move to the next frame.

Training and Inference Time

RTF = real-time factor, time in seconds required to synthesize one second of audio. So FastSpeech 2s takes 0.018 seconds to synthesize 1 second of audio on the hardware specified.

36 CPUs with 256 GB Memory
1 Nvidia V100 GPU (32 GB I assume)

To analyze they have some quantitative metrics of how aligned are the phoneme boundaries and how close the distributions are to the ground truth. For example Mean Absolute Error between the energy of the synthesized audio vs the actual audio.


End to end Text to Speech without external libraries is a very challenging problem, but is becoming achievable with these neural network architectures.

In my opinion they actually don’t train it on that much data compared to some of these foundation models for text. LJSpeech is only 24 hours of English audio. I’ve probably listened to more audio than that in the past week, and this model only took 17 hours to train.

You could imagine a pipeline that generates a large dataset from podcasts and videos using Whisper to translate the audio to text, and some sort of human verification or loop to validate the data.

I would love to see the stats on getting these models to run on CPU or mobile devices.

I would also love to see if we can use the internal representations of these generative models to improve speech recognition, text understanding, and other fields of AI.

Discussion Links




If you enjoyed this dive, please join us next week!

Arxiv Dives with Oxen.ai · Luma
Hey Nerd, join the Herd!... for a little book/paper review. Make sure to also join our Discord here (https://discord.gg/s3tBEn7Ptg) to share recommendations for future reads and more…

All the past dives can be found on the blog.

Arxiv Dives - Oxen.ai
Each week we dive deep into a topic in machine learning, data management, or general artificial intelligence research. These are notes from a live reading group we do every Friday. Captured for future reference.

The live sessions are posted on YouTube if you want to watch at your own leisure.

Each week we dive deep into a topic in machine learning or general artificial intelligence research. The sessions are live with a group of smart Oxen every Friday. Join the discussion: https://lu.ma/oxenbookclub

Best & Moo,

~ The herd at Oxen.ai

Who is Oxen.ai?

Oxen.ai is an open source project aimed at solving some of the challenges with iterating on and curating machine learning datasets. At its core Oxen is a lightning fast data version control tool optimized for large unstructured datasets. We are currently working on collaboration workflows to enable the high quality, curated public and private data repositories to advance the field of AI, while keeping all the data accessible and auditable.

If you would like to learn more, star us on GitHub or head to Oxen.ai and create an account.