Related Work

Language models of audio

AudioLM (Borsos et al., 2023) showed that audio can be generated by language-modeling discrete tokens, using a hierarchy of semantic and acoustic tokens. MusicLM (Agostinelli et al., 2023) conditioned this approach on text through a joint music-text embedding. MusicGen (Copet et al., 2023) simplified the hierarchy to a single Transformer over the codebooks of a neural codec, interleaved by a delay pattern so that all codebooks of a frame are predicted in one pass. StaleyNET adopts MusicGen’s delay pattern, top-k sampling and classifier-free guidance, but replaces the pretrained codec with a k-means quantizer over log-mel features and the pretrained text encoder with raw UTF-8 bytes, an approach studied for text by ByT5 (Xue et al., 2022).

Singing and lyrics

Jukebox (Dhariwal et al., 2020) generated singing conditioned on artist, genre and lyrics, using lyrics-to-audio attention trained on 1.2 million songs. Singing voice synthesis systems such as DiffSinger (Liu et al., 2022) instead condition on a musical score with phoneme durations, which gives precise control but requires annotated scores. StaleyNET follows the Jukebox direction at a much smaller scale: it conditions on lyrics as text and learns the alignment from data. The training signal for the alignment comes from JamendoLyrics (Stoller, Durand and Ewert, 2019; Durand, Stoller and Ewert, 2023), a lyrics-alignment benchmark whose line-level time stamps allow each training excerpt to be paired with the words sung in it.

Audio representations

Neural codecs (SoundStream, Zeghidour et al., 2021; EnCodec, Défossez et al., 2022; DAC, Kumar et al., 2023) learn an encoder, a residual vector quantizer and a decoder jointly, and reach high fidelity at a few kilobits per second. They require large training sets and substantial compute. StaleyNET retains the residual quantization principle but applies it, by k-means, directly to normalized log-mel frames, so that the tokenizer can be trained in minutes on an edge device. Waveforms are recovered with the Griffin-Lim algorithm (Griffin and Lim, 1984) in its accelerated form (Perraudin, Balazs and Søndergaard, 2013), rather than with a neural vocoder such as HiFi-GAN (Kong, Kim and Bae, 2020) or Vocos (Siuzdak, 2023). The consequences of both choices are quantified in Experiments.

Efficient Transformer computation

Key/value caching for autoregressive inference is standard practice (Pope et al., 2023). Register-blocked matrix multiplication on GPUs was analyzed by Volkov and Demmel (2008); FlashAttention (Dao et al., 2022) fuses attention to avoid materializing the score matrix. StaleyNET implements the first two in Rust through cuda-oxide and leaves fused attention to future work.