MusicGen represents an innovative music generation framework developed by Jade Copet and colleagues at Meta AI. This system leverages a unified language model architecture to produce high-quality musical compositions from textual descriptions or audio prompts. The underlying research is documented in the paper "Simple and Controllable Music Generation".

The core innovation lies in its single-stage transformer language model architecture that operates across multiple compressed discrete music representations (tokens). Unlike previous approaches requiring cascaded models through hierarchical or upsampling techniques, this design enables direct generation of both mono and stereo samples with enhenced output control based on textual descriptions or melodic features.

MusicGen specifically targets music generation, addressing the complexity inherent in musical compositions compared to environmental sounds. Maintaining coherent long-term structures proves crucial when creating novel musical pieces. This modeling approach naturally extends to stereo music synthesis capabilities.

The MusicGen framework introduces alternative approaches for controlling music generation, including a novel unsupervised melody-guided generation technique based on pitch contour maps. Given a musical sample, the system extracts the primary melody through pitch contour analysis. These contour maps capture harmonic and melodic characteristics while remaining robust to instrumental variations and timbre differences. The extracted signal guides the generation process to follow the specified melody while staying faithful to the provided textual description.

Architecture Overview

The MusicGen architecture follows a three-stage pipeline:

1. Textual descriptions pass through a fixed encoder model to produce latent state representations
2. The MusicGen decoder predicts discrete latent audio tokens
3. An audio compression model (such as EnCodec) decodes these tokens back into waveform audio

The implementation utilizes Google's t5-base model with pre-trained weights as the text encoder and EnCodec 32kHz for audio compression. The MusicGen decoder represents a language model architecture specifically trained from scratch for music generation tasks.

Recent advances with multi-band diffusion codec (MBD-EnCodec) can further enhance MusicGen's audio quality. While computationally more demanding, MBD-enhanced EnCodec produces samples with reduced audio distortion.

Installation and Setup

MusicGen offers three pre-trained weight variants: small, medium, and large. This guide uses the small variant which provides faster generation at the cost of lower audio quality:

%%capture captured_output
# Compatible with mindnlp 0.3.1 version
!pip install -i https://pypi.mirrors.ustc.edu.cn/simple mindnlp jieba soundfile librosa

from mindnlp.transformers import MusicgenForConditionalGeneration

pretrained_model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")

Music Generation Techniques

MusicGen supports two generation strategies: greedy decoding and sampling-based approaches. In practice, sampling typically yields superior results compared to greedy methods. By default, we enable sampling mode and explicitly specify it using do_sample=True when calling MusicgenForConditionalGeneration.generate.

Unconditional Generation

Random inputs can be obtained through the MusicgenForConditionalGeneration.get_unconditional_inputs method, followed by autoregressive generation using .generate with do_sample=True enabled:

%%time
random_inputs = pretrained_model.get_unconditional_inputs(num_samples=1)

generated_audio = pretrained_model.generate(**random_inputs, do_sample=True, max_new_tokens=256)

The audio output format appears as: a Torch tensor with dimensions (batch_size, num_channels, sequence_length).

Use the scipy library to save the generated audio as musicgen_output.wav:

import scipy
from IPython.display import Audio

sample_rate = pretrained_model.config.audio_encoder.sampling_rate
scipy.io.wavfile.write("musicgen_output.wav", rate=sample_rate, data=generated_audio[0, 0].asnumpy())

# Play the generated audio sample using Audio in notebook
Audio(generated_audio[0].asnumpy(), rate=sample_rate)

The max_new_tokens parameter specifies the number of tokens to generate. Using the frame rate of the EnCodec model, you can calculate the duration of the generated audio sample in seconds:

duration_seconds = 256 / pretrained_model.config.audio_encoder.frame_rate
duration_seconds

Text-Based Generation

First preprocess inputs based on textual prompts using AutoProcessor. Then pass the processed inputs to the .generate method to create text-conditioned audio samples. Again, enable sampling mode by setting do_sample=True.

The guidance_scale parameter controls classifier-free guidance (CFG), determining the balance between conditional logits (predicted from text prompts) and unconditional logits (predicted from empty text). Higher guidance_scale values result in outputs more closely aligned with input text. Enable CFG by setting guidance_scale > 1. For optimal results, use guidance_scale=3 (the default) for text prompt audio generation.

%%time
from mindnlp.transformers import AutoProcessor
from IPython.display import Audio

processor_instance = AutoProcessor.from_pretrained("facebook/musicgen-small")

input_data = processor_instance(
    text=["80s pop track with bassy drums and synth", "90s rock song with loud guitars and heavy drums"],
    padding=True,
    return_tensors="ms",
)

generated_audio = pretrained_model.generate(**input_data, do_sample=True, guidance_scale=3, max_new_tokens=256)
scipy.io.wavfile.write("musicgen_output_text.wav", rate=sample_rate, data=generated_audio[0, 0].asnumpy())

# Play the generated audio sample using Audio in notebook
Audio(generated_audio[0].asnumpy(), rate=sample_rate)

Audio Prompt Generation

AutoProcessor can also preprocess audio prompts for audio-based generation. In the following example, we first load an audio file, preprocess it, then feed it to the model for audio generation. Finally, save the generated audio as musicgen_output_audio.wav:

%%time
from datasets import load_dataset
from IPython.display import Audio

processor_instance = AutoProcessor.from_pretrained("facebook/musicgen-small")
dataset_source = load_dataset("sanchit-gandhi/gtzan", split="train", streaming=True)
sample_data = next(iter(dataset_source))["audio"]

# Use the first half of the audio sample
sample_data["array"] = sample_data["array"][: len(sample_data["array"]) // 2]

input_data = processor_instance(
    audio=sample_data["array"],
    sampling_rate=sample_data["sampling_rate"],
    text=["80s blues track with groovy saxophone"],
    padding=True,
    return_tensors="ms",
)

generated_audio = pretrained_model.generate(**input_data, do_sample=True, guidance_scale=3, max_new_tokens=256)
scipy.io.wavfile.write("musicgen_output_audio.wav", rate=sample_rate, data=generated_audio[0, 0].asnumpy())

# Play the generated audio sample using Audio in notebook
Audio(generated_audio[0].asnumpy(), rate=sample_rate)

To demonstrate batch audio prompt generation, we slice the sample audio at two different ratios to provide audio samples of varying lengths. Since input audio prompts have different lengths, they will be padded to match the longest audio sample in the batch before passing to the model.

Post-process the generated audio_values to remove padding using the processor class:

sample_data = next(iter(dataset_source))["audio"]

# Take the first quarter of the audio sample
sample_part_1 = sample_data["array"][: len(sample_data["array"]) // 4]

# Take the first half of the audio sample
sample_part_2 = sample_data["array"][: len(sample_data["array"]) // 2]

input_data = processor_instance(
    audio=[sample_part_1, sample_part_2],
    sampling_rate=sample_data["sampling_rate"],
    text=["80s blues track with groovy saxophone", "90s rock song with loud guitars and heavy drums"],
    padding=True,
    return_tensors="ms",
)

generated_audio = pretrained_model.generate(**input_data, do_sample=True, guidance_scale=3, max_new_tokens=256)

# Post-process to remove padding from batch audio
processed_audio = processor_instance.batch_decode(generated_audio, padding_mask=input_data.padding_mask)

Audio(processed_audio[0], rate=sample_rate)

Configuration Management

Default parameters controlling the generation process (such as sampling, guidance scale, and token count) can be found in the model's generation configuration and updated as needed. First, inspect the default generation configuration:

pretrained_model.generation_config

We observe that the model defaults to sampling mode (do_sample=True), guidance scale of 3, and maximum generation length of 1500 (equivalent to 30 seconds of audio). Update any of these properties to modify default generation parameters:

# increase the guidance scale to 4.0
pretrained_model.generation_config.guidance_scale = 4.0

# set the max new tokens to 256
pretrained_model.generation_config.max_new_tokens = 256

# set the softmax sampling temperature to 1.5
pretrained_model.generation_config.temperature = 1.5

Now re-running generation will utilize the newly defined values in the generation configuraton:

generated_audio = pretrained_model.generate(**input_data)

Note that any parameters passed to the generate method will override those in the ganeration configuration. Therefore, setting do_sample=False in the generate call will override the model.generation_config.do_sample setting.