Hugging Face's logo

Diffusers documentation

Stable Audio 3

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v0.40.0).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Stable Audio 3

Stable Audio 3 (SA3) is a text-to-audio model from Stability AI that generates high-quality stereo audio at 44.1 kHz. It uses a rectified-flow DiT conditioned on two signals:

  • Text — encoded by a frozen T5Gemma encoder and injected via cross-attention.
  • Duration — a float (seconds) embedded by StableAudio3DurationEmbedder and used as a global conditioning vector for adaptive layer normalisation.

Audio is decoded by the SAME (Semantically-Aligned Music Encoder) autoencoder, AutoencoderSAME.

Both checkpoints use FlowMatchEulerDiscreteScheduler with a log-SNR-uniform sigma schedule, differing only in stochastic_sampling and the default step count:

Checkpointdiffusion_objectivestochastic_samplingnum_inference_steps
stable-audio-3-medium-baserectified_flowFalse (deterministic Euler)100 (not distilled)
stable-audio-3-medium (distilled)rf_denoiserTrue (ping-pong re-noise)8 (distilled for 8 steps)

The correct scheduler config is baked into each converted checkpoint, so num_inference_steps defaults to the right value when you leave it unset. Only pass it to override.

Original codebase: Stability-AI/stable-audio-3.

Converting original checkpoints

The Stability AI checkpoints are not published in diffusers format, so convert them locally. The script downloads the checkpoint’s model_config.json and selects the scheduler from its diffusion_objective:

python scripts/convert_stable_audio_3_to_diffusers.py \
  --checkpoint_path stabilityai/stable-audio-3-medium-base \
  --text_encoder_repo google/t5gemma-b-b-ul2 \
  --output_dir /tmp/sa3-diffusers-euler \
  --dtype float32

stable-audio-3-medium-base is a gated repo. Run hf auth login with an account that has access before converting, otherwise the download fails with a 401.

Usage example

Load the converted checkpoint from its local output directory (install soundfile with pip install soundfile):

import torch
import soundfile as sf
from diffusers import StableAudio3Pipeline

pipe = StableAudio3Pipeline.from_pretrained("/tmp/sa3-diffusers-euler", torch_dtype=torch.float32)
pipe = pipe.to("cuda")

generator = torch.Generator("cuda").manual_seed(0)
audio = pipe(
    "A gentle piano melody with soft strings in a concert hall",
    duration=10.0,  # seconds; latent length is computed automatically
    generator=generator,
).audios

sf.write("sa3_output.wav", audio[0].T.cpu().float().numpy(), samplerate=44100)

The pipeline is also registered with AutoPipelineForText2Audio, which resolves the checkpoint to StableAudio3Pipeline automatically:

from diffusers import AutoPipelineForText2Audio

pipe = AutoPipelineForText2Audio.from_pretrained("/tmp/sa3-diffusers-euler", torch_dtype=torch.float32)

The examples use a local path because stabilityai/stable-audio-3-medium and stable-audio-3-medium-base are not yet published in diffusers format (loading by repo id returns a 404). Once published, the repo id works in place of the local path.

Tips

  • Use torch.float32 on CPU or MPS (Apple Silicon) — torch.float16 on MPS produces noise.
  • The distilled model (stable-audio-3-medium) is adversarially distilled — guidance is baked into the weights. Leave guidance_scale=1.0 (the default) and don’t pass a negative_prompt for that checkpoint; both only do something useful for the non-distilled stable-audio-3-medium-base checkpoint.
  • silence_padding_duration (default 0.0) adds silent headroom at the end of the latent sequence. Leave it at 0.0 unless the model is trained to mask that padding — otherwise the extra frames drain output energy and the result gets quiet.
  • Set num_waveforms_per_prompt > 1 to generate multiple clips per prompt.

StableAudio3Pipeline

class diffusers.StableAudio3Pipeline

< >

( vae: AutoencoderSAMEtext_encoder: T5GemmaEncoderModeltokenizer: GemmaTokenizerduration_embedder: StableAudio3DurationEmbeddertransformer: StableAudio3DiTModelscheduler: FlowMatchEulerDiscreteScheduler )

Parameters

  • vae ([AutoencoderSAME]) — SAME autoencoder used to encode and decode audio latents.
  • text_encoder ([~transformers.T5GemmaEncoderModel]) — Frozen T5Gemma text encoder (google/t5gemma-b-b-ul2).
  • tokenizer ([~transformers.GemmaTokenizerFast]) — Tokenizer for the text encoder.
  • duration_embedder ([StableAudio3DurationEmbedder]) — Maps duration in seconds to a global conditioning vector for AdaLN in each DiT block.
  • transformer ([StableAudio3DiTModel]) — The rectified-flow velocity-prediction DiT.
  • scheduler ([FlowMatchEulerDiscreteScheduler]) — Scheduler for the iterative denoising loop. The production (distilled) SA3 Medium checkpoint uses stochastic_sampling=True for exactly 8 ping-pong steps; the non-distilled base checkpoint uses stochastic_sampling=False for ~100 deterministic Euler steps.

Pipeline for text-to-audio generation using Stable Audio 3.

SA3 uses a distilled rectified-flow DiT with ping-pong sampling. Classifier-free guidance (guidance_scale / negative_prompt) is unnecessary for the distilled checkpoint (leave guidance_scale=1.0, the default) but is meaningful for the non-distilled stable-audio-3-medium-base checkpoint.

This model inherits from [DiffusionPipeline]. Check the superclass documentation for the generic methods implemented for all pipelines.

__call__

< >

( prompt: typing.Union[str, typing.List[str], NoneType] = Noneduration: float = 10.0num_inference_steps: typing.Optional[int] = Nonelogsnr_min: float = -6.2logsnr_max: float = 2.0silence_padding_duration: float = 0.0guidance_scale: float = 1.0negative_prompt: typing.Union[str, typing.List[str], NoneType] = Nonenum_waveforms_per_prompt: int = 1generator: typing.Union[torch.Generator, typing.List[torch.Generator], NoneType] = Nonelatents: typing.Optional[torch.Tensor] = Noneprompt_embeds: typing.Optional[torch.Tensor] = Noneencoder_attention_mask: typing.Optional[torch.LongTensor] = Nonenegative_prompt_embeds: typing.Optional[torch.Tensor] = Nonenegative_encoder_attention_mask: typing.Optional[torch.LongTensor] = Nonereturn_dict: bool = Truecallback_on_step_end: typing.Optional[typing.Callable[[int, int, dict], dict]] = Nonecallback_on_step_end_tensor_inputs: typing.List[str] = ['latents']output_type: str = 'pt' ) → [~pipelines.AudioPipelineOutput] or tuple

Parameters

  • prompt (str or list[str], optional) — Text prompt(s). Pass prompt_embeds instead to skip tokenization and encoding.
  • duration (float, defaults to 10.0) — Requested output duration in seconds.
  • num_inference_steps (int, optional) — Number of denoising steps. When None (default), the step count is chosen from the scheduler’s stochastic_sampling config: 8 when True (the distilled ping-pong-style checkpoint) and 100 when False (the non-distilled base checkpoint). Pass an explicit value to override.
  • logsnr_min (float, defaults to -6.2) — Minimum log-SNR value for the noise schedule — maps to the high-noise start of the schedule.
  • logsnr_max (float, defaults to 2.0) — Maximum log-SNR value for the noise schedule — maps to the low-noise end of the schedule.
  • silence_padding_duration (float, defaults to 0.0) — Extra seconds of latent context generated beyond the target content, giving the model headroom at the boundary; the output is trimmed back to duration. Defaults to 0.0 (disabled). Increase only if the model is trained/distilled to mask this padding — otherwise the extra frames drain output energy.
  • guidance_scale (float, defaults to 1.0) — Classifier-free guidance scale. 1.0 disables guidance (the default, and the only sensible value for the distilled SA3 Medium checkpoint, whose CFG is baked into the weights). Values > 1.0 are meaningful for the non-distilled stable-audio-3-medium-base checkpoint; higher values follow the prompt more closely at the cost of diversity.
  • negative_prompt (str or list[str], optional) — Prompt(s) describing what to steer away from when guidance_scale > 1.0. Defaults to an empty string (unconditional) when guidance_scale > 1.0 and neither this nor negative_prompt_embeds is given. Ignored when guidance_scale <= 1.0.
  • num_waveforms_per_prompt (int, defaults to 1) — Number of waveforms to generate per prompt.
  • generator (torch.Generator or list[torch.Generator], optional) — For deterministic generation and reproducible re-noise in the ping-pong loop.
  • latents (torch.Tensor, optional) — Pre-generated starting latents. If None a fresh Gaussian tensor is sampled.
  • prompt_embeds (torch.Tensor, optional) — Pre-computed text embeddings (batch, seq_len, 768).
  • encoder_attention_mask (torch.LongTensor, optional) — Boolean mask for pre-computed embeddings.
  • negative_prompt_embeds (torch.Tensor, optional) — Pre-computed negative text embeddings, as an alternative to negative_prompt.
  • negative_encoder_attention_mask (torch.LongTensor, optional) — Boolean mask for pre-computed negative embeddings.
  • return_dict (bool, defaults to True) — Return an AudioPipelineOutput or a plain tuple.
  • callback_on_step_end (Callable, optional) — Called at the end of each denoising step with (self, step_idx, timestep, callback_kwargs), where callback_kwargs contains the tensors listed in callback_on_step_end_tensor_inputs. Must return a dict with the (optionally modified) tensors to use for the rest of the loop.
  • callback_on_step_end_tensor_inputs (list[str], defaults to [“latents”]) — The tensors passed to callback_on_step_end. Must be a subset of self._callback_tensor_inputs.
  • output_type (str, defaults to "pt") — "pt" for a PyTorch tensor, "np" for a NumPy array, or "latent" to skip decoding and return the raw latents.

Returns

[~pipelines.AudioPipelineOutput] or tuple

.audios is a tensor / array of shape (batch * num_waveforms_per_prompt, audio_channels, samples).

Generate audio from a text prompt.

Examples:

>>> import torch
>>> import soundfile as sf
>>> from diffusers import StableAudio3Pipeline

>>> pipe = StableAudio3Pipeline.from_pretrained("stabilityai/stable-audio-3-medium", torch_dtype=torch.float16)
>>> pipe = pipe.to("cuda")

>>> generator = torch.Generator("cuda").manual_seed(0)
>>> audio = pipe(
...     "A gentle piano melody with soft strings in a concert hall",
...     duration=10.0,
...     generator=generator,
... ).audios

>>> sf.write("output.wav", audio[0].T.cpu().float().numpy(), samplerate=pipe.vae.config.sampling_rate)

encode_duration

< >

( duration: floatdevice: devicenum_waveforms_per_prompt: intbatch_size: int )

Parameters

  • duration — Duration in seconds, applied to every sample in the batch.
  • device — Target device.
  • num_waveforms_per_prompt — Tile factor.
  • batch_size — Number of prompts.

Embed the duration value into the global conditioning vector.

encode_prompt

< >

( prompt: typing.Union[str, typing.List[str], NoneType]device: devicenum_waveforms_per_prompt: intprompt_embeds: typing.Optional[torch.Tensor] = Noneencoder_attention_mask: typing.Optional[torch.LongTensor] = None )

Parameters

  • prompt — Text prompt or list of prompts. Ignored when prompt_embeds is provided.
  • device — Target device.
  • num_waveforms_per_prompt — How many output waveforms to generate per prompt; conditioning tensors are tiled accordingly.
  • prompt_embeds — Pre-computed text embeddings (batch, seq_len, hidden_size).
  • encoder_attention_mask — Boolean mask (batch, seq_len) for pre-computed embeddings; 1 = real token, 0 = pad.

Encode text prompt(s) into cross-attention conditioning tensors.

prepare_cross_attention

< >

( prompt_embeds: Tensorencoder_attention_mask: Tensorglobal_hidden_states: Tensor )

Build the cross-attention context by appending the duration embedding as an extra token.

SA3 routes the seconds_total conditioner to both the global (AdaLN) input and the cross-attention context (cross_attention_cond_ids = ["prompt", "seconds_total"]). The duration embedding is concatenated after the text tokens, and the attention mask is extended with one valid entry.

StableAudio3InpaintPipeline

class diffusers.StableAudio3InpaintPipeline

< >

( vae: AutoencoderSAMEtext_encoder: T5GemmaEncoderModeltokenizer: GemmaTokenizerduration_embedder: StableAudio3DurationEmbeddertransformer: StableAudio3DiTModelscheduler: FlowMatchEulerDiscreteScheduler )

Parameters

  • vae ([AutoencoderSAME]) — SAME autoencoder used to encode and decode audio latents.
  • text_encoder ([~transformers.T5GemmaEncoderModel]) — Frozen T5Gemma text encoder (google/t5gemma-b-b-ul2).
  • tokenizer ([~transformers.GemmaTokenizerFast]) — Tokenizer for the text encoder.
  • duration_embedder ([StableAudio3DurationEmbedder]) — Maps duration in seconds to a global conditioning vector for AdaLN in each DiT block.
  • transformer ([StableAudio3DiTModel]) — The rectified-flow velocity-prediction DiT.
  • scheduler ([FlowMatchEulerDiscreteScheduler]) — Scheduler for the iterative denoising loop. The production (distilled) SA3 Medium checkpoint uses stochastic_sampling=True for exactly 8 ping-pong steps; the non-distilled base checkpoint uses stochastic_sampling=False for ~100 deterministic Euler steps.

Audio inpainting pipeline for Stable Audio 3.

Shares its text-to-audio logic with [StableAudio3Pipeline] (kept in sync via # Copied from). When audio and mask are provided, encodes the reference audio with the frozen SAME encoder and injects masked_latent ∥ mask as local-additive conditioning into each DiT block via the transformer’s local_add_cond pathway (to_local_embed).

Call signature extension (see __call__): audio (torch.Tensor of shape (batch, channels, samples)): Reference audio waveform at vae.config.sampling_rate Hz. mask (torch.Tensor of shape (batch, 1, latent_length)): Per-frame binary mask in latent space. 1 = preserve original audio; 0 = region to be inpainted.

__call__

< >

( prompt: typing.Union[str, typing.List[str], NoneType] = Noneduration: float = 10.0audio: typing.Optional[torch.Tensor] = Nonemask: typing.Optional[torch.Tensor] = Nonemask_start_seconds: typing.Union[float, typing.List[float], NoneType] = Nonemask_end_seconds: typing.Union[float, typing.List[float], NoneType] = Nonenum_inference_steps: typing.Optional[int] = Nonelogsnr_min: float = -6.2logsnr_max: float = 2.0silence_padding_duration: float = 0.0num_waveforms_per_prompt: int = 1generator: typing.Union[torch.Generator, typing.List[torch.Generator], NoneType] = Nonelatents: typing.Optional[torch.Tensor] = Noneprompt_embeds: typing.Optional[torch.Tensor] = Noneencoder_attention_mask: typing.Optional[torch.LongTensor] = Nonereturn_dict: bool = Truecallback_on_step_end: typing.Optional[typing.Callable[[int, int, dict], dict]] = Nonecallback_on_step_end_tensor_inputs: typing.List[str] = ['latents']output_type: str = 'pt' )

Parameters

  • prompt (str or list[str], optional) — Text prompt(s).
  • duration (float, defaults to 10.0) — Output duration in seconds. Should match the reference audio.
  • audio (torch.Tensor, optional) — Reference waveform (batch, channels, samples) at vae.config.sampling_rate Hz. Required for inpainting.
  • mask (torch.Tensor, optional) — Per-frame latent-space mask (batch, 1, L) with 0 = inpaint region, 1 = preserve. Either mask or mask_start_seconds / mask_end_seconds must be provided.
  • mask_start_seconds (float or list[float], optional) — Start time(s) of the inpaint region in seconds.
  • mask_end_seconds (float or list[float], optional) — End time(s) of the inpaint region (must pair with mask_start_seconds).
  • num_inference_steps (int, optional) — Number of denoising steps. When None (default), the step count is chosen from the scheduler’s stochastic_sampling config, matching [StableAudio3Pipeline].
  • logsnr_min (float, defaults to -6.2) — Minimum log-SNR value for the noise schedule — maps to the high-noise start of the schedule.
  • logsnr_max (float, defaults to 2.0) — Maximum log-SNR value for the noise schedule — maps to the low-noise end of the schedule.
  • silence_padding_duration (float, defaults to 0.0) — Extra latent headroom after the target content.
  • num_waveforms_per_prompt (int, defaults to 1) — Waveforms per prompt.
  • generator — RNG for reproducibility.
  • latents — Pre-generated starting noise (None → sample fresh).
  • prompt_embeds — Pre-computed text embeddings.
  • encoder_attention_mask — Mask for pre-computed embeddings.
  • return_dict (bool, defaults to True) — Return AudioPipelineOutput or tuple.
  • callback_on_step_end (Callable, optional) — Called at the end of each denoising step with (self, step_idx, timestep, callback_kwargs). Must return a dict with the (optionally modified) tensors to use for the rest of the loop.
  • callback_on_step_end_tensor_inputs (list[str], defaults to [“latents”]) — The tensors passed to callback_on_step_end. Must be a subset of self._callback_tensor_inputs.
  • output_type (str, defaults to "pt") — "pt" / "np" / "latent".

Generate inpainted audio conditioned on a text prompt and reference.

Examples:

>>> import torch
>>> import soundfile as sf
>>> import torchaudio
>>> from diffusers import StableAudio3InpaintPipeline

>>> pipe = StableAudio3InpaintPipeline.from_pretrained(
...     "stabilityai/stable-audio-3-medium", torch_dtype=torch.float16
... )
>>> pipe = pipe.to("cuda")

>>> audio, sr = torchaudio.load("reference.wav")
>>> audio = torchaudio.functional.resample(audio, sr, pipe.vae.config.sampling_rate).unsqueeze(0).to("cuda")

>>> generator = torch.Generator("cuda").manual_seed(0)
>>> audio = pipe(
...     "A gentle piano melody with soft strings in a concert hall",
...     duration=10.0,
...     audio=audio,
...     mask_start_seconds=4.0,
...     mask_end_seconds=6.0,
...     generator=generator,
... ).audios

>>> sf.write("output.wav", audio[0].T.cpu().float().numpy(), samplerate=pipe.vae.config.sampling_rate)

encode_duration

< >

( duration: floatdevice: devicenum_waveforms_per_prompt: intbatch_size: int )

Parameters

  • duration — Duration in seconds, applied to every sample in the batch.
  • device — Target device.
  • num_waveforms_per_prompt — Tile factor.
  • batch_size — Number of prompts.

Embed the duration value into the global conditioning vector.

encode_prompt

< >

( prompt: typing.Union[str, typing.List[str], NoneType]device: devicenum_waveforms_per_prompt: intprompt_embeds: typing.Optional[torch.Tensor] = Noneencoder_attention_mask: typing.Optional[torch.LongTensor] = None )

Parameters

  • prompt — Text prompt or list of prompts. Ignored when prompt_embeds is provided.
  • device — Target device.
  • num_waveforms_per_prompt — How many output waveforms to generate per prompt; conditioning tensors are tiled accordingly.
  • prompt_embeds — Pre-computed text embeddings (batch, seq_len, hidden_size).
  • encoder_attention_mask — Boolean mask (batch, seq_len) for pre-computed embeddings; 1 = real token, 0 = pad.

Encode text prompt(s) into cross-attention conditioning tensors.

prepare_cross_attention

< >

( prompt_embeds: Tensorencoder_attention_mask: Tensorglobal_hidden_states: Tensor )

Build the cross-attention context by appending the duration embedding as an extra token.

SA3 routes the seconds_total conditioner to both the global (AdaLN) input and the cross-attention context (cross_attention_cond_ids = ["prompt", "seconds_total"]). The duration embedding is concatenated after the text tokens, and the attention mask is extended with one valid entry.

StableAudio3AudioToAudioPipeline

Generates a variation of a reference audio clip: the whole reference is noised to init_noise_level and denoised from there, unlike StableAudio3InpaintPipeline’s per-frame local-additive conditioning which preserves specific frames exactly.

class diffusers.StableAudio3AudioToAudioPipeline

< >

( vae: AutoencoderSAMEtext_encoder: T5GemmaEncoderModeltokenizer: GemmaTokenizerduration_embedder: StableAudio3DurationEmbeddertransformer: StableAudio3DiTModelscheduler: FlowMatchEulerDiscreteScheduler )

Parameters

  • vae ([AutoencoderSAME]) — SAME autoencoder used to encode and decode audio latents.
  • text_encoder ([~transformers.T5GemmaEncoderModel]) — Frozen T5Gemma text encoder (google/t5gemma-b-b-ul2).
  • tokenizer ([~transformers.GemmaTokenizerFast]) — Tokenizer for the text encoder.
  • duration_embedder ([StableAudio3DurationEmbedder]) — Maps duration in seconds to a global conditioning vector for AdaLN in each DiT block.
  • transformer ([StableAudio3DiTModel]) — The rectified-flow velocity-prediction DiT.
  • scheduler ([FlowMatchEulerDiscreteScheduler]) — Scheduler for the iterative denoising loop. The production (distilled) SA3 Medium checkpoint uses stochastic_sampling=True for exactly 8 ping-pong steps; the non-distilled base checkpoint uses stochastic_sampling=False for ~100 deterministic Euler steps.

Audio-to-audio variation pipeline for Stable Audio 3.

Shares its text-to-audio logic with [StableAudio3Pipeline] (kept in sync via # Copied from). Encodes the reference audio with the frozen SAME encoder, mixes it with fresh noise according to init_noise_level, and denoises from there — the whole signal is noised/denoised globally, unlike [StableAudio3InpaintPipeline]’s per-frame local-additive conditioning.

Call signature extension (see __call__): audio (torch.Tensor of shape (batch, channels, samples)): Reference audio waveform at vae.config.sampling_rate Hz. init_noise_level (float): How much noise to mix into the reference before denoising. 1.0 = full noise (equivalent to text-to-audio); lower values retain more of the reference.

__call__

< >

( prompt: typing.Union[str, typing.List[str], NoneType] = Noneduration: float = 10.0audio: typing.Optional[torch.Tensor] = Noneinit_noise_level: float = 1.0num_inference_steps: typing.Optional[int] = Nonelogsnr_min: float = -6.2logsnr_max: float = 2.0silence_padding_duration: float = 0.0num_waveforms_per_prompt: int = 1generator: typing.Union[torch.Generator, typing.List[torch.Generator], NoneType] = Nonelatents: typing.Optional[torch.Tensor] = Noneprompt_embeds: typing.Optional[torch.Tensor] = Noneencoder_attention_mask: typing.Optional[torch.LongTensor] = Nonereturn_dict: bool = Truecallback_on_step_end: typing.Optional[typing.Callable[[int, int, dict], dict]] = Nonecallback_on_step_end_tensor_inputs: typing.List[str] = ['latents']output_type: str = 'pt' )

Parameters

  • prompt (str or list[str], optional) — Text prompt(s).
  • duration (float, defaults to 10.0) — Output duration in seconds. Should match the reference audio.
  • audio (torch.Tensor, optional) — Reference waveform (batch, channels, samples) at vae.config.sampling_rate Hz. Required.
  • init_noise_level (float, defaults to 1.0) — Noise level (in (0, 1]) mixed into the reference before denoising: x_start = (1 - init_noise_level) * reference_latents + init_noise_level * noise. 1.0 discards the reference entirely (equivalent to [StableAudio3Pipeline]); lower values retain progressively more of the reference’s structure while still running the full step count.
  • num_inference_steps (int, optional) — Number of denoising steps. When None (default), the step count is chosen from the scheduler’s stochastic_sampling config, matching [StableAudio3Pipeline].
  • logsnr_min (float, defaults to -6.2) — Minimum log-SNR value for the noise schedule — maps to the high-noise start of the schedule.
  • logsnr_max (float, defaults to 2.0) — Maximum log-SNR value for the noise schedule — maps to the low-noise end of the schedule.
  • silence_padding_duration (float, defaults to 0.0) — Extra latent headroom after the target content.
  • num_waveforms_per_prompt (int, defaults to 1) — Waveforms per prompt.
  • generator — RNG for reproducibility.
  • latents — Pre-generated starting noise (None → sample fresh).
  • prompt_embeds — Pre-computed text embeddings.
  • encoder_attention_mask — Mask for pre-computed embeddings.
  • return_dict (bool, defaults to True) — Return AudioPipelineOutput or tuple.
  • callback_on_step_end (Callable, optional) — Called at the end of each denoising step with (self, step_idx, timestep, callback_kwargs). Must return a dict with the (optionally modified) tensors to use for the rest of the loop.
  • callback_on_step_end_tensor_inputs (list[str], defaults to [“latents”]) — The tensors passed to callback_on_step_end. Must be a subset of self._callback_tensor_inputs.
  • output_type (str, defaults to "pt") — "pt" / "np" / "latent".

Generate an audio variation conditioned on a text prompt and a reference waveform.

Examples:

>>> import torch
>>> import soundfile as sf
>>> import torchaudio
>>> from diffusers import StableAudio3AudioToAudioPipeline

>>> pipe = StableAudio3AudioToAudioPipeline.from_pretrained(
...     "stabilityai/stable-audio-3-medium", torch_dtype=torch.float16
... )
>>> pipe = pipe.to("cuda")

>>> audio, sr = torchaudio.load("reference.wav")
>>> audio = torchaudio.functional.resample(audio, sr, pipe.vae.config.sampling_rate).unsqueeze(0).to("cuda")

>>> generator = torch.Generator("cuda").manual_seed(0)
>>> audio = pipe(
...     "A gentle piano melody with soft strings in a concert hall",
...     duration=10.0,
...     audio=audio,
...     init_noise_level=0.6,
...     generator=generator,
... ).audios

>>> sf.write("output.wav", audio[0].T.cpu().float().numpy(), samplerate=pipe.vae.config.sampling_rate)

encode_duration

< >

( duration: floatdevice: devicenum_waveforms_per_prompt: intbatch_size: int )

Parameters

  • duration — Duration in seconds, applied to every sample in the batch.
  • device — Target device.
  • num_waveforms_per_prompt — Tile factor.
  • batch_size — Number of prompts.

Embed the duration value into the global conditioning vector.

encode_prompt

< >

( prompt: typing.Union[str, typing.List[str], NoneType]device: devicenum_waveforms_per_prompt: intprompt_embeds: typing.Optional[torch.Tensor] = Noneencoder_attention_mask: typing.Optional[torch.LongTensor] = None )

Parameters

  • prompt — Text prompt or list of prompts. Ignored when prompt_embeds is provided.
  • device — Target device.
  • num_waveforms_per_prompt — How many output waveforms to generate per prompt; conditioning tensors are tiled accordingly.
  • prompt_embeds — Pre-computed text embeddings (batch, seq_len, hidden_size).
  • encoder_attention_mask — Boolean mask (batch, seq_len) for pre-computed embeddings; 1 = real token, 0 = pad.

Encode text prompt(s) into cross-attention conditioning tensors.

prepare_cross_attention

< >

( prompt_embeds: Tensorencoder_attention_mask: Tensorglobal_hidden_states: Tensor )

Build the cross-attention context by appending the duration embedding as an extra token.

SA3 routes the seconds_total conditioner to both the global (AdaLN) input and the cross-attention context (cross_attention_cond_ids = ["prompt", "seconds_total"]). The duration embedding is concatenated after the text tokens, and the attention mask is extended with one valid entry.

StableAudio3DurationEmbedder

class diffusers.StableAudio3DurationEmbedder

< >

( output_dim: int = 768fourier_dim: int = 256min_val: float = 0.0max_val: float = 384.0min_freq: float = 0.5max_freq: float = 10000.0 )

Parameters

  • output_dim (int, defaults to 768) — Dimension of the output embedding. Must match the DiT’s global_cond_dim.
  • fourier_dim (int, defaults to 256) — Internal Fourier feature dimension (must be even).
  • min_val (float, defaults to 0.0) — Minimum duration value for normalization clamping.
  • max_val (float, defaults to 384.0) — Maximum duration value for normalization clamping. Values above this are clamped. 384 seconds is the production SA3 Medium upper bound for the seconds_total conditioner.
  • min_freq (float, defaults to 0.5) — Minimum frequency for the exponential Fourier basis.
  • max_freq (float, defaults to 10000.0) — Maximum frequency for the exponential Fourier basis.

Embeds a duration value (in seconds) into a global conditioning vector for the Stable Audio 3 DiT (used as the global_hidden_states AdaLN input).

Replicates NumberConditioner(fourier_features_type="expo") from the SA3 reference implementation.

forward

< >

( seconds: Tensor )

Parameters

  • seconds (torch.Tensor of shape (batch,)) — Duration values in seconds.
Update on GitHub

←Stable Audio Anima→