[Feat] Adds LongCat-AudioDiT pipeline

[RuixiangMa]

What does this PR do?

Adds LongCat-AudioDiT model support to diffusers.

Although LongCat-AudioDiT can be used for TTS-like generation, it is fundamentally a diffusion-based audio generation model (text conditioning + iterative latent denoising + VAE decoding) rather than a conventional autoregressive TTS model, so i think it fits naturally into diffusers.

Test

import soundfile as sf
import torch
from diffusers import LongCatAudioDiTPipeline

pipeline = LongCatAudioDiTPipeline.from_pretrained(
    "meituan-longcat/LongCat-AudioDiT-1B",
    torch_dtype=torch.float16,
)
pipeline = pipeline.to("cuda")

audio = pipeline(
    prompt="A calm ocean wave ambience with soft wind in the background.",
    audio_end_in_s=5.0,
    num_inference_steps=16,
    guidance_scale=4.0,
    output_type="pt",
).audios

output = audio[0, 0].float().cpu().numpy()
sf.write("longcat.wav", output, pipeline.sample_rate)

Result

longcat.wav

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[dg845]

Similarly, I think we should inline _pixel_shuffle_1d in UpsampleShortcut following #13390 (comment).

[dg845]

Suggested change

	self.time_embed = AudioDiTTimestepEmbedding(dim)
	self.input_embed = AudioDiTEmbedder(latent_dim, dim)
	self.text_embed = AudioDiTEmbedder(dit_text_dim, dim)
	self.rotary_embed = AudioDiTRotaryEmbedding(dim_head, 2048, base=100000.0)
	self.blocks = nn.ModuleList(
	self.time_embed = AudioDiTTimestepEmbedding(dim)
	self.input_embed = AudioDiTEmbedder(latent_dim, dim)
	self.text_embed = AudioDiTEmbedder(dit_text_dim, dim)
	self.rotary_embed = AudioDiTRotaryEmbedding(dim_head, 2048, base=100000.0)
	self.blocks = nn.ModuleList(

See #13390 (comment).

[dg845]

Suggested change

	batch_size = hidden_states.shape[0]
	if timestep.ndim == 0:
	timestep = timestep.repeat(batch_size)
	timestep_embed = self.time_embed(timestep)
	text_mask = encoder_attention_mask.bool()
	encoder_hidden_states = self.text_embed(encoder_hidden_states, text_mask)
	batch_size = hidden_states.shape[0]
	if timestep.ndim == 0:
	timestep = timestep.repeat(batch_size)
	timestep_embed = self.time_embed(timestep)
	text_mask = encoder_attention_mask.bool()
	encoder_hidden_states = self.text_embed(encoder_hidden_states, text_mask)

Can you also refactor forward here so that it is better organized, following #13390 (comment)? See for example the QwenImageTransformer2DModel.forward method:

diffusers/src/diffusers/models/transformers/transformer_qwenimage.py

Line 836 in d7bc233

def forward(

[RuixiangMa]

Reorganized parts of forward incrementally; kept the current structure otherwise to avoid unnecessary behavioral churn.

[dg845]

Thanks for your continued work on this! Left some suggestions that should help LongCatAudioDiTPipeline support model offloading, layerwise casting, etc.

[dg845]

@bot /style

[github-actions]

Style bot fixed some files and pushed the changes.

[yiyixuxu]

can you add a conversion script?
our pipeline should not define from_pretrained method

[RuixiangMa]

can you add a conversion script? our pipeline should not define from_pretrained method

Added it and tested.

[yiyixuxu]

@claude can you help with a review here?

[github-actions]

Claude Code is working…

I'll analyze this and get back to you.

View job run

[dg845]

@bot /style

[github-actions]

Style bot fixed some files and pushed the changes.

[dg845]

Thanks for working on this PR! Does a HF Hub repo with the diffusers-format checkpoint currently exist? If not, would you be willing to create one?

[RuixiangMa]

Thanks for working on this PR! Does a HF Hub repo with the diffusers-format checkpoint currently exist? If not, would you be willing to create one?

There isn't a diffusers-format checkpoint yet. I'll try to create one.

[dg845]

Merging as the CI failures are unrelated.