File size: 7,529 Bytes
93f2318 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 | ---
library_name: diffusers
tags:
- modular-diffusers
- diffusers
- cosmos3-omni
- text-to-image
---
This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.
**Pipeline Type**: Cosmos3OmniBlocks
**Description**: Modular pipeline blocks for Cosmos3 generation modes.
This pipeline uses a 5-block architecture that can be customized and extended.
## Example Usage
[TODO]
## Pipeline Architecture
This modular pipeline is composed of the following blocks:
1. **text_encoder** (`Cosmos3AutoTextEncoderStep`)
- Auto text encoder block for Cosmos3.
2. **vae_encoder** (`Cosmos3AutoVaeEncoderStep`)
- Auto VAE conditioning block for Cosmos3.
3. **denoise** (`Cosmos3AutoCoreDenoiseStep`)
- Selects the Cosmos3 core denoising workflow.
4. **decode** (`Cosmos3AutoDecodeStep`)
- Selects the Cosmos3 decode workflow.
5. **after_decode** (`Cosmos3ActionOutputStep`)
- Post-processes action latents into action outputs.
## Model Components
1. video_processor (`VideoProcessor`)
2. text_tokenizer (`AutoTokenizer`)
3. vae (`AutoencoderKLWan`)
4. transformer (`Cosmos3OmniTransformer`)
5. scheduler (`UniPCMultistepScheduler`)
6. sound_tokenizer (`Cosmos3AVAEAudioTokenizer`)
## Configuration Parameters
default_use_system_prompt (default: True)
enable_safety_checker (default: True)
use_native_flow_schedule (default: False)
## Workflow Input Specification
<details>
<summary><strong>text2image</strong></summary>
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
- `num_frames` (`int`, *optional*): Number of frames to generate.
</details>
<details>
<summary><strong>text2video</strong></summary>
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
</details>
<details>
<summary><strong>image2video</strong></summary>
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
- `image` (`None`, *optional*): Reference image for image-to-video conditioning.
</details>
<details>
<summary><strong>video2video</strong></summary>
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
- `video` (`None`, *optional*): Reference video for video-to-video conditioning.
</details>
<details>
<summary><strong>text2video_with_sound</strong></summary>
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
</details>
<details>
<summary><strong>image2video_with_sound</strong></summary>
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
- `image` (`None`, *optional*): Reference image for image-to-video conditioning.
</details>
<details>
<summary><strong>video2video_with_sound</strong></summary>
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
- `video` (`None`, *optional*): Reference video for video-to-video conditioning.
</details>
<details>
<summary><strong>action_policy</strong></summary>
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
- `action` (`CosmosActionCondition`): Action-conditioning metadata and its reference visual input.
</details>
<details>
<summary><strong>action_forward_dynamics</strong></summary>
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
- `action` (`CosmosActionCondition`): Action-conditioning metadata and its reference visual input.
</details>
<details>
<summary><strong>action_inverse_dynamics</strong></summary>
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
- `action` (`CosmosActionCondition`): Action-conditioning metadata and its reference visual input.
</details>
## Input/Output Specification
**Inputs:**
- `control_videos` (`dict`, *optional*): Mapping of hint name (edge/blur/depth/seg/wsm) to the control video for that modality.
- `height` (`int`, *optional*): Height of the generated video in pixels.
- `width` (`int`, *optional*): Width of the generated video in pixels.
- `num_frames` (`int`, *optional*): Optional cap on the number of output frames (defaults to the control video length).
- `num_video_frames_per_chunk` (`int`, *optional*): Number of pixel frames generated per autoregressive chunk.
- `num_conditional_frames` (`int`, *optional*, defaults to `1`): Number of frames each chunk reuses from the previous chunk's tail.
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
- `negative_prompt` (`str`, *optional*): The negative text prompt used for classifier-free guidance.
- `use_system_prompt` (`bool`, *optional*, defaults to `True`): Whether to prepend the Cosmos3 transfer system prompt.
- `action` (`CosmosActionCondition`, *optional*): Action-conditioning metadata and its reference visual input.
- `fps` (`float`, *optional*, defaults to `24.0`): Frame rate of the generated video.
- `add_resolution_template` (`bool`, *optional*, defaults to `True`): Whether to add resolution metadata to the prompt.
- `add_duration_template` (`bool`, *optional*, defaults to `True`): Whether to add duration metadata to the prompt.
- `video` (`None`, *optional*): Reference video for video-to-video conditioning.
- `condition_frame_indexes_vision` (`tuple | list`, *optional*, defaults to `(0, 1)`): Latent-frame indexes to preserve from the conditioning video.
- `condition_video_keep` (`str`, *optional*, defaults to `first`): Which end of a longer conditioning video to use: `first` or `last`.
- `image` (`None`, *optional*): Reference image for image-to-video conditioning.
- `chunk_id` (`int`, *optional*, defaults to `0`): Index of the current chunk.
- `previous_output` (`None`, *optional*): Decoded pixels of the previous chunk, used to seed later chunks.
- `num_first_chunk_conditional_frames` (`int`, *optional*, defaults to `0`): Number of frames the first chunk reuses from the input video.
- `generator` (`Generator`, *optional*): Torch generator for deterministic generation.
- `num_inference_steps` (`int`): The number of denoising steps.
- `**denoiser_input_fields` (`None`, *optional*): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc.
- `guidance_scale` (`float`, *optional*, defaults to `6.0`): Scale for text classifier-free guidance.
- `control_guidance` (`float`, *optional*, defaults to `1.0`): Scale for the control (structural) guidance axis.
- `guidance_interval` (`tuple`, *optional*): Timestep interval [lo, hi] over which text guidance is active (None = always).
- `control_guidance_interval` (`tuple`, *optional*): Timestep interval [lo, hi] over which control guidance is active (None = always).
- `output_chunks` (`list`, *optional*): Decoded pixel chunks accumulated so far.
- `x0_tokens_vision` (`Tensor`, *optional*): Vision latents encoded from the conditioning image or video.
- `vision_condition_frames` (`list`, *optional*): Latent-frame indexes fixed by visual conditioning.
- `latents` (`Tensor`): Pre-generated noisy vision latents.
- `sound_latents` (`Tensor`, *optional*): Pre-generated noisy sound latents.
- `action_condition_frame_indexes` (`list`, *optional*): Action-frame indexes fixed by action conditioning.
- `action_latents` (`Tensor`, *optional*): Pre-generated noisy action latents.
- `enable_sound` (`bool`, *optional*, defaults to `False`): Whether to generate a synchronized sound track.
- `output_type` (`str`, *optional*, defaults to `pil`): Output format: 'pil', 'np', 'pt'.
**Outputs:**
- `videos` (`list`): The generated videos.
- `sound` (`Tensor`): Generated waveform.
- `sampling_rate` (`int`): Sample rate of the generated waveform in Hz.
- `action` (`list`): Generated action vectors.
|