inclusionAI
/

LLaDA2.1-mini

Text Generation

text_generation

Model card Files Files and versions

fixes for transforemrs v5

#4

by kashif HF Staff - opened Mar 15

base: refs/heads/main

←

from: refs/pr/4

Discussion Files changed

Mar 15

The "default" rope_type was removed in recent transformers versions. Using "linear" (with no scaling factor) is functionally equivalent.

fix: use linear rope_type instead of removed default for transformers compat52747ba6

Mar 15

fixes https://huggingface.co/inclusionAI/LLaDA2.1-mini/discussions/3

fix: remap default rope_type to linear for newer transformers compat0435edb8

fix: add default factor=1.0 for linear rope compat with newer transformers220496f1

fix: align RotaryEmbedding with Qwen2Moe pattern for transformers compata0f450fc

fix: call super()._init_weights() to match Qwen2Moe convention1f0d143f

fix: align _init_weights with Qwen2Moe using nn.init API92a5aa65

Use create_bidirectional_mask for backend-agnostic attention mask handling (SDPA, FA2, flex)90c63c74

kashif changed pull request title from fix: use linear rope_type instead of removed default for transformers compat to fixes for transforemrs v5 about 1 month ago

inclusionAI org 5 days ago

LGTM. Standardizing the RoPE implementation and adding partial_rotary_factor support makes total sense. Approved for merge. Thanks for the work!

lccurious changed pull request status to merged 5 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment