Update documentation to Please add how to character swap for Ref2V/Image to Video.

#60
by CoolaidFun - opened

I see some videos online now where people are successfully replacing people in a video with their image references but I have failed to do this no matter how many times and specific I word the prompt.
It will not transfer the subject in image 1 to replace the person in the reference video especially not preserving them across cuts.
It will also not transfer the subject in image 1 to replace the person in image 2 and image 3 if using only images.
Minimax seems to always cut to image 2 and 3 in the shots using the people in those images instead and ignore me asking it to put the subject of image 1 into those shots replacing the people in the other images.
I don't understand why.

I have followed the document but it isn't working https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md
My current prompt has the person from image 1 only working for shot 1, shot 2 and 3 will not make the person in image 1 override the people in image 2 and 3.
#################
<Subject 1> is the woman whose appearance comes from <Picture 1> and whose spoken words will come from <Audio 1>.
<Picture 2> and <Picture 3> is a storyboard reference for [Shot 2] and [Shot 3], defining their viewpoint, subject placement, and shot order.
<Subject 1> (appears in [Shot 1], [Shot 2], [Shot 3]): fully_preserved.

[0.00s: Shot 1] Camera fade to black showing <Subject 1> woman's face as she walks into the room.
6.00 [Shot 2]: The camera cuts to her lying on her back sitting on the couch with the camera close up to her face as she says the words directly from <Audio 1> [English] Hey hows it going....Oh you are watching a movie?.
10.00 [Shot 3]: The camera cuts to her from a new close up angle as she says the words directly from <Audio 1> [English] I really hate this movie you know!.
#################

Result: Uses shot 1 subject 1 from the reference image correctly but for Shot 2 and 3 it relies on the image 2 and 3 perfectly and uses the same camera angle and closeups of those images but it also uses the person in those images and is not replacing that person with the subject 1 from image 1!

How to do this correctly thank you?

Sign up or log in to comment