Stable Video Diffusion Guide to Image-to-Video Generation

Stable Video Diffusion (SVD) animates a supplied still image using Stability AI's downloadable image-to-video checkpoints. Announced on November 21, 2023, the family includes SVD for 14-frame sequences and SVD-XT for 25-frame sequences. Run the documented Diffusers pipeline with an image input and export the returned frames at your chosen playback rate. launch xt

What are the key facts about Stable Video Diffusion?

How do SVD and SVD-XT animate a still image?

SVD gives you a direct way to test motion from an existing composition. Its conditioning image establishes the visual starting point, making it useful for experiments where the still image has already been selected. xt

Choose that starting image deliberately. Before animating, decide which subject should remain recognizable and which kind of visible change would make the result useful for your project.

SVD was trained to generate 14 frames, while SVD-XT was further fine-tuned to generate 25. Both model cards specify a training resolution of 576 pixels high by 1024 pixels wide. base

Frame count is different from playback frame rate. When reviewing an output, record both the generated sequence length and the rate used to export the video, so another person can view the same timing.

Diffusers exposes additional conditioning controls for frame rate, motion level, and noise added to the input image. The documentation explains these through fps, motion_bucket_id, and noise_aug_strength. guide

Use those controls to explore a particular source image. Keep the image fixed while investigating motion settings, and save a baseline video before trying a more active result.

For example, an original evaluation could use a still life with a clearly defined central object. Judge whether that object remains acceptable throughout the sequence and whether the visible motion supports the intended shot.

What are Stable Video Diffusion's limitations?

The model cards explicitly list limited or absent motion, problems with faces, illegible text, and imperfect photorealism. They also state that the released models cannot be controlled through text. base

A text description of a camera move is therefore not a documented control input for this pipeline.

Review the entire clip. A convincing first frame does not show whether the middle and final frames preserve the object or face that matters to your project.

The guide describes memory trade-offs involving CPU offloading, feed-forward chunking, and VAE decode chunks. It notes that decoding one frame at a time can introduce flicker. guide

The model's license should be checked at the selected checkpoint.

The cited SVD-XT file includes commercial-use registration and revenue-related conditions, so downloadable weights should not be described as unrestricted public-domain assets. license

Separate video duration from generation latency when measuring performance. Your timing note should specify the hardware, checkpoint, frame count, and whether model loading is included.

How do you run Stable Video Diffusion locally?

Start with the published Diffusers SVD-XT example in a compatible PyTorch environment with Diffusers, Transformers, and Accelerate. For MP4 export, the cited Diffusers implementation recommends imageio and imageio-ffmpeg; install them in the same environment. guide

python -m pip install imageio imageio-ffmpeg

Use a source image you have prepared at 1024 by 576 pixels and save it as input.png. Preparing the composition beforehand lets you inspect the exact image that will condition the video.

This example follows the documented pipeline interface and uses a local image file. The seed and playback rate are example settings for a reproducible trial, rather than claims about generation speed. guide

import torch
from diffusers import StableVideoDiffusionPipeline
from diffusers.utils import load_image, export_to_video

pipe = StableVideoDiffusionPipeline.from_pretrained(
    "stabilityai/stable-video-diffusion-img2vid-xt",
    torch_dtype=torch.float16, variant="fp16"
)
pipe.enable_model_cpu_offload()
image = load_image("input.png")
generator = torch.manual_seed(42)
frames = pipe(
    image, num_frames=25, decode_chunk_size=8,
    generator=generator
).frames[0]
export_to_video(frames, "svd-xt.mp4", fps=7)

There is no text prompt in this call. The pipeline receives the image and generation controls, then returns frames that the export utility writes to a video file. guide

Evaluate the result in a short sequence:

1. Confirm that the generated file plays, then compare the starting composition with your source image. Record any important difference before judging the motion.
2. Watch the whole clip at its intended playback rate. Look for changes to the subject, unstable details, or movement that does not suit the shot.
3. Inspect representative frames separately. Keep still exports of visible failures so you can compare another configuration against the same problem.
4. Change one motion control while retaining the image and other settings. Record whether that change improves the intended movement and preserves the subject.
5. If memory is insufficient, follow the guide's offloading and chunking options, then review the result again for temporal artifacts. guide

The guide says increasing noise augmentation can increase motion while reducing resemblance to the source image. Treat both outcomes as part of the same test instead of reviewing motion alone. guide

Keep the input image, output clip, script, and environment versions together. This makes a useful handoff when another person needs to reproduce the shot or assess whether it meets the project brief.

For workflow construction, see the ComfyUI guide.

The separate Stable Video access article covers the related access topic.

How do SVD, SVD-XT, and SDXL compare?

Choose the model according to the stage you need. SDXL can help create a starting image; SVD then provides a separate experiment in image-conditioned motion. Evaluate the still and animated outputs independently.

What else should you know about Stable Video Diffusion?

Can Stable Video Diffusion generate video directly from text?

The released SVD and SVD-XT checkpoints use an image as conditioning. Both model cards explicitly list text control as unavailable. base

Does SVD-XT generate at 25 frames per second?

SVD-XT's 25-frame specification describes the sequence length it was trained to generate. Playback frame rate is a separate export setting, shown as 7 fps in the Diffusers usage example. xt

How can I reduce Stable Video Diffusion's GPU memory use?

The Diffusers SVD guide documents CPU offloading, feed-forward chunking, and smaller VAE decode chunks. Decoding one frame at a time uses less memory but can introduce flicker. guide

Is Stable Video Diffusion available as open weights?

Stability AI publishes SVD and SVD-XT checkpoints on Hugging Face. The cited SVD-XT weight license includes commercial-use registration and revenue conditions, so inspect the license for the checkpoint you select. base license

Sources

• Stability AI's SVD launch announcement
• Official SVD model card
• Official SVD-XT model card
• SVD-XT license file
• Diffusers image-to-video usage guide
• Diffusers MP4 export implementation and dependencies
• Official SDXL model card

Related guides on PromptZone

• AI Model Releases Timeline