A Teaching Series

Building an Image-to-Video
Model from Scratch

Temporal consistency: objects must remain coherent across frames. A face cannot subtly shift shape between frame 5 and frame 6.
Causal structure: the model must understand that frame N depends only on frames 0 through N−1, not frames yet to come.
Scale: a 4-second 24fps video is 96 frames. Processing them jointly in pixel space is computationally infeasible without compression.
Conditioning fidelity: the generated video must look like the input image, not just be vaguely similar to it.

A step-by-step series from first principles to a working I2V pipeline. It covers video VAEs, diffusion transformers, flow matching, and conditioning, all built in PyTorch.

All Topics

Building an Image-to-VideoModel from Scratch

Building an Image-to-Video
Model from Scratch