SVFR: The First Multi-Task Video Face Restoration Framework Built on Stable Diffusion
Hook
Historical footage from the early 1900s often suffers from a perfect storm of degradation: low resolution, fading, scratches, and grayscale conversion. Until now, restoring such videos required chaining multiple specialized tools, each introducing its own artifacts and temporal inconsistencies.
Context
Face restoration has traditionally been treated as a collection of separate problems. Need to upscale a blurry face? Use one model. Want to colorize grayscale footage? Different model. Trying to remove scratches or occlusions? Yet another tool. This fragmented approach creates challenges for anyone working with severely degraded videos.
SVFR (Stable Video Face Restoration) offers a unified framework that handles blind face restoration (BFR), colorization, and inpainting within a single system. Built on top of Stable Video Diffusion, SVFR treats these tasks as different conditioning signals to the same temporal-aware diffusion process. This means you can potentially restore a scratched, low-resolution, black-and-white video in one pass. The framework has gained traction with 855 GitHub stars, and the demo videos show restoration of severely degraded historical footage.
Technical Insight
SVFR’s architecture builds on Stable Video Diffusion’s temporal priors for face-specific restoration. The system uses YOLO-based face detection (yoloface_v5m.pt) to locate and align faces in the input video. These face crops are processed through InsightFace embeddings (insightface_glint360k.pth) to extract identity features—ensuring the restored face maintains the same identity across all frames.
The framework uses task-ID conditioning, where different restoration tasks (BFR, colorization, inpainting) are signaled to the model through task IDs passed at inference time. The same weights handle all tasks, with behavior controlled through conditioning:
# Clone SVD weights
git lfs install
git clone https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt
# Required checkpoints according to README:
# - stable-video-diffusion-img2vid-xt/ (base temporal model)
# - face_align/yoloface_v5m.pt (face detection)
# - face_restoration/unet.pth (restoration weights)
# - face_restoration/id_linear.pth (identity preservation)
# - face_restoration/insightface_glint360k.pth (identity encoding)The multi-task capability works through task ID composition at inference:
# BFR only
python3 infer.py --task_ids 0 --input_path input.mp4
# BFR + Colorization
python3 infer.py --task_ids 0,1 --input_path input.mp4
# BFR + Colorization + Inpainting
python3 infer.py --task_ids 0,1,2 --input_path input.mp4 --mask_path mask.pngThe identity preservation mechanism uses InsightFace embeddings extracted from detected faces, which guide the restoration process to maintain the original person’s identity rather than generating generic faces—a common failure mode in face restoration models.
Temporal consistency comes from leveraging Stable Video Diffusion’s pre-trained temporal attention mechanisms. Unlike methods that process frames independently, SVFR’s diffusion process considers information from neighboring frames for coherent predictions about missing or corrupted details.
Gotcha
SVFR’s computational requirements are significant. The repository explicitly recommends 16GB+ VRAM. Running Stable Video Diffusion with face-specific conditioning requires substantial hardware, and there’s no mention of optimization techniques like model quantization that might make it accessible on lower-end GPUs.
The repository provides inference code and pre-trained checkpoints but no training code. If you want to fine-tune SVFR on your own dataset, there’s no documented path forward. The README doesn’t provide information about training dataset composition, hyperparameters, or training procedures.
Input constraints matter. The framework expects faces to be detectable and alignable. The README notes that input videos should have equal width and height, or you’ll need to use the --crop_face_region flag. This means preprocessing may be required for arbitrary aspect ratios common in real-world historical footage.
Verdict
Use SVFR if you’re working with severely degraded historical video footage that combines multiple restoration challenges—low resolution, grayscale, scratches, or occlusions—and you have access to GPU hardware with 16GB+ VRAM. The unified multi-task approach handles BFR, colorization, and inpainting in a single pass, and the temporal consistency makes it suitable for video work where frame-to-frame coherence matters. It’s also appropriate if you need identity preservation across frames, such as for archival restoration. Skip it if you have limited GPU resources (under 16GB VRAM), need to customize or train the model (no training code provided), only require single-image face restoration (lighter alternatives would be more efficient), or need to process videos with arbitrary aspect ratios without preprocessing. The computational requirements and lack of training documentation are the main barriers to adoption.
