CVPR 2026
Pushing the Frontier of Video Novel View Synthesisthrough Large-Scale Monocular Videos
Video Novel View Synthesis (VNVS) aims to render arbitrary novel viewpoints of dynamic scenes from a single-view video, but training VNVS models faces a major challenge: the lack of large-scale multi-view video datasets. Prior methods often train on monocular data by framing it as an inpainting task, which typically leads to a training-inference gap and visual artifacts. While synthetic multi-view data can partially alleviate the data scarcity issue, its high acquisition costs and limited diversity restrict scalability. To address these problems, we propose Scaling4D, a novel strategy that theoretically bridges the training-inference gap while leveraging large-scale monocular videos for training. Specifically, we take a higher-level perspective on the problem, reformulating VNVS into a general correspondence-guided generation task. Furthermore, in conjunction with extensive real-world data, we establish a synthetic data pipeline integrated with our training strategy to enhance precision. Qualitative and quantitative results demonstrate a positive correlation between performance and training data volume, confirming the scalability of our approach.
This example demonstrates the essential difference between inpainting and novel view synthesis. In the novel view, the area inside the red box should show the back of the person, but the inpainting result incorrectly fills it with background pixels.
Our correspondence control paradigm leverages optical flow to establish pixel correspondences between source and target clips extracted from any monocular video during training. By unifying training and inference under this single control condition, we ensure that inference scenarios are well covered by the diversity of real-world training data.
To complement the vast diversity of real-world data and further improve camera-pose control accuracy, we develop a specialized synthetic data pipeline. This highly scalable pipeline is constructed using Houdini's Procedural Dependency Graph (PDG). By representing all 3D assets in USD, we exploit its variant mechanism to efficiently generate randomized appearances and support extensible custom variants.
Our lightweight architecture features custom VNVS Blocks designed for seamless, plug-and-play integration with any pretrained foundation model. By explicitly omitting feed-forward (FFN) layers and strictly decoupling from text tokens, the design reduces computational overhead while preserving the foundation model's original prompt-following capabilities.
We investigate how our method scales with data volume (1k to 3000k) under a fixed total training iteration budget. Scaling yields a strong positive correlation with model performance but exhibits two distinct trends. First, visual quality metrics (FID, FVD, CLIP-V) improve continuously without reaching a saturation point, highlighting the framework's capability to translate massive data into superior fidelity. Second, pose accuracy metrics (RotError, TransError) plateau after 100k, suggesting that control precision eventually reaches an upper bound dictated by the current architecture or inherent flow estimation limits.
If you find Scaling4D useful for your work, please consider citing:
@inproceedings{cai2026scaling4d,
title = {Scaling4D: Pushing the Frontier of Video Novel View Synthesis through Large-Scale Monocular Videos},
author = {Cai, Hongrui and Luo, Junjie and Fu, Zhihong and Zhu, Shengnan and Wen, Jiawei and Feng, Wanquan and Zhao, Songtao and He, Qian},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2026}
}