Extrapolated Urban View Synthesis Benchmark

ICCV 2025

Xiangyu Han*^1,³, Zhen Jia*¹, Boyi Li², Yan Wang², Boris Ivanovic², Yurong You²,

Lingjie Liu³, Yue Wang^2,⁴, Marco Pavone^2,⁵, Chen Feng¹, Yiming Li^1,²

¹ NYU ² NVIDIA ³ University of Pennsylvania ⁴ USC ⁵ Stanford University
^* Equal Contribution

We introduce a novel, extensive real-world benchmark for quantitatively and qualitatively evaluating extrapolated novel view synthesis in large-scale urban scenes.

Our key contributions. Previous evaluations for urban view synthesis have primarily focused on interpolated poses, as the lack of ground truth data has made it challenging to evaluate extrapolated poses. We address this gap by providing real-world data that enables both quantitative and qualitative evaluations of extrapolated view synthesis in urban scenes. The quantitative results reveal a significant performance drop in Gaussian Splatting when handling extrapolated views, highlighting the need for more robust NVS methods.

Abstract

Photorealistic simulators are essential for the training and evaluation of vision-centric autonomous vehicles (AVs). At their core is Novel View Synthesis (NVS), a crucial capability that generates diverse unseen viewpoints to accommodate the broad and continuous pose distribution of AVs. Recent advances in radiance fields, such as 3D Gaussian Splatting, achieve photorealistic rendering at real-time speeds and have been widely used in modeling large-scale driving scenes. However, their performance is commonly evaluated using an interpolated setup with highly correlated training and test views. In contrast, extrapolation, where test views largely deviate from training views, remains underexplored, limiting progress in generalizable simulation technology. To address this gap, we leverage publicly available AV datasets with multiple traversals, multiple vehicles, and multiple cameras to build the first Extrapolated Urban View Synthesis (EUVS) benchmark. Meanwhile, we conduct quantitative and qualitative evaluations of state-of-the-art Gaussian Splatting methods across different difficulty levels. Our results show that Gaussian Splatting is prone to overfitting to training views. In addition, incorporating diffusion priors and improving geometry cannot fundamentally improve NVS under large view changes, highlighting the need for more robust approaches. We have released our data to help advance self-driving and urban robotics simulation technology.

Dataset Visualization

Dataset distribution. Our dataset comprises 90,810 frames distributed over 104 cases, capturing a diverse array of multi-traversal paths and multi-agent interactions across varying difficulty levels.

Results Comparison across Different Levels

Qualitative and quantitative results across three difficulty levels. The results show a clear degradation in performance as the difficulty level increases, highlighting the challenge of maintaining consistency and realism in complex urban scenarios.

Baseline Comparison Video

Location:

Translation-only experiment uses train and test traversal on different lanes.

Train

Test

Radar Chart of Baseline Comparison

Baseline Comparison Table

BibTeX

@inproceedings{han2025extrapolated,
      title={Extrapolated Urban View Synthesis Benchmark},
      author={Han, Xiangyu and Jia, Zhen and Li, Boyi and Wang, Yan and Ivanovic, Boris and You, Yurong and Liu, Lingjie and Wang, Yue and Pavone, Marco and Feng, Chen and Li, Yiming},
      booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
      year={2025},
      url={https://arxiv.org/abs/2412.05256}, 
}

Acknowledgements

This work was supported in part through NSF grants 2238968 and 2121391, and the NYU IT High Performance Computing resources, services, and staff expertise. Yiming Li is supported by NVIDIA Graduate Fellowship (2024-2025).