We introduce the novel problem of point-level 3D scene interpolation. Given observations of a scene at two distinct states from multiple views, the goal is to synthesize a smooth point-level interpolation between them, without any intermediate supervision. Our method, PAPR in Motion, builds upon the recent Proximity Attention Point Rendering (PAPR) technique, and generates seamless interpolations of both the scene geometry and appearance.
We compare our method to the latest dynamic scene reconstruction method, Dynamic Gaussian, on this task. The results show the point cloud representation of the scene geometry in the first row and the corresponding RGB rendering in the second row.
As shown in the results above, methods build on Gaussian Splatting encounter difficulties in generating smooth point-level scene interpolations. This is due to the following issues:
Consider a pixel with high loss that’s not covered by any splat. Splatting-based methods would fail to move splats to cover it because of vanishing gradients. PAPR avoids this issue by using an attention mechanism where the total attention weights always sum up to one, making it more suitable for learning large scene changes.
Gaussian splats may no longer be Gaussian after non-rigid transformations and would introduce gaps in rendering. In contrast, attention-based methods like PAPR render by interpolating nearby points and avoid gaps naturally.
Compared to Gaussian Splatting, PAPR can better preserve rendering quality after non-rigid transformation, allowing for rendering non-rigid scene changes during the interpolation process.
@inproceedings{peng2024papr,
title={PAPR in Motion: Seamless Point-level 3D Scene Interpolation},
author={Shichong Peng and Yanshu Zhang and Ke Li},
booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2024}
}