PAPR: Proximity Attention Point Rendering

Abstract

Learning accurate and parsimonious point cloud representations of scene surfaces from scratch remains a challenge in 3D representation learning. Existing point-based methods often suffer from the vanishing gradient problem or require a large number of points to accurately model scene geometry and texture. To address these limitations, we propose Proximity Attention Point Rendering (PAPR), a novel method that consists of a point-based scene representation and a differentiable renderer. Our scene representation uses a point cloud where each point is characterized by its spatial position, influence score, and view-independent feature vector. The renderer selects the relevant points for each ray and produces accurate colours using their associated features. PAPR effectively learns point cloud positions to represent the correct scene geometry, even when the initialization drastically differs from the target geometry. Notably, our method captures fine texture details while using only a parsimonious set of points. We also demonstrate four practical applications of our method, which are zero-shot geometry editing, object manipulation, texture transfer, and exposure control. More results and code are available at https://zvict.github.io/papr.

Publication
Conference on Neural Information Processing Systems 2023
@inproceedings{zhang2023papr,
    title={PAPR: Proximity Attention Point Rendering},
    author={Yanshu Zhang and Shichong Peng and Seyed Alireza Moazenipourasil and Ke Li},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
    year={2023}
}
Yanshu Zhang
Yanshu Zhang
Ph.D. Student

My research interests include 3D vision and Reinforcement Learning.

Shichong Peng
Shichong Peng
Ph.D. Student

My research focuses on generative model and 3D neural rendering. My goal is to develop methods that can better aid 3D object generation and animation.

Alireza Moazeni
Alireza Moazeni
Ph.D. Student

My research interests include Neural Rendering and Generative Models.

Ke Li
Ke Li
Assistant Professor of Computer Science

My research interests include Machine Learning, Computer Vision and Algorithms.