PermutoSDF: Fast Multi-View Reconstruction with
Implicit Surfaces using Permutohedral Lattices
CVPR 2023

Radu Alexandru Rosu, Sven Behnke
University of Bonn
Neural radiance-density field methods have become increasingly popular for the task of novel-view rendering. Their recent extension to hash-based positional encoding ensures fast training and inference with visually pleasing results. However, density-based methods struggle with recovering accurate surface geometry. Hybrid methods alleviate this issue by optimizing the density based on an underlying SDF. However, current SDF methods are overly smooth and miss fine geometric details. In this work, we combine the strengths of these two lines of work in a novel hash-based implicit surface representation. We propose improvements to the two areas by replacing the voxel hash encoding with a permutohedral lattice which optimizes faster, especially for higher dimensions. We additionally propose a regularization scheme which is crucial for recovering high-frequency geometric detail. We evaluate our method on multiple datasets and show that we can recover geometric detail at the level of pores and wrinkles while using only RGB images for supervision. Furthermore, using sphere tracing we can render novel views at 30 fps on an RTX 3090.

Video

Overview of our PermutoSDF pipeline. 1) For a batch of pixels from the posed images, we sample rays inside the volume of interest. 2) For each sample we slice features from a multi-resolution permutohedral lattice. 3) The features from all lattice levels are concatenated. For the color network, we also concatenate additional features regarding normal of the SDF, view direction and learnable features from the SDF network. 4) A small MLP decodes the SDF or a view-dependent RGB color. 5) The final output is an SDF and color field which is volumetrically rendered and supervised only with RGB images.

Highly Detailed Reconstruction

PermutoSDF can recover highly detailed reconstructions given only 2D images and camera poses. The recovered SDF and color can be rendered into novel-views that account for view-dependant effects. Using our proposed network regularization, we can recover details at the level of pores and wrinkles.

4D Spatio-Temporal Surface

The permutohedral lattice scales linearly with input dimensionality instead of exponentially as in the case of hyper-cubical voxels. This allows us to easily represent 4D surfaces which evolve in time. In this example we use a 4D lattice to encode an evolving shape represented as an SDF in (x,y,z,t). We traverse the time dimension of the network and show that it smoothly interpolates between the different time-steps of the shape.

Comparison

Other works that encode shape as an SDF fail to recover high-frequency detail. We show that using the permutohedral lattice combined with our proposed regularization method allows the network to recover significantly more geometric detail.

NeuS (8 hours) Ours (30min)

Citation

Acknowledgements


The website template was borrowed from Michaël Gharbi.