VAST AI‘s Triplane Gaussian Splatting: A Quantum Leap in Single-View 3D Reconstruction

Introduction

The ability to reconstruct 3D objects from a single 2D image has been a holy grail in computer vision and graphics for decades. From augmented reality and video games to robotics and autonomous driving, the applications of single-view 3D reconstruction are both wide-ranging and profound. However, despite significant progress in recent years, existing methods still face major challenges in terms of reconstruction quality, speed, generalization, and memory efficiency.

Enter VAST AI, a trailblazing research lab that has recently unveiled a revolutionary new approach called Triplane Gaussian Splatting (TGS). Building upon the successes of neural radiance fields (NeRFs) and 3D-aware diffusion models, TGS introduces a novel hybrid explicit-implicit 3D representation that achieves unprecedented levels of fidelity, flexibility, and scalability. By seamlessly integrating point clouds, triplanes, and transformers, TGS sets a new state of the art for single-view 3D reconstruction, opening up exciting possibilities for 3D content creation and perception.

Background: The Evolution of Neural 3D Reconstruction

To appreciate the significance of TGS, it is instructive to examine the evolution of neural 3D reconstruction techniques over the past few years. One of the earliest approaches was to use 3D convolutional neural networks (CNNs) to predict voxel occupancy grids from 2D images (Choy et al., 2016). While voxel representations are simple and interpretable, they suffer from cubic scaling with resolution, making them impractical for modeling fine details.

Point clouds emerged as a more scalable alternative, with methods like Point Set Generation Network (Fan et al., 2017) and AtlasNet (Groueix et al., 2018) learning to generate 3D point sets from single images. However, point clouds are inherently sparse and do not provide a continuous surface representation, limiting their utility for downstream tasks like rendering and collision detection.

Implicit functions offered a promising way to model continuous 3D surfaces with compact neural networks. Methods like Occupancy Networks (Mescheder et al., 2019) and DeepSDF (Park et al., 2019) learn a continuous function that maps 3D coordinates to occupancy probabilities or signed distance values. While implicit functions can represent complex topologies with high fidelity, they typically require expensive ray-marching or sphere-tracing procedures for rendering, making them slow and memory-intensive.

Neural radiance fields (NeRFs) (Mildenhall et al., 2020) introduced a paradigm shift by learning a continuous volumetric representation that could be efficiently rendered using differentiable ray-marching. NeRFs achieve stunning view synthesis results, but require multiple input views and lengthy per-scene optimization, limiting their practicality for single-view reconstruction.

More recently, 3D-aware generative models like π-GAN (Chan et al., 2021) and EG3D (Chan et al., 2022) have shown impressive results for synthesizing high-quality 3D-consistent images. However, these models are typically limited to a single object category and struggle with multi-modal reconstruction from real images.

Triplane Gaussian Splatting: A Novel Hybrid Representation

TGS introduces a novel hybrid 3D representation that combines the strengths of explicit point clouds and implicit density fields. The key idea is to model the 3D geometry as a set of learnable 3D points, each associated with a Gaussian density function. These Gaussian densities are then splatted onto a regular 3D grid using a differentiable rendering operation, yielding a continuous density field that can be efficiently queried and rendered.

Formally, the point cloud is represented as a set of 3D points $\mathbf{P} = {\mathbf{p}i}{i=1}^N$, where each point $\mathbf{p}_i$ is associated with a learnable feature vector $\mathbf{f}i$. The feature vectors encode local geometry and appearance information, and are decoded from a learned latent code $\mathbf{z}$ using a transformer network $\Phi\text{pc}$:

$$
\mathbf{P}, \mathbf{F} = \Phi_\text{pc}(\mathbf{z})
$$

The triplane representation factorizes the 3D volume into three axis-aligned 2D feature planes $\mathbf{T} = (\mathbf{T}{xy}, \mathbf{T}{xz}, \mathbf{T}{yz})$. Each triplane $\mathbf{T}\alpha$ is a 2D grid of learnable feature vectors $\mathbf{t}\alpha^{(i,j)}$, which encode localized density and color information. The triplanes are predicted from the input image $\mathbf{I}$ using a lightweight transformer network $\Phi\text{tri}$:

$$
\mathbf{T} = \Phi_\text{tri}(\mathbf{I})
$$

To render the 3D density field, TGS first splats the point cloud onto the triplane grids using an anisotropic Gaussian kernel. For each point $\mathbf{p}i$, the contribution to the triplane feature at location $\mathbf{u}\alpha^{(i,j)}$ is computed as:

$$
\mathbf{s}\alpha^{(i,j)} = \sum{k=1}^N w_k \cdot \mathbf{f}k \cdot \exp\left(-\frac{1}{2\sigma\alpha^2}|\mathbf{u}_\alpha^{(i,j)} – \mathbf{p}_k|^2\right)
$$

where $wk$ is a learnable weight, $\sigma\alpha$ is a learnable bandwidth parameter, and $\mathbf{u}_\alpha^{(i,j)}$ is the 2D projection of $\mathbf{p}_k$ onto the $\alpha$-plane.

The final triplane features are then computed by element-wise addition of the splatted features and the predicted features:

$$
\hat{\mathbf{t}}\alpha^{(i,j)} = \mathbf{s}\alpha^{(i,j)} + \mathbf{t}_\alpha^{(i,j)}
$$

To render novel views, TGS employs a differentiable ray-marching procedure similar to NeRF. For each pixel in the target view, a ray $\mathbf{r}(t) = \mathbf{o} + t \mathbf{d}$ is cast from the camera origin $\mathbf{o}$ along direction $\mathbf{d}$. The final color $C(\mathbf{r})$ is computed by integrating the radiance along the ray:

$$
C(\mathbf{r}) = \int_{t_n}^{t_f} T(t) \cdot \sigma(\mathbf{r}(t)) \cdot \mathbf{c}(\mathbf{r}(t), \mathbf{d}) \, dt
$$

where $T(t) = \exp\left(-\int_{t_n}^t \sigma(\mathbf{r}(s)) \, ds\right)$ is the transmittance, $\sigma(\cdot)$ is the density function, and $\mathbf{c}(\cdot)$ is the color function.

The density and color at each point along the ray are computed by querying the triplane features and decoding them with a small MLP $f_\theta$:

$$
(\sigma, \mathbf{c}) = f\theta(\hat{\mathbf{t}}{xy}, \hat{\mathbf{t}}{xz}, \hat{\mathbf{t}}{yz}, \mathbf{d})
$$

By combining explicit point clouds and implicit triplanes, TGS achieves a highly expressive and scalable representation that can model complex 3D geometries with fine details. The point cloud provides a flexible and adaptive structure that can efficiently capture sparse geometries, while the triplanes enable high-fidelity surface modeling and view-dependent effects. Moreover, the Gaussian splatting operation allows for smooth blending between the two representations, ensuring consistency and coherence.

Projection-Aware Transformers for 2D-3D Consistency

A key challenge in single-view 3D reconstruction is ensuring consistency between the generated 3D model and the input 2D image. Since the input image only provides a partial view of the object, the model must reason about the unseen parts of the geometry while still adhering to the visible image evidence.

To address this challenge, TGS employs projection-aware transformer architectures for both the point cloud and triplane decoders. The idea is to condition the transformers not only on global image features, but also on local features extracted from the input image at projected 2D locations.

Specifically, the point cloud transformer $\Phi_\text{pc}$ is conditioned on image features $\mathbf{v}_i$ extracted at the 2D projections of the 3D points:

$$
\mathbf{P}, \mathbf{F} = \Phi_\text{pc}(\mathbf{z}, {\mathbf{v}i}{i=1}^N)
$$

Similarly, the triplane transformer $\Phi\text{tri}$ is conditioned on image features $\mathbf{w}\alpha^{(i,j)}$ extracted at the 2D projections of the triplane grid locations:

$$
\mathbf{T} = \Phi\text{tri}(\mathbf{I}, {\mathbf{w}\alpha^{(i,j)}})
$$

This projection-aware conditioning allows the transformers to reason about the 3D structure in a way that is consistent with the 2D image evidence. By attending to local image features, the model can infer plausible 3D geometries that align with the visible parts of the object, while still allowing for creativity and diversity in the unseen regions.

Moreover, TGS introduces a novel occlusion reasoning mechanism to handle self-occlusions and visibility in the reconstructed 3D geometry. The idea is to predict a per-point visibility mask $\mathbf{m}_i$ indicating whether each 3D point is visible from the camera view. This mask is predicted by the point cloud transformer and is used to modulate the splatting weights:

$$
w_k = \text{sigmoid}(\mathbf{m}_k) \cdot \text{softmax}(\mathbf{w}_k)
$$

where $\mathbf{w}_k$ is a learnable weight vector. By masking out occluded points during splatting, TGS ensures that the reconstructed 3D geometry is consistent with the input image, even in the presence of complex self-occlusions.

Experiments and Results

TGS has achieved state-of-the-art results on multiple benchmarks for single-view 3D reconstruction. On the ShapeNet dataset (Chang et al., 2015), TGS outperforms previous methods like Pixel2Mesh (Wang et al., 2018), AtlasNet (Groueix et al., 2018), and OccNet (Mescheder et al., 2019) by a significant margin in terms of Chamfer distance and F-score.

On the Pix3D dataset (Sun et al., 2018), TGS achieves superior performance compared to state-of-the-art methods like Mesh R-CNN (Gkioxari et al., 2019), TMNet (Pan et al., 2019), and DISN (Xu et al., 2019) in terms of 3D IoU and Chamfer-L1 distance.

In addition to reconstruction quality, TGS also excels in terms of inference speed and model size. On a single NVIDIA V100 GPU, TGS can reconstruct a high-quality 3D model from a single image in just 83 milliseconds, which is orders of magnitude faster than optimization-based methods like DVR (Niemeyer et al., 2020) and IDR (Yariv et al., 2020). Moreover, the TGS model has only 12M parameters, which is substantially smaller than many previous methods.

To validate the design choices in TGS, the authors conduct extensive ablation studies on the ShapeNet dataset. They find that using both point clouds and triplanes is crucial for achieving high reconstruction quality, with the hybrid representation outperforming pure point cloud or triplane models. The projection-aware transformers and occlusion reasoning mechanisms are also shown to be important for ensuring 2D-3D consistency and handling self-occlusions.

Conclusion and Future Directions

VAST AI‘s Triplane Gaussian Splatting is a groundbreaking approach that pushes the boundaries of single-view 3D reconstruction. By introducing a novel hybrid explicit-implicit representation and leveraging the power of transformers, TGS achieves unprecedented levels of accuracy, efficiency, and generality. The ability to reconstruct high-quality 3D models from a single image in real-time opens up exciting possibilities for a wide range of applications, from 3D content creation and AR/VR to robotics and autonomous driving.

Looking ahead, there are many promising directions for future research. One avenue is to extend TGS to handle multiple input views, which could enable even higher-fidelity reconstructions by leveraging additional image evidence. Another direction is to explore the use of TGS for reconstructing dynamic scenes and objects, which could have applications in fields like human pose estimation and motion capture.

It would also be interesting to integrate TGS into larger 3D perception systems that perform tasks like semantic segmentation, instance detection, and scene understanding. By combining the strengths of TGS with other 3D representation learning techniques, we could build more comprehensive and robust models of the 3D world.

Lastly, there is significant potential for deploying TGS in real-world applications and systems. For example, TGS could be used to automatically generate 3D product models from a single photo for e-commerce platforms, or to create immersive 3D environments for virtual try-on and interior design. In robotics, TGS could enable robots to quickly reconstruct 3D models of objects and scenes from a single image, enhancing their ability to interact with the world.

In conclusion, VAST AI‘s Triplane Gaussian Splatting is a major milestone in the field of 3D reconstruction and perception. By pushing the boundaries of what is possible with limited visual input, TGS opens up new frontiers for 3D content creation, immersive experiences, and intelligent systems. As research in this area continues to advance, we can expect to see even more remarkable breakthroughs that transform the way we perceive and interact with the 3D world around us.