GStex: Per-Primitive Texturing of 2D Gaussian Splatting for Decoupled Appearance and Geometry Modeling

Victor Rong, Jingxiang Chen, Sherwin Bahmani, Kiriakos N. Kutulakos, David B. Lindell
University of Toronto
Paper arXiv Code (coming soon)

(Column 1) In 2DGS, each 2D Gaussian encodes both appearance (blue) and geometry (green), and so these properties are coupled. Our approach decouples appearance and geometry through use of per-Gaussian texture maps. Each tile of the checkboard pattern represents 10 × 10 texels. For visualization purposes, we show Gaussians with opacity > 0.5.

(Column 2) We visualize novel view renderings for 2DGS and GStex, where we observe higher fidelity for our proposed GStex.

(Column 3) When rendering the edited scene from novel views, our representation results in sharper details because it can model features that are smaller in size than a single Gaussian.

(Columns 4-6) We find that GStex renders edited textures more accurately than 2DGS, even when using 200,000 Gaussians.

Abstract

Gaussian splatting has demonstrated excellent performance for view synthesis and scene reconstruction. The representation achieves photorealistic quality by optimizing the position, scale, color, and opacity of thousands to millions of 2D or 3D Gaussian primitives within a scene. However, since each Gaussian primitive encodes both appearance and geometry, these attributes are strongly coupled—thus, high-fidelity appearance modeling requires a large number of Gaussian primitives, even when the scene geometry is simple (e.g., for a textured planar surface). We propose to texture each 2D Gaussian primitive so that even a single Gaussian can be used to capture appearance details. By employing per-primitive texturing, our appearance representation is agnostic to the topology and complexity of the scene's geometry. We show that our approach, GStex, yields improved visual quality over prior work in texturing Gaussian splats. Furthermore, we demonstrate that our decoupling enables improved novel view synthesis performance compared to 2D Gaussian splatting when reducing the number of Gaussian primitives, and that GStex can be used for scene appearance editing and re-texturing.

Method

We initialize the model using 2DGS and assign an RGB texture map to each 2D Gaussian primitive. We render a pixel by casting a ray into the scene; the ray passes through a textured 2D Gaussian that is oriented along and with position and normal. The color is given by mapping the intersection point to the uv coordinates of the Gaussian's texture map , bilinearly interpolating the resulting texture value, and adding it to the Gaussian's view-dependent color component (parameterized by spherical harmonics). The alpha value is given by evaluating the Gaussian function at the intersection point. Finally, we alpha composite the rendered colors of each intersected Gaussian, resulting in the rendered pixel color.

  architecture

Novel View Synthesis

We show comparisons against Texture-GS (Xu et al., 2024) on the Blender synthetic dataset.

Novel View Synthesis for Discrete Levels of Detail

We demonstrate GStex's visual performance in a discrete LOD setting compared to 2DGS (Huang et al., 2024) across the Blender synthetic dataset and DTU dataset.

Appearance Editing

We show texture painting on a GStex model compared to on a 2DGS model.

We can also provide texturing with procedural textures. Below, we show side-by-side renders on two DTU scenes. The left panel is the original RGB render; the right panel is after retexturing.

=

BibTeX


        @article{rong2024gstex,
          title={GStex: Per-primitive texturing of 2D gaussian splatting for decoupled appearance and geometry modeling},
          author={Rong, Victor and Chen, Jingxiang and Bahmani, Sherwin and Kutulakos, Kiriakos N and Lindell, David B},
          journal={arXiv preprint arXiv:2409.12954},
          year={2024}
        }