GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation

2024/03/21 by Yinghao Xu, Zifan Shi, Xu, Yinghao +13 · 51 citations
Computer Science · Engineering · #3D Shape Modeling and Analysis #Computer Graphics and Visualization Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Medical Image Segmentation Techniques

paper · pdf · doi:10.48550/arxiv.2403.14621

openalex publication_date 2024/03/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce GRM, a large-scale reconstructor capable of recovering a 3D asset from sparse-view images in around 0.1s. GRM is a feed-forward transformer-based model that efficiently incorporates multi-view information to translate the input pixels into pixel-aligned Gaussians, which are unprojected to create a set of densely distributed 3D Gaussians representing a scene. Together, our transformer architecture and the use of 3D Gaussians unlock a scalable and efficient reconstruction framework. Extensive experimental results demonstrate the superiority of our method over alternatives regarding both reconstruction quality and efficiency. We also showcase the potential of GRM in generative tasks, i.e., text-to-3D and image-to-3D, by integrating it with existing multi-view diffusion models. Our project website is at: https://justimyhxu.github.io/projects/grm/.

Cited by

• Geometric Context Transformer for Streaming 3D Reconstruction
• Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputs
• FlexAvatar: Flexible Large Reconstruction Model for Animatable Gaussian Head Avatars with Detailed Deformation
• ART: Articulated Reconstruction Transformer
• SplatPainter: Interactive Authoring of 3D Gaussians from 2D Edits via Test-Time Training
• Selfi: Self Improving Reconstruction Engine via 3D Geometric Feature Alignment
• MeshRipple: Structured Autoregressive Generation of Artist-Meshes
• Multi-view Pyramid Transformer: Look Coarser to See Broader
• DepthScape: Authoring 2.5D Designs via Depth Estimation, Semantic Understanding, and Geometry Extraction
• CC-FMO: Camera-Conditioned Zero-Shot Single Image to 3D Scene Generation with Foundation Model Orchestration
• ITS3D: Inference-Time Scaling for Text-Guided 3D Diffusion Models
• AnchorFlow: Training-Free 3D Editing via Latent Anchor-Aligned Flows
• Native 3D Editing with Full Attention
• CuriGS: Curriculum-Guided Gaussian Splatting for Sparse View Synthesis
• Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generation
• LSS3D: Learnable Spatial Shifting for Consistent and High-Quality 3D Generation from Single-Image
• Wonder3D++: Cross-domain Diffusion for High-fidelity 3D Generation from a Single Image
• HumanCrafter: Synergizing Generalizable Human Reconstruction and Semantic 3D Segmentation
• Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infilling
• ReconViaGen: Towards Accurate Multi-view 3D Object Reconstruction via Generation
• CUPID: Generative 3D Reconstruction via Joint Object and Pose Modeling
• From Volume Rendering to 3D Gaussian Splatting: Theory and Applications
• Capture, Canonicalize, Splat: Zero-Shot 3D Gaussian Avatars from Unstructured Phone Images
• VIST3A: Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generator
• SyncHuman: Synchronizing 2D and 3D Generative Models for Single-view Human Reconstruction
• Towards Scalable and Consistent 3D Editing
• GaussianLens: Localized High-Resolution Reconstruction via On-Demand Gaussian Densification
• Large Material Gaussian Model for Relightable 3D Generation
• Dark3R: Learning Structure from Motion in the Dark
• Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation
• SPFSplatV2: Efficient Self-Supervised Pose-Free 3D Gaussian Splatting from Sparse Views
• GWM: Towards Scalable Gaussian World Models for Robotic Manipulation
• SPGen: Spherical Projection as Consistent and Flexible Representation for Single Image 3D Shape Generation
• Align 3D Representation and Text Embedding for 3D Content Personalization
• DreamLifting: A Plug-in Module Lifting MV Diffusion Models for 3D Asset Generation
• SynthDrive: Scalable Real2Sim2Real Sensor Simulation Pipeline for High-Fidelity Asset Generation and Driving Data Synthesis
• Few-step Flow for 3D Generation via Marginal-Data Transport Distillation
• Complete Gaussian Splats from a Single Image with Denoising Diffusion Models
• Articulate3D: Zero-Shot Text-Driven 3D Object Posing
• Collaborative Multi-Modal Coding for High-Quality 3D Generation
• PhysGM: Large Physical Gaussian Model for Feed-Forward 4D Synthesis
• A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generation
• GAP: Gaussianize Any Point Clouds with Text Guidance
• Hi3DEval: Advancing 3D Generation Evaluation with Hierarchical Validity
• H3R: Hybrid Multi-view Correspondence for Generalizable 3D Reconstruction
• Dream-to-Recon: Monocular 3D Reconstruction with Diffusion-Depth Distillation from Single Images
• No Pose at All: Self-Supervised Pose-Free 3D Gaussian Splatting from Sparse Views
• iLRM: An Iterative Large 3D Reconstruction Model
• DepR: Depth Guided Single-view Scene Reconstruction with Instance-level Diffusion
• Learning Efficient and Generalizable Human Representation with Human Gaussian Model
• Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting

Related

• MVGamba: Unify 3D Content Generation as State Space Sequence Modeling
• GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting
• GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation
• iLRM: An Iterative Large 3D Reconstruction Model
• LRM: Large Reconstruction Model for Single Image to 3D