Structured 3D Latents for Scalable and Versatile 3D Generation
2024/12/02 by Jianfeng Xiang, Zhiqing Lv, Xiang, Jianfeng +17 · 2 voices · 220 citations
Computer Science · Engineering · #Computer Graphics and Visualization Techniques #3D Shape Modeling and Analysis #Image Processing and 3D Reconstruction
paper · pdf · doi:10.48550/arxiv.2412.01506
Abstract
We introduce a novel 3D generation method for versatile and high-quality 3D asset creation. The cornerstone is a unified Structured LATent (SLAT) representation which allows decoding to different output formats, such as Radiance Fields, 3D Gaussians, and meshes. This is achieved by integrating a sparsely-populated 3D grid with dense multiview visual features extracted from a powerful vision foundation model, comprehensively capturing both structural (geometry) and textural (appearance) information while maintaining flexibility during decoding. We employ rectified flow transformers tailored for SLAT as our 3D generation models and train models with up to 2 billion parameters on a large 3D asset dataset of 500K diverse objects. Our model generates high-quality results with text or image conditions, significantly surpassing existing methods, including recent ones at similar scales. We showcase flexible output format selection and local 3D editing capabilities which were not offered by previous models. Code, model, and data will be released.
Cited by
- MSVS-VAE: Multi-Scale Anchored VecSet for High-Fidelity 3D Reconstruction
- SceneActBench: Can Agents Act on the 3D Scenes They See?
- LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows
- Axolotl3D: a Unified Framework for Faithful 3D Shape Completion
- LATO.2: Factorized 3D Mesh Generation with Vertex and Topology Flow
- CNS-Edit++: Category-Agnostic 3D Editing with Coupled Neural Shape Representation
- HOMIE: Human-object Centric Video Personalization via Multimodal Intelligent Enhancement
- UniPhysGen: Unified Physical Grounding for Simulation-Ready 3D Assets
- GEAR: Reconstruction of Classical Paintings via Geometry Grounding and Appearance Restitution
- The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric
- SubdivAR: Autoregressive Next-Scale Prediction for Neural Mesh Subdivision
- Scene-SAM3D: Multi-View Scene Asset Generation Without Fine-Tuning
- Clarify Before Executing: A Self-Evolving Agent for Resolving Intent Asymmetry in 3D Tool Orchestration
- MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction
- TanGO: Training-Free 3D Editing via Tangent-Space Guidance and Optimization
- Instant NuRec: Feed-Forward 3D Gaussian Reconstruction for Driving Scene Simulation
- Depth Anything 3: Recovering the Visual Space from Any Views
- SAM 3D: 3Dfy Anything in Images
- Memorization in 3D Shape Generation: An Empirical Study
- ShapeR: Robust Conditional 3D Shape Generation from Casual Captures
- SCPainter: A Unified Framework for Realistic 3D Asset Insertion and Novel View Synthesis
- SAM 3D for 3D Object Reconstruction from Remote Sensing Images
- Pixal3D: Pixel-Aligned 3D Generation from Images
- UMI3D: Robust 3D Generation on Unconstrained Multi-Image Inputs via Simultaneous Focus Cross-Attention Routing
- Orient Anything V2: Unifying Orientation and Rotation Understanding
- Fashion-3DLR: A Controllable 3D Garment Generation Using Pairwise Fashion Elements for Intelligent Design
- Nova3D: Code-Native Generation of Programmable 3D Assets
- Text-based Tactile Graphics Generation for the Visually Impaired
- Feedforward 3D Editing Learns from Semantic-Part Transformation
- DreamCAD: Scaling Multi-modal CAD Generation using Differentiable Parametric Surfaces
- UltraShape 1.0: High-Fidelity 3D Shape Generation via Scalable Geometric Refinement
- MVInverse: Feed-forward Multi-view Inverse Rendering in Seconds
- Retrieving Objects from 3D Scenes with Box-Guided Open-Vocabulary Instance Segmentation
- EchoMotion: Unified Human Video and Motion Generation via Dual-Modality Diffusion Transformer
- InsertAnywhere: Geometrically Grounded and Optics-Aware Video Object Insertion
- Make-It-Poseable: Feed-forward Latent Posing Model for 3D Characters
- 3DProxyImg: Controllable 3D-Aware Animation Synthesis from Single Image via 2D-3D Aligned Proxy Embedding
- Native and Compact Structured Latents for 3D Generation
- CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction
- SS4D: Native 4D Generative Model via Structured Spacetime Latents
- Particulate: Feed-Forward 3D Object Articulation
- ViewMask-1-to-3: Multi-View Consistent Image Generation via Multimodal Discrete Diffusion Models
- I-Scene: 3D Instance Models are Implicit Generalizable Spatial Learners
- Feedforward 3D Editing via Text-Steerable Image-to-3D
- CoRe3D: Collaborative Reasoning as a Foundation for 3D Intelligence
- Animus3D: Text-driven 3D Animation via Motion Score Distillation
- SpaceControl: Introducing Test-Time Spatial Control to 3D Generative Modeling
- SplatPainter: Interactive Authoring of 3D Gaussians from 2D Edits via Test-Time Training
- SceneMaker: Open-set 3D Scene Generation with Decoupled De-occlusion and Pose Estimation Model
- SWiT-4D: Sliding-Window Transformer for Lossless and Parameter-Free Temporal 4D Generation
- XDen-1K: A Density Field Dataset of Real-World Objects
- PoseGAM: Robust Unseen Object Pose Estimation via Geometry-Aware Multi-View Reasoning
- Are We Ready for RL in Text-to-3D Generation? A Progressive Investigation
- UniPart: Part-Level 3D Generation with Unified 3D Geom-Seg Latents
- StereoWorld: Geometry-Aware Monocular-to-Stereo Video Generation
- Self-Evolving 3D Scene Generation from a Single Image
- Graph Deep Learning for Intracranial Aneurysm Blood Flow Simulation and Risk Assessment
- Photo3D: Advancing Photorealistic 3D Generation through Structure-Aligned Detail Enhancement
- Inferring Compositional 4D Scenes without Ever Seeing One
- Voxify3D: Pixel Art Meets Volumetric Rendering
- MoCA: Mixture-of-Components Attention for Scalable Compositional 3D Generation
- MeshRipple: Structured Autoregressive Generation of Artist-Meshes
- See Once, Then Act: Vision-Language-Action Model with Task Learning from One-Shot Video Demonstrations
- Value Gradient Guidance for Flow Matching Alignment
- Object Reconstruction under Occlusion with Generative Priors and Contact-induced Constraints
- LaFiTe: A Generative Latent Field for 3D Native Texturing
- GaussianBlender: Instant Stylization of 3D Gaussians with Disentangled Latent Spaces
- TEXTRIX: Latent Attribute Grid for Native Texture Generation and Beyond
- SPARK: Sim-ready Part-level Articulated Reconstruction with VLM Knowledge
- IGen: Scalable Data Generation for Robot Learning from Open-World Images
- Cue3D: Quantifying the Role of Image Cues in Single-Image 3D Generation
- Efficient and Scalable Monocular Human-Object Interaction Motion Reconstruction
- Asset-Driven Sematic Reconstruction of Dynamic Scene with Multi-Human-Object Interactions
- SplatFont3D: Structure-Aware Text-to-3D Artistic Font Generation with Part-Level Style Control
- CC-FMO: Camera-Conditioned Zero-Shot Single Image to 3D Scene Generation with Foundation Model Orchestration
- GENA3D: Generative Amodal 3D Modeling by Bridging 2D Priors and 3D Coherence
- UniArt: Unified 3D Representation for Generating 3D Articulated Objects with Open-Set Articulation
- Fast3Dcache: Training-free 3D Geometry Synthesis Acceleration
- ITS3D: Inference-Time Scaling for Text-Guided 3D Diffusion Models
- Wukong's 72 Transformations: High-fidelity Textured 3D Morphing via Flow Models
- AnchorFlow: Training-Free 3D Editing via Latent Anchor-Aligned Flows
- CaliTex: Geometry-Calibrated Attention for View-Coherent 3D Texture Generation
- MFM-point: Multi-scale Flow Matching for Point Cloud Generation
- MajutsuCity: Language-driven Aesthetic-adaptive City Generation with Controllable 3D Assets and Layouts
- GigaWorld-0: World Models as Data Engine to Empower Embodied AI
- ShapeGen: Towards High-Quality 3D Shape Synthesis
- Cloud4D: Estimating Cloud Properties at a High Spatial and Temporal Resolution
- LATTICE: Democratize High-Fidelity 3D Generation at Scale
- PartDiffuser: Part-wise 3D Mesh Generation via Discrete Diffusion
- Yo'City: Personalized and Boundless 3D Realistic City Scene Generation via Self-Critic Expansion
- NeAR: Coupled Neural Asset-Renderer Stack
- InfiniBench: Infinite Benchmarking for Visual Spatial Reasoning with Customizable Scene Complexity
- Native 3D Editing with Full Attention
- Refracting Reality: Generating Images with Realistic Transparent Objects
- Mesh RAG: Retrieval Augmentation for Autoregressive Mesh Generation
- RoomPlanner: Explicit Layout Planner for Easier LLM-Driven 3D Room Generation
- WorldGen: From Text to Traversable and Interactive 3D Worlds
- SVG360: Editable Multiview Vector Graphics from a Single SVG
- PartUV: Part-Based UV Unwrapping of 3D Meshes
- NaTex: Seamless Texture Generation as Latent Color Diffusion
- From Prompts to Printable Models: Support-Effective 3D Generation via Offset Direct Preference Optimization
- PhysX-Anything: Simulation-Ready Physical 3D Assets from Single Image
- Part-X-MLLM: Part-aware 3D Multimodal Large Language Model
- LSS3D: Learnable Spatial Shifting for Consistent and High-Quality 3D Generation from Single-Image
- Appreciate the View: A Task-Aware Evaluation Framework for Novel View Synthesis
- Rethinking Multimodal Point Cloud Completion: A Completion-by-Correction Perspective
- LARM: A Large Articulated-Object Reconstruction Model
- Sat2RealCity: Geometry-Aware and Appearance-Controllable 3D Urban Generation from Satellite Imagery
- LoG3D: Ultra-High-Resolution 3D Shape Modeling via Local-to-Global Partitioning
- SPIDER: Scalable Physics-Informed Dexterous Retargeting
- ScaleADFG: Affordance-based Dexterous Functional Grasping via Scalable Dataset
- AnchorDS: Anchoring Dynamic Sources for Semantically Consistent Text-to-3D Generation
- Determinism of Randomness: Prompt-Residual Seed Shaping for Diffusion Generation
- ProcGen3D: Learning Neural Procedural Graph Representations for Image-to-3D Reconstruction
- Robot Learning from a Physical World Model
- Faithful Contouring: Near-Lossless 3D Voxel Representation Free from Iso-surface
- Room Envelopes: A Synthetic Dataset for Indoor Layout Reconstruction from Images
- HiGS: Hierarchical Generative Scene Framework for Multi-Step Associative Semantic Spatial Composition
- FullPart: Generating each 3D Part at Full Resolution
- FreeArt3D: Training-Free Articulated Object Generation using 3D Diffusion
- NOVA3R: Non-pixel-aligned Visual Transformer for Amodal 3D Reconstruction
- Gamifying the Past: Embodied LLMs in DIY Archaeological Video Games
- Surflo: Consistent 3D Surface Flow Model with Global State
- Beyond Prompts: Unconditional 3D Inversion for Out-of-Distribution Shapes
- SpatialTraceGen: High-Fidelity Traces for Efficient VLM Spatial Reasoning Distillation
- TRELLISWorld: Training-Free World Generation from Object Generators
- Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infilling
- ReconViaGen: Towards Accurate Multi-view 3D Object Reconstruction via Generation
- Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method
- LAMP: Data-Efficient Linear Affine Weight-Space Models for Parameter-Controlled 3D Shape Generation and Extrapolation
- WorldGrow: Generating Infinite 3D World
- Topology Sculptor, Shape Refiner: Discrete Diffusion Model for High-Fidelity 3D Meshes Generation
- CUPID: Generative 3D Reconstruction via Joint Object and Pose Modeling
- OnlineSplatter: Pose-Free Online 3D Reconstruction for Free-Moving Objects
- From Masks to Worlds: A Hitchhiker's Guide to World Models
- Seed3D 1.0: From Images to High-Fidelity Simulation-Ready 3D Assets
- Advances in 4D Representation: Geometry, Motion, and Interaction
- Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks
- Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes
- Demystifying Transition Matching: When and Why It Can Beat Flow Matching
- GACO-CAD: Geometry-Augmented and Conciseness-Optimized CAD Model Generation from Single Image
- GuideFlow3D: Optimization-Guided Rectified Flow For Appearance Transfer
- Coupled Diffusion Sampling for Training-Free Multi-View Image Editing
- Terra: Explorable Native 3D World Model with Point Latents
- NANO3D: A Training-Free Approach for Efficient 3D Editing Without Masks
- CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation
- GS-Verse: Mesh-based Gaussian Splatting for Physics-aware Interaction in Virtual Reality
- InfiniHuman: Infinite 3D Human Creation with Precise Control
- Jigsaw3D: Disentangled 3D Style Transfer via Patch Shuffling and Masking
- SyncHuman: Synchronizing 2D and 3D Generative Models for Single-view Human Reconstruction
- DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videos
- HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video
- ShapeGen4D: Towards High Quality 4D Shape Generation from Videos
- Towards Scalable and Consistent 3D Editing
- Exploring Opportunities to Support Novice Visual Artists' Inspiration and Ideation with Generative AI
- NeoWorld: Neural Simulation of Explorable Virtual Worlds via Progressive 3D Unfolding
- UniLat3D: Geometry-Appearance Unified Latents for Single-Stage 3D Generation
- Sparse-Up: Learnable Sparse Upsampling for 3D Generation with High-Fidelity Textures
- ZeroScene: A Zero-Shot Framework for 3D Scene Generation from a Single Image and Controllable Texture Editing
- M3DLayout: A Multi-Source Dataset of 3D Indoor Layouts and Structured Descriptions for 3D Generation
- PartSAM: A Scalable Promptable Part Segmentation Model Trained on Native 3D Data
- Large Material Gaussian Model for Relightable 3D Generation
- Hunyuan3D-Omni: A Unified Framework for Controllable Generation of 3D Assets
- QuadGPT: Native Quadrilateral Mesh Generation with Autoregressive Models
- Dense Semantic Matching with VGGT Prior
- MeshMosaic: Scaling Artist Mesh Generation via Local-to-Global Assembly
- Articulated Object Reconstruction from Rest-State Observation
- Generative Relightable Avatars
- 3D-Fixer: Coarse-to-Fine In-place Completion for 3D Scenes from a Single Image
- Generating Human-AI Collaborative Design Sequence for 3D Assets via Differentiable Operation Graph
- Hierarchical Neural Semantic Representation for 3D Semantic Correspondence
- RoboSeek: You Need to Interact with Your Objects
- FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models
- IDU: Incremental Dynamic Update of Existing 3D Virtual Environments with New Imagery Data
- WorldForge: Unlocking Emergent 3D/4D Generation in Video Diffusion Model via Training-Free Guidance
- AToken: A Unified Tokenizer for Vision
- CraftMesh: High-Fidelity Generative Mesh Manipulation via Poisson Seamless Fusion
- Hunyuan3D Studio: End-to-End AI Pipeline for Game-Ready 3D Asset Generation
- SPGen: Spherical Projection as Consistent and Flexible Representation for Single Image 3D Shape Generation
- No Mesh, No Problem: Estimating Coral Volume and Surface from Sparse Multi-View Images
- WildSmoke: Ready-to-Use Dynamic 3D Smoke Assets from a Single Video in the Wild
- Stable Part Diffusion 4D: Multi-View RGB and Kinematic Parts Video Generation
- T2Bs: Text-to-Character Blendshapes via Video Generation
- X-Part: high fidelity and structure coherent shape decomposition
- DreamLifting: A Plug-in Module Lifting MV Diffusion Models for 3D Asset Generation
- O3Afford: One-Shot 3D Object-to-Object Affordance Grounding for Generalizable Robotic Manipulation
- Few-step Flow for 3D Generation via Marginal-Data Transport Distillation
- TRELLIS-Enhanced Surface Features for Comprehensive Intracranial Aneurysm Analysis
- Unifi3D: A Study on 3D Representations for Generation and Reconstruction in a Common Framework
- TeRA: Rethinking Text-guided Realistic 3D Avatar Generation
- Category-Aware 3D Object Composition with Disentangled Texture and Shape Multi-view Diffusion
- NeuralMeshing: Complete Object Mesh Extraction from Casual Captures
- 3D-LATTE: Latent Space 3D Editing from Textual Instructions
- SceneGen: Single-Image 3D Scene Generation in One Feedforward Pass
- Realistic and Controllable 3D Gaussian-Guided Object Editing for Driving Video Generation
- Droplet3D: Commonsense Priors from Videos Facilitate 3D Generation
- VoxHammer: Training-Free Precise and Coherent 3D Editing in Native 3D Space
- Articulate3D: Zero-Shot Text-Driven 3D Object Posing
- FastMesh: Efficient Artistic Mesh Generation via Component Decoupling
- SemLayoutDiff: Semantic Layout Generation with Diffusion Model for Indoor Scene Synthesis
- SAT-SKYLINES: 3D Building Generation from Satellite Imagery and Coarse Geometric Priors
- Follow My Hold: Hand-Object Interaction Reconstruction through Geometric Guidance
- Collaborative Multi-Modal Coding for High-Quality 3D Generation
- Snap-Snap: Taking Two Images to Reconstruct 3D Human Gaussians in Milliseconds
- MeshCoder: LLM-Powered Structured Mesh Code Generation from Point Clouds
- EAvatar: Expression-Aware Head Avatar Reconstruction with Generative Geometry Priors
- Say It, See It: A Systematic Evaluation on Speech-Based 3D Content Generation Methods in Augmented Reality
- Elastic Diffusion Transformer
- Hi3DEval: Advancing 3D Generation Evaluation with Hierarchical Validity
- MagicHOI: Leveraging 3D Priors for Accurate Hand-object Reconstruction from Short Monocular Video Clips
- HOLODECK 2.0: Vision-Language-Guided 3D World Generation with Editing
- DreamSat-2.0: Towards a General Single-View Asteroid 3D Reconstruction
- Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesis
- XSpecMesh: Quality-Preserving Auto-Regressive Mesh Generation Acceleration via Multi-Head Speculative Decoding
- DISCOVERSE: Efficient Robot Simulation in Complex High-Fidelity Environments
- HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels
- BANG: Dividing 3D Assets via Generative Exploded Dynamics
- FMimic: Foundation Models are Fine-grained Action Learners from Human Videos
- Detecting Visual Information Manipulation Attacks in Augmented Reality: A Multimodal Semantic Reasoning Approach
- CRUISE: Cooperative Reconstruction and Editing in V2X Scenarios using Gaussian Splatting
Discussions
Related