The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding

2025/12/22 by Weichen Fan, Fan, Weichen, Haiwen Diao +7
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis

paper · doi:10.48550/arxiv.2512.19693

openalex publication_date 2025/12/22 · openalex created_date 2025/12/24 · openalex updated_date 2026/07/28

Abstract

Deep representations across modalities are inherently intertwined. In this paper, we systematically analyze the spectral characteristics of various semantic and pixel encoders. Interestingly, our study uncovers a highly inspiring and rarely explored correspondence between an encoder's feature spectrum and its functional role: semantic encoders primarily capture low-frequency components that encode abstract meaning, whereas pixel encoders additionally retain high-frequency information that conveys fine-grained detail. This heuristic finding offers a unifying perspective that ties encoder behavior to its underlying spectral structure. We define it as the Prism Hypothesis, where each data modality can be viewed as a projection of the natural world onto a shared feature spectrum, just like the prism. Building on this insight, we propose Unified Autoencoding (UAE), a model that harmonizes semantic structure and pixel details via an innovative frequency-band modulator, enabling their seamless coexistence. Extensive experiments demonstrate that UAE effectively unifies semantic abstraction and pixel-level fidelity within a single latent space, achieving state-of-the-art performance. Moreover, we show that UAE can be directly applied to pixel-space modeling, significantly improving both FID and IS over the vanilla JIT baseline. Our code is avaliable at: https://github.com/WeichenFan/UAE.

Citations

• Latent Diffusion Model without Variational Autoencoder
• Diffusion Transformers with Representation Autoencoders
• UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation
• Next Visual Granularity Generation
• UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing
• Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation
• FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space
• ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement
• NFIG: Multi-Scale Autoregressive Image Generation via Frequency Ordering
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space
• TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation
• ZoomLDM: Latent Diffusion Model for multi-scale image generation
• An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels
• OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation
• Hierarchical Patch Diffusion Models for High-Resolution Video Generation
• Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation
• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
• SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers
• Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
• Generative Multimodal Models are In-Context Learners
• Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
• ImageBind: One Embedding Space To Bind Them All
• DINOv2: Learning Robust Visual Features without Supervision
• Generic-to-Specific Distillation of Masked Autoencoders
• Scalable Diffusion Models with Transformers
• MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis
• Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis
• Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
• GroupViT: Semantic Segmentation Emerges from Text Supervision
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
• High-Resolution Image Synthesis with Latent Diffusion Models
• Masked Autoencoders Are Scalable Vision Learners
• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• Alias-Free Generative Adversarial Networks
• BEiT: BERT Pre-Training of Image Transformers
• Cascaded Diffusion Models for High Fidelity Image Generation
• Emerging Properties in Self-Supervised Vision Transformers
• Learning Transferable Visual Models From Natural Language Supervision
• Focal Frequency Loss for Image Reconstruction and Synthesis
• Taming Transformers for High-Resolution Image Synthesis
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Implicit Neural Representations with Periodic Activation Functions
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Glow: Generative Flow with Invertible 1x1 Convolutions
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

Related

• From Pixels and Words to Waves: A Unified Framework for Spectral Dictionary vLLMs
• The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Training
• SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling
• Language Through a Prism: A Spectral Approach for Multiscale Language Representations
• Beyond Semantic Features: Pixel-level Mapping for Generalized AI-Generated Image Detection