Simplicity is Key: An Unsupervised Pretraining Approach for Sparse Radio Channels

2025/05/19 by Jonathan Ott, Ott, Jonathan, Maximilian Stahlke +7
Computer Science · Engineering · #FOS: Computer and information sciences #FOS: Electrical engineering #Indoor and Outdoor Localization Technologies #Machine Learning (cs.LG) #Signal Processing (eess.SP) #Sparse and Compressive Sensing Techniques #Wireless Signal Modulation Classification #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2505.13055

openalex publication_date 2025/05/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Unsupervised representation learning for wireless channel state information (CSI)reduces reliance on labeled data, thereby lowering annotation costs, and often improves performance on downstream tasks. However, state-of-the-art approaches take little or no account of domain-specific knowledge, forcing the model to learn well-known concepts solely from data. We introduce Sparse pretrained Radio Transformer (SpaRTran), a hybrid method based on the concept of compressed sensing for wireless channels. In contrast to existing work, SpaRTran builds around a wireless channel model that constrains the optimization procedure to physically meaningful solutions and induces a strong inductive bias. Compared to the state of the art, SpaRTran cuts positioning error by up to 28% and increases top-1 codebook selection accuracy for beamforming by 26 percentage points. Our results show that capturing the sparse nature of radio propagation as an unsupervised learning objective improves performance for network optimization and radio-localization tasks.

Citations

• Large Wireless Localization Model (LWLM): A Foundation Model for Positioning in 6G Networks
• A Multi-Task Foundation Model for Wireless Channel Representation Using Contrastive and Masked Autoencoder Learning
• WirelessGPT: A Generative Pre-trained Multi-task Learning Framework for Wireless Communication
• BERT4MIMO: A Foundation Model using BERT Architecture for Massive MIMO Channel State Information Prediction
• Large Wireless Model (LWM): A Foundation Model for Wireless Channels
• Radio Foundation Models: Pre-training Transformers for 5G-based Indoor Localization
• Improving Dictionary Learning with Gated Sparse Autoencoders
• Sparse Autoencoders Find Highly Interpretable Features in Language Models
• Sharp Convergence Rates for Matching Pursuit
• Self-Supervised and Invariant Representations for Wireless Localization
• Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions
• Indoor Localization with Robust Global Channel Charting: A Time-Distance-Based Approach
• Emerging Properties in Self-Supervised Vision Transformers
• Model-Based Deep Learning
• Characterizing the First-Arriving Multipath Component in 5G Millimeter\n Wave Networks: TOA, AOA, and Non-Line-of-Sight Bias
• Bootstrap your own latent: A new approach to self-supervised Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• CSI-based Positioning in Massive MIMO systems using Convolutional Neural\n Networks
• Momentum Contrast for Unsupervised Visual Representation Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• Towards Practical Indoor Positioning Based on Massive MIMO Systems
• DeepMIMO: A Generic Deep Learning Dataset for Millimeter Wave and Massive MIMO Applications
• A Very Brief Introduction to Machine Learning With Applications to Communication Systems
• Channel Charting: Locating Users within the Radio Environment using Channel State Information
• Approximating the Void: Learning Stochastic Channel Models from\n Observation with Variational Generative Adversarial Networks
• Deep Learning in Mobile and Wireless Networking: A Survey
• Attention Is All You Need
• Language Modeling with Gated Convolutional Networks
• Sparse Methods for Direction-of-Arrival Estimation
• SPICE: A Sparse Covariance-Based Estimation Method for Array Processing
• Signal Recovery From Random Measurements Via Orthogonal Matching Pursuit
• rm K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation
• Robust Uncertainty Principles: Exact Signal Reconstruction from Highly Incomplete Frequency Information
• Atomic Decomposition by Basis Pursuit

Related

• RadioDUN: A Physics-Inspired Deep Unfolding Network for Radio Map Estimation
• Estimating Sparsity Level for Enabling Compressive Sensing of Wireless Channels and Spectra in 5G and Beyond
• Unsupervised Representation Learning of Structured Radio Communication Signals
• Semi-Supervised Radio Signal Identification
• Radio Foundation Models: Pre-training Transformers for 5G-based Indoor Localization