2025/11/11 by Randall Balestriero, Yann LeCun, Balestriero, Randall +1 · 14 voices · 30 citations
Computer Science · #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #Explainable Artificial Intelligence (XAI) #cs.AI #cs.CV #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.2511.08544
openalex publication_date 2025/11/11 · openalex created_date 2025/11/13 · openalex updated_date 2026/07/28
LeJEPA replaces the heuristic machinery of modern self-supervised learning (asymmetric encoders, stop-gradients, momentum teachers, schedulers) with a single isotropic-Gaussian regulariser, SIGReg. Carrying SIGReg to sequence models exposes a documented hazard: a naive pooled application can drive every sequence to a constant vector along time, a time-axis collapse the objective fails to penalise. We build a heuristics-free time-series representation-learning library and compare three placements of SIGReg relative to the time axis (pooled, dual, structured). The collapse is real and placement-controlled: dual raises across-time variance by roughly an order of magnitude. Our central result is negative and clarifying: through a factorial crossing encoder architecture with placement and probe feature, the across-time variance collapse does not measure downstream order availability and is anticorrelated with it (Spearman rho = -0.64). What determines temporal-order retention is the encoder's positional structure, an axis orthogonal to the placement that controls the collapse: a position-free encoder gives a permutation-invariant pooled feature at exactly chance (0.500) on a content-matched probe over two datasets, while a positional transformer recovers order at 0.97-1.00 even when maximally collapsed. The dual placement is nonetheless valuable where the task needs the richer representation it produces: on UCI HAR it beats pooled by 8-11 accuracy points, and an eight-seed paired-significance forecasting study finds it significantly better on ETTh2 and ETTm1, worse on PEMS08, and indistinguishable on ETTh1. We propose a two-axis account: preventing the collapse does not change order availability (set by positional structure) but increases representation richness (effective rank), which helps tasks whose targets are structured rather than near-constant.