The Free Transformer

2025/10/20 by François Fleuret, Fleuret, François · 3 voices
#cs.LG

paper · pdf · doi:10.48550/arxiv.2510.17558

Abstract

We propose an extension of the decoder Transformer that conditions its generative process on random latent variables which are learned without supervision thanks to a variational procedure. Experimental evaluations show that allowing such a conditioning translates into substantial improvements on downstream tasks.

Discussions

• The Free Transformer [hn, 6 points, 0 comments]
• From Meta: Exciting breakthrough in AI. The Free Transformer, a new decoder Transformer extension by François Fleuret (FAIR at Meta), uses unsupervised latent variables to boost performance on downstr [bsky, 2 points, 0 comments]
• 👇 Deep Dive in das Paper "The Free Transformer" (arxiv.org/abs/2510.17558), das den Decoder‑Transformer zu einem Conditional Variational Autoencoder (VAE) erweitert, der pro Token auf gelernte latent [bsky, 0 points, 1 comments]

Related

• Pretrained Transformers as Universal Computation Engines
• Making the Most of your Model: Methods for Finetuning and Applying Pretrained Transformers
• A Variational AutoEncoder for Transformers with Nonparametric Variational Information Bottleneck
• Finetuning Pretrained Transformers into Variational Autoencoders
• Solo Connection: A Parameter Efficient Fine-Tuning Technique for Transformers