Large Concept Models: Language Modeling in a Sentence Representation Space

2024/12/11 by LCM team, Loïc Barrault, LCM Team +42 · 13 voices · 33 citations
Computer Science · #Advanced Text Analysis Techniques #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2412.08821

openalex publication_date 2024/12/11 · openalex created_date 2024/12/14 · openalex updated_date 2026/07/28

Abstract

LLMs have revolutionized the field of artificial intelligence and have emerged as the de-facto tool for many tasks. The current established technology of LLMs is to process input and generate output at the token level. This is in sharp contrast to humans who operate at multiple levels of abstraction, well beyond single words, to analyze information and to generate creative content. In this paper, we present an attempt at an architecture which operates on an explicit higher-level semantic representation, which we name a concept. Concepts are language- and modality-agnostic and represent a higher level idea or action in a flow. Hence, we build a "Large Concept Model". In this study, as proof of feasibility, we assume that a concept corresponds to a sentence, and use an existing sentence embedding space, SONAR, which supports up to 200 languages in both text and speech modalities. The Large Concept Model is trained to perform autoregressive sentence prediction in an embedding space. We explore multiple approaches, namely MSE regression, variants of diffusion-based generation, and models operating in a quantized SONAR space. These explorations are performed using 1.6B parameter models and training data in the order of 1.3T tokens. We then scale one architecture to a model size of 7B parameters and training data of about 2.7T tokens. We perform an experimental evaluation on several generative tasks, namely summarization and a new task of summary expansion. Finally, we show that our model exhibits impressive zero-shot generalization performance to many languages, outperforming existing LLMs of the same size. The training code of our models is freely available.

Cited by

• GRAPH-GRPO-LEX: Contract Graph Modeling and Reinforcement Learning with Group Relative Policy Optimization
• ParaScopes: What do Language Models Activations Encode About Future Text?
• From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
• Text Simplification with Sentence Embeddings
• Context-level Language Modeling by Learning Predictive Context Embeddings
• Chain-of-Conceptual-Thought Elicits Daily Conversation in Large Language Models
• Soft-Masked Diffusion Language Models
• The Idola Tribus of AI: Large Language Models tend to perceive order where none exists
• Multilingual Routing in Mixture-of-Experts
• Enabling Physical AI through Biological Principles
• Alternatives To Next Token Prediction In Text Generation -- A Survey
• Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks
• From Found to Designed: Concepts as a Design Axis for Large Language Models
• Charting trajectories of human thought using large language models
• LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures
• Neuro-Symbolic Predictive Process Monitoring
• SONAR-LLM: Autoregressive Transformer that Thinks in Sentence Embeddings and Speaks in Tokens
• Mitigating Resolution-Drift in Federated Learning: Case of Keypoint Detection
• Making Language Model a Hierarchical Classifier
• Improving Large Language Models with Concept-Aware Fine-Tuning
• Concept-Level AI for Telecom: Moving Beyond Large Language Models
• Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generation
• TALL -- A Trainable Architecture for Enhancing LLM Performance in Low-Resource Languages
• Context Is Not Comprehension
• Thinking with Generated Images
• Latent Reasoning via Sentence Embedding Prediction
• Distinctive Feature Codec: An Adaptive Efficient Speech Representation for Depression Detection
• TAG-INSTRUCT: Controlled Instruction Complexity Enhancement through Structure-based Augmentation
• Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space
• AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction
• Interpreting Black-Box Large Language Models with Sentence-Level Energy Landscapes
• The Rise of Small Language Models in Healthcare: A Comprehensive Survey
• Hierarchical Latent Prediction for Language Models
• On The Landscape of Spoken Language Models: A Comprehensive Survey
• Large Language Model (LLM) for Software Security: Code Analysis, Malware Analysis, Reverse Engineering

Discussions

• *Large Concept Models* by Barrault et al. Builds an autoregressive model in a "concept" space by wrapping the LLM in a pre-trained sentence embedder (also works with diffusion models). arxiv.org/abs [bsky, 6 points, 0 comments]
• Tech: Meta arbeitet an „Large Concept Models“, die statt Tokens der LLMs Sätze als Konzepte repräsentieren. Finde das spannenden Schritt, erinnert mich an das alte frame-based reasoning symbolischer K [bsky, 2 points, 0 comments]
• One way to do this is hierarchical LLMs like the Large Concept Model arxiv.org/abs/2412.08821, Byte Latent Transformer arxiv.org/abs/2412.09871, and Block Transformer arxiv.org/abs/2406.02657. [bsky, 1 points, 1 comments]
• Meta then proceeded to release 2 insane papers introducing Byte Linear Transformers and Large Concept Models, which aim to completely change the units of prediction in NLP. Expect Meta to outshine OAI [bsky, 1 points, 0 comments]
• 2/3 There are no embedding or logits flops in the LCM & the context length is much shorter (a sentence is on average 30 subwords), so a context length of 3000 subwords is only 100 in the LCM. See sec [bsky, 1 points, 1 comments]
• Interesting article from the FAIR lab at Meta on Large Concept Models. I think the general direction is interesting and that future models will need to work with things other than or in addition to to [bsky, 1 points, 0 comments]
• https://arxiv.org/pdf/2412.08821v2 [bsky, 1 points, 0 comments]
• Really interesting paper: The Large Concept Model is trained to perform autoregressive sentence prediction in an embedding space. arxiv.org/abs/2412.08821 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2412.08821 What do we think, #aisky? Is this feasible and will it break the mold of the #stochasticparrot view? #edusky [bsky, 0 points, 0 comments]
• Swapping individual tokens for predictable enough larger patches of meaning-constraining text. This is bloody brilliant, and that's not just because I've been calling LLMs a "sets of context made up [bsky, 0 points, 0 comments]
• It moves AI beyond word-by-word prediction toward higher-level reasoning. Instead of just predicting words, LCMs predict thoughts. That shift could make AI much more natural, scalable, and useful. [bsky, 0 points, 0 comments]
• This builds on two ideas: - DeepSeek V3 where they predicted next two tokens in advance - Meta's large concept models where they predict an embedding of paragraph, not a token arxiv.org/abs/2412.08821 [bsky, 0 points, 1 comments]
• Language Modeling in a Sentence Representation Space [Barrault+, 2024] Large Concept Models predict sentence embeddings instead of tokens. They inherently generalize to languages that their embedding [bsky, 0 points, 0 comments]

Related

• Towards Concept-Aware Large Language Models
• The Future of AI: Exploring the Potential of Large Concept Models
• A Survey on Large Language Models with some Insights on their Capabilities and Limitations
• Can Large Language Models Follow Concept Annotation Guidelines? A Case Study on Scientific and Financial Domains
• Scaling Sentence Embeddings with Large Language Models