PIQA: Reasoning about Physical Commonsense in Natural Language

2019/11/26 by Yonatan Bisk, Bisk, Yonatan, Rowan Zellers +7 · 763 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.1911.11641

AAAI 2020

arxiv created 2019/11/26 · arxiv updated 2019/11/27

Abstract

To apply eyeshadow without a brush, should I use a cotton swab or a toothpick? Questions requiring this kind of physical commonsense pose a challenge to today's natural language understanding systems. While recent pretrained models (such as BERT) have made progress on question answering over more abstract domains - such as news articles and encyclopedia entries, where text is plentiful - in more physical domains, text is inherently limited due to reporting bias. Can AI systems learn to reliably answer physical common-sense questions without experiencing the physical world? In this paper, we introduce the task of physical commonsense reasoning and a corresponding benchmark dataset Physical Interaction: Question Answering or PIQA. Though humans find the dataset easy (95% accuracy), large pretrained models struggle (77%). We provide analysis about the dimensions of knowledge that existing models lack, which offers significant opportunities for future research.

Cited by

• UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective
• Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Data
• Reservoir Computing inspired Matrix Multiplication-free Language Model
• Diversity or Precision? A Deep Dive into Next Token Prediction
• AFA-LoRA: Enabling Non-Linear Adaptations in LoRA with Activation Function Annealing
• Towards Efficient Post-Training via Fourier-Driven Adapter Architectures
• LLMBoost: Make Large Language Models Stronger with Boosting
• Raven: High-Recall Sequence Modeling with Sparse Memory Routing
• Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference
• CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data
• TriSP: Tri-Signal Structured Pruning for Large Language Models
• Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
• Compressing LLMs with MoP: Mixture of Pruners
• Deep Delta Learning
• Rethinking Output Alignment For 1-bit Post-Training Quantization of Large Language Models
• Memory-Efficient Acceleration of Block Low-Rank Foundation Models on Resource Constrained GPUs
• TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
• AraMix: Recycling, Refiltering, and Deduplicating to Deliver the Largest Arabic Pretraining Corpus
• Making Strong Error-Correcting Codes Work Effectively for HBM in AI Inference
• Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
• Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation
• Sigma-MoE-Tiny Technical Report
• Bolmo: Byteifying the Next Generation of Language Models
• T5Gemma 2: Seeing, Reading, and Understanding Longer
• Per-Axis Weight Deltas for Frequent Model Updates
• Dual-objective Language Models: Training Efficiency Without Overfitting
• VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reuse
• SASQ: Static Activation Scaling for Quantization-Aware Training in Large Language Models
• SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
• DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training
• MIDUS: Memory-Infused Depth Up-Scaling
• SIGMA: An AI-Empowered Training Stack on Early-Life Hardware
• SkipCat: Rank-Maximized Low-Rank Compression of Large Language Models via Shared Projection and Block Skipping
• CoRe3D: Collaborative Reasoning as a Foundation for 3D Intelligence
• Resting Neurons, Active Insights: Improving Input Sparsification for Large Language Models
• Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics
• Low-Rank Compression of Language Models via Differentiable Rank Selection
• Benchmarking the Generality of Vision-Language-Action Models
• Scaling Behavior of Discrete Diffusion Language Models
• LLaDA2.0: Scaling Up Diffusion Language Models to 100B
• Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
• MobileFineTuner: A Mobile-Native Framework for On-Device LLM Fine-Tuning in Real-World Embedded AI Applications
• GatedFWA: Linear Flash Windowed Attention with Gated Associative Memory
• PCMind-2.1-Kaiyuan-2B Technical Report
• Flash Multi-Head Feed-Forward Network
• Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs
• LIME: Making LLM Data More Efficient with Linguistic Metadata Embeddings
• Leveraging KV Similarity for Online Structured Pruning in LLMs
• Greedy Alignment Principle for Optimizer Selection
• SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs
• Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
• SignRoundV2: Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs
• Nexus: Higher-Order Attention Mechanisms in Transformers
• UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs
• Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in \± 1, ± i\
• Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedules
• PEFT-Factory: Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models
• HBLLM: Wavelet-Enhanced High-Fidelity 1-Bit Quantization for LLMs
• PerfMamba: Performance Analysis and Pruning of Selective State Space Models
• A Rosetta Stone for AI Benchmarks
• Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models
• Dripper: Token-Efficient Main HTML Extraction with a Lightweight LM
• Outlier Smoothing with Closed-Form Rotations for W4A4 Large Language Model Quantization
• Beyond URLs: Metadata Diversity and Position for Efficient LLM Pretraining
• IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference
• PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark
• CafeQ: Calibration-free Quantization via Learned Transformations and Adaptive Rounding
• ROOT: Robust Orthogonalized Optimizer for Neural Network Training
• SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot
• More Bias, Less Bias: BiasPrompting for Enhanced Multiple-Choice Question Answering
• Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models
• SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
• ModHiFi: Identifying High Fidelity predictive components for Model Modification
• FastForward Pruning: Efficient LLM Pruning via Single-Step Reinforcement Learning
• How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
• Xmodel-2.5: 1.3B Data-Efficient Reasoning SLM
• Blu-WERP (Web Extraction and Refinement Pipeline): A Scalable Pipeline for Preprocessing Large Language Model Datasets
• Layer-Wise High-Impact Parameter Ratio Optimization in Post-Training Quantization for Large Language Models
• R2Q: Towards Robust 2-Bit Large Language Models via Residual Refinement Quantization
• E³-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models
• Adaptive Layer-Wise Transformations for Post-Training Quantization of Large Language Models
• AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser
• Dynamic Nested Hierarchies: Pioneering Self-Evolution in Machine Learning Architectures for Lifelong Intelligence
• OTARo: Once Tuning for All Precisions toward Robust On-Device LLMs
• Learning from the Undesirable: Robust Adaptation of Language Models without Forgetting
• Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning
• AlignTree: Efficient Defense Against LLM Jailbreak Attacks
• FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Models
• Optimizing Mixture of Block Attention
• The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturation
• Why Should the Server Do It All?: A Scalable, Versatile, and Model-Agnostic Framework for Server-Light DNN Inference over Massively Distributed Clients via Training-Free Intermediate Feature Compression
• Safety-Preserving PTQ via Contrastive Alignment Loss
• Range Asymmetric Numeral Systems-Based Lightweight Intermediate Feature Compression for Split Computing of Deep Neural Networks
• SpecQuant: Spectral Decomposition and Adaptive Truncation for Ultra-Low-Bit LLMs Quantization
• Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs
• Learning to Focus: Focal Attention for Selective and Scalable Transformers
• Controllably Efficient Language Models
• Rethinking Parameter Sharing as Graph Coloring for Structured Compression
• Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
• MobileLLM-Pro Technical Report
• MuonAll: Muon Variant for Efficient Finetuning of Large Language Models
• Next-Latent Prediction Transformers Learn Compact World Models
• Motif 2 12.7B technical report
• PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
• Block Rotation is All You Need for MXFP4 Quantization
• DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization
• Memory- and Latency-Constrained Inference of Large Language Models via Adaptive Split Computing
• CryptoMoE: Privacy-Preserving and Scalable Mixture of Experts Inference via Balanced Expert Routing
• Epidemiology of Large Language Models: A Benchmark for Observational Distribution Knowledge
• IG-Pruning: Input-Guided Block Pruning for Large Language Models
• Random Initialization of Gated Sparse Adapters
• Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies
• FlashEVA: Accelerating LLM inference via Efficient Attention
• TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control
• LongCat-Flash-Omni Technical Report
• Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modeling
• Value Drifts: Tracing Value Alignment During LLM Post-Training
• 1+1>2: A Synergistic Sparse and Low-Rank Compression Method for Large Language Models
• MossNet: Mixture of State-Space Experts is a Multi-Head Attention
• NeuronMLP: Efficient LLM Inference via Singular Value Decomposition Compression and Tiling on AWS Trainium
• Gaperon: A Peppered English-French Generative Language Model Suite
• INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats
• When is Routing Meaningful? Diversity and Robustness in Language Model Societies
• Constitutional Midtraining: Content Presence Drives Alignment Gains
• Mixture-of-Depths Attention
• Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet
• KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Report
• Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization
• SSV: Sparse Speculative Verification for Efficient LLM Inference
• A Bitter Lesson for Data Filtering
• CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledge
• LOGICAL-COMMONSENSEQA: A Benchmark for Logical Commonsense Reasoning
• A Survey on Unlearning in Large Language Models
• Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale
• MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Sampling
• LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis
• Parallel Loop Transformer for Efficient Test-Time Computation Scaling
• BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction
• Calibrating and Rotating: A Unified Framework for Weight Conditioning in PEFT
• Beyond Line-Level Filtering for the Pretraining Corpora of LLMs
• Information-Theoretic Discrete Diffusion
• FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic
• MeCeFO: Enhancing LLM Training Robustness via Fault-Tolerant Optimization
• FourierCompress: Layer-Aware Spectral Activation Compression for Efficient and Accurate Collaborative LLM Inference
• ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
• A Survey on LLM Mid-Training
• Knocking-Heads Attention
• Simple Denoising Diffusion Language Models
• Softmax is 1/2-Lipschitz: A tight bound across all ℓ_p norms
• Offline Preference Optimization via Maximum Marginal Likelihood Estimation
• SeeDNorm: Self-Rescaled Dynamic Normalization
• Culturally Grounded Physical Commonsense Reasoning in Italian and English: A Submission to the MRL 2025 Shared Task
• PerCoR: Evaluating Commonsense Reasoning in Persian via Multiple-Choice Sentence Completion
• The Structural Scalpel: Automated Contiguous Layer Pruning for Large Language Models
• Label Smoothing Improves Gradient Ascent in LLM Unlearning
• Transformer Based Linear Attention with Optimized GPU Kernel Implementation
• A Coherence-Based Measure of AGI
• Context-level Language Modeling by Learning Predictive Context Embeddings
• Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation
• From Characters to Tokens: Dynamic Grouping with Hierarchical BPE
• Data-Centric Lessons To Improve Speech-Language Pretraining
• GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameters
• Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuning
• Latent Space Factorization in LoRA
• Energy-Efficient and Dequantization-Free Q-LLMs: A Spiking Neural Network Approach to Salient Value Mitigation
• ELUTQ: Efficient LUT-Aware Quantization for Deploying Large Language Models on Edge Devices
• CPSVD: Enhancing Large Language Model Compression via Column-Preserving Singular Value Decomposition
• Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall
• ARA: Adaptive Rank Allocation for Efficient Large Language Model SVD Compression
• Binary Quadratic Quantization: Beyond First-Order Quantization for Real-Valued Matrix Compression
• Learning from the Best, Differently: A Diversity-Driven Rethinking on Data Selection
• Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
• NeuroAda: Activating Each Neuron's Potential for Parameter-Efficient Fine-Tuning
• ScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameters
• Mapping Post-Training Forgetting in Language Models at Scale
• From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models
• Unbiased Gradient Low-Rank Projection
• MARS-M: When Variance Reduction Meets Matrices
• ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts
• Vocab Diet: Reshaping the Vocabulary of LLMs with Vector Arithmetic
• MergeMoE: Efficient Compression of MoE Models via Expert Output Merging
• Predicting Task Performance with Context-aware Scaling Laws
• Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing
• Closing the Gap Between Text and Speech Understanding in LLMs
• End-to-End Multi-Modal Diffusion Mamba
• Tahakom LLM Guidelines and Recipes: From Pre-training Data to an Arabic LLM
• Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Models
• Dr.LLM: Dynamic Layer Routing in LLMs
• CARVQ: Corrective Adaptor with Group Residual Vector Quantization for LLM Embedding Compression
• OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning
• Balancing Synthetic Data and Replay for Enhancing Task-Specific Capabilities
• Deconstructing Attention: Investigating Design Principles for Effective Language Modeling
• MeTA-LoRA: Data-Efficient Multi-Task Fine-Tuning for Large Language Models
• Neural Weight Compression for Language Models
• DND: Boosting Large Language Models with Dynamic Nested Depth
• ShishuLM: Lightweight Language Model with Hybrid Decoder-MLP Architecture and Paired Weight Sharing
• RePro: Training Language Models to Faithfully Recycle the Web for Pretraining
• Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization
• AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs
• Harnessing Consistency for Robust Test-Time LLM Ensemble
• Long Exposure: Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsity
• DixitWorld: Evaluating Multimodal Abductive Reasoning in Vision-Language Models with Multi-Agent Dixit Gameplay
• Tapered Language Models
• Decoupled DiLoCo for Resilient Distributed Pre-training
• FLRC: Fine-grained Low-Rank Compressor for Efficient LLM Inference
• KORMo: Korean Open Reasoning Model for Everyone
• RCPU: Rotation-Constrained Error Compensation for Structured Pruning of Large Language Models
• MeSH: Memory-as-State-Highways for Recursive Transformers
• SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networks
• Can Speech LLMs Think while Listening?
• Next Semantic Scale Prediction via Hierarchical Diffusion Language Models
• Native Hybrid Attention for Efficient Sequence Modeling
• Mid-Training of Large Language Models: A Survey
• Reusing Overtrained Language Models Saturates Scaling
• POME: Post Optimization Model Edit via Muon-style Projection
• Grouped Differential Attention
• PuzzlePlex: Benchmarking Foundation Models on Reasoning and Planning with Puzzles
• Training Dynamics Impact Post-Training Quantization Robustness
• Mixture of Neuron Experts
• Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM
• BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining
• Boomerang Distillation Enables Zero-Shot Model Size Interpolation
• Recover-LoRA: Data-Free Accuracy Recovery of Degraded Language Models via Low-Rank Adaptation
• SpikingMamba: Towards Energy-Efficient Large Language Models via Knowledge Distillation from Mamba
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• What Makes Diffusion Language Models Super Data Learners?
• Measuring Language Model Hallucinations Through Distributional Correctness
• Composer: A Search Framework for Hybrid Neural Architecture Design
• Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerability
• Learning Compact Representations of LLM Abilities via Item Response Theory
• Towards Ecologically Valid LLM Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitioners
• Thoughtbubbles: an Unsupervised Method for Parallel Thinking in Latent Space
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training
• CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Models
• Understanding the Mixture-of-Experts with Nadaraya-Watson Kernel
• The Flaw of Averages: Quantifying Uniformity of Performance on Benchmarks
• Layer-wise dynamic rank for compressing large language models
• MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources
• Fingerprinting LLMs via Prompt Injection
• Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs
• Conda: Column-Normalized Adam for Training Large Language Models Faster
• Negative Pre-activations Differentiate Syntax
• Short window attention enables long-term memorization
• Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining
• Pretraining with hierarchical memories: separating long-tail and common knowledge
• Tequila: Trapping-free Ternary Quantization for Large Language Models
• Beyond Outliers: A Study of Optimizers Under Quantization
• PATCH: Learnable Tile-level Hybrid Sparsity for LLMs
• Train Once, Answer All: Many Pretraining Experiments for the Cost of One
• SDQ-LLM: Sigma-Delta Quantization for 1-bit LLMs of any size
• A2D: Any-Order, Any-Step Safety Alignment for Diffusion Language Models
• PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space
• PT²-LLM: Post-Training Ternarization for Large Language Models
• MoE-PHDS: One MoE checkpoint for flexible runtime sparsity
• Quant-dLLM: Post-Training Extreme Low-Bit Quantization for Diffusion Large Language Models
• JE-IRT: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory
• Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attention
• Representing LLMs in Prompt Semantic Task Space
• JGU Mainz's Submission to the WMT25 Shared Task on LLMs with Limited Resources for Slavic Languages: MT and QA
• IIET: Efficient Numerical Transformer via Implicit Iterative Euler Method
• Stochastic activations
• HEAPr: Hessian-based Efficient Atomic Expert Pruning in Output Space
• Lightweight error mitigation strategies for post-training N:M activation sparsity in LLMs
• COSPADI: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning
• Induction Signatures Are Not Enough: A Matched-Compute Study of Load-Bearing Structure in In-Context Learning
• Blockwise Hadamard high-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning
• On Code-Induced Reasoning in LLMs
• CLUE: Conflict-guided Localization for LLM Unlearning Framework
• SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs
• Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment
• SpecMamba: Accelerating Mamba Inference on FPGA with Speculative Decoding
• Enhancing Linear Attention with Residual Learning
• RSAVQ: Riemannian Sensitivity-Aware Vector Quantization for Large Language Models
• Harnessing the Potential of Optimizing Data Mixtures via Bayesian Domain Reweighting
• GyRot: Leveraging Hidden Synergy Between Rotation and Fine-Grained Group Quantization for Low-Bit LLM Inference
• Subtract or Replay? Exact Deletion from Language-Model Memory
• Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs
• Σ-Mem: An Online Reliability Memory for LLM-based Multi-Agent Systems
• WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning
• Where and When to Commit: Candidate-Aware Decoding for Diffusion Language Models
• Averaged Evaluation Masks Capability Trade-Offs: Multi-Source Calibration for High-Sparsity LLM Pruning
• EMO: Pretraining Mixture of Experts for Emergent Modularity
• How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
• GradMAP: Faster Layer Pruning with Gradient Metric and Projection Compensation
• IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments
• Layerwise Importance Analysis of Feed-Forward Networks in Transformer-based Language Models
• When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models
• Prior-based Noisy Text Data Filtering: Fast and Strong Alternative For Perplexity
• HyperAdapt: Simple High-Rank Adaptation
• TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-training
• On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs
• Everyday Physics in Korean Contexts: A Culturally Grounded Physical Reasoning Benchmark
• QWHA: Quantization-Aware Walsh-Hadamard Adaptation for Parameter-Efficient Fine-Tuning on Large Language Models
• PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models
• Rethinking the Role of Text Complexity in Language Model Pretraining
• EG-MLA: Embedding-Gated Multi-head Latent Attention for Scalable and Efficient LLMs
• Distribution-Aligned Decoding for Efficient LLM Task Adaptation
• Evaluating the Effectiveness and Scalability of LLM-Based Data Augmentation for Retrieval
• How Good are Foundation Models in Step-by-Step Embodied Reasoning?
• Fair-GPTQ: Bias-Aware Quantization for Large Language Models
• AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguity
• TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge
• SBVR: Summation of BitVector Representation for Efficient LLM Quantization
• DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
• CBP-Tuning: Efficient Local Customization for Black-box Large Language Models
• AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Models
• Optimal Brain Restoration for Joint Quantization and Sparsification of LLMs
• Ko-PIQA: A Korean Physical Commonsense Reasoning Dataset with Cultural Context
• Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs
• InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoning
• ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transforms
• Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inference
• ENSI: Efficient Non-Interactive Secure Inference for Large Language Models
• Enabling Regulatory Multi-Agent Collaboration: Architecture, Challenges, and Solutions
• Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models
• Open-sci-ref-0.01: open and reproducible reference baselines for language model and dataset comparison
• Interpreting the Effects of Quantization on LLMs
• Causal Attention with Lookahead Keys
• COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens
• LoaQ: Layer-wise Output Approximation Quantization
• On Robustness and Reliability of Benchmark-Based Evaluation of LLMs
• A Probabilistic Inference Scaling Theory for LLM Self-Correction
• Cetvel: A Unified Benchmark for Evaluating Language Understanding, Generation and Cultural Capacity of LLMs for Turkish
• TeRA: Vector-based Random Tensor Network for High-Rank Adaptation of Large Language Models
• Calibration and Transformation-Free Weight-Only LLMs Quantization via Dynamic Grouping
• TRACE: Unlocking Effective CXL Bandwidth via Lossless Compression and Precision Scaling
• Implicit Reasoning in Large Language Models: A Comprehensive Survey
• GradES: Significantly Faster Training in Transformers with Gradient-Based Early Stopping
• Low-bit Model Quantization for Deep Neural Networks: A Survey
• LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving
• REFRAG: Rethinking RAG based Decoding
• Dream-Coder 7B: An Open Diffusion Language Model for Code
• LongCat-Flash Technical Report
• QueryBandits for Hallucination Mitigation: Exploiting Semantic Features for No-Regret Rewriting
• DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformers
• Universal Properties of Activation Sparsity in Modern Large Language Models
• Metis: Training LLMs with FP4 Quantization
• PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inference
• FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation
• TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference
• Diffusion Language Models Know the Answer Before Decoding
• Predicting the Order of Upcoming Tokens Improves Language Modeling
• SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version
• Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices
• Enabling MoE on the Edge via Importance-Driven Expert Scheduling
• UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning
• Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap
• Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
• Integral Transformer: Denoising Attention, Not Too Much Not Too Little
• DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstruction
• WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling
• End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost
• Dream 7B: Diffusion Large Language Models
• Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs
• NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model
• GLASS: Global-Local Aggregation for Inference-time Sparsification of LLMs
• Maximum Score Routing For Mixture-of-Experts
• MuDRiC: Multi-Dialect Reasoning for Arabic Commonsense Validation
• Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
• Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Models
• BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining
• GreenTEA: Gradient Descent with Topic-modeling and Evolutionary Auto-prompting
• TiMoE: Time-Aware Mixture of Language Experts
• Progressive Depth Up-scaling via Optimal Transport
• Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regime
• Pruning Large Language Models by Identifying and Preserving Functional Networks
• Align, Don't Divide: Revisiting the LoRA Architecture in Multi-Task Learning
• iFairy: the First 2-bit Complex LLM with All Parameters in \±1, ± i\
• LLM Data Selection and Utilization via Dynamic Bi-level Optimization
• Share Your Attention: Transformer Weight Sharing via Matrix-based Dictionary Learning
• FlexQ: Efficient Post-training INT6 Quantization for LLM Serving via Algorithm-System Co-Design
• Tensorized Clustered LoRA Merging for Multi-Task Interference
• Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models
• Coherent Multimodal Reasoning with Iterative Self-Evaluation for Vision-Language Models
• FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication
• CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis
• Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models
• Trainable Dynamic Mask Sparse Attention
• Kron-LoRA: Hybrid Kronecker-LoRA Adapters for Scalable, Sustainable Fine-tuning
• FPEdit: Robust LLM Fingerprinting through Localized Parameter Editing
• Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction
• Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
• Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
• EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
• Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product
• Unveiling Super Experts in Mixture-of-Experts Large Language Models
• Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked?
• OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration
• Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training
• Intent Aware Context Retrieval for Multi-Turn Agricultural Question Answering
• StackTrans: From Large Language Model to Large Pushdown Automata Model
• Inducing Causal World Models in LLMs for Zero-Shot Physical Reasoning
• DeltaLLM: A Training-Free Framework Exploiting Temporal Sparsity for Efficient Edge LLM Inference
• Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method
• Prune&Comp: Free Lunch for Layer-Pruned LLMs via Iterative Pruning with Magnitude Compensation
• Adaptive Block-Scaled Data Types
• Retrieval-Aware Distillation for Transformer-SSM Hybrids
• OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration
• Shaping capabilities with token-level data filtering
• A Comprehensive Evaluation on Quantization Techniques for Large Language Models
• BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMs
• WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
• Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
• MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning
• Pixels to Principles: Probing Intuitive Physics Understanding in Multimodal Language Models
• Language Models Improve When Pretraining Data Matches Target Tasks
• DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts
• PARAM-1 BharatGen 2.9B Model
• LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters
• Olica: Efficient Structured Pruning of Large Language Models without Retraining
• SDMPrune: Self-Distillation MLP Pruning for Efficient Large Language Models
• First-Order Error Matters: Accurate Compensation for Quantized Large Language Models
• AdaMuon: Adaptive Muon Optimizer
• Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
• GeLaCo: An Evolutionary Approach to Layer Compression
• Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMs
• DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models
• SLIM: A Heterogeneous Accelerator for Edge Inference of Sparse Large Language Model via Adaptive Thresholding
• Lizard: An Efficient Linearization Framework for Large Language Models
• KV Cache Steering for Controlling Frozen LLMs
• BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
• An Offline Mobile Conversational Agent for Mental Health Support: Learning from Emotional Dialogues and Psychological Texts with Student-Centered Evaluation
• Invariant-based Robust Weights Watermark for Large Language Models
• SAS: Simulated Attention Score
• COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation
• Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift
• FlexOlmo: Open Language Models for Flexible Data Use
• A Systematic Analysis of Hybrid Linear Attention
• Tokenizer-Agnostic Engram Module
• DocTalk: Scalable Graph-based Dialogue Synthesis for Enhancing LLM Conversational Capabilities
• Steering Information Utility in Key-Value Memory for Language Model Post-Training
• Train-before-Test Harmonizes Language Model Rankings
• Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients
• Nile-Chat: Egyptian Language Models for Arabic and Latin Scripts
• LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization
• GradOT: Training-free Gradient-preserving Offsite-tuning for Large Language Models
• DOTResize: Reducing LLM Width via Discrete Optimal Transport-based Neuron Merging
• RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling
• OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
• Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency
• MGAA: Multi-Granular Adaptive Allocation fof Low-Rank Compression of LLMs
• RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs
• Implicit Reasoning for Large Language Model-based Generative Recommendation
• Breaking the HBM Bit Cost Barrier: Domain-Specific ECC for AI Inference Infrastructure
• From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction
• High-Layer Attention Pruning with Rescaling
• MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining
• Eka-Eval: An Evaluation Framework for Low-Resource Multilingual Large Language Models
• La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
• MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE
• Private Memorization Editing: Turning Memorization into a Defense to Strengthen Data Privacy in Large Language Models
• Masked Gated Linear Unit
• Spectra 1.1: Scaling Laws and Efficient Inference for Ternary Language Models
• Residual Matrix Transformers: Scaling the Size of the Residual Stream
• Towards Distributed Neural Architectures
• AutoMixer: Checkpoint Artifacts as Automatic Data Mixers
• Training Language Model to Critique for Better Refinement
• Data Efficacy for Language Model Training
• Norm×Direction: Restoring the Missing Query Norm in Vision Linear Attention
• A Survey of AI for Materials Science: Foundation Models, LLM Agents, Datasets, and Tools
• GPTailor: Large Language Model Pruning Through Layer Cutting and Stitching
• DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs
• From 2D to 3D Cognition: A Brief Survey of General World Models
• Adapt Once, Thrive with Updates: Transferable Parameter-Efficient Fine-Tuning on Evolving Base Models
• DipSVD: Dual-importance Protected SVD for Efficient LLM Compression
• Tensor-Parallelism with Partially Synchronized Activations
• AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Models
• Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models
• Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations
• Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection
• CCI4.0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large Language Models
• A geometric framework for momentum-based optimizers for low-rank training
• Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque
• Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps
• Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights
• Revela: Dense Retriever Learning via Language Modeling
• SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity
• Long-Context Generalization with Sparse Attention
• Instruction Tuning with and without Context: Behavioral Shifts and Downstream Impact
• Alignment between Brains and AI: Evidence for Convergent Evolution across Modalities, Scales and Training Trajectories
• Learning-Time Encoding Shapes Unlearning in LLMs
• NeurIPS 2025 E2LM Competition : Early Training Evaluation of Language Models
• DBellQuant: Breaking the Bell with Double-Bell Transformation for LLMs Post Training Binarization
• RATTENTION: Towards the Minimal Sliding Window Size in Local-Global Attention Models
• Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
• MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation
• UltraSketchLLM: Saliency-Driven Sketching for Ultra-Low Bit LLM Compression
• Mixture of Cognitive Reasoners: Modular Reasoning with Brain-Like Specialization
• BOW: Training Language Models to Reason Over Plausible Next Words
• Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization
• Load Balancing Mixture of Experts with Similarity Preserving Routers
• TensorSLM: Energy-efficient Embedding Compression of Sub-billion Parameter Language Models on Low-end Devices
• EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimization
• GTA: Grouped-head latenT Attention
• Assessing the Role of Data Quality in Training Bilingual Language Models
• Unveiling Confirmation Bias in Chain-of-Thought Reasoning
• Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
• LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
• Curriculum-Guided Layer Scaling for Language Model Pretraining
• Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training
• One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers
• Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning
• Don't Pay Attention
• Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Language
• TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embedding
• Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events
• MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models
• dots.llm1 Technical Report
• Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data
• Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation
• Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
• DynamicMind: A Tri-Mode Thinking System for Large Language Models
• PCDVQ: Enhancing Vector Quantization for Large Language Models via Polar Coordinate Decoupling
• Exploring Diffusion Transformer Designs via Grafting
• MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
• FPTQuant: Function-Preserving Transforms for LLM Quantization
• SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs
• Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Information
• TokAlign: Efficient Vocabulary Adaptation via Token Alignment
• SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling
• A Statistical Physics of Language Model Reasoning
• PoLAR: Polar-Decomposed Low-Rank Adapter Representation
• CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
• DiaBlo: Diagonal Blocks Are Sufficient For Finetuning
• EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving
• Learning Together to Perform Better: Teaching Small-Scale LLMs to Collaborate via Preferential Rationale Tuning
• LocalGPT: Benchmarking and Advancing Large Language Models for Local Life Services in Meituan
• Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
• Beyond Text Compression: Evaluating Tokenizers Across Scales
• ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations
• Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition
• Exchangeability in Neural Network and its Application to Dynamic Pruning
• ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding
• zip2zip: Inference-Time Adaptive Tokenization via Online Compression
• Taming LLMs by Scaling Learning Rates with Gradient Grouping
• ReTern: Exploiting Natural Redundancy and Sign Transformations for Enhanced Fault Tolerance in Compute-in-Memory based Ternary LLMs
• Mamba Drafters for Speculative Decoding
• BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM Evaluation
• RLAE: Reinforcement Learning-Assisted Ensemble for LLMs
• SPAP: Structured Pruning via Alternating Optimization and Penalty Methods
• BadLingual: A Novel Lingual-Backdoor Attack against Large Language Models
• A Simple Linear Patch Revives Layer-Pruned Large Language Models
• Stepsize anything: A unified learning rate schedule for budgeted-iteration training
• SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Training
• GradPower: Powering Gradients for Faster Language Model Pre-Training
• Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM Evaluation
• Advantageous Parameter Expansion Training Makes Better Large Language Models
• LittleBit: Ultra Low-Bit Quantization via Latent Factorization
• ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration
• Recipes for Pre-training LLMs with MXFP8
• Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning
• TSENOR: Highly-Efficient Algorithm for Finding Transposable N:M Sparse Masks
• AC-ODM: Actor--Critic Online Data Mixing for Sample-Efficient LLM Pretraining
• A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning
• Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE
• DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration
• ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization
• Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training
• Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimization
• Weight Spectra Induced Efficient Model Adaptation
• LoLA: Low-Rank Linear Attention With Sparse Caching
• MAP: Revisiting Weight Decomposition for Low-Rank Adaptation
• Zero-Shot Vision Encoder Grafting via LLM Surrogates
• MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning
• SineLoRAΔ: Sine-Activated Delta Compression
• Highly Efficient and Effective LLMs with Multi-Boolean Architectures
• DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models
• SlimLLM: Accurate Structured Pruning for Large Language Models
• FireQ: Fast INT4-FP8 Kernel and RoPE-aware Quantization for LLM Inference Acceleration
• Pretraining Language Models to Ponder in Continuous Space
• RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale
• Sparsified State-Space Models are Efficient Highway Networks
• DenseLoRA: Dense Low-Rank Adaptation of Large Language Models
• QwT-v2: Practical, Effective and Efficient Post-Training Quantization
• Efficient Large Language Model Inference with Neural Block Linearization
• Hardware-Efficient Attention for Fast Decoding
• ResSVD: Residual Compensated SVD for Large Language Model Compression
• MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees
• MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs
• Radio: Rate-Distortion Optimization for Large Language Model Compression
• PiCa: Parameter-Efficient Fine-Tuning with Column Space Projection
• InFact: Informativeness Alignment for Improved LLM Factuality
• FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
• Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking
• GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretraining
• SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning
• Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selection
• BASE-Q: Bias and Asymmetric Scaling Enhanced Rotational Quantization for Large Language Models
• WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference
• ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
• GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
• An Empirical Study of Qwen3 Quantization
• RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
• Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization
• HD-PiSSA: High-Rank Distributed Orthogonal Adaptation
• AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping
• How Does Sequence Modeling Architecture Influence Base Capabilities of Pre-trained Language Models? Exploring Key Architecture Design Principles to Avoid Base Capabilities Degradation
• Data Mixing Can Induce Phase Transitions in Knowledge Acquisition
• FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling
• Training with Pseudo-Code for Instruction Following
• NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling
• Two-Stage Regularization-Based Structured Pruning for LLMs
• Fast Quiet-STaR: Thinking Without Thought Tokens
• COUNTDOWN: Contextually Sparse Activation Filtering Out Unnecessary Weights in Down Projection
• NeUQI: Near-Optimal Uniform Quantization Parameter Initialization for Low-Bit LLMs
• CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning
• Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMs
• LCD: Advancing Extreme Low-Bit Clustering for Large Language Models via Knowledge Distillation
• HOFT: Householder Orthogonal Fine-tuning
• LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
• PaTH Attention: Position Encoding via Accumulating Householder Transformations
• NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
• Understanding Differential Transformer Unchains Pretrained Self-Attentions
• Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs
• Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuning
• Logic-of-Thought: Empowering Large Language Models with Logic Programs for Solving Puzzles in Natural Language
• URLs Help, Topics Guide: Understanding Metadata Utility in LLM Training
• RAP: Runtime Adaptive Pruning for LLM Inference
• DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling
• Revealing Language Model Trajectories via Kullback-Leibler Divergence
• Pre-training Limited Memory Language Models with Internal and External Knowledge
• Set-LLM: A Permutation-Invariant LLM
• Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought
• SPhyR: Spatial-Physical Reasoning Benchmark on Material Distribution
• On the Generalization vs Fidelity Paradox in Knowledge Distillation
• Boost Post-Training Quantization via Null Space Optimization for Large Language Models
• Social Bias in Popular Question-Answering Benchmarks
• The Graph Language: How Knowledge Graphs Speak to Large Language Models
• Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference
• Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity
• Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization
• OSoRA: Output-Dimension and Singular-Value Initialized Low-Rank Adaptation
• Dual Decomposition of Weights and Singular Value Low Rank Adaptation
• ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models
• Safety Alignment Can Be Not Superficial With Explicit Safety Signals
• Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference
• Automatic mixed precision for optimizing gained time with constrained loss mean-squared-error based on model partition to sequential sub-graphs
• Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?
• GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection
• A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
• Towards Budget-Friendly Model-Agnostic Explanation Generation for Large Language Models
• SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment
• LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades
• Qronos: Correcting the Past by Shaping the Future... in Post-Training Quantization
• Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation
• Deep Research Pretraining via Predictive Navigation
• ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training
• A Systematic Analysis of Base Model Choice for Reward Modeling
• Addition is almost all you need: Compressing large language models with double binary factorization
• GenKnowSub: Improving Modularity and Reusability of LLMs through General Knowledge Subtraction
• Rethinking Prompt Optimizers: From Prompt Merits to Optimization
• Parallel Scaling Law for Language Models
• Motif-Mamba: network motif improved mamba for long-range sequence modeling
• What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs
• MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining
• Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compression
• GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
• GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models
• Improving Block-Wise LLM Quantization by 4-bit Block-Wise Optimal Float (BOF4): Analysis and Variations
• (How) Learning Rates Regulate Catastrophic Overtraining
• RAP: KV-Cache Compression via RoPE-Aligned Pruning
• A Switch-Centric In-Network Architecture for Accelerating LLM Inference in Shared-Memory Network
• When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
• Efficient Fine-Tuning of Quantized Models via Adaptive Rank and Bitwidth
• RWKVQuant: Quantizing the RWKV Family with Proxy Guided Hybrid of Scalar and Vector Quantization
• Grouped Sequency-arranged Rotation: Optimizing Rotation Transformation for Quantization for Free
• Don't be lazy: CompleteP enables compute-efficient deep transformers
• CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs
• Dynamic Linear Attention
• Leaderboard Incentives: Model Rankings under Strategic Post-Training
• Sphere Retraction Normalizations
• Data Darwinism Part II: DataEvolve -- AI can Autonomously Evolve Pretraining Data Curation
• Improved Large Language Diffusion Models
• Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation
• Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE
• On the Limits of Layer Pruning for Generative Reasoning in Large Language Models
• ICQuant: Index Coding enables Low-bit LLM Quantization
• Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing
• In-Place Test-Time Training
• The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning
• GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
• Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
• Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation
• Theory-optimal Quantization Based on Flatness
• Efficient LLMs with AMP: Attention Heads and MLP Pruning
• Softpick: No Attention Sink, No Massive Activations with Rectified Softmax
• Combatting Dimensional Collapse in LLM Pre-Training Data via Diversified File Selection
• HiFloat4 Format for Language Model Inference
• Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solving
• Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models
• Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
• Co-LMLM: Continuous-Query Limited Memory Language Models
• ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation
• The State-Prediction Separation Hypothesis
• Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
• Efficient Pre-Training with Token Superposition
• VLA Foundry: A Unified Framework for Training Vision-Language-Action Models
• PLDR-LLMs Reason At Self-Organized Criticality
• Dissecting Quantization Error: A Concentration-Alignment Perspective
• Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts
• Post-LayerNorm Is Back: Stable, ExpressivE, and Deep
• STEM: Scaling Transformers with Embedding Modules
• Nested Learning: The Illusion of Deep Learning Architectures
• R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
• Mixture of Experts for Decentralized Generative AI and Reinforcement Learning in Wireless Networks: A Comprehensive Survey
• HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing
• Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks
• BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs
• A Case Study of Selected PTQ Baselines for Reasoning LLMs on Ascend NPU
• Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation
• M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization
• Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling
• Representation-Aware Unlearning via Activation Signatures: From Suppression to Entity-Signature Erasure
• Scalable Frequency- and Length-Aware Subdocument Deduplication for Large Language Model Pretraining
• Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks
• LoopMTP: A looped transformer guided by latent multi-token prediction
• FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation
• Attention Needs to Focus: A Unified Perspective on Attention Allocation
• Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference
• MoEGen: Mixture-of-Experts for Instance-Adaptive LoRA Generation
• Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models
• Maglev: Sliding Recurrent Memory
• Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
• Cost-Aware Multi-Objective Bandits: Theory and Application to Budgeted LLM Configuration Evaluation
• Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
• Efficient Evaluation of Large Language Models via Collaborative Filtering
• Identifying and Evaluating Inactive Heads in Pretrained LLMs
• SHIELD: A Segmented Hierarchical Memory Architecture for Energy-Efficient LLM Inference on Edge NPUs
• QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining
• Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation
• AROMA: Autonomous Rank-one Matrix Adaptation
• Compression Laws for Large Language Models
• Saliency-driven Dynamic Token Pruning for Large Language Models
• Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism
• Efficient Pretraining Length Scaling
• Natural Fingerprints of Large Language Models
• NoWag: A Unified Framework for Shape Preserving Compression of Large Language Models
• CoLoTa: A Dataset for Entity-based Commonsense Reasoning over Long-Tail Knowledge
• Empirical Evaluation of Knowledge Distillation from Transformers to Subquadratic Language Models
• Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Accelerator
• D-GEN: Automatic Distractor Generation and Evaluation for Reliable Assessment of Generative Model
• DIDS: Domain Impact-aware Data Sampling for Large Language Model Training
• Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training
• It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization
• Unveiling Hidden Collaboration within Mixture-of-Experts in Large Language Models
• FLIP Reasoning Challenge
• Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs?
• EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding
• A Perplexity and Menger Curvature-Based Approach for Similarity Evaluation of Large Language Models
• Persona-Pruner: Sculpting Lightweight Models for Role-Playing
• Improving Instruct Models for Free: A Study on Partial Adaptation
• Enhancing Ultra-Low-Bit Quantization of Large Language Models Through Saliency-Aware Partial Retraining
• Can the capability of Large Language Models be described by human ability? A Meta Study
• Alleviating the Fear of Losing Alignment in LLM Fine-tuning
• A Survey of Reasoning with Foundation Models: Concepts, Methodologies, and Outlook
• C3PO: Critical-Layer, Core-Expert, Collaborative Pathway Optimization for Test-Time Expert Re-Mixing
• Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs
• LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation
• A Neuro-inspired Interpretation of Unlearning in Large Language Models through Sample-level Unlearning Difficulty
• Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation
• Graph-based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Survey
• Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs
• Lattice: Learning to Efficiently Compress the Memory

Related

• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
• Benchmarking Knowledge-Enhanced Commonsense Question Answering via Knowledge-to-Text Transformation
• How Additional Knowledge can Improve Natural Language Commonsense Question Answering?
• From Data to Commonsense Reasoning: The Use of Large Language Models for Explainable AI
• A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge