Kakade, Sham
- Matryoshka Representation Learning
2022/05/26 by Aditya Kusupati, Gantavya Bhatt, Kusupati, Aditya +20 · 4 voices · 55 citations
Computer Science · Medicine · #cs.LG #cs.CV
- DataComp-LM: In search of the next generation of training sets for language models
2024/06/17 by Jeffrey Li, Alex Fang, Alex Chengyu Fang +121 · 2 voices · 79 citations
Computer Science · #Natural Language Processing Techniques #Topic Modeling #cs.CL #cs.LG
- MatFormer: Nested Transformer for Elastic Inference
2023/10/11 by Devvrit, Sneha Kudugunta, Aditya Kusupati +20 · 5 voices · 19 citations
Computer Science · Social Sciences · #Caching and Content Delivery #Human Mobility and Location-Based Analysis #Opportunistic and Delay-Tolerant Networks #cs.CL #cs.CV #cs.LG
- Meta-Learning with Implicit Gradients
2019/09/10 by Aravind Rajeswaran, Rajeswaran, Aravind, Chelsea Finn +5 · 59 citations
Computer Science · Physics and Astronomy · #Domain Adaptation and Few-Shot Learning #Advanced Neural Network Applications #Model Reduction and Neural Networks
- SOAP: Improving and Stabilizing Shampoo using Adam
2024/09/17 by Nikhil Vyas, Depen Morwani, Vyas, Nikhil +14 · 2 voices · 42 citations
Engineering · Computer Science · Biochemistry, Genetics and Molecular Biology · #Tree Root and Stability Studies #Music Technology and Sound Studies #Animal Vocal Communication and Behavior
- Stochastic subgradient method converges on tame functions
2018/04/20 by Davis, Damek, Drusvyatskiy, Dmitriy, Kakade, Sham +1 · 19 citations
#65K05 #65K10 #90C15 #90C30 #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC)
- Efficient Learning of Generalized Linear and Single Index Models with\n Isotonic Regression
2011/04/11 by Sham M. Kakade, Adam Tauman Kalai, Kakade, Sham +5 · 15 citations
Mathematics · Computer Science · #Statistical Methods and Inference #Bayesian Modeling and Causal Inference #Bayesian Methods and Mixture Models
- Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
2018/11/05 by Kendall Lowrey, Aravind Rajeswaran, Lowrey, Kendall +7 · 22 citations
Computer Science · Physics and Astronomy · Decision Sciences · #Reinforcement Learning in Robotics #Model Reduction and Neural Networks #Simulation Techniques and Applications
- Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions
2025/02/10 by Jaeyeon Kim, Kim, Jaeyeon, Kulin Shah +7 · 63 citations
Engineering · #Architecture and Computational Design
- Revisiting the Polyak step size
2019/05/01 by Elad Hazan, Hazan, Elad, Sham M. Kakade +1 · 13 citations
Computer Science · Decision Sciences · Engineering · #Advanced Bandit Algorithms Research #FOS: Mathematics #Optimization and Control (math.OC) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques
- FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs
2020/06/18 by Agarwal, Alekh, Kakade, Sham, Krishnamurthy, Akshay +1 · 12 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)
- Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit
2022/07/18 by Boaz Barak, Benjamin Edelman, Barak, Boaz +9 · 13 citations
Computer Science · Materials Science · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning in Materials Science #Neural and Evolutionary Computing (cs.NE) #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques
- Online Meta-Learning
2019/02/22 by Chelsea Finn, Aravind Rajeswaran, Finn, Chelsea +5 · 15 citations
Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Healthcare #Multimodal Machine Learning Applications
- Inductive Biases and Variable Creation in Self-Attention Mechanisms
2021/10/19 by Benjamin Edelman, Edelman, Benjamin L., Surbhi Goel +5 · 10 citations
Computer Science · #Data Stream Mining Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Topic Modeling
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
2025/04/10 by Zhao, Rosie, Meterez, Alexandru, Kakade, Sham +3 · 38 citations
#FOS: Computer and information sciences #I.2.7 #Machine Learning (cs.LG)
- Optimal Regularization Can Mitigate Double Descent
2020/03/04 by Preetum Nakkiran, Nakkiran, Preetum, Prayaag Venkat +5 · 7 citations
Computer Science · Engineering · #FOS: Computer and information sciences #FOS: Mathematics #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #Sparse and Compressive Sensing Techniques #Statistics Theory (math.ST) #Stochastic Gradient Optimization Techniques
- The Implicit and Explicit Regularization Effects of Dropout
2020/02/28 by Colin Wei, Sham M. Kakade, Wei, Colin +3 · 7 citations
Computer Science · #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques
- Learning Features of Music from Scratch
2016/11/29 by John Thickstun, Zaïd Harchaoui, Thickstun, John +3 · 8 citations
Computer Science · #Music and Audio Processing #Music Technology and Sound Studies #Speech and Audio Processing
- PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning
2020/07/16 by Alekh Agarwal, Agarwal, Alekh, Mikael Henaff +5 · 8 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics
- Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems
2024/02/27 by Zhenting Qi, Qi, Zhenting, Hanlin Zhang +7 · 12 citations
Computer Science · #Algorithms and Data Compression #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Semantic Web and Ontologies #Web Data Mining and Analysis
- Soft Threshold Weight Reparameterization for Learnable Sparsity
2020/02/08 by Aditya Kusupati, Vivek Ramanujan, Kusupati, Aditya +11 · 7 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Infrastructure Maintenance and Monitoring #Machine Learning (cs.LG) #Machine Learning (stat.ML)
- Model-Based Reinforcement Learning with a Generative Model is Minimax\n Optimal
2019/06/10 by Alekh Agarwal, Agarwal, Alekh, Sham M. Kakade +3 · 9 citations
Computer Science · #Reinforcement Learning in Robotics #Explainable Artificial Intelligence (XAI)
- LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks
2024/10/16 by Akshara Prabhakar, Prabhakar, Akshara, Yuanzhi Li +9 · 13 citations
Computer Science · Engineering · #Computation and Language (cs.CL) #Context-Aware Activity Recognition Systems #FOS: Computer and information sciences #Intelligent Tutoring Systems and Adaptive Learning #Machine Learning (cs.LG) #Robotics and Automated Systems
- Understanding Contrastive Learning Requires Incorporating Inductive Biases
2022/02/28 by Nikunj Saunshi, Jordan T. Ash, Saunshi, Nikunj +13 · 6 citations
Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Machine Learning and Data Classification
- How Does Critical Batch Size Scale in Pre-training?
2024/10/29 by Hanlin Zhang, Zhang, Hanlin, Depen Morwani +13 · 14 citations
Psychology · #Human Resource Development and Performance Evaluation
- Interpreting the linear structure of vision-language model embedding spaces
2025/04/16 by Isabel Papadimitriou, Huangyuan Su, Papadimitriou, Isabel +8 · 2 voices · 11 citations
Computer Science · Social Sciences · Psychology · #Multimodal Machine Learning Applications #Language and cultural evolution #Language, Metaphor, and Cognition
- A New Perspective on Shampoo's Preconditioner
2024/06/25 by Depen Morwani, Morwani, Depen, Itai Shapira +9 · 10 citations
Engineering · #Acoustic Wave Phenomena Research #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Vibration Control and Rheological Fluids
- On Provable Copyright Protection for Generative Models
2023/02/21 by Vyas, Nikhil, Kakade, Sham, Barak, Boaz · 6 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)
- Information Theoretic Regret Bounds for Online Nonlinear Control
2020/06/22 by Kakade, Sham, Krishnamurthy, Akshay, Lowrey, Kendall +2 · 4 citations
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC) #Robotics (cs.RO)
- Robust and Differentially Private Mean Estimation
2021/02/18 by Xiyang Liu, Weihao Kong, Liu, Xiyang +5 · 4 citations
Computer Science · #Adversarial Robustness in Machine Learning #Cryptography and Security (cs.CR) #Distributed Sensor Networks and Detection Algorithms #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Privacy-Preserving Technologies in Data
- Gone Fishing: Neural Active Learning with Fisher Embeddings
2021/06/17 by Jordan T. Ash, Surbhi Goel, Ash, Jordan T. +5 · 4 citations
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Machine Learning and Data Classification
- Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
2018/03/20 by Wu, Cathy, Rajeswaran, Aravind, Duan, Yan +5 · 3 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)
- Learning from Logged Implicit Exploration Data
2010/02/27 by Strehl, Alex, Langford, John, Kakade, Sham +1 · 2 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)
- Random Scaling of Emergent Capabilities
2025/02/24 by Rosie Zhao, Zhao, Rosie, Tian Qin +8 · 2 voices · 5 citations
Decision Sciences · #Complex Systems and Decision Making #cs.LG
- When are Overcomplete Topic Models Identifiable? Uniqueness of Tensor Tucker Decompositions with Structured Sparsity
2013/08/13 by Anandkumar, Animashree, Hsu, Daniel, Janzamin, Majid +1 · 2 citations
#FOS: Computer and information sciences #FOS: Mathematics #Information Retrieval (cs.IR) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Numerical Analysis (math.NA) #Statistics Theory (math.ST)
- AdANNS: A Framework for Adaptive Semantic Search
2023/05/30 by Aniket Rege, Rege, Aniket, Aditya Kusupati +14 · 1 voice · 2 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Image Retrieval and Classification Techniques #Information Retrieval (cs.IR) #Machine Learning (cs.LG) #cs.IR #cs.LG
- From an Image to a Scene: Learning to Imagine the World from a Million 360 Videos
2024/12/10 by Matthew Wallingford, Anand Bhattad, Wallingford, Matthew +17 · 7 citations
Health Professions · Social Sciences · #Digital Storytelling and Education #Educator Training and Historical Pedagogy
- Towards Generalization and Simplicity in Continuous Control
2017/03/08 by Rajeswaran, Aravind, Lowrey, Kendall, Todorov, Emanuel +1 · 2 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Robotics (cs.RO) #Systems and Control (eess.SY) #electronic engineering #information engineering
- Prediction with a Short Memory
2016/12/08 by Vatsal Sharan, Sham M. Kakade, Sharan, Vatsal +5 · 2 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Algorithms and Data Compression #Artificial Intelligence (cs.AI) #Computational Complexity (cs.CC) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms
- Q-Probe: A Lightweight Approach to Reward Maximization for Language Models
2024/02/22 by Li, Kenneth, Jelassi, Samy, Zhang, Hugh +3 · 4 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG)
- Universal Length Generalization with Turing Programs
2024/07/03 by Kaiying Hou, David Brandfonbrener, Hou, Kaiying +7 · 5 citations
Computer Science · #Computability, Logic, AI Algorithms #FOS: Computer and information sciences #Machine Learning (cs.LG)
- Deconstructing What Makes a Good Optimizer for Language Models
2024/07/10 by Rosie Zhao, Depen Morwani, Zhao, Rosie +7 · 6 citations
Computer Science · #Natural Language Processing Techniques
- Feature emergence via margin maximization: case studies in algebraic tasks
2023/11/13 by Depen Morwani, Benjamin Edelman, Morwani, Depen +7 · 3 citations
Computer Science · Physics and Astronomy · #FOS: Computer and information sciences #I.2.6 #I.5.1 #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Neural Networks and Applications #Topological and Geometric Data Analysis
- Multi-Stage Episodic Control for Strategic Exploration in Text Games
2022/01/04 by Jens Tuyls, Tuyls, Jens, Shunyu Yao +5 · 2 citations
Computer Science · Psychology · #Artificial Intelligence in Games #Computation and Language (cs.CL) #Educational Games and Gamification #FOS: Computer and information sciences #Reinforcement Learning in Robotics
- A Study on the Calibration of In-context Learning
2023/12/07 by Hanlin Zhang, Yifan Zhang, Zhang, Hanlin +13 · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Education and Learning Interventions #FOS: Computer and information sciences #Machine Learning (cs.LG)
- A Complete Characterization of Linear Estimators for Offline Policy Evaluation
2022/03/08 by Perdomo, Juan C., Krishnamurthy, Akshay, Bartlett, Peter +1 · 2 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)
- CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training
2024/06/15 by David Brandfonbrener, Hanlin Zhang, Brandfonbrener, David +7 · 4 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Speech Recognition and Synthesis
- Any-Order Flexible Length Masked Diffusion
2025/08/31 by J.M. Kim, Kim, Jaeyeon, Lee Cheuk-Kit +13 · 12 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning in Healthcare
- Convergence Rates of Active Learning for Maximum Likelihood Estimation
2015/06/08 by Chaudhuri, Kamalika, Kakade, Sham, Netrapalli, Praneeth +1 · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)
- Scaling Laws for Imitation Learning in Single-Agent Games
2023/07/18 by Tuyls, Jens, Madeka, Dhruv, Torkkola, Kari +3 · 2 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)
- Learning Overcomplete HMMs
2017/11/07 by Sharan, Vatsal, Kakade, Sham, Liang, Percy +1 · 1 citation
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)
- Variance Reduction Methods for Sublinear Reinforcement Learning
2018/02/26 by Kakade, Sham, Wang, Mengdi, Yang, Lin F. · 1 citation
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)
- Cognitive models can reveal interpretable value trade-offs in language models
2025/06/25 by Sonia K. Murthy, Rosie Zhao, Murthy, Sonia K. +13 · 3 voices
Computer Science · #Natural Language Processing Techniques #Semantic Web and Ontologies
- How Important is the Train-Validation Split in Meta-Learning?
2020/10/12 by Yu Bai, Bai, Yu, Minshuo Chen +13 · 2 citations
Computer Science · Engineering · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Sparse and Compressive Sensing Techniques
- Provably Correct Automatic Subdifferentiation for Qualified Programs
2018/09/23 by Sham M. Kakade, Jason D. Lee, Kakade, Sham +1 · 1 citation
Computer Science · #Advanced Multi-Objective Optimization Algorithms #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Numerical Methods and Algorithms #Optimization and Control (math.OC)
- Robust Meta-learning for Mixed Linear Regression with Small Batches
2020/06/17 by Weihao Kong, Raghav Somani, Kong, Weihao +5 · 2 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Machine Learning and ELM
- Loss-to-Loss Prediction: Scaling Laws for All Datasets
2024/11/19 by David Brandfonbrener, Brandfonbrener, David, Nikhil Anand +7 · 3 citations
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #Bayesian Modeling and Causal Inference #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Healthcare #Statistical Methods and Inference
- Provable Representation Learning for Imitation Learning via Bi-level Optimization
2020/02/24 by Sanjeev Arora, Simon S. Du, Arora, Sanjeev +7 · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics
- Meta-learning for mixed linear regression
2020/02/20 by Kong, Weihao, Somani, Raghav, Song, Zhao +2 · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)
- Leverage Score Sampling for Faster Accelerated Regression and ERM
2017/11/22 by Naman Agarwal, Sham M. Kakade, Agarwal, Naman +9 · 2 citations
Computer Science · Engineering · #Stochastic Gradient Optimization Techniques #Sparse and Compressive Sensing Techniques #Matrix Theory and Algorithms
- Recurrent Convolutional Neural Networks Learn Succinct Learning Algorithms
2022/09/01 by Goel, Surbhi, Kakade, Sham, Kalai, Adam Tauman +1 · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)
- Fine-Tuning Masked Diffusion for Provable Self-Correction
2025/10/01 by Kim, Jaeyeon, Kim, Seunggeun, Lee, Taekyun +4 · 5 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG)
- Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning
2023/06/14 by Nikhil Vyas, Depen Morwani, Vyas, Nikhil +9 · 1 citation
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications
- Soup to go: mitigating forgetting during continual learning with model averaging
2025/01/09 by Anat Kleiman, Kleiman, Anat, Gintare Karolina Dziugaite +7 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Intelligent Tutoring Systems and Adaptive Learning #Machine Learning (cs.LG)
- Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luck
2023/09/07 by Benjamin Edelman, Edelman, Benjamin L., Surbhi Goel +7 · 1 citation
Computer Science · #Domain Adaptation and Few-Shot Learning #Neural Networks and Applications #Machine Learning and Algorithms
- (weak) Calibration is Computationally Hard
2012/02/20 by Elad Hazan, Hazan, Elad, Sham M. Kakade +1 · 1 citation
Decision Sciences · Economics, Econometrics and Finance · #Auction Theory and Applications #Economic theories and models #Game Theory and Applications
- The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton
2025/10/10 by Natalie Abreu, Abreu, Natalie, Nikhil Vyas +5 · 3 citations
Physics and Astronomy · #Magnetic confinement fusion research
- Selective Underfitting in Diffusion Models
2025/10/01 by Ki‐Whan Song, J.M. Kim, Song, Kiwhan +9 · 5 citations
Computer Science · Physics and Astronomy · Mathematics · #Generative Adversarial Networks and Image Synthesis #Model Reduction and Neural Networks #Markov Chains and Monte Carlo Methods
- EvoLM: In Search of Lost Language Model Training Dynamics
2025/06/19 by Zhenting Qi, Qi, Zhenting, Fan Nie +15 · 1 voice · 1 citation
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL #cs.LG
- Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling
2025/10/16 by Alexandru Meterez, Meterez, Alexandru, Depen Morwani +9 · 1 citation
#cs.LG #cs.AI #math.OC #stat.ML