Kaiyue Wen
- Fantastic Pretraining Optimizers and Where to Find Them
2025/09/02 by Kaiyue Wen, David Hall, Wen, Kaiyue +5 · 5 voices · 26 citations
#cs.LG #cs.AI #stat.ML
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
2025/05/10 by Zihan Qiu, Qiu, Zihan, Zekun Wang +23 · 3 voices · 42 citations
Computer Science · #Topic Modeling #Machine Learning in Healthcare #Big Data and Digital Economy
- Overtrained Language Models Are Harder to Fine-Tune
2025/03/24 by Jacob Mitchell Springer, Jacob M. Springer, Springer, Jacob Mitchell +14 · 8 voices · 29 citations
Computer Science · #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL
- Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization
2023/07/20 by Kaiyue Wen, Zhiyuan Li, Wen, Kaiyue +3 · 12 citations
Computer Science · #Neural Networks and Applications #Stochastic Gradient Optimization Techniques #Machine Learning and ELM
- RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval
2024/02/28 by Kaiyue Wen, Xingyu Dang, Wen, Kaiyue +3 · 13 citations
Computer Science · #Topic Modeling
- How Does Sharpness-Aware Minimization Minimize Sharpness?
2022/11/10 by Kaiyue Wen, Tengyu Ma, Wen, Kaiyue +3 · 9 citations
Computer Science · #Stochastic Gradient Optimization Techniques #Domain Adaptation and Few-Shot Learning #Machine Learning and ELM
- Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective
2024/10/07 by Kaiyue Wen, Wen, Kaiyue, Zhiyuan Li +9 · 17 citations
Computer Science · #Machine Learning and Data Classification #Machine Learning and Algorithms #Anomaly Detection Techniques and Applications
- Finding Skill Neurons in Pre-trained Transformer-based Language Models
2022/11/14 by Xiaozhi Wang, Wang, Xiaozhi, Kaiyue Wen +9 · 7 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques
- From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency
2024/10/07 by Kaiyue Wen, Wen, Kaiyue, Huaqing Zhang +5 · 9 citations
Neuroscience · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural dynamics and brain function
- Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
2025/01/21 by Zihan Qiu, Zeyu Huang, Qiu, Zihan +17 · 10 citations
Decision Sciences · Computer Science · #Simulation Techniques and Applications #Scientific Computing and Data Management #AI-based Problem Solving and Planning
- Weight Ensembling Improves Reasoning in Language Models
2025/04/14 by Christina Baek, Dang, Xingyu, Baek, Christina +6 · 16 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling
- Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images
2025/02/19 by Shengguang Wu, Wu, Shengguang, Fan-Yun Sun +5 · 6 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image Retrieval and Classification Techniques #Machine Learning (cs.LG) #Multimodal Machine Learning Applications
- Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars
2023/12/03 by Kaiyue Wen, Wen, Kaiyue, Yuchen Li +5 · 1 citation
Computer Science · #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Topic Modeling