Tricks and Plug-ins for Gradient Boosting with Transformers

2025/08/04 by Fang, Biyi, Vo, Truong, Utke, Jean +1
#68Q32 #68T07 #F.1.1 #FOS: Computer and information sciences #I.2.6 #I.5.1 #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2508.02924

Abstract

Transformer architectures dominate modern NLP but often demand heavy computational resources and intricate hyperparameter tuning. To mitigate these challenges, we propose a novel framework, BoostTransformer, that augments transformers with boosting principles through subgrid token selection and importance-weighted sampling. Our method incorporates a least square boosting objective directly into the transformer pipeline, enabling more efficient training and improved performance. Across multiple fine-grained text classification benchmarks, BoostTransformer demonstrates both faster convergence and higher accuracy, surpassing standard transformers while minimizing architectural search overhead.

Citations

• AutoFormer: Searching Transformers for Visual Recognition
• Language Models are Few-Shot Learners
• Transformer-Transducer: End-to-End Speech Recognition with Self-Attention
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Fine-grained Sentiment Classification using BERT
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• What Does BERT Look At? An Analysis of BERT's Attention
• Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese
• Not All Samples Are Created Equal: Deep Learning with Importance Sampling
• Hybrid Gradient Boosting Trees and Neural Networks for Forecasting Operating Room Data
• Attention Is All You Need
• Importance Sampling for Minibatches
• Variance Reduction in SGD by Distributed Importance Sampling
• Sequence to Sequence Learning with Neural Networks
• Neural Machine Translation by Jointly Learning to Align and Translate
• Stochastic Optimization with Importance Sampling
• Stochastic Gradient Descent, Weighted Sampling, and the Randomized\n Kaczmarz algorithm
• A theory of multiclass boosting
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost
• DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorization
• AutoTrans: Automating Transformer Design via Reinforced Architecture Search
• ETC: Encoding Long and Structured Inputs in Transformers
• Enhanced Transformer Architecture for Natural Language Processing