Multi-Item-Query Attention for Stable Sequential Recommendation

2025/09/29 by Xu, Mingshi, Zhu, Haoren, Ng, Wilfred Siu Hung
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2509.24424

Abstract

The inherent instability and noise in user interaction data challenge sequential recommendation systems. Prevailing masked attention models, relying on a single query from the most recent item, are sensitive to this noise, reducing prediction reliability. We propose the Multi-Item-Query attention mechanism (MIQ-Attn) to enhance model stability and accuracy. MIQ-Attn constructs multiple diverse query vectors from user interactions, effectively mitigating noise and improving consistency. It is designed for easy adoption as a drop-in replacement for existing single-query attention. Experiments show MIQ-Attn significantly improves performance on benchmark datasets.

Citations

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• Improving Sequential Recommendation Models with an Enhanced Loss Function
• A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation
• Effective and Efficient Training for Sequential Recommendation using Recency Sampling
• Modeling Sequences as Distributions with Uncertainty for Sequential Recommendation
• Efficient Large-Scale Language Model Training on GPU Clusters Using\n Megatron-LM
• Contrastive Learning for Sequential Recommendation
• Fast Transformer Decoding: One Write-Head is All You Need
• BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
• Self-Attentive Sequential Recommendation
• Next Item Recommendation with Self-Attention
• Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding
• Attention Is All You Need
• Joint Deep Modeling of Users and Items Using Reviews for Recommendation
• Session-based Recommendations with Recurrent Neural Networks
• Image-based Recommendations on Styles and Substitutes
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Sequential Recommendation via Adaptive Robust Attention with Multi-dimensional Embeddings
• Improving Sequential Query Recommendation with Immediate User Feedback
• Recursive Attentive Methods with Reused Item Representations for Sequential Recommendation
• Multi-Perspective Attention Mechanism for Bias-Aware Sequential Recommendation
• Attention-based sequential recommendation system using multimodal data