Bootstrapping your behavior: a new pretraining strategy for user behavior sequence data

2025/05/22 by Weichang Wu, Xiaolu Zhang, Wu, Weichang +7
Social Sciences · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Human Mobility and Location-Based Analysis #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2506.11053

openalex publication_date 2025/05/22 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

User Behavior Sequence (UBS) modeling is crucial in industrial applications. As data scale and task diversity grow, UBS pretraining methods have become increasingly pivotal. State-of-the-art UBS pretraining methods rely on predicting behavior distributions. The key step in these methods is constructing a selected behavior vocabulary. However, this manual step is labor-intensive and prone to bias. The limitation of vocabulary capacity also directly affects models' generalization ability. In this paper, we introduce Bootstrapping Your Behavior (\model), a novel UBS pretraining strategy that predicts an automatically constructed supervision embedding summarizing all behaviors' information within a future time window, eliminating the manual behavior vocabulary selection. In implementation, we incorporate a student-teacher encoder scheme to construct the pretraining supervision effectively. Experiments on two real-world industrial datasets and eight downstream tasks demonstrate that \model achieves an average improvement of 3.9% in AUC and 98.9% in training throughput. Notably, the model exhibits meaningful attention patterns and cluster representations during pretraining without any label supervision. In our online deployment over two months, the pretrained model improves the KS by about 2.7% and 7.1% over the baseline model for two financial overdue risk prediction tasks in the Alipay mobile application, which reduces bad debt risk by millions of dollars for Ant group.

Citations

• DINOv2: Learning Robust Visual Features without Supervision
• Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
• VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• InfoBehavior: Self-supervised Representation Learning for Ultra-long Behavior Sequence via Hierarchical Grouping
• Emerging Properties in Self-Supervised Vision Transformers
• PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Bootstrap your own latent: A new approach to self-supervised Learning
• Language Models are Few-Shot Learners
• Improved Baselines with Momentum Contrastive Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Behavior Sequence Transformer for E-commerce Recommendation in Alibaba
• Deep Clustering for Unsupervised Learning of Visual Features
• Attention Is All You Need
• Asynchronous Methods for Deep Reinforcement Learning
• Large Scale Purchase Prediction with Historical User Actions on B2C Online Retail Platform

Related

• PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision
• BehaveGPT: A Foundation Model for Large-scale User Behavior Modeling
• UserBERT: Contrastive User Model Pre-training
• UserBERT: Modeling Long- and Short-Term User Preferences via Self-Supervision
• Tuning Language Models for Robust Prediction of Diverse User Behaviors