LLM4ES: Learning User Embeddings from Event Sequences via Large Language Models

2025/08/06 by Aleksei Shestov, Shestov, Aleksei, Omar Zoloev +11
Computer Science · Decision Sciences · #Data Quality and Management #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Recommender Systems and Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2508.05688

openalex publication_date 2025/08/06 · openalex created_date 2025/10/15 · openalex updated_date 2026/07/28

Abstract

This paper presents LLM4ES, a novel framework that exploits large pre-trained language models (LLMs) to derive user embeddings from event sequences. Event sequences are transformed into a textual representation, which is subsequently used to fine-tune an LLM through next-token prediction to generate high-quality embeddings. We introduce a text enrichment technique that enhances LLM adaptation to event sequence data, improving representation quality for low-variability domains. Experimental results demonstrate that LLM4ES achieves state-of-the-art performance in user classification tasks in financial and other domains, outperforming existing embedding methods. The resulting user embeddings can be incorporated into a wide range of applications, from user segmentation in finance to patient outcome prediction in healthcare.

Citations

• Pooling And Attention: What Are Effective Designs For LLM-Based Embedding Models?
• Detecting the Future: All-at-Once Event Sequence Forecasting with Horizon Matching
• The Llama 3 Herd of Models
• LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
• Harnessing Large Language Models for Text-Rich Sequential Recommendation
• TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series
• Heterogeneous Knowledge Fusion: A Novel Approach for Personalized Recommendation via LLM
• simpleKT: A Simple But Tough-to-Beat Baseline for Knowledge Tracing
• Intensity-Free Learning of Temporal Point Processes
• Representation Learning with Contrastive Predictive Coding
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• LATTE: Learning Aligned Transactions and Textual Embeddings for Bank Clients
• User-LLM: Efficient LLM Contextualization with User Embeddings
• Encode Me If You Can: Learning Universal User Representations via Event Sequence Autoencoding
• Harnessing Generative LLMs for Enhanced Financial Event Entity Extraction Performance
• LLMs are Also Effective Embedding Models: An In-depth Overview