A Reinforcement-Learning-Enhanced LLM Framework for Automated A/B Testing in Personalized Marketing

2025/05/27 by Haoyang Feng, Feng, Haoyang, Yanjun Dai +3 · 1 citation
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Digital Marketing and Social Media #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Mobile Crowdsensing and Crowdsourcing #Recommender Systems and Techniques

paper · pdf · doi:10.48550/arxiv.2506.06316

openalex publication_date 2025/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

For personalized marketing, a new challenge of how to effectively algorithm the A/B testing to maximize user response is urgently to be overcome. In this paper, we present a new approach, the RL-LLM-AB test framework, for using reinforcement learning strategy optimization combined with LLM to automate and personalize A/B tests. The RL-LLM-AB test is built upon the pre-trained instruction-tuned language model. It first generates A/B versions of candidate content variants using a Prompt-Conditioned Generator, and then dynamically embeds and fuses the user portrait and the context of the current query with the multi-modal perception module to constitute the current interaction state. The content version is then selected in real-time through the policy optimization module with an Actor-Critic structure, and long-term revenue is estimated according to real-time feedback (such as click-through rate and conversion rate). Furthermore, a Memory-Augmented Reward Estimator is embedded into the framework to capture long-term user preference drift, which helps to generalize policy across multiple users and content contexts. Numerical results demonstrate the superiority of our proposed RL-LLM-ABTest over existing A/B testing methods, including classical A/B testing, Contextual Bandits, and benchmark reinforcement learning approaches on real-world marketing data.

Citations

• Psychological Health Knowledge-Enhanced LLM-based Social Network Crisis Intervention Text Transfer Recognition Method
• Research on Personalized Medical Intervention Strategy Generation System based on Group Relative Policy Optimization and Time-Series Data Fusion
• LOLA: LLM-Assisted Online Learning Algorithm for Content Experiments
• A Two-Part Machine Learning Approach to Characterizing Network Interference in A/B Testing

Cited by

• Multimodal Health Risk Prediction System for Chronic Diseases via Vision-Language Fusion and Large Language Models

Related

• Techniques Toward Optimizing Viewability in RTB Ad Campaigns Using Reinforcement Learning
• AdCraft: An Advanced Reinforcement Learning Benchmark Environment for Search Engine Marketing Optimization
• An Extremely Data-efficient and Generative LLM-based Reinforcement Learning Agent for Recommenders
• Reinforcement Learning from Statistical Feedback: the Journey from AB Testing to ANT Testing
• Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shopping