Enhancing Generative Auto-bidding with Offline Reward Evaluation and Policy Search

2025/09/19 by Zhiyu Mou, Yiqin Lv, Mou, Zhiyu +20 · 1 citation
Decision Sciences · #Artificial Intelligence (cs.AI) #Auction Theory and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2509.15927

openalex publication_date 2025/09/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Auto-bidding is a critical tool for advertisers to improve advertising performance. Recent progress has demonstrated that AI-Generated Bidding (AIGB), which learns a conditional generative planner from offline data, achieves superior performance compared to typical offline reinforcement learning (RL)-based auto-bidding methods. However, existing AIGB methods still face a performance bottleneck due to their inherent inability to explore beyond the static dataset with feedback. To address this, we propose AIGB-Pearl (\emphPlanning with EvaluAtor via RL), a novel method that integrates generative planning and policy optimization. The core of AIGB-Pearl lies in constructing a trajectory evaluator to assess the quality of generated scores and designing a provably sound KL-Lipschitz-constrained score-maximization scheme to ensure safe and efficient exploration beyond the offline dataset. A practical algorithm that incorporates the synchronous coupling technique is further developed to ensure the model regularity required by the proposed scheme. Extensive experiments on both simulated and real-world advertising systems demonstrate the state-of-the-art performance of our approach.

Citations

• Permutation Equivariant Model-based Offline Reinforcement Learning for Auto-bidding
• Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library
• Generative Models in Decision Making: A Survey
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• An Adaptable Budget Planner for Enhancing Budget-Constrained Auto-Bidding in Online Advertising
• Qwen2.5 Technical Report
• Doubly Mild Generalization for Offline Reinforcement Learning
• Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression
• HiBid: A Cross-Channel Constrained Bidding System with Budget Allocation by Hierarchical Offline Deep Reinforcement Learning
• Diffusion Models for Reinforcement Learning: A Survey
• Efficient Diffusion Policies for Offline Reinforcement Learning
• Training Diffusion Models with Reinforcement Learning
• Multi-channel Autobidding with Budget and ROI Constraints
• Better Training of GFlowNets with Local Credit and Incomplete Trajectories
• Is Conditional Generative Modeling all you need for Decision-Making?
• Sustainable Online Reinforcement Learning for Auto-bidding
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
• Classifier-Free Diffusion Guidance
• Robust Auction Design in the Auto-bidding World
• Offline Reinforcement Learning with Implicit Q-Learning
• Generative Adversarial Networks
• Multi-Agent Cooperative Bidding Games for Multi-Objective Optimization in e-Commercial Sponsored Search
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Towards Efficient Auctions in an Auto-bidding World
• Improved Denoising Diffusion Probabilistic Models
• Denoising Diffusion Probabilistic Models
• Conservative Q-Learning for Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• MOReL : Model-Based Offline Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Off-Policy Deep Reinforcement Learning without Exploration
• Deep Reinforcement Learning and the Deadly Triad
• Deep reinforcement learning from human preferences
• Attention Is All You Need
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Generalization of an Inequality by Talagrand and Links with the Logarithmic Sobolev Inequality
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS
• Reinforcement Learning: An Introduction

Cited by

• VAO: Validation-Aligned Optimization for Cross-Task Generative Auto-Bidding

Related

• Generative Auto-Bidding with Value-Guided Explorations
• AIGB-R1: Self-Evolving Generative Auto-Bidding via Hierarchical Planner-Executor Optimization
• AIGB: Generative Auto-bidding via Conditional Diffusion Modeling
• Trajectory-wise Iterative Reinforcement Learning Framework for Auto-bidding
• Sustainable Online Reinforcement Learning for Auto-bidding