When Alpha Disappears: A One-Switch Benchmark for Decision-Time Leakage in Financial Backtests

2026/05/12 by Fan Zhang, Zhen Li, Sijia Peng +1 · 1 voice
Economics, Econometrics and Finance · #q-fin.RM #q-fin.TR

paper · pdf

Abstract

We introduce When Alpha Disappears, a paired evaluation benchmark for diagnosing decision-time leakage in financial machine-learning backtests. Rather than treating leakage as a binary property, the benchmark estimates protocol-induced inflation by toggling one evaluation convention at a time around a clean t+1-open reference, while holding the data panel, walk-forward split, model family, horizon, portfolio rule, and cost convention fixed. Across two daily-OHLCV equity panels, six model families, and yearly tests from 2016--2024, we find that inflation is highly selective: centered temporal features and same-day-open execution with post-open daily-bar information cause large and stable increases in both predictive and trading metrics, whereas global normalization, future-informed graph structure, and same-day-close execution are weak in most settings. The benchmark is diagnostic rather than a claim of tradable alpha, and is intended to make evaluation assumptions, failure modes, and protocol fragility directly measurable.

Citations

• Spurious Predictability in Financial Machine Learning
• Look-Ahead-Bench: a Standardized Benchmark of Look-ahead Bias in Point-in-Time LLMs for Finance
• Detecting Lookahead Bias in LLM Forecasts
• Future Is Unevenly Distributed: Forecasting Ability of LLMs Depends on What We're Asking
• Profit Mirage: Revisiting Information Leakage in LLM-based Financial Agents
• Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking
• The Memorization Problem: Can We Trust LLMs' Economic Forecasts?
• The Pitfalls of Memorization: When Memorization Hurts Generalization
• Impact of Leakage on Data Harmonization in Machine Learning Pipelines in Class Imbalance Across Sites
• On Leakage in Machine Learning Pipelines
• Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysis
• Asset Pricing and Machine Learning: A critical review
• Qlib: An AI-oriented Quantitative Investment Platform
• Avoiding Backtesting Overfitting by Covariance-Penalties: an empirical investigation of the ordinary and total least squares cases
• Look-Ahead Benchmark Bias in Portfolio Performance Evaluation
• The Statistics of Sharpe Ratios

Discussions

• Fan Zhang, Zhen Li, Sijia Peng, Yu Chen: When Alpha Disappears: A One-Switch Benchmark for Decision-Time Leakage in Financial Backtests https://arxiv.org/abs/2605.23959 https://arxiv.org/pdf/2605.2395 [bsky, 0 points, 0 comments]

Related

• Leakage-Aware Benchmarking of LLM Forecasting: Real-Time Nowcasts as the Decision-Time Input for Macro Factor Ranking
• Predictive Extrema, Unprofitable Policies: An AI-Assisted Audit of Candle-Based Binance Spot Timing Models
• FinStressTS: A Parametric Synthetic Benchmark for Time-Series Forecasting in Finance
• AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models
• Increase Alpha: Performance and Risk of an AI-Driven Trading Framework