Transforming Causality: Transformer-Based Temporal Causal Discovery with Prior Knowledge Integration

2025/08/21 by Huang, Jihua, Yao, Yi, Divakaran, Ajay
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2508.15928

Abstract

We introduce a novel framework for temporal causal discovery and inference that addresses two key challenges: complex nonlinear dependencies and spurious correlations. Our approach employs a multi-layer Transformer-based time-series forecaster to capture long-range, nonlinear temporal relationships among variables. After training, we extract the underlying causal structure and associated time lags from the forecaster using gradient-based analysis, enabling the construction of a causal graph. To mitigate the impact of spurious causal relationships, we introduce a prior knowledge integration mechanism based on attention masking, which consistently enforces user-excluded causal links across multiple Transformer layers. Extensive experiments show that our method significantly outperforms other state-of-the-art approaches, achieving a 12.8% improvement in F1-score for causal discovery and 98.9% accuracy in estimating causal lags.

Citations

• CausalFormer: An Interpretable Transformer for Temporal Causal Discovery
• ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling
• Dynamic Causal Explanation Based Diffusion-Variational Graph Neural Network for Spatio-temporal Forecasting
• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers
• Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• High-recall causal discovery for autocorrelated time series with latent confounders
• DYNOTEARS: Structure Learning from Time-Series Data
• DAGs with NO TEARS: Continuous Optimization for Structure Learning
• Kernel Method for Nonlinear Granger Causality

Related

• CausalFormer: An Interpretable Transformer for Temporal Causal Discovery
• TC-GAT: Graph Attention Network for Temporal Causality Discovery
• Transformers with Sparse Attention for Granger Causality
• Causal Discovery with Inverted Self-attention for Multivariate Time Series
• Causal Discovery in Multivariate Time Series through Mutual Information Featurization