Causally-Guided Pairwise Transformer -- Towards Foundational Digital Twins in Process Industry

2025/08/18 by Michael Mayr, Mayr, Michael, Georgios C. Chasparis +1 · 1 voice
Computer Science · Engineering · Materials Science · #Digital Transformation in Industry #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Materials Science #cs.LG

paper · pdf · doi:10.48550/arxiv.2508.13111

openalex publication_date 2025/08/18 · arxiv published 2025/08/18 · arxiv updated 2025/09/22 · openalex created_date 2025/10/16 · openalex updated_date 2026/07/28

Abstract

Foundational modelling of multi-dimensional time-series data in industrial systems presents a central trade-off: channel-dependent (CD) models capture specific cross-variable dynamics but lack robustness and adaptability as model layers are commonly bound to the data dimensionality of the tackled use-case, while channel-independent (CI) models offer generality at the cost of modelling the explicit interactions crucial for system-level predictive regression tasks. To resolve this, we propose the Causally-Guided Pairwise Transformer (CGPT), a novel architecture that integrates a known causal graph as an inductive bias. The core of CGPT is built around a pairwise modeling paradigm, tackling the CD/CI conflict by decomposing the multidimensional data into pairs. The model uses channel-agnostic learnable layers where all parameter dimensions are independent of the number of variables. CGPT enforces a CD information flow at the pair-level and CI-like generalization across pairs. This approach disentangles complex system dynamics and results in a highly flexible architecture that ensures scalability and any-variate adaptability. We validate CGPT on a suite of synthetic and real-world industrial datasets on long-term and one-step forecasting tasks designed to simulate common industrial complexities. Results demonstrate that CGPT significantly outperforms both CI and CD baselines in predictive accuracy and shows competitive performance with end-to-end trained CD models while remaining agnostic to the problem dimensionality.

Citations

• Causal Time-Series Synchronization for Multi-Dimensional Forecasting
• Learning Paradigms and Modelling Methodologies for Digital Twins in Process Industry
• Unified Training of Universal Time Series Forecasting Transformers
• Timer: Generative Pre-trained Transformers Are Large Time Series Models
• Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series
• A decoder-only foundation model for time-series forecasting
• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers
• Are Transformers Effective for Time Series Forecasting?
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Digital Twin: Values, Challenges and Enablers
• Digital Twin: Values, Challenges and Enablers From a Modeling Perspective
• SGDR: Stochastic Gradient Descent with Warm Restarts

Discussions

• ..adaptable foundation for digital twins capable of learning and reasoning about complex industrial processes, helping industries improve efficiency and decision-making in an ever-changing environment [bsky, 0 points, 0 comments]

Related

• Causal Time-Series Synchronization for Multi-Dimensional Forecasting
• FaCTR: Factorized Channel-Temporal Representation Transformers for Efficient Time Series Forecasting
• Channel Dependence, Limited Lookback Windows, and the Simplicity of Datasets: How Biased is Time Series Forecasting?
• Transforming Causality: Transformer-Based Temporal Causal Discovery with Prior Knowledge Integration
• Beyond All-to-All: Causal-Aligned Transformer with Dynamic Structure Learning for Multivariate Time Series Forecasting