Scalable Linear Causal Inference for Irregularly Sampled Time Series with Long Range Dependencies

2016/03/10 by Francois Belletti, Francois W. Belletti, Belletti, Francois W. +9
Computer Science · Economics, Econometrics and Finance · Mathematics · Neuroscience · #Blind Source Separation Techniques #Complex Systems and Time Series Analysis #FOS: Computer and information sciences #Functional Brain Connectivity Studies #Machine Learning (cs.LG) #Methodology (stat.ME) #Neural Networks and Applications #cs.LG #stat.ME

paper · pdf · doi:10.48550/arxiv.1603.03336

arxiv created 2016/03/10 · openalex publication_date 2016/03/10 · arxiv updated 2016/03/11 · openalex created_date 2022/09/30 · openalex updated_date 2026/07/28

Abstract

Linear causal analysis is central to a wide range of important application spanning finance, the physical sciences, and engineering. Much of the existing literature in linear causal analysis operates in the time domain. Unfortunately, the direct application of time domain linear causal analysis to many real-world time series presents three critical challenges: irregular temporal sampling, long range dependencies, and scale. Moreover, real-world data is often collected at irregular time intervals across vast arrays of decentralized sensors and with long range dependencies which make naive time domain correlation estimators spurious. In this paper we present a frequency domain based estimation framework which naturally handles irregularly sampled data and long range dependencies while enabled memory and communication efficient distributed processing of time series data. By operating in the frequency domain we eliminate the need to interpolate and help mitigate the effects of long range dependencies. We implement and evaluate our new work-flow in the distributed setting using Apache Spark and demonstrate on both Monte Carlo simulations and high-frequency financial trading that we can accurately recover causal structure at scale.

Citations

• Least-squares frequency analysis of unequally spaced data
• Studies in astronomical time series analysis. II - Statistical aspects of spectral analysis of unevenly spaced data
• Investigating Causal Relations by Econometric Models and Cross-spectral Methods
• Embarrassingly Parallel Time Series Analysis for Large Scale Weak Memory Systems

Related

• Causal Discovery on Irregular Time Series
• Robust Causal Discovery in Real-World Time Series with Power-Laws
• Causal Discovery for Irregularly Time Series with Consistency Guarantees
• Learning latent causal relationships in multiple time series
• Causal Discovery from Subsampled Time Series Data by Constraint Optimization