A Transformer-based Framework for Multivariate Time Series\n Representation Learning

2020/10/06 by George Zerveas, Zerveas, George, Srideepika Jayaraman +7 · 46 citations
Computer Science · #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #Data Stream Mining Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Time Series Analysis and Forecasting

paper · pdf · doi:10.48550/arxiv.2010.02803

openalex publication_date 2020/10/06 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

In this work we propose for the first time a transformer-based framework for\nunsupervised representation learning of multivariate time series. Pre-trained\nmodels can be potentially used for downstream tasks such as regression and\nclassification, forecasting and missing value imputation. By evaluating our\nmodels on several benchmark datasets for multivariate time series regression\nand classification, we show that not only does our modeling approach represent\nthe most successful method employing unsupervised learning of multivariate time\nseries presented to date, but also that it exceeds the current state-of-the-art\nperformance of supervised methods; it does so even when the number of training\nsamples is very limited, while offering computational efficiency. Finally, we\ndemonstrate that unsupervised pre-training of our transformer models offers a\nsubstantial performance benefit over fully supervised learning, even without\nleveraging additional unlabeled data, i.e., by reusing the same data samples\nthrough the unsupervised objective.\n

Cited by

• SANet: A Semantic-aware Agentic AI Networking Framework for Cross-layer Optimization in 6G
• GraviBERT: Transformer-based inference for gravitational-wave time series
• DETACH : Decomposed Spatio-Temporal Alignment for Exocentric Video and Ambient Sensors with Staged Learning
• Chorus: Harmonizing Context and Sensing Signals for Data-Free Model Customization in IoT
• SDE-Attention: Latent Attention in SDE-RNNs for Irregularly Sampled Time Series with Missing Data
• Mechanistic Interpretability for Transformer-based Time Series Classification
• Unmasking Airborne Threats: Guided-Transformers for Portable Aerosol Mass Spectrometry
• Global Cross-Time Attention Fusion for Enhanced Solar Flare Prediction from Multivariate Time Series
• A Method for Characterizing Disease Progression from Acute Kidney Injury to Chronic Kidney Disease
• CURENet: Combining Unified Representations for Efficient Chronic Disease Prediction
• TransactionGPT
• Synheart Emotion: Privacy-Preserving On-Device Emotion Recognition from Biosignals
• Predicting Grain Growth in Polycrystalline Materials Using Deep Learning Time Series Models
• Frequency Matters: When Time Series Foundation Models Fail Under Spectral Shift
• Recurrence Plot and Change Quantile-based Deep Supervised and Semi-supervised Protection for Transmission Lines Connected to Photovoltaic Plants
• A Dual-Use Framework for Clinical Gait Analysis: Attention-Based Sensor Optimization and Automated Dataset Auditing
• Hydra: Dual Exponentiated Memory for Multivariate Time Series Analysis
• TS-Reasoner: Aligning Time Series Foundation Models with LLM Reasoning
• LLMComp: A Language Modeling Paradigm for Error-Bounded Scientific Data Compression (Technical Report)
• Beyond Hearing: Learning Task-Agnostic ExG Representations from Earphones via Physiology-Informed Tokenization
• Fighter: Unveiling the Graph Convolutional Nature of Transformers in Time Series Modeling
• Robotic Classification of Divers' Swimming States using Visual Pose Keypoints as IMUs
• Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challenges
• Cross-dataset Multivariate Time-series Model for Parkinson's Diagnosis via Keyboard Dynamics
• STaTS: Structure-Aware Temporal Sequence Summarization via Statistical Window Merging
• Learning Time-Series Representations by Hierarchical Uniformity-Tolerance Latent Balancing
• A Diffusion Model for Regular Time Series Generation from Irregular Data with Completion and Masking
• Training-Free Time Series Classification via In-Context Reasoning with LLM Agents
• Learning More with Less: A Generalizable, Self-Supervised Framework for Privacy-Preserving Capacity Estimation with EV Charging Data
• Cross-Modal Reconstruction Pretraining for Ramp Flow Prediction at Highway Interchanges
• TSKAN: Interpretable Machine Learning for QoE modeling over Time Series Data
• HealthCAT: An Interpretable Encoder-only Transformer Framework for Health Indicator Prediction and Temporal Interpretation of Wearable Sensor Data
• Rethinking EEG-Based Disease Diagnosis: Decoupling Instance Representation Learning from Subject-Level Supervision
• Conv-like Scale-Fusion Time Series Transformer: A Multi-Scale Representation for Variable-Length Long Time Series
• MTS-DMAE: Dual-Masked Autoencoder for Unsupervised Multivariate Time Series Representation Learning
• DeCoP: Enhancing Self-Supervised Time Series Representation with Dependency Controlled Pre-training
• DyWPE: Signal-Aware Dynamic Wavelet Positional Encoding for Time Series Transformers
• MIS-LSTM: Multichannel Image-Sequence LSTM for Sleep Quality and Stress Prediction
• PLanTS: Periodicity-aware Latent-state Representation Learning for Multivariate Time Series
• Decomposing and Revising What Language Models Generate
• Stage-Diff: Stage-wise Long-Term Time Series Generation Based on Diffusion Models
• A Learning-based Hybrid System Approach for Detecting Contingencies in Distribution Grids with Inverter-Based Resources
• Weather-Driven Agricultural Decision-Making Using Digital Twins Under Imperfect Conditions
• EnergyPatchTST: Multi-scale Time Series Transformers with Uncertainty Estimation for Energy Forecasting
• User Trajectory Prediction Unifying Global and Local Temporal Information
• A Dynamic Allocation Scheme for Adaptive Shared-Memory Mapping on Kilo-core RV Clusters for Attention-Based Model Deployment

Related

• CaSS: A Channel-aware Self-supervised Representation Learning Framework for Multivariate Time Series Classification
• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers
• Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer
• VSFormer: Value and Shape-Aware Transformer with Prior-Enhanced Self-Attention for Multivariate Time Series Classification
• Unsupervised Representation Learning for Time Series: A Review