Learning to Embed Time Series Patches Independently

2023/12/27 by Lee, Seunghan, Park, Taeyoung, Lee, Kibok · 8 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2312.16427

Abstract

Masked time series modeling has recently gained much attention as a self-supervised representation learning strategy for time series. Inspired by masked image modeling in computer vision, recent works first patchify and partially mask out time series, and then train Transformers to capture the dependencies between patches by predicting masked patches from unmasked patches. However, we argue that capturing such patch dependencies might not be an optimal strategy for time series representation learning; rather, learning to embed patches independently results in better time series representations. Specifically, we propose to use 1) the simple patch reconstruction task, which autoencode each patch without looking at other patches, and 2) the simple patch-wise MLP that embeds each patch independently. In addition, we introduce complementary contrastive learning to hierarchically capture adjacent time series information efficiently. Our proposed method improves time series forecasting and classification performance compared to state-of-the-art Transformer-based models, while it is more efficient in terms of the number of parameters and training/inference time. Code is available at this repository: https://github.com/seunghan96/pits.

Cited by

• TimePerceiver: An Encoder-Decoder Framework for Generalized Time-Series Forecasting
• TwinFormer: A Dual-Level Transformer for Long-Sequence Time-Series Forecasting
• Lightweight Foundation Model for Wireless Time Series Downstream Tasks on Edge Devices
• PhaseFormer: From Patches to Phases for Efficient and Effective Time Series Forecasting
• Learning More with Less: A Generalizable, Self-Supervised Framework for Privacy-Preserving Capacity Estimation with EV Charging Data
• EntroPE: Entropy-Guided Dynamic Patch Encoder for Time Series Forecasting
• Why Attention Fails: The Degeneration of Transformers into MLPs in Time Series Forecasting
• DeCoP: Enhancing Self-Supervised Time Series Representation with Dependency Controlled Pre-training

Related

• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers
• SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling
• MTSMAE: Masked Autoencoders for Multivariate Time-Series Forecasting
• Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning
• Ti-MAE: Self-Supervised Masked Time Series Autoencoders