Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data

2024/11/11 by Kai Kim, Kim, Kai, Rajat Sen +11 · 12 citations
Computer Science · #Advanced Text Analysis Techniques

paper · pdf · doi:10.48550/arxiv.2411.06735

Abstract

Current forecasting approaches are largely unimodal and ignore the rich textual data that often accompany the time series due to lack of well-curated multimodal benchmark dataset. In this work, we develop TimeText Corpus (TTC), a carefully curated, time-aligned text and time dataset for multimodal forecasting. Our dataset is composed of sequences of numbers and text aligned to timestamps, and includes data from two different domains: climate science and healthcare. Our data is a significant contribution to the rare selection of available multimodal datasets. We also propose the Hybrid Multi-Modal Forecaster (Hybrid-MMF), a multimodal LLM that jointly forecasts both text and time series data using shared embeddings. However, contrary to our expectations, our Hybrid-MMF model does not outperform existing baselines in our experiments. This negative result highlights the challenges inherent in multimodal forecasting. Our code and data are available at https://github.com/Rose-STL-Lab/Multimodal_ Forecasting.

Cited by

• LLM-Integrated Bayesian State Space Models for Multimodal Time-Series Forecasting
• TsLLM: Augmenting LLMs for General Time Series Understanding and Prediction
• AXIS: Explainable Time Series Anomaly Detection with Large Language Models
• CaTS-Bench: Can Language Models Describe Numeric Time Series?
• Multimodal Language Models with Modality-Specific Experts for Financial Forecasting from Interleaved Sequences of Text and Time Series
• Text Reinforcement for Multimodal Time Series Forecasting
• PA-RNet: Perturbation-Aware Reasoning Network for Multimodal Time Series Forecasting
• XFMNet: Decoding Cross-Site and Nonstationary Water Patterns via Stepwise Multimodal Fusion for Long-Term Water Quality Forecasting
• Diffusion Models for Time Series Forecasting: A Survey
• When Does Multimodality Lead to Better Time Series Forecasting?
• TimeMaster: Training Time-Series Multimodal LLMs to Reason via Reinforcement Learning
• Time-IMM: A Dataset and Benchmark for Irregular Multimodal Multivariate Time Series

Related

• When Does Multimodality Lead to Better Time Series Forecasting?
• Dual-Forecaster: A Multimodal Time Series Model Integrating Descriptive and Predictive Texts
• MoTime: A Dataset Suite for Multimodal Time Series Forecasting
• Unveiling the Potential of Text in High-Dimensional Time Series Forecasting
• FiCoTS: Fine-to-Coarse LLM-Enhanced Hierarchical Cross-Modality Interaction for Time Series Forecasting