QKCV Attention: Enhancing Time Series Forecasting with Static Categorical Embeddings for Both Lightweight and Pre-trained Foundation Models

2025/10/21 by Wang, Hao, Ma, Baojun
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2510.20222

Abstract

In real-world time series forecasting tasks, category information plays a pivotal role in capturing inherent data patterns. This paper introduces QKCV (Query-Key-Category-Value) attention, an extension of the traditional QKV framework that incorporates a static categorical embedding C to emphasize category-specific information. As a versatile plug-in module, QKCV enhances the forecasting accuracy of attention-based models (e.g., Vanilla Transformer, Informer, PatchTST, TFT) across diverse real-world datasets. Furthermore, QKCV demonstrates remarkable adaptability in fine-tuning univariate time series foundation model by solely updating the static embedding C while preserving pretrained weights, thereby reducing computational overhead and achieving superior fine-tuning performance.

Citations

• Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapters
• Considering Nonstationary within Multivariate Time Series with Variational Hierarchical Transformer for Forecasting
• VeRA: Vector-based Random Matrix Adaptation
• A decoder-only foundation model for time-series forecasting
• AutoGluon-TimeSeries: AutoML for Probabilistic Time Series Forecasting
• QLoRA: Efficient Finetuning of Quantized LLMs
• LLaMA: Open and Efficient Foundation Language Models
• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers
• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
• Training language models to follow instructions with human feedback
• A review on the attention mechanism of deep learning
• GLM: General Language Model Pretraining with Autoregressive Blank Infilling
• Transformer in Transformer
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• Temporal Fusion Transformers for Interpretable Multi-horizon Time Series\n Forecasting
• Temporal Fusion Transformers for interpretable multi-horizon time series forecasting
• Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
• N-BEATS: Neural basis expansion analysis for interpretable time series forecasting
• Attention Is All You Need
• DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks
• Long Short-Term Memory
• A Learning Algorithm for Continually Running Fully Recurrent Neural Networks

Related

• TS-RAG: Retrieval-Augmented Generation based Time Series Foundation Models are Stronger Zero-Shot Forecaster
• Attention as Robust Representation for Time Series Forecasting
• Revisiting Attention for Multivariate Time Series Forecasting
• Output Scaling: YingLong-Delayed Chain of Thought in a Large Pretrained Time Series Forecasting Model
• A decoder-only foundation model for time-series forecasting