Attend and Diagnose: Clinical Time Series Analysis using Attention Models

2017/11/10 by Huan Song, Song, Huan, Deepta Rajan +5 · 9 citations
Computer Science · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Healthcare #Time Series Analysis and Forecasting #Topic Modeling #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1711.03905

AAAI 2018

openalex publication_date 2017/11/10 · openalex created_date 2017/11/17 · arxiv created 2017/11/19 · arxiv updated 2018/07/17 · openalex updated_date 2026/07/28

Abstract

With widespread adoption of electronic health records, there is an increased emphasis for predictive models that can effectively deal with clinical time-series data. Powered by Recurrent Neural Network (RNN) architectures with Long Short-Term Memory (LSTM) units, deep neural networks have achieved state-of-the-art results in several clinical prediction tasks. Despite the success of RNNs, its sequential nature prohibits parallelized computing, thus making it inefficient particularly when processing long sequences. Recently, architectures which are based solely on attention mechanisms have shown remarkable success in transduction tasks in NLP, while being computationally superior. In this paper, for the first time, we utilize attention models for clinical time-series modeling, thereby dispensing recurrence entirely. We develop the SAnD (Simply Attend and Diagnose) architecture, which employs a masked, self-attention mechanism, and uses positional encoding and dense interpolation strategies for incorporating temporal order. Furthermore, we develop a multi-task variant of SAnD to jointly infer models with multiple diagnosis tasks. Using the recent MIMIC-III benchmark datasets, we demonstrate that the proposed approach achieves state-of-the-art performance in all tasks, outperforming LSTM models and classical baselines with hand-engineered features.

Citations

• Modeling Missing Data in Clinical Time Series with RNNs
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• A Deep Reinforced Model for Abstractive Summarization
• Convolutional Neural Networks for Sentence Classification
• Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks

Cited by

• Transparent Early ICU Mortality Prediction with Clinical Transformer and Per-Case Modality Attribution
• A Prototypical Network with an Attention-based Encoder for Drivers Identification Application
• Multimodal Disease Progression Modeling via Spatiotemporal Disentanglement and Multiscale Alignment
• BladderFormer: A Streaming Transformer for Real-Time Urological State Monitoring
• HyMaTE: A Hybrid Mamba and Transformer Model for EHR Representation Learning
• Bridging Graph and State-Space Modeling for Intensive Care Unit Length of Stay Prediction
• Fre-CW: Targeted Attack on Time Series Forecasting using Frequency Domain Loss
• FISformer: Replacing Self-Attention with a Fuzzy Inference System in Transformer Models for Time Series Forecasting
• DeepSelective: Interpretable Prognosis Prediction via Feature Selection and Compression in EHR Data

Related

• Multi-Time Attention Networks for Irregularly Sampled Time Series
• Learning to Diagnose with LSTM Recurrent Neural Networks
• No Black Box Anymore: Demystifying Clinical Predictive Modeling with Temporal-Feature Cross Attention Mechanism
• Clinical NLP with Attention-Based Deep Learning for Multi-Disease Prediction
• Position-based Content Attention for Time Series Forecasting with Sequence-to-sequence RNNs