Foundation Model Hidden Representations for Heart Rate Estimation from Auscultation

2025/05/27 by Jingping Nie, Dũng Trần Trung, Nie, Jingping +13 · 1 citation
Medicine · #Audio and Speech Processing (eess.AS) #ECG Monitoring and Analysis #FOS: Computer and information sciences #FOS: Electrical engineering #Healthcare Technology and Patient Monitoring #Machine Learning (cs.LG) #Phonocardiography and Auscultation Techniques #Sound (cs.SD) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2505.20745

openalex publication_date 2025/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Auscultation, particularly heart sound, is a non-invasive technique that provides essential vital sign information. Recently, self-supervised acoustic representation foundation models (FMs) have been proposed to offer insights into acoustics-based vital signs. However, there has been little exploration of the extent to which auscultation is encoded in these pre-trained FM representations. In this work, using a publicly available phonocardiogram (PCG) dataset and a heart rate (HR) estimation model, we conduct a layer-wise investigation of six acoustic representation FMs: HuBERT, wav2vec2, wavLM, Whisper, Contrastive Language-Audio Pretraining (CLAP), and an in-house CLAP model. Additionally, we implement the baseline method from Nie et al., 2024 (which relies on acoustic features) and show that overall, representation vectors from pre-trained foundation models (FMs) offer comparable performance to the baseline. Notably, HR estimation using the representations from the audio encoder of the in-house CLAP model outperforms the results obtained from the baseline, achieving a lower mean absolute error (MAE) across various train/validation/test splits despite the domain mismatch.

Citations

• Model-driven Heart Rate Estimation and Heart Murmur Detection based on Phonocardiogram
• Pre-Trained Foundation Model representations to uncover Breathing patterns in Speech
• Predicting Heart Activity from Speech using Data-driven and Knowledge-based features
• HeAR -- Health Acoustic Representations
• Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis
• Robust Speech Recognition via Large-Scale Weak Supervision
• Masked Autoencoders that Listen
• Speech Emotion Recognition using Self-Supervised Features
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Layer-wise Analysis of a Self-supervised Speech Representation Model
• HuBERT: Self-Supervised Speech Representation Learning by Masked\n Prediction of Hidden Units
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Cited by

• Breathing and Semantic Pause Detection and Exertion-Level Classification in Post-Exercise Speech

Related

• Predicting Heart Activity from Speech using Data-driven and Knowledge-based features
• Generative Deep Learning and Signal Processing for Data Augmentation of Cardiac Auscultation Signals: Improving Model Robustness Using Synthetic Audio
• Beyond Speech and More: Investigating the Emergent Ability of Speech Foundation Models for Classifying Physiological Time-Series Signals
• Assessing the Utility of Audio Foundation Models for Heart and Respiratory Sound Analysis
• Training one model to detect heart and lung sound events from single point auscultations