DenseHMM: Learning Hidden Markov Models by Learning Dense Representations

2020/12/17 by Joachim Sicking, Maximilian Pintz, Sicking, Joachim +5
Biochemistry, Genetics and Molecular Biology · Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning in Bioinformatics #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2012.09783

openalex publication_date 2020/12/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose DenseHMM - a modification of Hidden Markov Models (HMMs) that allows to learn dense representations of both the hidden states and the observables. Compared to the standard HMM, transition probabilities are not atomic but composed of these representations via kernelization. Our approach enables constraint-free and gradient-based optimization. We propose two optimization schemes that make use of this: a modification of the Baum-Welch algorithm and a direct co-occurrence optimization. The latter one is highly scalable and comes empirically without loss of performance compared to standard HMMs. We show that the non-linearity of the kernelization is crucial for the expressiveness of the representations. The properties of the DenseHMM like learned co-occurrences and log-likelihoods are studied empirically on synthetic and biomedical datasets.

Related

• Learning Hidden Markov Models using Non-Negative Matrix Factorization
• Density Ratio Hidden Markov Models
• Bridging the Usability Gap: Theoretical and Methodological Advances for Spectral Learning of Hidden Markov Models
• Parameter estimation of hidden Markov models: comparison of EM and quasi-Newton methods with a new hybrid algorithm
• Scaling Factorial Hidden Markov Models: Stochastic Variational Inference without Messages