Reduction of Maximum Entropy Models to Hidden Markov Models

2012/12/12 by Joshua Goodman, Goodman, Joshua
Computer Science · #Authorship Attribution and Profiling #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.1301.0570

Appears in Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence (UAI2002)

arxiv created 2012/12/12 · arxiv updated 2013/01/07

Abstract

We show that maximum entropy (maxent) models can be modeled with certain kinds of HMMs, allowing us to construct maxent models with hidden variables, hidden state sequences, or other characteristics. The models can be trained using the forward-backward algorithm. While the results are primarily of theoretical interest, unifying apparently unrelated concepts, we also give experimental results for a maxent model with a hidden variable on a word disambiguation task; the model outperforms standard techniques.

Related

• Maximum Entropy Estimator for Hidden Markov Models: Reduction to Dimension 2
• Maximum Entropy Modeling Toolkit
• Entropy Semiring Forward-backward Algorithm for HMM Entropy Computation
• Classes for Fast Maximum Entropy Training
• Hidden Markov Chains, Entropic Forward-Backward, and Part-Of-Speech Tagging