Understanding Shannon's Entropy metric for Information

2014/03/24 by Sriram Vajapeyam, Vajapeyam, Sriram · 1 voice · 3 citations
Computer Science · Neuroscience · #Cognitive Science and Education Research #Neural Networks and Applications #cs.IT

paper · pdf · doi:10.48550/arxiv.1405.2061

openalex publication_date 2014/03/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Shannon's metric of "Entropy" of information is a foundational concept of information theory. This article is a primer for novices that presents an intuitive way of understanding, remembering, and/or reconstructing Shannon's Entropy metric for information.

Cited by

• STEMTOX: From Collaborative Tags to Fine-Grained Toxic Meme Detection via Entropy-Guided Multi-Task Learning
• Cross-Attention Calibrated Deduplication for Retrieval-Augmented Generation System
• BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactions

Discussions

• Understanding Shannon's Entropy metric for Information [pdf] [hn, 100 points, 42 comments]

Related

• Shannon's entropy and its Generalizations towards Statistics, Reliability and Information Science during 1948-2018
• Information vs Thermodynamic Entropy
• DLITE: The Discounted Least Information Theory of Entropy
• What entropy really is : the contribution of information theory
• On the Structure of Information