2009/02/28 by Robert K. Niven, R. K. Niven
Physics and Astronomy · #Advanced Mathematical Theories and Applications #Advanced Thermodynamics and Statistical Mechanics #Axiom #Axiomatic system #Entropy (arrow of time) #Independent and identically distributed random variables #Multinomial distribution #Probabilistic logic #Probability distribution #Random variable #Statistical Mechanics and Entropy #cond-mat.stat-mech
paper · pdf · doi:10.1140/epjb/e2009-00168-5
Invited contribution to the SigmaPhi 2008 Conference; accepted by EPJB volume 69 issue 3 June 2009
arxiv created 2009/04/09 · openalex publication_date 2009/05/14 · arxiv updated 2015/05/12 · openalex created_date 2016/06/24 · openalex updated_date 2026/08/05
We examine the combinatorial or probabilistic definition ("Boltzmann's principle") of the entropy or cross-entropy function H ∝ ln \mathbbW or D ∝ - ln ℙ, where \mathbbW is the statistical weight and ℙ the probability of a given realization of a system. Extremisation of H or D, subject to any constraints, thus selects the "most probable" (MaxProb) realization. If the system is multinomial, D converges asymptotically (for number of entities N \back → \back ∞) to the Kullback-Leibler cross-entropy DKL; for equiprobable categories in a system, H converges to the Shannon entropy HSh. However, in many cases \mathbbW or ℙ is not multinomial and/or does not satisfy an asymptotic limit. Such systems cannot meaningfully be analysed with DKL or HSh, but can be analysed directly by MaxProb. This study reviews several examples, including (a) non-asymptotic systems; (b) systems with indistinguishable entities (quantum statistics); (c) systems with indistinguishable categories; (d) systems represented by urn models, such as "neither independent nor identically distributed" (ninid) sampling; and (e) systems representable in graphical form, such as decision trees and networks. Boltzmann's combinatorial definition of entropy is shown to be of greater importance for "probabilistic inference" than the axiomatic definition used in information theory.