Divergence measures based on the Shannon entropy

1991/01/01 by J. Lin, Jinfeng Lin · 250 citations
Computer Science · Physics and Astronomy · Decision Sciences · #Bayesian Modeling and Causal Inference #Statistical Mechanics and Entropy #Multi-Criteria Decision Making

paper · doi:10.1109/18.61115

Abstract

A novel class of information-theoretic divergence measures based on the Shannon entropy is introduced. Unlike the well-known Kullback divergences, the new measures do not require the condition of absolute continuity to be satisfied by the probability distributions involved. More importantly, their close relationship with the variational distance and the probability of misclassification error are established in terms of bounds. These bounds are crucial in many applications of divergence measures. The measures are also well characterized by the properties of nonnegativity, finiteness, semiboundedness, and boundedness.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Citations

• On Information and Sufficiency

Cited by

• Optimal Entropy-Transport problems and a new Hellinger–Kantorovich distance between positive measures
• Where ideology meets private interest: the three-part composition of climate obstruction in the United States
• Quantifying compositional variability in microbial communities with FAVA
• Detecting and diagnosing prior and likelihood sensitivity with power-scaling
• Markovian Continuity of the MMSE
• Drift-Aware RL-based Wavelet Denoising for Network-Traffic Anomaly Detection
• Isolation Failure From Shared Storage: Characterizing and Exploiting Page-Cache SCA Leakage Across Containers and VMs
• Clustering of multivariate tail dependence using conditional methods
• Securing Multimodal AI through Internal Information Decomposition
• GW240925 and GW250207: Astrophysical calibration of gravitational wave detectors
• On Estimating the Trace of Quantum State Powers
• AlphaFlex: Ensembles of the human proteome representing disordered regions
• Constructions are Revealed in Word Distributions
• Modeling Spatial and Temporal Cues for Multi-label Facial Action Unit Detection
• Beyond Application End-Point Results: Quantifying Statistical Robustness of MCMC Accelerators
• Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributions
• Unsupervised learning of dynamical and molecular similarity using variance minimization
• Similarity Detection Pipeline for Crawling a Topic Related Fake News Corpus
• Excitation-based Voice Quality Analysis and Modification
• A Divergence Minimization Perspective on Imitation Learning Methods
• Adversarial Numerical Analysis for Inverse Problems
• Compressed-domain visual saliency models: A comparative study
• ReDiF: Reinforced Distillation for Few Step Diffusion
• 1d-qt-ideal-solver: 1D Idealized Quantum Tunneling Solver with Absorbing Boundaries
• LU-500: A Logo Benchmark for Concept Unlearning
• Bounds on Bayes Factors for Binomial A/B Testing
• GW250114: testing Hawking's area law and the Kerr nature of black holes
• DICA: Dual-Indicator Guided Contrastive Alignment in Multimodal Large Language Models
• Coevolution of epidemic dynamics and network topology driven by disease fatality and waning immunity
• ShinyNeRF: Digitizing Anisotropic Appearance in Neural Radiance Fields
• What is Stochastic Supervenience?
• Simulation-based inference with neural posterior estimation applied to X-ray spectral fitting - III Deriving exact posteriors with dimension reduction and importance sampling
• Metanetworks as Regulatory Operators: Learning to Edit for Requirement Compliance
• Social Story Frames: Contextual Reasoning about Narrative Intent and Reception
• A new group of transformations related to the Kullback-Leibler and Rényi divergences and universal classes of monotone measures of statistical complexity
• Does Less Hallucination Mean Less Creativity? An Empirical Investigation in LLMs
• Topology-Guided Quantum GANs for Constrained Graph Generation
• Infusing Experimental Reality into Complex Many-Body Hamiltonians: The Observable-Constrained Variational Framework (OCVF)
• Disentangled and Distilled Encoder for Out-of-Distribution Reasoning with Rademacher Guarantees
• Enhancing Chinese-to-English legal translation by generative artificial intelligence: a corpus-informed prompt engineering approach
• DAO-GP Drift Aware Online Non-Linear Regression Gaussian-Process
• Elephants Don’t Pack Groceries: Robot Task Planning for Low Entropy Belief States
• Beyond Pattern Matching: Seven Cross-Domain Techniques for Prompt Injection Detection
• SAM3-I: Segment Anything with Instructions
• Identifying preferred routes of sharing information on social networks
• A Dual Approach for Hierarchical Information-Theoretic Tree Abstractions
• Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?
• A Needle in a Haystack: Intent-driven Reusable Artifacts Recommendation with LLMs
• Score-Regularized Joint Sampling with Importance Weights for Flow Matching
• OmniGround: A Comprehensive Spatio-Temporal Grounding Benchmark for Real-World Complex Scenarios
• Limits of Deepfake Detection: A Robust Estimation Viewpoint
• An Interpretability-Guided Framework for Responsible Synthetic Data Generation in Emotional Text
• Inherent Tradeoffs in Learning Fair Representations
• Monte Carlo convergence of rival samplers
• Structural asymmetry as a fraud signature: detecting collusion with Heron's Information Coefficient
• nuPlan-R: A Closed-Loop Planning Benchmark for Autonomous Driving via Reactive Multi-Agent Simulation
• Privacy-Preserving Explainable AIoT Application via SHAP Entropy Regularization
• Leaf mass per area: An investigation into the application of the ubiquitous functional trait from a paleobotanical perspective
• Breaking the Adversarial Robustness-Performance Trade-off in Text Classification via Manifold Purification
• The impact of precession and higher-order multipoles on cosmological inference
• Surprisingly Similar: The Mass Function of Gaia Neutron Stars and First-Born Double Neutron Stars
• Reasoning About Generalization via Conditional Mutual Information
• Non-Rival Data as Rival Products: An Encapsulation-Forging Approach for Data Synthesis
• Chain-of-Thought as a Lens: Evaluating Structured Reasoning Alignment between Human Preferences and Large Language Models
• Data-driven uncertainty-aware seakeeping prediction of the Delft 372 catamaran using ensemble Hankel dynamic mode decomposition
• System Identification of a Moored ASV with Recessed Moon Pool via Deterministic and Bayesian Hankel-DMDc
• Discrete and Continuous Action Representation for Practical RL in Video Games
• Information bounds the robustness of self-organized systems
• A Framework Based on Graph Cellular Automata for Similarity Evaluation in Urban Spatial Networks
• Beyond Single-Tokenomics: How Farcaster's Pluralistic Incentives Reshape Social Networking
• Generalized Divergence Measures and Weak Convergence for the Sets of Probability Measures
• Fused Lasso for Feature Selection using Structural Information
• Investigating the Validity Evidence of Automated Scoring Methods for Divergent Thinking Assessments
• Throughput-Optimal Topology Design for Cross-Silo Federated Learning
• Automatic scoring of apnea and hypopnea events using blood oxygen saturation signals
• Convolutional relation network for facial expression recognition in the wild with few-shot learning
• Nonlinear Chaotic Processing Model
• S-MGHSTN: Towards An Effective Streaming Traffic Accident Risk Prediction Framework
• T-Rep: Representation Learning for Time Series using Time-Embeddings
• From Motifs to Lévy Flights: Modeling Urban Mobility in Bogotá's Public Transport System
• Hearsay: Vision-Language Medical Diagnoses Without an Image
• To Split or Not to Split: The Impact of Disparate Treatment in Classification
• Analysis of symbolic sequences using the Jensen-Shannon divergence
• Distilling Audio-Visual Knowledge by Compositional Contrastive Learning
• Women also Snowboard: Overcoming Bias in Captioning Models
• Unsupervised Feature Learning for low-level Local Image Descriptors
• f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learning
• TVDIM: Enhancing Image Self-Supervised Pretraining via Noisy Text Data
• Near Real-time Learning and Extraction of Attack Models from Intrusion Alerts
• Exploiting Social Annotation for Automatic Resource Discovery
• A Study on the Uncertainty of Convolutional Layers in Deep Neural Networks
• DTGAN: Differential Private Training for Tabular GANs
• Dissecting Role Cognition in Medical LLMs via Neuronal Ablation
• Data-driven fault diagnosis for heterogeneous chillers using domain adaptation techniques
• Joint Contrastive Learning for Unsupervised Domain Adaptation
• Varieties of platform capitalism? Competition, regime types and the diversity of food delivery platforms across Europe and North America
• ELKI: A large open-source library for data analysis - ELKI Release 0.7.5 "Heidelberg"
• FR-Train: A Mutual Information-Based Approach to Fair and Robust Training
• On the Upper Limit of Separability
• Kullback-Leibler divergence between quantum distributions, and its upper-bound
• Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution
• ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Control
• Improving the Generation and Evaluation of Synthetic Data for Downstream Medical Causal Inference
• Randomized SMILES strings improve the quality of molecular generative models
• On the use of the Derivative Approximation for Likelihoods for Gravitational Wave Inference
• A unified framework for divergences, free energies, and Fokker-Planck equations
• Statistically Significant Detection of Linguistic Change
• ToolTweak: An Attack on Tool Selection in LLM-based Agents
• Regularizing Generative Adversarial Networks under Limited Data
• A Convex Cauchy-Schwarz DivergenceMeasure for Blind Source Separation
• Disentangling Options with Hellinger Distance Regularizer
• How Well do Diffusion Policies Learn Kinematic Constraint Manifolds?
• Can We Reliably Rank Model Performance across Domains without Labeled Data?
• On Learning Invariant Representation for Domain Adaptation
• GeoGen: A Two-stage Coarse-to-Fine Framework for Fine-grained Synthetic Location-based Social Network Trajectory Generation
• Improving Reliability of Latent Dirichlet Allocation by Assessing Its Stability Using Clustering Techniques on Replicated Runs
• Transferable Generative Models Bridge Femtosecond to Nanosecond Time-Step Molecular Dynamics
• On the Guaranteed Almost Equivalence between Imitation Learning from Observation and Demonstration
• SDQM: Synthetic Data Quality Metric for Object Detection Dataset Evaluation
• Multi-Agent Collaborative Intelligence: Dual-Dial Control for Reliable LLM Reasoning
• Relative Code Comprehensibility Prediction
• Dynamic roughening of cities driven by multiplicative noise
• Using Fourier Analysis and Mutant Clustering to Accelerate DNN Mutation Testing
• Learning Stylized Character Expressions from Humans
• Model Correlation Detection via Random Selection Probing
• A DNA-encoded recipe to direct multi-stage colloidal assembly
• Network entropy as a key to the past: A quantitative approach to complex social networks
• Metrics for more than two points at once
• Quantum skew divergence
• Concept-SAE: Active Causal Probing of Visual Model Behavior
• PANICL: Mitigating Over-Reliance on Single Prompt in Visual In-Context Learning
• A High-Capacity and Secure Disambiguation Algorithm for Neural Linguistic Steganography
• Designing Ethereum's Geographical (De)Centralization Beyond the Atlantic
• Robust AI-ECG for Predicting Left Ventricular Systolic Dysfunction in Pediatric Congenital Heart Disease
• Multi-task Learning by Leveraging the Semantic Information
• Quantifying Interdisciplinarity in Cognitive Science and Beyond
• CTAB-GAN: Effective Table Data Synthesizing
• Smoking-gun evidence for hierarchical black-hole mergers
• Continual Learning with Vision-Language Models via Semantic-Geometry Preservation
• The impact of glitches on young pulsar rotational evolution
• Assessing climate vulnerability and nonlinear rainfall dynamics in complex networks
• <scp>PEG–mCherry</scp> interactions beyond classical macromolecular crowding
• Two types of inversely graded bands in spaced stratification identified by microtextural analysis using a convolutional neural network
• Learning is a fundamental source of individuality
• Emerging App Issue Identification via Online Joint Sentiment-Topic Tracing
• LiDAR Point Cloud Image-based Generation Using Denoising Diffusion Probabilistic Models
• Similarity Field Theory: A Mathematical Framework for Intelligence
• Ambiguous Medical Image Segmentation Using Diffusion Schrödinger Bridge
• Topic-based Integrator Matching for Pull Request
• Effective and Privacy preserving Tabular Data Synthesizing
• MusicSwarm: Biologically Inspired Intelligence for Music Composition
• Evaluation of Black-Box XAI Approaches for Predictors of Values of Boolean Formulae
• Gradual Domain Adaptation via Normalizing Flows
• An Information-Theoretic Framework for Credit Risk Modeling: Unifying Industry Practice with Statistical Theory for Fair and Interpretable Scorecards
• Permutation-Based Distances for Groups and Group-Valued Time Series
• Infinite Stream Estimation under Personalized w-Event Privacy
• Machine Learning with Multitype Protected Attributes: Intersectional Fairness through Regularisation
• Are Humans as Brittle as Large Language Models?
• Efficient and Adaptive Recommendation Unlearning: A Guided Filtering Framework to Erase Outdated Preferences
• MS2: Multi-Document Summarization of Medical Studies
• Classification of Consumer Belief Statements From Social Media
• A Minimum Message Length Criterion for Robust Linear Regression
• FSMJ: Feature Selection with Maximum Jensen-Shannon Divergence for Text Categorization
• Trade-offs and Guarantees of Adversarial Representation Learning for\n Information Obfuscation
• Popularity Bias in Recommendation: A Multi-stakeholder Perspective
• Medium-range structural order in amorphous arsenic
• RiskOracle: A Minute-level Citywide Traffic Accident Forecasting Framework
• Weather-Dependent Variations in Driver Gaze Behavior: A Case Study in Rainy Conditions
• The Context Sensitivity Problem in Biological Sequence Segmentation
• Lectures on Entropy. Part I
• Collective Learning by Ensembles of Altruistic Diversifying Neural Networks
• A Case Study of Balanced Query Recommendation on Wikipedia
• LFD: Layer Fused Decoding to Exploit External Knowledge in Retrieval-Augmented Generation
• Delta-Audit: Explaining What Changes When Models Change
• Improving Recommendation Fairness via Graph Structure and Representation Augmentation
• Reconstruction Error Bounds for Compressed Sensing under Poisson Noise using the Square Root of the Jensen-Shannon Divergence
• Fed-TGAN: Federated Learning Framework for Synthesizing Tabular Data
• GWTC-4.0: Methods for Identifying and Characterizing Gravitational-wave Transients
• REG4Rec: Reasoning-Enhanced Generative Model for Large-Scale Recommendation Systems
• Unsupervised Label Refinement Improves Dataless Text Classification
• A Theory of Information, Variation, and Artificial Intelligence
• Memory erasure with finite-sized spin reservoir
• A Lightweight Privacy-Preserving Smart Metering Billing Protocol with Dynamic Tariff Policy Adjustment
• Evaluating Sparse Autoencoders for Monosemantic Representation
• Measuring LLM Code Generation Stability via Structural Entropy
• Constrained Centroid Clustering: A Novel Approach for Compact and Structured Partitioning
• Modeling Social Annotation: a Bayesian Approach
• TCUQ: Single-Pass Uncertainty Quantification from Temporal Consistency with Streaming Conformal Calibration for TinyML
• Familia: A Configurable Topic Modeling Framework for Industrial Text Engineering
• HyperKD: Distilling Cross-Spectral Knowledge in Masked Autoencoders via Inverse Domain Shift with Spatial-Aware Masking and Specialized Loss
• Characterizing English Variation across Social Media Communities with BERT
• On the Language Coverage Bias for Neural Machine Translation
• Challenging a binary neutron star merger interpretation of GW230529
• SynLLM: A Comparative Analysis of Large Language Models for Medical Tabular Synthetic Data Generation via Prompt Engineering
• Confronting general relativity with principal component analysis: Simulations and results from GWTC-3 events
• Addressing the Multistakeholder Impact of Popularity Bias in Recommendation Through Calibration
• ADSEL: Adaptive dual self-expression learning for EEG feature selection via incomplete multi-dimensional emotional tagging
• Two tales for a geometric Jensen--Shannon divergence
• Rapid parameter estimation with the full symphony of compact binary mergers using meshfree approximation
• Distribution-Based Categorization of Classifier Transfer Learning
• Explainable Deep Neural Network for Multimodal ECG Signals: Intermediate vs Late Fusion
• When Cars Have Stereotypes: Auditing Demographic Bias in Objects from Text-to-Image Models
• Beyond Manually Designed Pruning Policies with Second-Level Performance Prediction: A Pruning Framework for LLMs
• Versatile yet Efficient Network Traffic Analysis: Offloading Network Foundation Model to SmartNIC
• KCR: Resolving Long-Context Knowledge Conflicts via Reasoning in LLMs
• MPCA-based Domain Adaptation for Transfer Learning in Ultrasonic Guided Waves
• Can User Feedback Help Issue Detection? An Empirical Study on a One-billion-user Online Service System
• JSON-Bag: A generic game trajectory representation
• Closed-Form Beta Distribution Estimation from Sparse Statistics with Random Forest Implicit Regularization
• Graph-Guided Dual-Level Augmentation for 3D Scene Segmentation
• DenDrift: A Drift-Aware Algorithm for Host Profiling
• The Fourier Discrepancy Function
• Meaning-infused grammar: Gradient Acceptability Shapes the Geometric Representations of Constructions in LLMs
• Non-Parametric Methods Applied to the N-Sample Series Comparison
• Towards Locally Deployable Fine-Tuned Causal Large Language Models for Mode Choice Behaviour
• The Effect of Collective Attention on Controversial Debates on Social\n Media
• Do Not Mimic My Voice: Speaker Identity Unlearning for Zero-Shot Text-to-Speech
• Canoe : A System for Collaborative Learning for Neural Nets
• Neighborhood-Adaptive Accelerated Fuzzy C-Means for Three-Dimensional Millimeter-Wave Image Segmentation
• Calibrated Variant Effect Prediction at the Residue Level Using Conditional Score Distributions
• SimPersona: Learning Discrete Buyer Personas from Raw Clickstreams for Grounded E-Commerce Agents
• Manifold Interpolation for Large-Scale Multi-Objective Optimization via Generative Adversarial Networks
• On the Centroids of Symmetrized Bregman Divergences
• Collective behaviour of social bots is encoded in their temporal Twitter activity
• Evaluating Causal Models by Comparing Interventional Distributions
• Coding limits on the number of transcription factors. [europepmc]
• Improved coarse-graining of Markov state models via explicit consideration of statistical uncertainty. [europepmc]
• DiffSplice: the genome-wide detection of differential splicing events with RNA-seq. [europepmc]
• High-order social interactions in groups of mice. [europepmc]
• Cortical regulation of cell size by a sizer cdr2p. [europepmc]
• The vaginal microbiota of pregnant women who subsequently have spontaneous preterm labor and delivery and those with a normal delivery at term. [europepmc]
• Characterizing the Google Books Corpus: Strong Limits to Inferences of Socio-Cultural and Linguistic Evolution. [europepmc]
• DiffLogo: a comparative visualization of sequence motifs. [europepmc]
• Fast and accurate single-cell RNA-seq analysis by clustering of transcript-compatibility counts. [europepmc]
• Oxalobacter formigenes-associated host features and microbial community structures examined using the American Gut Project. [europepmc]
• The Inuit gut microbiome is dynamic over time and shaped by traditional foods. [europepmc]
• Strain profiling and epidemiology of bacterial species from metagenomic sequencing. [europepmc]
• Analyzing Immunoglobulin Repertoires. [europepmc]
• Quantifying size and diversity of the human T cell alloresponse. [europepmc]
• Quality and bias of protein disorder predictors. [europepmc]
• Benchmarking of alignment-free sequence comparison methods. [europepmc]
• Characteristics, service use and mortality of clusters of multimorbid patients in England: a population-based study. [europepmc]
• Identifying inhibitors of epithelial-mesenchymal plasticity using a network topology-based approach. [europepmc]
• Association of Mobile Phone Location Data Indications of Travel and Stay-at-Home Mandates With COVID-19 Infection Rates in the US. [europepmc]
• A multiresolution framework to characterize single-cell state landscapes. [europepmc]
• On the Jensen-Shannon Symmetrization of Distances Relying on Abstract Means. [europepmc]
• Randomized SMILES strings improve the quality of molecular generative models. [europepmc]
• Topological signatures in regulatory network enable phenotypic heterogeneity in small cell lung cancer. [europepmc]
• Predicting retrosynthetic pathways using transformer-based models and a hyper-graph exploration strategy. [europepmc]
• Reconciling functional differences in populations of neurons recorded with two-photon imaging and electrophysiology. [europepmc]

Related

• Characterization of Generalized Alpha-Beta Divergence and Associated Entropy Measures
• Formulation and properties of a divergence used to compare probability measures without absolute continuity and its application to uncertainty quantification
• Formulation and properties of a divergence used to compare probability measures without absolute continuity
• Normalized information-based divergences
• Sequences of Inequalities Among New Divergence Measures