Claude Shannon

Wednesday, March 25th, 2015

Room 202 in Packard Bldg., Stanford University
Parking Generally Free In Nearby Lots After 4:00 pm

Refreshments and Conversation at 6:00 P.M.
Presentation at 6:30 P.M.

How to estimate mutual information with insufficient sampling

Jiantao Jiao
Stanford University


Mutual information emerged in Shannon’s 1948 masterpiece as the answer to the most fundamental questions of compression and communication. Since that time, however, it has been adopted and widely used in a variety of other disciplines. In particular, its estimation has emerged as a key component in fields such as machine learning, computer vision, systems biology, medical imaging, neuroscience, genomics, economics, ecology, and physics. In practical applications, the underlying distribution is usually unknown, so it is of utmost importance to obtain accurate mutual information estimates from empirical data for inference.

We discuss a new approach to the estimation of mutual information between random objects with distributions residing in high-dimensional spaces (e.g., large alphabets), as is the case in increasingly many applications. We will discuss the shortcomings of traditional estimators, and suggest a new estimator achieving essentially optimum worst-case performance under L2 risk (i.e., achieves the minimax rates). We apply this new estimator in various applications, including the Chow--Liu algorithm and the Tree-Augmented Naive Bayes (TAN) classifier. Experiments with these and other algorithms show that replacing the empirical mutual information by the proposed estimator results in consistent and substantial performance boosts on a wide variety of datasets.


Photo of Jiantao Jiao Jiantao Jiao is a Ph.D. candidate in Stanford University's Electrical Engineering Department with research interests in information theory, probability theory and high dimensional and nonparametric statistics. He has an M.S. Electrical Engineering from Stanford University (2014) and a B.E. Electronic Engineering from Tsinghua University, Beijing, China (2012).



SCV IT Society Webmaster (
Last updated on

Return to SCV IT Society Homepage