Shannon Entropy
Finite discrete form, in bits
Shannon entropy measures the average information content of a discrete random variable, or the uncertainty in its outcome before it is observed. Each outcome contributes its information content weighted by its probability. Base-two logarithms express the result in bits.
Shannon entropy
Expected self-information
Notes
The random variable has possible outcomes with probabilities and . An individual outcome has self-information ; entropy is the probability-weighted average of these values, using the continuous extension for outcomes of zero probability.
Entropy is zero for a certain outcome and is maximized, for a fixed number of possible outcomes, by the uniform distribution, giving bits.
Related equations
- Bayes' Theoremrelated
Sources
- A Mathematical Theory of Communication — Claude E. ShannonShannon's communication theory paper.