Equationpedia

Shannon Entropy

Finite discrete form, in bits

Shannon entropy measures the average information content of a discrete random variable, or the uncertainty in its outcome before it is observed. Each outcome contributes its information content weighted by its probability. Base-two logarithms express the result in bits.

Shannon entropy
Expected self-information
H(X)=−∑i=1npi log⁡2pi\htmlData{region=shannon-entropy}{\htmlData{term=entropy-function}{H}(\htmlData{term=random-variable}{X})} = \htmlData{region=expected-self-information}{-\htmlData{term=outcome-sum}{\sum_{i=1}^{n}} \htmlData{term=outcome-probability}{p_i}\,\htmlData{term=binary-logarithm}{\log_2} \htmlData{term=outcome-probability}{p_i}}

Notes

The random variable XX has nn possible outcomes with probabilities pi≥0p_i \geq 0 and ∑ipi=1\sum_i p_i = 1. An individual outcome has self-information −log⁡2pi-\log_2 p_i; entropy is the probability-weighted average of these values, using the continuous extension 0log⁡20=00\log_2 0 = 0 for outcomes of zero probability.

Entropy is zero for a certain outcome and is maximized, for a fixed number nn of possible outcomes, by the uniform distribution, giving log⁡2n\log_2 n bits.

Related equations

Sources