Hellinger Distance

Hellinger distance

The Hellinger distance (sometimes called the Jeffreys distance) is a metric in the space of probability distributions. The Hellinger distance can be used to quantify the degree of similarity between two probability distributions. It is the probabilistic analog of Euclidean distance.

Let P and Q denote two probability measures on a measure space χ with probability densities px and gx. The Hellinger distance, HP,Q is then defined as l2 norm between px and qx.

The Hellinger distance forms a bounded metrics, meaning that it takes values in between 0 and 1. In other words, it satisfies the property 0⩽HP,Q⩽1. When distance is 0 the two distributions are identical.  The maximum distance 1 is achieved when the two probability distributions are furthest apart . The distance is therefore not able to differentiate between very strong shifts when distance values are maxed to 1.  The squared Hellinger distance is symmetric, meaning that it satisfies the property H2P,Q=H2Q,P.

Hellinger distance is closely related to, although different from, the Bhattacharyya Coefficient. In contrast to the Bhattacharyya Coefficient, the Hellinger distance is a metric which fulfills the triangular inequality, making it easier to interpret and work with. As a result, the Hellinger distance is often used as a replacement for the Bhattacharyya Coefficient.  

For continuous measures, the Hellinger distance is defined as: $$H(P,Q) = \left[\int (\sqrt{p(x)} - \sqrt{q(x)})^2 dx \right]^{\frac{{1}}{2}}.$$

For two discrete probability distributions $$P = \left (p_1, ..., p_k \right)$$ and $$Q = \left (q_1, ..., q_k \right)$$, the Hellinger distance is defined as:  $$ H\left(P,Q\right) = {\frac{{1}}{2}}{\Vert \sqrt{P} - \sqrt{Q} \Vert}_{2}. $$

Intuitively, the Hellinger distance quantifies overlap between the probabilities assigned to the same event by two distributions, as depicted in figure below. The Hellinger distance can also be used in a multivariate context, for example, when it is computed over the Gaussian copula representation of joint distributions.