Kullback-Leibler (KL) divergence

It is also known as relative entropy or information divergence (I-divergence).  It is not symmetric in the two distributions, meaning that the KL divergence between distribution P and distribution Q is generally not equal to the KL divergence between distribution Q and distribution P.

When comparing two distributions, KL divergence is helpful if one has a high variance or a limited sample size relative to the other.

Which is equivalent to

Additionally, KL divergence is asymmetric. In contrast to PSI, this indicates that the results will vary if the reference and production(compared) distributions are switched. Therefore, KL(P || Q) != KL(Q || P). This makes it handy for applications utilising Baye's theorem or when you have a large number of training (reference) samples but only a limited set of samples, resulting in increased variance in the comparison distribution.

A KL score can range from 0 to infinity, with 0 denoting equality between the two distributions. The output will be in "nats" if the natural log (base-e) is used, and "bits" if the KL formulae are taken to log base 2.

‍

KL divergence can be used for both numerical and categorical features. The default KL score threshold for drift in AryaXAI is '0.05'.

‍