KL Divergence Calculator
Kullback–Leibler divergence between two discrete distributions in both directions, plus Jensen–Shannon divergence, cross-entropy and entropy — in nats, bits or hartleys.
How it works
D_KL(P‖Q) = Σ pᵢ·log(pᵢ/qᵢ); JS = ½D_KL(P‖M) + ½D_KL(Q‖M), M = (P + Q)/2; H(P, Q) = H(P) + D_KL(P‖Q); H(P) = −Σ pᵢ log pᵢ
The Kullback–Leibler divergence measures how much information is lost when Q is used to approximate P — the extra bits (or nats) needed on average to encode samples from P with a code built for Q. It is zero only when the distributions are identical, it is not symmetric, and it becomes infinite if Q assigns zero probability to something P allows. The Jensen–Shannon divergence is a smoothed, symmetric and always-finite alternative. Values match SciPy’s scipy.stats.entropy(P, Q).
Worked example
P = (0.36, 0.48, 0.16) against an almost uniform Q = (0.333, 0.333, 0.334): D_KL(P ‖ Q) = 0.0858 nats, the reverse D_KL(Q ‖ P) = 0.0981 nats and the Jensen–Shannon divergence is 0.0226 nats.
Sources: Kullback & Leibler (1951), On information and sufficiency, Annals of Mathematical Statistics · SciPy — scipy.stats.entropy
Frequently asked questions
Why is KL divergence not symmetric?
D_KL(P‖Q) weights the log-ratio by P, so it punishes Q for missing outcomes that are likely under P; the reverse direction punishes the opposite. Use Jensen–Shannon if you need a symmetric measure.
Why do I get infinity?
Q gives probability 0 to an outcome that has positive probability under P. Add a small smoothing count to Q (e.g. +1 to every count) if that is not realistic.
How is KL related to cross-entropy loss?
Cross-entropy H(P, Q) = H(P) + D_KL(P‖Q). Because H(P) of the true labels is fixed, minimising cross-entropy loss is the same as minimising the KL divergence.
Nats, bits or hartleys?
They differ only by the logarithm base: 1 nat = 1.4427 bits = 0.4343 hartleys. Machine-learning libraries usually report nats.
Related calculators
This calculator is for educational purposes. Double-check important results before acting on them.