📊

Fleiss' Kappa Calculator

Fleiss’ kappa for agreement among three or more raters — κ, observed and chance agreement, z test, p-value, per-category kappa and the Landis–Koch strength label.

Rating counts (one row per subject)

Loading calculator…

How it works

pⱼ = Σᵢ nᵢⱼ ÷ (N·n); Pᵢ = [Σⱼ nᵢⱼ² − n] ÷ [n(n − 1)]; P̄ = mean Pᵢ; P̄e = Σ pⱼ²; κ = (P̄ − P̄e) ÷ (1 − P̄e); SE₀ = √2 ÷ [Σpⱼqⱼ·√(N·n(n−1))] · √[(Σpⱼqⱼ)² − Σpⱼqⱼ(qⱼ − pⱼ)]

Fleiss’ kappa (1971) measures how much a fixed number of raters agree when each assigns every subject to one of several categories, correcting for the agreement expected by chance. Unlike Cohen’s kappa it works for any number of raters, and the raters may differ from subject to subject. The z test uses the large-sample standard error under κ = 0 from Fleiss, Nee & Landis (1979); the strength label follows Landis & Koch (1977).

Worked example

Ten subjects, fourteen raters, five categories (the textbook example): P̄ = 0.3780, P̄e = 0.2128 and κ = 0.210 — fair agreement beyond chance.

Sources: Fleiss (1971), Measuring nominal scale agreement among many raters, Psychological Bulletin · Landis & Koch (1977), The measurement of observer agreement for categorical data · Fleiss' kappa — Wikipedia

Frequently asked questions

Fleiss’ kappa or Cohen’s kappa?

Cohen’s kappa is for exactly two raters; Fleiss’ kappa handles three or more raters (or a different set of raters for each subject) rating into nominal categories.

How do I read the value?

Landis & Koch: below 0 poor, 0–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial and 0.81–1 almost perfect agreement.

How do I enter my data?

One row per subject, one number per category: how many raters chose that category. Every row must add up to the number of raters.

Does it work for ordinal ratings?

It treats categories as nominal, so “1 vs 2” counts as much disagreement as “1 vs 5”. For ordinal or continuous ratings use a weighted kappa or the intraclass correlation.

Related calculators

Cohen’s Kappa Calculator

Inter-rater agreement for two raters and two categories from a 2×2 table — observed agreement, expected agreement, κ and its interpretation.

Intraclass Correlation (ICC) Calculator

Inter-rater reliability from a ratings table — ICC(1,1), ICC(2,1) absolute agreement and ICC(3,1) consistency with the ANOVA mean squares.

Cronbach’s Alpha Calculator

Estimate the internal-consistency reliability of a scale from the number of items, the sum of item variances and the variance of total scores.

Cochran's Q Test Calculator

Cochran’s Q test for three or more related yes/no outcomes — Q statistic, degrees of freedom, p-value, success rate of each treatment and the effect size η²Q.

Educational tool: check the test’s assumptions (sample size, independence, distribution) before relying on a result for a real decision.