Intraclass Correlation (ICC) Calculator
Inter-rater reliability from a ratings table — ICC(1,1), ICC(2,1) absolute agreement and ICC(3,1) consistency with the ANOVA mean squares.
How it works
ICC(2,1) = (MSᵣ − MSₑ) ÷ [MSᵣ + (k−1)MSₑ + k(MS꜀ − MSₑ)/n]; ICC(3,1) = (MSᵣ − MSₑ) ÷ [MSᵣ + (k−1)MSₑ]; ICC(1,1) = (MSᵣ − MS_w) ÷ [MSᵣ + (k−1)MS_w]
The intraclass correlation partitions the variance of the ratings: how much comes from real differences between subjects versus from raters and noise. ICC(2,1) treats raters as a random sample and penalises systematic rater bias (absolute agreement); ICC(3,1) treats these specific raters as fixed and only asks whether they rank subjects consistently; ICC(1,1) is for designs where each subject is rated by a different set of raters. Koo & Li’s guide: below 0.5 poor, 0.5–0.75 moderate, 0.75–0.9 good, above 0.9 excellent.
Worked example
Six subjects rated by four raters (the classic Shrout–Fleiss data): ICC(1,1) = 0.17, ICC(2,1) = 0.29, ICC(3,1) = 0.71.
Frequently asked questions
Which ICC should I report?
Absolute agreement ICC(2,1) if any trained rater could be used in practice; consistency ICC(3,1) if only these raters matter and a constant offset is acceptable.
Single or average measures?
Report the single-measure ICC when one rater will score subjects in practice; the (2,k) form applies only if the average of all k raters is used.
Related calculators
For education — check assumptions (sample size, distribution, independence) before using a test result in a real decision.