Definition
Inter-rater reliability concerns how consistently different reviewers assess the same material. In origination research it can help evaluate whether criteria, allowed answers, and evidence rules produce repeatable decisions. Agreement alone does not prove correctness: reviewers can share an ambiguous rule, a mistaken assumption, or an inadequate evidence set.
A meaningful comparison holds the task and information available to reviewers sufficiently constant. If one reviewer has fresh sources while another sees only a partial dossier, disagreement may reflect unequal evidence rather than inconsistent judgement. The measure and comparison design should match the kind of answers being assessed. Preserve the original decisions before discussion so consensus does not erase the disagreement the exercise was meant to investigate.
In origination work
Have reviewers apply the same version of a screening question to the same retained materials. Examine disagreements by cause: unclear definitions, missing context, evidence interpretation, or a genuine borderline case. Revise the rubric where appropriate and keep track of which version governed each assessment. An adjudicated answer resolves an operational case, while the disagreement record helps improve the process. High agreement should not justify removing evidence review or presenting unresolved facts as established merely because several readers accepted the same statement.
What it measures
Inter-rater reliability is the degree to which independent reviewers give the same answer to the same question about the same item. If two analysts classify a company as meeting or not meeting a criterion, reliability is the extent to which they agree, adjusted for the agreement expected by chance. A criterion that two competent people cannot apply the same way is not yet a usable criterion.
The best-known statistic is Cohen's kappa, introduced in 1960 as a coefficient of agreement for nominal scales. It compares observed agreement with chance agreement, so a value of one means perfect agreement and zero means agreement no better than chance. A widely cited 1977 paper proposed verbal labels for ranges of kappa; they are conventions, and which level is adequate depends on the cost of error.
Why it matters for screening
Disagreement has two typical causes: unclear criteria and unclear evidence. If two reviewers disagree because they interpret "recurring revenue" differently, the criterion needs a tighter definition and an accepted-values list. If they disagree because one found a page the other did not, the process for retrieving evidence needs attention. Measuring agreement shows which problem you have.
Research on professional judgement shows that unexplained variability between experts is common and costly, which the authors call noise, and that structured approaches reduce it. Reliability measurement is the tool for detecting that noise in a screening process. It is also the right way to evaluate an automated system: compare it with human reviewers using the same statistic, and compare human reviewers with each other to learn the ceiling.
Running a reliability check
Draw a random sample of items, have two reviewers classify each independently without seeing each other's answers, compute raw agreement and kappa per criterion, and examine every disagreement. Resolve disagreements with a third reviewer or a written rule, and update the criterion definitions. Repeat after changes. A small sample can mislead, so report the sample size along with the statistic.
Model-based reviewers can be assessed the same way. Work on using language models as judges reports high agreement with human preferences on some tasks while identifying systematic biases, which is a reason to measure agreement on your own criteria, not to trust reported figures from other settings. If a model reviewer agrees with humans about as well as humans agree with each other, it may serve as a second reviewer under supervision.
Related terms
- Screening rubric — A defined set of questions, accepted answers, evidence requirements, and decision rules for assessing candidates.
- Evidence trail — A reviewable chain connecting a research conclusion to its supporting material, interpretation, and decisions.
- Grounding — Basing generated research answers on supplied evidence while still checking that the evidence supports each answer.
Further reading
- Make screening decisions repeatable
- Screening criteria guide
- Screening rubric
- Disqualifier
- Analysts versus AI agents
- Article: how many records to sample for an audit
- Article: benchmarking AI agents for acquisition research
Sources and further reading
- A Coefficient of Agreement for Nominal Scales, Educational and Psychological Measurement (1960). The original definition of kappa, agreement corrected for chance.
- The Measurement of Observer Agreement for Categorical Data, Biometrics (1977). The source of the widely quoted benchmarks for reading kappa values.
- Noise: How to Overcome the High, Hidden Cost of Inconsistent Decision Making, Harvard Business Review (2016). Unwanted variability between judges of the same case, and the structured protocols that reduce it.
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, arXiv (2023). Documents position, verbosity and self-enhancement bias in model-graded evaluation.
- Clinical versus mechanical prediction: a meta-analysis, Psychological Assessment (2000). 136 studies. Mechanical combination of cues equals or beats expert holistic judgement roughly nine times out of ten.
To apply these distinctions to your own acquisition criteria, discuss your research brief or inspect the demo platform.