AI Evaluation Is a Measurement Problem

Psychometrics spent a century learning how to measure things you can't see directly. AI evaluation keeps rediscovering the same lessons the hard way.

A familiar scene: a team builds an "LLM-as-judge" pipeline to grade model outputs, checks it against a few hundred human ratings, finds 85% agreement, and calls the evaluator validated. The eval ships. Everyone moves on.

In the field I trained in, that 85% isn't the finish line. It's closer to the first item on a long list of questions you're supposed to ask before trusting a score that affects a decision.

I came to AI evaluation from industrial-organizational psychology, where the entire job is measuring things that have no ruler. Job performance, cognitive ability, conscientiousness, "culture fit." You can't observe conscientiousness directly. You build an instrument you hope captures it, and then you spend most of your effort proving the instrument actually does what you claim. That discipline is psychometrics, and it has been sharpening its methods for roughly a hundred years, partly because the stakes (hiring, promotion, admissions) have always been high enough to invite lawsuits.

Watching AI evaluation grow up over the last few years, I keep recognizing problems psychometrics already named and partly solved. The field is reinventing the wheel, and occasionally shipping a square one. Here's the case for borrowing from the older discipline.

A score is a measurement, and measurement starts with reliability

The moment you use a model or a panel of humans to score outputs, you've built a measurement instrument. The first thing psychometrics asks of any instrument is whether it's reliable: does it produce consistent results?

For a rating task, that means inter-rater reliability. If you're going to treat human ratings as ground truth, the humans need to agree with each other enough to be worth agreeing with. Two raters who barely line up aren't a benchmark; they're noise wearing a lab coat. There are standard tools for this, including intraclass correlation and Cohen's kappa, and for ordinal scores you want a weighted version so that "rated a 4 instead of a 5" counts as nearly right rather than simply wrong. None of this is exotic. It's a first-week-of-class idea that a lot of eval work skips entirely, usually by reporting a single agreement percentage on a sample too small to mean much.

Reliability also sets a ceiling. An evaluator can't be more valid than it is reliable. If your judges are inconsistent, no amount of clever prompting downstream will rescue the measurement.

Reliability is not validity

Here's the distinction the field most often collapses: an instrument can be perfectly consistent and still measure the wrong thing.

Psychometrics treats validity as a separate, harder question, captured well by Samuel Messick's argument that validity is about whether the interpretation you put on a score is justified. A benchmark is an operationalization of some fuzzy construct: "reasoning," "helpfulness," "safety." High scores get read as "the model is good at reasoning." But the benchmark is a proxy, and proxies drift. When a model trains on data resembling the test, or when the benchmark rewards a surface pattern rather than the underlying ability, you get high reliability and low validity at the same time. The number is stable and the interpretation is wrong.

The psychometric habit here is to ask, out loud and in writing, what construct a benchmark claims to measure, and to gather evidence that it does. Does it cover the content domain? Does it predict an external criterion anyone cares about? Those questions feel slow. They're also the difference between a leaderboard and a measurement.

Fairness is not a new problem

The part that surprises people most: the legal and statistical groundwork for "is this scoring system fair across groups" was largely laid decades ago, in employment testing.

Because hiring tests in the United States have been subject to legal scrutiny since the 1970s, I/O psychology has spent generations on adverse impact: the four-fifths rule, group difference statistics, and the careful separation of "the test predicts equally well across groups" from "the groups score equally." More relevant to AI than any of that is measurement invariance, the formal test of whether an instrument means the same thing for different groups of people. Differential item functioning asks whether a specific item behaves differently for two equally able groups. That is precisely the question an AI fairness audit should be asking about a benchmark or an automated scorer, and most don't, because the people building them were never taught the concept existed.

AI fairness research is rediscovering a lot of this under new names. The rediscovery would go faster with the people who already have the vocabulary.

What this looks like in practice

You don't need a psychometrics degree to adopt the discipline. A few habits go a long way:

• Pilot your rubric before you trust it, and measure how well your raters (human or model) agree.
• Report reliability alongside performance, not instead of it.
• State the construct each eval claims to measure, and at least one piece of evidence that it does.
• When the eval informs a decision about people, check for measurement invariance across groups, and treat a gap as a question to investigate rather than a number to bury.

These steps are cheap relative to the cost of trusting a measurement that doesn't hold.

Who should be in the room

AI evaluation is one of the most important problems in the field right now, and it's a measurement problem dressed in machine-learning clothes. The people who have spent careers measuring unobservable human qualities reliably, validly, and defensibly have a great deal to contribute, and at the moment most of them aren't in the room.

I think that's a mistake worth fixing, and it's the work I'm trying to do.

Amira Ghazy is an I/O psychologist with a focus on advanced quantitative methods, working at the intersection of measurement science and applied AI. Project code and portfolio: amira-ghazy.github.io