A new Microsoft paper reveals that even top engineers are stuck in pre-scientific measurement—and why that should concern every coach and CHRO
In April 2026, a team at Microsoft Research published "The Art of Building Verifiers for Computer Use Agents" (Rosset et al., 2026). It's a careful, transparent, technically sophisticated paper. The team built a system that evaluates whether AI agents successfully completed their assigned tasks. They iterated through 32 experiments. They achieved something impressive: their verifier agrees with human judges about as often as human judges agree with each other.
And they entirely missed the fundamental problem.
This isn't a criticism of their engineering. It's an observation about the measurement theory their engineering assumes—and what that assumption reveals about the state of evaluation, not just in AI, but in every field that makes high-stakes decisions about human capability.
The Microsoft team, like most practitioners in coaching, HR, and AI evaluation, is working in what metrologists would call a pre-scientific paradigm. They're optimizing agreement between observers without first establishing whether those observers are actually measuring something real. They're treating ordinal rankings as if they were measurements. They're assuming that human judges are valid instruments without testing that assumption.
This is exactly the problem that plagues executive coaching, leadership development, and talent decisions in organizations worldwide. And it's why we wrote this piece.
Here's what the Microsoft paper demonstrates without intending to: their system and human raters correlate at about the level you'd see between two managers independently rating the same employee's performance—moderate agreement, meaningful overlap, but plenty of room for disagreement on critical cases.
This is real engineering progress. It is not measurement.
Consider what happens when a CHRO asks whether a $500K leadership development initiative produced measurable growth. "Our raters agreed moderately well" doesn't survive a CFO's scrutiny. The question demands something ordinal classification cannot provide: units of measurement that support arithmetic operations, trend analysis, and process quality control—the same rigor we apply to financial reporting or manufacturing quality.
You can say "our raters agreed." You cannot say "your client developed 0.8 logits to Stage 11.1 on the Harvard Model of Hierarchical Complexity for strategy, with a standard error of 0.15"
Only one of those statements survives a CFO's question.
The Microsoft paper treats human agreement as the gold standard without examining whether those human judges are actually measuring something real.
This isn't a new problem. For three decades, measurement scientists have developed sophisticated methods for evaluating whether human raters contribute useful information at all. Linacre's Many-Facet Rasch Model (Linacre, 1989) allows you to separate rater severity, rater consistency, and the underlying trait being measured—and to identify raters whose judgments are so erratic or biased that they add noise rather than signal. The Multidimensional Random Coefficients Multinomial Logit model (Adams, Wilson, & Wang, 1997) extends this further, enabling you to verify whether your measurement structure is valid before you trust any scores.
The Microsoft team, like most coaching and HR practitioners, assumes raters are valid instruments without testing them. In metrology, that's called pre-metrological—measurement before we had standards for whether our instruments actually measure what we think they measure.
The Universal Verifier paper documents this assumption clearly. The authors report that their system agrees with human labels about as often as humans agree with each other. But they never ask the prior question: are the human labels themselves valid measurements, or merely consistent opinions?
When two thermometers agree, that's consistency. When they agree and they're both calibrated against a traceable standard, that's measurement. The Microsoft paper achieves the former. The coaching and HR fields, by and large, haven't even asked for the latter.
In executive coaching, we face this daily. A client produces results but damages relationships in the process. Another client follows the development plan faithfully but hasn't reached the target capability stage. These aren't binary outcomes. They're positions on a developmental scale—and the distance between them matters.
The Model of Hierarchical Complexity (Commons et al., 1998) defines invariant stages of cognitive and behavioral complexity. A leader at Stage 10 can compare systems. A leader at Stage 12 can integrate multiple paradigms into unifying frameworks. The sequence is fixed: you cannot perform Stage 12 tasks without consolidated Stage 11 operations. This isn't theory—it's a four-decade empirical regularity.
When the Microsoft paper awards "process credit" for correct steps, it's making an implicit judgment about what correctness means. Without anchoring that judgment to a developmental standard, two agents scoring 0.8 on process may be operating at fundamentally different complexity levels—and the rubric cannot tell them apart.
The same is true for every 360 assessment, competency rating, and leadership evaluation in your organization. Two executives both rated "4 out of 5" on strategic thinking may be operating at different developmental stages entirely. The ordinal scale collapses that distinction. Your succession decision doesn't.
The Microsoft paper's two-pass scoring system elegantly catches AI agents claiming things that visual evidence contradicts. This is valuable. It is also insufficient for high-stakes contexts.
In credentialing examinations and succession decisions, the question isn't whether you can catch fabrication after the fact. It's whether your system is structurally resistant to gaming.
Rasch methodology addresses this through fit statistics—indices that flag response patterns that are too consistent (suggesting memorization) or too erratic (suggesting random responding). These aren't post-hoc checks; they're integral to scoring. A person whose responses misfit the model receives a larger standard error, honestly reflecting the measurement system's uncertainty about their true position.
The Microsoft team has no equivalent mechanism. They can detect contradictions between claims and screenshots, but they cannot detect the agent that produces consistently fluent outputs at a complexity level it hasn't actually achieved—the AI equivalent of someone who aces the interview but can't do the job.
This is the same vulnerability that plagues every assessment system that optimizes for agreement without checking for fit. A coaching client who gives socially desirable responses on a 360 will show high agreement between raters who all see the same polished performance—and low fit to a genuine developmental measurement model.
For coaches, I/O psychologists, and CHROs, the gap between agreement and measurement isn't academic. It's the difference between saying "our raters agreed" and "your client developed 0.8 logits on the Strategy dimension, with a standard error of 0.15, and fit statistics confirming valid measurement." Only one of those statements supports the decisions organizations actually make.
What high-stakes evaluation requires—whether of AI agents or human leaders:
Interval-scale measurement with known uncertainty. A score of 3.4 logits ± 0.2 is a measurement. A score of 0.8 on a locally generated rubric is a classification. Only measurements support the trend analysis, quality control, and financial valuation that organizational decision-makers require.
Traceability to published standards. When you report a client at "Stage 11, Step 3 on Strategy," that score is traceable to decades of empirical research. It's citable, defensible, and comparable across contexts. A rubric score generated from a task description is none of these things.
Rater quality diagnostics. Before you optimize agreement with human judges, you need to verify those judges are actually measuring something. The Many-Facet Rasch Model and MRCML model provide this. Assuming raters are valid instruments without testing them is pre-metrological practice—whether you're at Microsoft or in a Fortune 500 HR department.
Structural resistance to gaming. When assessment items are generated fresh from parameterized models rather than drawn from a fixed bank, there's nothing to memorize. This is measurement security by design, not monitoring.
Process capability analysis. Six Sigma quality control requires interval-scale data. You cannot perform it on ordinal classifications. Yet this is exactly what organizations need to determine whether a development program or AI system is producing output within acceptable quality bounds.
Right now, TruMind.ai delivers this measurement to executive coaches and organizations. Our AI Precision Measurement system analyzes coaching session transcripts and produces interval-scale assessments across nine leadership dimensions and all eight ICF Coaching Competencies—with 15× the precision of traditional high-stakes credentialing exams.
Every score carries a standard error. Every assessment includes fit statistics that flag whether the measurement is valid. Every client gets a Goldilocks Zone—the developmental range where challenge produces growth rather than frustration or stagnation.
Coaches use it to prove ROI to sponsors. CHROs use it to make succession decisions with financial-grade evidence. I/O psychologists use it because it solves the rater problem they've known about for decades but couldn't fix with ordinal tools.
This is hyperpersonalized measurement: calibrated to the individual, anchored to a universal standard, updated after every coaching session.
Here's what's coming next.
Every TruMind measurement is traceable to published calibration parameters. Every confidence interval is computed from the standard error of the latent trait estimate. Every dimension score carries measurement uncertainty that a client, coach, or legal proceeding can interrogate.
Now imagine that traceability made permanent. What if every measurement certificate was cryptographically signed, permanently auditable, and impossible to alter after the fact? What if the fit statistics that detect gaming—responses that are too consistent, suggesting memorization, or too erratic, suggesting random responding—were verified not by a single organization's claim, but by a network that couldn't be corrupted?
What if the standard itself—the Model of Hierarchical Complexity, the Rasch calibration parameters, the measurement science that makes interval-scale assessment possible—was embedded in infrastructure that no single party could dilute?
That's the direction we're building. Not because blockchain is trendy. Because the measurement gap costs organizations real money, real talent, and real credibility—and the solution requires trust infrastructure that doesn't depend on trusting us.
The Microsoft paper is a gift. It demonstrates, with careful engineering and transparent reporting, that even the world's best AI researchers are working with pre-metrological tools. They've optimized agreement. They haven't achieved measurement.
This is the same gap that exists in your coaching practice, your leadership development programs, your succession planning. You have rater agreement. You don't have measurement-grade evidence.
If you're an I/O psychologist who's known for decades that ordinal scales can't support the decisions being made with them, we see you.
If you're a coach who dreads the sponsor question because you have anecdotes but not evidence, we've built what you need.
If you're a CHRO who's been asked for financial-grade evidence of leadership development ROI and couldn't provide it—not because you didn't want to, but because the tools didn't exist—you now have them.
The measurement gap isn't inevitable. It's a choice between pre-metrological practice and what's possible when you use instruments calibrated to universal standards with known uncertainty.
If you've hit the ceiling of what ordinal scales can deliver—in coaching, succession, or AI evaluation—we'd like to collaborate. Reach out at [contact information]. The infrastructure for measurement-grade people decisions exists. Let's build what comes next together.
Adams, R. J., Wilson, M., & Wang, W. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21(1), 1–23. https://doi.org/10.1177/0146621697211001
Commons, M. L., Trudeau, E. J., Stein, S. A., Richards, F. A., & Krause, S. R. (1998). Hierarchical complexity of tasks shows the existence of developmental stages. Developmental Review, 18(3), 237–278. https://doi.org/10.1006/drev.1998.0467
Linacre, J. M. (1989). Many-facet Rasch measurement. MESA Press. ISBN: 978-0941938017
Rosset, C., Sharma, P., Zhao, A., Gonzalez-Fernandez, M., & Awadallah, A. (2026). The art of building verifiers for computer use agents. arXiv preprint arXiv:2604.06240v1. https://arxiv.org/abs/2604.06240