Research note | Checked October 4, 2026
Keep the cases behind the pilot number.
A headline result cannot show what was missing. Retain the agreed cases, every supplied observation and the decision that a responsible human actually made.
What the primary source says.
NIST's 2024 generative-AI profile proposes context-specific measurement, sharing test results with release decision makers, avoiding broad claims from narrow assessments and checking output sources. It is a voluntary framework companion, not an institutional acceptance. Read NIST AI 600-1, Measure 2.3 and 2.5.
A missing case must remain visible.
Our recommendation: agree the case register independently of the results. Then account for each supplied row as a mapped observation, an owned exclusion or an unresolved record. Keep repeated attempts rather than silently picking the best result.
Consider a fictional pilot with four required cases and results for three. A clean summary of those three says nothing about the fourth. Keep its missing result in the file; do not convert it into a success or make it disappear.
Keep evaluation and acceptance separate.
A provider result marked Pass records the evaluator's output under that method. It is not the institution reviewer's answer-quality judgment, source permission or deployment approval.
If effort is compared, measure the same task and include preparation, operation, review, correction, re-entry and export. A faster model response alone does not measure that work. Keep incomplete observations out of an eligible comparison and retain them in the report.
The useful retained file.
Required cases, complete source rows, result methods, explicit mappings, failures, owned exclusions, missing records and pending human reviews should be inspectable together. That is our method recommendation, not a guarantee of safety or a new regulatory obligation.
Inspect the existing supplied-export preparation method Read the pilot review scope
A file to inspect. A decision your institution owns.
Public tools run on supplied inputs and fictional examples. An engagement uses only sources you are allowed to share. We do not connect to a core, compile or execute institution code, take a financial action, certify compliance, give a legal opinion or claim achieved savings.