Two models answer the same question. One receives the full policy document, and the other receives the first page. The difference in their answers may tell you more about the document pipeline than about the models. Before comparing scores, decide what you are comparing and record what each system actually received.
Choose what the comparison can answer
To compare models under matched conditions, give both the same task, source excerpts and output requirements. Where a setting has no equivalent, record the difference. If you are choosing between complete products, allow each product to use the retrieval and tools you would actually deploy. Record those choices and describe the result as a comparison of the workflows. Both tests can inform a purchase or rollout, but they answer different questions.
NIST’s AI Risk Management Framework calls for documenting test sets, metrics and evaluation tools and measuring performance under conditions similar to deployment. Write down whose work the test represents before deciding which examples to include. NIST AI RMF Core, Measure 2.
Write the expected answer before seeing either model
Suppose a support team wants to draft answers about shipping. In a fictional policy packet, a table says standard domestic delivery usually takes four business days. A note below it says orders placed after noon begin processing the following business day. Ask both models to explain what a customer who ordered at 2 p.m. can infer about timing. The expected answer preserves the cutoff, distinguishes processing from delivery and avoids promising an arrival date. Record the table and note as the supporting passages.
Include a companion case where the cutoff note is absent. Decide in advance what an appropriately qualified answer looks like. This pair helps you spot a model that follows evidence when it is available but guesses when it runs out. For a separate check on whether the cited passage supports the answer, use the source-checking worksheet.
Keep one record per matched pair
| Field | What to save |
|---|---|
| Case | Task, input version and required answer elements |
| Evidence | Exact excerpts and their order for each run |
| Configuration | Model version, date, instructions, settings and tool access |
| Outcome | Each answer, its supporting passages and any omitted condition |
| Decision | Which answer passed, what failed and whether it requires review |
Save the text actually delivered to each model, especially when software trims long inputs. Capture the output before editing it. If a person scores the answers, hide model names where practical and use the same scoring guide for both. Record ties and disagreements so another reviewer can reconstruct the result.
Put consequential mistakes beside the score
Imagine that both models pass eight of ten fictional cases. One drops a cutoff twice. The other asks for missing information twice and therefore fails to complete the task. The counts tie, but the failures have different costs. Decide how many unnecessary clarifications a support team can tolerate, and which unsupported commitments should send a draft to review. Write those rules before the next comparison.
Report the counts by case type and retain the paired examples. A small convenience sample can guide your next test, especially when it shows a repeatable failure, but it gives limited evidence about how often that failure will occur in production. If speed influences the decision, measure waiting, generation and tool work separately and keep the same definition of usable completion.
AI-generated editorial illustration. The shipping policy and scores are fictional worksheet examples.
