The document fits in the prompt. That answers one capacity question, but it does not establish that the assistant will find the passage you need, connect it to the right subject or preserve an important exception.
A longer context window can make useful workflows possible. Evaluating one still requires more than uploading a large file and checking whether the request succeeds. Test what the system does with the information, not just whether it accepts it.
Capacity, retrieval and interpretation are separate checks
First ask whether the relevant material was actually included in the input. Then ask whether the answer used the right part. Finally, ask whether it interpreted that part correctly. A failure at any of these stages can look like “the model forgot,” even though the remedies differ.
For example, a document may have been cut off before reaching the model. Alternatively, the full document may be present but a similar passage may distract the answer. Or the assistant may quote the correct sentence and still miss the condition attached to it. Save enough evidence to distinguish those cases.
What the classic position study showed
The 2023 paper Lost in the Middle: How Language Models Use Long Contexts, by Nelson F. Liu and colleagues, examined multi-document question answering and key-value retrieval. In the models and tasks studied, changing the position of relevant information could substantially change performance, with relevant material in the middle often harder to use.
That is evidence about the evaluated systems and settings, not a performance forecast for every model available today. It motivates a useful testing question: does your own workflow depend on where the relevant passage appears?
A position test you can actually inspect
Create a small synthetic document set with no private records. Include one target passage and several plausible distractors. Keep a separate answer key. Make versions that put the same target near the beginning, middle and end while preserving its wording and the task.
Here is a fictional target passage:
Project Ember’s pilot is planned for November 18, subject to approval. The final launch date has not been agreed.
A distractor might give Project Cedar a confirmed date. Another might mention an Ember draft deadline. Ask two questions: “What date is proposed for Ember’s pilot?” and “Is Ember’s final launch date confirmed?” The first answer should retain the approval condition; the second should preserve the unresolved launch date.
Neither question requires a model to remember a real business event. The examples are invented to make the expected answer inspectable.
Keep the comparison fair
Use the same model version, output instructions and scoring rules across the variants. Record which text was supplied and which part supports the answer. If you change the prompt while moving the passage, you will not know which change mattered.
Try more than one question and repeat runs where appropriate. A single successful response is a demonstration, not a reliable estimate of performance. Also include a missing-information version. A system should not manufacture a launch date simply because your other examples contained dates.
| Record | Why it matters |
|---|---|
| Target position and input version | Shows which context the model received |
| Expected answer and supporting passage | Makes the scoring rule explicit |
| Answer, qualification and source reference | Separates retrieval from interpretation |
| Failure category and repetition | Avoids hiding distinct errors in one score |
Do not stop at finding one sentence
A simple lookup test checks only one kind of use. Real work may require comparing two revisions, following an exception across sections or synthesizing evidence from several files. Add those tasks separately and give each its own expected result. Passing the lookup does not establish that the harder task will succeed.
If you use a retrieval system to select passages before answering, evaluate its selection as well. An answer cannot reliably use a relevant passage that the surrounding workflow never supplied. The retrieval-versus-fine-tuning guide explains why diagnosing the failure should precede an architecture change.
Choose context deliberately
Removing irrelevant duplication, identifying authoritative versions and asking a bounded question can make a workflow easier to inspect. These are proposed workflow improvements, not a guarantee that shorter input always wins. Compare them on the tasks you actually need.
For the final answer, use the source-checking worksheet. For a document summary, also check important omissions. A capacity number tells you what may fit; evidence from your own task tells you whether the system used it well.
AI-generated conceptual illustration. The test protocol is an editorial proposal; no new model benchmark was run for this article.
