#203此视频及其文本为英文。
Option order bias: when an AI picks a position, not an answer
A small AI choosing from four options may favour a spot in the list over the right answer. Learn how order bias works and how rotating options keeps it out of tests.
详细说明
What option order bias is
When you ask an AI model to choose from a list, say four categories or four multiple-choice answers, it should decide based on what the question says. Option order bias is when the choice also depends on where each option sits in the list. Move the options around and the answers move with them, even though nothing else has changed.
What the video shows
A small AI is asked to pick one of four options, and it keeps favouring whatever sits in a particular spot. The lesson of the video: rotate the order of the options across your tests, so that you measure whether the model is reading, not which position it happens to like.
The numbers come from the course. The task is sorting customer support tickets into one of four queues, such as "technical" or "billing". With the four queue names always listed in one fixed order, the model answered "technical" 38 times out of 60. With the list frozen in a different order, it answered "billing" 34 times out of 60. Same sixty tickets, same model. Only the order of the list changed, and the favourite answer changed with it.
An everyday example
Imagine a student guessing on a multiple-choice test who, whenever unsure, picks the last option. If the teacher happened to put most correct answers last, that student would look brilliant. If not, hopeless. Neither grade says anything about what the student actually knows. Shuffle the answer positions from question to question and the habit stops distorting the result.
An AI model can carry a similar habit. The difference is that it never tells you it is guessing.
How to spot it and fix it
- Count the answers. If one option comes out far more often than your data can explain, be suspicious before you celebrate or panic.
- Reorder and rerun. Run the same cases with the options in a different order. If the favourite answer follows the position instead of the content, you have found order bias.
- Rotate by default. The course's test bench cycles the four queue names through different positions across its sixty cases, so no queue is always first or always last. Any preference for a position then spreads evenly instead of piling onto one answer.
This is about the order of the answer options. The order of a prompt's own building blocks (rules, data, instruction) is a separate effect with its own story.
Why it matters
Researchers (Zhao and colleagues) named this a calibration problem: the model's confidence is tilted toward certain answers for reasons that have nothing to do with the question in front of it. In the course, this tilt was larger than every other effect the chapter measured. So if you compare two prompts without rotating the options, you may be measuring the list, not the prompt, and pick the wrong winner.
The honest caveat: these numbers come from a small open model with about half a billion parameters (the adjustable internal values a model learns). The course notes that bigger models are more robust to most of these effects. But rotating the options costs nothing. It does not cure the model's habit; it stops that habit, which on this model was bigger than anything else being tested, from deciding your results.
Learn it step by step in Chapter 15 of our free course AI From Scratch: Prompt Engineering, Measured: What Changes the Output.
也可在
本章更多内容
#203
#205Prompt order: the same words, rearranged, can double AI accuracy
#206System prompt: why where you put AI instructions changes results
#207Prompt formatting: how quotes and tags quietly shift AI accuracy
#208Few-shot prompting: how many examples an AI prompt really needs
#209Prompt myths: what tips, pep talks and threats did in a real test
文案由 AI 辅助撰写,基于我们免费课程第 15 章。