Self-consistency: ask an AI several times, then trust the majority

What self-consistency is

Self-consistency is a simple idea. Instead of asking an AI model a question once, you ask it several times, let it write out its reasoning each time, and keep the answer that comes up most often. The hope is that wrong answers wander off in different directions while right answers agree with each other, so a majority vote filters out the mistakes.

To get different attempts, you turn up the model's temperature, a setting that adds randomness to its word choices. With no randomness at all (called greedy decoding, because the model always grabs its single most likely next word), asking again would just give you the same answer every time.

What the video shows

The video starts from a simple recipe: ask an AI five times and go with the majority. Then it gives the catch. Voting can fix mistakes, but it costs several times more than a single answer, and sometimes one careful try was already right. The course's numbers behind that message are below.

An everyday example

Imagine a pub quiz. Your team's most careful player works the question out and gives the right answer. Now picture another approach: you ask five teammates who are each a little distracted, so every one of them slips now and then, and you go with the most common answer. The slips mostly point in different directions, so the majority usually lands on the right answer, but only once you have heard from several people. You spent five people's effort to end up where the careful player already was. The vote earns its keep on the night your careful player is confidently wrong and the rest of the table is not.

What the course measured

The course tried it on 16 easy two-step word problems with a small model, with randomness set to a temperature of 0.8:

| Attempts per problem | Correct | Tokens per problem, all attempts | |---|---|---| | 1 | 81% | 185 | | 3 | 100% | 526 | | 5 | 100% | 871 |

(A token is a word or a piece of a word, so tokens are a fair measure of cost.)

Two lessons stand out.

• It levels off early. By the third attempt, accuracy was already at its ceiling. The fourth and fifth attempts added cost and nothing else.
• One careful try was already perfect. In the chapter's opening test, greedy decoding solved these problems 100% of the time with a single chain of about 145 tokens. Adding randomness dropped a single attempt to 81%. Voting then needed three attempts, about 526 tokens, to climb back to where that one careful attempt already was: about 3.6 times the cost, or about six times if you run all five attempts without knowing where the curve flattens.

The course adds a fair warning about its own numbers: 16 problems is a small sample, so the margins of error around 81% and 100% overlap. Trust the shape of the curve, not the exact attempt where it flattens.

When it pays off

Randomness buys variety by adding errors, and voting removes those errors again. That trade is worth making when the model's most likely answer is the wrong one and a less likely line of reasoning is right. That is the situation the technique was designed for. When the careful single answer already works, you pay several times more to break even.

Why it matters

Self-consistency is a real and useful technique, but "more attempts means better answers" is not automatic. The cases where it only adds cost rarely get written up, so the honest move is to test it on your own task and compare it against one careful attempt before you pay for it.

Learn it step by step in Chapter 12 of our free course AI From Scratch: Chain of Thought, RLVR and Test-Time Compute, Measured.