Tree of thoughts: how an AI searches through ideas, and what it costs

What tree of thoughts is

Tree of thoughts is a way of getting an AI model to search for an answer instead of writing it in one straight line. The model proposes a few possible next steps, scores how promising each one looks, keeps going down the best ones and drops the rest. If a path hits a dead end, it can back up and try another branch. Drawn on paper, the attempts form a tree: one starting point, a branch for each option, and smaller branches for each follow-up.

It grows out of a simpler idea: asking the model the same question several times and voting on the final answer. Those attempts run separately from start to finish. Tree of thoughts works on half-finished attempts instead, and decides at every step which ones deserve more work.

What the video shows

The video's hook is an AI exploring a maze of ideas. Its message: for puzzles and planning, an AI can branch out, score each idea and follow the best ones. It works, but a single answer can take dozens to hundreds of AI calls.

An everyday example

Imagine planning a weekend trip. You list three cities. For each one, you check trains. For the cities with good trains, you check hotels. If every hotel in one city is full, you do not start the whole plan again: you step back to the last good choice and try the next city. That is branching, scoring and backing up, which is exactly the kind of problem tree of thoughts is built for.

Where it shines

The course places it in problems that need search:

• puzzles, where some moves lead nowhere;
• planning, where an early choice limits the later ones;
• anything where you may need to undo a step and try again.

For a question with one straight path to the answer, a tree adds little but cost.

What it costs

Cost is the reason it is not the default. Each point in the tree needs at least one call to the model to grow it, and often a second call to judge how good it is. And the number of points grows fast: a tree that considers a few options at each step, several steps deep, multiplies them at every level. For example, 3 options at each of 4 steps already allows up to 81 points. In the published experiments, one answer took anywhere from dozens to hundreds of calls to the model.

Those calls hurt in ways a longer answer does not. One long answer is one request with a predictable wait. A hundred-call tree means:

• a hundred round trips to the server;
• a hundred chances to hit a rate limit, the cap a service puts on how many requests it accepts in a given time;
• a total wait that adds up, because many calls cannot start until an earlier one has finished.

The course treats that last point as an engineering problem about speed, not a question of how clever the model is.

Why it matters

Tree of thoughts is a good reminder that smarter strategies are not free. For a hard puzzle or a plan full of dead ends, searching can be worth it. For everyday questions, the bill in calls and waiting time usually outweighs the benefit. The useful question to ask first: does my problem really need backtracking?

Learn it step by step in Chapter 12 of our free course AI From Scratch: Chain of Thought, RLVR and Test-Time Compute, Measured.