Universal approximation theorem: what it promises and what it does not

What the theorem says

The universal approximation theorem is one of the most quoted results about neural networks. Proved by Cybenko in 1989 and Hornik in 1991, it says that a network with a single hidden layer, given enough neurons in that layer and a suitable activation (the small function each neuron applies), can imitate any continuous relationship, meaning any curve without sudden jumps, within a limited range of inputs, as closely as you like.

That is a real and important result. It tells you the basic design of a neural network is not what holds it back: with enough units, a good enough answer exists.

What the video shows

The video contrasts the promise with practice. The famous theorem says a simple network can mimic any unbroken curve. It does not say you will find that network: in the course, one training run in four never solved a puzzle with just four cases.

What the theorem leaves out

It is often quoted as if it said far more. It is silent on three things:

• How many neurons you need. "Enough" can mean an astronomically large number.
• Whether training can find the right settings. The theorem proves a solution exists. Training starts from random settings and walks downhill step by step, and nothing guarantees it arrives.
• How the network behaves on new data. Matching the examples you have says nothing about examples it has never seen.

An everyday example

Imagine hiking down a hilly landscape in thick fog, trying to reach the lowest valley. A map proves the deepest valley exists. Useful, but in the fog you can only feel the slope under your feet. You may walk into a small hollow where every direction goes uphill, and stop there. The map was right. You still did not get there.

The experiment in the course

The course takes the XOR puzzle (four cases; the answer is yes when exactly one of two inputs is on). For each network size it runs 50 trainings of 1,000 steps, each from a different random start. Only the number of hidden neurons changes:

| Hidden neurons | Runs that got all 4 cases right | |---|---| | 2 | 38 of 50 (76%) | | 3 | 49 of 50 (98%) | | 4 | 50 of 50 (100%) | | 8 | 47 of 50 (94%) |

Three lessons sit in that table:

1. Two neurons are enough, as the 38 successes prove. Yet about one run in four gets stuck in a spot it cannot walk down from, the foggy hollow from the example.
2. One extra neuron nearly removes the failures, not because the network became more capable, but because an extra dimension gives training more directions to escape through.
3. Eight neurons do slightly worse than four. With the same step size and the same number of steps, a bigger network is not automatically a better one.

Why it matters

"A solution exists" and "training will find it" are different claims. A common piece of advice says a stuck network just needs to be bigger; the table backs that only in its middle rows, from two neurons up to four, and it fails at eight. The theorem tells you the destination exists. Getting there is where the real engineering work happens.

Learn it step by step in Chapter 5 of our free course AI From Scratch: Backpropagation From Scratch: The Engine, Then the Network.