The Perceptron From Scratch: What a Neuron Computes

A perceptron takes the measurements, multiplies each by a number it controls, adds them up, adds one more number, and looks at the sign.

Write the measurements of one part as a vector $\mathbf{x} = (x_1, x_2)$ — width and weight. The perceptron holds a weight vector $\mathbf{w} = (w_1, w_2)$ and a bias $b$. Its score is

and its answer is the sign of that score: accept if $s(\mathbf{x}) \geq 0$, reject otherwise.

That is the entire model. Everything the perceptron will ever know about the factory lives in three numbers.

The geometry is worth pausing on, because it is the picture that keeps working for the next twenty-nine chapters even when the equations stop fitting on a line. The set of points where $s(\mathbf{x}) = 0$ — where the perceptron is exactly undecided — is a straight line in the plane. On one side the score is positive and everything is accepted; on the other it is negative and everything is rejected. Learning, for a perceptron, means moving that line.

Two facts about that line follow directly from the algebra, and both matter later:

• $\mathbf{w}$ is perpendicular to it. The weight vector does not lie along the boundary, it points across it, toward the accepted side.
• $b$ slides it without turning it. Without a bias the line would be forced through the origin, which for a factory measuring millimetres and grams would be an absurd constraint — it would mean a part of zero width and zero weight sits exactly on the fence.

The perceptron starts knowing nothing: $\mathbf{w} = (0, 0)$ and $b = 0$. Every score is zero, so it accepts everything.

Now show it one example at a time. Label the accepted parts $y = +1$ and the rejected ones $y = -1$. For each example, ask one question: did the sign come out right? The compact way to write that question is to check whether $y \cdot s(\mathbf{x})$ is positive — if the label and the score agree in sign, their product is positive, and if they disagree it is negative.

If the answer is yes, change nothing. If the answer is no, nudge:

That is the whole algorithm, and it is worth understanding why it is the right nudge rather than memorising it. Suppose a part should have been accepted ($y = +1$) and the score came out negative. Adding $\mathbf{x}$ to $\mathbf{w}$ changes the score on that same part by

which is a positive number. The score on the part it just got wrong goes up, which is the direction it needed to go. The rule is not a heuristic somebody guessed; it is the smallest change that provably improves the case in front of it. It may of course break a different case, which is why you go round again.

Notice what is absent. There is no derivative anywhere. This is not an oversight, and it is the first genuinely important idea in the course.

The thing you would want to differentiate is the error — the count of misclassified parts. But that count is a staircase: it sits flat at 4 while you nudge the line, then drops to 3 the instant the line crosses a point. Its derivative is zero almost everywhere and undefined at the steps. Calculus has nothing to grip. The perceptron rule works around that by not asking for a slope at all: it asks only "right or wrong?", and moves in a direction it can justify geometrically.

That is a genuine solution, and it is also a dead end. In Chapter 2 we will want a loss that comes from somewhere rather than being chosen, in Chapter 4 a model that reports how sure it is, and in Chapter 5 something with more than one layer — and neither is reachable from a rule that only knows "wrong". Getting a usable slope back is what forces the next two chapters. But the perceptron gets to do something none of its successors can: learn without calculus at all.

Pure Python, no NumPy. Lists and a loop. NumPy arrives in the next chapter, where the arithmetic stops fitting in a loop you would want to read; introducing it now would hide the arithmetic behind a library at exactly the moment you want to see it.

def score(w, b, x):
    return w[0] * x[0] + w[1] * x[1] + b


def predict(w, b, x):
    return 1 if score(w, b, x) >= 0 else -1


def train(data, epochs=200):
    """Returns (w, b, epoch_it_converged) — or None for the epoch if it never did."""
    w, b = [0.0, 0.0], 0.0
    for epoch in range(epochs):
        mistakes = 0
        for x, y in data:
            if y * score(w, b, x) <= 0:          
                w[0] += y * x[0]                 
                w[1] += y * x[1]                 
                b += y                           
                mistakes += 1
        if mistakes == 0:
            return w, b, epoch + 1
    return w, b, None

The four highlighted lines are the algorithm. Everything else is bookkeeping.

And the belt, with eight parts measured off it — four that shipped and four that came back:

BELT = [
    ((18.0, 47.0), +1), ((19.5, 52.0), +1), ((20.2, 49.0), +1), ((21.0, 55.0), +1),
    ((24.0, 61.0), -1), ((25.5, 66.0), -1), ((23.0, 70.0), -1), ((26.0, 58.0), -1),
]

w, b, epoch = train(BELT, epochs=200)
print(epoch, w, b)

These eight parts are separable by a straight line — every accepted part is under 22 mm and every rejected one is 23 mm or more. A vertical fence at 22 millimetres does the job. So the perceptron should find it.

Run it:

None [-142.1, -13.0] 54.0

Two hundred epochs, 454 corrections, and it has not converged. The weights are large and the wrong sign. Something is wrong — except that nothing is wrong, and the reason is the most useful thing in this chapter.

The perceptron has a guarantee, proved by Novikoff in 1962.¹ If the data can be separated by a line at all, the algorithm makes at most

corrections before it stops making any — where $R$ is the radius of the data, the length of the longest example vector, and $\gamma$ is the margin: the distance from the separating hyperplane to the closest point in the augmented space where the bias is a third coordinate. That is why centring the data changes it while the distance in millimetres does not.

The guarantee is unconditional and it does not mention epochs, learning rates, or luck. It also does not mention time, and that omission is the point.

Put our numbers in. Measured directly from the eight parts, with the bias folded in as a constant feature:

The theorem was never violated. Run the raw version for long enough and it does converge — at epoch 11,976, after 29,870 corrections — comfortably inside its bound of 2,633,550, and that gap is itself the point: the theorem bounds the worst case, not the typical one. It simply needed sixty times more epochs than anyone would sit through.

The second row is the same eight parts, the same twenty lines of code, with three lines added to subtract the mean width and the mean weight from every measurement. That is it. That is the entire change. It moves the cloud of points so it straddles the origin instead of floating out at (22, 57), and the effect on the bound is a factor of fifteen thousand, because both terms improve at once: $R$ falls from 74 to 13 because the points are no longer measured from a far-away origin, and $\gamma$ rises from 0.045 to 0.989 because the margin is measured against a weight vector that no longer has to carry a huge bias to reach the data.

mean_w = sum(x[0] for x, _ in BELT) / len(BELT)   # 22.15
mean_g = sum(x[1] for x, _ in BELT) / len(BELT)   # 57.25
CENTRED = [(((x[0] - mean_w), (x[1] - mean_g)), y) for x, y in BELT]

w, b, epoch = train(CENTRED, epochs=200)
print(epoch, w, b)

2 [-4.15, -10.25] 1.0

Converged in two epochs, having corrected itself exactly once.

There is a real lesson here and it is not "remember to normalise your inputs", although you should. It is that a guarantee about whether an algorithm finishes tells you nothing about whether you will be there when it does, and that the gap between the two is usually geometry. This is the first appearance of a pattern you will meet again in Chapter 6 with initialisation, in Chapter 10 with learning-rate schedules, and in Chapter 13 with quantisation: the mathematics says the thing is possible, and the engineering decides whether it is practical. A course that teaches you only the theorem hands you a model that trains for three days and blames you.

Now the failure that ended the first era of neural networks, and it fits in four rows.

Forget the factory. Take two inputs that are each either 0 or 1, and ask for the answer to be $+1$ when exactly one of them is 1:

This is XOR — exclusive or. Before reading on, draw the four points on paper: three corners of a unit square and the fourth. Mark the two diagonal corners $(0,1)$ and $(1,0)$ as accept, and $(0,0)$ and $(1,1)$ as reject. Now draw one straight line with the two accepted points on one side and the two rejected points on the other.

You cannot. It is not that it is hard, or that you need a cleverer algorithm; it is that the line does not exist. Three lines of algebra show why. If a perceptron got all four right, then reading the four rows in order gives

Add the middle two inequalities: $w_1 + w_2 + 2b \geq 0$, so $w_1 + w_2 \geq -2b$. The last one says $w_1 + w_2 < -b$. Together: $-2b \leq w_1 + w_2 < -b$, which requires $-2b < -b$, which requires $b > 0$. And the first inequality says $b < 0$. There is no such $b$, so there are no such weights. No perceptron, with any numbers whatsoever, classifies XOR.

Run it anyway, because watching an algorithm fail is worth more than being told it will:

     100 epochs -> converged=None  w=[0.0, 0.0] b=0.0  correct=2/4
   1,000 epochs -> converged=None  w=[0.0, 0.0] b=0.0  correct=2/4
 100,000 epochs -> converged=None  w=[0.0, 0.0] b=0.0  correct=2/4

It does not diverge, and it does not thrash around near a decent answer. It cycles: it walks a short loop through weight space and comes back to exactly where it started, forever, getting two of four right — which is what you would get by guessing. A hundred thousand epochs and a hundred are indistinguishable, because the algorithm is not making progress that a longer run could finish. Compare that to the belt, which looked stuck at 200 epochs and was in fact grinding toward a real answer. From the outside the two look similar for the first few seconds. Telling them apart, without the theorem, is impossible — which is one more argument for knowing the theorem.

In 1969 Marvin Minsky and Seymour Papert published Perceptrons, a book-length mathematical study of exactly what this model can and cannot represent.² XOR is its most quoted result, and the quotation is usually deployed as an accusation: that the book killed neural network research for fifteen years out of rivalry or spite.

The mathematics in the book is correct, and it is more interesting than the XOR example. Minsky and Papert were not primarily interested in whether a single perceptron could do XOR; they were interested in what happens when perceptrons are given limited receptive fields — each unit seeing only part of the input — and they proved that certain global properties of an image, such as whether a figure is connected, cannot be computed that way regardless of how many units you use. That is a genuinely deep result about locality, and it has nothing to do with the popular story.

The popular story is also wrong on the history. Minsky and Papert explicitly discuss multi-layer perceptrons and say that the question of their power is open — they suspected extending the theory would be "sterile", which is a prediction, not a proof, and it was wrong. What was missing in 1969 was not the idea of stacking layers; it was a way to train a stack. The perceptron rule cannot do it: it needs to know how wrong each unit is, and for a unit buried in the middle there is no label to compare against. That gap stayed open until backpropagation was popularised in 1986,³ and closing it is what Chapter 5 does.

So the honest summary is this. The book proved a real limitation of a real model. The field's funding collapse in the seventies had many causes, of which one was that the promises made for perceptrons in the early sixties had been extravagant. And the technical obstacle was solvable, but nobody had the tool yet.

The perceptron is sixty-eight years old and you have just written one. It is worth being precise about which parts of it are still in the machine you will finish this course with, because the answer is: more than you would guess.

The perceptron got stuck on two things at once, and they turn out to be the same thing.

It cannot represent XOR, because one line is not enough. Fixing that means stacking layers — a first layer that bends the space, a second that draws the line in the bent space. That is Chapter 5.

But you cannot train a stack with the perceptron rule, because it only knows "wrong", and a unit in the middle of a network has no label of its own to be wrong about. To train a stack you need to know how wrong, and in which direction, for every weight — you need a slope. And the perceptron's error function, the staircase, does not have one.

So before the stack there has to be a loss function with a usable derivative. Not one chosen because it is convenient to differentiate, either: one that comes from somewhere, that says something true about the data, and whose gradient falls out of that meaning rather than being reverse-engineered to look tidy.

That is Chapter 2, and it starts by asking a question the perceptron never had to answer: not "is this part good?", but "how likely are these readings, if this is the truth?"

Also worth reading alongside this chapter: Rosenblatt's original paper, The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain (Psychological Review 65(6), 1958), which is more readable than its reputation suggests; McCulloch and Pitts, A Logical Calculus of the Ideas Immanent in Nervous Activity (Bulletin of Mathematical Biophysics 5, 1943), the paper that first modelled a neuron as a threshold over a weighted sum; the perceptron section of Hal Daumé III's A Course in Machine Learning, which derives the same update with a different emphasis; and chapters 2 and 3 of Deisenroth, Faisal and Ong's Mathematics for Machine Learning for the linear algebra, if the box above left you wanting more than it gave.