The Thinking Machine Chronicles #0014: Neurons That Fire Together Wire Together: Hebb's Learning Rule

Era 1 · The Foundations (1936–1955) A psychologist at McGill University proposes, in one sentence, the mechanism by which biological brains learn. The sentence is so simple a child can memorise it. Its consequences took fifty years to fully unfold, and are still unfolding.

The World in 1949

The year 1949 was the year the postwar world hardened into the Cold War world. On 4 April, twelve nations signed the North Atlantic Treaty in Washington, creating NATO, the alliance that would define Western security for the next seven decades. On 29 August, the Soviet Union detonated its first atomic bomb in a remote Kazakh steppe, ending the American nuclear monopoly four years earlier than Western intelligence had estimated; the arms race that followed would consume trillions of dollars and shape every geopolitical decision until the Soviet collapse. On 1 October, Mao Zedong stood at Tiananmen Gate and proclaimed the People's Republic of China, completing the revolution that had begun in 1927 and reorganising a quarter of the world's population under a new government. On 8 June, George Orwell's Nineteen Eighty-Four was published, a dystopian novel written from a Scottish island by a dying man that named the century's darkest tendencies,surveillance, doublethink, the memory hole, with a precision that seemed to derive from supernatural foresight. The Berlin Blockade, which had begun in June 1948, ended on 12 May 1949 after the airlift demonstrated that the West would not be coerced.

The cognitive sciences were undergoing their own revolution. Alan Turing had published his paper on "Intelligent Machinery" in 1948, describing learning machines based on modifiable networks. John von Neumann was writing The Computer and the Brain, unfinished at his death. Norbert Wiener's Cybernetics had put feedback and communication at the centre of discussions about mind and machine. Into this ferment came a 44-year-old Canadian psychologist named Donald Hebb, whose book The Organization of Behavior proposed a mechanism for learning at the level of individual synapses, a mechanism so local, so simple, and so biologically grounded that it would survive intact into the age of deep learning.

Hebb's Obsession

Donald Olding Hebb had spent much of the 1940s thinking about a problem that no one had solved: how does repetition create memory? It was empirically obvious that the brain learned by experience and that learned patterns persisted. It was equally obvious that this persistence had to be implemented in the physical structure of the brain, in the connections between neurons. But how? What changed when you learned something, and why did the change persist?

The dominant theory at the time was that memory was stored in reverberating circuits, loops of neurons that kept firing indefinitely, maintaining a pattern in the manner of an electrical oscillator. Hebb thought this was wrong for storage of long-term memories, which persisted even when the brain was cooled to the point of electrical silence. Something structural had to change, not just something functional.

His answer was synaptic modification. When a presynaptic neuron $i$ and a postsynaptic neuron $j$ fire at nearly the same time, the synaptic connection between them should be strengthened. In Hebb's own words, from The Organization of Behavior (1949):

"When an axon of cell A is near enough to excite cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased."

This is the Hebb rule. In the mathematical form that became standard through the work of Rosenblatt, Hopfield, and Rumelhart:

$\Delta w_{ij} = \eta \cdot a_i \cdot a_j$

where $w_{ij}$ is the synaptic weight from neuron $i$ to neuron $j$, $a_i$ and $a_j$ are their activation levels (firing rates), and $\eta$ is a learning rate. Weights grow when both neurons are active simultaneously.

The Mathematics of Hebbian Learning

For a network of $n$ binary neurons with activation vector $\mathbf{x} \in \{0, 1\}^n$ (or $\{-1, +1\}^n$ in the bipolar convention), the weight matrix update for a single pattern $\mathbf{x}^{(\mu)}$ is:

$\Delta W = \eta \cdot \mathbf{x}^{(\mu)} \left(\mathbf{x}^{(\mu)}\right)^T$

with the diagonal set to zero (a neuron does not connect to itself). For $p$ patterns stored sequentially:

$W = \frac{1}{n} \sum_{\mu=1}^{p} \mathbf{x}^{(\mu)} \left(\mathbf{x}^{(\mu)}\right)^T$

This is the outer product rule. It is the oldest and simplest form of unsupervised learning, no error signal, no target, no teacher. The weights are shaped entirely by the co-occurrence statistics of the input patterns.

Retrieval and Associative Memory

Given a stored weight matrix $W$ and a noisy or partial cue $\tilde{\mathbf{x}}$, a Hebbian network retrieves the closest stored pattern by iterating:

$x_j \leftarrow \text{sign}\!\left(\sum_i w_{ij} x_i\right)$

This is pattern completion, the network fills in missing bits of a partially observed pattern. If the stored patterns are sufficiently dissimilar and the number of stored patterns is small relative to $n$, the network converges to the nearest stored memory.

The capacity limit is a critical result. For $n$ neurons storing $p$ patterns with bipolar activations, the network recalls correctly when the noise in the cue $\epsilon < \epsilon^*$, where the critical capacity is approximately:

$p_{\max} \approx 0.138 \cdot n$

This result, due to Hopfield (1982), who extended Hebb's rule into the Hopfield network,tells us that a network of 1,000 neurons can reliably store roughly 138 patterns. The capacity scales linearly with network size. Beyond $p_{\max}$, patterns begin to interfere, and the network produces spurious memories, attractors that correspond to no stored pattern but to a mixture of several.

Hebb's Rule and Oja's Normalisation

Pure Hebbian learning has an instability: weights grow without bound. If both neurons are consistently active, $w_{ij}$ grows without limit. Oja's rule (1982) adds a weight decay term that enforces normalisation:

$\Delta w_{ij} = \eta \left( a_i a_j - a_j^2 w_{ij} \right)$

At equilibrium, this converges to the principal eigenvector of the input covariance matrix, the direction of maximum variance. Oja's rule is, in other words, a local Hebbian algorithm that performs principal component analysis. It was one of the first demonstrations that a biologically plausible local learning rule could implement a globally meaningful statistical computation.

The Code: Hebbian Learning Network

The companion project implements Hebbian learning in pure Python: the outer-product weight rule, bipolar pattern storage, associative recall with noise, and Oja's normalised rule for comparison. The core storage and recall loop:

def store(self, pattern: list[int]) -> None:
    """Hebbian outer-product update for one bipolar pattern."""
    x = self._to_bipolar(pattern)
    for i in range(self.n):
        for j in range(self.n):
            if i != j:
                self.W[i][j] += x[i] * x[j] / self.n

def recall(self, cue: list[int], steps: int = 10) -> list[int]:
    """Synchronous update until convergence or step limit."""
    state = self._to_bipolar(cue)[:]
    for _ in range(steps):
        new_state = [
            1 if sum(self.W[i][j] * state[j] for j in range(self.n)) >= 0 else -1
            for i in range(self.n)
        ]
        if new_state == state:
            break
        state = new_state
    return [1 if s == 1 else 0 for s in state]

The full project stores multiple patterns in a 100-neuron network, retrieves them from noisy cues at varying corruption levels, plots the capacity limit empirically, and compares pure Hebb with Oja's normalised rule.

Why It Mattered

Hebb's rule mattered because it was local and biologically plausible. A synapse only needs to know the activity of the two neurons it connects, it does not need a global error signal, a teacher, or knowledge of the network's output. This locality made Hebbian learning a serious candidate for how biological brains actually work, which in turn made it a blueprint for artificial networks.

The direct lineage is clear. The perceptron learning rule (Rosenblatt, 1958) is a supervised variant of Hebb: weights increase when input and output co-activate in the desired direction. The delta rule (Widrow-Hoff, 1960) generalises this with an error signal. Backpropagation (Rumelhart, Hinton, Williams, 1986) extends the delta rule through multiple layers. Every one of these algorithms has the form $\Delta w_{ij} = f(a_i, a_j, \text{error})$, the Hebb update with an error correction appended. Without Hebb's 1949 formulation, the conceptual vocabulary for all of them would have been missing.

Hebb also introduced the concept of the cell assembly, a group of neurons that form a closed loop, fire together in response to a stimulus, and persist as a coherent unit after the stimulus is removed. The cell assembly is the neural substrate of a percept, an idea, a memory. Modern neuroscience has confirmed its existence under the name ensemble coding, and the study of hippocampal place cells and cortical columns is largely the study of cell assemblies in vivo.

What Came Next

Hebb gave learning a mechanism at the synaptic level. The next question was whether intelligence itself could be formalised, whether a machine could demonstrate behaviours we recognise as intelligent, and whether "thinking" was even a coherent concept to apply to a machine. In 1950, Alan Turing,who had been working on computing machinery at the NPL and then at Manchester, within walking distance of the Baby's stable, published the most provocative question in the history of artificial intelligence: Can machines think? His answer was indirect, operational, and remains disputed to this day. That is the next story: The Thinking Machine Chronicles #0015: Can Machines Think? Turing's Imitation Game and the Test That Changed Philosophy.

---

References

1. Hebb, D.O. (1949). The Organization of Behavior: A Neuropsychological Theory. Wiley. The source text; Chapter 4 contains the synaptic modification rule.
2. Hopfield, J.J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8), 2554–2558. The paper that turned Hebb's rule into a rigorous associative memory model.
3. Oja, E. (1982). A simplified neuron model as a principal component analyser. Journal of Mathematical Biology, 15(3), 267–273. The normalised Hebbian rule that converges to the principal eigenvector.
4. Siegelbaum, S.A. & Bhatt, D.L. (2013). Synaptic Plasticity, Long-Term Potentiation, and Long-Term Depression. In Principles of Neural Science (5th ed., ed. Kandel et al.). McGraw-Hill. The modern neuroscience of LTP, the physical mechanism underlying Hebb's rule.
5. Dayan, P. & Abbott, L.F. (2001). Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems. MIT Press. Chapter 8 covers Hebbian learning, Oja's rule, and pattern completion in depth.
6. Brown, T.H. et al. (1990). Hebbian Synapses: Biophysical Mechanisms and Algorithms. Annual Review of Neuroscience, 13, 475–511. The bridge between Hebb's psychological theory and LTP at the molecular level.