Series 1 left us with a deliberately vague definition: a model is “a general set of formulas that depends on a set of free parameters”. Vague because the real question was left open — which formulas? With three input variables you can still look at the data, get a sense of the shape of the relationship, and pick a family of functions by hand. With a million input variables you can’t. You need a generic structure, one able to take on any shape without anyone having to specify it.
The answer that prevailed — to the point that in deep learning it is by now essentially the only one in use — is the neural network.
Nodes and layers. A neural network is made of elementary processing units — artificial neurons, or nodes — organized into layers and joined to one another by weighted connections: each connection carries a number and multiplies it by a constant, the weight. The layers come in three kinds:
- the input layer receives the N variables of the prediction problem — in the example from the first chapter: age, gender, body mass index;
- one or more hidden layers process the information. Hidden because they face neither the input nor the output: no one observes them, and what they represent has, in general, no human-readable interpretation;
- the output layer returns the M predicted variables — life expectancy, total health-care cost.
In the simplest topology, information flows in a single direction, from input toward output (a feedforward network), and every node in a layer is connected to all the nodes in the next one (a fully connected network). Many others exist, and they are the subject of chapter 10.
The name is an analogy, and it is worth knowing how far it goes. The textbook definition says that a neural network is inspired by the structure and functioning of the human brain, and that is true as a matter of genealogy: the artificial neuron descends from the formal model of McCulloch and Pitts (1943) and from Rosenblatt’s perceptron (1957), both explicitly inspired by the neurophysiology of their day. But the limit of the analogy should be set straight away: an artificial neuron is not a model of the biological neuron, and a neural network does not explain how the brain works. It is a kinship of inspiration, not of substance.
Where the weights live. On the connections — and this is where everything ties back to chapter 3. Those weights are exactly the free parameters that training has to determine by minimizing the loss. Counting them is immediate: between a layer of n nodes and one of m nodes, if they are fully connected, there are n×m connections, hence n×m weights — plus one parameter per node, the bias, which we will come to in the next chapter. It takes only a few layers of decent size for the count to reach the millions; in the language models we will discuss later on, it runs to hundreds of billions.
This is where “deep” takes on its meaning. In chapter 2, deep learning appeared as “machine learning with multiple processing layers”: the depth in question is simply the number of hidden layers. Each additional layer does not just add weights, it adds a step of reprocessing: the next layer no longer works on the raw data, but on a representation already built by the one before it.
Why this architecture won. Two reasons. The first is modularity: the artificial neuron is an elementary building block, always identical to itself, and by combining enough of them you can — at least in principle — represent any relationship between input and output. This is a precise mathematical result, the universal approximation theorem, which however comes with an important caveat: we will return to it in chapter 9. The second reason is less elegant but just as decisive: the layered structure makes it possible to compute the gradient of the loss extraordinarily efficiently, one layer at a time, with an algorithm called backpropagation. Without it, the gradient descent of chapter 3 would remain a fine idea, impractical on models of this size. That is the subject of chapter 8.
One constraint to keep in mind. A neural network takes real numbers as input and returns real numbers as output. Always. That opens a problem that is anything but secondary for a model that has to read and write text — or look at images, or listen to audio. Assigning a number to each word is not enough: what is needed is a numerical representation that carries meaning with it. That is the subject of Series 3.
But first we have to open one of those network nodes and look inside: what, exactly, does a single artificial neuron do?