How Machines Learn: Weights, Loss, and Gradient Descent — Behind the AI — RI4AI

Three example models mapping the same three inputs — age, level of education, annual income — to two outputs through formulas of different shape, illustrating that a model is just a mathematical relationship between inputs and outputs.

“Training a model” sounds mysterious until you see the machinery. It always comes down to five ingredients and one very concrete goal.

Start with the simplest possible example: one input x (age), one output y (income), and a linear model y = w₀ + w₁x with only two weights. Given a training set (a table of genuinely observed age/income pairs), how do we find the right two numbers?

Initialize. Start with random weights and run the model over every input in the training set (age) to get its predicted incomes.
Measure the error. For each record in the training set, measure the prediction error. There are several formulas for this. The most intuitive is the sum of squared errors: take the difference between the predicted income and the real one and square it, so that overshooting and undershooting count the same and big errors weigh more than small ones. Then add these values up across the whole training set. That single total is the loss — one number measuring the prediction error the model is making with those weights.
Improve. Change the weights to make the loss smaller.

The model is run over a training set of age/income pairs; the squared error on each example is summed into a single loss L to be minimized. — The heart of it: run the model, measure the prediction error on every example, and sum it into a single loss L. The data is just fixed numbers — only the weights are free to change.

The whole of learning is step 3: find the weights that make the loss as small as possible. And here’s the elegant part — because the training data is just fixed numbers, the loss depends on the weights alone. With two weights you can literally plot it as a surface and go looking for its lowest point — its minimum.

A 3D loss surface plotted against two weights, with different weight choices giving very different fits to the age/income data. — With two weights the loss becomes a surface. Different weight choices fit the data very differently; the best pair sits at the minimum of that surface.

You can’t find that lowest point by trying random combinations forever — real models have billions of weights (GPT-3 had 175 billion; Brown, T. et al. (2020), Language Models are Few-Shot Learners). Instead you use an optimization algorithm, and the most widely used one is gradient descent: start somewhere on the surface and repeatedly step in the direction of steepest descent until you reach a minimum — not necessarily the deepest one, but in practice good enough.

The five ingredients of machine learning: a task, a model, a training set, a loss function, and an optimization algorithm. — The five ingredients every machine-learning system needs — a task, a model, a training set, a loss function, and an optimization algorithm. Change the ingredients and you change the application; the recipe stays the same.

Pulling it together, every machine-learning system needs the same five ingredients: a task (regression, classification…), a model with parameters, a training set to learn from, a loss function that quantifies the prediction error, and an optimization algorithm to minimize that loss. Change the ingredients and you change the application — but the recipe never changes. And crucially, the model’s behavior is learned from the data, not written by a person.