There’s a trap hiding in “make the loss as small as possible.” Taken literally, you can almost always drive the loss to zero — just make the model complex enough, with enough weights, and it will be able to reproduce a training-set output exactly from its input. In our simplified example with a single input and a single output, that amounts to saying that as the number of weights grows there will always be a function — the model — that passes exactly through every single point of the training set.
The problem is what happens between the points.
Look at the age/income data again (above), this time fitted with a model more complex than a simple straight line. It passes through every point perfectly — a training loss of zero — but to do so it swings up and down markedly in between. Ask it to predict the income of a 50-year-old, a value it wasn’t trained on, and it produces something absurd. It has memorized the data instead of learning the pattern. This is overfitting.
What we actually want is generalization: good predictions on data the model has never seen. That is the entire point of a predictor. The real skill is finding the right trade-off between fitting the data and generalizing beyond it:
- The more training data you have, the more complex the model can afford to be — and the more accurately it can fit.
- But minimum loss does not necessarily have to mean zero loss.
How do you tell whether a trained model can really generalize? You split the data you have into a training set (80–90%) and a test set (10–20%). You train the model on the training set, then measure performance on the test set — data the model has never seen, but whose real-world outputs we know. If the model predicts the test set accurately enough, it has generalized; if it excels on the training data but fails on the test set, we are looking at overfitting.
This is also why a machine-learning model has two distinct phases: a training phase, where the weights are learned, and a use (or inference) phase, where the “frozen” model makes predictions on new inputs.