Chapter 12 closed with a promise of surprising properties and limitations. Let’s start with the limitations, because Word2Vec, elegant as it is, carries two structural flaws.
The first is that it assigns a single embedding per word, not per meaning. Take the word “bank”: it can mean a financial institution, or the bank of a river, two completely distinct meanings. Word2Vec doesn’t know that, though: it looks at every sentence where “bank” appears, mixes them all together, and derives a single vector that’s a kind of average between the two senses, unable to represent either one well.
The second limitation is that the model completely ignores word order within the context. Remember the mechanism from chapter 12: the context word vectors get summed. But a sum is inherently commutative, it has no memory of position: “the dog bites the man” and “the man bites the dog” would produce exactly the same context sum, even though the meaning is reversed.
On top of these two Word2Vec-specific limitations sits a more fundamental one, inherited from neural networks in general: the opacity we talked about in chapter 9. Nobody decides in advance what the dimensions of the space we’ve built are supposed to mean: it’s all learned during training, and the result isn’t something immediately intelligible to a human being.
And yet, within these limits, the space that Word2Vec builds hides a surprising property: you can do arithmetic on word vectors, and the result still makes sense. The most famous example: take the vector for “king”, subtract the vector for “man”, add the vector for “woman”. The point you land on in the dense space sits very close to the vector for “queen”.
It’s not an isolated case. The same pattern shows up for a great many other pairs of concepts: geographic relationships, verb tenses, comparatives, and superlatives. It’s as if certain directions inside the space had acquired a precise meaning on their own, one direction representing, in a sense, the concept of “gender”, another the concept of “royalty”, without anyone having established it explicitly. It’s an emergent effect of the geometry that training found, just by looking at contexts.
But that same geometry, which captures such elegant relationships, has a flip side. If the directions inside the space encode real concepts, they end up absorbing the biases present in the texts the model was trained on. A now-classic case, documented by Bolukbasi and colleagues in 2016 and then by Caliskan, Bryson, and Narayanan in Science in 2017: in the space, the vector for “doctor” turns out to be closer to the vector for “man” than to the vector for “woman” — the distance between doctor and man is smaller than the distance between doctor and woman.
The model didn’t invent this bias out of nowhere: it simply learned it, with extreme fidelity, from the way those words appeared together in the training texts. And it’s a finding that has had a curious spin-off: embeddings have stepped outside the boundaries of artificial intelligence and become a research tool in their own right for linguists, who use them precisely to quantitatively measure the biases present in a language or a text corpus, exploiting exactly these geometric distances.
There remains a practical problem, though, and it’s one that goes well beyond academia: if these embeddings end up inside a system that, say, screens candidates for a job, that geometric bias risks translating into real discrimination.
Despite all of this, embeddings remain the foundation of every modern large language model. But an LLM doesn’t reuse Word2Vec’s vectors: what survives is the idea, not those numbers. Every model learns its own embedding table from scratch, together with the rest of the network, and it learns it not over whole words but over pieces of words — which is exactly the subject of chapter 15. That table then lives inside far more complex architectures, able to overcome word order and the ambiguity of multiple meanings as well. The architecture that has become dominant in this area is the Transformer, which we’ll see in Series 4.