Embeddings: The Geometry of Meaning
Give a model a word or a picture and it hands back a list of numbers: a direction. Training points things that mean alike in directions that lie close, so “how alike” becomes “how wide an angle”. This globe is a real three-dimensional space: every picture and tag on it sits where a small model on this page puts it.
Chapters · 12
Scroll
One Direction, One Meaning
Each small picture on the globe is a 16 × 16 pixel image; each brass tag is a two-word caption, like “red circle”. A small model turns each into three numbers, scaled to length 1: a point on the sphere, the tip of an arrow from the centre. A window cut in the globe shows the arrows: each fine steel needle runs from the centre to one point.
The model was trained on this page. Now each picture points almost where its own caption points, and a fine silver wire joins the two. Models like CLIP do the same, with hundreds of numbers to a point, 512 or 768 for example.
16 captions · 32 pictures · 3 numbers each · all of length 1
The Angle Is the Likeness
The angle between two arrows says how alike they are. The brass arc laid between two points reads it. The cosine of that angle is the dot product of the two unit vectors: multiply the three numbers pair by pair and add. The cosine of 0° is 1, of 90° is 0, of 180° is −1.
This picture of a red square is 3.9° from its own caption, 56.7° from “red circle” and 109.6° from “yellow triangle”: cosines of about 1.00, 0.55 and −0.34.
cosine similarity = cos θ = a · b (a, b of length 1)
Words Are Directions Too
word2vec made the idea popular in 2013: a model learns to guess a word’s neighbours from the word, or the word from its neighbours, and words used alike get vectors alike. On the globe now are three GloVe word vectors, learned from 6 billion words of Wikipedia and newswire, 100 numbers each.
A hundred dimensions can’t be drawn, but any three directions span only a three-dimensional space, and they sit on its unit sphere with every angle between them unchanged. cat and dog are 28° apart; cat and car 72°.
Three 100-dimensional word vectors, in the space they span · angles computed
King − Man + Woman
Take the step from man to woman and add it to king: where does it land? The open ring on the globe is the direction of king − man + woman. The nearest word to it, apart from king itself, is queen. The word2vec authors used this example in 2013 to show that differences between word vectors carry meaning too.
It is less reliable than it looks. As the test is usually scored, king, man and woman are not allowed as answers; allow them, and the word nearest the ring is king. Of word2vec’s analogy questions, 3,859 use only words in this page’s vectors: with the question words left out, 77% come out right; with them in, 44%.
Open ring = king − man + woman · nearest: king 0.84 · queen 0.77 (cosines, computed)
Pictures and Words in One Space
CLIP was published by OpenAI in 2021. It has two encoders, one for images and one for text, and their vectors share one space. It was trained on 400 million pairs of images and their text, collected from the internet.
The small model here has two halves too. The picture half is a single layer: 768 pixel values in, 3 numbers out. The caption half gives each word an arrow, and a caption is the sum of its words’ arrows. Before training, the pictures mostly crowd into one region: the median angle between two of them is only 39°, against 97° between captions. To the untrained layer they all look alike, mostly paper.
pictures: 768 → 3 · captions: 3 numbers per word, added · 2,331 numbers in all
The Pairs Lie on the Diagonal
Each training step takes a batch of pairs: 12 here, 32,768 for CLIP. Every picture is compared with every caption, and the cosines form a square: pictures down the side, captions across the top. On the diagonal are the true pairs; everything else is a mismatch.
A softmax along each row gives the chance that this picture picks each caption in the batch; one down each column, the other way round. The loss asks for the diagonal to come as close to 1 as it can. This is CLIP’s symmetric cross-entropy, of the kind a 2018 paper called InfoNCE. A CLIP batch has 32,768² ≈ 1.07 billion cells.
Each cell: the chance picture i picks caption j · diagonal = pairs · loss = mean of the two cross-entropies
Pulled Together, Pushed Apart
The one loss does two things at once: it pulls each picture towards its own caption, and pushes it away from the other captions in the batch. Nobody tells the model what “red” or “circle” means; it is only given the pairs.
After 1,500 steps, a picture is on average 5.6° from its caption; at the start it was 96°. The 12 trained captions end at least 53° apart. The chance on the square’s diagonal rises from 0.03 to 0.97 on average.
Step 0 → step 1,500 · 12 pairs a step · temperature 0.1
Temperature
Before the softmax, the cosines are divided by a number called the temperature, τ. At a low temperature the closest match takes nearly all the chance; at a high one the chance spreads flat. The knob in front changes only the temperature: the chances in the square change with it, and the points on the globe stay put.
CLIP learns its temperature as well, starting from 0.07, with 1/τ kept at or under 100 because training became unstable otherwise. A 2022 study found that after training, CLIP’s images and texts still occupy two separate regions of the space, the “modality gap”; one cause is that each kind starts out crowded into its own small patch, and a low temperature keeps them apart.
p = softmax(cos / τ) · CLIP: τ learned from 0.07, 1/τ ≤ 100
Pairs It Never Saw
Four combinations never appear in training: a red cross, a yellow circle, a green square and a blue triangle. To ask what one of their pictures shows, write out all 16 captions and see which is closest. This is zero-shot classification.
On this three-dimensional globe it gets 53% right (a guess would get 1 in 16, about 6%); trained twenty times from different starts, it averages 49.5%, and 4% at worst. Give the same model 8 numbers, and the twenty runs average 99.8%; keep 3 numbers but set the temperature to 1, and they average 87.8%. How big the space is, and how hard training pushes, both change whether it can carry what it learned over to new pairs. CLIP does it the same way: without seeing any of ImageNet’s 1.28 million labelled training images, it gets 76.2% of ImageNet right, as well as the ResNet-50 trained on them.
Zero-shot: write the classes as captions, pick the closest · chance is 1 in 16
Finding Pictures by Words
The other way round, given a caption, find the pictures nearest it: that is search. A picture library only needs each picture turned into a vector in advance; a query then compares nothing but angles. Many image searches and “more like this” features work this way.
Raised on the globe are the pictures nearest “blue circle”. On Flickr30k, with no training for the task, CLIP finds the right picture first for 68.7% of captions, and the right caption first for 88.0% of pictures.
Nearest neighbours: by cosine, largest first
Where It Goes Wrong
A model always answers. Show this one a purple square, a colour it has never seen, and it still picks one of its 16 captions: red square, at 87%. A softmax always adds up to 1, and “none of these” is not among the choices. CLIP can be fooled by writing: one version of it calls a Granny Smith apple a Granny Smith with 85.6% chance; stick on a paper label saying iPod, and it calls it an iPod with 99.7%.
Vectors learn from what people wrote, prejudice included. In this page’s GloVe vectors the word nearest doctor − man + woman is nurse; allow the question words as answers, and the nearest is doctor itself. A 2020 paper showed that some famous “biased analogies” come out only because the question words are barred. The bias is real; examples of it can be overstated. The CLIP paper reports that when classes such as “gorilla” were added for a set of face photos, 4.9% of the photos went into non-human classes: about 14% of photos of Black people, under 8% for every other group.
Purple square: a colour never seen · the answer is still one of the 16
The Bench
Train again at another temperature, turn the training back and forth, show the model a picture (in colours it has never seen, too), or put other words on the globe. Every number on the globe and the board is computed again from your choice.
Try thisOn the bench, set the temperature to 1 and train again. How many of the four unseen pairs does it get right now? Try a few new starts.
What the model leaves out
The model here is tiny: pictures of 16 × 16 pixels in four colours and four shapes; a picture half of one layer, and a caption half that only adds up its words’ arrows, ignoring their order; a space of three dimensions, to fit on a globe. CLIP reads real photographs and whole sentences from the web; its image encoder is a ResNet or a Vision Transformer and its text encoder a Transformer (63 million parameters at the base size); it trains on batches of 32,768 pairs, 32 passes over 400 million pairs. The three words on the globe sit exactly because any three directions span three dimensions; the open ring (king − man + woman) lies in the space of those three words, so it is exact too; queen lies outside it, and the globe keeps only its angles to king and to the ring true. The GloVe vectors here are the 6,000 or so most common of 400,000 words, stored as 8-bit integers, and the analogy test runs among those words only: fewer words make the questions easier.
A common misunderstanding
The numbers of a vector don’t each stand for a meaning, as in “number 37 means gender”. What carries meaning are directions and angles: turn the whole space to new axes and every angle stays the same, and so does everything the model does. And a small angle only says the model puts two things close, not that it understands them: the small model here has only seen colours and shapes, and it will push a purple picture into one of its captions all the same.
Common questions · 4
- What is cosine similarity?
The cosine of that angle is the dot product of the two unit vectors: multiply the three numbers pair by pair and add.
02The Angle Is the Likeness ↑- Does king − man + woman really give queen?
As the test is usually scored, king, man and woman are not allowed as answers; allow them, and the word nearest the ring is king.
04King − Man + Woman ↑- What does CLIP’s training do?
The one loss does two things at once: it pulls each picture towards its own caption, and pushes it away from the other captions in the batch.
07Pulled Together, Pushed Apart ↑- What is the temperature in contrastive learning?
Before the softmax, the cosines are divided by a number called the temperature, τ. At a low temperature the closest match takes nearly all the chance; at a high one the chance spreads flat.
08Temperature ↑
Sources
- Radford et al. (2021): Learning Transferable Visual Models From Natural Language Supervision (CLIP), arXiv:2103.00020 (400 million pairs, batch 32,768, temperature, loss, zero-shot 76.2%, retrieval, bias)
- OpenAI: CLIP code and model card (MIT licence)
- van den Oord et al. (2018): Representation Learning with Contrastive Predictive Coding, arXiv:1807.03748 (InfoNCE)
- Mikolov et al. (2013): Efficient Estimation of Word Representations in Vector Space, arXiv:1301.3781 (word2vec, the analogy test)
- Mikolov, Yih and Zweig (2013): Linguistic Regularities in Continuous Space Word Representations, NAACL (king − man + woman)
- Mikolov et al. (2013): Distributed Representations of Words and Phrases and their Compositionality, arXiv:1310.4546
- word2vec code and analogy questions (Apache License 2.0)
- Pennington, Socher and Manning (2014): GloVe, EMNLP; word vectors glove.6B (PDDL 1.0)
- Linzen (2016): Issues in evaluating semantic spaces using word analogies, arXiv:1606.07736
- Nissim, van Noord and van der Goot (2020): Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor, Computational Linguistics 46(2)
- Liang et al. (2022): Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning, arXiv:2203.02053
- Goh et al. (2021): Multimodal Neurons in Artificial Neural Networks, Distill (the typographic attack: an apple labelled iPod)
- Kingma and Ba (2014): Adam, arXiv:1412.6980 (the optimiser of this page’s model)
Data
Word vectors: GloVe (Jeffrey Pennington, Richard Socher and Christopher D. Manning, Stanford, 2014), glove.6B, 100 dimensions, trained on Wikipedia 2014 and Gigaword 5; a slice of 6,008 words, made available under the Public Domain Dedication and License v1.0. Analogy questions: the test set of the word2vec project (Tomas Mikolov et al., Google), Apache License 2.0; the 3,859 questions whose words are all in the slice. The small model on the globe, its pictures and its training are the site’s own: src/topics/embeddings/sim.js (one layer for pictures, one arrow per word, CLIP’s loss, Adam).
The text, pictures and models on this page are licensed CC BY-NC 4.0: free to use and adapt for teaching and other non-commercial purposes, with credit to world.lapla.si. Third-party material listed above keeps its own licence.
Cite this plate
Laplace’s Mini World, Plate F22, “Embeddings: The Geometry of Meaning”, https://world.lapla.si/en/embeddings/, CC BY-NC 4.0, updated 2026-10-19.
Follow the question
Plate F13Attention in TransformersA two-layer, eight-head Transformer reads a sentence in your browser, one character at a time, and guesses the next. Each thread is one act of attention; its brightness is the weight.
Plate 43More Features Than NeuronsA single neuron in a neural network often stands for several unrelated things. A model with just two neurons can arrange five features into a pentagon and keep them all.
Plate F18How a Neural Network LearnsAn aluminium relief milled to a network’s real error, a steel ball that steps down it by the real gradient, and a small network in glass and brass: light runs forward with the numbers and back with each weight’s share of the blame.
Plate 44Pictures Out of NoiseMany AI image generators start from random static and remove it, one step at a time. A small network does the same in your browser, and a cloud of dots slowly condenses into a knot.