Embedding similarity

Learn it

Words to know

embedding

A vector that a trained model produces for a piece of text. Texts with similar meaning get vectors that point in similar directions.

vector length

How long a vector is as an arrow, not how many numbers it holds: the square root of the sum of its squared numbers. Also called its norm.

cosine similarity

The dot product divided by the two vector lengths multiplied together. It is 1 for the same direction, 0 for unrelated directions and -1 for opposite ones.

How it works

1. Compute the dot product of a and b.
2. Compute the length of each vector: square every number, add the squares, and take the square root. For [3, 4] that is sqrt(9 + 16) = 5.
3. Divide the dot product by the two lengths multiplied together.
4. If either length is 0, the vector has no direction. Return 0 instead of dividing by zero.

Worked example

Input
a = [3, 4]
b = [4, 3]

Output
0.96

0.96 is close to 1, so the two vectors point in nearly the same direction.

Why it matters

Vector search ranks documents by this number: turn the question and every piece of every document into an embedding, and return the pieces with the highest cosine. Chapter 2 builds the same ranking with a simpler score.

Your turn

Return the cosine similarity of two vectors with the same number of positions.

a

a list of numbers, possibly empty

b

a list with as many numbers as a

Returns

A number from -1 to 1, give or take a rounding error in the last digit: the dot product divided by both lengths. If either vector is all zeros or empty, 0.

Input
a = [3, 4]
b = [4, 3]

Output
0.96

Input
a = [1, 0]
b = [5, 0]

Output
1

Input
a = [1, 0]
b = [0, 1]

Output
0

Input
a = [0, 0]
b = [1, 2]

Output
0

The checks also try

• opposite directions
• an angle of 45 degrees
• the same direction, ten times longer
• zeros in the second vector
• three positions
• numbers that add up to zero
• a zero in the first position only
• two empty vectors

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   Three numbers are needed: the dot product, the length of a, and the length of b. The dot product is the previous lesson.

2. Hint 2: The key step

   A length is the square root of the sum of the squares. A small helper for it saves writing the same loop twice.

3. Hint 3: The outline

   Compute the dot product as the sum of a[i] * b[i]. Compute each length as the square root of the sum of its squared numbers. If either length is 0, return 0. Otherwise return the dot product divided by the two lengths multiplied together.

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. The length of a vector: square, add, take the root. This is the distance from the corner to the tip of the arrow. A helper keeps the same loop from being written twice.
2. The dot product, as in the previous lesson. It measures how much the two vectors agree, but it still grows with their lengths.
3. A vector of zeros has no direction. Its length is 0, and dividing by 0 does not give a usable number. Returning 0 treats it as unrelated to everything.
4. Divide by both lengths so that only direction is left. Dividing by one length is not enough: the result would still grow with the size of the other vector.

The same numbered steps are the comments in the solution.

function solve(a, b) {
  // 1. The length of a vector: square, add, take the root
  const length = (vector) => {
    let squares = 0;
    for (const x of vector) squares += x * x;
    return Math.sqrt(squares);
  };
  // 2. The dot product, as in the previous lesson
  let dot = 0;
  for (let i = 0; i < a.length; i++) dot += a[i] * b[i];
  // 3. A vector of zeros has no direction
  const lengths = length(a) * length(b);
  if (lengths === 0) return 0;
  // 4. Divide by both lengths so that only direction is left
  return dot / lengths;
}

import math


def solve(a, b):
    # 1. The length of a vector: square, add, take the root
    def length(vector):
        squares = 0
        for x in vector:
            squares += x * x
        return math.sqrt(squares)

    # 2. The dot product, as in the previous lesson
    dot = 0
    for i in range(len(a)):
        dot += a[i] * b[i]
    # 3. A vector of zeros has no direction
    lengths = length(a) * length(b)
    if lengths == 0:
        return 0
    # 4. Divide by both lengths so that only direction is left
    return dot / lengths

Watch out

• Do not forget the square root. Without it you divide by the squared lengths, and the answer for [3, 4] and [4, 3] becomes 0.0384 instead of 0.96.
• A high cosine is a hint, not proof. It says two texts are about similar things; it does not say the document answers the question.
• In JavaScript: 0 / 0 does not raise an error in JavaScript. It gives NaN, which is not equal to anything, so the check fails without any message about a division.
• In Python: 0 / 0 raises ZeroDivisionError in Python, so a vector of zeros stops the run unless you check for it first.

Check yourself