Represent text with word counts

Learn it

Words to know

vocabulary

The full list of tokens a model knows. A token's place in the list gives its ID.

bag of words

A text described only by how often each word appears. Word order is ignored.

feature

One of the numbers that describe an example, such as one pixel of an image or one word count.

baseline

A simple model that sets the score a more complex model has to beat.

How it works

1. Make the text lowercase, so Good and good count as the same word.
2. Split it on whitespace. Punctuation stays attached: good! is its own word and does not match good.
3. For each vocabulary word, in order, count how many times it appears. Words that are not in the vocabulary are ignored.

Worked example

Input
text = "Good good bad"
vocabulary = ["good", "bad", "other"]

Output
[2, 1, 0]

The answer has one number per vocabulary word, in vocabulary order. A word with no match still gets its place, with a 0.

Why it matters

Counts like these are a cheap baseline to beat. The sentiment lesson next uses the same split, and its weakness is why models moved on to embeddings and attention.

Your turn

Turn a text into word counts for a fixed vocabulary.

text

any string; may be empty

vocabulary

a list of lowercase words, each appearing once

Returns

A list as long as vocabulary: how many times each word appears in the lowercased text, split on whitespace.

Input
text = "Good good bad"
vocabulary = ["good", "bad", "other"]

Output
[2, 1, 0]

Input
text = "good!"
vocabulary = ["good"]

Output
[0]

Input
text = ""
vocabulary = ["a"]

Output
[0]

Input
text = "good\nbad  good"
vocabulary = ["good", "bad"]

Output
[2, 1]

Hints

Each hint gives away a little more. Open one only when you need it.

1. Hint 1: Where to start

   Prepare the text once: lowercase it, then split it into a list of words.

2. Hint 2: The key step

   Count the words of the text first, then read the count for each vocabulary word. collections.Counter does the counting.

3. Hint 3: The outline

   Set counts = Counter(text.lower().split()). Return [counts[word] for word in vocabulary]. A Counter gives 0 for a word it has not seen.

Solution

From here on the page gives the method away. If you are still solving, go back to the hints.

Step by step

1. Lowercase, then split on whitespace. split() with no argument splits on any run of spaces, tabs or line breaks, and gives an empty list for an empty text.
2. Count every word of the text once. One pass over the text. Counting inside the vocabulary loop would read the whole text again for every vocabulary word.
3. Read the counts in vocabulary order; a missing word gives 0. The position of each number is fixed by the vocabulary, so the model always finds the same word in the same place.

The same numbered steps are the comments in the solution.

from collections import Counter


def solve(text, vocabulary):
    # 1. Lowercase, then split on whitespace
    words = text.lower().split()
    # 2. Count every word of the text once
    counts = Counter(words)
    # 3. Read the counts in vocabulary order; a missing word gives 0
    return [counts[word] for word in vocabulary]

Watch out

• Lowercase the text before splitting. Good and good are different strings.
• Do not strip punctuation. This exercise defines a word as whatever sits between spaces, and the tests rely on that.

Check yourself