Distributed Representations of Words and Phrases and their Compositionality

Inside the paper

The problem

Word vectors learned from large text collections could capture relationships between words, but training costs were high and individual word vectors could miss the meaning of a phrase as a unit.

The idea

The paper extends skip-gram training with cheaper objectives and sampling choices, and treats selected phrases as tokens. It also asks what simple arithmetic on the learned vectors can reveal.

How it works

• Skip-gram learns a representation for a center word by predicting words found nearby in a text sequence.
• Negative sampling trains the model to distinguish observed word-context pairs from pairs formed using sampled noise words.
• Frequent words are randomly discarded during training, reducing their dominance and speeding up processing of the corpus.
• A data-driven score identifies word combinations that occur together unusually often; selected combinations are joined and learned as individual tokens.

What it showed

The experiments compare training choices on word analogy tasks and show that phrase vectors can also answer some analogy questions. Examples of vector addition suggest that combining representations can retrieve meaningful words or phrases, although the paper does not claim universal compositionality.

Read it for

Start with Figure 1 and Sections 2.2–2.3 for the training changes, then read Section 4 for phrase examples. The hierarchical softmax derivation in Section 2.1 can wait until you need that comparison.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.