Efficient Estimation of Word Representations in Vector Space

Inside the paper

The problem

Neural language models could learn useful word vectors, but their training cost limited the size of the vocabulary and the text they could use. The authors wanted to study the quality of vectors learned at much larger scale.

The idea

They simplify the prediction architecture and train word representations using nearby words. Two variants predict a center word from its surroundings or predict surrounding words from the center.

How it works

• The continuous bag-of-words model combines representations of nearby words to predict the word at the center of a context window.
• The skip-gram model instead uses the current word to predict words found within a window around it.
• A hierarchical softmax reduces the cost of predicting over a large vocabulary by organizing output words in a tree.
• The learned vectors are tested with semantic and syntactic analogy questions, using vector offsets to select candidate answers.

What it showed

The experiments compare the proposed models with earlier neural language models and show that larger training sets and vectors can improve analogy accuracy. The two models differ in their performance on semantic and syntactic relationships.

Read it for

Read Sections 3.1–3.2 for the two prediction tasks and Section 4.1 for the analogy test. Table 4 compares models; the parallel training discussion in Section 4.4 can wait.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.