Word2Vec

Definition

The Word2Vec model learns vector representations of words that effectively capture the semantic relationships them using large corpus of text.

Continuous Bag-Of-Words

Continuous bag-of-words (CBOW) predicts a target word given its context words. The model takes a window of context words around a target word. The context words are fed into the network, and it tries to predict the target word. The model is trained to maximize the probability of the target word given the context words.

Algorithm

1. Generate one-hot encodings $\{x_1,x_2,\dots ,x_C\}$ of the context words $v_{I_1},v_{I_2},\dots ,v_{I_C}$ of window size $C$.
2. Get embedded words vectors $\{h_1,h_2,\dots ,h_C\}$ using linear transformation, and take average of them $h=\frac{1}{C}{\sum }_{i=1}^C{x}_{i}W$ where the weight $W$ is shared
3. Generate a score vector and turn the scores into probability with Softmax Function $y=\mathrm{softmax}(h{W}^{\prime })$
4. Adjust the weights $W$ and $W^{\prime }$ to match the result $y$ to the one-hot encoding of actual output word $v_O$ by minimizing the loss $L=-\ln P(v_O|v_{I_1},v_{I_2},\dots ,v_{I_C})=-\sum _{i=1}^{|V|}{v}_{O_i}\ln (y_i)$ where $|V|$ is the dimension of the vectors, and $v_{O_i}$ is the $i$-th element of the one-hot encoded output word vector.

Skip-Gram

The Skip-gram model does the opposite of CBOW. It predicts the context words given a target word The model takes a single target word as input. The target word is fed into the network, and it tries to predict the surrounding context words. The model is trained to maximize the probability of each context word given the target word.

Algorithm

1. Generate one-hot encoding $x$ of the input word $v_I$
2. Get an embedded word vector $h$ using linear transformation $h=xW$
3. Generate a score vector and turn the scores into probability with Softmax Function $y=\mathrm{softmax}(h{W}^{\prime })$
4. Adjust the weights $W$ and $W^{\prime }$ to match the result $y$ to the $C$ many one-hot encodings of the actual output word $v_{O_1},v_{O_2},\dots ,v_{O_C}$ by minimizing the loss $L=-\ln P(|v_{O_1},v_{O_2},\dots ,v_{O_C}|v_I)=-{\sum }_{j=1}^C{\sum }_{i=1}^{|V|}{v}_{O_{j,i}}\ln (y_i)$ where $|V|$ is the dimension of the vectors, and $v_{O_{j,i}}$ is the $i$-th element of the one-hot encoded output $j$-th context word vector.