FastText Word Embeddings

Word embeddings, trained on large unlabeled corpora are useful for many natural language processing tasks. FastText (Facebook AI) in contrast to Word2vec model accounts for sub-word information. FastText trains embeddings for sub-word sequences of length n (n-grams). For example, instead of only embedding entire sequence “where”, FastText also embed 3-grams: wh, whe, her, ere, re. FastText word representation is the word embedding vector plus sum of n-grams contained in it.

How FastText Works?

FastText embeds words by adding word’s n-grams to the word embedding and then normalizes by total token count:

\( \mathrm{fastText}(\mathrm{word}) = v_{\mathrm{word}} + \sum_{g \in \mathrm{ngrams}(\mathrm{word})} \frac{v_g}{| \mathrm{ngrams} | } \)

If the word is not present in the dictionary, is out of vocabulary (OOV), then \( v_{\mathrm{oovWord}} = 0 \) and only n-grams summed up:

\( \mathrm{fastText}(\mathrm{oovWord}) = \sum_{g \in \mathrm{ngrams}(\mathrm{oovWord})} \frac{v_g}{| \mathrm{ngrams} | } \)

FastText embedding vectors can then be used for word analogy tasks, text classification, or ranking.

StarSpace vs FastText

StarSpace a general-purpose embeddings is generalization of FastText to objects with some hierarchy e.g. user-content for content recommendation.

Word2vec vs FastText

Word2vec has single word embedding for each word, while FastText has also embeddings for sub-word n-grams and performs the summation above to get the final word embedding. FastText generally outpeforms Word2vec, while having similar requirements.

Embedding Norms

Word2vec Embedding Norms

Word2vec vector norms have been shown (Schakel & Wilson, 2015) to be correlated to word significance. Speculation: If we look on the chart below, then the “middle frequency” words seem to contributing the most, thanks to their large norm, to predict other context words. The very frequent words cannot add much as their appear in too ambiguous contexts. And we don’t have much data about the context of very infrequent words. We can think about the word norms a bit like analogous to TF-IDF, where the IDF is defined on the word2vec window of size 10 words. For this reason when using Word Movers Distance with TF-IDF weights, Word Rotator paper suggests to use consine instead of euclidean distance.

FastText Embedding Norms

How above chart looks in case of FastText? For purpose of studying OOV words this asymmetry between vocabulary and out of vocabulary words is removed by only utilizing word’s n-grams regardless if the word is OOV or not.

In order to study contrast between common english words e.g. “apple” and noise-words (usually some parsing artifacts or unusual tokens with very specific meaning) e.g. “wales-2708” or “G705” MIT 10K Common words dataset is used.

Entire code for this post in available in this repository in file “main.py”. FastText model used is 5-gram English 2M “cc.en.300.bin”.

Standard Vector Norm

Standard vector norm as defined in Gensim implementation is used in this section. Common words are located mostly on the right in the term-frequency spectrum and clustered in three different areas in the norm spectrum. On both axis common words are clustered approximatelly in 4 areas.

From below samples it is not clear what clusters correspond to:

• bottom left cluster: now, three, month, News, Big, picked, votes, signature, Challenge, Short, trick, Lots, 68, priorities, upgrades
• bottom right cluster: our, home, game, won, control, law, common, Street, speed, Tuesday, direct, helped, passed, condition, Date, signed
• middle right cluster: via, companies, necessary, straight, menu, kinds, Championship, relief, periods, Prize, minimal, Rated, 83, wears
• top right cluster: position, wonderful, shooting, switch, â, Atlantic, ladies, vegetables, tourist, HERE, prescription, upgraded, Evil

No N-Gram Norm

As mentioned above each FastText vocab word has its vector representation regardless its size. Norms of those vectors are plotted in this section. The shape of the distribution seems to match closely the shape of the same plot for Word2Vec (Schakel & Wilson, 2015). The vector norm as measure of word significance seems to hold even for FastText in terms of this norm as can be seen from labeled samples in the scatter plot (same frequency bin with increasing vector norm: authors, Alfine, numbertel).

NG_Norm (N-Grams Times Count Norm)

As mentioned above FastText uses average of word vectors used. However for detection of noise-words number of n-grams seems to useful. For that purpose NG_Norm is defined ng_norm(word)= || Σ_{g ∈ ngrams(word)}v_g ||. Using this norm common words are clustered in narrower band on ng_norm axis.

Explicitly aggregated distribution on ng_norm axis is plotted in histogram below.

Probability distribution of given FastText vocabulary word being common word is plotted below. The distribution is well approximated by t-distribution.

Norms of Hyponyms vs Hypernyms

To evaluate thesis of (Shakel 2015) that word specificity in given term-frequency norm is correlated with vector norm for FastText 67 pairs of hyponyms and hypernyms are used. From just these few examples we see that No-NGram norm with 77% accuracy predicts which word is hyponym and which hypernym disregarding their term-frequencies. Below is the data used. The norm colums contain relative percent differences i.e. (hypo-hyper) / hyper * 100.

Detecting non-english words using NG_Norm

Ability to detect noisy-words is evaluated on simple task of splitting two concatenated words back apart below. For example let’s split back concatenation ‘inflationlithium’:

Above approach yielded around 48% accuracy on 3000 random two-word samples from MIT 10k common words. A more efficient method in this specific case would be to search vocabulary instead of calculating vector norms. More appropriate comparison however would be for more general task involving OOV words e.g. using Edit Distance performed also on OOV words and words with typos.

Conclusion

FastText vector norms and their term-frequency were visualized and investigated in this post.

Standard Norm Term-Frequency plot revealed potentially interesting clustering of common vectors in three to four main areas.

No-N-Gram Norm has very similar Norm-TF distribution as Word2Vec shown in (Schakel & Wilson, 2015). The word significance correlation does seem to hold even for FastText embeddings in terms of No-N-Gram Norm.

NG_Norm shows that n-gram count could be potentially useful feature and that simple averaging over n-gram vectors may not be optimal. Perhaps some approach akin to (Zhelezniak et al., 2019) could be used.

• Full code is here.
• Continue: StarSpace - a general-purpose embeddings inspired by FastText