Wen we dey train classifiers wey base on BoW or TF/IDF, we dey use high-dimensional bag-of-words vectors wey get length vocab_size, and we dey change from low-dimensional positional representation vectors go sparse one-hot representation. But dis one-hot representation no dey save memory well. Plus, e dey treat each word as if dem no relate, meaning say one-hot encoded vectors no fit show any semantic similarity between words.
Di idea of embedding na to represent words wit lower-dimensional dense vectors, wey go somehow show di meaning of di word. Later, we go discuss how to build better word embeddings, but for now, just think say embeddings na way to reduce di size of word vector.
So, di embedding layer go take one word as input, and e go produce output vector wey get di embedding_size wey you set. In one way, e be like Linear layer, but instead of using one-hot encoded vector, e fit take word number as input, so we no go need to create big one-hot-encoded vectors.
If we use embedding layer as di first layer for our classifier network, we fit change from bag-of-words to embedding bag model. For here, we go first convert each word for our text into di embedding wey match am, then we go calculate one aggregate function like sum, average or max for all di embeddings.
Image by di author
Continue to learn wit di following notebooks:
Even though di embedding layer dey map words to vector representation, di representation no really get plenty semantical meaning. E go make sense if we fit learn vector representation wey similar words or synonyms go dey close to each other based on vector distance (like Euclidean distance).
To do dis, we need to pre-train our embedding model on big text collection in one special way. One way to train semantic embeddings na Word2Vec. E get two main architectures wey dem dey use to create distributed representation of words:
-
Continuous bag-of-words (CBoW) — for dis architecture, we dey train di model to predict one word from di words wey dey around am. If we get di ngram
$(W_{-2},W_{-1},W_0,W_1,W_2)$ , di model go try predict$W_0$ from$(W_{-2},W_{-1},W_1,W_2)$ . - Continuous skip-gram na di opposite of CBoW. Di model go use di words wey dey around di main word to predict di main word.
CBoW dey faster, but skip-gram dey slow small, though e dey represent words wey no dey common better.
Image from this paper
Word2Vec pre-trained embeddings (and other models like GloVe) fit replace embedding layer for neural networks. But we go need to handle vocabularies, because di vocabulary wey dem use train Word2Vec/GloVe fit no match di vocabulary for our text corpus. Check di Notebooks above to see how to solve dis problem.
One big wahala wey traditional pretrained embedding representations like Word2Vec get na di problem of word sense disambiguation. Even though pretrained embeddings fit capture some meaning of words for context, dem dey put all di possible meanings of one word inside di same embedding. Dis fit cause problem for downstream models, because plenty words like 'play' get different meanings based on di context.
For example, di word 'play' for dis two sentences get different meanings:
- I go watch one play for di theatre.
- John wan play wit im friends.
Di pretrained embeddings wey we talk about before go represent di two meanings of di word 'play' as di same embedding. To solve dis problem, we need to build embeddings wey base on language model, wey dem don train on big text corpus, and e sabi how words dey work together for different contexts. To talk about contextual embeddings no dey inside dis tutorial, but we go come back to am wen we dey talk about language models later for di course.
For dis lesson, you don learn how to build and use embedding layers for TensorFlow and Pytorch to better show di semantic meanings of words.
Word2Vec don dey used for some interesting things, like creating song lyrics and poetry. Check dis article wey explain how di author use Word2Vec to create poetry. Watch dis video by Dan Shiffmann too to see another way dem explain dis technique. Then try use dis techniques for your own text corpus, maybe from Kaggle.
Read dis paper on Word2Vec: Efficient Estimation of Word Representations in Vector Space
Disclaimer:
Dis dokyument don use AI transleshion service Co-op Translator do di transleshion. Even as we dey try make am correct, abeg make you sabi say automatik transleshion fit get mistake or no dey accurate well. Di original dokyument wey dey for im native language na di one wey you go take as di main source. For important informashon, e good make you use professional human transleshion. We no go fit take blame for any misunderstanding or wrong interpretashon wey fit happen because you use dis transleshion.