Preparation

1. Install Gensim Library

Install using pip:

!pip install gensim

2. Tokenize the Raw Corpus

We use the novel "In the Name of the People" as the corpus and apply jieba for tokenization.

import jieba
import jieba.analyse

# Add terms to improve jieba segmentation accuracy
jieba.suggest_freq('Sha Ruijin', True)
jieba.suggest_freq('Tian Guofu', True)
jieba.suggest_freq('Gao Yuliang', True)
jieba.suggest_freq('Hou Liangping', True)
jieba.suggest_freq('Zhong Xiaoai', True)
jieba.suggest_freq('Chen Yanshi', True)
jieba.suggest_freq('Ouyang Jing', True)
jieba.suggest_freq('Yi Xuexi', True)
jieba.suggest_freq('Wang Dalu', True)
jieba.suggest_freq('Cai Chenggong', True)
jieba.suggest_freq('Sun Liancheng', True)
jieba.suggest_freq('Ji Changming', True)
jieba.suggest_freq('Ding Yizhen', True)
jieba.suggest_freq('Zheng Xipo', True)
jieba.suggest_freq('Zhao Donglai', True)
jieba.suggest_freq('Gao Xiaoqin', True)
jieba.suggest_freq('Zhao Ruilong', True)
jieba.suggest_freq('Lin Huahua', True)
jieba.suggest_freq('Lu Yike', True)
jieba.suggest_freq('Liu Xinjian', True)
jieba.suggest_freq('Liu Qingzhu', True)
jieba.suggest_freq('Zhao Dehan', True)

with open(r'C:\Users\zhangjh46\Downloads\in_the_name_of_people.txt', 'r', encoding='UTF-8') as f:
    tokenized_lines = []
    for line in f.readlines():
        tokenized_lines.append(list(jieba.cut(line)))

Output:

stop_words = ["，", "。", "\n", "\u3000", " ", "：", "！", "？", "..."]

def remove_stopwords(tokens):
    return [word for word in tokens if word not in stop_words]

filtered_lines = [remove_stopwords(line) for line in tokenized_lines if remove_stopwords(line)]
print(filtered_lines[100:103])

Output:

[['Hou Liangping', 'also', 'very', 'humorous', 'grab', 'hold', 'of', 'Zhao Dehan', 's', 'hand', 'hey', 'Zhao', 'director', 'I', 'since', 'came', 'really', 'reluctant', 'to', 'part', 'from', 'you', 'immediately', 'we', 'go', 'next', 'spot', 'saying', 'finished', 'from', 'Zhao', 's', 'table', 'miscellaneous', 'basket', 'accurately', 'took', 'out', 'a', 'white', 'key card', 'inserted', 'into', 'Zhao Dehan', 's', 'shirt', 'pocket'], ['Zhao Dehan', 'panicked', 'hurriedly', 'pulled', 'the', 'card', 'out', 'this', '...', '...', 'what', 'is', 'this'], ['Your', 'Dijingyuan', 'mansion', 'key', 'card', 'please', 'continue', 'to', 'cooperate', 'with', 'us', 'on', 'official', 'business']]

Training the Word2Vec Model

from gensim.models import Word2Vec

model = Word2Vec(
    filtered_lines,
    vector_size=100,
    window=5,
    min_count=1
)

Model Application

1. Compute Word Similarity

Use the similarity() method to compute cosine similarity between two words.

print(model.wv.similarity('Sha Ruijin', 'Ji Changming'))
print(model.wv.similarity('Sha Ruijin', 'Tian Guofu'))

Output:

0.9995858
0.99938

# Select the top 5 most similar words
for word, score in model.wv.most_similar(positive=['Sha Ruijin'], topn=5):
    print(word, score)

Output:

this 0.9998286962509155
Li Dakang 0.9998038411140442
Chen Qingquan 0.9997875094413757
some 0.999778687953949
Chen Yanshi 0.9997553825378418

2. Find the Odd One Out

Use doesnt_match() to identify the word that doesn't match the group.

odd_word = model.wv.doesnt_match(["apple", "banana", "orange", "book"])
print(f"The odd word in this set: {odd_word}")

Output:

The odd word in this set: book

3. Get Word Frequency

Use get_vecattr() to retrieve the frequency of a word.

word_frequency = model.wv.get_vecattr("Sha Ruijin", "count")
print(f"Sha Ruijin: {word_frequency}")

Output:

Sha Ruijin: 353

Summary

Gensim is efficient and easy too use for word embedding training.