1. BERT Architecture

BERT stands for Bidirectional Encoder Representations from Transformers, a language model developed by Google in late 2018. It leverages the Transformer encoder architecture combined with bidirectional training to capture contextual information from both directions.

The key innovation lies in the pretraining methodology, which employs two complementary techniques: Masked Language Modeling and Next Sentence Prediction. These enable the model to learn both word-level and sentence-level representations.

Masked Language Model (MLM): During training, approximately 15% of tokens in the corpus are masked. The masking strategy consists of three approaches: 80% replaced with [MASK], 10% substituted with random tokens, and 10% left unchanged.

Next Sentence Prediction (NSP): This pretraining task helps the model understand sentence relationships. For a given sentence pair A and B, where B is randomly chosen to be the next sentence or from elsewhere in the corpus, the model predicts whether B follows A.

After pretraining, BERT stores its embedding table and either 12 layers of Transformer weights (BERT-BASE) or 24 layers (BERT-LARGE). These pretrained weights can be fine-tuned for downstream tasks including text classification, semantic similarity, and reading comprehension.

2. Emotion Detecsion

Emotion detection focuses on identifying user sentiment in intelligent conversation scenarios. The system automatically classifies input text into emotional categories with corresponding confidence scores. Typical classifications include positive, negative, and neutral emotions.

This technology applies to chat applications and customer service systems, enabling enterprises to monitor conversation quality, enhance user experience, and reduce manual quality inspection costs.

3. Implementation

3.1 Dataset

This example uses a labeled Chinese conversational dataset from PaddlePaddle, preprocessed with word segmentation. The data format consists of two columns separated by tabs, where the first column contains the emotion label (0: negative, 1: neutral, 2: positive) and the second column contains whitespace-separated tokenized text.

# download dataset
!wget https://baidu-nlp.bj.bcebos.com/emotion_detection-dataset-1.0.0.tar.gz -O emotion_detection.tar.gz
!tar xvf emotion_detection.tar.gz

Data Loading

import os
import numpy as np

import mindspore
from mindspore.dataset import text, GeneratorDataset, transforms
from mindspore import nn, context

from mindnlp._legacy.engine import Trainer, Evaluator
from mindnlp._legacy.engine.callbacks import CheckpointCallback, BestModelCallback
from mindnlp._legacy.metrics import Accuracy
from mindnlp.transformers import BertTokenizer

class EmotionDataset:
    """Emotion Classification Dataset"""

    def __init__(self, data_path):
        self.data_path = data_path
        self.emotion_labels = []
        self.text_content = []
        self._load_data()

    def _load_data(self):
        with open(self.data_path, "r", encoding="utf-8") as file_handle:
            raw_data = file_handle.read()
        data_lines = raw_data.split("\n")
        for entry in data_lines[1:-1]:
            emotion_tag, text_seg = entry.split("\t")
            self.emotion_labels.append(int(emotion_tag))
            self.text_content.append(text_seg)

    def __getitem__(self, idx):
        return self.emotion_labels[idx], self.text_content[idx]

    def __len__(self):
        return len(self.emotion_labels)


def prepare_dataset(source, tokenizer, max_length=64, batch_size=32, shuffle=True):
    use_ascend = mindspore.get_context('device_target') == 'Ascend'

    output_columns = ["label", "text_a"]
    
    data_generator = GeneratorDataset(source, column_names=output_columns, shuffle=shuffle)
    
    type_cast_op = transforms.TypeCast(mindspore.int32)
    
    def tokenize_text(text):
        if use_ascend:
            tokenized_result = tokenizer(text, padding='max_length', truncation=True, max_length=max_length)
        else:
            tokenized_result = tokenizer(text)
        return tokenized_result['input_ids'], tokenized_result['attention_mask']
    
    data_generator = data_generator.map(operations=tokenize_text, input_columns="text_a", output_columns=['input_ids', 'attention_mask'])
    data_generator = data_generator.map(operations=[type_cast_op], input_columns="label", output_columns='labels')
    
    if use_ascend:
        data_generator = data_generator.batch(batch_size)
    else:
        data_generator = data_generator.padded_batch(batch_size, pad_info={'input_ids': (None, tokenizer.pad_token_id),
                                                         'attention_mask': (None, 0)})

    return data_generator


tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')
train_data = prepare_dataset(EmotionDataset("data/train.tsv"), tokenizer)
validation_data = prepare_dataset(EmotionDataset("data/dev.tsv"), tokenizer)
test_data = prepare_dataset(EmotionDataset("data/test.tsv"), tokenizer, shuffle=False)

3.2 Model Construction

The BertForSequenceClassification class provides a BERT model configured for sentiment classification. By loading pretrained weights and specifying the number of emotion categories, the model architecture is automatically constructed.

from mindnlp.transformers import BertForSequenceClassification, BertModel
from mindnlp._legacy.amp import auto_mixed_precision

# initialize model with three-class classification head
classifier = BertForSequenceClassification.from_pretrained('bert-base-chinese', num_labels=3)
classifier = auto_mixed_precision(classifier, 'O1')

# configure optimizer with low learning rate for fine-tuning
optimizer = nn.Adam(classifier.trainable_params(), learning_rate=2e-5)

performance_metric = Accuracy()

# configure callbacks for checkpoint management
checkpoint_saver = CheckpointCallback(save_path='checkpoint', ckpt_name='bert_emotect', epochs=1, keep_checkpoint_max=2)
best_model_saver = BestModelCallback(save_path='checkpoint', ckpt_name='bert_emotect_best', auto_load=True)

trainer = Trainer(network=classifier, train_dataset=train_data,
                  eval_dataset=validation_data, metrics=performance_metric,
                  epochs=5, optimizer=optimizer, callbacks=[checkpoint_saver, best_model_saver])

# initiate training process
trainer.run(tgt_columns="labels")

3.3 Inference

from mindspore import Tensor


def predict_sentiment(text, expected_label=None):
    sentiment_mapping = {0: "negative", 1: "neutral", 2: "positive"}

    tokenized_input = Tensor([tokenizer(text).input_ids])
    output_logits = classifier(tokenized_input)
    predicted = output_logits[0].asnumpy().argmax()
    result = f"inputs: '{text}', predict: '{sentiment_mapping[predicted]}'"
    if expected_label is not None:
        result += f" , label: '{sentiment_mapping[expected_label]}'"
    print(result)


inference_data = EmotionDataset("data/infer.tsv")
for emotion, content in inference_data:
    predict_sentiment(content, emotion)