Emotion recognition and its application in learning scenarios supported by a smart classroom (2023)

In today's society, the student-centred method of teaching has gradually become mainstream, with more and more attention being paid to individualized and contextualized learning [1-5]. Therefore, it is necessary for teachers to understand and correct students' learning emotions over time in order to achieve the best learning effect [6-11]. The smart classroom is an important product of the new educational technology that uses the latest computer, communication and Internet technologies to achieve deep perception, intelligent analysis and accurate conduct of the educational process [12-17]. Among them, emotion recognition technology is one of the important applications of artificial intelligence and machine learning in the field of education [18-21]. By recognizing student emotions in learning scenarios, teachers can better understand students' learning status and provide them with personalized learning and support resources. Emotions are closely related to learning outcomes. By recognizing and adapting to students' emotions in real time, you can improve their learning motivation and performance.

Recognizing student emotions in the classroom plays an important role in improving classroom effectiveness. Currently, recognition methods based on traditional image processing generally suffer from problems such as low recognition accuracy and difficult feature extraction. In order to effectively solve these problems, Su and Wang [22] proposed a method of recognizing students' emotions based on deep learning. This method first introduces MobileNet's convolutional structure to replace DarkNet-53 with YOLO v3, making the model lighter and reducing the number of parameters. Then use GIOU loss to improve the model loss function. Experimental results show that the mAP of the improved model increases by 4%, the F1 score increases by 3.2%, and the detection time is reduced by 1/3. Liang et al. [23] focused on the teacher's speech signal and designed a sound processing system to detect emotions. The teacher's speech determines their mood. The classification model for speech emotion recognition was built using a recursive neural network (RNN) algorithm. Refining the traditional cepstral Mel-frequency coefficient (MFCC) feature extraction process, adding a second-order differential process to remove MFCC convolutional noise, and adding one-dimensional energy features to 39-dimensional MFCC coefficients for experiments. The experimental results show that the improved MFCC eigenvalue and neural network are more effective in improving the speech emotion recognition rate than the traditional speech emotion recognition method and can be used in speech emotion recognition in teaching. Putra and Arifin [24] built a real-time facial emotion recognition system that allows teachers to track students' emotions during classroom activities. The system should be reliable enough when running on a mid-range computer. Students will receive a questionnaire to measure stress. The results of the survey will be used to analyze whether using the system can reduce students' stress. The results of the survey showed that the system was able to detect student emotions early, which allowed teachers to minimize student stress.

Facial emotions are very complex and subtle cues, and people can show multiple facial emotions at the same time. Moreover, different people can express the same emotions in different ways, which makes it very difficult to recognize facial emotions. Moreover, facial emotions are dynamic, and the beginning, duration and ending of facial expressions contain rich emotional information. However, current emotion recognition methods rely mainly on static facial emotions, ignoring the temporal characteristics of facial emotions, which can lead to inaccurate recognition results. To meet these challenges, it is necessary to develop more advanced and precise methods of emotion recognition. To that end, this study investigates emotion recognition and its application in learning scenarios that support smarter classrooms.

The traditional recursive neural network (RNN) faces the problem of gradient decay and explosion when processing long sequences, while the Transformer model uses its self-attention mechanism to capture long-term dependencies in sequences, allowing the model to better understand and analyze dynamic changes in facial emotions. In addition, compared to RNNs that must sequentially process each time step, the Transformer model can process all time steps simultaneously, significantly improving computational efficiency, which is critical for processing large amounts of learning scene data. Based on the self-attention mechanism, the Transformer model can assign different weights based on the importance of facial expressions at different points in time, which helps the model capture key emotional changes. Therefore, this study used the Transformer encoder to partially extract the temporal features of students' facial expressions based on the learning scene, that is, the encoder-based self-attention module extracts the temporal features of students' facial expressions in the learning scene. .

In the self-attention module, it is assumed that the input sequence of the learner's facial expressions is mapped usingentrance entrance, bring to doOther₁IOther₂, then go$Q^w, Q^i$ i $Q^c$received matrix$w^u, j^u$, IC^vas.

Where,won behalf of the query,Jrepresentkey, ICRepresents information extracted fromOther.When we assume that the inputOther₁IOther₂, array parameters enabled$Q^w$, Then:

$w^1=s_1 Q^w, w^2=s_2 Q^w$ (1)

Since models can be parallel, there are also:

$\left(\begin{array}{l}w^1 \\ w^2\end{array}\right)=\left(\begin{array}{l}s_1 Q^w \\ s_2 Q^w \end{数组}\right)$ (2)

Similar to, (J¹J²) I (C¹C²) you can get. In the mechanism of self-attention W (w¹w²), (J¹J²) her (C¹C²) SoC.next connect w1 to eachJto perform the dot product operation. As the dot product is processed, the gradient becomes very smallMecca liverfunction that needs to be scaled to get the right oneBThe calculation process is as follows:

$\beta_{1,1}=\frac{w^1 j^1}{\sqrt{f}}, \beta_{1,2}=\frac{w^2 j^2}{\sqrt{f }}$ (3)

So is stackingw²AllJ,$\beta_{2, u}$ You can get:

$\left(\begin{array}{ll}\beta_{1,1} & \beta_{1,2} \\ \beta_{2,1} & \beta_{2,2}\end{array}\右)=\frac{\left(\begin{array}{c}w^1 \\ w^2\end{array}\right)\left(\begin{array}{l}j^1 \\ j ^2\end{tablica}\right)^Y}{\sqrt{f}}$ (4)

after applicationMecca liverWorking on each row of the matrix in the formula above ($\beta_{1,1}^{\klin} \beta_{2,1}^{\klin}$) I ($\beta_{2,1}^{\klin} \beta_{2,2}^{\klin}$) you can get. by$\beta^{\klin}$the weight of eachC, you can get the output of the self-monitoring mechanism, namely:

$A T T(W, J, C)=\Omega\left(\frac{W J^Y}{\sqrt{f_j}}\right) C$ (5)

The multi-head self-service module was transformed in a similar way$\beta_u$ gets $w^u, j^u$ and $c^u$ over $Q^w, Q^j$ and $Q^c$Figure 1 shows the architecture of the multi-headed attention mechanism. The specific calculation is shown in the formula below.

$\operatorname{ATT}\left(W_u, J_u, C_u\right)=\Omega\left(\frac{W_u J_u^Y}{\sqrt{f_j}}\right) C_u$ (6)

picture 1.Architecture of the multi-headed mechanism of attention

Using this module to extract a time series of a learner's facial expressions, it is assumed that a set of features has been learnedvisual converterrepresent$\lambda_u^k$.Nurse$\lambda_u^k$NursetransformerThe module extracts the temporal characteristics of the emotions on the faces of students in a series of educational scenes. Then the vector scores of several categories are obtained by a fully connected layer, expressed as$z_u$.

hypothetical resultvasThe output of the Transformer module class is represented as$z_u$, the initial probability vectorMecca liverfunction expressed aso_vas, the number of marked samples is expressed asOther, and the corresponding ground truth label is marked withHis_vasand the cross-entropy loss of the probability vectoro_vascomeHis_vasrepresentRice_ICome in finally$z_u$NurseMecca liverfunction and formula of the entropy loss function to obtain the model loss value.

$\left\{\begin{array}{l}o_u=\frac{r^{z_u}}{\sum_{j=1}^B r^{z_j}} \\ M_Y=-\sum_{u= 1}^B t_u L O\lijevo(o_u\desno)\end{niz}\desno.$ (7)

Existing facial emotion recognition algorithms have many limitations in data collection and suppression of non-emotional features. Delayed attention networks can bring out deep features and record subtle changes in images that are crucial for recognizing facial emotions.transformersIt can process time series data and record the dynamic changes of facial emotions. Non-local neural networks can capture long-term relationships in images, which is especially useful for understanding complex facial emotions. By integrating these three networks, facial emotion features can be extracted from different perspectives and levels. Figure 2 shows the general architecture of the network model.

photo 2.Network model architecture

The residual attention network consists of two main branches: the main branch and the masked branch. These two branches work together to extract useful sentiment characteristics and learn the importance of attention from input images. Figure 3 illustrates the architecture of the residual attention network.

The master branch is mainly responsible for extracting features from the input image. It usually includes a series of convolutional layers, activation functions and pool layers, forming the deep structure of the neural network. The design of the main branch mainly refers to the structure of the residual network (ResNet), where each residual block contains multiple convolutional layers, and aabbreviationa combination that effectively brings out deep features in images and prevents the problem of gradient fade. In addition, the main branch can extract the image features of different receptive fields by changing the convolution nucleus size and step size to better understand the local and global information in the image.

packet normalization layer irecoveryA line activation unit is placed after each winding layer in the main branch. Hypothetical outputRice-this convolutional layer is expressed asX^Rice, the output after series normalization is expressed asul^Rice, the activation function is expressed asD( ), weights and deviations are expressed asaskINgive the formula for calculating the linear activation unit as:

$p^m=d\left(x^m\right)=d\left(Qo^{m-1}+n\right)$ (8)

The first task of the mask branch is to learn the importance of the input image's attention, directing the master branch to pay more attention to the key areas of the image. The mask branch generally consists of a light convolutional layer and an activation function, and finally transmits the attention weight of each pixel throughThe sigmoid colonFunction. These weights are applied to master branch feature maps to enable dynamic feature-level attention adjustments. The mask branch design enables the model to adaptively focus on areas of the image that are more critical to the task, thereby improving model performance.

Do a point multiplication operation on the features and output weights from the two branches to get a careful feature map corresponding to the learning scene. Suppose the input of the residual attention network is expressed asz, the exit of the main branch is marked as$Y_{u, v}(z)$, the output of the mask branch is marked as$L_{u, v}(z)$, Then:

$G_{u, v}(z)=L_{u, v}(z) * Y_{u, v}(z)$ (9)

In order to avoid affecting the transmission of the extracted facial features of the students, this study adopts a method similar to residual connection to combine the output of the two branches. The output expression of the residual attention network model is:

$\lijevo(1+L_{u, v}(z)\desno) * Y_{u, v}(z)$ (10)

In facial emotion recognition tasks, it is often necessary to deal with multidimensional objects, such as three-channel RGB images and object maps from different convolutional layers.coupled neural network (FFNN) can effectively process these multidimensional features and extract useful information. By incorporating multi-headed attention networks, feature relationships can be modeled in multidimensional space to better understand and recognize facial emotions.

The workflow of the adopted multi-head attention network is as follows:

$A T T((J, C), W)=A T T\lijevo((J, C), w_1\desno) \oplus \cdots \oplus A T T\lijevo((J, C), w_g\desno)$ (11)

Hypothetical input featuresRice- this layer consists ofX^Rice, output functionsRice- this layer consists oful^Rice, matrix of weightsRice-this layer is represented asask^Riceand biasX-this layer is represented asN^RiceThe iterative feedback neural network process is presented as follows:

$x^m=Q^m p^{m-1}+n^m$ (12)

$p^m=d^m\lewo(x^m\prawo)$ (13)

In facial emotion recognition, global feature relationships (e.g. eye-mouth interaction) are crucial to understanding overall facial emotion. The non-local neural network calculates the relationship between any two points, models display global relationships and help extract more complete emotional information. As non-local neural networks can capture richer spatial information, they can significantly improve the performance of facial emotion recognition tasks. Combining the previous network of residual attention andtransformernetwork that enables the model to better understand complex facial emotions, thereby improving the accuracy of emotion recognition. Figure 4 shows the architecture of a non-local neural network. Suppose the input features are given byz, the output at position u is marked asHis_vas, the normalization factor is expressed asV(z) and all element positions required for related calculationsvasDepends onkThe definition of a foreign operation is as follows:

$t_u=\frac{1}{V(z)} \sum_{\forall_k} d\left(z_u, z_k\right) h\left(z_k\right)$ (14)

The correlation scores between u and all items can be computed using$d\left(z_u, z_k\right)$and input representationkcan be calculated usingH(z_k).

In this study, the results of the correlation betweenvasIkIn the object map, it is obtained by downloadingaskIzAs the input of a non-local neural network, expressed as$d\left(z_u, z_k\right)=\left(Q_\phi z_u\right)^Y\left(Q_\rho z_k\right)$The detailed process of calculating a non-local neural network is described as follows:

Step 1: Make a 1×1 weave on the map of objects, mapzResults in the ranges $\phi, \rho$ and $h 3$.

Step 2: Calculate the correlation score betweenvasIkusing the pattern$d\left(z_u, z_k\right)=\left(Q_\phi z_u\right)^Y\left(Q_\rho z_k\right)$.

photo 3.Network architecture for residual attention

Figure 4.Architecture of a non-local neural network

Step 3: Weighted sum of correlation results and feature representation functionsH(z_k) zkDownload dependency learning resultsvasIk.

In research,visual converter(Wertera) is used as the main frame.Werterais an image processing model designed to extract spatial features from images. thisGOVERNMENT networkModules have also been introduced to extract more valuable features from images using grouping methods. At last,transformerThe module is used to learn the temporal features of the image sequence and further extract the features of the temporal dimension. This enables a comprehensive extraction of the space-time features of the learning screen for more accurate identification of student emotions. ConnectionWerteraIGOVERNMENT networkIt allows the model to learn data features from multiple perspectives, thus improving the generalizability of the model and making it work well on unseen data. This can be represented by the following formula:

$M_c=M_B+\eta M_Y+\frac{\eta_2}{2}\|q\|$ (15)

From the formula above, it can be seen that the unified loss functionRice_Cswitch onWerterastrataRice_OtherIGOVERNMENT networkstrataRice_I.ten₁Iten₂is a balancing hyperparameterRice_IIRice_Other, this is.

To improve model robustness and maintain high recognition accuracy across different environments and people, this study usesStarGANGenerate student images in the top frame of the learning screen sequence. generated imageStarGANIt can increase the diversity of data, which can be considered as a data improvement method. Data augmentation can improve the model's ability to generalize, making it perform better on unseen data. Finally, the generated image and the top image of the frame are mixed together and fed into themWerteranetwork, allowing models to understand and learn facial emotions from different perspectives, which helps improve model performance in facial emotion recognition tasks.

At this point, the mixed input image is expressed as$z_u^l \u Z$mark by$t_u\kod T$foundedsoftmaxSMfunction expressed aso_vas, and the corresponding ground truth label is marked withHis_vas, the number of markers is expressed asOtherand cross entropy lossRice_Hprobability vectoro_vaseyeHis_vasbe represented. Then it uses the mainframe gridMecca liverfunction and cross entropy function as follows:

$M_H=-\sum_{u=1}^B t_u \log \left(o_u\right)$ (16)

Finally, the general loss function of the model has three parts:Rice_Other,Rice_I, IRice_H, as follows:

$M=M_H+M_B+\eta_1 M_Y+\frac{\eta_2}{2}\|q\|$ (17)

Figure 5 shows a comparison of emotion recognition accuracy before and after the introduction of the NetVLAD module. It can be seen that after the introduction of the NetVLAD module, the accuracy of recognition of all categories of emotions has improved. For 'fear' recognition, the accuracy rate increased by 0.02 from 0.71 to 0.73 after the introduction of the NetVLAD module. In the case of “anger” recognition, after the introduction of the NetVLAD module, the accuracy index increased by 0.02, from 0.68 to 0.7. For recognition of "disgust", after the introduction of the NetVLAD module, the accuracy index increased by 0.03, from 0.61 to 0.64. For "lucky" recognition, the accuracy rate increased by 0.08 after the introduction of the NetVLAD module, from 0.89 to 0.97. This is the biggest improvement in all categories of feeling. For the recognition of "sadness", the accuracy index increased by 0.06 after the introduction of the NetVLAD module, from 0.65 to 0.71. For forward recognition, the accuracy increased by 0.015 after the introduction of the NetVLAD module, from 0.695 to 0.71. For "neutral" recognition, the accuracy factor increases by 0.06 after the introduction of the NetVLAD module, from 0.84 to 0.9. From these results it can be seen that after the introduction of the NetVLAD module, the accuracy of recognition of all categories of emotions improved, especially the recognition of "happy", which has the largest increase. This shows that the NetVLAD module plays a positive role in feature extraction and effectively improves the model's ability to recognize emotions. This result also confirms the advantage of the NetVLAD module in obtaining more valuable features with the clustering method, thus improving the discriminating ability of the model.

In addition, this study analyzed the RMSE of changes in emotion recognition scores in terms of emotional arousal and emotional valence with increasing repetitions. Figure 6 shows that the RMSE of both emotional arousal and emotional valence decreased during the model iterations and stabilized after about 30 iterations. For emotional arousal, the RMSE decreased from the initial 0.75 to the final 0.42, indicating that the model's prediction error for emotional arousal is gradually decreasing and the model's performance is improving. For emotional valence, the RMSE decreased from the initial 1.68 to the final 0.49, which also shows that the model's prediction error for emotional valence is gradually decreasing and the model performance is improving. In addition, the error reduction for emotional valence was greater than for emotional arousal, suggesting that the model outperformed emotional arousal in predicting emotional valence. In general, as the number of iterations increases, the model's prediction error gradually decreases, indicating a good learning performance. Especially after 30 iterations, the error is basically stable, which means that the model has reached convergence and further refinement cannot significantly improve the performance of the model. This result also confirms the effectiveness of our earlier strategy of introducing the NetVLAD module and using the StarGAN and ViT networks to recognize facial emotions.

Similar conclusions can be drawn by observing the curve of change in the coefficient of correlation coefficient. As the number of iterations increased, the consistency of the model's predictions of emotional arousal and emotional valence improved, indicating that the model has good learning outcomes (Figure 7). However, it should be noted that although the consistency correlation coefficient increased, it did not reach a value of 1, indicating that there is some discrepancy between the model's prediction results and the actual labels, and the model needs further improvement. These results further prove that the strategy of introducing the NetVLAD module and using the StarGAN and ViT networks for facial emotion recognition can effectively improve the consistency of model predictions, thus improving model performance.

Figure 5.Comparison of emotion recognition accuracy before and after the introduction of the NetVLAD module

Figure 6.Root mean square error (RMSE) variation curve.

Figure 7.Coherence correlation coefficient variation curve

Table 1.Comparing the performance of different network models

Table 2.Comparison of recognition accuracy of different models