Multimodal Efficient Computing: A learning evaluation application based on joint graphics and text processing (2023)

With the rapid development of artificial intelligence and deep learning, significant progress has been made in the field of computer vision and natural language processing [1-5], which has created the background for the emergence of MAC for joint text and image processing [6-12]. Traditional teaching evaluation emphasizes that students' mastery of knowledge is greater than emotions [13-15], and the state of emotions has a very significant impact on the learning process and student performance. MAC can comprehensively analyze all kinds of information about students in the classroom, including facial expressions, gestures, text feedback, etc., to help teachers detect students' emotional problems in time, adjust teaching methods and teaching methods. Appropriate strategies [16-18]. The research results obtained in this study can be directly used in the actual work of teaching evaluation, can help teachers to understand the educational needs and emotional states of students more fully and accurately, and provide useful evidence for educational decision-makers, thereby improving learning outcomes and promoting a sensible allocation of educational resources to achieve educational equity and improve the overall level of education.

It was proposed by scientists Pham and Wang [19].An attentive student 2, a multi-modal smart teacher that runs on an unmodified smartphone, complementing today's MOOC (Massive Open Online Course) click-through learning analytics. thisAn attentive student 2Using the front and rear cameras of smartphones as two complementary channels for accurate real-time feedback: the rear camera monitors students' photoplethysmographic (PPG) signals, and the front camera monitors their facial expressions during MOOC learning and indirectly to infer students' emotional abilities and cognitive states, learning from PPG cues and facial expressions. Barron-Estrada et al. [20] presented the initial implementation of a multimodal emotion recognition system using mobile devices and the creation of an emotion database using a mobile application. Recognizers work in mobile educational apps to recognize the user's emotions when interacting with the device. The emotions identified by the system are commitment and boredom. The emotion database was created based on the spontaneous emotions of students interacting with the educational mobile application Duolingo and the information-gathering mobile application EmoData. Hu and Flaxman [21] proposed a new method of multimodal sentiment analysis using deep neural networks combined with visual analysis and natural language processing. Their purpose differs from the target mood of standard sentiment analysis, which predicts whether a sentence expresses positive or negative moods. , aimed to infer users' underlying emotional states by focusing on predicting emotional word tags that users attach to their Tumblr posts, seeing them as "self-descriptive emotions." Yin et al. [22] pointed out that the use of deep learning methods to analyze multimodal physiological signals and recognize human emotions is becoming more and more attractive, but traditional deep emotion classifiers may not be able to determine the structure of the model and combine multimodal feature abstractions. oversimplification. Therefore, they propose a multi-layered, stacked autoencoder assembly classifier for emotion recognition, where deep structures are identified based on a physiological data-driven approach.

A thorough review of existing research shows that while MAC has some potential in evaluating learning, its shortcomings and challenges are also obvious. Existing MAC technology can cause unstable or erroneous judgments for complex emotional expressions, which in turn leads to inaccurate results of evaluating students' emotional states, which affects the effectiveness of learning evaluation. In addition, most existing MAC techniques rely on deep learning models that have poor interpretation, which can confuse teachers' understanding and trust in the evaluation results provided by the models. Given these issues, this article uses MAC to assess learning based on a combination of graphics and text. In the second chapter, the input text is split into two parts, the text and the hash tag, and feature extraction is done separately. The third chapter extracts image features from two angles of the subject and the scene, which can provide different levels of information about the image. Chapter 4 divides the MAC model into modality-based tasks and modality-based tasks to better accommodate new instructional evaluation scenarios. Experimental results confirm the effectiveness of the proposed method.

MAC can play an important role in teaching evaluation as it integrates text and image processing, detects and assesses students' emotional state based on provided feedback and facial expression images, and provides personalized learning resources and suggestions according to their situation. Emotional state and learning requirements by analyzing text responses and images posted by students in online tests or surveys can be used to help students understand and accept course content, and can also assess teacher performance and help mental health professionals Students analyze "conditions affective. These applications are conducive to improving the quality of teaching, equity in education and all-round development of students.

The use of MAC for learning evaluation requires the collection of the following textual and graphical data: 1) Students' textual data: including homework and exam answers, textual data from classroom discussions, forums, chat tools, and other scenarios, as well as their feedback and assessment by teachers and courses and teaching methods to understand their satisfaction and needs; 2) data on the student's image, including facial expressions, gestures and movements of students in classes and images used in classes, such as drawing, design, handicrafts, etc.; 3) text data about teachers, including lesson plans, brochures and text data from teacher-student interactions; 4) pictorial data about teachers, including facial expressions, positions and actions of teachers in the classroom.

In this study, the input text is divided into two parts: body and hash tags. Teaching assessment textual data tend to have high dimensionality, and the vector word segmentation expression can transform high-dimensional textual information into a low-dimensional vector form, thereby reducing computational complexity and improving the performance of training and prediction models. When the text is converted to a vector form by the word segmentation algorithm, the semantic information of the original text can be well preserved, which is convenient for improving the accuracy of the MAC model in the analysis of text features. The idea behind this method is to convert text into mathematical vectors, support comparison and calculation of similarities between texts, facilitate the identification of texts with similar emotional characteristics in a learning evaluation scene, and help improve the accuracy of emotional calculations. Figure 1 shows the working principle of the word segmentation algorithm.

picture 1.Principle of word segmentation algorithm

assumptions:vasrepresentvas- NO. a sample,I^Q_vasrepresents the evaluation text content input string,I^I_vasrepresents a string of hash tags, then the two parts of the text (content tags and hash tags) can be saved asI_vas={I^Q_vas,I^I_vas}; Let's assume thatQ_krepresentk- the first word in the text string of the learning assessment text,lostIndicates the length of the subject string, then the input subject string of the teaching assessment text can be saved asI^Q_vas={Q₁,Q₂,...,Q_k,...,Q_lost}; Similarly in principleI_krepresentk- prvi hashtag u nizu hashtagova,JIndicates the length of the hashtag string, then the hashtag string can be saved asI^I_vas={I₁,I₂,...,I_k,...,I_J}.

The process of converting text for teaching assessment into vectors is described by the following formula:

$G_{0}^{y}={{d}_{TE}}\lijevo( Y_{u}^{q};{{\phi}_{TE}} \desno),G_{0}^ {y}\w {{E}^{l\razy f}}$ (1)

After preprocessing the input learning assessment text, it is converted into a word vector in two steps. assumptions:Q_krepresents the input word,CIndicates the size of the corresponding generated dictionary, valueCequal to the size of the single vector, {0,0,0,...,0,1,0,...,0}∈Other^CRepresents each vector of words in one-time coding; if the dimension of the word vector is expressed asF, then the weight initialization matrix between the input layer and the output layer can be represented by the matrixask_Rthe size isC×F.

$Q_r=\lijevo(\begin{niz}{cccc}q_{11} & q_{12} & \cdots & q_{1 n} \\ q_{21} & q_{22} & \cdots & q_{2 n} \\ \cdots & \cdots & \cdots & \cdots \\ q_{C 1} & q_{C 2} & \cdots & q_{C n}\end{tablica}\right)$ (2)

Hash tags (such as: #keyword) can contain important emotional information in the text, i.e. keywords or phrases closely related to the content of the text, which help to analyze the emotional characteristics of the text in more detail. A routing based encoder can efficiently extract this key information and provide more valuable input for later efficient computations. In addition, keywords in hashtags may have similar or identical semantics to words in the text. By encoding and distinguishing hash tags, you can increase the weight of these keywords in the text vector to better capture the semantic information of the text.

assumptions:I_krepresentk- the first hashtag in the sample,JIt indicates the number of hash marks, and the operation to remove the "#" symbol for each hash character can be written asI^G_vas={I₁...,I_k,...,I_J}; Meanwhile, let's assumeG^G₀represents the hidden state of the generated hashtag,D_tenrepresents a text processing network,Phi_BMIndicates the parameters that the network needs to learn,G^G_vasrepresent eigenvectorsMean,D_BMRepresents a network that encodes hash tags, then the hash tag encoding process can be expressed by the following formula:

$G_{0}^{g}={{d}_{BM}}\left( Y_{u}^{g};{{\varphi}_{BM}}\right),g_{0}^ {g}\w {{E}^{l\razy f}}$ (3)

As with content processing, hash tags should also be represented in vector form, assumingC_Grepresents the number of hashtags,ask_G∈Other^CG×FRepresents a learnable matrix, then:

$r_k^g=Q_g \cdot y_k$ (4)

The initialization parameters of the neural network can be obtained by uniformly sampling over the interval such thatB_SRIB_SCShow the number of neurons in the input layer and the output layer:

$q \sim I\left(-\sqrt{\frac{6}{b_{S R}+b_{S C}}}, \sqrt{\frac{6}{b_{S R}+b_{S C}}} \右)$ ∀ (5)

photo 2.Hashtag extraction algorithm principle

In order to use MAC for learning evaluation, it is necessary to ensure which hash tags automatically learned by the model during training are associated with mood. In this way, when the model processes the learning assessment text, it can focus on hash tags that contain emotional information, thereby improving the accuracy of the emotional calculations. In order to implement the above algorithm, a soft routing method is used to optimize the encoding method based on the routing network. Instead of simply classifying the tags into two categories (pass or fail), the soft gate control method allows multiple hash tags to be passed with a certain probability, which can maintain the relative weight between the tags so that the model can be better The difference between hash tags can be accurately captured, and the effect of affective calculations can be enhanced. Figure 2 shows how the hash tag extraction algorithm works.

Using vectorsI_w ∈E^FProvide a representation of the global textual context for learning evaluation and calculate the similarity between current textsI_kand properI_wcan be done using a probability-based routing mechanismI_kThis can be judged by the closing mechanism. Specifically, let's assumeR^Grepresent a vector characterI_k, transform it non-linearly and find out its hidden expression; SupposeI^G_krepresents a hidden expressionk-th hashtag, transformation matrix and bias term can be expressed asask_Krab∈Other^F×FII^I_G∈Other^FThis means that the process can be described by the following formula:

$i_{k}^{g}=\nazwa operatora{Tanh}\left({{Q}_{gg}}\cdot r_{k}^{g}+n_{g}^{y}\right)$ (6)

tenThe sigmoid colonSelect a function to calculate the probabilityB_kEach hash tag selected by the gateway check engine:

${{\beta}_{k}}=\frac{1}{2+{{e}^{-i_{k}^{Y}\cdot {{i}_{v}}}}}$ (7)

Then take the weighted sum of the hidden expressions of all the hash tags:

${{r}^{g}}=\sum\limits_{k=1}^{j}{{{\beta}_{k}}\cdot r_{k}^{g}}$ (8)

Finally, a linear transformation of the hashtag representation is performed to incorporate information from all samples into the same semantic space. assumptions:G^G₀∈Other^FHashtag representing the final result,ask_Krab∈Other^F×FRepresents the transformation matrix, then there are:

$G_{0}^{g}={{Q}_{gg}}\cdot {{r}^{g}}$ (9)

In order to obtain more complete emotional information in order to apply MAC to learning evaluation, this study extracts image features from the two perspectives of the scene and the object, as these two perspectives can provide different levels of information about the image. Object features focus on specific objects in an image and the relationships between them, helping to identify objects and events in an image, while scene elements reveal the overall background and surroundings of an image, which can provide background and scene information. Object and scene features can complement each other, giving images richer semantic representations. For learning evaluation, image data can contain multiple objects and scenes. By considering the characteristics of these two aspects, the model's generalization ability to deal with new image data can be strengthened. Figure 3 shows the principle of operation of the image feature extraction algorithm.

photo 3.The principle of operation of the image feature extraction algorithm

The local target characteristics of the learning assessment scene images can be obtained using a target detection algorithm. assumptions:G^ul₀Represents the characteristics of an object-oriented image,Xrepresents the number of objects,Frepresents the dimensions of features,D_Leadrepresents the object detection network,Phi_LeadIt indicates the parameters that the network needs to learn, and then this process can be expressed as:

$G_{0}^{p}={{d}_{PB}}\lijevo( L;{{\varphi}_{PB}} \desno),G_{0}^{p}\w { E}^{x\razy f}}$ (10)

The local object image area of ​​the final teaching evaluation scene can be saved asR^ul={R^ul₁,R^ul₂,...,R^ul_X}, on the assumptionG^ul₀∈Other^X×FRepresents the characteristics of the local end object of the learning evaluation scene result,ask_{general practitioner}∈Other^F×FIt represents the transformation matrix, the non-linear transformation of the image features to the semantic space of the text is given by:

$G_{0}^{p}={{Q}_{gp}}\cdot {{r}^{p}}$ (11)

assumptions:G^A₀Represents scene-oriented image features,D_SCa grid representing a separate scene,Phi_SCIt denotes the parameters that the network needs to learn, and then the global scene feature extraction process is given by the following formula:

$G_{0}^{a}={{d}_{SC}}\left( L;{{\varphi}_{SC}} \right),G_{0}^{a}\u {{ E}^{f}}$ (12)

assumptions:G^A₀∈Other^FRepresents the characteristics of an object-oriented image of the final output,ask_ga∈Other^F×FIt represents the transformation matrix, the non-linear transformation of the image features to the semantic space of the text is given by:

$G_{0}^{a}={{Q}_{ga}}\cdot {{r}^{a}}$ (13)

After the feature extraction of the collected image and text data was completed, in order to further implement the use of MAC in the assessment of common classes of image and text processing, the MAC model was divided into modal shared tasks and modal private tasks in order to achieve better adaptability to new classroom scenarios. The modality sharing task can integrate information between different modalities and extract cross-modal sharing features, which is useful for the model to better capture the relationship between text and images and improve the accuracy of affective calculations. Private modal tasks focus on the intrinsic characteristics of each modality and can retain unique information within the modality, so that when the model combines multimodal information, it does not lose the characteristics of each modality, which may better reflect the assessment of the emotional situation of the learning scene information in. Thanks to such a division of tasks, the generalization ability of the model can be improved. This is because modality-shared tasks can learn the common features of different modalities, while modality-private tasks focus on the functions of different modalities. This design makes the model more flexible for new learning evaluation scenarios.

Figure 4.The modal private task principle

The key to achieving the modality-private task (Figure 4) is to predict and analyze each modality so that unique information within each modality can be extracted, namely Dirichlet distribution parameters and subjective opinions. This helps prevent the loss of modality characteristics when combining multimodal information and thus better reveal emotional information in learning evaluation scenarios. Then the prediction results for each mode are combined according to D-S proof theory. Since Dirichlet distribution parameters and subjective opinions can provide an uncertainty matrix for each model's prediction results, we need to apply D-S evidence theory to deal effectively with this incomplete, uncertain and conflicting information so that the model can flexibly combine ways of communicating with each other, improving thus the adaptability and accuracy of the model in teaching evaluation scenarios.

assumptions:big^IIbig^Crepresents the subjective output of each mode,B^IIB^Crepresents the Dirichlet distribution parameter,Otherrepresents the number of samples,JayIndicates pattern type number, $\hat{t}_v=\left[\hat{o}_1, \hat{o}_2, \ldots, \hat{o}_J\right]$ indicates that the final result prediction is result based on the D-S theory of proof, which includes any wayIRepresents an uncertainty estimate, then the loss function can be expressed as:

$Los{{s}_{UN}}=-\sum\limits_{u=1}^{B}{\sum\limits_{k=1}^{J}{{{t}_{uk}} \log \left( {{{\kapelusz{o}}}_{uk}} \right)}}$ (14)

The key to the task of sharing modalities (Fig. 5) is to build a feature fusion network based on tensor decomposition and use it to effectively combine features with different modalities, integrating multimodal text and picture information into one feature in the model's ability to capture emotional information in evaluation scenarios teaching. Using tensor decomposition, you can reduce the computational complexity of the feature-combination process and improve model performance. By performing a low-order approximation on multimodal features, the dimensionality of the combined features can be significantly reduced, thus reducing the computational load on the model. Finally, based on the results of feature fusion, multilayer perception (MLP) is used for predictive classification. MLP can effectively integrate multimodal functions, reduce computational complexity, capture high-level interactive information, and provide comprehensive training. It has good flexibility and scalability. Using MLP can provide a more valuable model design for your application. MAC in the classroom Rate the app in the scenario.

Figure 5.The principle of modal division of tasks

assumptions:G_IIG_CAfter the modal representation, it represents the feature vector of text and image; when combined, the feature tensor can be expressed asX=X_I⊗X_C, the tensor product can be expressed as ⊗, and the dimension of the feature vector can be expressed asF_IIF_C, and isX∈Other^fly×fc, then the eigenvectors of each model can be extended to one dimension by the formula above:

$\begin{align} & {{x}_{y}}=\Gamma \lijevo( {{G}_{y}},1 \desno),{{x}_{y}}\w {{ E}^{{{f}_{y}}}} \\& {{x}_{c}}=\Gama \lijevo( {{G}_{C}},1 \desno),{{ x}_{c}}\w {{E}^{{{f}_{c}}}}\\\end{align}$ (15)

assumptions:askRepresents the linear weight of the layer,Nrepresent prejudiceXrepresents a third-order tensor,ask∈Other^fly×fc×fGIt represents a fourth-order tensor, and the extra dimension is the dimension of the output vectorF_G,Goutput through the line layerH(""), these are:

$G=h\lijevo( X;Q,n \desno)=Q\cdot X+n,g,n\in {{E}^{{{f}_{G}}}}$ （16）

askcan be viewed as a superpositionF_Gsecond order tensorask^~_J∈Other^fly×fc.for everyoneask^~_J∈Other^fly×fs×fc, which can be broken down by rankOtheras follows:

${{\tylda{Q}}_{j}}=\sum\limits_{u=1}^{E}{q_{y,j}^{\left( u \right)}\czasami q_{c ,j}^{\lijevo( u \desno)}}$ （17）

assumptions:Q^(vas)_I,J∈Other^flyIQ^(vas)_C,J∈Other^fcRepresents the tensor factorization factorask^~_JCapital is classOtherRelevant text and images hereOtheris then set to constantOther- rank factorization factor {Q^(vas)_I,J,Q^(vas)_C,J}^Othervas₌₁,J=1,...F_Gcan be used to reconstruct tensorsask^~_J.allowQ^(vas)_I=[Q^(vas)_I1,Q^(vas)_I,2,...Q^(vas)_I,fG],Q^(vas)_Other=[Q^(vas)_Other1,Q^(vas)_Other,2,...Q^(vas)_Other,fG],Q^(vas)_C=[Q^(vas)_C1,Q^(vas)_C,2,...Q^(vas)_C,fG] followed by the weightaskin equation 16 can be represented by the following formula:

$Q=\sum\limits_{u=1}^{E}{q_{y}^{\left( u \right)}\otimes q_{s}^{\left( u \right)}\otimes q_ {c}^{\lijevo( u \desno)}}$ (18)

In addition, equation 16 can be written as:

$\begin{align} & g=\left( \sum\limits_{u=1}^{E}{q_{y,j}^{\left( u \right)}\czasami q_{s,j} ^{\left( u \right)}\otimes q_{c,j}^{\left( u \right)}} \right)\cdot X=\sum\limits_{u=1}^{E}{ \lijevo( q_{y,j}^{\lijevo( u \right)}\otimes q_{s,j}^{\lijevo( u \right)}\otimes q_{c,j}^{\lijevo( u \right)}\cdot X \right)} \\& =\sum\limits_{u=1}^{E}{\left( q_{y,j}^{\left( u \right)}\ ponekad q_{s,j}^{\lijevo( u \right)}\otimes q_{c,j}^{\lijevo( u \desno)}\cdot {{x}_{y}}\otimes {{ x}_{s}}\otimes {{x}_{c}} \right)} \\& =\lijevo( \sum\limits_{u=1}^{E}{q_{y}^{\左( u \right)}\cdot {{x}_{y}}} \right)\otimes \left( \sum\limits_{u=1}^{E}{q_{s}^{\left( u \right)}\cdot {{x}_{s}}} \right)\otimes \left( \sum\limits_{u=1}^{E}{q_{c}^{\left( u \右)}\cdot {{x}_{c}}} \右) \\\end{align}$ (19)

allowask_w∈Other^fG×1, the deviation is expressed asN_w, the emotion prediction result is represented by $\hat{t}_v$ , and finally the fully connected layer is used for multimodal emotion prediction, and the result can be expressed as:

${{\kapelusz{t}}_{v}}=Q_{v}^{Y}G+{{n}_{v}}$ (20)

The loss function of the modality sharing task can be expressed as:

$Los{{s}_{CO}}=\frac{1}{B}\sum\limits_{u=1}^{B}{{{\left( {{{\kapelusz{t}}}_ {u}}-{{t}_{u}}\右)}^{2}}}$ (21)

assumptions:BRepresents a hyperparameter, then the total loss function of the MAC model based on joint text and image processing can be expressed as:

${{s}_{MA}}={{s}_{CO}}+\beta {{s}_{UN}}$ (22)