Facial and facial expression recognition integrated into the Bottleneck module using deep learning and SNet architecture (2023)

Facial recognition is a biometric application used to access various applications such as unlocking mobile phones, timekeeping, health and security systems. Initially, there were many challenges with many traditional methods of authentication and security. For example, passwords and PINs are hard to remember and can easily be assumed to be stolen or forgotten. Smart cards, chip cards, and plastic cards can be lost or duplicated, and magnetic cards can darken or become illegible. An efficient technique is needed to overcome the ambiguity found in traditional authentication methods. To overcome these difficulties, biometrics and human characteristics such as fingerprint, iris and facial recognition are used for authentication. Among these options, facial recognition is the more important biometric element that promises authentication and security. The widespread use of smartphones and digital cameras has made the face recognition process simpler and more effective. Facial recognition can provide accurate, secure and fast authentication for secure access and surveillance.

Many researchers have conducted research in the visible (RGB) and infrared spectrum, but the above face recognition methods are affected by such factors as lighting, light, darkness or occlusion [1]. Therefore, in order to overcome these problems, thermal images are used for facial recognition, which work well in all lighting conditions. Thermal imaging images were recorded in the wavelength range from 8 μm to 12 μm [2]. Thermal images of the human face are created on the basis of thermal patterns emitted by the body, which are not affected by ambient lighting conditions [3].

Artificial intelligence (AI) refers to human behavior as described by a system or machine. Programming computers to mimic human behavior is the main task of artificial intelligence, and this is achieved using vast amounts of data from past patterns of the same behavior. Artificial intelligence is distinguished by various algorithms that use machine learning (ML) and deep learning (DL). Machine learning is a branch of artificial intelligence that automates the building of analytical models and prepares machines to independently adapt to new models. On the other hand, deep learning is a subset of ML that performs tasks better than traditional machine learning methods. Deep learning mainly combines two key techniques: supervised and unsupervised learning [4]. DL implements an artificial neural network with multiple hidden layers that takes the output of the previous layer as the input of the next layer. Deep learning is considered as a tool to improve performance and optimize processing time in various computational processes. Today, deep learning (DL) is becoming an effective facial recognition tool with impressive results. Convolutional Neural Network (CNN) is a deep neural network that automatically extracts visual features [5] DL automates the filter selection process to extract the best features from the image and provides better accuracy. Deep learning applications include image recognition, natural language processing, recommendation systems, and speech recognition [6]. The article focuses on the recognition of thermal images of human faces using deep learning (DL) embedded in the SNet architecture. The SNet architecture consists of a special block called the Squeeze and Excite (SE) block, which can improve the representational power of any network by explicitly modeling interdependencies between channels. The SE network is built by creating a series of several SE blocks. The function of the SE block is to calculate channel attention and provide increased efficiency at low cost. SENet provides an effective method to calculate channel attention with excellent performance. The main purpose of SENet is to impose inter-channel relationships by learning modulation weights in each channel [7]. While the SNet architecture is suitable for face recognition, the proposed SENet-Bottleneck Attention Module (SN-BNAM) method improves accuracy and reduces the error rate. The Bottleneck Attention Module (BNAM) is located between each SE block where information flow is critical. Before we get to BNAM, the concept of the attention module needs to be explained. Attention modules or attention mechanisms are DL techniques that provide additional emphasis on specific components of particular importance. Attention modules are used to improve the performance of convolutional neural networks (CNNs) by focusing only on more important functions and removing unnecessary functions [8]. Typically, attentional mechanisms apply to both channel and spatial dimensions. The proposed SN-BNAM method uses the attention module of Bottleneck Park et al. [9] The bottleneck attention module can be applied to any CNN. In the proposed system, BAM is incorporated into the SENet architecture to achieve operational accuracy. BAM divides the default 3D object map into two attention modules, called Channel Attention Modules and Spatial Attention Modules, which can be viewed as feature detectors to gain clear knowledge of where and what to focus on. Experimentation on various underlying architectures using our own Thermal Face Dataset (THFDS) shows that performance can be improved by adding SN-BNAM to the baseline. The effectiveness of the proposed method was also assessed by its implementation on CIFAR-100 and ImageNet-1K data sets, and the results are presented. Finally, performance improvements were achieved in object detection tasks in the VOC 2007 and MS COCO datasets, demonstrating the superiority of SN-BNAM.

1.1 Statement of the problem

There is a lot of research in the field of facial recognition systems using different algorithms and architectures. Facial recognition based on RGB images has been extensively studied with practical success. This research addresses the challenges of illusions, occlusions, light changes, and camouflage. Previous studies used multiple images of faces with different poses and lighting conditions to achieve maximum accuracy. This process requires a large database containing many images of the same person, leading to higher storage requirements and complex calculations. Calculations on such a large database are error-prone, time-consuming, and reduce performance accuracy. Although thermal images have been used for facial recognition in many studies, the proposed method can recognize not only facial images, but also basic facial expressions in any situation. Many traditional algorithms have been used for face recognition, but they are computationally intensive and unnecessary functions may be included in the recognition process, reducing the accuracy rate and increasing the error rate.

To overcome these problems, the proposed method uses thermal infrared images to recognize faces.

1.2 Contribution

The main contributions of this work are as follows:

(1) This work uses thermal infrared images for facial recognition, which overcomes the problems of facial illusion, repositioning, occlusion, expression and aging.

(2) The proposed SN-BNAM method implements the SNet architecture that integrates a lightweight bottleneck attention module.

(3) In the proposed method, the SE block increases the interdependence of the channels and improves the efficiency of the recognition process only at the computational cost.

(4) BottleNeck lightweight attention module focuses on combining channel attention with space attention to improve efficiency.

(5) The proposed system also detects basic facial expressions: happy, sad, natural, scared and camouflaged.

(6) Our proposed method works well under various environmental conditions, such as indoor, outdoor, day and night.

The rest of this article is arranged as follows. Part 2 presents a literature review, Part 3 presents the methodology of the proposed system, Part 4 explains the results of the proposed system and Part 5 presents the conclusions.

The performance of the infrared facial recognition system is affected by the temporal fluctuations present in the thermal images of the face, which are mainly caused by various environmental factors, physiological changes of the subject, and changes in the response of the infrared detectors at the time of registration. These five techniques Local Binary Pattern (LBP), Weber Linear Descriptor (WLD), Gabor Jet Descriptor, Scale Invariant Feature Transformation and Accelerated Robust Features were used to develop a thermal facial recognition system [1]. Recent advances in deep learning techniques in the field of thermal infrared facial recognition have enabled numerous research groups working on this topic to achieve many groundbreaking results [2]. In addition to identifying faces that cannot be seen in visible light, thermal infrared facial recognition can also identify facial blood vessel structures. It reviews previous research on temperature changes, mathematical equations, wave types, and infrared facial recognition techniques [3]. A new field of research on machine learning (ML) is deep learning. For large databases, deep learning methods use non-linear transformations and advanced model abstractions. Deep learning architectures have recently made significant advances in several areas, and these developments have had a significant impact on AI [4]. However, recognizing faces in everyday presence using convolutional neural networks (CNNs) remains a challenge due to sampling challenges. Samples can be increased by data augmentation, which is used for multi-stage learning [5]. Advances in machine learning in MFR have greatly aided the intelligent process of identifying and authenticating people with hidden faces. Using the characteristics of deep network architecture and deep feature extraction strategies to introduce state-of-the-art technology [6]. Triple attention encodes inter-channel and spatial information with minimal computational load on the input tensor while creating cross-dimensional relationships through rotation operations and residual transformations [7]. A convolutional neural network with a convolutional block attention module (CBAM) for finger vein recognition can achieve a more accurate capture of the visual structure by the attention mechanism and take into account the different meaning of pixels [8]. The Bottleneck Attention Module (BAM) is a simple and powerful attention module that can be integrated into any convolutional feedback neural network. Wide application of BAM is achieved by continuous improvement of the classification and detection efficiency of various models [9]. The Squeeze-and-Excitation Network (SENet) architecture excels at generalizing difficult datasets. Most importantly, we found that SE blocks significantly improve the performance of current state-of-the-art deep architectures at low additional computational costs [10]. Convolutional neural networks solve both of these problems together by creating AW convolutions where the shape of the attention map corresponds to the shape of the weights, not the activations. The attention module we propose [11] is a complementary approach to earlier attention-focused strategies, such as methods using attention mechanisms to study the relationship between channel features and spatial features. The Convolutional Block Attention Module (CBAM) is an Attention Module that we propose for use with all convolutional feedback neural networks. CBAM can be seamlessly integrated into any CNN architecture with minimal overhead as it is a lightweight general purpose module [12]. Residual meshes are easier to optimize and can improve accuracy with greatly increased depth. On the ImageNet dataset, we tested residual networks up to 152 layers, 8 times deeper than VGG networks, and at the same time less complex [13]. ResNeXt additionally uses the COCO detection kit and the ImageNet-5K kit, both of which show better performance than ResNet [14]. As a very deep family of architectures with impressive accuracy and good convergence, deep residual networks emerged. Using identity maps as skip links and post-add activations, we analyze the propagation formulation of residual building blocks and find that forward and reverse signals can be directly transmitted from one block to another [15]. An overall summary of the literature review is presented in Table 1.

Table 1.Literature review summary

This article proposes an SNet architecture that integrates a facial recognition method using the Bottleneck Attention Module (SN-BNAM). The proposed method uses a single face for facial recognition, and for the dataset, our own Thermal Face Dataset (THFDS), which consists of thermal face images. Thermal imaging overcomes other spectral challenges such as optical illusions, facial camouflage, aging, etc. This work also focuses on face recognition in various environmental conditions such as indoors, outdoors, day and night. In addition to face recognition, basic expressions such as happiness, sadness, nature, fear, and camouflage are also taken into account when recognizing facial expressions. Experiments are conducted taking into account all of the above factors and the results are presented in terms of accuracy, error rate, power, precision, recall and F1 score. Figure 1 shows a block diagram of the proposed method.

picture 1.Proposed SN-BNAM method

3.1 Architektura SENet

Figure 2 shows a model of the extrusion and excitation block. As can be seen in the figure, each Ftr transform mapping input X to U (feature map), where U$\in$R H×W×C, must perform recalibration of the feature, which is performed in the SE block [10 ].

photo 2.Squeeze i Excite blokovi SNet-a [10]

Input X is mapped to the feature map U, where U$\in$ is ℝ^{Height x width x depth}Initially, a compression operation is applied to U features to generate channel descriptors by collapsing feature maps along the spatial dimension (H x W). This descriptor performs the function of generating globally distributed channel response integration and enables all layers in the network to use information from the global receiver field in the network. This aggregation is followed by a gain operation, which is aided by a basic self-monitoring tool that accepts embedding as input and clusters modulation weights per channel. These weights are now used in the U feature map to obtain the output of the compression and excitation block, which can be fed directly to subsequent layers of the network.

3.1.1 Operation of the Extrusion and Excitation Block (SEB)

SEB creates a computational unit that is set to transform F_specialThe mapping from input X to the object map U $\in$ ℝ^{Height x width x depth}In the equation below F_specialTreated as the convolution operator, the set of filter kernels is denoted as V=[v₁, w₂,...,w_CThe result can be written as U=[u₁, To₂,...,was_C] where [10]:

$u_c=\mathbf{v}_c * \mathbf{X}=\sum_{s=1}^{C^{\prime}} \mathbf{v}_c^s \mathbf{X}^s$ (1 )

In the above formula, the operation * represents convolution and vs c is a two-dimensional spatial kernel, representing one channel, acting on the corresponding X and v channels_C=[w_C¹, w_C²CV_C^Cʹ], X=[x₁, X₂,...,X^Cʹ] And you_C$\w$ ℝ^{height x width}_.The output is generated by implementing the sum of all channels, so channel dependencies are embedded in v_Cimplicitly. Convolutional models describe channel relationships as implicit and local, but it would be better if these convolutional features were made explicit and the network had the ability to amplify their sensitive features. It is necessary to provide an unambiguous model for solving the issue of information features that are used by subsequent transformations. The SE block provides this functionality by allowing access to global information, and the filter response is recalibrated in two passes, blocking the excitation from entering subsequent passes.

3.1.2 Ironing procedure

To overcome the channel dependencies, all learned filters operate using the local receiving field so that each unit of the transformed output U cannot use the channel dependencies outside this region. The whole process is performed by a compression operation in which global spatial information is compressed into channel descriptors. Global average aggregation is used to generate statistics for each channel. Stats from $\in$ ℝ^CThis is achieved by reducing U along its spatial dimension HxW, where c^IZ elements were calculated according to [10]:

$z_c=\mathbf{F}_{s q}\left(\mathbf{u}_c\right)=\frac{1}{H \times W} \sum_{i=1}^H \sum_{j= 1}^Wu_c(i, j)$ (2)

The output U is treated as a set of local descriptors whose statistics represent the whole picture.

3.1.3 Excitation work

The information collected during the compression operation is used in the next operation where the channel correlations are fully recorded. This goal is achieved according to the following criteria: (1) the function should be able to learn non-linear interactions between channels and (2) it must learn non-linear mutual exclusion relationships. These criteria are met by using the following basic routing mechanism [10]:

$s=F_{e x}(\mathbf{z}, W)=\mho(y(z, W))=\mho\lijevo(W_2 \delta\lijevo(W_1 z\desno)\desno)$ (3 )

The upper activation of the $\delta$ sigmoid is a function of ReLU, W1$\in$ℝC/r X C and W2$\in$ℝC×C/r. The routing mechanism is parameterized by the bottleneck of two fully connected layers, a dimensionality reduction layer with a factor of r and a dimensionality enhancement layer that returns the dimension of the output channel U. The final output of the block is activated by [10]:

$\widetilde{X_c}=F_{\text {skala}}\left(u_c, s_c\right)=s_c u_c$ (4)

F_measure(was_C, Other_C) refers to the multiplication of channels between scalars_Cand map of objects in_Cℝ^{height x width}_.

3.2 Attention module about bottlenecks

Reminders module:Attention mechanism techniques capture long-term function interactions and improve CNN representations [11]. For any input image, the two modules of attention, channel and space, compute complementary attention to focus on what and where. Two modules can be mounted parallel to each other or one behind the other. Experimental results show that keeping channel attention modules in a sequential manner leads to better results first [12].

Note that the modulus is constant when a bottleneck occurs when sampling the feature map. BAM constructs a hierarchical bottleneck of attention that can be trained using a feedback model. In the proposed SN-BNAM method, this module is integrated with the SNet architecture, thanks to which the advantages of SNet and BAM are used in the face recognition process to achieve greater accuracy.

3.2.1 Channel Focus

This module performs feature extraction and reduces data loss by compressing feature maps. This process is performed using a global pool averaging layer that generates general feature map informationFand creates a channel vector F_C$\w$ ℝ^C×1×1.Using the F channel vector_C,Attention is estimated using multi-layered perception (MLP) with one hidden layer and an M-channel attention map_C(F) $\w$ ℝ^C×1×1is produced. Reduce parameter loading by setting hidden activation size to ℝ^C×1×1where r is the reduction factor. After the mesh is applied, a volume normalization (BN) layer is added to scale the result in space. Channel attention is calculated as follows:

Rice_C(F)=BN(MLP(medium set(F)))=BN(W1(w0ŚredniaPula(F)+b0)+b1) (5)

U gornjoj jednadžbi, $\mathrm{W}_0 \in \mathbb{R}^{\mathrm{C}/\mathrm{r}\times\mathrm{C}}, \mathrm{W}_1 \in\mathrm {C}} {R}^{\mathrm{C}\times \mathrm{C}/\mathrm{r}}, \mathrm{b}_0 \in \mathrm{R}^{\mathrm{C}/ \mathrm{ r }}$ i $\mathrm{b}_1 \in \mathbb{R}^{\mathrm{C}}$. Ze sądza na MLP network dijeleja network, dva ulaza dijele waży $W_0$ i $W_1 $ .

3.2.2 Spatial attention

The Spatial Attention Module provides a spatial M map of attention_small(F) $\w$ ℝ^{height x width}Disable items in different spatial locations. This module is used to determine which spatial locations to focus on to retain important contextual information. A large-scale receptive field that can effectively use contextual information is required. Perform an extended turn to increase the receptive field more effectively. Extended convolutions construct an efficient spatial map compared to standard convolutions. The spatial branch adopts a bottleneck structure, which reduces the number of parameters and computational load. The feature F belongs to ℝ^{height x height x width}Reduced to size ℝ^{C/ Receiver height x width}Assemble using 1×1 convolutions and compressed feature maps in the channel dimension. After the reduction process is completed (reduction factor r), two expanded 3×3 weaves are used to use the information efficiently. Again, the features in the spatial attention map have been reduced to r1xhxw using a 1×1 convolution. Mass normalization is applied at the end of the spatial scaling branch. Spatial attention is calculated as follows:

$M_s(F)=B N\left(f_3^{1 X 1}\left(f_2^{3X3}\left(f_1^{3X3}\left(f_0^{1X1}(F) \右)\右)\右)\右)$ (6)

Where,FDenotes the convolution operation, the BM series normalization operation, with superscript asFIndicates the size of the convolution filter. Two 1×1 turns are used to shrink the channel, and the middle 3×3 extended weave is used to gather contextual information in a large receptive field.

3.2.3 BAM wheel

Figure 3 shows the general structure of the bottlenecked attention module. Given the input object map F$\in$ ℝ^{length x height x width}, which derives the three-dimensional attention map M(F) $\in$ ℝ^{Length x Height x Width.}To develop an effective module, first focus on M_C(F) $\w$ ℝ^Cand spatial attention M_small(F) $\w$ ℝ^{height x width}As two independent branches, and finally compute M(F), the attention map looks like this:

M(F)为：M(F)=σ(Mc(F)+Ms(F)) (7)

Once channel attention and spatial attention are gathered, they are combined to create the final 3D attention mapmen and women).The two attention maps have different shapes, so they are elongated to ℝ before merging^{length x height x width}This fusion is performed using elemental summation followed by a sigmoid function to create the final 3D attention mapmen and women)Between 0 and 1.

Combining channel attention Mc and spatial attention Ms, the attention map M(F) is computed by the following equation:

M(F)为：M(F)=σ(Mc(F)+Ms(F)) (8)

A 3D attention map was obtainedmale female)ℝ^{length x height x width}and enter the object mapFare multiplied by elements and added to the original input feature map to produce a refined feature map, using the following equation, where ⊗ stands for multiplication by elements.

F'=F+F⊗M(F) (9)

photo 3.Detailed view of the BottleNeck attention module

To evaluate the proposed SN-BNAM method, other network models (such as ResNet18, ResNet50) were implemented using our THFDS dataset and the experimental results are shown in Table 2. As shown in Table 2, the results of architectures such as ResNet18, ResNet50 and ResNet50 are compared with the results of other modules when the proposed SN-BNAM module is integrated with other modules. Integration of the proposed modules with other architectures can reduce the error rate and provide better performance. The proposed SN-BNAM method outperforms other architectures and improves facial recognition accuracy while reducing the error rate. Then our proposed system is also implemented on CIFAR-100 and ImageNet-1K datasets for effective evaluation. Table 3 and Table 4 show the performance evaluation of our SN-BNAM method on the CIFAR-100 and ImageNet-1K datasets. The results show that our method outperforms other traditional methods and lightweight network architectures in performance and error reduction.

Table 2.Classification results of other network models and our THFDS dataset

Table 3.Results of the SN-BNAM classification on the CIFAR-100 dataset

Table 4.Results of the SN-BNAM classification in the ImageNet-1K dataset

table 5.MS COCO Object Detection [ECCV Reference Document] (Completed)

Table 6.mAP for object detection on VOC 2007 test set Apply StairNet detection framework and apply SE, CBAM and SN-BNAM to the detector

(A)

(two)

(C)

(four)

(pet)

Figure 4.Performance comparison with state-of-the-art algorithms. (a) Comparison of accuracy with other traditional algorithms. (b) Performance of different algorithms under different environmental conditions. (c) and (d) show a comparison of the power and accuracy of the algorithms for detecting the 5 basic facial expressions. (e) recall and F1 score of different algorithms

4.1 Detecting MS COCO Objects

The performance of the proposed system was evaluated in terms of object detection using the Microsoft (MS) COCO dataset [22]. Our model is trained using all training images, a subset of validation images, and 5,000 sample validation images. Adopt Faster-RCNN [22] as our discovery method and select pre-trained ResNet101 [15] ImageNet as the underlying network. Improve performance by including SN-BNAM in the underlying architecture. As can be seen in Table 5, our proposed model, SN-BNAM, shows a significant improvement and outperforms other methods.

4.2 VOC Object Detection 2007

The experiments are carried out on the PASCAL VOC 2007 test set, where the SN-BNAM is superimposed on the detector. The proposed SN-BNAM method adopts the StairNet framework [23], which is the strongest multi-scale method based on SSD [24]. The SN-BNAM is placed in front of each classifier to refine the final features before the prediction process and force the model to select significant features.

Experimental results are listed in Table 6 and it is clear that SN-BNAM achieves higher accuracy compared to strong underlying architectures. The increase in precision is achieved with a small increase in parameters. The light network results also show that SN-BNAM can also be implemented in less expensive devices.

4.3 Performance comparison with state-of-the-art algorithms

The performance of the proposed system is compared with the most modern algorithms, such as Support Vector Machine (SVM), K-Nearest Neighbors (KNN), Naive Bayes (NB), Random Forest (RF), Linear Regression (LR). ) and the SNet architecture. The results are shown in Figure 4. Performance comparisons are made using metrics such as accuracy, power, precision, memory, and F1 score. Experiments are conducted in a variety of environments, such as indoors, outdoors, day and night, and it is clear that the proposed SN-BNAM method is superior to other state-of-the-art algorithms. The strength and accuracy of five basic facial expressions are reported, which are happy, sad, natural, scared and camouflaged. Compared to all other algorithms, the method we propose performs better in all aspects.