Facial expression recognition using convolutional neural network along with data augmentation and transfer learning

Abstract

Facial expression recognition (FER) is a most interested popular area of the modern technological world. Some of the applications of the FER system are patient condition monitoring, games, human-computer interaction, security, automated access control and video surveillance. Modern research uses transfer learning models for facial expression recognition using convolutional neural network (CNN). The used datasets are CK+ and JAFFE. A large number of training images are required as input for training FER with CNN. Therefore, different data augmentation methods were used such as flip, crop, Gaussian blur, contrast normalization, additive Gaussian noise, multiply, scale, translate percent, shear, and rotate. In this study synthesized 10 images from one existing image in a test dataset. The first step is to call a transfer learning model by using "tenserflow.kera.application" and "tenserflow.keras.layers" to import layers into the network. The second step is freezing all layers of the transfer learning model using "layer.trainable=False" to avoid the problem of over-fitting and to avoid training the entire network. Unfreeze the final eight layers to detect fine information in the images such as edges of the images After this step, the model fit well. Then it can be fine-tuned using "layer.trainable=True". The next step is to feed the CNN model the test images and label list, and then perform stratified 5-fold cross validation. The proposed method uses transfer learning models such as DenseNet121, DenseNet201, DenseNet169 and InceptionV3 and fine tuning for augmented datasets CK+ and JAFFE. This proposed method achieves 99.36% for the CK+ dataset and 99.14% for the JAFFE datasets.