Emotion Recognition of Foreign Language Teachers in College English Classroom Teaching

In order to improve students’ learning effect, more and more universities favor foreign language teachers who are native speakers of English. Based on the analysis and summary of the research status of emotion recognition, this paper proposes that, in college English classroom teaching, foreign language teachers can reduce the communication barriers with Chinese students through emotion recognition. Based on literature review and actual situation investigation, this study identified four influencing factors on emotion recognition of foreign language teachers, namely, interactive action, facial expression, vocal emotion, and body posture. In our opinion, in the teaching process, teachers can adjust the four factors of emotion recognition to achieve better teaching effect. Further, improve students’ learning efficiency. Analytic Hierarchy Process (AHP) is chosen as the research method in this study. After building the analysis model, we collected the questionnaire using the Questionnaire Star, and finally got 12 valid data. After determining the importance of different factors by pairwise comparison, we draw the following conclusions: the influence degree of emotion recognition factors of foreign language teachers is in descending order, interactive action (43%), facial expression (28%), vocal emotion (21%), and body posture (9%). Our research adds to the body of knowledge on emotion recognition among college English teachers. Furthermore, this research assists students in improving their grasp of course content based on the emotions of foreign English lecturers. Based on the findings, we recommend that foreign language teachers in college English classrooms alter their interactive behaviors, facial expressions, and vocal emotions in response to various instructional materials and emphases.

Introduction

From the perspective of internationalization, the exchanges in economy, culture, and other aspects of the world are deepening. Therefore, there are higher requirements for Chinese people’s intercultural and intercultural communication skills. English is the most common language in the international market. In China, the majority of universities set English as a compulsory course for students’ future development in the world, so as to improve students’ English level and enhance their ability to communicate in English. As an additional language course, English requires students to spend a lot of time learning basic vocabulary and sentences and applying them in daily life communication. However, it is challenging for students to express more specialized and special usages through word selection, sentence combination, and the use of the culture hidden behind the language (Australian Curriculum, Assessment and Reporting Authority, 2016). Foreign teachers who speak English as their mother tongue have certain advantages in vocabulary use and cultural understanding. Therefore, more and more universities choose to hire them as English course teachers. However, in the process of learning a second language, especially a foreign language, students will inevitably have anxiety in the face of knowledge they do not understand (Chen and Lee, 2011). Moreover, anxiety is an important reason for students’ poor learning effect (Woodrow, 2006).

Emotion recognition (Cai et al., 2021) is considered to be an effective method to promote classroom teaching, reduce students’ anxiety, and enhance students’ learning effect (Chen and Lee, 2011; Xiao et al., 2021). Generally speaking, emotion recognition includes facial expression analysis (Yu, 2020), speech analysis (Ando et al., 2021), physiological signs analysis (Kort et al., 2002), observable behavior analysis (Vicente and Pain, 2002), and so on (Cai and Wei, 2021). It will make the English course more attractive to students by constructing the relationship between language, pronunciation, gesture, and other meaningful ways of expression.

With the vigorous development of information technology (Miao et al., 2021; Wu and Chu, 2021; Zhao et al., 2021), there is a growing tendency for universities to employ foreigners as English teachers. More and more scholars have begun to study the emotion recognition of foreign language teachers in English curriculum teaching. For example, Yu (2020) pointed out that emotion recognition is helpful for English course teaching quality assessment. In terms of English writing, and Unsworth and Mills (2020) pointed out that creating an appropriate body context through the combination of multiple modes such as body language and sight line can effectively help students to expand and use language ability. However, most of the research on Chinese university curriculum focuses on mandarin teaching. Therefore, the emotion recognition of foreign language teachers in English courses has not received much attention. Therefore, this study tries to answer the following questions: (1) what factors can affect the emotion recognition of foreign language teachers? (2) to what extent these factors can affect the emotion recognition of foreign language teachers? (3) how should foreign language teachers adjust the emotion recognition factors to help students to improve their comprehensive English level? We first establish a research model that can be used to analyze the emotion recognition of college English teachers. Then the analytic hierarchy process (AHP) is used to analyze the collected data and determine the weight of each factor used to represent emotion recognition. Our research adds to the body of knowledge on emotion recognition among college English teachers. Furthermore, this research assists students in improving their grasp of course content based on the emotions of foreign English lecturers.

The rest of the paper is arranged as follows: the second part describes the work related to emotion recognition of foreign language teachers; the third part presents the analytical method and experimental results; the fourth part discusses the results; and, in the fifth and last part, we put forward the research conclusions, limitations and possible future research directions.

Related Work

Emotion Recognition in Teaching

In the field of teaching, emotion recognition is often used to test students’ learning effect. For example, in the virtual teaching environment, Yang et al. (2018) pointed out that teachers can pay attention to students’ state through emotion recognition, and then adjust teaching strategies to achieve better teaching effects. Similarly, Zhang and Zhang (2020) believe that emotion recognition has advantages in improving the effect of O2O English teaching. In real class, teachers can also monitor students’ English learning emotions in real time through facial emotion recognition machines to switch teaching content and teaching focus (Cui et al., 2021). Alternatively, long-term collection of facial images can be used for emotion recognition and analysis to find patterns (Oo et al., 2019). In addition, emotion recognition has also been applied in the training of pre-service teachers. Through the virtual situation teaching simulation system, the teaching ability of pre-service teachers is cultivated in order to achieve better learning in the future (Park and Ryu, 2018).

Factors of Emotion Recognition

Voice is an important part of emotion recognition. Neumann and Vu (2018) took English and French as examples to construct a cross-lingual and multilingual voice emotion recognition classification model. Ando et al. (2021) studied the differences in the perception of voice emotions by different listeners and considered as the individuality of voice emotion recognition. In addition, facial expressions (Zhang et al., 2021) play an important role in cross-cultural communication. In addition to voice, people tend to recognize emotions through visual appearance (Miyasato and Nakatsu, 1997). Facial expression is an indispensable part of emotion recognition (Park and Ryu, 2018; Oo et al., 2019). Some scholars also pay attention to the influence of posture and behavior on emotion recognition. Human behavior often carries more emotional information than language (Cui et al., 2021). Brone and Jeroen (2017) point out that it is necessary for teachers to guide students to construct English comprehension schema by using the body mechanism in the teaching process to help students to understand language. In English courses, teachers’ use of language, vision, gesture, and other communication methods is conducive to reducing students’ learning anxiety and improving their application ability of complex words and sentences (Unsworth and Mills, 2020).

Teaching emotion recognition is an important research field in emotion recognition. According to the above literatures, teachers can adjust teaching methods and strategies by identifying students’ emotions. This provides theoretical reference for our research. Accordingly, can teachers adjust their teaching methods according to students’ identification and evaluation of teachers’ emotions? However, there are relatively few studies from this perspective. And, this is a question worth thinking about in college English courses. Because when students are faced with difficulties in learning a second language and communication barriers with foreign language teachers, foreign language teachers can help students to obtain better learning results by adjusting the expression forms of emotion recognition.

Methods and Results

Identification of Emotion Recognition Factors

Combined with literature research and practical investigation, we identify four factors of emotion recognition in college English classroom teaching. They are facial expression, voice mood, physical gestures, and body movement, as shown in Table 1. We believe that in English class teaching, teachers can adjust the content identified by students’ emotion by adjusting these four factors, and further, let students obtain better learning effect.

TABLE 1

Table 1. Factors of foreign language teachers’ emotion recognition.

Selection of Methods

In order to explore the influence of different factors on emotion recognition of foreign language teachers, we choose AHP as the main research method. AHP is a decision-making and selection method, which combines qualitative and quantitative research. This method is generally used to solve complex multi-objective decision issues. It takes the form of expert scoring by comparing different factors in pairs. Then calculate the influence weight of each factor according to the collected data.

At present, the AHP is widely used in the research on the weight analysis of influential factors. For example, Su et al. (2015) used AHP to determine the influencing factors of inter-task dependence. Park and Lee (2020) chose AHP to analyze the influencing factors of airport passenger station efficiency. Akbar et al. (2020) consider this approach to be an effective tool in identifying and prioritizing DevOps success factors. It can be seen that this method has been widely used in different fields in judging the weight of influencing factors. Therefore, AHP is a feasible and effective way to determine the weight of influencing factors of foreign language teachers’ emotion recognition.

Data Collection and Conformance Check

Based on the influencing factors of foreign language teachers’ emotion recognition identified above and the principle of AHP, we constructed an emotion recognition model of foreign language teachers in college English classroom teaching, as shown in Figure 1.

FIGURE 1

Figure 1. Emotion recognition model.

After the model was built, YAANP software was used as a research tool and a questionnaire was designed, as shown in Table 2. The purpose of this questionnaire is to determine the relative weight of each influencing factor of emotion recognition. The questionnaire was designed according to the method of AHP. This method compares the importance of influencing factors in pairs at the same level. The measurement scale is divided into five grades, among which the values of 5, 4, 3, 2, and 1 correspond to absolute importance, very important, relatively important, slightly important and equally important, respectively. The hierarchical cells on the left indicate that the left-column factors are more important than the right-column factors. If you select the right rank cell, it means that the right column is more important than the left column.

TABLE 2

Table 2. Scale for comparison of factors in pairs.

Students that match the following criteria are chosen to participate in the survey: (a) students who have taken 1-year college English courses and (b) the students’ foreign language teacher is a foreigner who speaks English as a first language. Questionnaires were sent by mail to students who met the requirements. The subject of the email is to evaluate the influence of four factors of foreign language teachers’ emotion recognition on the improvement of students’ learning effect in college English class. Finally, after half a month, 14 questionnaires were collected, in which the names and schools were replaced by numbers in the questionnaires.

We imported the collected questionnaires into YAANP software. Furthermore, the consistency of the questionnaire data is tested. The results showed that two of the 14 questionnaires collected failed the consistency test (CR>0.1). Thus, we have a total of 12 valid data sets (CR<0.1). The original data we obtained are shown in Table 3.

TABLE 3

Table 3. Original data of valid samples.

Data Analysis

In YAANP software, we used arithmetic average to process the 12 valid questionnaires collected. The judgment matrix of the target layer and criterion layer of emotion recognition of foreign language teachers in college English classroom teaching calculated by us is shown in Table 4. Among them, the target layer is the influence of foreign language teachers’ emotion recognition on improving students’ learning efficiency, and the criterion layer is the influence of physical gesture, facial expression, voice mood, and body movement.

TABLE 4

Table 4. Judgment matrix table of target layer and criterion layer.

According to the judgment matrix data in Table 3, we calculate the matrix λmax=4, CR=0. This shows that the judgment matrix meets the consistency requirements. Therefore, the aggregate data collected are valid. Furthermore, we use YAANP software to calculate group decision-making and finally determine the weight of the four influencing factors of foreign language teachers’ emotion recognition in college English classroom teaching. The results showed that facial expressions were weighted at 28%, vocal emotions at 21%, body postures at 8% and interactive movements at 43%, as shown in Figure 2. Therefore, we find that foreign language teachers’ body movements and facial expressions have a great influence on students’ learning effect, with a total weight of more than 70%.

FIGURE 2

Figure 2. Weight of each factor. The abscissa is the weight of each factor.

Discussion

The vast majority of human communication is closely related to emotion recognition. For example, facial expressions, gestures, body posture, and tone of voice can reflect people’s emotions to a certain extent, thus affecting communication. More and more scholars are attaching importance to the research of emotion recognition in teaching application.

Through literature analysis, this study identified four specific factors affecting the emotion recognition of foreign language teachers in college English classroom teaching, which are facial expression, voice mood, body movement and physical gestures. Then, we use the AHP and YAANP software’s group decision calculation method to determine the weight of the influence of each factor of foreign language teachers’ emotion recognition on students’ learning effect. Data analysis results show that when foreign language teachers take body movements in college English classroom teaching, they have the greatest influence on students’ learning effect, accounting for 43% of the emotion recognition. Facial expressions are much easier to regulate and control than voice mood. In this study, the facial expressions of foreign language teachers have 28% impact on students’ learning effect during the teaching process. The influence degree of voice mood on students’ learning effect was 21%. Finally, the teacher’s physical gesture had the least impact on students’ learning outcomes, at just 8%. The findings add to the body of knowledge on emotion recognition among college English teachers. In addtion, this study can assist students in improving their grasp of course content based on the emotions of foreign English lecturers.

In order to ensure a high level of college English classroom teaching and improve the quality of teaching, foreign language teachers need to reasonably adjust the factors of emotion recognition to provide students with a better English classroom experience. First, from the above data analysis results, it is not difficult to find that body movements play an important role in the emotion recognition of foreign language teachers. Teachers should reasonably use interactive actions to create body context and guide students to understand complex language and the culture contained in language according to the teaching content and focus in class. This can help students to better understand the knowledge in class. For example, when explaining travel-related knowledge points, foreign language teachers can describe the differences between Chinese and foreign countries in taking taxis through body language. Chinese hail taxis by waving their hands. However, in parts of the United States, the thumbs-up is the correct gesture for getting a taxi. This teaching method can help students to enter the teaching situation faster and to better understand the teaching content.

Second, we suggest that foreign language teachers interact with students through expressions such as eyes and smiles, so as to reduce the sense of distance with students. In the classroom, a teacher with a serious face tends to increase the sense of distance with students. Further, it will have a negative impact on the initiative of communication between students and teachers. A teacher who smiles and often makes eye contact with students will be more popular. In this teaching environment, students’ interest in learning and love for college English courses will be improved.

Finally, voice mood can help students to establish associations (Park and Ryu, 2018). For the complex sentences that are difficult to understand, it is helpful for students to understand the knowledge points better and faster by driving students into the specific context through the voice mood. In the process of teaching, foreign language teachers change the voice mood according to the teaching content. For example, the teachers will use happy, sad and depressed pronunciation to teach. Furthermore, by guiding the learners’ emotional state, they can be attracted to participate in learning better.

Conclusion

English is a compulsory course for most Chinese college students in their first and second years. In order to improve the teaching quality, more and more schools choose to hire foreign teachers to teach English courses. However, the differences in teaching methods, teaching environment and classroom teaching habits between China and foreign countries inevitably cause difficulties for students in communicating with teachers and receiving knowledge. In order to reduce the barriers between teaching and learning, this paper studies the factors of emotion recognition of foreign language teachers in college English teaching. Through literature analysis, we believe that the influencing factors of foreign language teachers in emotion recognition include interactive action, facial expression, voice emotion, and body posture. In our opinion, in the teaching process, teachers can adjust the four factors of emotion recognition to achieve better teaching effect. The data analysis results show that the weight proportion of the four factors on students’ learning effect is body movements (43%), facial expression (28%), voice mood (21%) and physical gestures (9%). English courses play an important role in college courses. How to make students learn English better and improve their English scores is the focus of many colleges and universities. Therefore, we suggest that foreign language teachers should reasonably adjust their interactive behaviors, facial expressions and vocal emotions in college English classes according to different teaching contents and teaching emphases. This can improve students’ sense of immersion in class, so that students are better engaged in learning.

It is worth noting that part of our research has not been fully considered, and there are many aspects that need to be further explored and analyzed. First, in terms of the influence of foreign language teachers’ emotion recognition on students’ knowledge mastery, our study ignores the differences between English majors’ learning in compulsory English courses and students’ learning in general English courses. Whether there is a difference between the two and how big the difference is, these are questions worthy of further study. Second, we consider four factors of foreign language teachers’ facial expressions, voice mood, body movements, and physical gestures, but the subdivision of each factor is also important. For example, sound includes speed, pitch, amplitude, etc., and it is also worth further studying to what extent these sub-factors influence foreign language teachers in college English classroom teaching. Third, our overall sample size is small, so we can increase the sample survey in the future. Finally, the interaction between the factors and the relationship between them and teacher’s emotion recognition is also worth discussing.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

LL was responsible for designing the framework of the entire manuscript from topic selection to solution to experimental verification.

Conflict of Interest

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Akbar, M. A., Mahmood, S., Shafiq, M., Alsanad, A., Abdul-Aziz Alsanad, A., and Gumaei, A. (2020). Identification and prioritization of DevOps success factors using fuzzy-AHP approach. Soft. Comput. 1–25. doi: 10.1007/s00500-020-05150-w

CrossRef Full Text | Google Scholar

Ando, A., Mori, T., Kobashikawa, S., and Toda, T. (2021). Speech emotion recognition based on listener-dependent emotion perception models. APSIPA Trans. Signal Inf. Process. 10, 1–11. doi: 10.1017/ATSIP.2021.7

CrossRef Full Text | Google Scholar

Australian Curriculum, Assessment and Reporting Authority (2016). The Australian curriculum: English. 8.2. Available at: http://www.australiancurriculum.edu.au/ (Accessed October 30, 2021).

Google Scholar

Brone, G., and Jeroen, V. (2017). Cognitive Poetics: Goals, Gains, Andgaps. Shanghai: Shanghai Foreign Language Education Press.

Google Scholar

Cai, W., Gao, M., Liu, R., and Mao, J. (2021). MIFAD-net: multi-layer interactive feature fusion network with angular distance loss for face emotion recognition. Front. Psychol. 12:762795. doi: 10.3389/fpsyg.2021.762795

PubMed Abstract | CrossRef Full Text | Google Scholar

Cai, W., and Wei, Z. (2021). PiiGAN: generative adversarial networks for pluralistic image inpainting. IEEE Access 8, 48451–48463. doi: 10.1109/ACCESS.2020.2979348

CrossRef Full Text | Google Scholar

Chen, C. M., and Lee, T. H. (2011). Emotion recognition and communication for reducing second-language speaking anxiety in a web-based one-to-one synchronous learning environment. Br. J. Educ. Technol. 42, 417–440. doi: 10.1111/j.1467-8535.2009.01035.x

CrossRef Full Text | Google Scholar

Cui, Y., Zhao, R., and Wang, S. (2021). Machine learning-based student emotion recognition for business English class. Int. J. Emerg. Technol. Learn. 16, 94–104. doi: 10.3991/ijet.v16i12.23313

CrossRef Full Text | Google Scholar

Kort, B., Reilly, R., and Picard, R. W. (2002). “An affective model of interplay between emotions and learning: reengineering educational pedagogy-building a learning companion.” in Proceedings IEEE International Conference on Advanced Learning Technologies; August 6–8, 2001. 43–46.

Google Scholar

Krause, F. C., Linardatos, E., Fresco, D. M., and Moore, M. T. (2021). Facial emotion recognition in major depressive disorder: a meta-analytic review. J. Affect. Disord. 293, 320–328. doi: 10.1016/j.jad.2021.06.053

PubMed Abstract | CrossRef Full Text | Google Scholar

Matsumoto, D., Leroux, J., Wilson-Cohn, C., Raroque, J., Kooken, K., Ekman, P., et al. (2000). A new test to measure emotion recognition ability: Matsumoto and Ekman's Japanese and Caucasian brief affect recognition test (JACBART). J. Nonverbal Behav. 24, 179–209. doi: 10.1023/A:1006668120583

CrossRef Full Text | Google Scholar

Miao, M., Cai, W., and Li, X. (2021). Parameter estimation of gamma-gamma fading with generalized pointing errors in FSO systems. Wirel. Commun. Mob. Comput. 2021:1301878. doi: 10.1155/2021/1301878

CrossRef Full Text | Google Scholar

Miyasato, T., and Nakatsu, R. (1997). “Facial emotion recognition using multi-modal information, communications and signal processing.” in ICICS Proceedings of 1997 International Conference; September 12–12, 1997.

Google Scholar

Neumann, M., and Vu, N. T. (2018). “Cross-lingual and multilingual speech emotion recognition on English and French.” in ICASSP; April 15–20, 2018. 5769–5773.

Google Scholar

Oo, T. T., Boonroungrut, C., and One, K. (2019). Exploring classroom emotion with cloud-based facial recognizer in the Chinese beginning class: a preliminary study. Int. J. Instr. 12, 947–958. doi: 10.29333/iji.2019.12161a

CrossRef Full Text | Google Scholar

Park, S. U., and Lee, J. D. (2020). Analyzing factors that influence the efficiency of airport passenger terminal by using AHP (analytic hierarchy process) method (case study of Juanda airport – Surabaya, Indonesia). KSCE J. Civ. Eng. 24, 3856–3865. doi: 10.1007/s12205-020-0880-x

CrossRef Full Text | Google Scholar

Park, S., and Ryu, J. (2018). Exploring preservice teachers' emotional experiences in an immersive virtual teaching simulation through facial expression recognition. Int. J-Hum-Comput. Interact. 35, 1–13. doi: 10.1080/10447318.2018.1469710

CrossRef Full Text | Google Scholar

Su, X., Mahadevan, S., Xu, P., and Deng, Y. (2015). Dependence assessment in human reliability analysis using evidence theory and AHP. Risk Anal. 35, 1296–1316. doi: 10.1111/risa.12347

PubMed Abstract | CrossRef Full Text | Google Scholar

Tsiourti, C., Weiss, A., and Wac, K. (2019). Multimodal integration of emotional signals from voice, body, and context: effects of (In) congruence on emotion recognition and attitudes towards robots. Int. J. Soc. Robot. 7, 555–573. doi: 10.1007/s12369-019-00524-z

CrossRef Full Text | Google Scholar

Unsworth, L., and Mills, K. A. (2020). English language teaching of attitude and emotion in digital multimodal composition. J. Second. Lang. Writ. 47, 1–17. doi: 10.1016/j.jslw.2020.100712

CrossRef Full Text | Google Scholar

Vicente, A. D., and Pain, H. (2002). Informing the detection of the students' motivational state: an empirical study. Lect. Notes Comput. Sci, 79–86. doi: 10.1007/3-540-47987-2_93

CrossRef Full Text | Google Scholar

Woodrow, L. (2006). Anxiety and speaking English as a second language. RELC J. 37, 308–328. doi: 10.1177/0033688206071315

CrossRef Full Text | Google Scholar

Wu, Z., and Chu, W. (2021). “Sampling strategy analysis of machine learning models for energy consumption prediction.” in 2021 IEEE 9th International Conference on Smart Energy Grid Engineering (SEGE); August 11–13, 2021. 77–81.

Google Scholar

Xiao, T., Zhang, P., Zhang, Y., Li, D., and Shen, J. (2021). A research on the application of college students' physique data mining based on logistic regression algorithm. ASP Trans. Comput. 1, 12–18. doi: 10.52810/TC.2021.100042

CrossRef Full Text | Google Scholar

Yang, D., Alsadoon, A., Prasad, P., Singh, A. K., and Elchouemi, A. (2018). An emotion recognition model based on facial recognition in virtual learning environment. Procedia Comput. Sci. 125, 2–10. doi: 10.1016/j.procs.2017.12.003

CrossRef Full Text | Google Scholar

Yu, H. (2020). Online teaching quality evaluation based on emotion recognition and improved AprioriTid algorithm. J. Intell. Fuzzy Syst. 40, 1–11. doi: 10.3233/JIFS-189534

CrossRef Full Text | Google Scholar

Zhang, L., Sun, L., Yu, L., Dong, X., Chen, J., Cai, W., et al. (2021). “ARFace: attention-aware and regularization for face recognition with reinforcement learning.” in IEEE Transactions on Biometrics, Behavior, and Identity Science; August 11, 2021.

Google Scholar

Zhang, M., and Zhang, L. (2020). Cross-cultural O2O English teaching based on AI emotion recognition and neural network algorithm. J. Intell. Fuzzy Syst. 40, 1–12. doi: 10.3233/JIFS-189546

CrossRef Full Text | Google Scholar

Zhao, M., Jha, A., Liu, Q., Millis, B. A., Mahadevan-Jansen, A., Lu, L., et al. (2021). Faster mean-shift: GPU-accelerated clustering for cosine embedding-based cell segmentation and tracking. Med. Image Anal. 71:102048. doi: 10.1016/j.media.2021.102048

PubMed Abstract | CrossRef Full Text | Google Scholar