A STEM Course Analysis During COVID-19: A Comparison Study in Performance and Affective Domain of PSTs Between F2F and F2S Flipped Classroom

Due to the worldwide COVID-19 pandemic, university education has faced a significant challenge that requires adaptation to virtual and online education. Here, a fruitful flipped methodology with increased popularity can support adaption to and improvement of the current pandemic situation. This research presents a comparison of two different instruction situations with an identical teaching methodology, face-to-face (F2F) and face-to-screen (F2S) flipped methodology, in terms of students' performance and affective domain in a science, technology, engineering and mathematics (STEM) course. It was considered and designed as an examination of 132 pre-service teachers (PSTs), with 68 and 64 PSTs respectively for each group. The first group before the pandemic was applied by F2F flipped classroom and the second group after the pandemic was applied by F2S flipped classroom. The results after pertaining various data analyses of class activities and questionaries showed that performance had been improved for both groups toward the course. In addition, F2F had a significant difference in PSTs' emotion and perception toward the course and made classes more interactive. The mean score values of students' emotion and perception between two groups showed that the difference between these mean values were significant, suggesting a very large effect. Particularly, the effect size (ES) showed that positive emotions were more significant with different variables and the items Q7–Q9 of questionnaires indicated more significant different perceptions for both F2F and F2S after completing the course. Finally, the principal component analysis (PCA) test described that F2F answers were located mainly in the positive emotion, while F2S answers were grouped in the negative emotion, while no differences were observed for PSTs perceptions to the flipped methodology. Consequently, although F2F–F2S transition was an effective process, instructors and PSTs faced difficulties in the platform usage for online lectures reflecting emotions' results in F2S group. Thus, by solving the problems raised, it will allow PSTs to be more interactive in a virtual and online context for their future implementation by giving them active instruction methodology and educating future students to teach STEM contents.

Introduction

Due to the COVID-19 pandemic, the United Nations Educational, Scientific, and Cultural Organization (UNESCO) indicated that educational institutions had been providing over 70% of their classes and assessments through various on-line and virtual platforms (Stub, 2020; UNESCO, 2020). Before the appearance of COVID-19, e-learning showed a consistent tendency of continuous growing, about 15.4% per year, in educational institutions (Wang et al., 2020). There was not any pressure or uncertainty for either institutions or students (Azeiteiro et al., 2015; Garg and Jain, 2017). However, learning environments were forced to change due to circumstances during the COVID-19 pandemic, which saw millions of cases confirmed in more than 216 countries according to the World Health Organization (WHO) (Cao et al., 2020; WHO, 2020a,b). The situation that began in the middle of the spring semester was totally unexpected and unplanned for by all people working in academic institutions, especially instructors and students. Due to precautious actions before the COVID-19 spread and lockdown, many institutions had already switched to complete e-learning teaching/learning (Cao et al., 2020; Crawford et al., 2020; WHO, 2020b). So, it resulted in a suspension of the current educational procedures and developments in many worldwide institutions (Fauci et al., 2020; WHO, 2020a). The traditional instruction method was changed to an e-learning model that allowed students to continue and finish their classes and activities (Crawford et al., 2020). Each institute adopted different and various e-learning systems due to the action of social distance regulation and directives, which the WHO strongly recommended to halt the COVID-19 spread amongst persons and countries (WHO, 2020a,b). Previous studies have indicated that instructors and students in educational areas had a close relationship and improvements in the on-line and virtual system (Moura et al., 2010; Azeiteiro et al., 2014; Islas-Pérez et al., 2015). However, this new and unfamiliar home-based education system was implemented to foster instructors and students to have a considerable responsibility of on-line and virtual skills and experiences (Bacelar-Nicolau et al., 2009; Crawford et al., 2020). Despite the enormous efforts to solve the glitches among the different teaching strategies implemented, the efficiency of on-line and virtual teaching was still falling behind expectations (Zare et al., 2016; Yang et al., 2017). Even though the instructors had undergone the training required in a brief period, there was a requirement to change the existing and paper-based materials to on-line and virtual teaching resources (Parkes et al., 2014; Cao et al., 2020). Therefore, it would be necessary to find out a proper methodology and system, which could achieve its objectives in education accomplishing the WHO social distancing suggestion during the COVID-19 pandemic.

In a teaching-learning process, especially considering current situations, on-line and virtual learning can be considered as a proper educational model (Eneroth, 2000; Paechter et al., 2010). This model, along with the information and communication technologies (ICTs), allows flexible and relevant student-focused education in science, technology, engineering, and mathematics (STEM) education (Pereira et al., 2008; Shee and Wang, 2008; Chujitarom and Piriyasurawong, 2019). ICTs can provide scaffolding among teacher to teacher, teacher to student, and student to student interactions that can help on-line and virtual systems as a virtual teaching-learning platform and multi-faceted communications (Garrison, 2000; Narciss et al., 2007; Pereira et al., 2008). The on-line and virtual system has various advantages that can be used without the consideration of time and place and can require a self-regulated learning practice (Garrison, 2000; Lee and Lee, 2008; Lozano et al., 2013). In teaching-learning forming fundamental factors for STEM education, it can be considered as inter- and multi-disciplinary development (Narciss et al., 2007; Lee and Lee, 2008; Lozano et al., 2013). According to Hansen (2001), the students in an on-line and virtual system typically had a greater sense of comprehension and experience, which led to fruitful transformative teaching-learning (Arbaugh, 2000; Schramm et al., 2001). Also, the students' achievements for learning as Paechter et al. (2010) indicated were closely connected to the on-line and virtual systems' characteristics, which were multi-directional communications for learning strategy flexibility and experience transmission, related with all significant issues of STEM education (Narciss et al., 2007; Moura et al., 2010; Lambrechts et al., 2018). On the other hand, these innovative systems proposed could be of great help for STEM education development in long-term teaching-learning (Garrison, 2000; Azeiteiro et al., 2014). Here, the technological integration to STEM education could fill a current educational niche, although there are many existing challenges, which will be integrated to transformational STEM teaching-learning (Pavlova, 2013; McVey, 2016; Nowotny et al., 2018). However, general on-line and virtual learning systems are still required to examine more specific models' efficiency in-depth, such as higher STEM education through e-learning systems (McVey, 2016; Nowotny et al., 2018). Thus, active methodologies based on on-line and virtual system are necessary to achieve their objectives in adopting STEM education during COVID-19.

The flipped classroom methodology, a form of active education methodology, recently gained a great level of attention in higher education along with the STEM courses (Roach, 2014; Blair et al., 2016; Ye et al., 2018). This methodology can provide a more suitable teaching-learning environment to reach significant and fruitful achievements together with the great availability of digital materials that comply with the WHO social distancing suggestion during the COVID-19 pandemic (Roach, 2014; Blair et al., 2016; Cao et al., 2020). In the view of students, O'Flaherty and Phillips (2015) indicated that this methodology required them to take responsibility for their learning. Sams and Bergmann (2013) mentioned that a flipped classroom course would be effective not only for a big group of students but also for individual students, unlike a traditional direct lecture. Particularly, in higher STEM education, traditional teaching methodology was better suited as an instructor-centered methodology to be delivered to students (Williams et al., 2018; Jeong and González-Gómez, 2020a). Here, the flipped classroom methodology aforementioned can be an effective and alternative approach delivering a student-centered methodology (Dooley et al., 2018; Zamora-Polo et al., 2019; Jeong and González-Gómez, 2021). In a basic flipped course setting, students can receive their lectures at home in the format of videos, tasks, quizzes, and written materials in on-line spaces such as Moodle. Reversely, students can do class activities that are conventionally done at home with instructors' supervision (Jeong and González-Gómez, 2020b). Here, more student-centered activities can be performed in-class time along with providing just-in-time lectures and collaborative tasks, which can address detailed questions, and realize more efficient chances for learning (Mattis, 2015; Moraros et al., 2015; Namaziandost and Çakmak, 2020). For this research, the performance in the flipped classroom methodology can be improved in the context of students' learning due to there being more in-class time along with active learning (Akçayir and Akçayir, 2018; Jeong and González-Gómez, 2020a). Kemp and Grieve (2014) indicated in their study that similar academic performance was achieved in both an on-line and face-to-face learning environment, but a higher preference of face-to-face settings was observed when students-centered activities were carried out. Although learning and affective domains could be considered as interdependent, the affective domains' influence on learning should be theoretically analyzed and practically investigated (Bower, 1981; Schwarz, 1990; Abele, 1995; Hascher and Edlinger, 2009). The learning process is the outcome of the cognitive and affective domain interplay (Pintrich et al., 1993). Currently, several theories show at least a certain amount of empirical evidence. Bower (1981) indicated a theory, “mood-congruence-hypothesis,” that information can be more easily remembered in a positive mood than in a negative mood. Thus, Schwarz (1990) suggested the theory “mood as information,” in which the pertainable point is the role that mood itself plays for learners. STEM education research was prevailing in the cognitive aspects of teaching/learning procedures without attention on the affective domain (Mellado et al., 2014). However, a growing interest has been seen on understanding the influence of emotions in the teaching/learning process by many studies in recent years (Dos Santos and Mortimer, 2003; Zembylas, 2007; Abrahams, 2009; Ritchie et al., 2011; Schutz and Zembylas, 2011; Bellocchi et al., 2013). Owing to support and higher interest in affective domain, students' perception can be analyzed for their opinions of flipped classroom methodology (Blair et al., 2016; Akçayir and Akçayir, 2018). Many studies have been carried out in different educational levels that have measured the students' perceptions toward the flipped classroom methodology (Bishop and Verleger, 2013; Roach, 2014; Gilboy et al., 2015; Sowa and Thorsen, 2015; Long et al., 2016). Abele (1995) demonstrated a positive mood can even increase the pace of perception along with performance and processing. Jeong et al. (2019) showed that perception and emotion had a significant relationship to students' learning in various learning environments in a STEM course, including face-to-face and face-to-screen learning settings, particularly the confirmation of more face-to-face education with different researchers (Marshall, 2011; Baker, 2012; Blair et al., 2016; Williams et al., 2018; Ye et al., 2018). The instruction methodology should also encourage students' positive perceptions and emotions toward STEM, especially in pre-service teachers (Osborne et al., 2003; Jarvis and Pell, 2004), who will be future instructors after their development and formation. Hence, based on the previous literatures and reasons confirmed, an active flipped methodology can be evaluated in different situations, face-to-face (F2F) and face-to-screen (F2S) flipped methodology in a STEM course, due to COVID-19, in terms of the students' performance and affective domains.

In this work, a comparison of two different instruction situations along with an identical teaching methodology is presented: F2F and F2S flipped methodology in terms of PSTs' performance and emotion and perception in a STEM course. A total of 132 PSTs participated in this study across two different years, 2018/19 and 2019/2020 course (68 and 64 PSTs, respectively). Students were randomly assigned to the studied group and they agreed to participated in this study. Particularly, before and after the COVID-19 pandemic, the first group was applied to the F2F flipped classroom and second group was applied to the F2S flipped classroom. With the various data analyses of class activities and questionaries, the results expose the performance variation and the significant change of emotion and perception of PSTs toward the course implemented. Also, it can show their effect size (ES) and principal component analysis (PCA) differences based on PSTs' data that allow PSTs to be more interactive and adopted in different instruction contexts.

Materials and Methods

A flipped classroom instruction methodology was applied in a STEM class during two different courses with an identical instruction methodology before and after the COVID-19 pandemic. Precisely, a F2F flipped instruction methodology was followed in the first course and F2S flipped instruction methodology in the second one. The class used to study this methodology had a course syllabus containing overall themes of science along with the didactic method and strategies to teach these contents for primary education. For each course, PSTs were randomly assigned and agreed to participate in this research.

Sample

For the course proposed, a total of 141 PSTs from two groups enrolled for this course. The PSTs were randomly assigned into individual courses, 70 and 71 PSTs, respectively. Here, the PSTs before registering for the subject did not have any knowledge of the flipped methodology for the course and choices were not based on any preconceived prejudices. The instructors imposed a constraint that there must be a similar quantity of participants for both groups. Each group had an identical instruction methodology in two different environments due to COVID-19: F2F flipped instruction methodology and F2S flipped instruction methodology, correspondingly. There were some PSTs who did not participate actively, which indicated the final response rate was 68 PSTs (97.14%) for F2F group and 64 PSTs (90.14%) for F2S group. The final participation rate was 132 PSTs (93.62%) for both groups, which was representative of the entire course. As displayed in Table 1, descriptive demographic information of PSTs showed a total participant number, gender distribution, average age, pre-test grade point average (GPA) that the average grade student had before starting the course, and educational background with social sciences, sciences, technology, arts, and others. Particularly, their average age was 20.1 and 21.0, respectively, with the total average age being 20.55 years old. In the case of gender distribution, it was a different pattern for each group. The pre-test GPA was 6.93 and 6.91 in the 0–10 scale, respectively. Finally, regarding the PSTs' educational background, both groups had a similar percentage, such as 68.3 and 65%, which showed the majority of PSTs did not have a strong STEM background from their previous education.

TABLE 1

Table 1. The comparison of demographic background information between PSTs participating in the F2F and F2S flipped classroom study.

Instructional Design

This study was conducted in a general STEM course across two different years. The STEM course is called “Teaching of Matter and Energy” and is taught to the PSTs as a mandatory subject. The course consists of 3 h per week on theoretical contents and 1 h per week for laboratorial contents. Particularly, in the theoretical classes, all students attend simultaneously, in the same classroom for the F2F or remotely from their homes for the F2S group. In experimental contents, the class is divided into three groups, which allows for the provision of better instruction for the laboratorial activities. Together with the theoretical and laboratory class hours, the PSTs can arrange an individualized tutorial period with the instructors to clarify contents. In both F2F and F2S courses, the same syllabus was followed, and the course had the same structure. As is shown in Table 2, the course structure consists of five units including overall features and views of the matter and the energy. In this methodology in two different environments, an active and participatory atmosphere was promoted for their learning.

TABLE 2

Table 2. The general information of the subject implemented into the course for F2F and F2S flipped classroom.

In both F2F and F2S courses, the flipped paradigm was presented from the beginning of the course and all PSTs had access to the virtual interface of Moodle that contains the course flowchart with all the principal dates and subject activities scheduled. In both cases, the flipped material consisted of pre-recorded video-lessons and lab demonstrations and texts that PSTs received, based on the syllabus, 1 week before working on them synchronously (F2F and F2S). So, PSTs can prepare for the class while watching the flipped materials. Here, they can retrieve all of the material for the whole course. Particularly, PSTs also had access to an online quiz based on a multiple-choice format that examined the subject contents. Then, they can give feedback to the instructors before the actual class, which can be considered as a “just-in-time” lecture if necessary. In this active class environment, PSTs can use in-class time to engage more with the class activities than just passively participating in the class. Figure 1 displays a class session structure for F2F and F2S flipped instruction methodology, which incorporates a schematic vision of two groups' learning processes.

FIGURE 1

Figure 1. A conceptual framework of a STEM course to examine performance and affective domain between F2F and F2S flipped classroom.

Comparison of Students' Emotions and Perceptions Toward F2F and F2S

The comparison study between the same instruction methodology in two different environments was realized by a questionnaire survey that PSTs completed at the end of the course. The questionnaire was formed on the basis of previously published research by Roach (2014) and then was adapted to the current work after considering the course syllabus and contents. Thus, an expert panel of professors and researchers working in this topic along with the university bioethics board validated the questionnaire before collecting the information.

The designed questionnaire consisted of three different sections. The first section (section 1 Introduction) was for gathering PSTs' demographic information such as age, gender, current GPA, and educational background (see Table 1). The second one (section Materials and Methods) was for collecting the data of PSTs' emotions when F2F and F2S flipped instructional methodology were followed. Here, emotions were divided into two groups, positive and negative emotions (Bisquerra, 2005). Fun, confidence, enthusiasm, and tranquility were used for the positive emotions and nervousness, concern, boredom, and fear were used for the negative emotions (Dunbar et al., 2016). PSTs answered their opinions based on the 0–10 scale about their emotions assessed toward the course, with the lowest incidence indicating 0 and the highest incidence indicating 10.

As shown in Table 3, the third section (section Results and Discussion) was dedicated to collecting the data of PSTs' perceptions when F2F and F2S flipped instructional methodology was followed. Here, the questionnaire consisted of nine questions that were closed type and could be defined and arranged in two groups. The first group consisted of six questions Q1 to Q6, which inquired about flipped video and other activity materials' suitability and how these flipped materials were valuable to achieving the learning and proficiencies of course goals. The second group consisted of three questions Q7 to Q9, which inquired about the entire flipped classroom and the valuable studying experience of the course (Roach, 2014; González-Gómez et al., 2019). Also, PSTs answered their opinions based on 1–5 as a five-point Likert-type scale about their perceptions assessed toward the course, which ranged from strongly disagreed (SD), disagreed (D), neutral (N), agreed (A), and strongly agreed (SA).

TABLE 3

Table 3. Five-points Likert-type survey used in this study to compare the F2F and F2S in terms of PST's perception change (Section Results and discussion).

Statistical Analysis

The gathered data throughout the instruments implemented were analyzed in a quantitative manner. Firstly, a descriptive analysis was used to represent the sample data conclusions as the most suitable method to describe, characterize, and draw (Etxeberria and Tejedor, 2005; Jeong et al., 2019). Then, a Cronbach alpha test was used to check the reliability of questionnaires (Pintrich and De Groot, 1990; Biggs et al., 2001; Ahlfeldt et al., 2005). Table 4 shows the value of Cronbach Alpha test for the two divisions, emotion and perception, and indicates emotion's question validity was 0.89 and the perceptions' question validity was 0.93. Consequently, for both questionnaire sections, the Cronbach alpha test can be determined as acceptable as it is close to reliable when making an important decision (Biggs et al., 2001).

TABLE 4

Table 4. Cronbach Alpha test in this study for the questionnaire Sections Materials and Methods and Results and Discussion.

The Kolmogorov–Smirnov test for normality was used to check whether data collected were normally distributed or not. Here, the data gathered were normally distributed, so we conducted a parametric statistical test. To find a significant difference and relationship between data of F2F and F2S, t-test as a parametric statistical analysis was performed at 95% confidence level. Both emotion and perception data were examined by score mean values that were compared and showed the significant differences' presence by means of t test at 95% confidence level. Then, the effect size (ES) estimation was executed in accordance with the Rosenthal method (Rosenthal, 1991). According to Cohen (1988), the ES was applied to gauge the treatment effect extent. Finally, principal component analysis (PCA) was used to deduce whether all data gathered had an objective to conduct. As a useful tool, the PCA can summarize large quantities of data. Also, it can conclude how samples collected are different from each other (F2F and F2S data), how variables can serve more significantly to the variance, and how variables can correlate with each other (Peres-Neto et al., 2005; González-Gómez et al., 2019). Finally, SPSS statistics 22.0 software was used to find out all information.

Results and Discussions

Through the different environments with an identical flipped methodology, the results obtained showed various examinations together with performance, emotion, and perception comparison. Particularly and firstly, sample homogeneity and performance comparison were checked to complete the comparative manner of this work. Then, the comparison of emotion and perception analysis was accomplished to figure out a keener vision of PSTs' affective domain. Consequently, the results showed the principal patterns and outlines for directing performance and affective domain analysis of a STEM course during COVID-19 with a comparison study between F2F and F2S flipped classroom.

Sample Homogeneity and Performance Comparison

Table 1 describes the interesting aspect of a sample that nearly three-fourths of PSTs during the mid- and high-school stage did not take science subjects. Particularly, 20.1% of the F2F group and 15.1% of the F2S group did take science subjects during mid- and high-school stage. So, a high percentage of PSTs already lacked an understanding of the fundamental science concepts that would create many difficulties in understanding the subject. In order to finish the comparative study, the sample homogeneity if normally distributed or not was proven with reference to F2F's and F2S's emotions and perceptions. Here, the significant differences were detected between them.

In accordance with the university's statistical data stipulated during the previous 10 years, PSTs with a science background or not always had complications to finish the subject compared with the entire degree program. In reality, many PSTs took 2.5 years on average to finish this subject. Furthermore, there was an even smaller number of PSTs who took more than 4 years to finish this subject satisfactorily. Here, the performance results to gauge the proposed methodology success were compared between F2F and F2S groups. Table 5 summarized the results gotten for F2F and F2S teaching methodology. The final grade for each group shows 7.54 (F2F) and 7.23 (F2S), respectively. For flipped instruction methodology of F2F and F2S, even though the final pass rate of PSTs increased enough (67.1 and 70.3%), around 30% of students still needed to take the course again. Finally, the information about the pass rate in the two attempts that the university provided is also summarized in Table 5. Although for both groups, the pass rate were similar, higher scores were observed for the F2S groups, although no significant differences were established.

TABLE 5

Table 5. Performance comparison between F2F and F2S.

Course Emotions Comparison

Figure 2 summarized the PSTs' comparison of emotions toward the flipped instruction methodology following F2F and F2S after finalizing the course. All scores of positive and negative emotions based on the statistical comparison analysis were significantly different in both F2F and F2S instructional settings. The mean score values for positive emotions for the F2F group was 27.6 (std dev = 4.24), whereas the mean score value for the F2S group was 14.7 (std dev = 8.71). The t-test showed that the difference between these mean values were significant (p<0.001, d = 1.91), suggesting a very large effect. On the other hand, the score values for the negative emotions for the F2F group was 15.7 (std dev = 9.17), while the mean score value for the F2S group was 29.0 (std dev = 7.19). Again, the t-test showed that the difference between the mean values of both groups were significant (p<0.001, d = 1.44), which corresponded with a very large effect.

FIGURE 2

Figure 2. Emotion analysis for F2F and F2S flipped classroom after the course completion.

To assess the difference among emotions, each one of the assessed emotions were analyzed. The main results are summarized in Figure 3. Thus, with respect to the positive emotions, the F2F group showed high scores that had an average 3.1 points difference of the points specified by the F2S PSTs. Particularly, the positive tranquility emotion had a notable score, which had more difference among the positive emotions (6.53 in F2F and 2.92 in F2S course). On the contrary, F2S had a high score of negative emotions, with the negative concern emotion indicating more difference among the negative emotions (4.90 in F2F and 8.53 in F2S course). Also, the negative fear emotion along with negative concern emotion pointed out a big difference between F2F and F2S groups (3.84 and 7.05, respectively). The negative boredom emotion in both groups showed the lowest value, which was 3–4 points less than other negative emotions recognized.

FIGURE 3

Figure 3. Single item assessment of the emotions for F2F and F2S flipped classroom after the course completion.

Table 6 described the PSTs' comparison of emotions toward the flipped instruction methodology following F2F and F2S after finalizing the course. There are some scores of positive and negative emotions based on the statistical comparison analysis that were significantly different in both F2F and F2S instructional settings. Thus, the ES analysis specified that the instruction methodology had a medium to large effect in the whole emotion measured as indicated in the table (Cohen, 1988). Particularly, positive emotions' ES was all very large effects of significant different variables and negative emotions' ES was situated from medium to very large effect of significant different variables. Here, we have used the Cohen's d to find out the significant difference for each variable between two means divided by a standard deviation for the data as shown in Equation (1). The value of d's magnitude indicates the ES size: very small is between 0 and 0.01, small is between 0.02 and 0.20, medium is between 0.21 and 0.50, large is between 0.51 and 0.80, very large is between 0.81 and 1.20, and huge is between 1.21 and 2.00.

$\begin{array}{ll}s=\sqrt{\frac{(n_1-1)s_1^2+(n_2-1)s_2^2}{n_1+n_2-2}}& (1)\end{array}$

TABLE 6

Table 6. Emotion comparison between F2F and F2S flipped instruction methodology.

Course Perceptions Comparison

Table 7 represented the PSTs' comparison of perceptions toward the flipped instruction methodology following F2F and F2S after finalizing the course. The main values of each questionnaire can be found in this table. Moreover, to catch a closer observation of PSTs' perceptions for both F2F and F2S instruction methodologies, Figure 4 summarizes the responses collected for the perception questionnaire part. The mean score values students' perception for F2F group was 40.0 (std dev = 5.26), whereas the mean score value for F2S group was 31.9 (std dev = 3.79). The t-test showed that the difference between these mean values were significant (p<0.001, d = 1.76), suggesting a very large effect.

TABLE 7

Table 7. Perception comparison between F2F and F2S flipped instruction methodology of the Likert-type test.

FIGURE 4

Figure 4. Perception analysis of F2F and F2S flipped classroom after the course completion.

Again, to have a detailed view of the perceptions, each item was also analyzed. According to the statistical comparison, Table 7 showed that some perception scores are significantly different in both F2F and F2S instructional environments. However, Q3 and Q4 questionnaires' scores indicated that there were significant differences in statistical assessment by both F2F and F2S groups. The rest of the questions for perception scores provided were significantly different in both F2F and F2S instructional environments. In addition, the ES analysis indicated that the instruction methodology had a small (no ES) to huge ES in the perception assessed as indicated in the table (Cohen, 1988). Particularly, Q1–Q6 perception items were located in between small (no ES) to large variables and the Q7–Q9 perception items were located in between very large to huge variables. Here, we used the same manner of Cohen's d to find out their significant difference for each variable between two means divided by a standard deviation for the data aforementioned.

Figure 5 showed the PSTs' perception part of the questionnaire with a closer view for both F2F and F2S instruction methodologies provided. According to the statistical comparison analysis, all perception questionnaires provided were significantly different in both instructional environments. Particularly, Q1, Q2, Q5, and Q6 items had around a 0.5 increase between F2F and F2S while Q3 and Q4 items had 0.14 and 0.19 difference, respectively, in the Group 1 questionnaires. In the Group 2 questionnaires, Q7–Q9 had more than 1 point and even close to 2 points difference between F2F and F2S groups after the course completion. Specifically, the Q8 item described PSTs' perceptions about the course learning experience as a whole. In this item, the average score of F2F was 4.47 points while the F2S showed 2.56 points. So, the F2F offered a higher positive perception about the learning procedure for the identical contents and could be considered as a significant contribution in PSTs' learning involvement and practice. Thus, both F2F and F2S groups agreed or strongly agreed the practicality of video lectures and other flipped materials before class improved learning target attainment and allowed PSTs to achieve the in-class works more confidently and easily. Particularly, those PSTs enrolled in the F2F course agreed or strongly agreed to have more flipped and presential course as the teaching methodology as a whole, which assisted in realizing their learning purposes.

FIGURE 5

Figure 5. Single item assessment of the perceptions of F2F and F2S flipped classroom after the course completion.

PCA Test

Figure 6 showed the PCA analysis in order to get a clear image of the instruction methodology's effect over the PSTs' emotion and perception toward STEM between F2F and F2S flipped instruction methodology. Here, it indicated a PCA loadings diagram about emotion and perception, in which the X and Y axes showed principal component 1 and principal component 2, that explained 52.3 and 23.1% of the total variance, respectively. According to the PCA results, the first PC was able to group the sample in to two groups (F2F and F2S answers). Precisely, prediction ellipses were also added to the PCA scores plot to show the probability that a new observation from the same group (F2F or F2S) will fall inside the ellipse with a 95% probability. Thus, F2F answers were located mainly in the positive axis of PC1, while F2S answers were grouped in the negative part of PC1. When comparing the score and loading plots, it is clear that PC1 represented the effect of the instruction methodology in the emotion toward science of PSTs, and it was able to distinguish between F2F and F2S groups. F2F scores were in the positive axis of PC1 which corresponds with positive emotion, while the F2S scores were in the negative axis of PC1 that corresponds with the negative emotions. Regarding the loadings corresponding to the students' perception, they are also located in the positive axis of PC1, and therefore are more correlated with the F2F scores.

FIGURE 6

Figure 6. The PCA diagram: (A) loading corresponding to the questionnaires of emotion and perception; (B) scores plots corresponding to the PSTs distribution of F2F and F2S flipped classroom.

Discussion

The obtained outcomes show the information that can be considered as a novel approach to examine PSTs performance and affective domains of F2F and F2S flipped instruction methodology during the COVID-19 pandemic. This research stipulates an exclusive comparison for a specific flipped STEM education of PSTs and can fill a niche/gap of different environments with an identical methodology to measure performance and affective domains.

Although on-line and virtual learning showed a consistent growing tendency, many institutes were not prepared for a significant transition due to the COVID-19 pandemic (UNESCO, 2020; WHO, 2020a,b). Many researchers confirmed that on-line and virtual learning along with the ICTs could be considered as a proper educational model in a teaching-learning process of STEM course (Lee and Lee, 2008; Pereira et al., 2008; Shee and Wang, 2008). Here, among teacher to teacher, teacher to student, and student to student interactions, ICTs can act as a scaffolding for communications (Garrison, 2000; Lee and Lee, 2008; Lozano et al., 2013). Thus, the technological integration to STEM education could fill a current educational niche, although there are existing many challenges, which will be integrated to transformational STEM teaching-learning (Eneroth, 2000; Paechter et al., 2010; Nowotny et al., 2018). On the other hand, these innovative systems proposed could be of great help for STEM education development in long-term teaching-learning (Garrison, 2000; Azeiteiro et al., 2014). However, general on-line and virtual learning systems are still required to examine more specific models' efficiency in-depth such as higher STEM education through e-learning systems (McVey, 2016; Nowotny et al., 2018). Therefore, a proper methodology and system proposed could achieve its objectives adapting to STEM education while complying with COVID-19 guidelines.

Together with the confirmation aforementioned, the flipped classroom methodology along with digital materials in STEM can provide a more suitable teaching-learning environment to reach significant and fruitful achievements, while complying with the WHO social distancing suggestions during the COVID-19 pandemic (Blair et al., 2016; Cao et al., 2020; WHO, 2020a). Previous studies also found that video-lectures supported shaping conceptual understanding and was valuable the convenience of students (Imran, 2013; Roach, 2014; Long et al., 2016). The performance can be improved in the context of students' learning due to having more in-class time along with active learning integration and consequence (Akçayir and Akçayir, 2018). Pintrich and De Groot (1990) indicated that positive emotions were vital for promoting significant learning in the STEM course along with the theories of emotions and learnings analyzed and investigated by previous researchers (Bower, 1981; Schwarz, 1990; Abele, 1995; Hascher and Edlinger, 2009). Here, students' perception can be analyzed for their opinions of flipped classroom methodology (Blair et al., 2016; Akçayir and Akçayir, 2018). Kemp and Grieve (2014) also concluded that, although same academic performance was achieved in an F2F and F2S environment, students do prefer to accomplish specific activities in F2F settings. Thus, in the context of affective domain, Marshall (2011) indicated a significant relationship of students' learning favoring to F2F along with previous research in F2F and F2S setting (Baker, 2012; Blair et al., 2016; Jeong et al., 2019). Hence, based on the previous literature and reasons confirmed, a proper methodology and system proposed could achieve its objectives with the comparison of face-to-face (F2F) and face-to-screen (F2S) flipped methodology. Thus, it can overcome the COVID-19 situation in terms of the students' performance and affective domains.

Consequently, although the transition from the F2F to F2S classes could be considered as a positive process in institutes, the instructors and students confronted struggles and difficulties in the platform use for on-line and virtual classes. Along with these problems, the performance and affective domain results obtained by the PSTs specified the direction to follow; there were crucial considerations that future teachers were required to reflect on their on-line and virtual classes along with improvements from these instructions to better equip themselves for future classes. Particular attention was required in students' emotions during on-line and virtual class when instructors engaged. Also, specific comparison results obtained with the methodologies and objectives in the PSTs performance and affective domain of flipped STEM education could be reapplied to various educational areas and contexts when there were available data required due to its flexible characteristics.

Conclusions

The research shows an examination of two different situations' comparison of an identical flipped instruction methodology, F2F and F2S, in terms of students' performance, emotion, and perception in a STEM course before and after the COVID-19 pandemic. It was designed and considered as a randomization examination with 132 PSTs, 68 and 64 PSTs, for the primary education bachelor's degree in Spain. Here, various statistical analyses were applied to data and questionnaires proposed.

According to the results obtained in this study, both groups of PSTs' increased their grade without significant difference after the course completion. The F2F had a significant effect on PSTs' perception and emotion toward the course and created classes that were more interactive. Particularly, in the comparison of students' emotions toward the instruction methodology, the ES analysis indicated that positive emotions' ES was all very large for significant different variables and negative emotions' ES was situated from medium to very large for significant different variables for both F2F and F2S after completing the course. Then, in the comparison of students' perceptions toward the instruction methodology, section 3 of the questionnaire (items Q7 to Q9) indicated a significant difference as ES showed a very large to huge index after the course was completed for both groups of F2F and F2S. Finally, the PCA test described that F2F answers were located mainly in the positive emotion part of PC1, while F2S answers were grouped in the negative axis of PC1. The scores plots indicated that positive emotions and perceptions were located in the positive axis of PC1 while negative emotions were grouped in the negative axis of PC1.

Consequently, although the conversion from F2F–F2S in the higher education was a fruitful procedure, students and instructors confronted difficulties in the use of online classes through a platform. In this research, the comparison demonstrated how to reflect the challenges and drawbacks meaningfully by the results obtained, especially emotions in the F2S group. Here, they emphasized emotions as the crucial criteria that instructors needed to consider when teaching virtual and online classes and taking advantage of these findings to better equip themselves for future classes. Although the F2F flipped classroom has enough virtual and online content, for the F2S classroom, more appropriate adoption and transition is required to promote both performance and affective domains of PSTs. Thus, it will allow PSTs to be more interactive in virtual and online context for their future implementation with active instruction methodology to educate future students to teach STEM contents. Finally, the main limitations of this study could be found in the sample size and the lack of analysis of other variables that might influence the results, such as participants' gender.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by La Comisión de Bioética y Bioseguridad de la Universidad de Extremadura, Spain. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

Funding

This work was funded through the Research Projects IB18004 and GR18004 by Consejería de Economía, Ciencia y Agenda Digital de la Junta de Extremadura (Spain) and FEDER funds.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors gratefully acknowledge the Consejerería de Economía e Infraestructura de la Junta de Extremadura y FEDER funds (Projects IB18004), whose support made this research possible.

References

Abele, A. (1995). Stimmung und Leistung [Mood and achievement]. Göttingen: Hogrefe.

Abrahams, I. (2009). Does practical work really motivate? a study of the affective value of practical work in secondary school science. Int. J. Sci. Educ. 31, 2335–2353. doi: 10.1080/09500690802342836

CrossRef Full Text | Google Scholar

Ahlfeldt, S., Mehta, S., and Sellnow, T. (2005). Measurement and analysis of student engagement in university classes where varying levels of PBL methods of instruction are in use. Higher Educ. Res. Develop. 24, 5–20. doi: 10.1080/0729436052000318541

CrossRef Full Text | Google Scholar

Akçayir, G., and Akçayir, M. (2018). The flipped classroom: a review of its advantages and challenges. Comput. Educ. 126, 334–345. doi: 10.1016/j.compedu.2018.07.021

CrossRef Full Text | Google Scholar

Arbaugh, J. B. (2000). Virtual classroom versus physical classroom: an exploratory study of class discussion patterns and student learning in an asynchronous internet-based MBA course. J. Manage. Educ. 24, 213–233. doi: 10.1177/105256290002400206

CrossRef Full Text | Google Scholar

Azeiteiro, U. M., Bacelar-Nicolau, P., Caetano, F. J., and Caeiro, S. (2015). Education for sustainable development through e-learning in higher education: experiences from Portugal. J. Clean. Prod. 106, 308–319. doi: 10.1016/j.jclepro.2014.11.056

CrossRef Full Text | Google Scholar

Azeiteiro, U. M., Walter Leal, F., and Sandra, S. C. (2014). E-learning and Education for Sustainability. Bern: Interactive Factory.

Google Scholar

Bacelar-Nicolau, P., Caeiro, S., Martinho, A., Azeiteiro, U. M., and Amador, F. (2009). E-Learning for the environment. The Universidade Aberta (Portuguese open distance university) experience in the environmental sciences Post-Graduate courses. Int. J. Sustain. Higher Educ. 10, 354–367. doi: 10.1108/14676370910990701

CrossRef Full Text | Google Scholar

Baker, J. D. (2012). “Affective and cognitive learning in the online classroom.” In: Encyclopedia of the Sciences of Learning, ed N. M. Seel (New York, NY: Springer). Available online at: https://doi.org/10.1007/978-1-4419-1428-6_762

Google Scholar

Bellocchi, A., Ritchie, S. M., Tobin, K., Sandhu, M., and Sandhu, S. (2013). Exploring emotional climate in preservice science teacher education. Cult. Stud. Sci. Educ. 8, 529–552. doi: 10.1007/s11422-013-9526-3

CrossRef Full Text | Google Scholar

Biggs, J. B., Kember, D., and dan Leung, D. Y. P. (2001). The revised two-factor study process questionnaire: R-SPQ-2F. Br. J. Educ. Psychol. 71, 133–149. doi: 10.1348/000709901158433

PubMed Abstract | CrossRef Full Text | Google Scholar

Bishop, J. L., and Verleger, M. (2013). “The flipped classroom: a survey of the research,” in Paper presented in the 120th ASEE Annual Conference and Exposition, Atlanta, GA.

PubMed Abstract | Google Scholar

Bisquerra, R. (2005). Emotional Education and Well-Being. Barcelona: Praxis.

Blair, E., Maharaj, C., and Primus, S. (2016). Performance and perception in the flipped classroom. Educ. Inform. Tech. 21, 1465–1482. doi: 10.1007/s10639-015-9393-5

CrossRef Full Text | Google Scholar

Bower, G. H. (1981). Mood and memory. Am. Psychol. 36, 129–148. doi: 10.1037/0003-066X.36.2.129

CrossRef Full Text | Google Scholar

Cao, W., Fang, Z., Hou, G., Han, M., Xu, X., Dong, J., et al. (2020). The psychological impact of the COVID-19 epidemic on college students in China. Psychiatry Res. 287:112934. doi: 10.1016/j.psychres.2020.112934

PubMed Abstract | CrossRef Full Text | Google Scholar

Chujitarom, W., and Piriyasurawong, P. (2019). The effect of the STEAM-GAAR field learning model to enhance Grit. TEM J. 8, 255–263. doi: 10.5539/ies.v11n11p23

PubMed Abstract | CrossRef Full Text | Google Scholar

Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Erlbaum, Hillsdale, NJ.

Google Scholar

Crawford, J., Butler-Henderson, K., Rudolph, J., and Glowatz, M. (2020). COVID-19: 20 Countries' higher education intra-period digital pedagogy responses. J. Appl. Teach. Learn. 3, 9–28. doi: 10.37074/jalt.2020.3.1.7

CrossRef Full Text | Google Scholar

Dooley, L. M., Frankland, S., Boller, E., and Tudor, E. (2018). Implementing the flipped classroom in a veterinary pre-clinical science course: student engagement, performance and satisfaction. J. Vet. Med. Educ. 45, 195–203. doi: 10.3138/jvme.1116-173r

CrossRef Full Text | Google Scholar

Dos Santos, F. M. T., and Mortimer, E. F. (2003). How emotions shape the relationship between a chemistry teacher and her high school students. Int. J. Sci. Educ. 25, 1095–1110. doi: 10.1080/0950069032000052216

PubMed Abstract | CrossRef Full Text | Google Scholar

Dunbar, R. L., Dingel, M. J., Dame, L. F., Winchip, J., and Petzold, A. L. (2016). Student social self-efficacy, leadership status, and academic performance in collaborative learning environments. Stud. Higher Educ. 38, 1507–1523. doi: 10.1080/03075079.2016.1265496

CrossRef Full Text | Google Scholar

Eneroth, C. (2000). E-learning for Environment. Improving e-learning as a Tool for cleaner production education (Licentiate Dissertation). Lund University, Lund, Sweden.

Google Scholar

Etxeberria, J., and Tejedor, F. J. (2005). Descriptive data analysis in education. La Muralla Madrid 11, 1–12. Available online at: https://ojs.uv.es/index.php/RELIEVE/article/view/4275/3897

Fauci, A. S., Lane, H. C., and Redfield, R. R. (2020). COVID-19—Navigating the uncharted. N. E. J. Med. 382, 1268–1269. doi: 10.1056/NEJMe2002387

CrossRef Full Text | Google Scholar

Garg, R., and Jain, D. (2017). Prioritizing e-learning websites evaluation and selection criteria using fuzzy set theory. Manage. Sci. Lett. 7, 177–184. doi: 10.5267/j.msl.2017.1.002

CrossRef Full Text | Google Scholar

Garrison, R. (2000). Theoretical challenges for distance education in the 21st century: a shift from structural to transactional issues. Int. Rev. Res. Open Distan. Learn. 1, 1–17. doi: 10.19173/irrodl.v1i1.2

CrossRef Full Text | Google Scholar

Gilboy, M. B., Heinerichs, S., and Pazzaglia, G. (2015). Enhancing student engagement using the flipped classroom. J. Nutr. Educ. Behav. 47, 109–114. doi: 10.1016/j.jneb.2014.08.008

CrossRef Full Text | Google Scholar

González-Gómez, D., Jeong, J. S., and Cañada-Cañada, F. (2019). Enhancing science self-efficacy and attitudes of Pre-Service Teachers (PST) through a flipped classroom learning environment. Interact. Learn. Environ. 1–12, doi: 10.1080/10494820.2019.1696843

PubMed Abstract | CrossRef Full Text | Google Scholar

Hansen, B. (2001). Distance learning. CQ Researcher 993–1016.

Google Scholar

Hascher, T., and Edlinger, H. (2009). Positive emotions and well-being in school—an overview of methods and results. Psychologie in Erziehung und Unterricht 56, 105–122.

Imran, M. (2013). Increasing the interaction time in a lecture by integrating flipped classroom and just-in-time teaching concepts. J. Learn. Teach. 7, 1–13. doi: 10.21100/compass.v4i7.84

CrossRef Full Text | Google Scholar

Islas-Pérez, E., Pérez, Y. H., Pérez-Ramírez, M., García-Hernández, C. F., and Pérez, B. Z. (2015). Multicriteria decision making for evaluation of e-learning tools. Res. Comput. Sci. 106, 27–37. doi: 10.13053/rcs-106-1-3

CrossRef Full Text | Google Scholar

Jarvis, T., and Pell, A. (2004). Primary teachers' changing attitudes and cognition during a tow-year science inservice program and their effect on pupils. Int. J. Science Education 26, 1787–1811. doi: 10.1080/0950069042000243763

CrossRef Full Text | Google Scholar

Jeong, J. S., and González-Gómez, D. (2020a). Adapting to PSTs' pedagogical changes in sustainable mathematics eduation through flipped e-learning: Ranking its criteria with MCDA/F-DEMATEL. Mathematics 8(5):858. doi: 10.3390/MATH8050858

CrossRef Full Text | Google Scholar

Jeong, J. S., and González-Gómez, D. (2020b). Assessment of sustainability science education criteria in online-learning through fuzzy-operational and multi-decision analysis and professional survey. Heliyon 6(8):e04706. doi: 10.1016/j.heliyon.2020.e04706

CrossRef Full Text | Google Scholar

Jeong, J. S., and González-Gómez, D. (2021). Flipped-OCN method in mathematics learning to analyze the attitudes of pre-service teachers. Mathematics 9:6077. doi: 10.3390/math9060607

PubMed Abstract | CrossRef Full Text | Google Scholar

Jeong, J. S., González-Gómez, D., and Cañada-Cañada, F. (2019). How does a flipped classroom course affect the affective domain toward science course? Interactive. Learn. Environ. 1–13, doi: 10.1080/10494820.2019.1636079

CrossRef Full Text | Google Scholar

Kemp, N., and Grieve, R. (2014). Face-to-face or face-to-screen? Undergraduates' opinions and test performance in classroom vs. online learning. Front. Psycholog. 12, 1–11. doi: 10.3389/fpsyg.2014.01278

CrossRef Full Text | Google Scholar

Lambrechts, W., Ghijsen, P. W., Jacques, A., Walravens, H., Liedekerke, L. V., and Petegem, p.V. (2018). Sustainability segmentation of business students: toward self-regulated development of critical and interpretational competences in a post-truth era. J. Clean. Produc. 202, 561–570. doi: 10.1016/j.jclepro.2018.07.303

CrossRef Full Text | Google Scholar

Lee, J., and Lee, W. (2008). The relationship of e-learner's self-regulatory efficacy and perception of e-learning environmental quality. Comput. Hum. Behav. 24, 32–47. doi: 10.1016/j.chb.2006.12.001

CrossRef Full Text | Google Scholar

Long, T., Logan, J., and Waugh, M. (2016). Students' perceptions of the value of using videos as a pre-class learning experience in the flipped classroom. TechTrends 60, 245–252. doi: 10.1007/s11528-016-0045-4

CrossRef Full Text | Google Scholar

Lozano, R., Lozano, F., Mulder, K., Huisingh, D., and Waas, T. (2013). Advancing higher education for sustainable development: international insights and critical reflections. J. Clean. Product. 48, 3–9. doi: 10.1016/j.jclepro.2013.03.034

CrossRef Full Text | Google Scholar

Marshall, S. (2011). More Face to Face, Less Face to Screen. The Chronicle of Higher Education. Available online at: https://www.chronicle.com/article/More-Face-to-Face-Less/126163 (accessed March 31, 2021).

Mattis, K. V. (2015). Flipped classroom versus traditional textbook instruction: assessing accuracy and mental effort at dfferent levels of mathematical complexity. Tech. Knowl. Learn. 20, 231–248. doi: 10.1007/s10758-014-9238-0

CrossRef Full Text | Google Scholar

McVey, M. (2016). E-learning and education for sustainability. Int.l Rev. Educ. 62, 117–121. doi: 10.1007/s11159-016-9533-1

CrossRef Full Text | Google Scholar

Mellado, V., Borrachero, A. B., Brígido, M., Melo, L. V., Dávila, M. A., Cañada, F., et al. (2014). Emotions in the teaching of science. Enseñanza de las Ciencias 32, 11–36. doi: 10.5565/rev/ensciencias.1478

CrossRef Full Text | Google Scholar

Moraros, J., Islam, A., Yu, S., Banow, R., and Schindelka, B. (2015). Flipping for success: evaluating the effectiveness of a novel teaching approach in a graduate level setting. BMC Med. Educ. 15, 1–10. doi: 10.1186/s12909-015-0317-2

CrossRef Full Text | Google Scholar

Moura, A. P. M., Cunha, L. M., Azeiteiro, U. M., Aires, L., and de Almeida, M. D. V. (2010). Food consumer science post-graduate courses: comparison of face-to-face versus online delivery systems. Br. Food J. 112, 544–556. doi: 10.1108/00070701011043781

CrossRef Full Text | Google Scholar

Namaziandost, E., and Çakmak, F. (2020). An account of EFL learners' self-efficacy and gender in the Flipped Classroom Model. Educ. Inform. Tech. 25, 4041–4055. doi: 10.1007/s10639-020-10167-7

CrossRef Full Text | Google Scholar

Narciss, S., Proske, A., and Körndle, H. (2007). Promoting self-regulated learning in web-based learning environments. Comput. Hum. Behav. 23, 1126–1144. doi: 10.1016/j.chb.2006.10.006

CrossRef Full Text | Google Scholar

Nowotny, J., Dodson, J., Fiechter, S., Gür, T. M., Kenndy, B., Macyk, W., et al. (2018). Towards global sustainability: education on environmentally clean energy technologies. Renew. Sustain. Energy Rev. 81, 2541–2551. doi: 10.1016/j.rser.2017.06.060

CrossRef Full Text | Google Scholar

O'Flaherty, J., and Phillips, C. (2015). The use of flipped classrooms in higher education: a scoping review. Inter. High. Educ. 25, 85–95. doi: 10.1016/j.iheduc.2015.02.002

CrossRef Full Text | Google Scholar

Osborne, J., Simon, S., and Collins, S. (2003). Attitudes towards science: A review of the literature and its implications. Instrum. J. Sci. Educ. 25, 1049–1079. doi: 10.1080/0950069032000032199

CrossRef Full Text | Google Scholar

Paechter, M., Maier, B., and Macher, D. (2010). Students' expectations of and experiences in e-learning: their relation to learning achievements and course satisfaction. Comput. Educ. 54, 222–229. doi: 10.1016/j.compedu.2009.08.005

CrossRef Full Text | Google Scholar

Parkes, M., Stein, S., and Reading, C. (2014). Student preparedness for university e-learning environments. Intern. High. Educ. 25, 1–10. doi: 10.1016/j.iheduc.2014.10.002

CrossRef Full Text | Google Scholar

Pavlova, M. (2013). Teaching and learning for sustainable development: ESD research in technology education. Intern. J. Tech. Design Educ. 23, 733–748. doi: 10.1007/s10798-012-9213-9

CrossRef Full Text | Google Scholar

Pereira, A., Mendes, A. Q., Morgado, L., Amante, L., and Bidarra, J. (2008). Universidade Aberta's Pedagogical Model for Distance Education (Universidade Aberta's pedagogical model for distance education). Lisbon: Universidade Aberta.

Peres-Neto, P. R., Jackson, D. A., and Somers, K. M. (2005). How many principal components? stopping rules for determining the number of non-trivial axes revisited. Comput. Stat. Data Anal. 49, 974–997. doi: 10.1016/j.csda.2004.06.015

CrossRef Full Text | Google Scholar

Pintrich, P. R., and De Groot, V. (1990). Motivational and self-regulated learning components of classroom academic performance. J. Educ. Psychol. 82, 33–40. doi: 10.1037/0022-0663.82.1.33

CrossRef Full Text | Google Scholar

Pintrich, P. R., Marx, R. W., and Boyle, R. A. (1993). Beyond cold conceptual change: the role of motivational beliefs and classroom contextual factors in the process of conceptual change. Rev. Educ. Res. 63, 167–199. doi: 10.3102/00346543063002167

CrossRef Full Text | Google Scholar

Ritchie, S. M., Tobin, K., Hudson, P., Roth, W. M., and Mergard, V. (2011). Reproducing successful rituals in bad times: exploring emotional interactions of a new science teacher. Sci. Educ. 95, 745–765. doi: 10.1002/sce.20440

CrossRef Full Text | Google Scholar

Roach, T. (2014). Student perceptions toward flipped learning: new methods to increase interaction and active learning in economics. Int. Rev. Econ. Educ. 17, 74–84. doi: 10.1016/j.iree.2014.08.003

CrossRef Full Text | Google Scholar

Rosenthal, R. (1991). Meta-analytic procedures for social research. Newbury Park, CA: Sage.

Sams, A., and Bergmann, J. (2013). Flip your students' learning. Technol. Rich Learn. 70, 16–20. Available online at: https://eric.ed.gov/?id=EJ1015329

Google Scholar

Schramm, R. M., Wagner, R. J., and Werner, J. M. (2001). Student perceptions of the effectiveness of web-based courses. NABTE Revista 27, 57–62. Available online at: https://eric.ed.gov/?id=EJ620166

Google Scholar

Schutz, P. A., and Zembylas, M. (2011). Advances in Teacher Emotion Research. Dordrecht: Springer.

Schwarz, N. (1990). “Feeling as information. informational and motivational functions of affective states,” in Handbook of Motivation and Cognition. Foundations of Social Behavior, eds E.T. Higgins and R. Sorrentino (New York, NY: Guilford Press).

Google Scholar

Shee, D. Y., and Wang, Y. S. (2008). Multi-criteria evaluation of the web-based e-learning system: a methodology based on learner satisfaction and its applications. Comput. Educ. 50, 894–905. doi: 10.1016/j.compedu.2006.09.005

CrossRef Full Text | Google Scholar

Sowa, L., and Thorsen, D. (2015). “An assessment of student learning, perceptions and social capital develpment in undergraduate, lower-division STEM courses employing a flipped classroom pedaggoy,” in Paper Presented at the 122nd ASEE Annual Conferencie and Exposition, Seattle, WA.

Google Scholar

Stub, S. T. (2020). Countries Face an Online Education Learning Curve. Available online at: https://www.usnews.com/news/bestcountries/articles/2020-04-02/coronavirus-pandemic-tests-countries-abilities-to-createeffective-online-education. (accessed April 2, 2020).

UNESCO (2020). COVID-19 Educational Disruption And response. Available online at: https://en.unesco.org/covid19/educationresponse (accessed May 19, 2020).

Google Scholar

Wang, C., Cheng, Z., Yue, X. G., and McAleer, M. (2020). Risk management of COVID-19 by 657 universities in China. J. Risk Financ. Manage. 13:36. doi: 10.3390/jrfm13020036

CrossRef Full Text | Google Scholar

WHO (2020a). Coronavirus Disease (COVID-19) Advice for the Public. Available online at: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/advice-forpublic (accessed April 29, 2020).

Google Scholar

WHO (2020b). Coronavirus Disease (COVID-19) Pandemic. Available online at: https://www.who.int/emergencies/diseases/novel-coronavirus-2019 (accessed May 22, 2020).

Google Scholar

Williams, C., Perlis, S., Gaughan, J., and Phadtare, S. (2018). Creation and implementation of a flipped jigsaw activity to stimulate interest in biochemistry among medical students. Biochem. Molecul. Biol. Educ. 46, 343–353. doi: 10.1002/bmb.21126

CrossRef Full Text | Google Scholar

Yang, M., Su, C., and Wang, W. (2017). The use of a DANP with VIKOR approach for establishing the model of e-learning service quality. EURASIA J. Math. Sci. Tech. Educ. 13, 5927–5937. doi: 10.12973/eurasia.2017.01041a

CrossRef Full Text | Google Scholar

Ye, S. H., Hsiao, T. H., and Sun, C. T. (2018). Using commercial video games in flipped classroom to support physical concept construction. J. Comput. Assist. Learn. 34, 602–614. doi: 10.1111/jcal.12267

CrossRef Full Text | Google Scholar

Zamora-Polo, F., Corrales-Serrano, M., Sánchez-Martín, J., and Espejo-Antúnez, L. (2019). university students training in general science using an active-learning merged pedagogy: gamification in a flipped classroom. Educ. Sci. 9:297. doi: 10.3390/educsci9040297

CrossRef Full Text | Google Scholar

Zare, M., Pahl, C., Rahnama, H., Nilashi, M., Mardani, A., Ibrahim, O., et al. (2016). Multi-criteria decision making approach in e-learning: a systematic review and classification. Appl. Soft Comput. 45, 108–128. doi: 10.1016/j.asoc.2016.04.020

CrossRef Full Text | Google Scholar

Zembylas, M. (2007). Emotional ecology: the intersection of emotional knowledge and pedagogical content knowledge in teaching. Teach. Teach. Educ. 23, 355–367. doi: 10.1016/j.tate.2006.12.002

CrossRef Full Text | Google Scholar