Modeling higher order thinking skills and metacognitive awareness in English reading comprehension among university learners

This study aimed to examine the levels of Pakistani university learners' higher order thinking skills (HOTS) in English reading comprehension and to explore how explicit assessments of HOTS can reveal learners' strengths and weaknesses in reading as compared to the traditional assessment of reading. Secondly, the study analyzed the effect of learners' metacognitive awareness on reading on the relationship between their HOTS levels and reading performance. It used a mixed-method approach. In the quantitative phase, an explicit HOTS-based reading test was designed using Bloom's taxonomy. Using random stratified sampling, it was administered to the learners of Bachelor of Science in English (n = 200). Stratified sampling was performed for investigating the gender gap in HOTS levels. The learners' metacognitive awareness of reading was analyzed using Survey of Reading Strategies. Descriptive analysis, independent sample t-test, correlation analysis, and moderator analysis were conducted. It was found that the learners' HOTS levels correlated with their reading performance where creative skill was the strongest, and evaluating skill was the weakest. Metacognitive awareness of reading positively moderated the relationship between the learners' HOTS levels and reading performance. In the qualitative phase, purposive stratified sampling was applied. A subsample (n = 6) was extracted from the core sample based on the high, average, and low reading performance, and three groups were formed. A semi-structured interview was conducted to know learners' opinions on using HOTS-based assessment in reading and their cognitive and metacognitive abilities. The thematic analysis of the qualitative data revealed that three out of six learners did not approve of the way they were assessed by the university. They informed that they had issues with cognitive and metacognitive control of English texts. The findings revealed that the majority of learners suffered low HOTS levels in English reading. The study offers recommendations for English as a second language (ESL) teachers, assessors, and policymakers to improve learner performance to the possible optimal level, especially in underdeveloped contexts that use traditional reading assessments.

Introduction

Every university learner should be able to comprehend advanced English reading texts to access the academic and global stock of knowledge, specifically in contexts where English is the sole medium of university instruction (Galloway and Ruegg, 2020). However, higher order thinking (e.g., analyzing the textual organization, evaluating arguments, and synthesizing information using previous knowledge) underlying reading comprehension differs from learner to learner. Prior studies have mainly focused on investigating the level of essential processual involvement of higher order thinking skills (HOTS) in instruction and materials of reading comprehension. Studies focusing on explicit taxonomic modeling of learners' HOTS levels in reading comprehension, targeting the cognitive skills specific to the process of reading, are still scarce (Saido et al., 2018). In underdeveloped education systems, such as in Pakistan, assessment practices for English as a second language reading comprehension are illusory and traditional relying largely on surface scoring. Stagnated low reading performance among university learners the past two decades is of great concern (OECD, 2014; University of the Punjab, 2020). Despite that, learners' HOTS levels in reading comprehension have never been modeled. Thus, the essential role of HOTS in producing effective reading comprehension seems to have been overlooked. In pursuance of investigating these university learners' strengths and deficits in HOTS that underlie reading comprehension, this study explicitly models their HOTS levels using higher order thinking skills specified in Bloom's revised taxonomy (Anderson et al., 2018).

Before we elaborate on the literature regarding the effect of using HOTS-based reading comprehension assessment, it is necessary to describe the purpose of modeling HOTS, and the corresponding context. Understanding the context of the current study can facilitate prospective research in reading assessment in developing countries while promoting consensus on why explicit modeling of learners' cognitive abilities is valuable in teaching-learning scenarios (Reynders et al., 2020). Contrary to developed countries, reading comprehension pedagogy and assessment in the Pakistani context (i.e., the context of the current study) entails a memorization-based system. The cognitive approaches to teaching and assessment have yet to take a firm ground in the tertiary education of the state. While researchers (Brookhart, 2017; Elleman and Oslund, 2019; Soto et al., 2019; Spencer et al., 2020; Cai and Chen, 2022) have largely asserted the pivotal role of HOTS and the explicit cognitive approach in producing effective reading comprehension, the Pakistani education system has not yet shown any significant response to this call of cognitive reforms (Hoodbhoy, 2021). According to Flippo et al. (2018), we can exploit the complementarities between HOTS and the reading comprehension process in assessing learners' cognitive strengths and deficits in reading comprehension. The core processes of advanced level reading comprehension include analyzing, evaluating, and creating textual information. These processes occur while a reader interacts with a text. Systematic and explicit modeling of these HOTS levels in learners can offer valuable solutions for low reading performance by identifying the levels that require remedial measures. Furthermore, the cognitive skills which contribute to reading performance can be optimized by such refined assessment. The cognitive processes are the basic components of comprehension since they simultaneously represent the cognitive and reading ability of the learner (Dewitz and Graves, 2021). For that matter, modern reading assessment should be designed on cognitive principles so that learners' HOTS levels can be explicitly mapped. The results obtained from traditional surface-scoring do not reveal latent reading abilities and deficits in the learners, hence, may be categorized as unauthentic assessments (Renandya et al., 2018). Reading assessment with the purpose of modeling learners' HOTS levels, combine the benefits of revealing the learners' actual latent performance and understanding their learning needs. Taken together, they can improve learner performance by initiating targeted remedial programs.

Research on reading comprehension, thus far, has been largely produced in the context of developed countries focusing on how to increase the learners' performance by engaging them more in cognitive activities. For example, Elleman and Oslund (2019), in the US context, examined the reasons why a number of adolescent learners had flat reading performance, and what policies and models could stimulate the growth of HOTS in these learners. Similarly, Kim et al. (2018), investigated the US beginner learners' cognitive sensitivity toward inconsistencies in reading comprehension and the role of online monitoring in it. Chow et al. (2021) investigated the reading performance of Chinese English as a second language (ESL) undergraduates. In the context of developing countries, most studies have focused on examining HOTS levels in academic reading materials and teaching (Aziz et al., 2017; Atiullah et al., 2019; Singh and Shaari, 2019; Ulum, 2021), and investigating the relationship between HOTS and reading performance, using critical thinking tests (Thamrin and Widodo, 2019; Tanudjaya and Doorman, 2020). The studies on explicit modeling of HOTS levels of the learner, instead of classroom materials or discourse, using HOTS-based reading tests are scarce in the context of developing countries (Mogaji et al., 2022) such as Pakistan (Hoodbhoy, 2021).

Link HOTS with reading comprehension and its curriculum design could, however, be further used to measure learners' HOTS abilities to bring about a difference in the teaching and learning of reading. It can prove beneficial at six broad levels: (1) knowing learners' cognitive levels can provide the teacher with valuable information about the learners' contributive and compromised HOTS; (2) using the information about the learning needs, teachers can target weak HOTS that require remedial measures (3) targeting deficient HOTS can potentially help struggling learners who are intimidated by higher-level reading comprehension activities; (4) explicit modeling can be used as an authentic design for future reading assessment, and (5) targeting weak HOTS can increase the learner performance (Allen and McNamara, 2020). Therefore, this study models university learners' HOTS levels in English reading comprehension to obtain detailed profiles of the learners' strengths and weaknesses in reading. This could be used to introduce an explicit design of HOTS-based reading assessment to university teachers to optimize learner performance. If teachers know the learning needs of their students, they can align their instruction and remedial programs to increase learner performance. For this purpose, this study has also included learners' metacognitive awareness of reading into the scope of this research. We argue that combining the inquiry of cognitive and metacognitive skills can provide a more accurate picture of the status of the Pakistani university learners about their HOTS in English reading. Knowing the level of the learners' metacognitive awareness of reading can reveal if the learners are able to apply their current cognitive skills to reading texts. As the literature on HOTS in reading comprehension is specifically limited to reading materials and classroom discourse, we will discuss below the cognitive approach and call for cognitive reforms to model learners' HOTS for assessing their reading abilities using context-valid assessment. We will also discuss other prominent and relevant HOTS concepts.

HOTS-based reading comprehension

Greater performance in university-level reading comprehension has largely been associated with greater academic achievement (Kosimov, 2022). Effective reading comprehension, especially in ESL contexts, is seen as an essential study skill in today's age of hyper-information and globalization (Moyon et al., 2017; Clinton-Lisell et al., 2022). In ESL contexts with English as the sole medium of university instruction, such as Pakistan (Irfan, 2019), proficiency in English reading comprehension is a tool to engage in academic activities as well as a doorway to better future opportunities for undergraduates. In a way, both academic and professional success is linked with effective reading comprehension abilities (Barber and Klauda, 2020). Many researchers (Brookhart, 2017; Köksal and Ulum, 2018; Horváthová and Nadová, 2021; Wu and Chen, 2021) agree that analytical, evaluative, and creative skills are the three core cognitive skills (or HOTS) that are highly represented in the process of reading comprehension. In simple words, reading comprehension includes analyzing, evaluating, and creating textual meanings. Ness (2019), following the principles of cognitive theory of reading comprehension and using think-aloud protocols, modeled learners' reading strategies such as inferring, evaluating, questioning, and synthesizing information. She mapped the development of the learners' reading comprehension and found a fair increase in their reading performance. Furthermore, Nappu and Hambali (2022), using the cognitive approach to reading, conducted a quasi-experiment by implementing HOTS in their classroom discourse. They concluded that the learners who were taught using the HOTS strategy in reading comprehension showed better learning outcomes as compared to the learners who were taught by the traditional method.

Collectively, the cognitive reforms of the 1970s' cognitive revolution in (second language) L2 education, changed the way reading comprehension was conceptualized. It emerged that it was a cognitive process based completely on the functioning of HOTS in the reader's cognitive dimension. As a result, teachers were exceedingly required to align the objectives, instruction, and assessment of reading with the cognitive principles of comprehension. As stated earlier, a considerable volume of empirical research in reading has asserted that effective reading comprehension is essentially linked with the learner's HOTS levels. Recently, Thamrin and Widodo (2019) conducted a study to improve learners' reading comprehension by developing their higher-order thinking. They found that students' poor reading comprehension performance improved with increasing their HOTS levels. Similarly, Lazonder et al. (2020) contend that there is a complementarity between HOTS and reading comprehension because both share the traits of problem-solving. However, processes within traditional education systems are still far behind in explicitly integrating HOTS into reading comprehension teaching, learning, and assessment. Where HOTS has been implemented in teaching reading comprehension in developing countries, it is mostly in an implicit manner. Moreover, HOTS-based reading assessment and diagnosis of learners' cognitive abilities that underly reading are scarce. Brookhart (2010) contends that reading should be assessed using an explicit HOTS strategy even if higher-order thinking is implicitly incorporated into instruction.

Recent studies (Cottrell, 2017; Carlson et al., 2022) propose that learners with low comprehension ability, in fact, lack HOTS. Thus, explicitly modeling learners' HOTS levels in reading can provide abundant opportunities for educators to know the exact skill(s) and subskill(s) that should be targeted to improve performance (Olson et al., 2018). Unfortunately, modeling these levels for assessing reading comprehension is not a frequent practice despite its dire need in the context of those with weak comprehension. The aforementioned studies, together with those that indicate enhanced neural activity while learners use their cognitive reserves (Hilton and Hilton, 2020), stipulate that it is important for the teacher to obtain detailed insight into students' mental models of reading and their engagement with texts by mapping HOTS levels. It becomes crucial at the university level since university learners need effective reading skills for securing a good level of academic achievement, and accessing the global stock of knowledge, research, and career prospects. Merely obtaining surface scores does not provide insights into learners' latent levels of higher-order thinking. Such simplistic assessment techniques do not stimulate learners' focus on learning specific skills for performing better hence, failing the objective of assessment for learning (Af L). Moreover, traditional assessments do not have the contextual affordance of reflecting on HOTS levels for skill diagnosis. On the contrary, detailed cognitive modeling in comprehension is parallel to cognitive diagnostic assessment that can provide each learner's developmental trajectory in reading (Tang and Zhan, 2021). It provides ample systematic information to the teacher and learner about how and where to focus on improvement. Information on learners' current HOTS levels can be used to simulate and reinforce their latent critical reading mechanism. As reading comprehension is an act of conceptual processing, this process should be modeled through relevant assessment to tap learners' reading abilities.

Analytical, evaluative, and creative skills as HOTS

Higher order thinking skills are a set of cognitive processes such as analytical, evaluative, and creative skills in the mental dimension of an individual, which are used in enterprises related to problem-solving (Fuchs et al., 2018). In the context of reading comprehension, processes such as inferring, analyzing textual formation, evaluating inconsistencies and consistencies, analyzing arguments, and creating information using previous cognitive frameworks, involve higher-order thinking. Additionally, the reader uses HOTS to comprehend and select relevant information in the text by de-noising it (Butterfuss and Kendeou, 2018; Shin et al., 2019). These higher-level processes compute reasoning in the mental workspace while the reader/learner interacts with the text. Therefore, they are crucial for the comprehension process as the whole process takes place in the cognitive dimension. Numerous studies (Brookhart, 2017; Morton and Colbert-Getz, 2017; Singh and Shaari, 2019; Surtantini, 2019; Lestari and Suhandi, 2020) have found a positive effect of higher-order thinking on learner achievement in reading. They have used the taxonomic specification of HOTS presented by Bloom's revised taxonomy (Anderson et al., 2018). In this regard, these studies have a consensus that the taxonomy's categorization of HOTS as analytical, evaluative, and creative skills is valid. Figure 1 illustrates the taxonomic levels of these skills.

FIGURE 1

Figure 1. HOTS in Bloom's taxonomy (conceptualized by the researchers).

Figure 1 shows that the taxonomic flow of HOTS is from analytical skill to evaluative skill to creative skill. These skills are complex mental processes that are used in advanced reading comprehension and other higher-order activities. Taxonomizing these skills have assisted teachers and researchers to understand the cognitive impact of materials, classroom discourse, and type of assessment. However, studies on modeling learners' HOTS levels and designing authentic and explicit assessments using skill categorization are scarce. Explicit categorization stimulates successful learning in a manner that is related to the structure and automation of skills (Stiller and Schworm, 2019). In the context of learners, these cognitive levels can be used as indicators of learners' reading comprehension performance. A recent study (Aziz et al., 2018) modeled learners' cognitive levels in analytical, evaluative, and creative skills for seeing the effect of HOTS on reading performance in the context of Pakistan. However, the interaction of HOTS with metacognitive skills in the context of Pakistani learners requires research attention. Acquiring a better understanding of the learners' latent cognitive abilities in the taxonomic form, also involving metacognitive awareness of reading, can offer valuable insights that can be used in increasing ESL learners' reading comprehension.

Effect of metacognitive awareness on reading

Closely related to higher-order thinking (the cognitive factor) in the comprehension process is metacognitive awareness of reading (the metacognitive factor), which is frequently defined as “cognition about cognition” or “thinking about thinking.” It is a set of metacognitive processes that are used to monitor ongoing thinking (Rhodes, 2019). Previous studies have investigated the direct effect of metacognitive skills on reading, yet few of them have analyzed the moderating effect of reading metacognitive skills on the relationship between learners' HOTS and reading performance. Prior studies (Darjito, 2019; Soto et al., 2019; Teng and Reynolds, 2019; Teng, 2020) have found a positive relationship between learners' metacognitive controls on the texts and reading performance. However, a recent study (McCarthy et al., 2018) did not replicate the findings of most prior studies. Instead, they found that explicit metacognitive prompts did not increase learners' reading performance. They argue that implicit integration of metacognitive strategies may produce better results. Burin et al. (2020) also indicated the possibility of metacognitive overload in learners that might have a negative effect on their performance. In this study, we considered the above contradictory findings and decided to evaluate the nature and level of the Pakistani learners' metacognitive awareness in English reading comprehension. Gaining detailed information about learners' reading metacognitive awareness along with their HOTS levels can reveal if they have metacognitive blocks or deficits in case of low HOTS levels.

Gender gap in HOTS levels and reading performance

Gender gap is a significant factor to consider in the context of reading comprehension performance. Numerous studies (Alvarado, 2017; Aditomo and Hasugian, 2018; Torppa et al., 2018; Reilly et al., 2019; Scholes, 2019) have found females outperforming males in reading. They conclude that female learners tend to be better at creative skills and overall comprehension than male learners. However, research on the gender gap at the university level reading performance is not rich in the context of Pakistan. Most studies in the Pakistani context conclude that the gender gap in reading translates to the underperformance of male learners (Shah and Armstrong, 2019). More investigation is needed to assess the validity of these findings. Pakistan has co-education institutions as well as gender-specific (i.e., males-only, and females-only) universities, therefore, investigating the gender gap holds valuable implications for gender-specific teaching-learning initiatives.

Van Hek et al. (2019) conclude that female learners have outperformed male learners in reading comprehension in all member countries of the Organization of Economic Cooperation and Development (OECD) showing a sizable score gap. Therefore, in this study, we were interested to explore the reasons behind the gender gap that favors females with a particular focus on the learner's cognitive abilities since reading comprehension is considered a cognitive enterprise. Moreover, cross-context findings can provide comparative opportunities in this regard. On a further note, it has been found recently that gender stereotypes may be a significant factor behind this gender gap as stereotypical thinking builds learners' self-concepts (Muntoni et al., 2021). Challenging the findings of previous research, a recent study (Alisaari et al., 2018) found that males outperformed females in reading comprehension. Therefore, the gender gap in reading remains unclear and many times confounding. This makes gender an important construct to be investigated. There is a possibility that it links further with female and male learners' self-efficacy, beliefs, and attitudes toward learning and problem-solving. Thus, gaining a closer insight into the gender gap in HOTS levels can help understand the differences in learners' reading comprehension mechanisms.

Explicit modeling of learners' HOTS levels in reading

Explicitly modeling HOTS levels of learners in reading comprehension to diagnose skill differences and deficits is rather an unexplored area specifically when it comes to developing countries with traditional educational systems such as Pakistan. The studies measuring ESL learners' HOTS levels using a learning taxonomy or valid guideline are noticeably scarce (Ichsan et al., 2019). Most prior studies in the Pakistani context have focused on determining HOTS levels of English reading textbook questions but the cognitive levels of the learner have never been directly investigated. At the university level, learners have been showing flat and low English reading performance for the past several years, yet their HOTS levels have not been explicitly investigated to determine individual differences and difficulties in HOTS (Din, 2020; Hoodbhoy, 2021). It can be a significant predictor of learners' reading performance and can play an important role in increasing it by remedying the diagnosed deficits. However, if this area of the investigation remains neglected, a lack of knowledge of learners' cognitive processes in reading may produce cognitive fixation in their reading comprehension. To elaborate, learners must be given robust and authentic HOTS-based evaluation in comprehension so that their learning needs can be understood. Learners, regularly assessed in analytical, evaluative, and creative skills (Anderson et al., 2018) can overcome reading deficits through self-regulation and focused attention. Explicit modeling of HOTS can immensely help teachers to customize their instruction according to each student's learning needs.

In the case of Pakistani university learners, the core reasons behind low reading performance are still unclear. Therefore, the nature and magnitude of these learners' higher-order thinking must be investigated since a vast body of literature has found the vital role of cognitive skills in producing effective comprehension. Furthermore, authentic modeling can remove general misconceptions among Pakistani teachers that rote-learning, memorization, and text replication can cultivate any quality reading comprehension.

Current study

Combining the research on HOTS and reading performance highlights the need to model learners' varying HOTS levels in reading using authentic guidelines. Furthermore, the application of higher-order thinking requires metacognitive awareness of reading in learners so investigating metacognitive awareness can also reveal significant information about learners' reading mechanisms. This study, therefore, addressed the following research questions:

1. How do the learners' HOTS levels in the English reading comprehension model against Bloom's taxonomy?

2. Is there a gender gap in the learners' HOTS levels?

3. How are the learners' HOTS levels related to their English reading comprehension performance?

4. How does the learners' metacognitive awareness of reading affect the relationship between their HOTS and reading performance?

5. What is the learners' perception of explicit assessment of HOTS in reading comprehension?

Based on the above literature review, we hypothesize that the learners' HOTS levels have a directly proportional relationship with reading comprehension performance, and metacognitive awareness of reading has a positive moderating influence on the relationship between the learners' HOTS levels and reading performance. Figure 2 illustrates the theoretical framework and moderation path of the study.

FIGURE 2

Figure 2. Theoretical framework of the study.

Figure 2 shows that HOTS (analytical, evaluative, and creative skills) is a set of independent variables while the learners' reading comprehension performance is a dependent variable. On the other hand, metacognitive awareness of reading is a moderating variable.

Methods

Design

A mixed-method design was selected for this investigation with the purpose of triangulation (Noble and Heale, 2019). Combining quantitative and qualitative approaches offered an enriched and deep understanding of the learners' HOTS, metacognitive skills, and opinions on explicit modeling for assessing English reading comprehension. The study was conducted in three phases. In the first phase, the learners' HOTS levels and reading scores were obtained. In the second phase, their metacognitive awareness levels relating to reading were obtained for conducting a moderator analysis. In the last phase, the learners were interviewed.

Participants

The target population of this study was university-level ESL learners from reading comprehension courses. Using random stratified sampling, 200 learners studying Bachelor of Science in English were selected from a university in Lahore, Pakistan. Random sampling was opted to reduce the risk of research bias to a minimum while a stratified technique was adopted for analyzing the gender gap in learners' performance. The learners were assumed to be a homogeneous sample because they had a similar academic background from which they were answering English reading comprehension questions; all participants had enrolled in a compulsory course of English academic reading comprehension at the selected university. The course outline claimed that it was designed for BSc (English) undergraduates. The participants were from the VII and VIII semesters that mark the final year of this bachelor's program. The final-year students were selected because, in Pakistan, after their first university graduation, students generally seek or intend to seek a job whether they continue their higher education or not. Moreover, effective English comprehension occupies a significant status in the skills demanded from fresh graduates for any job. This way, English comprehension plays quite an important role in the academic and prospective professional careers of these learners.

For conducting the interview, a subsample of six learners was extracted from the main sample employing purposive stratified sampling. Purposive sampling was performed to form three groups of learners based on reading performance while stratified sampling (n = 3 females; 3 males) was employed to understand gendered perspectives. It was considered important to explore the opinions of female and male learners to benefit both co-education and gender-specific institutions along with the findings of the current study. The groups were marked as High-Performance Group (HPG), Average-Performance Group (APG), and Low-Performance Group (LPG). While forming the groups, it was ascertained that a large gap existed between their reading performances.

Research instruments

The first instrument was a HOTS-based reading comprehension test battery comprising comprehension questions based on Bloom's HOTS—analyzing, evaluating, and creating. The items were adapted from IELTS reading tests and organized according to the three HOTS. The test battery was divided into three tests (i.e., Analyzing Skill Test, Evaluating Skill Test, and Creating Skill Test). The objective of these tests was to measure the levels of analytical, evaluative, and creative skills in the learners' reading comprehension. The items were selected based on the skills they assessed. Each test had one reading passage. To adjust the selected reading passages to university-level texts, their readability scores were calculated using three widely accepted measures namely the Flesch-Ease readability score (Flesch, 1948; Kincaid et al., 1975), the new Dale-Chall readability formula (Chall and Dale, 1995), and McAlpine EFlaw readability test (McAlpine, 2012). In total, nine readability tests were conducted. The readability scores of two passages loaded within the standard range were already recommended for university undergraduates. The third passage was modulated to the standard level by decreasing the cognitive load of its vocabulary. This objective was achieved by substituting the low-frequency words in the passage with high-frequency occurrences (Schurer et al., 2020), and running the reading tests again till the cognitive load matched the standard range. The instructions of the readability tests were followed for text modulation. Table 1 depicts the final readability scores of each passage.

TABLE 1

Table 1. Readability scores for the text passages.

Table 1 presents the final readability scores of each passage in the test battery. The Flesch-Ease scores of the passages fall within the standard range (30.0–50.0) recommended for university-level learners. The Dale-Call readability scores were also within the standard range (9.0–9.9) for university learners. Similarly, the McAlpine readability score falls within the recommended range (>30) for university learners.

Skill tests for “Analyzing,” “Evaluating,” and “Creating” consisted of 15, 10, and 15 multiple-choice items, respectively. Jovanovska (2018) contends that multiple-choice questions can be used as a dependable tool to measure learner performance. She states that creative skills can also be examined through multiple-choice items in case the items are well-written and taken to a higher order. Therefore, the items were designed to keep a large margin for the learners' application of creative ability in the shape of the synthesis of thought. Each test item had four distractors which fulfilled the criteria of the tests that measured cognitive abilities in reading. To substantiate reliability, the Kudor Richardson-20 Alpha test (Allen, 2017) was performed. The obtained value was 0.89, which was close to +1. It showed that the test was reliable as good reliability is close to +1 yet not greater than that (Adeniran, 2019).

The second instrument was a questionnaire titled “Survey of Reading Strategies” developed by Mokhtari and Sheorey (2002) for obtaining the level of ESL learners' metacognitive awareness of reading. The questionnaire has been reported valid and being widely used in several ESL contexts (Par, 2020). However, its reliability was re-determined to ensure it was appropriate in the Pakistani context. The obtained reliability value (α = 0.89) indicated its contextual suitability for further investigation.

The third instrument was a semi-structured in-depth interview in which the questions were designed according to the guidelines of the “Framework for Developing Interview Guide” (Kallio et al., 2016). Firstly, it was explored whether the in-depth interview can serve the objectives of the research questions. Secondly, the background knowledge on using the HOTS strategy in Pakistan was utilized to develop rational questions that could elicit ample information. Thirdly, a working draft was formulated. Fourthly, the questions were piloted on two students in the research setting. At this stage, certain changes were made to the draft based on the piloting. Lastly, the guide was updated. The questions were designed to gain descriptive insights into the learners' perception of their ability to analyze, evaluate and create textual information, explicitly modeling HOTS levels to assess their English reading comprehension, and applying their metacognitive skills to comprehend English texts. Asking these questions was possible since the interviewees were adult learners and could talk about what they could and could not do in comprehension. The interview was semi-structured to provide ample time and room for the learners to explicate their experience with HOTS modeling and opinions about metacognitive reading strategies.

Procedures

In the quantitative testing phase, both descriptive and inferential statistical analyses were performed on the quantitative data. To answer the research questions, the overall descriptives were obtained. Later, a correlation analysis, independent sample t-test, and moderator analysis were conducted. In the qualitative testing phase, the data gathered through the interviews were thematically analyzed to understand the learners' opinions and perceptions about HOTS modeling in reading.

Analysis

Quantitative analysis

The primary aim of the quantitative analysis was to explicitly model HOTS levels in female and male learners by analyzing the data that was obtained from their Analyzing, Evaluating, and Creating Skill Tests (the test battery). The descriptive analysis revealed the learners' latent levels of analytical, evaluative, and creative skills and prepared the data for inferential analysis. The analysis was performed in stratified form to analyze the levels of female and male learners separately. For this, the marks were converted into percentages of total marks because each HOT skill test had different total marks due to the different number of subskills. Then, the average percentage of the learners' scores in each skill and subskill was calculated. Expressing the learners' scores by percentages removed the problem of unequal total scores in each test. There was no missing data (n = 200) while the results represented the performance of 100 female and 100 male learners. Reading performance was estimated by totaling the scores and obtaining the average percentage of each skill as it was hypothesized that HOTS (e.g., analytical, evaluative, and creative skills) represents the reading comprehension process (Brookhart, 2017; Allen and McNamara, 2020). This data was then used for conducting a part-whole correlation analysis to determine the strength of the relationship between HOTS levels and the learners' scores in reading comprehension. Through correlation analysis normality the regression analysis was conducted to find the most contributive skill set from HOTS to learners' reading comprehension. Thus, regression analysis helped determine the learners' weakest HOTS.

After the regression analysis, learners' metacognitive awareness level related to reading was investigated through descriptive analysis of the questionnaire data. The values obtained were further used in moderator analysis (see Figure 1). For the moderator analysis, we fitted a regression model analyzing which we investigated the effect of metacognitive awareness of reading (Z) on the relationship between the learners' HOTS levels [X = Xa (analytical skill) + Xb (evaluative skill) + Xc (creative skill)] and reading comprehension performance (Y) according to the common guidelines.

Qualitative analysis

The interview was conducted after administering the HOTS test battery. All interviews were recorded after acquiring the permission of the interviewees. Later, the gathered data was transcribed verbatim. A thematic analysis (TA) was conducted to find out the significant themes in the learners' opinions and perceptions. The themes were discussed in detail among the researchers. The listed themes were selected through consensus. In TA, the guidelines of Braun and Clarke (2022) were followed. Accordingly, only coherent and rich themes were analyzed keeping in view their relevance to the research questions of this study. To begin with, the transcripts were organized for data reduction. Data reduction was performed to refine, organize, converge, dispose of, and prepare the data for identifying significant themes and patterns of relevance. It was done without compromising the quality of the general “truth space” of the data. After that, the data were assigned open codes related to the interview themes. The recurrent themes were obtained by matching codes. In the end, the main themes were defined and labeled.

Results

Before investigating the correlation and moderation of the variables, we considered it necessary to determine the normality of data distribution. For that purpose, in the preliminary analysis, the values of data skewness and kurtosis were determined. These values were determined for the tests (analytical, evaluative, and creative skills) and the questionnaire (Survey of Reading Strategies data). Tables 2, 3 show the results, respectively.

TABLE 2

Table 2. Skewness and kurtosis values of Survey of Reading Strategies.

TABLE 3

Table 3. Skewness and kurtosis values of reading comprehension tests.

The skewness and kurtosis values indicated that the data were normally distributed since all skewness values were < 3, and kurtosis values < 10, which falls into the standard range. The normal distribution of the data indicated that the data were fit for further analysis. The findings of the analysis are described against each research question.

Learners' modeled HOTS levels

For the first research question, the HOTS levels of the learners were obtained. The learners' HOTS (analyzing, evaluating, and creating) levels are shown in Table 4.

TABLE 4

Table 4. HOTS levels of the learners.

The findings in Table 4 demonstrate that the mean of the level of creative skill, both in females and males (F = 21.60; M = 19.28) was the highest among other skills. The mean of the level of evaluating skill was the lowest, both in females and males (F = 6.95; M = 5.79). Both genders scored the highest in creative skills, and the lowest in evaluative skills. The mean level of analytical skill was in the middle of the other two mean values both in females and males (F = 11.02; M = 9.23). From these results, it can be inferred that creative skill is the strongest HOT skill both in female and male learners whereas evaluating skill is the weakest. However, the overall HOTS levels of the female and male learners are low since the levels equaled C to D+ grades against the assessment key that the university of the learners uses (Prospectus UOL, 2022, p. 293). It is noticeable that HOTS levels of the female learners are higher than male learners, with the mean difference greater in analytical skills than in the other two skills. This finding was further substantiated empirically by conducting an independent sample t-test on the levels' data.

Gender gap in learners' HOTS levels

Referring to the second research question, the t-test analysis of the female and male learners' HOTS levels revealed a statistically significant gender gap favoring females. Table 5 shows the findings of the t-test.

TABLE 5

Table 5. Gender gap in the learners' HOTS levels.

It can be seen in Table 5 that p-values for the gender gap in analytical, evaluating, and creative skills are statistically significant. Moreover, the p-value of creative skills (p = 0.028) is higher than that of analytical skills (p = 0.030) and evaluating skills (0.028). It implies that evaluating skills have the lowest significance and shows that the gender gap is more pronounced in creative skills and least pronounced in evaluating skills. Furthermore, the difference between the statistical significance of the gap between creating and other two skills is greater than that between analytical and evaluating skills. It implies that the level of performance of the female and male learners in analytical and evaluating skills is less different than that in creative skills.

Relationship between ESL learners' HOTS and reading performance

To answer the third research question, first, the reading comprehension performance of the learners was analyzed in its overall and stratified form from the learners' total scores in the test battery. The descriptive statistics of this variable are shown in Table 6.

TABLE 6

Table 6. Learners' reading comprehension performance.

The minimum reading scores of the females were 22 whereas the maximum was 85 (M = 53.50; SD = 14.010). In males, the minimum scores were 18 whereas the maximum was 75 (M = 46.50; 14.380). It shows that the reading scores of females are higher than the scores of males while the difference is also reflected in the minimum and maximum scores. Interpreted against the assessment key, the overall mean scores of the learners are low.

After analyzing the reading performance, a correlation analysis was conducted between HOTS levels and reading comprehension scores of the learners. As it was a part-whole correlation, Lee (2000) statistical formula was used in the correlation analysis to remove spuriousness in the correlation values. In the formula (see below), Tx represented a HOT skill every time the formula was applied since there were three skills for computing correlation, Tz represented the overall test battery, and E represented the estimate of error:

$\begin{array}{r}\rho (Tx,Tz)=\frac{\sigma (Tx,Tz)}{\sigma (Tx)\alpha Tz}\\ =\frac{\sigma \left[(X-Ex)(Z-Ez)\right]}{\sigma (X)\sigma (Z)\frac{\sigma (Tx)\sigma (Tz)}{\sigma (X)\sigma (Z)}}\\ =\frac{\sigma (X,Z)-\sigma (X,Ez)-\sigma (Z,Ex)+\sigma (Ex,Ez)}{\sigma (X)\sigma (Z)\sqrt{}Pxx,Pzz}\end{array}$

The obtained results of the correlation between analytical, evaluative, and creative skills and the reading comprehension performance of the learners are shown in Table 7.

TABLE 7

Table 7. Correlation between learners' reading comprehension performance and HOTS.

The findings of the correlation analysis demonstrated the strength of the relationship of the learners' HOT skills in their individual form with the learners' reading comprehension performance. It can be seen in Table 7 that analytical, evaluative, and creative skills have a positively significant correlation with learners' reading performance. The highest correlation is between creative skill and reading performance (r = 0.665) whereas the lowest correlation is between evaluating skill and reading performance (r = 0.577). The correlation of analytical skills with reading performance (r = 0.609) is in the middle of the correlation values of creating and evaluating skills. This supports the findings of the first question that also indicated creative skill as the strongest and evaluating skill as the weakest in the learners. The positive correlation between the learners' HOTS levels and reading performance implies that the learners' cognitive ability influences their reading performance in a directly proportional manner. If these learners' HOTS levels are increased, their comprehension performance will also increase, and deficits in HOTS will decrease their reading performance. If the learners' performance in HOTS and reading comprehension are compared, both are low. The presence of a positive correlation indicates that the reading comprehension low scores are due to HOTS deficits among these learners. It supports Braun et al. (2020), who asserted a vital positive link between HOTS and reading comprehension.

After performing the correlation analysis, a step-wise regression analysis was performed to determine the most contributive skillset from HOTS to the learners' reading comprehension. According to Pallant (2020), for conducting a regression analysis, no value of prior correlations between the independent variables should exceed the cutoff value of 0.70. Table 7 shows that no correlation value exceeded this cutoff value. Another condition for conducting regression analysis is that n ≥ 40 for every single variable (Pallant, 2020). This condition was already satisfied in the sample size of this study (n = 200). In the analysis, HOTS levels were the independent variables whereas the learners' reading performance was the dependent variable. Table 8 shows the results of the regression analysis.

TABLE 8

Table 8. Regression analysis of HOTS levels and reading performance.

According to Table 8, the learners' HOTS significantly contributes to the reading comprehension performance (F sig. < 0.05). However, creative skill (β = 0.575) was the best predictor of the learner's performance in reading comprehension because of the highest value of the standardized beta (β). It implies that creative skill is the most contributive skill to the learners' reading performance. It is followed by analytical skills (β = 0.278) and evaluation skills (β = 0.212). This shows that evaluating skills are the weakest HOT skill in the learners' reading comprehension.

It confirms the primary results of the mean of the learners' HOTS levels (see Table 4) in which creative skill is the strongest whereas evaluating skill is the weakest.

Effect of learners' reading metacognitive awareness on HOTS and reading performance

The fourth research question was answered by constructing a path diagram (see Figure 1) to investigate the moderating effect of learners' metacognitive awareness of reading on the relationship between their HOTS levels and reading comprehension performance. Before conducting a moderator analysis, the learners' levels of metacognitive awareness of reading were analyzed using the data obtained from the “Survey of Reading Strategies.” This questionnaire had three categories namely Global Reading Strategies, Support Strategies, and Problem-Solving Strategies. The obtained levels of the categories are presented in Table 9.

TABLE 9

Table 9. Levels of metacognitive awareness of reading in learners.

Table 9 shows that the learners' use of Global Reading Strategies (Mean = 3.259) is higher than that of Support Strategies (M = 2.823) and Problem-Solving strategies (M = 3.243). The least used metacognitive strategies were Support Strategies. It shows that learners usually purposively read and use tables and figures to understand what they are reading. However, they are weak in restating the ideas in their own words, which indicates a weakness in comprehension (Mokhtari and Sheorey, 2002). Finding relationships, or in other words, evaluating and critiquing may pose problems to them (Brdarevic Celjo et al., 2021). It is noticeable that evaluating is the weakest of these learners' HOTS. On account of this finding, it was hypothesized that these learners' metacognitive skills of reading would have a relationship and affect their use of HOTS in reading. For empirical confirmation, and referring to the fourth research question, a moderator analysis was performed using the learners' modeled HOTS levels, reading comprehension performance, and metacognitive awareness. The findings demonstrated that the learners' metacognitive awareness of reading positively influenced the relationship between their HOTS levels and reading performance. Tables 10–12 show the results of the moderator analysis relating to analytical, evaluative, and creative skills, respectively.

TABLE 10

Table 10. Effect of reading metacognitive awareness on analyzing skill.

TABLE 11

Table 11. Effect of reading metacognitive awareness on evaluating skill.

TABLE 12

Table 12. Effect of reading metacognitive awareness on creating skill.

The results of the moderator analysis are important as they reveal patterns of significance similar to the results of the correlation analysis. According to these results, the learners' metacognitive awareness of reading moderates the connection between their analytical, evaluative, and creative skills and reading comprehension. The moderation value (p = 0.008) for creative skill and reading performance is the highest. The effect of the metacognitive awareness of reading (p = 0.41) on the relationship between evaluating skill and reading performance is the lowest. The significance value (p = 0.020) for analytical skills lies in the middle of the p-values of creative and evaluating skills. Overall, it indicates that the effect of reading metacognitive awareness is positive on the learners' HOTS and reading relationship. The relationship is promoted by the learners' metacognitive controls of texts. It also implies that low metacognitive awareness negatively affects learners' textual understanding. Additionally, the learners will not be able to proficiently apply their HOTS to the text to comprehend it. In simple words, it supports Teng (2020) who found that reading metacognitive strategies are positively linked with enhancing textual comprehension to which HOTS are central.

Discussion

This study analyzed how explicitly modeled HOTS levels of ESL learners are related to their reading performance while also considering learners' reading metacognitive awareness. This study helped in filling the relevant research gap in the Pakistani context. To achieve this objective, quantitative data analysis (to model the learners' HOTS levels in reading), as well as qualitative data analysis (to discern the learners' opinions about using HOTS-based assessment in reading), were performed.

The findings of this study indicated that the learners' HOTS—analytical, evaluative, and creative skills—highly correlate with their reading performance. Moreover, their reading metacognitive awareness has a positive effect on the relationship between the learners' HOTS and reading performance. The importance of modeling HOTS levels was also suggested by prior studies (Tyas et al., 2019; Xiangping and Xiaodong, 2019; Yunita et al., 2020; Al Maawali, 2021) because these studies analyzed the use of HOTS strategy in either classroom instruction or textbook questions. The current study supports the findings of the previous studies (Brookhart, 2017; Elleman and Oslund, 2019; Spencer et al., 2020) that found a vital link between HOTS and reading performance. On the other hand, the detailed modeling of HOTS levels also revealed that Pakistani learners' HOTS levels have some divergences from Bloom's taxonomic level-up (see Figure 1). The learners' HOTS levels did not replicate the level-up scheme of “analyzing to evaluating to creating.” Instead, they revealed a trajectory of “evaluating to analyzing to creating.” However, female and male learners showed a similar pattern of skill levels even while a significant gender gap existed in their reading performance. This study elucidated that in the Pakistani context, the university learners' creative skill is the most contributive to their reading comprehension whereas their evaluating skill is the least contributive.

Based on the positive relationship between HOTS and reading comprehension, we hypothesized that the HOTS levels of the majority of the participants in this study would be low because the reading comprehension results of Pakistani university students had been discouraging for the past many years (University of the Punjab, 2020). In simple words, we expected the low ability comprehenders to have low HOTS levels, which was later confirmed by the findings. The analysis of the learners' interview responses also revealed HOTS deficits in these learners. However, a more generalizable sample may produce different results. For that matter, future investigations, engaging a larger sample size, are required.

The findings about the learners' metacognitive awareness of reading demonstrated that its effect on the relationship between the learners' HOTS and reading performance is promotional in nature. The increase in the learners' reading metacognitive awareness is likely to increase these learners' effective use of HOTS to comprehend English reading texts. It is in line with previous studies (Darjito, 2019; Soto et al., 2019; Teng and Reynolds, 2019; Teng, 2020) which found that ESL learners' comprehension is geared up by effective reading metacognitive skills. It, in turn, implies that learners' lack of metacognitive control of the text can negatively affect their critical reading performance (Lindholm and Tengberg, 2019).

The subsample for the interview was extracted from the main sample, and three proficiency groups were formed based on their overall reading performance. As stated in the Method section, these groups were called HPG, APG, and LPG based on high, average, and low performance. Female and male learners from HPG had different opinions about being assessed in reading comprehension by explicit HOTS strategy. One female learner had a positive opinion about using this assessment method and thought that it would add to her learning and focus on reading. The male counterpart was half-willing toward being assessed explicitly in reading. However, both these learners were self-confident that they could comprehend any advanced level text in English. They found working with their creative ability in comprehending English texts enjoyable. They also expressed that while analyzing and evaluating texts was easy, however, evaluating arguments was harder than synthesizing information. These learners are also aware of their metacognitive skills to be effective. They said that they did not feel any anxiety while working with English texts independently. This indicates that these learners can self-regulate their reading.

The second APG group secured average marks in reading comprehension but contrary to our expectations, both female and male learners wanted to improve their reading using “any available technique.” We expected this group to be less enthusiastic about using explicit HOTS assessment for examining reading comprehension, however, their responses indicated a positive opinion about HOTS modeling. According to these learners, they found advanced English texts challenging while stumbling often in evaluating arguments and inconsistencies. This pointed to their weak evaluating skill in reading, which requires immediate remedial measures. This group had a negative opinion about the way they were assessed by their university. They said that explicit reading assessment based on HOTS could increase their comprehension skills and should be included in the reading comprehension curriculum. These learners revealed that they experienced anxiety while working independently on a difficult text. A possible explanation for such anxiety is that these learners lack adequate metacognitive skills through which they can regulate their comprehension. Nervousness and anxiety while reading also indicates a lack of reading self-efficacy, which requires investigation.

The opinions of the third group, labeled as LPG, were close to the quantitative findings because the majority of the sample had low HOTS levels. One female learner from the LPG group informed that her reading comprehension abilities were low, and she often fell short of vocabulary and what “lies at the heart of the sentences.” The male learner said that he tried but he could not put together the deep meaning of the text. This shows a lack of analytical and organizing skills according to Anderson et al. (2018). Moreover, both learners perceived their creative skills to be weak and found English comprehension taxing. They informed that they feared explicit HOTS assessment, but they anticipated that with time it will increase their reading performance. A possible explanation for this is the long-term exposure of LPG learners to traditional assessments that encourages memorization (Malik, 2021). With fossilized abilities, learners find modern, self-regulatory, formative, or diagnostic methods of assessment intimidating. These learners also revealed that their metacognitive skills were low as they felt confused about the questions relating to metacognitive techniques. However, the learners' abilities cannot be disregarded considering their exposure to traditional assessment. It may be a matter of practice and the lack of explicit instruction and assessment in reading. However, it needs more investigation if such learners can improve within a short period after they are exposed to explicit instruction or/and assessment.

The overall current findings are in line with Din (2020), who found that Pakistani university learners scored low on Watson-Glazer critical thinking test. It is also in line with Ali et al. (2017) who found that adult Pakistani learners were weak at proficiently using HOTS in inferential reading comprehension in English. Taking together the findings of the prior studies and the current investigation, it can be inferred that Pakistani university learners' low reading performance is connected with the absence of an explicit HOTS strategy in teaching and assessing English reading. HOTS-based assessment, administered explicitly, can make these learners invest in garnering specific critical thinking skills. This way, the learner will focus on the most relevant reading skill while the teachers will have a tool for diagnosing the students' HOTS deficits to improve them.

Limitations and recommendations for future research

This study had two limitations. The first limitation is the sample size, which by the current provincial population of Lahore was found fit as we calculated the sample size using Cohen's d (cited in Goulet-Pelletier and Cousineau, 2018). However, broadening the periphery of the population as well as the sample could provide more generalizable results. In the current study, using a larger sample size would have provided more reliable and generalizable conclusions. However, valuable information can still be found in this study for gathering relevant implications. The second limitation is that more learners could have been interviewed in the qualitative data collection, but this study was conducted under COVID-19 lockdown measures that challenged our access to a larger sample. However, given that the assessment reliability was efficiently determined with no relevant loophole, the students' responses and performance in the assessment can be efficiently used for modeling learners' HOTS levels in reading comprehension.

During data analysis, the findings revealed that improving the learners' evaluating skills is more pressing than improving their analytical and creative skills. Future research models can focus exclusively on evaluation skills along with their subskills for more in-depth analysis. Similarly, analytical and creative skills can be separately investigated focusing on their relevant subskills. Such detailed investigations are necessary because general assessments cannot capture the learner's performance in an in-depth manner (Kruit et al., 2018).

Against this background, future studies should extend the framework of this study by integrating more mediating or/and moderating variables that relate to the complex process of reading comprehension such as self-efficacy and aptitude. Furthermore, think-aloud protocols involving English texts should be used to investigate learners' HOTS levels in reading comprehension. It may provide detailed insight into the learners' minds while they try to comprehend English texts.

Implications

Pedagogical implications

The current findings have five implications for ESL pedagogy, particularly in the context of developing educational systems. Firstly, the findings of the quantitative analysis emphasize the significance of the vital role of HOTS in reading comprehension, which has, thus far, been overlooked in Pakistani tertiary education (Hoodbhoy, 2021). Secondly, empirical information regarding learners' reading abilities and higher-order thinking skills may prove to be a significant tool for teachers to understand the student's learning needs and tailor their teaching accordingly (Schlatter et al., 2020). The findings also indicate that explicitly modeling learners' HOTS levels can elicit relevant data on learners' strengths and weaknesses in reading comprehension. Thirdly, in-depth information on their cognitive skills can help learners to focus on improving their specific skills and self-regulate them in reading. Lack of proper assessment in reading can misdirect learners toward rote learning and text replication in comprehension. Fourthly, the findings endorse prior studies which have indicated the inevitable role of higher order thinking in effective reading comprehension. Understanding this role can increase teachers' assessment literacy in ESL reading comprehension.

Policy implications

This study offers two significant recommendations for ESL education policymakers. Firstly, policymakers should reform the guidelines for instruction and assessment of reading comprehension in light of cognitive principles of comprehension. The findings of this study also demonstrate that the learner's performance in reading largely depends on emphasizing the cognitive facet of comprehension. Secondly, policymakers should develop standardized schemes and guidelines for the explicit practice of HOTS in reading comprehension assessments. They should make effective policies for increasing learners' metacognitive awareness in reading, as indicated by Veenman (2017).

Conclusion

This study modeled Pakistani university learners' HOTS levels in English reading comprehension with the purpose to diagnose their cognitive strengths and weaknesses. The study was motivated by the stagnated low performance of the learners for the past several years. The quantitative findings, extending on prior studies, demonstrated the significance of HOTS-based assessment of reading comprehension by tapping learners' cognitive and metacognitive skills. The qualitative findings revealed learners' opinions about using HOTS-based assessment in reading comprehension. This study's findings show that within the context of HOTS in Pakistani learners' reading comprehension, their creative skills were the strongest while their evaluation skills were the weakest. The study recommends integrating explicit assessment of HOTS in assessing reading comprehension at the university level. It also has implications for ESL/EFL education policymakers.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Ethics statement

Ethical review and approval was not required for the study involving human participants in accordance with the local legislation and institutional requirements. Written informed consent to participate in this study was not required from the participants in accordance with the national legislation and the institutional requirements.

Author contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

Funding

This research was funded by the Fundamental Research Grant Scheme (FRGS) awarded by the Ministry of Higher Education Malaysia, under Grant No: 13133.

Acknowledgments

Our appreciation is also extended to the University of Utara Malaysia, Sintok, Malaysia.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Adeniran, A. O. (2019). Application of Likert scale's type and Cronbach's alpha analysis in an airport perception study. Scholar J. Appl. Sci. Res. 2, 1–5.

Google Scholar

Aditomo, A., and Hasugian, E. J. (2018). Indonesian adolescents' EFL reading comprehension: Gender differences and the influence of parental background. Ind. J. Appl. Ling. 8, 325–335. doi: 10.17509/ijal.v8i2.13279

CrossRef Full Text | Google Scholar

Al Maawali, J. (2021). The Learning and Assessment of Higher-Order Thinking Skills (HOTS) in the Omani English Language Curriculum (Doctoral dissertation). Sydney: UNSW.

Google Scholar

Ali, A., Javed, M., and Shabbir, G. (2017). Assessing ESL students' literal, reorganization and inferential reading comprehension abilities. J. Educ. Res. 20, 42–53.

Google Scholar

Alisaari, J., Turunen, T., Kajamies, A., Korpela, M., and Hurme, T. R. (2018). Reading comprehension in digital and printed texts. Educ. Stud. Lang. Lit. 18, 1–18. doi: 10.17239/L1ESLL-2018.18.01.15

CrossRef Full Text | Google Scholar

Allen, L. K., and McNamara, D. S. (2020). “Defining deep reading comprehension for diverse readers,” in Handbook of Reading Research, Volume V (London: Routledge), 261–276.

Google Scholar

Allen, M. (2017). The SAGE Encyclopedia of Communication Research Methods. Thousand Oaks: Sage Publications. doi: 10.4135/9781483381411.n493

CrossRef Full Text | Google Scholar

Alvarado, L. K. A. (2017). Gender SCORE GAps of Colombian students in the Pisa Test (No. 015675). Bogotá: Universidad Javeriana.

Google Scholar

Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruiskshant, K. A., Mayer, R. E., Pintrich, P. R., et al. (2018). A Taxonomy for Learning, Teaching and Assessing: A Revision of Bloom's Taxonomy of educational Objectives. Longman.

PubMed Abstract | Google Scholar

Atiullah, K., Fitriati, S. W., and Rukmini, D. (2019). Using revised bloom's taxonomy to evaluate higher order thinking skills (hots) in reading comprehension questions of english textbook for year x of high school. English Educ. J. 9, 428–436. doi: 10.15294/eej.v9i4.31794

CrossRef Full Text | Google Scholar

Aziz, A. A., Ismail, F., Ibrahim, N. M., and Samat, N. A. (2017). Investigating the implementation of higher order thinking skills in malaysian classrooms: insights from l2 teaching practices. Sains Humanika 9, 4–2. doi: 10.11113/sh.v9n4-2.1361

CrossRef Full Text | Google Scholar

Aziz, M., Al-Hasani, A. M. A., and Buriro, G. A. (2018). A correlation analysis of ESL learners' textual comprehension performance, higher order thinking skills and type of background schooling. Res. J. Social Sci. 16, 83–95.

Google Scholar

Barber, A. T., and Klauda, S. L. (2020). How reading motivation and engagement enable reading achievement: policy implications. Policy Insights Behav. Brain Sci. 7, 27–34. doi: 10.1177/2372732219893385

CrossRef Full Text | Google Scholar

Braun, H. I., Shavelson, R. J., Zlatkin-Troitschanskaia, O., and Borowiec, K. (2020). Performance assessment of critical thinking: conceptualization, design, and implementation. Front. Educ. 5, 156. doi: 10.3389/feduc.2020.00156

CrossRef Full Text | Google Scholar

Braun, V., and Clarke, V. (2022). Conceptual and design thinking for thematic analysis. Qual. Psychol. 9, 3. doi: 10.1037/qup0000196

CrossRef Full Text | Google Scholar

Brdarevic Celjo, A., Becirovic, S., and Dubravac, V. (2021). An examination of perceived reading strategy use among university level students. Eur. J. Contemp. Educ. 10, 595–608. doi: 10.13187/ejced.2021.3.595

CrossRef Full Text | Google Scholar

Brookhart, S. M. (2010). How to Assess Higher-Order Thinking Skills in Your Classroom. ASCD.

Google Scholar

Brookhart, S. M. (2017). How to Give Effective Feedback to Your Students. Alexandria, VA: ASCD.

Google Scholar

Burin, D. I., Gonzalez, F. M., Barreyro, J. P., and Injoque-Ricle, I. (2020). Metacognitive regulation contributes to digital text comprehension in E-learning. Metacogn. Learn. 15, 391–410. doi: 10.1007/s11409-020-09226-8

CrossRef Full Text | Google Scholar

Butterfuss, R., and Kendeou, P. (2018). The role of executive functions in reading comprehension. Educ. Psychol. Rev. 30, 801–826. doi: 10.1007/s10648-017-9422-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Cai, Y., and Chen, H. (2022). The fluctuating effect of thinking on language performance: new evidence for the island ridge curve. Lang. Assess. Q. 2022, 1–15. doi: 10.1080/15434303.2022.2080553

CrossRef Full Text | Google Scholar

Carlson, S. E., Broek, P. V. D., and McMaster, K. L. (2022). Factors that influence skilled and less-skilled comprehenders' inferential processing during and after reading: exploring how readers maintain coherence and develop a mental representation of a text. Elem. School J. 122, 719477. doi: 10.1086/719477

CrossRef Full Text | Google Scholar

Chall, J. S., and Dale, E. (1995). Readability Revisited: The New Dale-Chall Readability Formula. Brookline: Brookline Books.

Google Scholar

Chow, B. W. Y., Mo, J., and Dong, Y. (2021). Roles of reading anxiety and working memory in reading comprehension in English as a second language. Learn. Indiv. Diff. 92, 102092. doi: 10.1016/j.lindif.2021.102092

CrossRef Full Text | Google Scholar

Clinton-Lisell, V., Taylor, T., Carlson, S. E., Davison, M. L., and Seipel, B. (2022). Performance on reading comprehension assessments and college achievement: a meta-analysis. J. College Read. Learn. 2022, 1–21. doi: 10.1080/10790195.2022.2062626

CrossRef Full Text | Google Scholar

Cottrell, S. (2017). Critical Thinking Skills: Effective Analysis, Argument and Reflection. London: Bloomsbury Publishing.

Google Scholar

Darjito, H. (2019). Students' metacognitive reading awareness and academic english reading comprehension in EFL context. Int. J. Instruct. 12, 611–624. doi: 10.29333/iji.2019.12439a

CrossRef Full Text | Google Scholar

Dewitz, P., and Graves, M. F. (2021). The science of reading: four forces that modified, distorted, or ignored the research finding on reading comprehension. Reading Res. Q. 56, S131–S144. doi: 10.1002/rrq.389

CrossRef Full Text | Google Scholar

Din, M. (2020). Evaluating university students' critical thinking ability as reflected in their critical reading skill: a study at bachelor level in Pakistan. Thinking Skills Creat. 35, 100627. doi: 10.1016/j.tsc.2020.100627

CrossRef Full Text | Google Scholar

Elleman, A. M., and Oslund, E. L. (2019). Reading comprehension research: implications for practice and policy. Policy Insights Behav. Brain Sci. 6, 3–11. doi: 10.1177/2372732218816339

CrossRef Full Text | Google Scholar

Flesch, R. (1948). A new readability yardstick. J. Appl. Psychol. 32, 221. doi: 10.1037/h0057532

PubMed Abstract | CrossRef Full Text | Google Scholar

Flippo, R. F., Armstrong, S. L., and Schumm, J. S. (2018). “Reading tests,” in Handbook of College Reading and Study Strategy Research (London: Routledge), 340–366

Google Scholar

Fuchs L. S. Gilbert J. K. Fuchs D. Seethaler P. M. and N. Martin B. (2018). Text comprehension and oral language as predictors of word-problem solving: insights into word-problem solving as a form of text comprehension. Sci. Stud. Read. 22, 152–166. doi: 10.1080/10888438.2017.1398259

PubMed Abstract | CrossRef Full Text | Google Scholar

Galloway, N., and Ruegg, R. (2020). The provision of student support on English Medium Instruction programmes in Japan and China. J. Eng. Acad. Purpose 45, 100846. doi: 10.1016/j.jeap.2020.100846

CrossRef Full Text | Google Scholar

Goulet-Pelletier, J. C., and Cousineau, D. (2018). A review of effect sizes and their confidence intervals, Part I: the Cohen's d family. Quant. Methods Psychol. 14, 242–265. doi: 10.20982/tqmp.14.4.p242

CrossRef Full Text | Google Scholar

Hilton, A., and Hilton, G. (2020). “Higher order thinking,” in Teaching Middle Years (London: Routledge), 223–242. doi: 10.4324/9781003117780-16

CrossRef Full Text | Google Scholar

Hoodbhoy, P. (2021). “Pakistan's Higher Education System 38,” in Handbook of Education Systems in South Asia, eds P. M. Sarangapani and R. Pappu (Singapore: Springer), 977. doi: 10.1007/978-981-15-0032-9_64

CrossRef Full Text | Google Scholar

Horváthová, B., and Nadová, L. (2021). Developing critical thinking in reading comprehension of texts for specific purposes at all levels of Bloom's taxonomy. J. Teach. English Specific Acad. Purp. 2021, 001–016. doi: 10.22190/JTESAP2101001H

CrossRef Full Text | Google Scholar

Ichsan, I. Z., Sigit, D. V., Miarsyah, M., Ali, A., Arif, W. P., and Prayitno, T. A. (2019). HOTS-AEP: higher order thinking skills from elementary to master students in environmental learning. Eur. J. Educ. Res. 8, 935–942. doi: 10.12973/eu-jer.8.4.935

CrossRef Full Text | Google Scholar

Irfan, H. (2019). The Policy and Practice of English Medium of Instruction (EMI) in Pakistani Universities. Newcastle upon Tyne: Cambridge Scholars Publishing.

Google Scholar

Jovanovska, J. (2018). Designing effective multiple-choice questions for assessing learning outcomes. Infotheca 18, 25–42. doi: 10.18485/infotheca.2018.18.1.2

CrossRef Full Text | Google Scholar

Kallio, H., Pietilä, A. M., Johnson, M., and Kangasniemi, M. (2016). Systematic methodological review: developing a framework for a qualitative semi-structured interview guide. J. Adv. Nurs. 72, 2954–2965. doi: 10.1111/jan.13031

PubMed Abstract | CrossRef Full Text | Google Scholar

Kim, Y. S. G., Vorstius, C., and Radach, R. (2018). Does online comprehension monitoring make a unique contribution to reading comprehension in beginning readers? Evidence from eye movements. Sci. Stud. Read. 22, 367–383. doi: 10.1080/10888438.2018.1457680

PubMed Abstract | CrossRef Full Text | Google Scholar

Kincaid, J. P., Fishburne Jr, R. P., Rogers, R. L., and Chissom, B. S. (1975). Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Millington TN: Naval Technical Training Command Research Branch. doi: 10.21236/ADA006655

CrossRef Full Text | Google Scholar

Köksal, D., and Ulum, Ö. G. (2018). Language assessment through Bloom's taxonomy. J. Lang Linguist. Stud. 14, 76–88.

Google Scholar

Kosimov, A. (2022). “The role of critical reading strategies on reading comprehension in EFL students and its impact to second language acquisition,” in International Journal of Conference Series on Education and Social Sciences, Vol. 2.

Google Scholar

Kruit, P. M., Oostdam, R. J., van den Berg, E., and Schuitema, J. A. (2018). Assessing students' ability in performing scientific inquiry: instruments for measuring science skills in primary education. Res. Sci. Technol. Educ. 36, 413–439. doi: 10.1080/02635143.2017.1421530

CrossRef Full Text | Google Scholar

Lazonder, A. W., Walraven, A., Gijlers, H., and Janssen, N. (2020). Longitudinal assessment of digital literacy in children: findings from a large Dutch single-school study. Comput. Educ. 143, 103681. doi: 10.1016/j.compedu.2019.103681

CrossRef Full Text | Google Scholar

Lee, G. (2000). A disattenuated part-whole correlation formula. J. Educ. Meas. 37, 279–280. doi: 10.1111/j.1745-3984.2000.tb01087.x

CrossRef Full Text | Google Scholar

Lestari, A., and Suhandi, A. (2020). “An analysis of hots in the 5th grade elementary school students' learning with radec model with the theme of “electricity around us,” in International Conference on Elementary Education, Vol 2, 1574–1582.

Google Scholar

Lindholm, A., and Tengberg, M. (2019). The reading development of Swedish L2 middle school students and its relation to reading strategy use. Read. Psychol. 40, 782–813. doi: 10.1080/02702711.2019.1674432

CrossRef Full Text | Google Scholar

Malik, M. K. A. (2021). ELT assessment patterns dictate teaching-learning approaches: a hindrance to map out employability and life skills. Asian ESP J. 128, 74. doi: 10.5539/ijel.v11n3p74

CrossRef Full Text | Google Scholar

McAlpine, R. (2012). From Plain English to Global English. Available online at: http://www.webpagecontent.com/arc_archive/139/5/#score (accessed June 28, 2022).

McCarthy, K. S., Likens, A. D., Johnson, A. M., Guerrero, T. A., and McNamara, D. S. (2018). Metacognitive overload!: Positive and negative effects of metacognitive prompts in an intelligent tutoring system. Int. J. Artif. Intell. Educ. 28, 420–438. doi: 10.1007/s40593-018-0164-5

CrossRef Full Text | Google Scholar

Mogaji, E., Jain, V., Maringe, F., and Hinson, R. E. (2022). “Reimagining educational Futures in developing countries: an introduction,” in Re-imagining Educational Futures in Developing Countries (Cham: Palgrave Macmillan), 3–14. doi: 10.1007/978-3-030-88234-1_1

CrossRef Full Text | Google Scholar

Mokhtari, K., and Sheorey, R. (2002). Measuring ESL students' awareness of reading strategies. J. Dev. Educ. 25, 2–10.

Google Scholar

Morton, D. A., and Colbert-Getz, J. M. (2017). Measuring the impact of the flipped anatomy classroom: the importance of categorizing an assessment by Bloom's taxonomy. Anatom. Sci. Educ. 10, 170–175. doi: 10.1002/ase.1635

PubMed Abstract | CrossRef Full Text | Google Scholar

Moyon, K., Hiner, A., and Richardson, K. (2017). “Cultivating academic habits of mind: key skills for collegiate success,” in Winthrop Conference on Teaching and Learning. 3. Available online at: https://digitalcommons.winthrop.edu/tlcconference/2017/schedule/3 (accessed June 15, 2022).

Google Scholar

Muntoni, F., Wagner, J., and Retelsdorf, J. (2021). Beware of stereotypes: are classmates' stereotypes associated with students' reading outcomes? Child Dev. 92, 189–204. doi: 10.1111/cdev.13359

PubMed Abstract | CrossRef Full Text | Google Scholar

Nappu, S., and Hambali, U. (2022). The implementation of higher-order thinking skill (HOTS) in learning reading for EFL learners. English Lang. Teach. Methodol. 2, 54–64.

Google Scholar

Ness, M. K. (2019). “Think aloud, think along, think alone: gradually releasing students to use comprehension strategies in elementary classrooms,” in The Gradual Release of Responsibility in Literacy Research and Practice (Bradford: Emerald Publishing Limited). doi: 10.1108/S2048-045820190000010014

CrossRef Full Text | Google Scholar

Noble, H., and Heale, R. (2019). Triangulation in research, with examples. Evid. Based Nurs. 22, 67–68. doi: 10.1136/ebnurs-2019-103145

PubMed Abstract | CrossRef Full Text | Google Scholar

OECD (2014). PISA 2012 Technical Report. Paris: OECDPublishing.

Google Scholar

Olson, G. M., Duffy, S. A., and Mack, R. L. (2018). “Thinking-out-loud as a method for studying real-time comprehension processes,” in New Methods in Reading Comprehension Research (London: Routledge), 253–286. doi: 10.4324/9780429505379-11

CrossRef Full Text | Google Scholar

Pallant, J. (2020). SPSS Survival Manual: A Step by Step Guide to Data Analysis Using SPSS for Windows Fifth Edition. Berkshire: Open University Press.

Google Scholar

Par, L. (2020). The relationship between reading strategies and reading achievement of the EFL students. Int. J. Instruct. 13, 223–238. doi: 10.29333/iji.2020.13216a

PubMed Abstract | CrossRef Full Text | Google Scholar

Prospectus UOL (2022). Available online at: https://uol.edu.pk/wp-content/uploads/2021/06/prospectus-2021-2022.pdf (accessed July 5, 2022).

Reilly, D., Neumann, D. L., and Andrews, G. (2019). Gender differences in reading and writing achievement: evidence from the National Assessment of Educational Progress (NAEP). Am. Psychol. 74, 445. doi: 10.1037/amp0000356

PubMed Abstract | CrossRef Full Text | Google Scholar

Renandya, W. A., Hamied, F. A., and Nurkamto, J. (2018). English language proficiency in Indonesia: issues and prospects. J. Asia TEFL 15, 618. doi: 10.18823/asiatefl.2018.15.3.4.618

CrossRef Full Text | Google Scholar

Reynders, G., Lantz, J., Ruder, S. M., Stanford, C. L., and Cole, R. S. (2020). Rubrics to assess critical thinking and information processing in undergraduate STEM courses. Int. J. STEM Educ. 7, 1–15. doi: 10.1186/s40594-020-00208-5

CrossRef Full Text | Google Scholar

Rhodes, M. G. (2019). Metacognition. Teach. Psychol. 46, 168–175. doi: 10.1177/0098628319834381

CrossRef Full Text | Google Scholar

Saido, G. M., Siraj, S., Nordin, A. B. B., and Al_Amedy, O. S. (2018). Higher order thinking skills among secondary school students in science learning. MOJES 3, 13–20.

Google Scholar

Schlatter, E., Molenaar, I., and Lazonder, A. W. (2020). Individual differences in children's development of scientific reasoning through inquiry-based instruction: Who needs additional guidance? Front. Psychol. 11, 904. doi: 10.3389/fpsyg.2020.00904

PubMed Abstract | CrossRef Full Text | Google Scholar

Scholes, L. (2019). Differences in attitudes towards reading and other school-related activities among boys and girls. J. Res. Read. 42, 485–503. doi: 10.1111/1467-9817.12279

CrossRef Full Text | Google Scholar

Schurer, T., Opitz, B., and Schubert, T. (2020). Working memory capacity but not prior knowledge impact on readers' attention and text comprehension. Front. Educ. 5, 26. doi: 10.3389/feduc.2020.00026

CrossRef Full Text | Google Scholar

Shah, S. A., and Armstrong, G. (2019). Exploring the Gender Gap in Reading in Pakistan. Teaching and Teacher Education. Cham: Palgrave Macmillan, 113–149.

Google Scholar

Shin, J., Dronjic, V., and Park, B. (2019). The interplay between working memory and background knowledge in L2 reading comprehension. Tesol Q. 53, 320–347. doi: 10.1002/tesq.482

CrossRef Full Text | Google Scholar

Singh, R. K. V., and Shaari, A. H. (2019). The analysis of higher-order thinking skills in english reading comprehension tests in Malaysia. Geografia 15, 12–26. doi: 10.17576/geo-2019-1501-02

CrossRef Full Text | Google Scholar

Soto, C., Gutiérrez de Blume, A. P., Jacovina, M., McNamara, D., Benson, N., and Riffo, B. (2019). Reading comprehension and metacognition: the importance of inferential skills. Cogen. Educ. 6, 1565067. doi: 10.1080/2331186X.2019.1565067

CrossRef Full Text | Google Scholar

Spencer, M., Richmond, M. C., and Cutting, L. E. (2020). Considering the role of executive function in reading comprehension: a structural equation modeling approach. Sci. Stud. Read. 24, 179–199. doi: 10.1080/10888438.2019.1643868

PubMed Abstract | CrossRef Full Text | Google Scholar

Stiller, K. D., and Schworm, S. (2019, March). Game-based learning of the structure and functioning of body cells in a foreign language: effects on motivation, cognitive load, and performance. Front. Educ. 4, 18. doi: 10.3389/feduc.2019.00018

CrossRef Full Text | Google Scholar

Surtantini, R. (2019). Reading comprehension question levels in grade X English students' book in light of the issues of curriculum policy in Indonesia. PAROLE J. Ling. Educ. 9, 44–52. doi: 10.14710/parole.v9i1.44-52

CrossRef Full Text | Google Scholar

Tang, F., and Zhan, P. (2021). Does diagnostic feedback promote learning? Evidence from a longitudinal cognitive diagnostic assessment. AERA Open 7, 23328584211060804. doi: 10.1177/23328584211060804

CrossRef Full Text | Google Scholar

Tanudjaya, C. P., and Doorman, M. (2020). Examining higher order thinking in Indonesian lower secondary mathematics classrooms. J. Math. Educ. 11, 277–300. doi: 10.22342/jme.11.2.11000.277-300

CrossRef Full Text | Google Scholar

Teng, F. (2020). The benefits of metacognitive reading strategy awareness instruction for young learners of English as a second language. Literacy 54, 29–39. doi: 10.1111/lit.12181

CrossRef Full Text | Google Scholar

Teng, F., and Reynolds, B. L. (2019). Effects of individual and group metacognitive prompts on EFL reading comprehension and incidental vocabulary learning. PLoS ONE 14, e0215902. doi: 10.1371/journal.pone.0215902

PubMed Abstract | CrossRef Full Text | Google Scholar

Thamrin, N. R., and Widodo, P. (2019). Developing higher order thinking skills (HOTS) for reading comprehension enhancement. J. Phys. Conf. Ser. 1179, 012073.

Google Scholar

Torppa, M., Eklund, K., Sulkunen, S., Niemi, P., and Ahonen, T. (2018). Why do boys and girls perform differently on PISA Reading in Finland? The effects of reading fluency, achievement behaviour, leisure reading and homework activity. J. Res. Read. 41, 122–139. doi: 10.1111/1467-9817.12103

CrossRef Full Text | Google Scholar

Tyas, M. A., Nurkamto, J., Marmanto, S., and Laksani, H. (2019). “Developing higher order thinking skills (HOTS)-Based questions: Indonesian EFL teachers' challenges,” in Proceeding of the 2nd International Conference on Future of Education, vol. 2, 52–63. doi: 10.17501/26307413.2019.2106

CrossRef Full Text | Google Scholar

Ulum, Ö. G. (2021). A critical approach to the inclusion of the revised Bloom's Taxonomy in active skills for reading: Book 1. RumeliDE Dil ve Edebiyat Araştirmalari Dergisi 25, 273–296. doi: 10.29000/rumelide.1037134

CrossRef Full Text | Google Scholar

University of the Punjab (2020). Results. Available online at: http://pu.edu.pk/home/results (accessed July 5, 2022).

Van Hek, M., Buchmann, C., and Kraaykamp, G. (2019). Educational systems and gender differences in reading: a comparative multilevel analysis. Eur. Sociol. Rev. 35, 169–186. doi: 10.1093/esr/jcy054

CrossRef Full Text | Google Scholar

Veenman, M. V. (2017). Assessing metacognitive deficiencies and effectively instructing metacognitive skills. Teachers College Record 119, 1–20. doi: 10.1177/016146811711901303

CrossRef Full Text | Google Scholar

Wu, Y. J., and Chen, J. C. (2021). Stimulating innovation with an innovative curriculum: a curriculum design for a course on new product development. Int. J. Manag. Educ. 19, 100561. doi: 10.1016/j.ijme.2021.100561

CrossRef Full Text | Google Scholar

Xiangping, W., and Xiaodong, Z. (2019). Multi-modal analysis of textbook discourse—taking the high school English textbook of pep (2019 edition) as an example. Int. J. New Dev. Educ. 4, 1–7. doi: 10.25236/IJNDE.2022.040601

CrossRef Full Text | Google Scholar

Yunita, W., Syahrial, S., and Hati, G. M. (2020). English teachers' knowledge on Higher Order Thinking Skills (HOTS). Engl. Rev. J. Engl. Educ. 9, 205–216.

Google Scholar