Measurement invariance of the teachers’ awareness scale in content-integrated education for multilingual learners

This study validates the scale, Critical Awareness toward Content-Language Integrated Education, recently designed to assess teachers’ awareness and beliefs toward Multilingual Learners (MLs) in content classrooms. By analyzing survey responses from a total of 458 teacher participants in the U.S., we examined evidence of validity, reliability, and measurement invariance. Confirmatory Factor Analysis (CFA) confirmed the three-factor structure, consisting of 34 items, with the suggestion to exclude one item out of 35 items. Multiple-group CFA (MG-CFA), considering teachers’ roles and for grade-levels of teaching, confirmed partial metric invariance and scalar invariance, with the constraints on three non-invariant items being released. Following the exclusion of the three items, subsequent data analyses were performed to compare teacher subgroups based on the finalized 31-item scale. The overall findings endorse the scale’s reliability and validity, supporting its applicability for researchers and practitioners in related fields.

1 Introduction

As the new content standards have raised expectations for all students’ academic achievement regardless of their background (Grapin, 2019), schools have been pressured to prove their instructional capacity in closing the growing achievement and engagement gaps between multilingual learners (MLs) and their mainstream peers in content-heavy classrooms (Caswell et al., 2016). It has been questioned whether teachers are adequately prepared to address the academic needs of MLs in the content classroom (Cochran-Smith et al., 2016). Surveys have been previously developed or adapted to measure latent constructs related to teachers’ culturally responsive teaching in “regular” (also called as “general” or “mainstream”) classrooms. The Language Attitudes of Teachers Scale (LATS; Byrnes and Kiger, 1994), comprising 13 items, delved into teachers’ beliefs and attitudes toward MLs. Using a principal component analysis (PCA) with oblique rotation, the scale is divided by three components, including teachers’ intolerance, support for MLs, and attitudes regarding language politics (e.g., “To be considered American, one should speak English.”). In a similar vein, Reeves (2006) designed a 16-item scale to measure secondary teachers’ attitudes regarding MLs’ integration into mainstream classrooms toward inclusion, coursework modification, professional development, and perceptions of language learning. Other researchers aimed to assess teachers’ utilization of instructional strategies in content classrooms to support MLs. Schall-Leckrone and McQuillan (2012) introduced a 21-item scale, focusing on language-based strategies employed in history classrooms (e.g., “I know approaches to teaching English Learners (ELs) how to write papers with arguments based on historical analysis”). Bacon (2020) designed a survey to assess teachers’ language ideologies, monolingualism, and instructional practices, unveiling subconstructs such as pedagogical confidence, agency, and language resource validation. Siwatu (2007) developed the Culturally Responsive Teaching Self-Efficacy (CRTSE) scale, rooted in Bandura’s social cognitive theory Bandura (1977). CRTSE assessed teachers’ competence in implementing specific teaching practices (e.g., “Greet English Language Learners with a phrase in their native language”) across four subconstructs: curriculum and instruction, classroom management, assessment, and cultural enrichment.

1.1 Critical awareness toward content-language integrated education for MLs

The CA-CIEML scale (Kim et al., 2023; Kim and Park, 2024) focuses on content teachers’ dual role: promoting MLs’ English language development and content knowledge acquisition within content classrooms. It is guided by the Linguistically and Culturally Responsive Content Teaching (LCRCT; Song et al., 2019) framework, emphasizing linguistically and culturally responsive teaching, which includes content-related and metacognitive competencies. LCRCT involves educators using pedagogical strategies that integrate content and language for MLs and exploring cross-cultural and sociopolitical beliefs in teaching MLs in content classrooms. The initial CA-CIEML scale version was created by adaptation and synthesis of validated scales and resources (Byrnes and Kiger, 1994; Reeves, 2006; Siwatu, 2007; Durgunoglu and Hughes, 2010; Schall-Leckrone and McQuillan, 2012; Grapin, 2019; Bacon, 2020; Thomas-Browne et al., 2023). Our goal is to provide an instrument for assessing teachers’ attitudes and beliefs related to LCRCT by diverse teacher characteristics. Each scale item is tailored to teachers working with MLs in content classrooms. The initial version of the scale consisted of a total of 38 items. Following an Exploratory Factor Analysis (EFA) conducted with responses from N = 307 K-12 teachers in the U.S., the scale was refined to 35 items (Kim et al., 2023), unveiling three prominent latent factors within the survey as follows:

• Language-integrated Content Teaching (Factor 1: 13 items) assesses teachers’ beliefs concerning the integration of ELs/MLs and their provision of supporting English language development within content classrooms (e.g., “Content teachers should provide additional language supports for ELs/MLs at all English proficiency level.”).

• English-only Monolingual Pedagogy (Factor 2: 9 items) assesses teachers’ perspectives on English-only instruction and students’ utilization of their native languages within and outside the classroom environment (e.g., “Using a student’s home language(s) in school will likely slow his or her progress in learning English.”).

• Sensitivity to ELs/MLs Backgrounds (Factor 3: 13 items) assesses teachers’ grasp of multilingual development and their consideration of the backgrounds of ELs/MLs’ in the context of content instruction and assessment (e.g., “I know my ELs’/MLs’ and their families’ backgrounds in terms of their national origin, ethnicity, and years of living in the U.S.”).

We note that the three salient factors reflect the major concerns in contemporary U.S. content classrooms, particularly with a growing Multilingual Learner (ML) population (de Jong and Naranjo, 2019): (1) Content teachers now should share the responsibility to support MLs’ content learning and literacy development simultaneously (Kim and Park, 2024), (2) English-only monolingual education policies and practices may impede, rather than enhance, MLs’ learning and identity development, necessitating a shift toward heteroglossic language ideologies (Pulinx et al., 2017; Bacon, 2020), and (3) New content-language integrated education should be grounded in students’ cultural and linguistic backgrounds, addressing their specific needs (Lucas and Villegas, 2011). See Supplementary Table S1 in Supplementary material (from Supplementary File) for item contents and descriptive statistics.

1.2 Research questions

After its initial validation, further research is required to assess the psychometric properties of the CA-CIEML scale. Previous research in the field has indicated a connection between educators’ attitudes and beliefs toward MLs and their ESOL training and certification (Youngs and Youngs, 2001; Martínez and Borko, 2009). Additionally, differences in pedagogical focus between elementary and secondary classrooms may lead secondary school teachers to perceive literacy development as beyond their scope, assuming it has already been addressed in elementary school or by specialized language teachers. Therefore, our study aims to validate the scale across teacher roles and grade levels to ensure the scale avoids potential item bias against specific subgroups. Specifically, we examine configural invariance to confirm that the 3-factor CFA fits the subgroups of the teachers, metric (weak) to ensure equivalency of factor loadings, and scalar (strong) invariances to determine the intercepts for each specific response option are the same between the two groups.

2 Methods

2.1 Participants

In June 2022, we conducted an online survey among K-12 teachers in the U.S., with participants receiving $20 compensation. The survey gathered responses from a total of 458 participants. Among these, 177 participants were classified as ESOL teachers, possessing or having held ESOL certification or endorsement, while the remaining 281 participants were categorized as non-ESOL or content teachers. In terms of grade levels, 175 teachers who taught K-5 in the past year were designated as elementary teachers, while those instructing upper grades were considered secondary teachers. On average, participants had 12.93 years of teaching experience (SD = 8.89) and 11.22 years of working with ELs/MLs (SD = 9.19).

2.2 Measure

The instrument consisted of 35 items representing three latent constructs: language-integrated content teaching (factor 1; 13 items), English-only monolingual pedagogy (factor 2; 9 items), and sensitivity to Els/MLs backgrounds (factor 3; 13 items). Each item used a 5-point Likert response options: 1 (“strongly disagree”), 2 (“disagree”), 3 (“neither disagree nor agree”), 4 (“agree”), and 5 (“strongly agree”).

2.3 Data analysis

First, we fitted three-factor confirmatory factor analysis (CFA) models to the data, considering teacher subgroups (teacher roles and grade-level of teaching). To account for the non-normality of ordinal responses, we employed a Weighted Least Squares Mean and Variance (WLSMV) estimator. We evaluated the goodness-of-fit of the models using robust fit indices, including Comparative Fit Index (CFI; Bentler, 1990) with a criterion of >0.95, the Tucker–Lewis Index (TLI; Tucker and Lewis, 1973) with a criterion of >0.95, and the Root Mean Square Error of Approximation (RMSEA; Steiger, 1990) with a criterion of <0.8 as well as $X^2$ /degrees of freedom (df) < 3.0. Second, we examined measurement invariance across distinct teaching roles and grade levels, fitting two-group three-factor multiple-group CFAs (MG-CFAs) with varying constraints: (a) configural invariance, (b) metric (weak) invariance, and (c) scalar (strong) invariance. R packages, lavaan (Rosseel, 2012) and semTools (Jorgensen et al., 2022) were used. As the chi-square test is sensitive to sample size, alternative model-fit indices such as CFI and RMSEA were utilized, with the cut-off values, $\Delta$CFI > 0.01 and $\Delta$RMSEA >0.015 (Chen, 2007), to determine both metric and scalar non-invariance. If non-invariance is found at each step, we released equality constraints between the two groups based on modification indices and the Lagrange multiplier test results to enhance model fit.

3 Results

3.1 Confirmatory factor analysis

Table 1 presents the single-group CFA results for both teacher roles, distinguishing between ESOL (n = 177) and content teachers (n = 281), as well as for grade-levels, i.e., elementary (n = 175) and secondary teachers (n = 283). Notably, one item within the first factor, exhibited unexpected results, showing negative factor loadings for specific subgroups (ESOL teachers and elementary teachers), unlike the remaining items within the factor. Following consultation with content experts, we decided to remove the item due to a conceptual inconsistency where a negative association for particular subgroups did not make sense. The remaining items maintain their content validity. Consequently, the results based on 34 items indicate that the three-factor structure met all criteria for acceptability in all subgroups (e.g., $X^2/df$ = 1.46, CFI = 0.957, TLI = 0.954, and RMSEA = 0.074 [0.065, 0.083] for ESOL teachers, see Table 1).

Table 1

Table 1. Three-factor CFA by teacher subgroups (34 items).

3.2 Measurement invariance across teaching roles

Table 2 (first two rows) presents the measurement invariance assessment based on teachers’ roles (ESOL versus content). The configural model demonstrated adequate model fit according to CFI, TLI, and RMSEA criteria, confirming the adequacy of the three-factor structure for both subgroups. However, the metric invariance model proved too restrictive, surpassing the cutoff value of CFI ( $\Delta$CFI = 0.018). Based on the modification indices from metric (weak) invariance model, we identified that imposing a constraint on the factor loading for an item within factor 3 (item #8) decreased model fit, exceeding the cutoff values (i.e., $\Delta$CFI = 0.018, $\Delta$RMSEA = 0.013). The result of factor loadings from MG-CFA suggests that the item was loaded on the factor (sensitivity to ELs/MLs backgrounds) weakly specifically for content teacher sample (less than 0.3). We examined the remaining 33 items to assess their suitability for both scalar invariance and metric invariance. During the metric invariance stage, we released the constraints on the item (item #8) while ensuring that the factor loadings and slopes for all other items remained equivalent. We found from the result that the changes in both CFI and RMSEA fell below the cutoff criteria ( $\Delta$CFI =0.012, $\Delta$RMSEA = 0.007), ensuring partial weak invariance. Furthermore, we assessed scalar invariance for the invariant items, while allowing item #8 to have different loading and intercept. We found that the scalar model remained tenable ( $\Delta$CFI = 0.001, $\Delta$RMSEA <0.001).

Table 2

Table 2. Multi-group CFA for Measurement Invariance (34 items).

3.3 Measurement invariance across grade-level of teaching

In Table 2 (bottom two rows), the configural model met all three criteria, while the metric invariance model did not ( $\Delta$CFI = 0.021, $\Delta$RMSEA = 0.023). Modification indices from the metric model revealed that constraining the factor loadings for an item within factor 2 (item #3) and another item within factor 1 (item #12) resulted in a notable decrease in model fit. The factor loading results from the MG-CFA indicate that the previous item exhibited weak loadings on Factor 2 for the secondary teacher sample, and the subsequent item showed weak loadings on Factor 1 for the elementary teacher sample. These loadings did not meet the minimal cutoff criteria for factor loadings (i.e., less than 0.3). Upon releasing the constraints on both slopes and intercepts, both CFI and RMSEA met the cutoff criteria ( $\Delta$CFI = 0.010, $\Delta$RMSEA = 0.008), confirming partial weak invariance. We also examined the scalar model while retaining the released factor loadings and intercepts for the two items to ensure that the remaining 32 items were persistent with regard to both metric invariance and scalar invariance. The results indicated that the scalar model (as well as the metric model) remained tenable ( $\Delta$CFI < 0.001, $\Delta$RMSEA = 0.001).

3.4 Internal consistency

Table 3 shows internal consistency reliability measures, including Cronbach’s alphas and Revelle’s omegas, calculated for different teaching roles and grade levels. Across all teacher subgroups, after excluding the three problematic items, the internal consistency of the finalized 31-item scale exceeded acceptable levels (>0.7).

Table 3

Table 3. Cronbach’s Alpha, and Revelle’s Omega (using the finalized 31-item scale).

3.5 Descriptive statistics and independent t-tests

The scores for the three subscales were computed by taking the averaging of the items associated with each respective factor, with a range from 1 to 5. As seen in Table 4, the descriptive statistics (means and standard deviations of the scores) are provided; and independent t-tests indicate significant differences between ESOL teachers and content teachers across all three subscales. Specifically, ESOL teachers were more inclined to support language-integrated content teaching in classrooms (t(456) = 7.892, p < 0.001) and displayed a higher degree of sensitivity to ELs/MLs’ backgrounds (t(456) = 7.978, p < 0.001), while showing less support for English-only monolingualism (t(456) = −4.357, p < 0.001). In general, the positive beliefs and attitudes expressed by ESOL teachers suggest a need for ESOL training and the integration of ESOL components into teacher preparation programs to better support multilingual learners (MLs). Similarly, significant differences were observed between elementary and secondary teachers in one out of three subscales: elementary teachers demonstrated less support for English-only monolingualism [t(456) = −6.255, p < 0.001]. The findings indicate that secondary teachers focus primarily on content-specific pedagogical training, lacking emphasis on language-supportive or multilingual education. In contrast, programs aimed at supporting the linguistic and academic development of U.S. English Learners are mainly implemented in elementary classrooms, with a growing emphasis on recognizing students’ cultural and linguistic backgrounds.

Table 4

Table 4. Means, standard deviation, and t-test for teacher subgroups (using the finalized 31-item scale).

4 Discussion

We conclude that the current version of the CA-CIEML scale (comprising 31 items) effectively assesses beliefs and attitudes concerning working with MLs in content classrooms. Additionally, significant differences in subscale scores between teacher roles (ESOL and Content) and grade levels imply the necessity for a more customized program to support teachers with varying backgrounds at the practical level.

As limitation, we acknowledge that there needs to be additional evidence to collect convergent validity of the scale by correlating it with an established teacher efficacy scale. Also, considering the population of K-12 teachers in the U.S., current sample size could potentially limit the generalizability of our findings.

As part of our ongoing validation efforts, future studies will focus on the development of a short version of the scale, considering its practical integration into regular use within teacher training programs. Result of its trajectories over multiple time during the programs will enable us to assess its effectiveness in modifying curriculum and teaching methods throughout the program. Also, measurement invariance will also be reevaluated over time by maintaining contact with the same teacher participants. These steps are crucial for addressing the growing research interest in understanding the extent to which ideological beliefs and attitudes influence teachers’ decisions to implement critical pedagogy in their everyday classroom practices. This will be achieved by testing the significance of longitudinal trajectories in the scales.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

JP: Writing – review & editing, Writing – original draft, Validation, Methodology, Formal analysis. SK: Writing – review & editing, Writing – original draft, Investigation, Funding acquisition, Conceptualization. XC: Writing – original draft, Validation, Resources, Methodology, Formal analysis.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. The research reported in this manuscript was supported by the Office of English Language Acquisition of the U.S. Department of Education under award number T365Z210097.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2024.1309476/full#supplementary-material

References

Bacon, C. K. (2020). “It’s not really my job”: a mixed methods framework for language ideologies, monolingualism, and teaching emergent bilingual learners. J. Teach. Educ. 71, 172–187. doi: 10.1177/0022487118783188

Crossref Full Text | Google Scholar

Bandura, A. (1977). Self-efficacy: toward a unifying theory of behavioral change. Psychol. Rev. 84, 191–215. doi: 10.1037/0033-295X.84.2.191

PubMed Abstract | Crossref Full Text | Google Scholar

Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychol. Bull. 107, 238–246. doi: 10.1037/0033-2909.107.2.238

Crossref Full Text | Google Scholar

Byrnes, D. A., and Kiger, G. (1994). Language attitudes of teachers scale (LATS). Educ. Psychol. Meas. 54, 227–231. doi: 10.1177/0013164494054001029

Crossref Full Text | Google Scholar

Caswell, L., Martinez, A., Lee, O., Berns, B. B., and Rhodes, H. (2016). Analysis of the National Science Foundation's discovery research K–12 on mathematics and science education for English learners. Teach. Coll. Rec. 118, 1–48. doi: 10.1177/016146811611800502

Crossref Full Text | Google Scholar

Chen, F. F. (2007). Sensitivity of goodness of fit indices to lack of measurement invariance. Struct. Equ. Model. 14, 464–504. doi: 10.1080/10705510701301834

Crossref Full Text | Google Scholar

Cochran-Smith, E. F., Grudnoff, L., Haigh, M., Hill, M., and Ludlow, L. (2016). Initial teacher education: what does it take to put equity at the center? Teach. Teach. Educ. 57, 67–78. doi: 10.1016/j.tate.2016.03.006

Crossref Full Text | Google Scholar

de Jong, E., and Naranjo, C. (2019). General education teacher educators and English language learner teacher preparation: infusion as curricular change. New Educ. 15, 331–354. doi: 10.1080/1547688X.2019.1663331

Crossref Full Text | Google Scholar

Durgunoglu, A. Y., and Hughes, T. (2010). How prepared are the US preservice teachers to teach English language learners? Int. J. Teach. Learni. High. Educ. 22, 32–41.

Google Scholar

Grapin, S. (2019). Multimodality in the new content standards era: implications for English learners. TESOL Q. 53, 30–55. doi: 10.1002/tesq.443

Crossref Full Text | Google Scholar

Jorgensen, T. D., Pornprasertmanit, S., Schoemann, A. M., and Rosseel, Y. (2022). semTools: Useful tools for structural equation modeling. R package version 0.5–6. Available at: https://CRAN.R-project.org/package=semTools

Google Scholar

Kim, S., and Park, J. Y. (2024). Critical awareness toward content-language integrated education for multilingual learners (CA-CIEML): a survey study about teachers’ ideological beliefs and attitudes. Lang. Aware. 9, 1–23. doi: 10.1080/09658416.2024.2321895

Crossref Full Text | Google Scholar

Kim, S., Park, J.Y., Chen, X., Ayik, B., Zan, Y., and Kim, W. (2023). The scale of content-integrated education for multilingual learners for critical awareness (CIEML-CA): a validation study. Paper presentation at the annual meeting of American Educational Research Association, Chicago, IL.

Google Scholar

Lucas, T., and Villegas, A. M. (2011). “A framework for preparing linguistically responsive teachers” in Teacher preparation for linguistically diverse classrooms: a resource for teacher educators. ed. T. Lucas (Routledge).

Google Scholar

Martínez, S., and Borko, H. (2009). Classroom assessment practices, teacher judgments, and student achievement in mathematics: evidence from the ECLS. Educ. Assess. 14, 78–102. doi: 10.1080/10627190903039429

Crossref Full Text | Google Scholar

Pulinx, R., Van Avermaet, P., and Agirdag, O. (2017). Silencing linguistic diversity: the extent, the determinants and consequences of the monolingual beliefs of Flemish teachers. Int. J. Biling. Educ. Biling. 20, 542–556. doi: 10.1080/13670050.2015.1102860

Crossref Full Text | Google Scholar

Reeves, J. R. (2006). Secondary teacher attitudes toward including English-language learners in mainstream classrooms. J. Educ. Res. 99, 131–143. doi: 10.3200/JOER.99.3.131-143

Crossref Full Text | Google Scholar

Rosseel, Y. (2012). Lavaan: an R package for structural equation modeling. J. Stat. Softw. 48, 1–36. doi: 10.18637/jss.v048.i02

Crossref Full Text | Google Scholar

Schall-Leckrone, L., and McQuillan, P. J. (2012). Preparing history teachers to work with English learners through a focus on the academic language of historical analysis. J. Engl. Acad. Purp. 11, 246–266. doi: 10.1016/j.jeap.2012.05.001

Crossref Full Text | Google Scholar

Siwatu, K. O. (2007). Preservice teachers’ culturally responsive teaching self-efficacy and outcome expectancy beliefs. Teach. Teach. Educ. 23, 1086–1101. doi: 10.1016/j.tate.2006.07.011

Crossref Full Text | Google Scholar

Song, K., Kim, S., and Zhao, Y. (2019). Manifesting multidimensional creativity in a technology mediated online TESOL practicum course. TESOL J. 11, 1–17. doi: 10.1002/tesj.472

Crossref Full Text | Google Scholar

Steiger, J. H. (1990). Structural model evaluation and modification: an interval estimation approach. Multivar. Behav. Res. 25, 173–180. doi: 10.1207/s15327906mbr2502_4

PubMed Abstract | Crossref Full Text | Google Scholar

Thomas-Browne, C. G., Boston, M. D., and Parke, C. S. (2023). Developing a tool to capture productive and unproductive mindsets about teaching mathematics to African American students: an initial review. Urban Educ. 58, 1799–1826. doi: 10.1177/0042085920934859

Crossref Full Text | Google Scholar

Tucker, L. R., and Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika 38, 1–10. doi: 10.1007/BF02291170

Crossref Full Text | Google Scholar

Youngs, C. S., and Youngs, G. A. (2001). Predictors of mainstream teachers’ attitudes toward ESL students. TESOL Q. 35, 97–120. doi: 10.2307/3587861

Crossref Full Text | Google Scholar