A School-Based Measure of Culturally Responsive Practices

Asil, Mustafa

doi:10.3389/feduc.2017.00017

ORIGINAL RESEARCH article

Front. Educ., 17 May 2017

Sec. Assessment, Testing and Applied Measurement

Volume 2 - 2017 | https://doi.org/10.3389/feduc.2017.00017

A School-Based Measure of Culturally Responsive Practices

$\r\nMustafa Asil*\r\n$ Mustafa Asil*

Educational Assessment Research Unit, College of Education, University of Otago, Dunedin, New Zealand

This study aimed to address a gap in current research on cultural responsiveness by introducing the Culturally Responsive Practices for Māori Scale (CRPMS), a self-report instrument designed to measure schools’ culturally responsive practices in relation to Māori students. Within this paper, the development and analysis of the factorial structure and psychometric properties of the CRPMS are presented. The validity evidence findings from this study supported the utility of the CRPMS as a measure of cultural responsiveness for mainstream English-speaking New Zealand schools. Potential uses and applications of the CRPMS by researchers and educational stakeholders are discussed.

Introduction

The latest national and international assessment results [National Education Monitoring Project, National Monitoring Study of Student Achievement (NMSSA), Trends in International Mathematics and Science Study, Programme for International Student Assessment] have identified New Zealand (NZ) as having one of the widest spread of achievement among students within schools rather than between schools. Māori students in New Zealand have been identified by the Ministry of Education as a “priority learner group” because they have lower achievement outcomes, are more likely to leave school earlier without formal qualifications, and are less likely to be identified as being gifted and talented than their majority ethnic counterparts (Macfarlane, 2004; Bishop et al., 2007, 2009; Marie et al., 2008; Savage et al., 2014). To address these disparities, which are reflected at all levels of the education system, the Ministry of Education has embarked on creating culturally responsive learning environments to raise Māori student success.

The culturally responsive learning environments are an integral part of the New Zealand Curriculum (NZC) and is detailed and operationalized in various policies, publications, and initiatives, such as Ka Hikitia—Accelerating Success 2013–2017 (Ministry of Education, 2013), Te Kotahitanga (Bishop et al., 2009), Practising Teacher Criteria (Education Council New Zealand, 2015), Tātaiako: cultural Competencies for Teachers of Māori Learners (Ministry of Education, 2011), He Kākano (University of Waikato and Te Whare Wānanga o Awanuiārangi, 2010), Pasifika Education Plan (Ministry of Education, 2012a), and success for all: every school, every child (Ministry of Education, 2012b).

Previous research (Gay, 2000; Bishop et al., 2007, 2009; Macfarlane et al., 2007; Education Review Office, 2010; Meyer et al., 2010; Aronson and Laughter, 2016) has demonstrated that all students, and not only indigenous students, benefit from and perform better in culturally responsive educational settings. However, the achievement gap between Māori and non-Māori students has remained large to date (Controller and Auditor-General, 2016). In their study, Meyer et al. (2010), p. 21, argued that “there was considerable variability in the cultural responsiveness of schools, from being highly responsive to things Māori to minimal evidence of awareness of Māori students’ cultural backgrounds and experiences at the school level.”

Cultural Responsiveness and the New Zealand Context

Today’s schools are becoming increasingly diverse with regard to culture, language, ethnicity, religion, and socioeconomic status. Consequently, schools are challenged with providing an effective learning environment for every student in response to an increasingly diverse and multicultural student population. Culturally relevant pedagogy (Ladson-Billings, 1995), culturally responsive teaching (Gay, 2000), or culturally sustaining pedagogy (Paris, 2012) are some of the terms or concepts that appear in the education literature to emphasize collective empowerment and social justice (Aronson and Laughter, 2016) fostering and promoting educational equity and excellence in schools through cultural diversity.

Gay (2000), p. 29, defined culturally responsive teaching as “using the cultural knowledge, prior experiences, frames of reference, and performance styles of ethnically diverse students to make learning encounters more relevant and effective for them.” She further notes that “being culturally responsive is more than being respectful, empathetic, or sensitive” (p. 3).

According to Gay (2013), successful implementation of culturally responsive teaching requires: eradicating the deficit view of students and communities; expecting, understanding, and resisting the opposition to culturally responsive teaching; teachers’ understanding how and why culture and difference are essential ideologies and foundations; and establishing pedagogical connections between culturally responsive teaching and specific subjects and skills taught in schools. The ability of principals and teachers to engage in cultural responsiveness requires that they have a certain degree of cultural competence themselves as well (Castagno and Brayboy, 2008).

Like New Zealand, many Western educational systems continue to result in poorer learning outcomes for indigenous, minority, or vulnerable students. However, there are countries such as Japan and some Nordic countries that have more equitable learning outcomes than others (Pickett and Wilkinson, 2009), which indicate that it is achievable.

Schools need to provide high-quality teaching for heterogeneous groups of students (Alton-Lee, 2003). There is an increasing recognition of the importance of cultural responsiveness and its influence on student outcomes.

Although the Ministry of Education is continuing to build a culturally responsive education system where all children can progress and achieve to their very best, Māori and non-Māori disparity in learning outcomes persists [Organisation for Economic Co-operation and Development (OECD, 2010, 2013); Cavanagh et al., 2012; Educational Assessment Research Unit and New Zealand Council for Educational Research, 2013a,b, 2014a,b, 2015a,b]. In this regard, a closer examination of New Zealand schools’ culturally responsive practices is vital to address this achievement disparity and Māori people’s concerns about current pedagogic practices in mainstream schools.

National Monitoring Study of Student Achievement (NMSSA)

As part of gaining an understanding of schools’ cultural responsiveness and its association with the achievement of priority learner groups, the NMSSA: Wānangatia Te Putanga Tauira developed a new self-report inventory, the Culturally Responsive Practices for Māori Scale (CRPMS), drawing on the educultural wheel (Macfarlane, 2004).

National Monitoring Study of Student Achievement is a collaboration between the Educational Assessment Research Unit at the University of Otago, the New Zealand Council for Educational Research, and the Ministry of Education. This program has been operating since 2012 with a 5-year cycle. NMSSA is sample based monitoring project designed to assess and understand student achievement at year 4 and year 8 in New Zealand primary schools.

A nationally representative sample of approximately 5,000 students in 200 schools are randomly selected across year 4 and year 8 each year to take part in NMSSA. Assessments are administered by experienced and trained teacher assessors. NMSSA uses complementary forms of assessments: group administered paper and pencil or computer-based assessments and in-depth interviews and performance assessments.

To help understand factors that are related to student achievement, contextual information is collected via student, teacher, and principal questionnaires. Approximately 600 teachers and 200 principals from participating schools are invited to respond to teacher and principal questionnaires, respectively. NMSSA has a responsibility to focus on three priority learner groups: Māori students, Pasifika students, and students with special education needs (Allan, 2012).

Most of the New Zealand research related to cultural responsiveness is qualitative in nature and uses checklists and case studies [e.g., Education Review Office (ERO)’s school evaluation indicators]. No psychometric instrument has been developed to specifically assess cultural responsiveness of schools at a national level. Therefore, a new self-report inventory, CRPMS, was developed as part of the NMSSA project.

A review of the related research literature revealed that only a few instruments have been developed specifically to measure cultural responsiveness, including Siwatu’s (Siwatu, 2007) Culturally Responsive Teaching Self-Efficacy and Culturally Responsive Teaching Outcome Expectancy scales. However, these measures and most of the others elicit information from teachers regarding their beliefs or expectations rather than their practices.

The purpose of the study reported in this paper is to present the development and analysis of the factorial structure and psychometric properties of the CRPMS designed to measure and understand schools’ cultural responsiveness practices in relation to Māori.

Theoretical Background

The theoretical framework for the development of CRPMS was grounded within the educultural wheel proposed by Macfarlane (2004) and guided by Ka Hikitia—Accelerating Success 2013–2017 (Ministry of Education, 2013).

Macfarlane’s (2004) educultural wheel framework was designed to support the development of positive interactions between teachers and Māori students. It consists of five interrelated concepts (or factors) that encompass key aspects of culturally responsive educational settings. These concepts include: Whanaungatanga (building relationships), Manaakitanga (ethic of caring), Rangatiratanga (teacher effectiveness), Kotahitanga (ethic of bonding), and Pumanawatanga (school morale, tone, pulse). Bishop et al. (2007) are consistent with Macfarlane’s (2004) model in identifying Manaakitanga and Kotahitanga for improving learning outcomes for Māori learners. According to Macfarlane (2004), p.64, “these concepts do not exist in isolation from each other—more often than not, they coexist or are amalgamated. Since Māori insist on wholeness, this is quite natural.”

According to Macfarlane (2004), whanaungatanga is about building relationships with students, parents, and the community. It refers to the extent to which the school generates opportunities to build mutual respect within their school community. Manaakitanga is the extent to which the school demonstrates a “duty of care” based on kindness and respect. It is not optional but obligatory. Rangatiratanga is about developing teachers’ skills and knowledge in order to develop the mana (integrity and dignity) of Māori students. Kotahitanga refers to the extent to which the school has developed practices and approaches to build a sense of unity and inclusiveness that support a culturally safe environment. Pumanawatanga is in the center of the model and represents whether the school has developed an infrastructure of care and support for students and teachers that is consistent with other four concepts.

Ka Hikitia–Accelerating Success 2013–2017 (Ministry of Education, 2013, p. 9) is the government’s strategy “to rapidly change how education performs so that all Māori students gain the skills, qualifications, and knowledge they need to succeed and to be proud in knowing who they are as Māori.” Ka Hikitia provides an educational framework, which guides all educational stakeholders to enable Māori students reach their full potential.

This study aimed to address a gap in current research on cultural responsiveness by introducing the CRPMS, a self-report instrument designed to measure schools’ culturally responsive practices in relation to Māori students. A practical contribution of this study is to develop a useful analytical tool to inform educators and researchers on whether schools are effectively responding to the specific learning and cultural needs of Māori learners. Such information will be invaluable for education stakeholders and will also be able to contribute to the current debate on whether Māori students achieve better in schools that are more culturally responsive.

Materials and Methods

Item Generation

The process of item generation began by reviewing the literature and empirical studies that discussed models and indicators of cultural responsiveness. To generate the initial set of items for the CRPMS, the NMSSA team worked with three local principals and discussed the nature of cultural responsiveness and current practices at their schools. At this meeting, a total of 34 Likert-type items were crafted under the five concepts. It is important to note here that all the items are about schools’ practices rather than principals’ beliefs.

Following this, the team took this draft to the NMSSA Māori reference group, which is a group of about 10 Māori educationalists from New Zealand including teachers, principals, ministry officials, advisors, and academic researchers who are leading figures in Māori education. The reference group members discussed and re-shaped the items within each concept quite substantially to ensure that the concepts and items validly represented the construct of culturally responsive practices for Māori students. To ensure that the items would be understood by the potential respondents (principals), the revised version was reviewed once again by the Māori reference group and Ministry of Education officials. Then the revised version was piloted with principals from a range of schools. At the end of this process, 27 items were retained for the final version and were measured on a 4-point scale with response options of 1 = not at all like our school, 2 = a little like our school, 3 = moderately like our school, and 4 = very like our school. These 27 items were hypothesized to load on the above-mentioned five factors: Whanaungatanga, Manaakitanga, Rangatiratanga, Kotahitanga, and Pumanawatanga. The items in the CRPMS are shown in Table A1 in Appendix.

Development and Validation of the CRPMS

Sample

Culturally Responsive Practices for Māori Scale items were presented to principals as part of the 2014 NMSSA principal questionnaire. Data were collected from a national sample of 199 school principals. However, 34 principals did not return a questionnaire and two principals did not respond to any of the CRPMS items. No systematic pattern was observed in the types of schools that these principals came from. Thus, responses from 163 schools (80 principals at year 4 and 83 at year 8) principals were used in this study. The distribution of these schools with regard to region and school decile was consistent with the national sample drawn for the 2014 study. The number of students attending these schools ranged from 70 to 1,361 with a mean of school size of 432. The percentage of Māori students in each school ranged from 3 to 97 percent with a mean of 22 percent.

Data Analysis

Data analysis was conducted in two stages. First, the pattern of missing data was investigated using Little’s MCAR test (Little, 1988) and the expectation maximization (EM) algorithm was used to impute missing cases. Second, the factorial structure of the CRPMS scale was assessed using confirmatory factor analysis (CFA) with robust weighted least squares mean and variance adjusted (WLSMV) estimation method. WLSMV is a robust estimation method recommended for ordinal questionnaire data (Flora and Curran, 2004).

As suggested by Noar (2003), several models were examined to compare differing conceptualizations of the factor structure. These included: (a) a null model (model 1) that assumed all of CRPMS factors are unrelated, (b) a one-factor model (model 2) that tested whether CRPMS measured one factor rather than five factors, and (c) a correlated factor model (model 3) that tested whether the five factors of CRPMS are related to one another. Support for model 2 would suggest that the respondents do not differentiate between the cultural responsiveness factors. That is, what is being measured is a unidimensional construct. Support for model 3 would suggest that respondents discriminate between the five factors but the factors are related to each other.

Goodness-of-model fit was assessed using a number of different indices (Cheung and Rensvold, 2002; Fan and Sivo, 2005, 2007) to reflect different aspects of model fit as not all of them are stable under various model conditions. The indices used and their commonly accepted standards of “acceptable” and/or “good” fit included:

• Due to the sensitivity of χ² to sample size (Hu and Bentler, 1998), the ratio of χ² to its degrees of freedom (χ²/df) with values less than 3 being indicative of an acceptable fit (Carmines and Mclver, 1981).

• Root mean square error of approximation (RMSEA) with values less than 0.08 (acceptable) or 0.05 (good).

• Comparative fit index (CFI) and Tucker–Lewis index (TLI) with values greater than 0.90 (acceptable) or 0.95 (good) (Hu and Bentler, 1998; Hair et al., 2010).

• Weighted root mean square residual (WRMR) with acceptable values being close to 1 (Yu, 2002).

Results

Descriptive Statistics

The proportion of overall missing data was 2.04 percent. Little’s MCAR test results for each factor suggested that the data were missing completely at random (p > 0.05), which supported the use of EM imputation. The descriptive statistics are presented in Table 1. The mean scores for all items ranged from 2.40 to 3.84, suggesting that most principals indicated that the statements, on average, were “moderately like our school.” The SDs ranged from 0.40 to 1.15. The lowest mean score was observed for the Rangatiratanga statements (teacher effectiveness) whereas the highest mean score was for the Whanaungatanga statements (building relationships). Examination of the means for each concept indicated that schools had different perceptions of their culturally responsive practices. For example, the mean of teacher effectiveness concept ranged from 1 (not at all like our school) to 4 (very like our school).

TABLE 1

Table 1. Descriptive statistics for culturally responsive practices for Māori scale items.

Confirmatory Factor Analysis

MPlus 7 (Muthén and Muthén, 2012) was used for the CFA of each measurement model and the relationships between latent factors. The goodness-of-fit criteria of alternative models were examined. Table 2 presents the results of the model comparisons.

TABLE 2

Table 2. Confirmatory factor analysis of alternative models.

The CFI, TLI, and WRMR values for model 1 and model 2 did not meet the acceptable fit criteria. Among alternative models tested, a measurement model consistent with Macfarlane’s framework (model 3) provided adequate fit within the recommended values of goodness-of-fit criteria and was retained as the model of best fit. For this model, factor correlations are given in Table 3.

TABLE 3

Table 3. Culturally responsive practices for Māori scale factor correlations.

The correlations between CRPMS factors ranged from 0.52 to 0.90 indicating that the evidence for discriminant validity was not strong. However, the level of relationship amongst factors was consistent with Macfarlane’s theory of “wholeness” appropriate for Māori cultural responsiveness. The highest correlations were found between Kotahitanga and Pumanawatanga, and Kotahitanga and Manaakitanga.

The unstandardized and standardized factor loadings and factor reliabilities are presented in Table 4. The limitations and problems of coefficient alpha (Cronbach, 1951) as a measure of reliability estimate is well established in the literature (Sijtsma, 2009; Teo and Fan, 2013), but this index is still widely used. Therefore, the greatest lower bound (glb) to reliability (Jackson and Agunwamba, 1977; Woodhouse and Jackson, 1977) was calculated and reported as a better estimate of reliability (Ten Berge and Sočan, 2004; Sijtsma, 2009). Greatest lower bound estimates were computed by means of the FACTOR program (Lorenzo-Seva and Ferrando, 2006), which can be downloaded from http://psico.fcep.urv.es/utilitats/factor/Download.html. Sijtsma (2009), however, states that alpha should also be used alongside glb as a better practice for reporting reliability (p. 118). This is the procedure followed in this study.

TABLE 4

Table 4. Unstandardized and standardized factor loadings and reliabilities.

All item parameter estimates were statistically significant at p < 0.001 level. Factor loadings ranged from 0.35 to 0.86, which provided support for convergent validity evidence. Two of the coefficient alpha reliability estimates were slightly less than the recommended level (0.70) for instrument development. Given the small number of items for those factors; however, these values were considered acceptable. On the other hand, all glb estimates were greater than the 0.70 threshold. The CFA analyses, factor correlations, and reliabilities provided support for the factorial and construct validity evidence of the CRPMS.

Discussion and Conclusion

The aim of this study was to provide validity evidence on the internal structure of the CRPMS self-report instrument designed by the NMSSA team to measure the degree to which New Zealand English-speaking primary schools were culturally responsive to Māori students. The present study adds to the culturally responsive education research literature specifically by providing empirical evidence via CFA supporting the factorial structure of the CRPMS comprising five factors: Whanaungatanga (building relationships), Manaakitanga (ethic of caring), Rangatiratanga (teacher effectiveness), Kotahitanga (ethic of bonding), and Pumanawatanga (school morale, tone, pulse). In general, the validity evidence findings from this study support the utility of CRPMS as a measure of cultural responsiveness for mainstream New Zealand schools.

In line with the previous findings of Meyer et al. (2010), we found that school principals had different perceptions/interpretations of their schools’ culturally responsive practices. The range of school means for CRPMS constructs indicated that there was some variability between schools with respect to cultural responsiveness practices. In general, however, the overall scale means revealed that most principals responded as either or “moderately like our school” or “very like our school.”

It is important to note that this is a bold project that spans a significant breadth of literature. What becomes difficult to preserve is the rich conceptual depth of tea o Māori/Māori worldview, which may be diluted by attempts at analytical specificity. The moderate-to-high correlations among the five factors of the CRPMS suggest that these factors tapped distinct but interrelated aspects of the same construct, which is consistent with Macfarlane’s (Macfarlane, 2004) theory.

This study supports Sleeter’s (Sleeter, 2012), p. 578, claims that “small-scale case studies illustrate what is possible, but we also need research on the impact of scaled-up work in culturally responsive pedagogy, including research showing how teachers can learn to use it in their classrooms.” This study has several important implications. First, gaining a better understanding of schools’ culturally responsive practices will increase our understanding of its nature and its relationship to student learning outcomes. Such knowledge has relevance for educators and researchers in bilingual, multilingual, and multicultural societies. Second, this CRPMS instrument may be used in a school’s self-review by principals and teachers rating their schools’ relative strengths and weaknesses in their responsiveness to Māori students and identifying areas that need to be addressed. This is particularly important given that the ERO (Education Review Office, 2010, p. 1) reported that “not all educators have yet recognized their professional responsibility to provide a learning environment that promotes success for Māori students.” Third, CRPMS outcomes may provide empirical evidence that can guide school policy decisions, which may lead to more satisfaction with school by Māori students as Māori.

Further replication and validation studies with the CRPMS is needed to provide further evidence for its validity in different settings. Research is also necessary to investigate the relationships between the CRPMS and significant student outcomes, and how this measure can be used with teachers who have the potential to change the educational outcomes of Māori students (Hattie, 2008). We believe that further replication and validation studies are needed to make sure that the instrument works well in different settings. Although developed for Māori cultural responsiveness, this model may be able to be adapted for other ethnic groups reflecting their cultural perspectives.

Cultural responsiveness in an important component of the ERO’s school evaluation indicators. As part of their review process, ERO evaluates schools and makes recommendations regarding their responsiveness to the needs of Māori students. Further studies may use ERO reports to CRPMS results.

One limitation of the study is the marginal sample size available in this study. As mentioned in the literature (Fan and Sivo, 2007), various fit indices and parameter estimates are sensitive to sample size. The other limitation pertains to a possible response bias that may exist in principals’ responses. Principals may have provided socially desirable responses. Future research may consider collecting cultural responsiveness data by means of independent ratings (e.g., ERO) or multiple ratings from the school community, such as teachers, students, whanau, and/or iwi rather than relying only on principals’ self-reporting.

Ethics Statement

This study was approved by the “University of Otago Human Ethics Committee.”

Author Contributions

CRPMS was developed by the National Monitoring Study of Student Achievement (NMSSA) team. The author designed this validation study, developed the methodology, performed the analysis, and wrote the manuscript. The author agrees to be accountable for all aspects of the work.

Conflict of Interest Statement

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer, JS, declared a shared affiliation, though no other collaboration, with the author to the handling editor, who ensured that the process nevertheless met the standards of a fair and objective review.

Acknowledgments

The author wish to acknowledge the support and contributions of Alison Gilmore, Denise Quinlan, and NMSSA project team and thank all principals, teachers, and students who participated in the 2014 NMSSA study.

Funding

The National Monitoring Study of Student Achievement (NMSSA) project is funded by the Ministry of Education, New Zealand.

References

Allan, R. (2012). National monitoring study of student achievement: wanangatia te putanga tauira. Set: Res. Info. Teach. 3, 78–80.