Assessment through a cross-cultural lens in North American higher education

Mortaz Hejri, Sara; Ivan, Rodica; Jama, Nimo

doi:10.3389/feduc.2022.1012722

OPINION article

Front. Educ., 20 October 2022

Sec. Language, Culture and Diversity

Volume 7 - 2022 | https://doi.org/10.3389/feduc.2022.1012722

This article is part of the Research Topic Teaching and Learning in a Global Cultural Context View all 5 articles

Assessment through a cross-cultural lens in North American higher education

$\nSara Mortaz Hejri$ Sara Mortaz Hejri

Rodica Ivan^*

Nimo Jama

Altus Assessments, Toronto, ON, Canada

Introduction

Cross-cultural perspectives are of paramount importance for educational institutions in increasingly diverse communities. Although this notion has been discussed extensively in teaching and learning contexts (e.g., Sugahara and Boland, 2010; Gay, 2013; Cortina et al., 2017), it has often been overlooked when assessing learners (Solano-Flores, 2019). For that reason, in this paper, we focus on the intersection of assessments and cross-cultural perspectives and review three of its interrelated facets. For reasons of space, we restrict our discussion to the context of higher education in North America, but these findings may also inform other culturally diverse educational environments. First, we discuss cross-cultural assessments and highlight some of the challenges they face. Then, we turn to the development of culturally sensitive assessments and present some strategies that can be used for this purpose. Finally, we shift toward discussing how cross-cultural perspectives have been incorporated into competency-based education and assessments as cultural competence, which professionals need to demonstrate alongside specialized knowledge and practical skills.

Challenges to cross-cultural assessments

Cross-cultural assessments generally refer to the use of standardized tests for culturally and linguistically diverse populations (Ortiz and Lella, 2005). A starting point for cross-cultural assessments was the language classroom, where individuals would have to learn English, for instance, as well as adapt to Anglo-Saxon and/or North American culture (Upshur, 1966; Phillipson, 1992). However, any assessment administered to students from diverse cultural backgrounds, is in effect a cross-cultural assessment (Lyons et al., 2021). Thus, assessments are required to be sensitive to cultural differences¹ and free from cultural bias [American Psychological Association (APA), (2002); Solano-Flores, 2019]. While a crucial goal to strive toward, the process of achieving a cross-cultural assessment faces a number of challenges.

One difficulty cross-cultural assessments face is the very definition of culture, which is by no means fixed (Lang, 1997). An individual's cultural background is not merely a matter of race or language, but at the intersection of heritage, language, beliefs, knowledge, behavior, common experience, and self-identity (gender, sexual orientation, etc.) (Ortiz and Lella, 2005; Montenegro and Jankowski, 2017). A culturally sensitive assessment must consider all the aspects that make up cultural diversity, as well as their complex interactions.

An additional challenge is that, like all human artifacts, assessments are affected by the cultural background of their developers (Cole, 1999; Solano-Flores, 2019). For instance, tests developed in North America or in the UK will invariably be imbued with content that reflects mainstream North American / Western European values (Phillipson, 1992; Ortiz and Lella, 2005), which cater to White Western conceptions of learning and assessments rather than to cross-cultural pedagogies and ways of knowing (Graham, 2020). Individuals who do not adhere to mainstream views, learning strategies, and life experiences are likely to be disadvantaged by such assessments. Not only does socioeconomic privilege lead to higher scores for racial majority applicants (Smith and Reeves, 2020; Whitcomb et al., 2021), but students from different cultural backgrounds have also been shown to prefer different methods of learning (Oxford, 1996; Hong-Nam and Leavell, 2007; Sugahara and Boland, 2010; Arbuthnot, 2020; Habók et al., 2021). In this sense, it is unlikely for any standardized test to be devoid of demographic group differences (Ortiz and Lella, 2005; Lyons et al., 2021), either when comparing applicants from different countries, or different demographic subgroups within the same country.

Societal inequities reflected in assessment scores

Assessments tend to reflect existing systemic inequities, To illustrate, we analyzed publicly available aggregate data for a standardized test commonly used in graduate school admissions in the US, namely the Graduate Record Examination (GRE) General Test. Here we focus on scores from the 2020 to 2021 application cycle (GRE Snapshot, 2022) for each of the three sections of the GRE test: Verbal Reasoning, Quantitative Reasoning, and Analytical Writing. We investigated race and citizenship, as two major demographic variables for which data was available². We selected the three racial subgroups with the largest sample sizes in the GRE Snapshot report (2022): Asian, Black, and White, and two major citizenship subgroups: US citizens and non-US citizens. We compared subgroup average scores using pairwise t-tests and reported the results of our analysis in Cohen's D effect size estimates (Cohen, 1992; Lakens, 2014) alongside descriptive statistics from the GRE Snapshot report (2022) in Table 1. Pairwise comparisons revealed moderate to large differences for all GRE sections between White and Black applicants, as well as between Black and Asian applicants; the Black applicant subgroup was associated with lower average scores. Additionally, large differences were found between US citizens and non-US citizens for Quantitative Reasoning and Analytical Writing; non-US citizens scored higher for Quantitative Reasoning and lower for Analytical Writing, on average. These results highlight significant demographic differences between racial minority and majority subgroups within the US, as well as between subgroups with different US citizenship statuses. Although the analysis was performed on individual demographic variables (as opposed to the intersection of multiple demographic variables, such as race and income), these differences are nevertheless reflective of the difficulty of designing a culturally sensitive assessment for all applicant populations, irrespective of race, nationality, or cultural background.

TABLE 1

Table 1. Descriptive statistics and Cohen's d estimates for GRE 2020–2021 scores for subgroup variables of interest.

Steps toward mitigating inequities in cross-cultural assessments

Given the challenges discussed in the sections above, for an assessment to be designed through a cross-cultural lens, several measures are recommended.

A crucial component is inviting diverse voices at all stages of assessment development: when creating the assessment, determining its efficacy, and interpreting its results (Lyons et al., 2021). This is achieved by building multidisciplinary teams of professionals from diverse backgrounds and identities and considering multiple perspectives when developing and rating an assessment.

While diversifying the cultural make-up of assessment teams is one potential strategy, research has shown that including items which allow students to connect the content to their lived experiences leads to improved performance (Solano-Flores and Nelson-Barber, 2001, Mislevy and Oliveri, 2019). Additionally, student feedback can be used to inform the suitability of specific measures within assessments (Montenegro and Jankowski, 2020), such as item phrasing (i.e., how pieces of specific content might be phrased for each question). Actively seeking students' feedback and incorporating their perspectives can be done in a variety of ways. For instance, a study conducted with the Centre for Global Programs and studies at Wake Forest University (Brocato et al., 2021) convened a student advisory board with students from a range of backgrounds to gather data on their perceptions of “culture” via semi-structured interviews.

Another critical aspect of designing culturally sensitive assessments is creating opportunities for meaningful student contribution to their own assessment and inviting them to showcase their strengths and display their learning outside of standardized testing. An example includes the assessments carried out longitudinally throughout the 4-year degree program at Portland State University via electronic portfolios (Carpenter et al., 2020). For these portfolios, students would submit assignments from the course that showed evidence of their learning as well as reflections on their progress during the academic year. In turn, faculty would provide feedback on the portfolios while collaboratively reviewing student work within the context of the course.

In order for an assessment to be culturally sensitive, contextual and structural factors must also be considered. Systemic issues related to culture, bias, power, and oppression influence society as a whole, including the institutions which carry out education and assessments. Such factors are also reflected in institutional norms and resource constraints, which may affect the interpretations of student learning outcomes. Thus, the ultimate objective is to understand not only how students are performing, but to also explore the underlying structures that students perform in and those that affect their learning (Montenegro and Jankowski, 2020). One US based example of an organizational effort to achieve this goal is the National Institute for Learning Outcomes Assessment (NILOA), who support the creation and use of culturally responsive assessments which take into account students' needs and the context in which the assessment takes place. Aiming to foster equitable outcomes, NILOA also conducts case studies with various institutions, such as those mentioned above (Carpenter et al., 2020; Brocato et al., 2021).

Assessment of cultural competence

In addition to investing in designing culturally sensitive assessments, training programs can also focus on developing cultural competence³ in students, i.e., the ability to effectively interact with people of various cultural backgrounds (DeAngelis, 2015). Aligning with increasingly diverse populations (Mills, 2016), cultural competence has been introduced as an important skill set among professionals that enables them to work effectively across cultural boundaries (Office of Minority Health (OMH), 2000; The Royal Australasian College of Physicians (RACP), 2018; Chun and Jackson, 2021). The notion of cultural competence is currently adapted across different fields; yet, the initial idea seems to be rooted in the healthcare system (Cross et al., 1989; Frawley et al., 2020). Healthcare has become increasingly culturally diverse over the last decades, and clinicians are expected to demonstrate cultural awareness, sensitivity, and competence as they encounter patients with a variety of perspectives, beliefs, and behaviors (Betancourt, 2003; Elminowski, 2015). In the 2000s, medical organizations and accrediting bodies, such as the Association of American Medical Colleges (AAMC) and the Liaison Committee on Medical Education (LCME), brought together experts to develop new standards regarding cultural competence. Since then, a large number of training programs have been designed and delivered for health professions trainees to foster the development of knowledge, skills, and attitudes required to care for culturally diverse patients (Gozu et al., 2007).

Despite the interest toward cultural competence training (Gozu et al., 2007; Vasquez Guzman et al., 2021), the assessment of cultural competence has remained one of the main challenges (Blue Bird Jernigan, 2016). One issue concerns the risk of including test content based on societal stereotypes (Campinha-Bacote, 2018). In addition, most instruments used for measuring cultural competence within health professions education have not been rigorously validated (Gozu et al., 2007), and the measurement of knowledge has been overemphasized (Blue Bird Jernigan, 2016). Researchers have also found it difficult to determine if a student is truly culturally competent either by observing their performance in simulated settings (Chun, 2010), or by administering attitude surveys (Gozu et al., 2007).

An additional point of contention is whether the assessment of cultural competency and professionalism overlap (Chun, 2010). While some view these two as independent concepts, others argue that there is no need for a separate measurement of cultural competence (Chun, 2010). One example of specialized assessments for cultural competence is the Tool for Assessing Cultural Competence Training (TACCT) (Lie et al., 2008). Although initially developed for curriculum development, TACCT also provides a guide on the assessment of cultural competency. Alternatively, collective tools that assess several aspects of professional performance, such as situational judgment tests (SJTs), could be used to measure cultural competence as an integrated part of a broader construct (i.e., professionalism). Such SJTs could highlight culture as one aspect of doctor-patient interactions, and also provide a significant practical contribution as an alternative performance-based assessment tool across different fields, including higher education, management, military, and engineering (Biga, 2007; Rockstuhl et al., 2015; Reinerman-Jones et al., 2016; Jesiek et al., 2020). Given the nuanced complexity of cultural competence and its various elements, one tool might not cover all aspects; however, SJTs provide an opportunity for students to express the rationale behind their behaviors and decisions. Additionally, open-response format SJTs, as opposed to closed-response format, also allow students to connect their answers to their lived experiences, and thus, allow raters to gain a deeper understanding of students' perspectives when assessing their responses.

Summary

We identified three focal points for developers of cross-cultural assessments that intend to be sensitive to individuals' diverse cultural perspectives. Although our source was North American higher education, these insights can extend to multicultural environments more broadly. We highlighted how cultural backgrounds and societal privilege are reflected in assessment scores and reflected on the difficulty of designing a culturally sensitive assessment. Then, we discussed how the inclusion of multiple perspectives at different stages in the assessment process can help alleviate differences in performance between students from different cultures. These issues also point to the need for moving beyond designing culturally sensitive assessments and toward also incorporating measures of students' cultural competence. Fostering the ability to work effectively across diverse communities and cultures is a prerequisite toward achieving a more equitable and inclusive society.

Author contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

Acknowledgments

We are grateful to the research team at Altus Assessments for all of their insightful feedback and valuable suggestions on the earlier versions of this manuscript.

Conflict of interest

Authors SH, RI, and NJ were employed by Altus Assessments.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^Various terminology is used to reference this concept: individuals and tools must be culturally “sensitive”, “informed”, “responsive”, “aware”, etc. While these terms come with different shades of meaning, they greatly overlap in their use (Montenegro and Jankowski, 2017; Frawley et al., 2020; Vasquez Guzman et al., 2021; a.m.o).

2. ^The three race subcategories are broadly used in research on demographic differences in assessments. We have also selected US citizenship status to showcase the disadvantages faced by test takers who are not native to the country where the assessment is designed. As mentioned above, these are only two aspects of cultural background, and there are many different other aspects (e.g., gender, socioeconomic status, etc.) and their complex interactions which we do not discuss here due to space constraints and unavailability of data.

3. ^In this paper we employ the term “cultural competence” due to its wide usage in the literature. More recently, however, this term has raised some concerns, given that becoming fully competent in other cultures is next to impossible (Chun, 2010; Blue Bird Jernigan, 2016). Instead, some alternative and complementary approaches have been introduced that could offer real potential to mitigate biases and create structural changes. This includes, but is not limited to the notion of cultural humility (which encourages lifelong commitment to reflective practices and continuous learning) and the notion of structural competency (which promotes efforts aiming to eliminate racial and ethnic disparities in the healthcare system).

References

American Psychological Association (APA) (2002). Ethical principles of psychologists and code of conduct. Am. Psychol. 57, 1060–1073. doi: 10.1037/0003-066X.57.12.1060