Development and validation of a self-perception instrument for critical media literacy in Chilean pre-service teachers

Introduction: The influence of mass media and social media has profoundly impacted the way individuals access, interpret, and engage with information. It is becoming increasingly important to gain an understanding of media literacy practices, particularly among those who influence children and young people. In Spanish, there is little knowledge about critical literacy practices related to media content. This study aims to develop and validate a self-perception instrument designed to measure critical media literacy practices among Chilean pre-service teachers.

Method: A structured written questionnaire with closed-ended questions was designed and validated. The instrument encompasses three dimensions: situating discourse within sociocultural contexts, recognizing and participating in discursive practices, and assessing the impact of media on the community and the individual. It comprises 28 first-person statements, rated on a five-point Likert scale.

Results: The content was validated by a panel of experts in education, linguistics, and journalism in two phases, resulting in a high inter-rater agreement (κ = 0.78) and a content validity coefficient (CVC) of 0.97. A pilot study was conducted with 92 undergraduate students enrolled in teacher education programs. Exploratory factor analysis confirmed a three-factor structure that aligns with the proposed dimensions, explaining 51% of the total variance. Internal consistency analyses showed strong reliability, with a Cronbach’s alpha of 0.84 and McDonald’s omega of 0.85.

Discussion: The results indicate that the instrument is valid and reliable for assessing self-perceived critical media literacy practices. The findings underscore the need for educational interventions to enhance students’ ability to critically engage with media content.

1 Introduction

The current era is characterized by the pervasive accessibility of information and the simplicity of expressing opinions through mass media, which have given rise to novel reading and writing practices. The media, and social networks, provide a plethora of discourses whose objective may be to inform, express one’s view, or even misinform. Frequently, the distinction between these objectives is not readily apparent. The influence of mass media has been amplified by the advent of continually evolving technologies, which raised questions about credibility, and about the need for education in media and information (Buckingham, 2019).

Therefore, developing critical literacy competencies becomes a necessity. Buckingham (2020) states that educators need to teach about media and technology, “asking critical questions about how they represent the world, and how they create meanings” and “engage much more directly and systematically with how students are using media and technology outside the classroom” (p. 235). Cassany (2020) posits that education should prioritize cultivating citizens proficient in reading, writing, and critical thinking, capable of making a constructive contribution to the development of inclusive, tolerant, and advanced societies.

Critical literacy has been the subject of study from several different perspectives in Hispano-American context, with consideration given to a variety of discursive genres and sociocultural contexts. From a didactic perspective, a few proposals have been put forth. For instance, in the context of higher education, Pinchao (2020) studied the use of multiple sources for complex reading tasks. The studies by Romano and López (2016) and Serrano de Moreno (2008) proposed the implementation of strategies for facilitating experiential learning, with the objective of fostering the development of critical reading skills among undergraduate students. In the high school context, a critical media literacy program was designed, implemented and evaluated by Mesquita et al. (2022).

There is a relative paucity of research examining the self-perception of critical literacy in Spanish. Thera have been studies on students’ perceptions of critical reading as a learning tool (Gómez López et al., 2016), as well as the self-perceived critical reading skills of advanced students of Educational Sciences (Díaz et al., 2016). In terms of digital competence, García-Vandewalle et al. (2023) explored teachers’ level through a self-assessment questionnaire based on the Spanish adaptation of the European Framework for Digital Competence of Educators, while Monzón (2019) analyzed teachers’ self-perceived performance of digital literacy in the classroom. Pérez-Escoda et al. (2021) assessed media, social media consumption, and the phenomenon of fake news among university students in Spain using a self-report questionnaire adapted from the report Media Use in the European Union and Digital News Report.es 2020.

A greater number of studies have been conducted in the English-speaking context. In the United States, the level of media literacy instruction in schools has been determined through the Media Literacy Implementation (MLI) scale, which assesses the extent to which media literacy learning activities are integrated into the curriculum at both the school and community levels (Hobbs et al., 2022). In university-level, Hallaq (2016) conducted research to assess the digital online media literacy of students through the development of a quantitative survey.

In the United Kingdom, the Office of Communication (OFCOM) is responsible for the promotion of media literacy and the facilitation of research activities, which include the administration of quantitative surveys to both UK adults and children, as well as their parents (Rumble, 2011). Since 2003, this organization has conducted extensive research into the media use and attitudes of both adults and children in the UK, with a particular focus on critical evaluation skills and online knowledge and understanding among internet users (OFCOM, 2024).

However, no instrument in Spanish has yet been identified which assesses self-perceived critical literacy regarding media content.

Given the vital role that educators play in fostering media literacy in students of all ages, with this competence spanning multiple domains, our primary focus is on those undergoing training to become teachers. Therefore, to ascertain whether student teachers are critical readers of media discourses, it is necessary to ask: What dimensions should be included in an instrument to explore student teachers’ self-perceptions of their media literacy practices? To answer this question, we conducted research whose objective was to validate an instrument for assessing student teachers’ media literacy practices by examining their self-perceptions.

2 Theoretical framework

Critical reading implies an active reader who is willing to reach the deep meaning of the text, its underlying ideas and implicit ideology to question the text, which requires an understanding of the topic, experience, interest and expectations regarding the read text (Serrano de Moreno, 2008). The development of critical reading practices requires subjects to “read with a critical mind and a critical eye to understand, interpret, identify, accept, or reject written material” (Olifant et al., 2020). According to the study by Henry and Mohamad (2021), the development and constant practice of critical reading leads to greater interest in reading and deeper knowledge provided by texts. Thus, good readers are able to evaluate their reading practices as they develop their critical reading skills.

From a pedagogical standpoint, a distinction can be made between critical reading and critical literacy. Luke (2014) acknowledges that critical reading is a systematic approach to interpreting a text’s multiple meanings, discerning the author’s perspective, and evaluating the veracity and truthfulness of a text. However, there is little recognition that texts engage particular cultural and political positions. Critical literacy incorporates ideological and political elements into the analysis of texts, and as a practical approach, “it melds social, political and cultural debate and discussion with the analysis of how texts and discourses work, where, with what consequences, and in whose interests” (Luke, 2014, p. 21).

Similarly, Andreotti (2014) demonstrates the progression from traditional reading to critical literacy. In the context of traditional reading, the process of decoding is a primary concern, focusing on the construction of the macrostructure. In critical reading, the reader would delve deeper into the context and sociopolitical context. A critical literacy approach would extend this analysis further, examining the production of knowledge and power relations.

Weng (2023) defines critical literacy as a sociocultural perspective on language that conceptualizes language learning as a dynamic process of critical awareness and action. Reading, in this view, entails a critical examination of linguistic texts and multimodal resources.

Cassany (2006) 2012 presents a comprehensive framework for critical literacy within the context of the Spanish-speaking community engaged in research on reading and writing. It is based on multiple sources, including Paulo Freire’s critical pedagogy, Critical Discourse Analysis, and New Literacy Studies. The author outlines that critical literacy seeks to comprehend the intricacies of human, psychological, and social aspects through a multifaceted, dynamic, adaptable, and evolving process, empowering individuals to engage in reasoned discourse.

According to Cassany (2006) 2012 and Cassany and Castellà (2010), critical literacy involves the following skills and actions:

• Situate the discourse in its original sociocultural context, which includes recognizing the context of the text produced, characterizing the author and identifying his or her purpose, identifying included and omitted voices, and identifying ideological positions.

• Recognize and participate in the discursive practice, interpreting the text according to its discursive genre and recognizing the sociocultural characteristics of the genre.

• Calculate the effects of a discourse on the community and oneself, i.e. the reader becomes aware of his situationality, his reading context, the individuality of his interpretation, and calculates the interpretations of others, adding diverse interpretations to his/her own.

Thus, Cassany’s framework (Cassany, 2006, 2012; Cassany and Castellà, 2010) highlights the interaction between reader, text and context, establishing as a starting point the importance of the socio-cultural environment of production and reading, without neglecting cognitive and metacognitive factors in the development of reading practices.

Critical literacy and critical thinking are closely related and fundamental to effective text comprehension. Weng (2023) postulates that critical literacy is a more expansive concept than critical thinking because “critical thinking is often related to the higher order cognitive thinking skill required for reading comprehension, critical literacy goes beyond higher-order cognitive thinking and is committed to the sociological and political orientation of literacy skills” (p. 199). For Bobkina and Stefanova (2016), critical thinking is a type of literacy because it involves interpreting and understanding different points of view in the construction of meaning.

According to Andrews and Leonard (2023), the development of critical thinking skills improves the quality and fluency of comprehension. This is because critical literacy involves analyzing evidence and using the logic of induction and deduction to understand texts, as suggested by Paige et al. (2024).

In the digital age, Hernández y Hernández et al. (2018) emphasize the importance of adapting literacy practices to new technological environments. They note that it is not entirely clear how students use digital technologies; what they write, what they read, what they learn, what they search for and how they organize information, aspects that could provide an empirical basis for generating pedagogical actions. Thus, the concept of media literacy emerges.

Media literacy is defined by Buckingham (2009) as “a skill or a form of competency; but it is also about critical thinking, and about cultural dispositions or tastes (…) about old media and new media, about books and mobile phones” (p. 4). Potter (2004) describes it as “a set of perspectives that we actively use to expose ourselves to the mass media to process and interpret the meaning of the messages we encounter” (p. 58). Livingstone (2004) affirms that media literacy is “the ability to access, analyze, evaluate and create messages across a variety of contexts” (p. 18), understanding media as “pathways of communication that are used to reach, inform, entertain, or influence people on a wide scale” (De Abreu, 2019: 25). This includes, but is not limited to, television, radio, newspapers, magazines, advertising, movies, videos, websites, and social media.

The media literacy promotes a critical analysis of media content, with a particular focus on four key concepts: language, representation, production, and audiences (Buckingham, 2020). The study of media language examines the ways in which media create meaning, including the use of linguistic and audio-visual codes that extend beyond the conventional understanding of the internet as a platform for spontaneous expression. Representation can be defined as the form in which media claims to represent reality, extending beyond the view of digital media as merely “information” technologies. The concept of production points to who makes the medium (individuals or large corporations), and how it is distributed and regulated. Finally, in exploring audiences, the questions that arise are who uses this media, how they use them and why, reflecting on how the use of digital and social media has social, ethical and psychological consequences.

Media literacy education is essential in developing critical skills for judging information from the media and promoting comprehension skills (Zhong, 2015). De Abreu (2019) stresses that media literacy learning environments provide opportunities for students to discuss many media topics, analyze media messages, and critically examine the role that media plays in their lives. Analysis involves “deconstructing messages, detecting bias and propaganda, and evaluating how a wide range of media messages portray or construct reality for the viewer” (De Abreu, 2019, p. 32).

The proliferation of media, particularly social media, is accompanied by a number of potential risks. One such risk is the dissemination of misinformation and the harmful consequences of fake news. Also, readers have greater control over the media and sources to which they are exposed, leading to the formation of a “filter bubble.” Consequently, as Buckingham (2020) states, rather than facilitating deliberation and informed debate, social media are in fact contributing to their decline.

The pervasive presence of social media in the lives of younger individuals represents an unprecedented social phenomenon (Pérez-Escoda et al., 2021). Media literacy education, then, responds to the new challenges to the assessment of reliability and credibility of sources that online media has introduced (Buckingham, 2019, 2020). It is crucial to equip digital users with the skills needed to distinguish fact from falsehood, with the assumption that those with greater media literacy tend to consume false or dubious stories in a more critical manner, thereby mitigating the impact of fake news on society (Jones-Jang et al., 2021). Nevertheless, Buckingham (2020) points out that “media education is not enough on its own. It may be part of the solution to the problem, but we also need media regulation and reform” (p. 238).

3 Methods

In order to gain insight into the critical literacy practices of media discourse among university students, a survey was conducted. A structured written questionnaire with closed-ended questions was designed and validated (Quispe Limaylla and Sánchez Moreno, 2011). The procedure for the design of the instrument is described below.

3.1 Definition of the construct of interest

We defined critical media literacy as the construct of interest. Three dimensions of critical literacy framework (Cassany, 2006, Cassany and Castellà, 2010) were considered: (1) situating discourse in socio-cultural context, (2) recognizing and participating in discursive practice, and (3) calculating the effects of discourse on the community and on oneself. This framework is consistent with the principles of media literacy. Buckingham (2020) advocates for a critical examination of media content, emphasizing four core concepts: language, representation, production, and audiences (Buckingham, 2020). The concepts of media language and representation are closely tied to the discursive practices of mass media, while production is inextricably linked to the context of mass media discourse. Audiences, on the other hand, are shaped by the impact of discourse on communities and readers.

It is anticipated that the self-perception instrument will elucidate students’ practices when they engage with mass media texts.

3.2 Questionnaire development

A questionnaire was constructed with 28 affirmative statements written in the first person. Participants rated their level of agreement with each statement using a five-point Likert scale (strongly agree, agree, neither agree nor disagree, disagree, and strongly disagree). Each item is designed to identify readers’ practices regarding information and opinions presented in the mass media. The questionnaire is available in its original Spanish version in Supplementary Appendix 1 and in an English version in Supplementary Appendix 2.

The items are grouped into three dimensions according to the aforementioned critical literacy framework and media literacy critical examination. Seven statements correspond to dimension 1 (situating the discourse in the socio-cultural context), thirteen to dimension 2 (recognizing and participating in the discursive practice), and eight to dimension 3 (calculating the effects of a discourse on the community and oneself).

3.3 Content validation of the questionnaire

Content validation was conducted through expert judgement involving specialists in education, linguistics and journalism. The experts were selected based on their knowledge and experience with the literacy process and genres of mass media.

In the first phase, the relevance of the items was evaluated according to the defined dimensions. Nine experts participated in the evaluation process, responding to a checklist indicating whether each item was deemed appropriate or measured the defined dimension. If the response was negative, the evaluator provided a justification, including a comment or suggestion for improving the statement.

Fleiss’ kappa coefficient, a statistical measure used to assess inter-judge reliability in studies with nominal variables and more than two raters, was used to measure the degree of agreement among the raters. Unlike Cohen’s kappa, which is only appropriate for two raters, Fleiss’ kappa extends this analysis to any number of raters. Each rater assigns a category to each item, and the resulting matrix is used to calculate the frequency of each category per item. Fleiss’ kappa adjusts the observed proportion of agreement to account for expected agreement by chance, providing a more robust measure of inter-rater reliability.

Fleiss’ kappa is calculated considering the observed agreement proportion (Po) and the expected agreement proportion by chance (Pe), with the formula:

$\mathrm{k}={\displaystyle \frac{\mathrm{Po}-\mathrm{Pe}}{1-\mathrm{Pe}}}$

where:

Po: observed agreement percentage obtained from the average observed concordance for each item.

Pe: expected proportion of agreement at chance, calculated from the marginal distribution of categories.

In the second phase, after the adjustments made by the agreement of the judges in the first phase, the content validity coefficient (CVC) was verified (Hernández-Nieto, 2002), which allowed to evaluate the degree of agreement among the experts (the author recommends involving three to five experts) regarding each item and the instrument in general. For this instrument, the expert judges’ evaluations considered the criteria of sufficiency, coherence, relevance and clarity (Escobar-Pérez and Cuervo-Martínez, 2008). The mean obtained for each item is used to calculate the CVC for each item:

$\mathrm{CVC}{}_1=\mathrm{M}{}_{\mathrm{x}}/{\mathrm{V}}_{\max }$

where:

Mx: measure of the element in judges’ scoring.

V_max: maximum score that the item can reach.

Additionally, the error assigned to each item (Pe₁) must be calculated to reduce potential bias introduced by judges. This is obtained from:

${\text{Pe}}_{\text{1}}\text{ = }{\left(\text{1/j}\right)}^{\text{j}}$

where:

J: number of judges.

Finally, the content validity coefficient is calculated by applying:

${\text{CVC = CVC}}_1-{\text{ Pe}}_1$

Regarding the results of this coefficient, the authors recommend retaining items with values above CVC = 0.80 (Pedrosa et al., 2014). Based on received observations, the instrument was adjusted for the last time before the pilot test.

3.4 Sample

The population of this study was undergraduate students in teacher education programs. We concentrated on pre-service teachers for two reasons. First, they are important role models for the younger generation. Second, they can play an influential part in developing media literacy in the classroom.

A propositional sample of 92 university students between the ages of 18 and 22 was surveyed using a questionnaire. The sample included 68 women, 19 men, and 5 individuals who did not identify with a gender. The participants were freshmen enrolled in three teaching programs: Language and Communication (Spanish as a first language), Music Education, and Special Education.

3.5 Analysis of internal validity and instrument consistency

Cronbach’s alpha and McDonald’s omega statistics were used to determine the internal consistency of the instrument. Cronbach’s alpha, one of the most used measures for analyzing the internal consistency of an instrument, is useful for evaluating the reliability of a set of items in a questionnaire or test. It indicates how closely related the items are as a group. It is calculated using the variance of each item and the total test variance, providing an estimate of the proportion of the total test variance that is attributable to the common variance among the items (Pedrosa et al., 2014).

In contrast, McDonald’s omega reliability measure also assesses the internal consistency of a set of items in a scale. However, unlike Cronbach’s alpha, omega does not assume that all items contribute equally to the scale construction and can therefore provide a more accurate estimate in the presence of multiple factors. While Cronbach’s alpha may underestimate reliability, Omega provides a more robust and less tau-equivalence dependent assessment (Hayes and Coutts, 2020).

3.6 Analysis of construct validity

To assess the construct validity of the questionnaire, an exploratory factor analysis (EFA) was conducted using SPSS 29. The suitability of the data for factor analysis was assessed using the Kaiser-Meyer-Olkin (KMO) measure and Bartlett’s test of sphericity. A KMO value greater than 0.6 and a significant Bartlett’s test (p < 0.05) indicated adequate conditions for EFA. Equamax rotation was used to achieve a clearer factor structure, promoting orthogonality among factors. The rotated factor solution was examined for interpretability.

4 Results

4.1 Expert judgment

As indicated in the methodology section, the expert judgment was configured in two phases.

In the first phase, a panel of nine experts in the fields of education, linguistics, and journalism evaluated the relevance of the proposed items on three dimensions present in the theoretical framework for measuring critical media literacy practices. The results are displayed in Table 1.

TABLE 1

Table 1. Item relevance evaluation (phase 1).

The objective of the evaluation was to ascertain the relevance of the items in question in relation to each theoretical dimension. Each judge assigned a category (either “yes” or “no”) to each item, indicating whether the item was relevant for measuring a dimension. The resulting matrix was then used to calculate the frequency of each category per item. To evaluate the agreement among the experts, the Fleiss kappa coefficient was employed, which is appropriate for categorical variables. This coefficient adjusts the proportion to account for expected chance agreement, thereby providing a more robust measure of inter-judge reliability.

A κ close to 1 indicates a high degree of inter-rater agreement, while a value close to 0 indicates that agreement is no better than chance. Intermediate values are interpreted according to the guidelines of Landis and Koch (1977), which categorize the degree of agreement from poor to almost perfect. For this instrument, the value κ = 0.78, indicating a high level of agreement among expert judges.

In addition, the coefficient was calculated for each dimension, taking into account the adjustment process of the instrument, since if the item evaluated was not relevant, each judge provided a comment to improve or modify the item, allowing the reformulation of items with lower agreement among experts and with more comments. Table 2 illustrates the adjustments applied to the items in accordance with expert judgement.

TABLE 2

Table 2. Example of items adjustment.

In the second phase, four of the nine experts who participated in the first phase were convened to evaluate the items according to the criteria of sufficiency, coherence, relevance, and clarity (Escobar-Pérez and Cuervo-Martínez, 2008). A Likert scale was employed for this purpose.

Through the calculation of the content validity coefficient (CVC) (Hernández-Nieto, 2002), it was found that the degree of agreement between the experts on each of the different items is high, since the items have CVC values above 0.89, as shown in Table 3. The same applies to the instrument, with a CVC = 0.97.

TABLE 3

Table 3. Content validity coefficient (phase 2).

In conclusion, the experts evaluated the questionnaire as an appropriate instrument for measuring the self-perception of critical media literacy practices among university students enrolled in teaching programs. They also confirmed that the items are correctly classified within the specified dimensions, in alignment with the theoretical framework.

4.2 Construct validation

The questionnaire was applied to a propositional sample of 92 university students between 18 and 22 years old, including 68 women, 19 men and 5 people who did not identify with a gender, studying teaching programs at a Chilean university. The survey was administered online using a Microsoft form. The average response time was 11 min.

An exploratory factor analysis (EFA) was conducted using the Principal Components method and an Equamax rotation to assess the construct validity of the critical media literacy questionnaire in a sample of 92 university student teachers. The Kaiser-Meyer-Olkin sampling adequacy index (KMO) was 0.79, indicating that the data were adequate for factor analysis. Additionally, Bartlett’s test of sphericity yielded a significant result (χ² = 725.60, p < 0.001), confirming the feasibility of factor analysis.

In line with the model put forward in this study, three factors with eigenvalues above one was extracted, accounting for 51.0% of the accumulated variance. The first factor accounted for 30.2% of the variance, grouping items related to critical verification of information and evaluation of source reliability (Dimension 1, “Situating the Discourse in a Sociocultural Context”). The second factor explained 10.9% of the variance and was only associated with trust in various sources of information (Dimension 2, “Recognizing and Participating in Discursive Practice”). The third factor explained 9.9% of the variance and focused on information consumption behaviors and critical evaluation of texts (Dimension 3, “Calculating the Impact of a Discourse on the Community and Oneself”).

The rotated factor loadings matrix demonstrated that most of items exhibited a significant loading (> 0.4) on their respective factors, except for a few specific items:

• Item 27 (“I mainly rely on the opinions of people I follow on social networks to form my opinion”)

It demonstrated low communality (0.24) and factor loadings below 0.4 for all factors. This indicates that this item does not contribute significantly to the assessed construct.

• Item 5 (“When I read news on social networks, I ensure that it presents contingent facts”)

Also demonstrated low communality (0.27), indicating potential issues with the representation in the factor model.

Overall, the results indicate that the questionnaire has a coherent factorial structure, with a clear differentiation of the constructs measured by each factor. However, further work is needed to improve the internal consistency and construct validity of the instrument, specifically the elimination or reformulation of items with low communalities.

4.3 Analysis of internal validity and instrument consistency

Reliability analyses of the questionnaire show a reasonable level of consistency, with a Cronbach’s alpha of 0.84 and an omega coefficient of 0.85, as shown in Table 4. These results indicate that the instrument is reliable and that its items are well suited to measuring critical media reading practices. Internal consistency and item correlation analyses provide evidence of the validity of the instrument, so that the questionnaire can be used with confidence in future research or practical applications.

TABLE 4

Table 4. Reliability statistics.

The results demonstrate a high level of internal consistency and reliability for the developed questionnaire, with a Cronbach’s alpha of 0.84 and an Omega coefficient of 0.85.

4.4 Pilot application

The instrument was answered by 92 university students between the ages of 18 and 22, who were freshmen enrolled in one of three teaching programs: Language and Communication (Spanish as a first language), Music Education, and Special Education. The population of this study was undergraduate students in teacher education programs.

The pilot application revealed that 90% of the sample reported that they primarily obtained information via social networks, including Instagram (97%), WhatsApp (97%), and YouTube (86%). This indicates a group that prefers to get information through social networks over traditional media. Next, some of the most reliable items¹ of the questionnaire are analyzed.

In dimension 1 (situating the discourse in the socio-cultural context), the most reliable items are: 2. I think that the author of a news story influences how information is presented, 3. I think that news stories are objective and do not include the point of view of the media or the author, and 7. When I read opinion texts, I look for information about the author to understand his/her social, professional, and political perspective, etc. For item 2, 34.8% of participants always agreed and 38% almost always agreed, indicating that 72.8% of the sample recognized that the author of the news influences how the information is presented, what Cassany calls ideological positioning (Cassany, 2006, 2012; Cassany and Castellà, 2010). These data are consistent with responses to item 3, where 46.7% say sometimes and 25% say almost never, indicating that news stories are not objective but take into account the perspective of the author. However, the responses to item 7 are spread across all options: 33.7% say occasionally, 23.9% almost never, 16.3% almost always, 17.4% never, and 8.7% always. This means that although students know that the author’s perspective influences how a news story is presented, they do not spend time investigating the author of the text they read, or at least it does not represent a practice or strategy they use in their reading practices.

In dimension 2 (recognizing and participating in discursive practice), the most reliable items are 9. I get informed by chance when I come across news in my social networks, 10. I mainly trust information published by recognized or traditional media, and 12. I mainly trust information published by people or groups on social networks. The data from item 9 shows that most of the sample is informed by chance through information found on their social networks, with 34.8% saying almost always, 29.3% sometimes, and 26.1% always. This is important because we are dealing with a group that maintains its interaction in these media, and from a pedagogical perspective it is necessary to investigate how they engage with these texts. In point 10, the participants show divided answers: 29.3% say occasionally, 25% almost always, 22.8% always, and 17.4% almost never; this reflects the diversity of media they use to inform themselves, although the majority tends toward traditional informational media. Responses to item 10 are consistent with those observed in item 12, where students get information through social networks, but trust traditional media more and distrust information posted by people or groups on social networks.

In dimension 3 (calculating the impact of a discourse on the community and on oneself), the most reliable items are 23. I only get information from mass media that treat events from a perspective close to mine, 25. When I read opinions in forums, news comments or social networks, I reject those of people with ideologies different from mine, and 27. I mainly rely on the opinions of people I follow on social networks to form my own opinions. For item 23, 35.9% occasionally, 32.6% almost never, 14.1% never, and 14.1% almost always get information from media with a perspective close to their own, indicating that students have a pluralistic attitude toward receiving information, which is confirmed by the responses to item 25, where 29.3% occasionally, 29.3% almost never, and 23.9% never dismiss the opinions or ideas of people with different ideologies. In item 27, most participants recognize that their opinions are not based on people they follow on social networks.

In conclusion, the pilot study indicates that, while students are aware of the impact of the author’s perspective on the presentation of information, they frequently neglect to examine the background. Moreover, the majority of students obtain information through social networks, yet tend to be circumspect about data originating from non-traditional sources. These findings indicate a need for educational interventions to enhance students’ critical engagement with media and promote a more discerning approach to information consumption.

5 Discussion

The results of the internal validity and consistency analyses indicate that this instrument is valid, reliable, and consistent. The proposal of a questionnaire to measure critical media literacy practices represents a valuable contribution to the field, as there are currently no other instruments that assess this phenomenon in Spanish.

This questionnaire is distinguished from other instruments developed in Spanish by its focus on critical literacy practices with regard to media discourses. In the course of our review, we have identified tests designed to assess critical literacy skills (Gómez López et al., 2016; Díaz et al., 2016) and questionnaires that aim to evaluate self-perceptions of digital competence (García-Vandewalle et al., 2023; Pérez-Escoda et al., 2021). Nevertheless, no other instrument has been identified that assesses self-perception of media literacy practices.

The instrument’s relevance is based on two key strengths: its foundation in a well-defined theoretical framework and its consideration of a range of written and multimodal media, including the press and social networks, as the object of reading practices.

The design of the instrument places an emphasis on reading practices situated within the concepts of critical literacy and media literacy. From critical literacy, it is established that the practice of reading is situated within a social, political, and cultural context. This context is then analyzed to understand how texts and discourses function, where and with what consequences, and from whose interests (Luke, 2014). Thus, the statements in the questionnaire point to reading as a social practice that involves recognizing and questioning the information and opinions published in the media, as well as being self-critical as a reader.

Cassany’s comprehensive framework for critical literacy (Cassany, 2004; Cassany, 2006, Cassany and Castellà, 2010) constituted the main frame of reference for the definition of the assessment dimensions utilized in the design of the questionnaire. The author delineates a framework comprising three dimensions of critical reading practice, each of which can be disaggregated into actions that are essential for reading critically.

In the process of developing the questionnaire, the three dimensions of critical media literacy permitted the classification of statements according to specific reading practices. Dimension 1, situating the discourse in a sociocultural context, encompasses statements that seek to ascertain readers’ recognition of the author(s) of the text, their characteristics, ideological positions, and the extent to which this recognition affects the veracity or positioning of the information presented. Furthermore, statements seek to ascertain whether the reader considers the date and publication medium of the discourse, which are fundamental data in the analysis of mass media texts. The dissemination of outdated texts from different platforms often occurs because of readers accessing texts from various sources, which may not reflect the most recent information.

Dimension 2, recognizing and participating in discursive practice, encompasses statements that prompt readers to identify the discursive genre and structure, as well as the social conventions, when reading a mass media text. Furthermore, the statements aim to ascertain whether the subject can distinguish between the main ideas and secondary details, as well as between facts and opinions. It is of the most importance for the reader to be able to comprehend the structural elements and discursive frameworks utilized in media discourse, as well as how disparate subjects are approached. Moreover, it is essential to gain insight into the student’s self-perception.

Dimension 3, calculating the impact of a discourse on the community and oneself, incorporates questions that prompt the reader to engage in metacognition, or the ability to recognize and distinguish their own point of view and ideological position from those of others. An understanding of the student’s self-perception in this dimension enables the determination of their level of reflection and self-evaluation.

Cassany’s framework (Cassany, 2004, 2006, Cassany and Castellà, 2010), which is based on multiple sources, including Paulo Freire’s critical pedagogy, Critical Discourse Analysis, and New Literacy Studies, represents a significant contribution to the field of critical literacy within the Spanish-speaking context.

On the other hand, media literacy is conceived as a set of competencies that are put into practice when engaging with the mass media, with the objective of processing and interpreting the messages published therein (Potter, 2004). In this sense, the critical reading of media is considered as a complex and integral activity involving cognitive, social, cultural, and metacognitive components.

Buckingham (2020) outlines a critical analysis of media content, with a particular focus on four key concepts: language, representation, production, and audiences. Therefore, in interpreting media discourses, the reader uses his or her skills and knowledge, paying attention to the register used, the ideas and content of the discourses, the production context, and the socio-cultural environment. Thus, the dimensions defined in this questionnaire incorporate these key concepts.

This instrument is designed to facilitate a comprehensive understanding of the media sources that readers prefer and trust. This understanding enables the delineation of the role of different media for specific reader groups (e.g., by age, social status, or profession). The results of the pilot application corroborate this assertion, indicating that the majority of surveyed students reported obtaining information through social networks.

The pilot application yielded some trends, which allowed us to develop a general vision of these readers’ practices. For instance, students acknowledge the impact of the author’s viewpoint in disseminating information, yet they seldom delve into the identity of the text’s creator. The respondents indicated that they primarily obtain information from social networks on an informal basis, despite expressing reservations about the credibility of information shared by individuals or groups on these platforms. They demonstrated a willingness to consider diverse perspectives when processing information, indicating a pluralistic approach to information acquisition. This suggests that the group is aware of the importance of reliable information sources but does not actively seek them out or appraise the reliability of the media engage with. Moreover, they place a high value on the plurality of voices.

In the contemporary global context, where a plethora of information and, arguably, misinformation sources are readily available, the development of critical media literacy skills is of paramount importance. It is therefore imperative to gain an understanding of the reading practices associated with mass media. The initial focus of this study was on pedagogy students, given the pivotal role they will play in future classrooms, in the education of children and adolescents.

5.1 Limitations

The study’s sample consisted of 92 undergraduate students, which may not be sufficiently large to generalize findings across the entire population of university students. Additionally, given that the sample focuses on teacher training programs, the general applicability of the findings may be limited.

The use of self-reported measures may introduce a potential for response bias. Participants may have provided answers that are socially desirable rather than an accurate reflection of their beliefs and practices regarding media discourse. Therefore, it is advisable to incorporate other measures that allow for a more comprehensive observation of the phenomenon of critical media literacy.

5.2 Implications

This study illustrates the importance of having effective assessment tools to evaluate students’ critical media literacy skills. The design and validation of the instrument described in this study contribute to an understanding of the students’ self-perceived critical media practices. The development and validation of new instruments can assist educators in measuring growth and identifying areas for improvement.

Areas for further research were identified, including an analysis of the strategies employed by students when reading mass media texts and an investigation of the efficacy of particular instructional strategies in fostering critical media literacy. Longitudinal studies may offer a more comprehensive understanding of how students’ media literacy skills develop over time.

One challenge is the application of this questionnaire to a larger and more diverse sample of university students, followed by its use with other samples representing different populations. Cross-cultural and longitudinal studies are recommended to validate the findings and extend their applicability.

The findings highlight the necessity for the integration of critical media literacy into teacher education programs. Further and larger applications of the instrument will facilitate the acquisition of additional insights, which will be instrumental in the design of curricula that enhance students’ ability to analyze and engage with media discourse in a critical manner.

6 Conclusion

This research proposes a reliable and valid instrument to measure pre-service teachers’ self-perceptions of critical media reading practices based on critical literacy and media literacy frameworks.

Returning to the question: What dimensions should be included in an instrument to explore student teachers’ self-perceptions of their media literacy practices? We can affirm that the validation of this instrument has enabled us to confirm that the three proposed dimensions, based on the theoretical framework of reference, facilitate an understanding of the self-perception of student teachers regarding their critical media literacy practices.

In future lines of inquiry, it would be beneficial to consider the role of quantitative data analysis in enhancing reading practices and formulating pedagogical strategies for students at varying academic levels. To this end, the questionnaire serves as a crucial diagnostic tool for assessing reading practices and informing decision-making based on the findings.

In sum, this study provides a foundation for promoting critical media literacy among future educators and underscores its importance in developing informed, engaged citizens.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by the Comité de bioética Facultad de Educación. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

Author contributions

AN: Writing – original draft, Writing – review and editing, Conceptualization, Data curation, Funding acquisition, Investigation, Methodology, Project administration, Supervision, Validation. KF-R: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review and editing. AV: Conceptualization, Data curation, Investigation, Methodology, Writing – original draft, Writing – review and editing. FG: Conceptualization, Writing – original draft. GN: Writing – review and editing.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This article had been financed by the project “Aumento de la capacidad investigativa y de vinculación con el medio del programa de Doctorado en Educación de la Universidad de Concepción con foco en la paridad de género.” Proyecto Basal FB0003. Center for Advanced Research in Education, CIAE, University of Chile.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2024.1476500/full#supplementary-material

Footnotes

1. ^ The items have been translated from the original Spanish questionnaire, which is available for reference in Supplementary Appendix 1.

References

Andreotti, V. O. (2014). Critical literacy: Theories and practices in development education. Policy Pract. Dev. Educ. Rev. 19, 47–79.

Google Scholar

Andrews, G., and Leonard, S. Y. (2023). Exploring the lived experiences of middle grades teacher candidates engaging in critical consciousness to inform equity-oriented and responsive teacher education. Educ. Sci. 13:658. doi: 10.3390/educsci13070658

Crossref Full Text | Google Scholar

Bobkina, J., and Stefanova, S. (2016). Literature and critical literacy pedagogy in the EFL classroom: Towards a model of teaching critical thinking skills. Stud. Second Lang. Learn. Teach. 6, 677–696. doi: 10.14746/ssllt.2016.6.4.6

Crossref Full Text | Google Scholar

Buckingham, D. (2009). The future of media literacy in the digital age: Some challenges for policy and practice. Medienimpulse 47, 1–18.

Google Scholar

Buckingham, D. (2019). Teaching media in a ‘post-truth’ age: Fake news, media bias and the challenge for media/digital literacy education. Cult. Educ. 31, 213–231. doi: 10.1080/11356405.2019.1603814

Crossref Full Text | Google Scholar

Buckingham, D. (2020). Rethinking digital literacy: Media education in the age of digital capitalism. Digit. Educ. Rev. 37, 230–239.

Google Scholar

Cassany, D. (2004). Explorando las necesidades actuales de comprensión. Aproximaciones a la comprensión crítica. Lectura y vida XXV, (2), 6–23.

Google Scholar

Cassany, D. (2006). Tras las Líneas. Sobre la Lectura Contemporánea. Barcelona: Anagrama.

Google Scholar

Cassany, D. (2012). En_línea. Leer y Escribir en la Red. Barcelona: Anagrama.

Google Scholar

Cassany, D. (2020). La lectura crítica en la era digital. Rev. Educ. 374, 113–126.

Google Scholar

Cassany, D., and Castellà, J. (2010). Aproximación a la literacidad crítica. Perspect. Florianópolis 28, 353–374.

Google Scholar

De Abreu, B. (2019). Teaching Media Literacy. Chicago: American Library Association.

Google Scholar

Díaz, J., Bar, A., and Ortiz, M. (2016). Autopercepción de habilidades de lectura crítica en estudiantes de ciencias de la educación. Actual. Investig. Educ. 16, 1–23.

Google Scholar

Escobar-Pérez, J., and Cuervo-Martínez, A. (2008). Validez de contenido y juicio de expertos: Una aproximación a su utilización. Avances Medición 6, 27–36.

Google Scholar

García-Vandewalle, J., García-Carmona, M., Trujillo, M., and Moya, P. (2023). Analysis of digital competence of educators (DigCompEdu) in teacher trainees: The context of Melilla, Spain. Tech. Know Learn. 28, 585–612. doi: 10.1007/s10758-021-09546-x

Crossref Full Text | Google Scholar

Gómez López, V., Rosales Gracia, S., and García Galaviz, J. (2016). Percepción de la utilidad de la lectura crítica en el aprendizaje de la investigación médica en posgrado. Educ. Méd. Super. 30, 270–277.

Google Scholar

Hallaq, T. (2016). Evaluating online media literacy in higher education: Validity and reliability of the Digital Online Media Literacy Assessment (DOMLA). J. Med. Lit. Ed. 8, 62–84. doi: 10.23860/jmle-8-1-5

Crossref Full Text | Google Scholar

Hayes, A. F., and Coutts, J. J. (2020). Use Omega rather than Cronbach’s Alpha for estimating reliability. But…. Commun. Methods Meas. 14, 1–24. doi: 10.1080/19312458.2020.1718629

Crossref Full Text | Google Scholar

Henry, C. C., and Mohamad, M. (2021). The use of mobile technology in enhancing the critical reading skills of Pre-University ESL students. Creat. Educ. 12, 678–695. doi: 10.4236/ce.2021.123047

Crossref Full Text | Google Scholar

Hernández y Hernández, D., Cassany, D., and López González, R. (2018). Prácticas de Lectura y Escritura en la era Digital. Córdoba: Editorial Universidad de Córdoba.

Google Scholar

Hernández-Nieto, R. A. (2002). Contributions to Statistical Analysis. Mérida: Universidad de Los Andes.

Google Scholar

Hobbs, R., Rongwei, M., and Steager, P. (2022). Measuring the implementation of media literacy statewide: A validation study. Edu. Media Int. 59, 189–208. doi: 10.1080/09523987.2022.2136083

Crossref Full Text | Google Scholar

Jones-Jang, S., Mortensen, T., and Liu, J. (2021). Does media literacy help identification of fake news? Information literacy helps, but other literacies don’t. Am. Behav. Sci. 65, 371–388. doi: 10.1177/0002764219869406

Crossref Full Text | Google Scholar

Landis, J. R., and Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33, 159–174.

Google Scholar

Livingstone, S. (2004). What is media literacy? Intermedia 32, 18–20.

Google Scholar

Luke, A. (2014). “Defining critical literacy,” in Moving Critical Literacies Forward: A New Look at Praxis Across Contexts, eds J. Avila and J. Zacher Pandya (New York, NY: Routledge), 19–31.

Google Scholar

Mesquita, W., Fernández, M., and Cebreiro, B. (2022). Alfabetización mediática crítica para mejorar la competencia del alumnado. Comunicar 70, 47–57.

Google Scholar

Monzón, E. (2019). “Alfabetización digital en el aula,” in Proceedings of the Digital World Learning Conference CIEV, (Guatemala), 89–98.

Google Scholar

OFCOM, (2024). Making Sense of Media. OFCOM’S Programme of Work to Help Improve the Media Literacy of UK Adults and Children. Available online at: https://www.ofcom.org.uk/media-use-and-attitudes/media-literacy/making-sense-of-media/ (accessed September 20, 2024).

Google Scholar

Olifant, T., Cekiso, M., and Rautenbach, E. (2020). Critical reading perceptions and practices of English First Additional Language learners in Gauteng, Tshwane South district. Read. Writ. 11:281. doi: 10.4102/rw.v11i1.281

Crossref Full Text | Google Scholar

Paige, D., Rupley, W. H., and Ziglari, L. (2024). Critical thinking in reading comprehension: Fine tuning the simple view of reading. Educ. Sci. 14:225. doi: 10.3390/educsci14030225

Crossref Full Text | Google Scholar

Pedrosa, I., Suárez-Álvarez, J., and García-Cueto, E. (2014). Evidencias sobre la validez de contenido: Avances teóricos y métodos para su estimación. Acción Psicol. 10, 3–18.

Google Scholar

Pérez-Escoda, A., Pedrero-Esteban, L. M., Rubio-Romero, J., and Jiménez-Narros, C. (2021). Fake news reaching young people on social networks: Distrust challenging media literacy. Publications 9:24. doi: 10.3390/publications9020024

Crossref Full Text | Google Scholar

Pinchao, L. (2020). Estrategia pedagógico-didáctica para promover la lectura crítica. Educación 29, 146–169. doi: 10.18800/educacion.202001.007

Crossref Full Text | Google Scholar

Potter, W. J. (2004). Theory of Media Literacy: A Cognitive Approach. Thousand Oaks, CA: SAGE Publications.

Google Scholar

Quispe Limaylla, A., and Sánchez Moreno, G. (2011). Encuestas y entrevistas en investigación científica. Rev. Actual. Clín. Investiga 10, 490–494.

Google Scholar

Romano, M. B., and López, E. (2016). Lectura crítica de medios en estudiantes universitarios. Praxis Educ. 20, 25–33. doi: 10.19137/praxiseducativa-2016-200203

Crossref Full Text | Google Scholar

Rumble, J. (2011). “OFCOM’S media literacy work,” in Media Literacy: Ambitions, Policies and Measures, ed. S. Livingstone , London: COST, 11–19.

Google Scholar

Serrano de Moreno, S. (2008). La alfabetización crítica, conceptualización de las competencias y estrategias de lectura crítica. Letras 76, 101–130.

Google Scholar

Weng, T. (2023). Creating critical literacy praxis: Bridging the gap between theory and practice. RELC J. 54, 197–207. doi: 10.1177/0033688220982665

Crossref Full Text | Google Scholar

Zhong, H. L. (2015). Education of media literacy and ideology & politics. Creat. Educ. 6, 2422–2426. doi: 10.4236/ce.2015.622248

Crossref Full Text | Google Scholar