Jorge Gaete
Jesus Montero-Marin
Cristian A. Rojas-Barahona
Esterbina Olivares
Ricardo Araya- 1Department of Epidemiology and Population Health, London School of Hygiene and Tropical Medicine, London, UK
- 2Department of Public Health and Epidemiology, Universidad de los Andes, Santiago, Chile
- 3Faculty of Health Sciences and Sports, University of Zaragoza, Zaragoza, Spain
- 4Primary Care Prevention and Health Promotion Research Network (RedIAPP), Zaragoza, Spain
- 5Faculty of Education, Pontificia Universidad Católica de Chile, Santiago, Chile
- 6School of Nursing (Campus San Felipe), Faculty of Medicine, Universidad de Valparaíso, San Felipe, Chile
School membership appears to be an important factor in explaining the relationship between students and schools, including school staff. School membership is associated with several school-related outcomes, such as academic performance and expectations. Most studies on school membership have been conducted in developed countries. The Psychological Sense of School Membership (PSSM) scale (18 items: 13 positively worded items, 5 negatively worded items) has been widely used to measure this construct, but no studies regarding its validity and reliability have been conducted in Spanish-speaking Latin American countries. This study investigates the psychometric properties, factor structure and reliability of this scale in a sample of 1250 early adolescents in Chile. Both exploratory and confirmatory factor analyses provide evidence of an excellent fit for a one-factor solution after removing the negatively worded items. The internal consistency of this new abbreviated version was 0.92. The association analyses demonstrated that high school membership was associated with better academic performance, stronger school bonding, a reduced likelihood of school misbehavior, and reduced likelihood of substance use. Analyses showed support for the reliability and validity of the PSSM among Chilean adolescents.
Introduction
During adolescence, and especially during the early adolescence (ages 10–14), individuals begin to explore their own identity and how they fit in the world. People also begin to project themselves to the future, becoming more distant from their own families (Steinberg and Silverberg, 1986) and relying more on friendships and non-kin relationships for help and guidance (Steinberg and Monahan, 2007).
School is the place where students spend an average of 6 to 7 h every weekday until they complete their secondary education between the ages of 16 to 18. Therefore, school becomes a place where social and contextual influences may have a huge impact on adolescent behaviors. Different variables have been studied to explore the students' connection to school, such as school bonding, school engagement, school involvement, and school membership (Maddox and Prinz, 2003; Libbey, 2004). Each one of them refers to a particular aspect of this relationship. For example, school bonding seems to be a broader concept under which we find the emotional link with the school (school attachment), students' involvement in academic activities (school commitment), students' participation in extra-curricular activities (school involvement), beliefs about school rules (Jenkins, 1997); and school engagement which refers to how much the student engaged in activities related to academic success such as doing homework or achieving better grades (Simons-Morton and Crump, 2003). School membership refers to the students' perceptions on how much teachers and other classmates respect and accept them, and students' feelings of inclusion and belonging. School membership increases engagement in healthy behaviors and reduces the likelihood of engagement in risky behaviors (McNeely and Falci, 2004). High levels of school membership have been associated with better academic motivation and performance (Goodenow and Grady, 1993; Ahn, 2010; Irvin et al., 2011; Moallem, 2013) and better health-promoting behaviors (Gaete et al., 2014). On the contrary, low levels of school membership have been associated with increased negative affect (Shochet et al., 2011), increased suicidality (Stallard et al., 2013), higher school drop-out rates (Finn and Rock, 1997; Ream and Rumberger, 2008), increased school misconduct (Demanet and Van Houtte, 2012), and increased substance use (West et al., 2004). In the last several years, there has been an increasing interest in the evaluation of school-based interventions that aim to improve school climate and school membership; these interventions boast promising results (Bonell et al., 2014; Langford et al., 2014, 2015). However, much less research on this topic has been conducted in Spanish-speaking countries, especially in Latin America. Thus, there is a clear need for a good research instrument in Spanish to measure school membership.
The Psychological Sense of School Membership (PSSM) scale is a self-reported instrument developed to assess the sense of school belonging. The original PSSM scale comprises 18 items: 13 positively worded statements and 5 negatively worded statements. For each statement, students answer on a 5-point scale (1 = not at all true; 5 = completely true). All of these items are related to students' perceptions of being “accepted, respected, included and supported by others in the school social environment” (p. 80)(Goodenow, 1993). It has been widely used—mainly in English-speaking countries—and it has been associated with several variables related to academic achievement, such as increased competence and self-efficacy (Ibañez et al., 2004), increased school attendance (Sánchez et al., 2005), and higher grades (Booker, 2007). Several U.S.-based studies have gathered information from Latino/a or Hispanic students using a Spanish version. However, these studies only report on a few measures of internal reliability (Goodenow, 1993; Robertson et al., 1998; Ibañez et al., 2004; Sánchez et al., 2005; Kuperminc et al., 2008) and do not provide any information about the construct validity or factor structure of the Spanish version.
We have identified eight studies that explored the internal item structure of the 18-item PSSM scale. Three studies were conducted in the U.S. (Hagborg, 1994; O'farrell and Morrison, 2003; Ye and Wallace, 2014); two were conducted in China (Cheung and Hui, 2003; Cheung, 2004); one was conducted in Australia (You et al., 2011); one was conducted in Japan (Togari et al., 2011); and one was a multi-country (Netherland, Kenya, Indonesia, and Spain) comparison study (Abubakar et al., 2015). The latter study included a sample of 14- to 18-year old adolescents from Spain, but some of them responded to the PSSM scale in Spanish while others responded to the scale in Basque (the authors did not provide the exact proportion of respondents for each language) (Abubakar et al., 2015). There is some variability regarding the number of extracted factors that show a good fit, the items included in each factor and the total number of items to be included in the scale. For example, one study found that the school membership scale should include just 5 items from the original scale (O'farrell and Morrison, 2003). A different study found that a one-factor model showed a good fit, using all the items but combining items per target (e.g., school, teachers) (Abubakar et al., 2015). However, two latent factors appear in the Chinese version (Cheung and Hui, 2003; Cheung, 2004), and three factors in other studies (Hagborg, 1994; You et al., 2011). Some studies have suggested that the negatively worded items may constitute a unique factor (Hagborg, 1994; Cheung and Hui, 2003); this does not necessarily imply the existence of a real latent factor in the construct, but rather a methodological issue (Ye and Wallace, 2014). Cultural differences seem to be important when considering a scale in different countries (Abubakar et al., 2015). Noticeably few studies have used a comprehensive data analysis, (i.e., both Exploratory or Confirmatory Factor Analyses) to explore the factor structure of the scale, which may also explain the differences in the findings between the studies (You et al., 2011; Abubakar et al., 2015).
The original purpose of this scale (Goodenow, 1993) was to develop an instrument to assess the social and contextual influences on students' behaviors. The scale was intended to be an instrument with the ability to identify students who are at-risk either for exhibiting risk behaviors and/or dropping out from school. In addition, the scale was initially conceptualized as an unidimensional measure (Goodenow, 1993; You et al., 2011).
The aims of this study are as follows: (1) to explore the factor structure of the Spanish version of the PSSM scale in Chilean early adolescents; (2) to examine the reliability of the resulting scale; and (3) to assess possible relationships between school membership and other school-related outcomes. We hypothesized that school membership will have a positive association with academic variables (academic performance and expectations), and school climate variables (school bonding, school attachment, and school support); conversely, we predicted a negative association between school membership and school misbehavior variables (cheating, truancy, and substance use).
Materials and Methods
Study Design
This is an analytical cross-sectional survey using self-reported information.
Setting, Sample
The population consisted of students in 5th to 8th grade (ages 10–15) from seven schools in San Felipe, a city in the center of Chile. A total of 2108 eligible students were invited to participate; the research team sent a letter to their parents/main caregivers, and informed consent was required to participate. Of those eligible students that were invited, 69.5% consented and participated in the study, resulting in a sample of 1465 students. Completed data for all items of the PSSM scale was gathered from 1250 students. The Ethical Committee at the Faculty of Education of Pontificia Universidad Católica de Chile approved the study protocol. The study followed Helsinki Convention norms.
A qualified nurse (EO) trained several research assistants, who worked in pairs or trios, to administer paper-and-pencil questionnaires to each class. Each school provided a suitable place to conduct the study, assuring that only students with the consent of their parents participated. The study was undertaken in March 2012, the beginning of the academic year in Chile. The day of the survey, research assistants introduced the study to the students, explained the aims and answered questions from the students. All students were asked to sign an informed assent form to participate; no student declined to participate. After the students completed the study, the questionnaires were collected by the research assistants and kept in a sealed envelope to maintain confidentiality.
Measures
The Psychological Sense of School Membership (PSSM) scale was developed by Goodenow (1993) and consisted of 18 items that measure a perception of membership to the school: 13 positively worded statements intercalated with 5 negatively worded statements. This scale has been used in a large number of studies in different contexts, and it has been translated into several languages (Bernard-Bonnin et al., 2008; Ahn, 2010; Shochet et al., 2011; You et al., 2011; Sari, 2012; Chang et al., 2013; Stallard et al., 2013; Ye and Wallace, 2014; Abubakar et al., 2015; Alkan, 2016). Internal consistency has ranged from 0.73 to 0.95 across samples and countries (Goodenow, 1993; You et al., 2011; Abubakar et al., 2015). The original English version of the scale was received from Carol Goodenow with the authorization to adapt and translate the PSSM to Spanish. Two professional translators produced two Spanish versions. Later, a bilingual expert panel (which included some of the authors) resolved discrepancies from the two Spanish versions, creating one final version. Finally, an independent bilingual translator, whose mother tongue was English, translated the scale back to English, and the expert panel compared the English versions, and a satisfactory final Spanish version was agreed.
Sociodemographic Variables
Data were collected on age, gender (0 = Male; 1 = Female), grade, school type, religiosity, parental education and occupation, parental marital status, and socioeconomic status (1 = low income; 2 = medium income; 3 = high income). The socioeconomic status was based on the criteria of the 2009 National System for the Measurement of Education Quality (Ministry of Education (Chile), 2010), which gathers information from parents or main caregivers about their household income.
School Variables
In order to explore the relationship between school membership and other school-related factors, we collected the following data:
Self-Reported Academic Performance
We collected each student's grade point average (GPA) from the previous school year. The Chilean GPA scale ranges from 1 to 7, and the approval cutoff for passing is 4.0. We asked students to select their GPA from one of the following categories: 1 = Below 4.0; 2 = Between 4.1 and 4.4; 3 = Between 4.5 and 4.9; 4 = Between 5.0 and 5.4; 5 = Between 5.5 and 5.9; 6 = Between 6.0 and 6.4; 7 = Between 6.5 and 7.0.
Academic Expectations
We asked if students were concerned about failing in school this year on a 4-point scale (1 = not at all concerned to 4 = extremely concerned). This item is part of a questionnaire of adolescent health that has been validated in Chile (Blum et al., 1988; Florenzano et al., 1993).
School Bonding
We used a scale similar to the one used by Hawkins et al. (2001) to assess school bonding. It included the following items: “I like school,” “Most mornings I look forward to going to school,” “I do extra school work on my own,” “When I have an assignment to do, I keep working on it until it is finished,” and “I like my classes this year.” Response options for each item were as follows: 1 = NO!; 2 = no; 3 = yes; and 4 = YES!. Each student's responses to the 5 items were averaged together to create their total score. A higher score indicates a higher level of school bonding. The Cronbach's alpha of this scale in our sample was 0.77. This scale measures concepts similar to school bonding, such as school attachment and school commitment, following Maddox and Printz's conceptualization (Maddox and Prinz, 2003). Therefore, the construct is similar to school membership, but not quite the same. We followed a similar procedure to obtain the Spanish version of this scale as the procedure we followed with the PSSM scale.
School Liking
We included a single item exploring the concept of school attachment: “How do you feel about your school?” Response options were: 1 = I hate my school; 2 = I don't like it; 3 = I like it somewhat; and 4 = I like it very much. This item is part of a questionnaire of adolescent health that has been validated in Chile (Blum et al., 1988; Florenzano et al., 1993).
School Staff Support
We asked students if they believe that school staff (e.g., school staff, administrator, teachers) care about them. Response options were: 1 = not at all; 2 = somewhat; 3 = very much. This item is part of a questionnaire of adolescent health that has been validated in Chile (Blum et al., 1988; Florenzano et al., 1993).
School Misbehavior
We assessed school misbehavior using two items. The first item was about cheating: “During the last year, how many times did you copy answers from others?” The second item was about truancy: “During the last month, how many days did you skip classes or did not attend school without permission?” Students responded to each item using a 5-point scale (1 = never to 5 = more than 10 times). This item is part of a questionnaire of adolescent health that has been validated in Chile (Blum et al., 1988; Florenzano et al., 1993).
Substance Use
We asked students about their use of tobacco, alcohol, marijuana, inhalants, stimulants, cocaine, and other drugs during the past year. We operationalized this construct as a binary variable (0 = no usage of any drug in the last year; 1 = usage of some drug in the last year).
Data Analysis
All analyses were performed using STATA 12.1, Factor 9.02, and Amos 7. Firstly, we examined the sociodemographic variables, as well as the items' psychometric characteristics by using descriptive statistics (i.e., means, standard deviations, frequencies, percentages, skewness, and kurtosis).
Dimensionality
To analyze the factor structure, we randomly split the data into two halves: The first half (n1 = 625) was used for the initial Exploratory Factor Analysis (EFA), and the second half (n2 = 625) was used for the Confirmatory Factor Analysis (CFA). We estimated Mardia's coefficients (Mardia, 1974) to assess the multivariate distribution of the variables. Polychoric correlations are advised for factor analysis when the distributions of ordinal items are asymmetric with excess of kurtosis or high item-total correlations (Muthen and Kaplan, 1992). For each sub-sample, a polychoric correlation matrix of PSSM scale items was estimated, and the unweighted least squares factor analysis (ULS) was used for factor extraction, due to its robustness (Jöreskog, 1977). This procedure does not require any distributional assumptions; it usually converges because of its efficiency in terms of computation; it tends to provide less biased estimates of the true parameter value; and it works well with polychoric matrices (Briggs and Maccallum, 2003). We firstly observed the unrotated solution of the EFA, but we also used the promin rotation method to facilitate interpretations, given the correlated solution expected (Lorenzo-Seva, 1999).
We used parallel analysis (Horn, 1965) to identify the number of factors to include in the solution. This identification was performed by replacing the raw data method with optimal implementation based on minimum rank factor analysis (Timmerman and Lorenzo-Seva, 2011), generating 500 random correlation matrices. With this analysis, a factor is significant if the associated eigenvalue is larger than that corresponding to a given percentile, such as the 95th percentile of the distribution of eigenvalues derived from a random dataset. This method is the best available solution to decide the number of factors to retain for a given scale (Hayton et al., 2004; Ledesma and Valero-Mora, 2007). To select the items to be included in each latent factor, we used the criterion of factorial loading (i.e., <0.5) (Comrey and Lee, 2013). If any item loaded lower than 0.5, we removed those items, then we conducted the EFA again, in an iterative way, repeating this procedure until all the items loaded higher than 0.5 in their corresponding factor (i.e., until they showed a clear and simple structure of weightings). Factor weights (w) and the percentage of explained variance in each item by means of communality values (h2) were calculated. For each EFA, we tested the goodness of fit of the exploratory model using the goodness of fit index (GFI) (Maiti and Mukherjee, 1990) and the root mean square of residuals (RMSR) (Harman, 1962).
Using CFA and by applying ULS from the second polychoric matrix, we examined the goodness of fit of the proposed PSSM scale. From a general perspective, we used the goodness of fit index (GFI), the adjusted goodness of fit index (AGFI), the root mean square of the standardized residuals (RMSR), the normed-fit-index (NFI), and Bollen's relative-fit-index (RFI). GFI and AGFI refer to explained variance; values greater than 0.90 are considered acceptable (Byrne, 2001). SRMR is the standardized difference between the observed and the predicted covariance; values lower than 0.08 indicate a good fit (Hu and Bentler, 1999). NFI measures the proportional reduction in the adjustment function when going from null to the proposed model; values greater than 0.90 are considered acceptable (Lévy et al., 2006). RFI takes into account the discrepancy for the proposed model and for the baseline model; values that are close to 1 indicate a good fit (Bollen, 1986). Standardized saturations and the proportion of explained variance were also examined.
Reliability
We examined the reliability of the scale using congeneric, tau-equivalent and parallel models for the whole sample (n = 1250), using the ULS method. The three models assume that all items measure the same latent factor. However, the congeneric model is the least restrictive model, with potentially different scales, degrees of precision, and magnitude of error. The tau-equivalent implies that items are on the same scale, with the same degree of precision, but with possibly different degrees of error. The parallel model is the most restrictive, where all items are on the same scale, with the same degree of precision, and with the same amount of error (Raykov, 1997). The reliability value was estimated by squaring the implied correlation between the composite latent true variable and the composite observed variable, to arrive at the percentage of the total observed variance that was accounted for by the “true” variable (Graham, 2006). The mean of item-total values (discrimination coefficients) was also assessed.
Associations
We explored the associations between school membership (as an independent variable) and the school-related factors (as dependent variables) using regression models and controlling for age, gender, and socioeconomic status. In most cases, the dependent variable was operationalized as a continuous variable; therefore, simple regression models were used. Standardized slopes (beta) and coefficients of determination (R2) were also calculated. For the case of substance use (binary variable), a logistic regression was performed, and odds ratios were examined. The cutoff for statistical significance was established at p < 0.05 and 95% confidence intervals were calculated and presented.
Results
Description of Participants
Table 1 shows the general features of the total sample and for each sub-sample. The sample consisted of students between the ages of 10–15 (mean = 11.9 years: SD = 1.2 years), and 45% were girls. Most of the students came from low socioeconomic status (45%) and there was a fairly similar distribution of grade level (20.5–27.4%). Nearly one-fifth (18.3%) of the students considered themselves as very religious. Most students lived with both parents (63.8%); further, over half the parents completed secondary education (58.7% of mothers and 54.2% of fathers). Almost 70% of fathers worked full-time, while 44.1% of mothers worked full-time.
Table 1. Characteristics of participants in total sample (N = 1250), and in the sub-samples used (n1 = 625; n2 = 625).
Descriptives and Dimensionality
The polychoric matrix of the PSSM items using the first sub-sample revealed that 52.9% of the coefficients on the diagonal were greater than 0.30. The determinant of the matrix was 0.003. KMO test had a value of 0.90 (very good) and Bartlett's statistic was 3841.7 (df = 153; p < 0.001). Table 2 shows the descriptive statistics of all 18 items and the loading factors for each step of the iterative EFA that was explained in the methods section. The unrotated solution was difficult to interpret in terms of meaning, so that we relied on the promin rotation to facilitate this task.
Table 2. Iterative process of discarding inadequate PSSM scale items through Exploratory Factor Analysis (n1 = 625).
We needed to conduct the EFA three times. In the first EFA, we observed two factors that explained 40.9% (F1) and 13.3% (F2) of the variance (GFI = 0.95; RMSR = 0.10). Most items loaded onto F1; only two items (items 6 and 9) loaded onto F2. Three other items (items 3, 12, and 16) did not load onto any of the factors over 0.5, so they were removed. When we conducted the second EFA with 15 items, only one factor appeared; this factor explained 51.6% of the variance (GFI = 0.98; RMSR = 0.07). In this factor, all items but two (6 and 9) had a factor loading greater than 0.5; therefore, items 6 and 9 were removed. Next, we conducted a third EFA with 13 items, finding one factor that explained 54.2% of the variance, (GFI = 0.99; RMSR = 0.06). Finally, all items had a factor loading greater than 0.5. Thus, the best fit was found in the latter exploratory factor model.
The polychoric matrix of the PSSM scale items using the sub-sample 2 revealed that 100% of the coefficients on the diagonal were over 0.30. The determinant of the matrix was 0.005. The KMO test had a value of 0.94 (very good) and Bartlett's statistic was 3167.4 (df = 783; p < 0.001). Table 3 shows the descriptive statistics of the 13-item scale and the PSSM one-factor structure using CFA from both an analytical and standardized point of view. The factor loadings for the items were very high (ranges from 0.59 to 0.79). This structure possessed adequate fit indices (Table 4), with no correlations between the error terms (GFI = 0.99; AGFI = 0.99; NFI = 0.99; RFI = 0.99; RMSR = 0.04).
Table 3. Descriptive statistics for items selected for CFA (n2 = 625).
Table 4. Adjustment indices for CFA and models of reliability of the PSSM scale.
Reliability
Table 4 shows the reliability models tested for the PSSM structure. As it can be seen, the best fit indices were for the congeneric model, although the parallel model also indicates a good fit for the data (GFI = 0.92; AGFI = 0.98; NFI = 0.98; RFI = 0.98; RMSR = 0.07). The reliability values were very good in all the models (R = 0.92). The mean of the item-total values was 0.60 (range: 0.49 to 0.66).
Associations
We found that school membership was significantly associated with all measured school-related factors, controlling for age, gender and socioeconomic status, although the effect size varied considerably across variables. Specifically, a higher score in school membership was associated with higher levels of school performance (β = 0.29; p < 0.001), school bonding (β = 1.60; p < 0.001), school attachment (β = 0.21; p < 0.001); school expectations (β = 0.11; p < 0.001), and school staff support (β = 0.29; p < 0.001). A higher score in school membership was also associated with lower levels of cheating (β = −0.12; p < 0.001), and truancy (β = −0.04; p = 0.007). We also found that a high score in school membership predicted a reduced risk of having used any substance during the last 12 months (OR = 0.60; p = 0.010). Effect sizes were high for school bonding (R2 = 0.29), being moderate for school performance (R2 = 0.18), school support (R2 = 0.16), cheating (R2 = 0.14), and school attachment (R2 = 0.11), and low for substance use (R2 = 0.05), school expectations (R2 = 0.02), and truancy (R2 = 0.01). Additionally, we found that older students tended to have lower grades, lower school bonding, lower school attachment, and higher risk for cheating. Females, compared to their male counterparts, had higher grades but also reported more cheating. Higher socioeconomic status was related to higher school performance and lower likelihood cheating (Table 5).
Table 5. Regression models of the association between school membership and school-related factors; controlling for age, gender, and socioeconomic status (SES).
Discussion
We found that the Spanish version of the PSSM scale measures one underlying factor, sense of school belonging. However, our analyses suggested that not all items performed well, particularly those that were negatively worded. We found that when using all 18 items, the negatively worded items may have created a methodological artifact—one that has already been found in some studies exploring the same scale (Ye and Wallace, 2014; Abubakar et al., 2015) or in studies using other scales (Marsh, 1996; Hankins, 2008). However, when removing all negatively worded items, the structure and the loading of the items in the scale appear to be consistent. Therefore, after the EFA analyses, one latent factor was found, and the CFA of these items results in excellent goodness of fit indices.
Several authors have found different item-structures of this scale. Initially, Goodenow did not report any analysis of the structure of the scale, but she was using it as unidimensional scale with good reliability (Goodenow, 1993). Later, using a small sample, Hagborg identified three factors, and one of them was labeled as “belonging” which consisted of 11 items (Hagborg, 1994), all of which were positively worded. In a subsequent study, Hagborg used this abbreviated scale of 11 items and obtained good results, supporting the idea that the PSSM scale should be considered unidimensional. Nonetheless, other studies have identified more than one latent factor; however, in many cases, these factors did not include the same items. Two recent studies, each with a good sample size (You et al., 2011; Ye and Wallace, 2014), used a combination of EFA and CFA conducted in different samples; the results suggested that there were three latent factors. However, the items that loaded onto particular factors were different across the two studies.
We noted that Cheung and Hui's study found 2 underlying factors, one with the 13 positively-worded items and another with the 5 negatively worded items (Cheung and Hui, 2003). In You et al. (2011) after removing six items because of cross-loading, three latent factors were identified: (1) caring relationships with teachers (4 items: 3 positively worded and 1 negatively worded); (2) acceptance (5 positively worded items); and (3) rejection (three negatively worded items). Taking this information into account, Ye and Wallace (2014) conducted a survey with U.S. secondary students from ages 13 to 19. The researchers explored whether a method effect caused previous studies to identify a factor including only negatively worded items, or if there is more than one substantive factor in the scale (Ye and Wallace, 2014). The authors found that there was a method effect related to the wording of the items and they advised that further research needed to explore the implications of this effect. However, the researchers removed 3 items, and they decided to keep this factor in the scale, thus proposing a structure solution with 3 factors: (1) identification with and participation in the school; (2) perception of fitting in among peers; and (3) generalized connections to teachers.
Finally, the most recent published study gathered information from four countries (Abubakar et al., 2015); in one of those countries, a Spanish version of the scale was used. The authors explored several models for the relationship between items, and they also studied the potential methodological artifact introduced by the negatively-worded items. These researchers found evidence that the PSSM scale has a unidimensional structure, and they supported the idea that the negatively-worded items were not producing an artificial multidimensional structure that was found in other studies. The researchers also posited that it is possible to consider the items as forming three sub-scales according to the targets or contexts to which the items were referring (Abubakar et al., 2015). The authors did not propose removing any items but instead considered that there may be a need to refine the scale further to ensure clarity in measurement. Our study differed from this one because the adolescents in their study were older (ages 14–18) and they also had a smaller sample (n = 590).
In our study, the first examination of the psychometric properties of the PSSM scale in Latin American early adolescents, we found that the negatively worded items did not load into the main factor or into any other factor. Furthermore, the latent factor uncovered in our study was very similar to the one originally found by Hagborg (1994). One potential explanation for our results may be related to the age range of our sample. It is possible that, due to the cognitive development and proficiency in reading comprehension, students who were ages 10–14 might have had difficulty understanding the answer format or the direction of the intercalated questions. Many of the studies reviewed here explored the psychometric properties of the scale with older adolescents (Abubakar et al., 2015). However, in a study with 4–6th graders in the U.S., the negatively-worded items appeared to load onto a separate factor than the positively worded items (O'farrell and Morrison, 2003), giving some support to the idea that there could be a methodological artifact.
The reliability of the final proposed items to form the PSSM scale was very high (0.92), which could be considered one of the highest internal consistency obtained so far (You et al., 2011). Moreover, each item seemed to be measuring the corresponding latent variable with possibly the same degree of precision and the same amount of error. Therefore, we propose the use of an abbreviated version of the PSSM scale, which includes only the positively worded items (13 items), to allow for the assessment of school membership among adolescents (ages 10–19) (Sawyer et al., 2012) and we encourage the comparison of the results across cultures.
The exploration of the associations between the sense of school membership and other relevant school-related outcomes supports the idea that school membership seems to be a good measure of the relationship between students and schools, either as a predictor of academic outcomes (achievement, truancy, drop-out), as a potential mediator for interventions aiming to increase school climate or even to improve misbehavior. It is important to continue studying this construct to better understand potential changes over time while students get older and to determine how they respond to different interventions.
One limitation of this study is related to its cross-sectional design, which makes it difficult to establish a longitudinal and truly predictive value of this measure. Another limitation is the use of retrospective, self-reported measures in an adolescent population, which could have influenced the accuracy of the reporting. For instance, cognitive factors might introduce reporting bias (related to comprehension of questions and decision-making issues) (Pokorski et al., 1994) or retrieval errors (especially for long periods of time)(Bachman and O'malley, 1981). Situational factors that affect the self-reporting may include students' perceptions of the confidentiality of the information (Hedges and Jarvis, 1998) or social desirability (Brittingham et al., 1998). When administering the PSSM scale, research assistants did not report any complaints about the comprehension of the items. Regarding the rest of the questionnaire, many of the questions have been used in other studies in Chile for many years (Senda, 2014) without any reported problems. Regarding the situational factors, numerous measures were taken out to ensure anonymity and confidentiality. Furthermore, there is evidence some that the cognitive and situational factors mentioned above do not threaten the validity of self-reported measurements among students (Brener et al., 2003).
Conclusions
The Psychological Sense of School Membership scale (PSSM), using 13 of the 18 original items, appears to have a good structure, validity, and a high reliability when used to assess Latin American adolescent students. We have provided evidence that this seems to be a unidimensional scale, and using only the positively worded items could provide a clearer solution among adolescents of a wide age range.
Author Contributions
JG, CR, and EO conceived and designed the study and supervised the collection of data. JG and JM analyzed and interpreted the data, and produced the drafting of the manuscripts. RA supervised all steps in the study. All authors provided a critical revision of the manuscript.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
We thank all personal staff involved in the study. Special thanks are due to all students who participated.
References
Abubakar, A., Van De Vijver, F. J. R., Alonso-Arbiol, I., Suryani, A. O., Pandia, W. S., Handani, P., et al. (2015). Assessing sense of school belonging across cultural contexts using the PSSM: measurement and functional invariance. J. Psychoeduc. Assess. 34, 380–388. doi: 10.1177/0734282915607161
Ahn, S. (2010). The Effects of School Membership on Academic and Behavioral Performance of At-Risk Students. PhD, University of Kansas.
Alkan, N. (2016). Psychological sense of university membership: an adaptation study of the PSSM scale for Turkish university students. J. Psychol. 150, 431–449. doi: 10.1080/00223980.2015.1087373
Bachman, J. G., and O'malley, P. M. (1981). When four months equal a year: inconsistencies in student reports of drug use. Public Opin. Q. 45, 536–548. doi: 10.1086/268686
Bernard-Bonnin, A. C., Hebert, M., Daignault, I. V., and Allard-Dansereau, C. (2008). Disclosure of sexual abuse, and personal and familial factors as predictors of post-traumatic stress disorder symptoms in school-aged girls. Paediatr. Child Health 13, 479–486.
Blum, R. W., Geer, L., Hutton, L., Mckay, C., Resnick, M. D., Rosenwinkel, K., et al. (1988). The minnesota adolescent health survey. Implications for physicians. Minn. Med. 71, 143–145, 149.
Bollen, K. (1986). Sample size and bentler and Bonett's nonnormed fit index. Psychometrika 51, 375–377. doi: 10.1007/BF02294061
Bonell, C., Humphrey, N., Fletcher, A., Moore, L., Anderson, R., and Campbell, R. (2014). Why schools should promote students' health and wellbeing. BMJ 348:g3078. doi: 10.1136/bmj.g3078
Booker, K. (2007). Likeness, comfort, and tolerance: examining african american adolescents' sense of school belonging. Urban Rev. 39, 301–317. doi: 10.1007/s11256-007-0053-y
Brener, N. D., Billy, J. O., and Grady, W. R. (2003). Assessment of factors affecting the validity of self-reported health-risk behavior among adolescents: evidence from the scientific literature. J. Adolesc. Health 33, 436–457. doi: 10.1016/S1054-139X(03)00052-1
Briggs, N. E., and Maccallum, R. C. (2003). Recovery of weak common factors by maximum likelihood and ordinary least squares estimation. Multivariate Behav. Res. 38, 25–56. doi: 10.1207/S15327906MBR3801_2
Brittingham, A., Tourangeau, R., and Kay, W. (1998). Reports of smoking in a national survey: data from screening and detailed interviews, and from self- and interviewer-administered questions. Ann. Epidemiol. 8, 393–401. doi: 10.1016/S1047-2797(97)00237-8
Byrne, B. M. (2001). Structural Equation Modeling with Amos: Basic Concepts, Applications and Programming. Mahwah, NJ: Erlbaum.
Chang, W. W., Su, H., Wang, J., Wang, C. C., Shan, X. W., and Han, Q. (2013). Problem behaviours of middle school students in eastern China and its associated factors. Child Care Health Dev. 39, 660–667. doi: 10.1111/j.1365-2214.2012.01417.x
Cheung, H. (2004). Comparing shanghai and Hong Kong students' psychological sense of school membership. APER 5, 34–38. doi: 10.1007/bf03026277
Cheung, H., and Hui, S. (2003). Mainland immigrant and Hong Kong local students' psychological sense of school membership. APER 4, 67–74. doi: 10.1007/bf03025553
Comrey, A. L., and Lee, H. B. (2013). A First Course in Factor Analysis. New York, NY: Taylor & Francis.
Demanet, J., and Van Houtte, M. (2012). School belonging and school misconduct: the differing role of teacher and peer attachment. J. Youth Adolesc. 41, 499–514. doi: 10.1007/s10964-011-9674-2
Finn, J. D., and Rock, D. A. (1997). Academic success among students at risk for school failure. J. Appl. Psychol. 82, 221–234. doi: 10.1037/0021-9010.82.2.221
Florenzano, R., Pino, P., and Marchandón, A. (1993). Risk behavior in adolescent students in Santiago de Chile. Rev. Med. Chil. 121, 462–469.
Gaete, J., Olivares, E., Rojas-Barahona, C. A., Labbé, N., Rengifo, M., Silva, M., et al. (2014). Factors associated with health promoting behaviors among Chilean adolescents. Rev. Med. Chil. 142, 418–427. doi: 10.4067/S0034-98872014000400002
Goodenow, C. (1993). The psychological sense of school membership among adolescents: scale development and educational correlates. Psychol. Sch. 30, 79–90. doi: 10.1002/1520-6807(199301)30:1<79::AID-PITS2310300113>3.0.CO;2-X
Goodenow, C., and Grady, K. E. (1993). The relationship of school belonging and friends' values to academic motivation among urban adolescent students. J. Exp. Educ. 62, 60–71. doi: 10.1080/00220973.1993.9943831
Graham, J. M. (2006). Congeneric and (essentially) tau-equivalent estimates of score reliability: what they are and how to use them. Educ. Psychol. Meas. 66, 930–944. doi: 10.1177/0013164406288165
Hagborg, W. J. (1994). An exploration of school membership among middle- and high-school students. J. Psychoeduc. Assess. 12, 312–323. doi: 10.1177/073428299401200401
Hankins, M. (2008). The factor structure of the twelve item General Health Questionnaire (GHQ-12): the result of negative phrasing? Clin. Pract. Epidemiol. Ment. Health 4:10. doi: 10.1186/1745-0179-4-10
Hawkins, J. D., Guo, J., Hill, K. G., Battin-Pearson, S., and Abbott, R. D. (2001). Long-term effects of the seattle social development intervention on school bonding trajectories. Appl. Dev. Sci. 5, 225–236. doi: 10.1207/S1532480XADS0504_04
Hayton, J. C., Allen, D. G., and Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: a tutorial on parallel analysis. Organ. Res. Meth. 7, 191–205. doi: 10.1177/1094428104263675
Hedges, B., and Jarvis, M. (1998). “Cigarette smoking,” in Health Survey for England: The Health of Young People, 1995-1997, eds P. Prescott-Clarke and P. Primatesta (London: The Stationery Office), 191–221.
Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika 30, 179–185. doi: 10.1007/BF02289447
Hu, L. T., and Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct. Equ. Modeling 6, 1–55. doi: 10.1080/10705519909540118
Ibañez, G., Kuperminc, G., Jurkovic, G., and Perilla, J. (2004). Cultural attributes and adaptations linked to achievement motivation among latino adolescents. J. Youth Adolesc. 33, 559–568. doi: 10.1023/B:JOYO.0000048069.22681.2c
Irvin, M. J., Meece, J., Byun, S.-Y., Farmer, T., and Hutchins, B. (2011). Relationship of school context to rural youth's educational achievement and aspirations. J. Youth Adolesc. 40, 1225–1242. doi: 10.1007/s10964-011-9628-8
Jenkins, P. H. (1997). School delinquency and the school social bond. J. Res. Crime Delinq. 34, 337–367. doi: 10.1177/0022427897034003003
Jöreskog, K. (1977). “Factor analysis by least-squares and maximum-likelihood methods,” in Statistical Methods for Digital Computers, eds K. Enslein, A. Ralston, and H. S. Wilf (New York, NY: Wiley), 125–153.
Kuperminc, G. P., Darnell, A. J., and Alvarez-Jimenez, A. (2008). Parent involvement in the academic adjustment of Latino middle and high school youth: teacher expectations and school belonging as mediators. J. Adolesc. 31, 469–483. doi: 10.1016/j.adolescence.2007.09.003
Langford, R., Bonell, C. P., Jones, H. E., Pouliou, T., Murphy, S. M., Waters, E., Campbell, R., et al. (2014). The WHO Health Promoting School framework for improving the health and well-being of students and their academic achievement. Cochrane Database Syst. Rev. 4:CD008958. doi: 10.1002/14651858.cd008958.pub2
Langford, R., Bonell, C., Jones, H., Pouliou, T., Murphy, S., Waters, E., et al. (2015). The World Health Organization's Health Promoting Schools framework: a Cochrane systematic review and meta-analysis. BMC Public Health 15:130. doi: 10.1186/s12889-015-1360-y
Ledesma, R. D., and Valero-Mora, P. (2007). Determining the Number of Factors to Retain in EFA: an easy-to-use computer program for carrying out Parallel Analysis. Pract. Assess. Res. Eval. 12, 1–11. Available online at: http://pareonline.net/getvn.asp?v=12&n=2
Lévy, J. P., Martín, M. T., and Román, M. V. (2006). “Optimización según estructuras de covarianzas,” in Modelización con Estructuras de Covarianzas en Ciencias Sociales, eds J. P. Lévy and J. Varela (Coruña: Netbiblo), 21–22.
Libbey, H. P. (2004). Measuring student relationships to school: attachment, bonding, connectedness, and engagement. J. Sch. Health 74, 274–283. doi: 10.1111/j.1746-1561.2004.tb08284.x
Lorenzo-Seva, U. (1999). Promin: a method for oblique factor rotation. Multivariate Behav. Res. 34, 347–365. doi: 10.1207/S15327906MBR3403_3
Maddox, S. J., and Prinz, R. J. (2003). School bonding in children and adolescents: conceptualization, assessment, and associated variables. Clin. Child Fam. Psychol. Rev. 6, 31–49. doi: 10.1023/A:1022214022478
Maiti, S., and Mukherjee, B. (1990). A note on distributional properties of the Jöreskog-Sörbom fit indices. Psychometrika 55, 721–726. doi: 10.1007/BF02294619
Mardia, K. V. (1974). Applications of some measures of multivariate skewness and kurtosis in testing normality and robustness studies. Sankhya 36, 115–128.
Marsh, H. W. (1996). Positive and negative global self-esteem: a substantively meaningful distinction or artifactors? J. Pers. Soc. Psychol. 70, 810–819. doi: 10.1037/0022-3514.70.4.810
McNeely, C., and Falci, C. (2004). School connectedness and the transition into and out of health-risk behavior among adolescents: a comparison of social belonging and teacher support. J. Sch. Health. 74, 284–292. doi: 10.1111/j.1746-1561.2004.tb08285.x
Ministry of Education (Chile) (2010). SIMCE: Results 2009 (Original work in Spanish [SIMCE: Informe de Resultados 2009]). (Santiago: Gobierno de la República de Chile, MINEDUC).
Moallem, I. (2013). A Meta-Analysis of School Belonging and Academic Success and Persistence. PhD, Loyola University Chicago.
Muthen, B., and Kaplan, D. (1992). A comparison of some methodologies for the factor analysis of non-normal Likert variables: a note on the size of the model. Br. J. Math. Stat. Psychol. 45, 19–30. doi: 10.1111/j.2044-8317.1992.tb00975.x
O'farrell, S., and Morrison, G. (2003). A factor analysis exploring school bonding and related constructs among upper elementary students. Calif. Sch. Psychol. 8, 53–72. doi: 10.1007/BF03340896
Pokorski, T. L., Chen, W. W., and Bertholf, R. L. (1994). Use of urine cotinine to validate smoking self-reports in U.S. Navy recruits. Addict. Behav. 19, 451–454. doi: 10.1016/0306-4603(94)90067-1
Raykov, T. (1997). Estimation of composite reliability for congeneric measures. Appl. Psychol. Meas. 21, 173–184. doi: 10.1177/01466216970212006
Ream, R. K., and Rumberger, R. W. (2008). Student engagement, peer social capital, and school dropout among Mexican American and Non-Latino white students. Sociol. Educ. 81, 109–139. doi: 10.1177/003804070808100201
Robertson, L. M., Harding, M. S., and Morrison, G. M. (1998). A comparison of risk and resilience indicators among latino/a students: differences between students identified as at-risk, learning disabled, speech impaired and not at-risk. Educ. Treat. Children 21, 333–353.
Sánchez, B., Colón, Y., and Esparza, P. (2005). The role of sense of school belonging and gender in the academic adjustment of latino adolescents. J. Youth Adolesc. 34, 619–628. doi: 10.1007/s10964-005-8950-4
Sari, M. (2012). Sense of school belonging among elementary school students. Cukurova Univ. Faculty Educ. J. 41, 01–11.
Sawyer, S. M., Afifi, R. A., Bearinger, L. H., Blakemore, S. J., Dick, B., Ezeh, A. C., et al. (2012). Adolescence: a foundation for future health. Lancet 379, 1630–1640. doi: 10.1016/S0140-6736(12)60072-5
Senda (2014). Décimo Estudio Nacional de Drogas en la Población Escolar 2013. Santiago: Ministerio de Interior de Chile.
Shochet, I. M., Smith, C. L., Furlong, M. J., and Homel, R. (2011). A prospective study investigating the impact of school belonging factors on negative affect in adolescents. J. Clin. Child Adolesc. Psychol. 40, 586–595. doi: 10.1080/15374416.2011.581616
Simons-Morton, B. G., and Crump, A. D. (2003). Association of parental involvement and social competence with school adjustment and engagement among sixth graders. J. Sch. Health 73, 121–126. doi: 10.1111/j.1746-1561.2003.tb03586.x
Stallard, P., Spears, M., Montgomery, A. A., Phillips, R., and Sayal, K. (2013). Self-harm in young adolescents (12-16 years): onset and short-term continuation in a community sample. BMC Psychiatry 13:328. doi: 10.1186/1471-244X-13-328
Steinberg, L., and Monahan, K. C. (2007). Age differences in resistance to peer influence. Dev. Psychol. 43, 1531–1543. doi: 10.1037/0012-1649.43.6.1531
Steinberg, L., and Silverberg, S. B. (1986). The vicissitudes of autonomy in early adolescence. Child Dev. 57, 841–851. doi: 10.2307/1130361
Timmerman, M. E., and Lorenzo-Seva, U. (2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychol. Methods 16, 209–220. doi: 10.1037/a0023353
Togari, T., Sato, M., Yagasaki, Y., and Otemori, R. (2011). The development of Japanese 13-item version of psychological sense of school membership scale for Japanese urban high school students. Sch. Health 7, 62–72.
West, P., Sweeting, H., and Leyland, A. (2004). School effects on pupils' health behaviours: evidence in support of the health promoting school. Res. Pap. Educ. 19, 261–291. doi: 10.1080/02671522.2004.10058645
Ye, F., and Wallace, T. L. (2014). Psychological sense of school membership scale: method effects associated with negatively worded items. J. Psychoeduc. Assess. 32, 202–215. doi: 10.1177/0734282913504816
Keywords: validity, school membership, school bonding, substance use, adolescents, truancy
Citation: Gaete J, Montero-Marin J, Rojas-Barahona CA, Olivares E and Araya R (2016) Validation of the Spanish Version of the Psychological Sense of School Membership (PSSM) Scale in Chilean Adolescents and Its Association with School-Related Outcomes and Substance Use. Front. Psychol. 7:1901. doi: 10.3389/fpsyg.2016.01901
Received: 10 May 2016; Accepted: 21 November 2016;
Published: 06 December 2016.
Edited by:
Meryem Yilmaz Soylu, Formerly affiliated with Meliksah University, TurkeyReviewed by:
Kathy Ellen Green, University of Denver, USADouglas Kauffman, Boston University School of Medicine, USA
Copyright © 2016 Gaete, Montero-Marin, Rojas-Barahona, Olivares and Araya. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Jorge Gaete, jorge.gaete@lshtm.ac.uk; jgaete@uandes.cl