Measurement invariance of the empowering and disempowering motivational climate questionnaire-coach in youth sport

The purpose of this study was to test the measurement invariance (across five languages, two time points, and two experimental conditions) of the empowering and disempowering motivational climate questionnaire-coach (EDMCQ-C; Appleton et al., 2016) when completed by 9256 young sport participants (M age = 11.53 years, SD = 1.39 years; 13.5% female). Exploratory Structural Equation Modeling was used to test the validity of a 2-factor (empowering and disempowering) model running a multiple group analysis without any equality constraint (configural invariance) followed by measurement invariance of factor loadings and thresholds (scalar invariance). Findings provided support for partial invariance across languages and scalar invariance across time and experimental groups. The factors were interpretable across the analyses, and items loaded as intended by theory except for item 15. This study provides further evidence regarding the psychometric properties of the EDMCQ-C and suggests this scale (minus item 15) can be used to provide meaningful latent mean comparisons (Marsh et al., 2013) of empowering and disempowering coach-created climates across athletes speaking the five targeted languages, across time, and across experimental groups.

Introduction

For several decades, the coach-created motivational climate has been one of the most studied topics within sport psychology (see Duda et al., 2018). The motivational climate captures the social psychological environment that is created by significant others in sport, including coaches, and is relevant to variability in athletes’ cognitions, affect and behavior. This motivational environment concerns what the coach does, says and how he/she structures the environment in training and competitions (Duda, 2001).

Much of the work conducted on the coach-created motivational climate has been guided by achievement goal theory (AGT; Ames, 1992; Nicholls, 1989) and self-determination theory (SDT; Deci and Ryan, 1985, 2000). AGT and SDT identify specific facets of the motivational climate that have implications for athletes in and outside of sport. Whereas AGT places emphasis on task- and ego-involving climates, SDT recognizes dimensions of the climate that supports (e.g., autonomy-supportive, socially-supportive) or frustrates (e.g., controlling) an athlete’s psychological needs. Recently, Duda (2013) and Duda and Appleton (2016) proposed a multidimensional conceptualization of the coach-created motivational climate that integrates the social environmental dimensions forwarded by AGT and SDT. Duda’s framework considers that the coach-created motivational climate can be more or less ‘empowering’ and ‘disempowering’ in nature. Empowering climates are task-involving, autonomy-supportive and socially-supportive in nature and are theorized to be adaptive. Conversely, disempowering climates are ego-involving and controlling and have negative implications for athletes.

Within Duda’s conceptualization, empowering climates reflect adaptive coaching strategies emphasized by AGT and SDT. According to AGT, a task-involving climate is characterized by athletes perceiving that trying hard, skill development and cooperative learning between teammates are valued by the coach (Newton et al., 2000). Within SDT, an autonomy-supportive climate includes athletes’ preferences being recognized and their perspectives considered, their feelings are acknowledged, they are provided with meaningful choices, their input into decision-making (when and where possible) is welcomed, and a rationale is provided when they are asked to do something by the coach (Mageau and Vallerand, 2003). In a socially-supportive environment, every athlete feels cared for and is empathized with, and is valued as an athlete and as a person (Mageau and Vallerand, 2003; Reinboth et al., 2004).

In contrast, disempowering climates are marked by the negative and debilitating coaching strategies recognized in AGT and SDT. An ego-involving climate, for example, is characterized by athletes perceiving that mistakes result in punishment, the coach providing differential treatment based on the ability level of athletes, and that intra-team member rivalry is encouraged on the team (Newton et al., 2000). Relatedly, a controlling climate is evident when coaches pressure, coerce and intimidate their athletes (Bartholomew et al., 2010).

To enable researchers to measure empowering and disempowering motivational climates, Appleton et al. (2016) developed the 34-item Empowering and Disempowering Motivational Climate Questionnaire-Coach (EDMCQ-C) using data from samples of young British sport participants. The theoretical model proposed by Duda (2013; Duda and Appleton, 2016) that informed the development of the EDMCQ-C includes five lower order factors (task- and ego-involving, autonomy- and social-supportive, and controlling) and two higher-order dimensions (empowering and disempowering). Moreover, although it is possible to capture data on the motivational climate using a range of distinct albeit related sources (e.g., coach self-report; objective observations), the EDMCQ was originally developed by Appleton et al. to measure athletes’ perceptions of their coaches’ empowering and disempowering strategies in training and competition. Capturing athletes’ views of empowering and disempowering features of the coach-created motivational climate is important given they are the most proximal predictor to variability in athletes’ cognitions, affect and behaviors. Furthermore, compared to coaches’ self-reports of their motivational climate, athletes’ own reports tend to be closer to reality (i.e., what coaches are actually doing; Smith et al., 2016).

The 34 items included in the EDMCQ-C were identified from a pool of 67 statements from established measures of the five lower order climate factors using confirmatory factor analyses. Appleton et al. (2016) subsequently tested a series of factor structures using exploratory structural equation models (ESEM). ESEM integrates the principles of Exploratory Factor Analysis (i.e., items permitted to load on intended factor and crossload on non-intended factors) within the SEM framework (i.e., fit indices to assess model fit) (Asparouhov and Muthén, 2009). The first ESEM model tested by Appleton et al. was a lower-order five factor (task, autonomy-supportive, socially-supportive, controlling, ego) model. The second model tested the proposed hierarchical structure of the EDMCQ-C using Hierarchical ESEM (HESEM) in which the aforementioned lower-order dimensions were modeled onto their respective higher order factor (i.e., task, autonomy-supportive, socially-supportive loaded on an empowering higher-order dimension; control and ego loaded on a disempowering higher-order dimension). Finally, Appleton et al. also tested a bi-factor ESEM (BESEM) which is represented by two higher-order (or “general”) factors (e.g., empowering and disempowering climates), five lower-order (or “group”) factors (e.g., task- and ego-involving climates, autonomy- and social-supportive climates, and controlling climates), and a pattern matrix in which items loads onto the general and group factors.

All ESEM models resulted in acceptable model fit (e.g., CFI > 0.94; TFI > 0.91; RMSEA ≤ 0.30), with the better fit achieved via the BESEM model. However, across all ESEM models, inspection of the standardized factor loadings revealed that the majority of autonomy-supportive and some controlling and socially-supportive items failed to load significantly on their intended factor and demonstrated elevated and significant factor loadings on their non-intended climate dimension. More specifically, autonomy- and socially-supportive items loaded onto the task-involving dimension and controlling items loaded significantly onto the ego-involving factor.

In sum, the findings reported by Appleton et al. (2016) do not lend support for the multidimensional, hierarchical structure proposed to underpin the EDMCQ-C. Rather than abandon the scale, however, researchers have proposed that the scale’s structure may be best represented by two main climate factors (empowering and disempowering). This is evident in the Appleton et al. (2016) study where many of the items from the empowering lower-order dimensions (task, autonomy-supportive, socially-supportive) loaded significantly onto one factor, and many items from the disempowering lower-order dimensions (control, ego) loaded onto a second factor. This two-factor model has also been tested using ESEM (in which the 34-items are permitted to load on both factors) in subsequent research (Milton et al., 2018; Solstad et al., 2020; Sukys et al., 2020).

Although adopting a two-factor model prevents researchers from using the scale to examine the multidimensional and hierarchical structure of the climate as emphasized in Duda’s framework, adopting a two-factor approach to modeling the EDMCQ-C would still permit researchers the opportunity to examine empowering and disempowering coach-created motivational climates and examine their association with athletes’ cognitions, affect and behavior (e.g., Appleton and Duda, 2016; Smith et al., 2016; Fenton et al., 2017). In addition, this simpler two-factor model (in comparison to the HESEM and BESEM models tested by Appleton et al., 2016) would reduce the complexities associated with establishing the psychometric properties (e.g., invariance) of the scale (Milton et al., 2018; Solstad et al., 2020).

Adopting the two-factor structure supported in previous research, the main purpose of this study was to further contribute to evidence regarding the psychometric properties of the EDMCQ. This was achieved by using ESEM to test measurement invariance using data from young football players from five European countries (i.e., France, Greece, Norway, Spain, and England) who spoke the main language in each country (e.g., English in England; Spanish in Spain etc.), across time (start and end of season), and between experimental conditions (intervention and control groups).

Materials and methods

Participants

The data come from the large European-wide PAPA project (see Duda et al., 2013) that sought to promote healthy and sustained physical activity in children and adolescents via participation in grassroots football. Player questionnaire data from the PAPA project on the motivational climate has been used in other studies, albeit limited to Time 1 data in England (Appleton et al., 2016) or one facet of the climate (i.e., autonomy-support) in five countries (Quested et al., 2013). 9256 football players (13.5% female) aged between 9 and 15 years (mean = 11.53, SD = 1.39) from five European countries (France = 1426, Greece = 1707, Norway = 1998, Spain = 2335, and the England = 1790) participated in this study. The participants played for 638 teams at the grassroots level, reported to have been playing for their teams between the present season and the last 10 seasons (median = 3 seasons, interquartile range = 4), and trained between 0.5 and 10 h a week (median = 4, interquartilerange = 2). 61.4% of the participants were from the intervention group in the PAPA project. The intervention (Empowering Coaching™; see Duda, 2013) involved the participants’ coaches attending a 6-hour education workshop concerning the creation of empowering (and less disempowering) motivational climates in training and matches. The coaches of the participants in the control condition engaged in their normal coaching practices. See Table 1 for an overview of participants’ demographic information by country.

TABLE 1

Table 1. Participants’ demographic information by country.

Procedures

The protocol outlining the procedures associated with data collection in the PAPA project were described in Duda et al. (2013) and employed in previous studies (e.g., Quested et al., 2013). Ethical approval was granted by the Universities of each team of researchers working on the PAPA project. Prior to collecting data, parents had the opportunity to opt out their son and/or daughter from the project. Participants were fully informed about the study before it took place, and a parental written opt out form was used in accordance with national legislation and the institutional requirements.

The EDMCQ-C was one of a number of scales included in the overall player questionnaire employed in the PAPA project. Time 1 data was collected at the start of the competitive football season (spring 2011 in Norway and between autumn and winter 2011/2012 in the other countries). Time 2 data was collected at the end of the same season, approximately 20–28 weeks after Time 1 (in late summer 2011 in Norway, and spring 2012 in the other countries). Data was collected by trained research assistants working on the PAPA project. Each participants completed the questionnaire individually albeit were grouped with their teammates typically during a training session. The overall questionnaire took between 20–40 min to complete, and the EDMCQ-C took between 5–10 min (see Duda et al., 2013 for further details).

Measure

All text in the questionnaire was initially drafted in English, translated into Spanish, Norwegian, Greek, and French by a native speaker, and then back-translated into English by a second dual-language speaker. The translation-back translation procedure was based on the recommendations from mainstream (Harkness, 1999; Hambleton, 2005) and sport psychology literature (Duda and Hayashi, 1998).

The EDMCQ-C (Appleton et al., 2016) includes 17 empowering items measuring task-involving (e.g., “My coach encouraged athletes to try new skills”), autonomy-supportive (e.g., “My coach gave athletes choices and options”) and socially-supportive (e.g., “My coach really appreciated athletes as people, not just as a sport participants”) coaching. 17 disempowering items are also included measuring ego-involving (e.g., “My coach yelled at athletes for messing up”) and controlling (e.g., “My coach paid less attention to athletes if they displeased him or her”) climate dimensions. Participants were instructed to “think about what it has usually been like on this team/club during the last 3–4 weeks” when providing their responses, which were measured on a 5-point scale (i.e., 1 = strongly disagree, 5 = strongly agree).

Data management and analysis

The data were validated and screened for patterns of missing values prior to the creation of an international file for main analysis. Although the majority of the sample (9194 cases) responded to the EDMCQ-C at Time 1 and/or Time 2, only 35% of the sample had complete data for all items and both time points. To detect any bias attributable to data missingness, we analyzed the data twice: first using the whole sample and second with the subsample that provided complete data. As the results of both analyses were similar, only the results obtained using the whole sample are presented.

To test for measurement invariance, target rotation (Asparouhov and Muthén, 2009; Marsh et al., 2013) in the ESEMs conducted in Mplus (Muthén and Muthén, 1998–2015) were employed. A target rotation consists of defining which factor loadings will be freely estimated (those for items on their intended factor) and which factor loadings will be restricted to values as close as possible to zero (those for items on their non-intended factor/s; Muthén, and Muthén, 1998-2017). In our study, task-involving, autonomy-supportive and socially-supportive items were freely estimated on an empowering factor and permitted to cross-load on the disempowering factor but with factor loadings restricted to values close as possible to zero. Controlling and ego involving items were freely estimated on the disempowering factor and permitted to cross-load on the empowering factor but with factor loadings restricted to values close as possible to zero. We followed the recommendations proposed by Millsap and Yun-Tein (2004) for categorical variables, and Marsh et al. (2013) and Marsh et al. (2020) for ESEMs. First, we tested the validity of a 2-factor model in each country using ESEM running a multiple group analysis without any equality constraint (configural invariance). We then tested measurement invariance of factor loadings and thresholds (scalar invariance; Muthén and Muthén, 1998–2015; Wang and Su, 2013) across countries, time, and experimental condition in separate steps. Total or partial scalar invariance of items ensures meaningful latent mean comparisons across groups and over time (e.g., Marsh et al., 2013).

Due to the complexity of the analyses (5 countries * 2 conditions * 2 waves), we did not employ the “interactional” tests of invariance outlined in Grouzet et al. (2006). Moreover, we did not have reason to test invariance in any specific order (Wang and Su, 2013). For the test of invariance across time, we correlated the error terms from the same items on different occasions to account for within subjects’ data (Marsh and Hau, 1996). In all analyses, factor loadings and thresholds were constrained to be equal in tandem as it is recommended for categorical-ordered indicators (Muthén and Muthén, 1998–2015).

Due to the categorical nature of the data and the presence of missing values (see Results section), the weighted least-squares mean and variance-adjusted estimator was used with pairwise deletion for missing values, both of them being the Mplus defaults for categorical data. The Goodness-of-fit Indices were χ2, Comparative Fit Index (CFI), Tucker–Lewis Index (TLI), and root mean square error of approximation (RMSEA). In an independent clusters confirmatory factor analysis model with quantitative indicators, CFI and TLI values >0.95 and RMSEA <0.06 are considered as indicators of excellent fit (Hu and Bentler, 1995), and CFI and TLI values >0.90 and RMSEA <0.08 are considered as indicators of acceptable fit (Marsh et al., 2004). Little simulation data are available on the behavior of these cut-off values in categorical data ESEM analysis, but Yu (2002) suggested using a CFI >0.96 for categorical data and most papers using ESEM, including categorical data ESEM (Myers et al., 2011), rely on them with some caution (but see Maydeu-Olivares et al., 2017). Standard errors and fit indices were calculated taking into account that players’ responses were clustered within their teams.

In order to compare nested models, we employed ad hoc guidelines to evaluate differences in fit, including the difference in CFI (ΔCFI) and RMSEA (ΔRMSEA). As a cut-off value, a ΔCFI < 0.01 (Cheung and Rensvold, 2002) and changes in RMSEA <0.015 (Chen, 2007) are considered as evidence for the more parsimonious model. Marsh (2007) also proposed that the more parsimonious models is supported if the TLI and RMSEA are as good as or better than that for the more complex model. However, Marsh also proposed that these above proposals for assessing invariance should be considered as rough guidelines rather than golden rules. All these recommendations were considered in this paper.

Regarding reliability, we followed the advice of Viladrich et al. (2017) regarding the coefficient H, also known as maximal reliability (Raykov, 2012), that should be used to get an adequate reliability estimate when the intended measures are factors instead of unweighted composite measures. As H is strongly related to the size of factor loadings but not directly applicable to ordinal data, we used the size of the factor loadings to gauge the reliability of the EMP and DISEMP factor scores.

Results

Responses to the questionnaire showed sizeable floor (disempowering items) or ceiling (empowering items) effects (see Table 2), confirming the decision to treat the data as categorical using the WLSMV estimator in MPlus. As reported in Table 2, the most frequent response in each country was “agree” (16–52% of responses) and “strongly agree” (17–72% of responses) for the majority of the empowering items at both time points, with fewer participants responding “strongly disagree” (1–12% of responses) or “disagree” (1–13% of responses). One exception was empowering item three “My coach offers choices and options”, where the proportion of the French sample was similar across the five response categories. For the disempowering items, the proportion of the sample was higher for the “strongly disagree” (10–57% of responses), “disagree” (11–39% of responses) and “neutral” (13–38% of responses) responses across the countries and at each time point with fewer participants responding “strongly agree” (1–7% of responses) or “agree” (5–33% of responses), albeit there were some exceptions. For example, for disempowering item 15 “My coach only allowed something we like to do at the end of training if players had done well during the session,” there was a larger proportion of the sample who responded “agree” and fewer who responded “strongly disagree” compared to the other disempowering items.

TABLE 2

Table 2. Percentage of responses to each category by time and language.

Language invariance

At Time 1, indexes of fit for configural invariance across languages were: χ²(2470) = 5932.68, CFI = 0.952, TLI = 0.945, RMSEA = 0.030 (CI 95% = 0.029–0.031) and scalar invariance χ²(3126) = 10644.48, CFI = 0.895, TLI = 0.906, RMSEA = 0.039 (CI 95% = 0.039–0.042). Results were similar for Time 2: indexes of fit for configural invariance were: χ²(2470) = 5908.73, CFI = 0.953, TLI = 0.946, RMSEA = 0.034 (CI95% = 0.032–0.035) and scalar invariance χ²(3126) = 9362.04, CFI = 0.914, TLI = 0.923, RMSEA = 0.040 (CI95% = 0.039–0.041). Although ΔRMSEA met the adopted criteria of <0.015 (Chen, 2007), ΔCFIs were > 0.01 and TLI and RMSEA values were not as good or better for scalar invariance compared to configural invariance (see Table 3). Inspection of the modification indices revealed non-invariant thresholds for a number of items. We therefore freed the thresholds of these items in partially invariant models. At Time 1, indexes of fit for partial invariant model were χ²(3086) = 8950.36, CFI = 0.918, TLI = 0.926, RMSEA = 0.035 (CI95% = 0.034–0.036) and Time 2 χ²(3086) = 7966.11, CFI = 0.933, TLI = 0.939, RMSEA = 0.036 (CI95% = 0.035–0.037) (see Table 3).

TABLE 3

Table 3. Model fit of the approximate invariance test across language, time and condition.

Non-invariant thresholds emerged for nine items (see Figure 1). Empowering item three was non-invariant in French with predicted proportions of responses more uniformly distributed across all categories compared to the other countries with distributions relatively skewed towards agreeing with this item. Disempowering items 10 and 15 were both more uniformly distributed for Greek than for the other countries. Empowering item 13 and disempowering items nine, 20 and 25 were more uniformly distributed in Norwegian. Disempowering items 25 and 29 were relatively more uniformly distributed in English. As an exception, item 31 in Spanish was non-invariant due to a predicted distribution of responses more skewed toward strongly disagree responses compared to the other languages.

FIGURE 1

Figure 1. Predicted proportions of responses for non-invariant items from the empowering and disempowering motivational climate questionnaire-coach (EDMCQ-C). NBE, empowering item; D, disempowering item. F, French; G, Greek; N, Norwegian; S, Spanish; En, English; 0, strongly disagree; 1, disagree; 2, neutral; 3, agree; 4, strongly agree.

Non-standardized factor loadings were statistically invariant across languages, thus only factor loadings for the reference language (i.e., English) are reported (see Table 4) in a standardized form. They revealed all empowering and disempowering items loaded positively and more strongly on their intended factor than on the non-intended factor at times 1 and 2 except for disempowering item 15 (“My coach only allowed something we like to do at the end of training if players had done well during the session”) which loaded more strongly onto the empowering factor in all countries. For the non-reference languages (i.e., Spanish, Norwegian, Greek, French), standardized factor loadings showed no major discrepancies (around 0.10) in each factor loading, except for autonomy-supportive item one (“My coach gave players choices and options”) in French with standardized factor loadings of 0.15 at Time 1 and 0.10 at Time 2. Both values were lower than those of the other countries which ranged from 0.40 to 0.57 at Time 1 and from 0.46 to 0.61 at Time 2. Overall, the majority of factor loadings were high (>0.50) suggesting indirect evidence for internal consistency (reliability) for the empowering and disempowering latent variables across the five countries.

TABLE 4

Table 4. Standardized factor loadings for the 34 items from the empowering and disempowering motivational climate questionnaire-coach (EDMCQ-C) across language, time, and condition.

Finally, the correlation between the empowering and disempowering factor at Time 1 was –0.48 (p < 0.001) in English, –0.67 (p < 0.001) in Spanish, –0.68 (p < 0.001) in Norwegian, –0.28 (p < 0.001) in Greek, and –0.36 (p < 0.001) in French. At Time 2, the correlation was –0.58 (p < 0.001) in English, –0.82 (p < 0.001) in Spanish, –0.66 (p < 0.001) in Norwegian, –0.48 (p < 0.001) in Greek, and –0.67 (p < 0.001) in French.

Time invariance

Indexes of fit for configural invariance were χ²(2106) = 8982.89, CFI = 0.937, TLI = 0.931, RMSEA = 0.019 (CI95% = 0.018–0.019) and scalar invariance χ²(2307) = 10229.91, CFI = 0.927, TLI = 0.928, RMSEA = 0.019 (CI95% = 0.019–0.020). ΔCFI = 0.01, ΔRMSEA <0.015 and TLI and RMSEA values were as good for scalar invariance compared to configural invariance, offering support for scalar invariance (see Table 3).

Non-standardized factor loadings were statistically invariant across time, thus only factor loadings for Time 1 are reported in standardized form. Inspection of the factor structure revealed that empowering and disempowering factors were clearly distinguishable, with 33 items loading ranging from 0.35 to 0.67 (p < 0.001) on the intended factor (see Table 4). Standardized factor loadings of the non-intended factor ranged from –0.14 to 0.38. As per the language analyses, item 15 loaded more strongly on the empowering factor (0.383) than disempowering factor (0.206). However, as the majority of factor loadings were high (>0.50) for items on their intended factor, there is indirect evidence for internal consistency (reliability) for the empowering and disempowering latent variables across time. The correlations between the error terms from the same items on different occasions ranged from 0.04 to 0.38.

The correlation between the empowering climate factors at Time 1 and Time 2 was 0.52 (p < 0.001), and between disempowering climate factors at Time 1 and Time 2,0.55 (p < 0.001). The correlation between the empowering and disempowering factors was –0.51 (p < 0.001) at Time 1 and –0.55 (p < 0.001) at Time 2. The correlation between empowering factor at Time 1 and disempowering factor at Time 2 was –0.30 (p < 0.001). Finally, the correlation between the disempowering factor at Time 1 and the empowering factor at Time 2 was –0.34 (p < 0.001).

Condition invariance

Indexes of fit for configural invariance across condition at Time 1 one were χ²(988) = 5997.15, CFI = 0.925, TLI = 0.915, RMSEA = 0.034 (CI95% = 0.033–0.035) and scalar invariance χ²(1152) = 4465.67, CFI = 0.945, TLI = 0.947, RMSEA = 0.027 (CI95% = 0.026–0.028). ΔCFI = 0.02, ΔRMSEA < 0.015 and TLI and RMSEA values were as good for scalar invariance compared to configural invariance, offering support for scalar invariance. Likewise, indexes of fit for configural invariance at Time 2 were χ²(988) = 5656.37, CFI = 0.924, TLI = 0.913, RMSEA = 0.039 (CI95% = 0.038–0.040) and scalar invariance were χ²(1152) = 4195.22 CFI = 0.950, TLI = 0.952, RMSEA = 0.029 (CI95% = 0.028–0.030). ΔCFI = 0.026, ΔRMSEA < 0.015 and TLI and RMSEA values were as good for scalar invariance compared to configural invariance, offering support for scalar invariance (see Table 3).

Non-standardized factor loadings were statistically invariant across the condition groups, thus only standardized factor loadings for the control group are reported. Standardized factor loadings (see Table 4) revealed all empowering and disempowering items for the control group loaded positively and strongly on their intended factor. Standardized factor loadings for empowering items ranged from 0.41 to 0.65 (p < 0.001) (Time 1) and 0.53 to 0.70 (p < 0.001) (Time 2) on the empowering factor (see Table 4) and –0.12 to 0.21 (Time 1) and –0.13 to 0.19 (Time 2) on the non-intended disempowering factor. The standardized factor loadings for the disempowering items ranged from 0.18 to 0.63 (p < 0.001) (Time 1) and 0.20 to 0.73 (p < 0.001) (Time 2) on the disempowering factor and –0.16 to 0.35 (Time 1) and –0.11 to 0.40 (p < 0.001) (Time 2) on the non-intended empowering factor. Item 15 loaded positively and more strongly onto the empowering compared to the disempowering factor.

For the non-reference (intervention) group, standardized factor loadings for empowering items ranged from 0.41 to 0.65 (p < 0.001) (Time 1) and 0.53 to 0.65 (p < 0.001) (Time 2) on the empowering factor and –0.14 to 0.20 (Time 1 and 2) on the disempowering factor. The standardized factor loadings for the disempowering items ranged from 0.20 to 0.61 (p < 0.001) (Time 1) and 0.21 to 0.71 (p < 0.001) (Time 2) on the disempowering factor and –0.14 to 0.34 (Time 1) and –0.10 to 0.37 (Time 2) on the empowering factor. Again, item 15 loaded positively and more strongly onto the empowering compared to the disempowering factor. As the majority of factor loadings for items on their intended factor were above 0.50, indirect support for the internal consistency of the empowering and disempowering latent variables across conditions is provided.

Finally, the correlation between the empowering and disempowering factor was –0.48 (p < 0.001) in the control condition and –0.40 (p < 0.001) in the experimental condition at Time 1. At Time 2, the correlation was –0.58 (p < 0.001) in the control condition and –0.53 (p < 0.001) in the experimental condition.

Discussion

The purpose of this study was to further contribute to the development and validation process of the EDMCQ-C (Appleton et al., 2016) by exploring measurement invariance of a two-factor model across five languages, longitudinally, and across intervention groups in a multi-national sample of young footballer players.

In testing for invariance, total or partial scalar invariance is required to ensure future comparisons of latent mean scores (across the groups and over time) are meaningful. ESEM analyses indicated that the two-factor model tested in this study showed partial invariance across language and full scalar invariance across time and experimental condition groups. Moreover, the analyses confirmed that the majority (33 of 34) of items from the EDMCQ-C loaded positively onto their intended factor (with lower scores on the non-intended factor). Taken together, the invariance findings and factor loadings reported in this study provide additional evidence that the EDMCQ-C provides a sound measure of the array of coaching strategies central to Duda (2013) theory-informed model of the motivational climate. The questionnaire can also be used with confidence to provide meaningful comparisons of latent mean empowering and disempowering climate values in the sample of junior participants recruited in the PAPA project.

Item 15 (“My coach only allowed something we like to do at the end of training if players had done well during the session”) was the only item that failed to load as hypothesized across all tests of invariance in this study. Item 15 attempts to capture a controlling reward that coaches may employ during training, and loaded most strongly on the empowering factor across all analyses. This finding is consistent with the ESEM findings reported by Appleton et al. (2016) and Milton et al. (2018). Thus, it seems that junior athletes in the PAPA project may not interpret this particular coaching strategy as disempowering as intended by the scale’s authors. Rather, the young footballers seem to perceive that being allowed to do a favorite activity at the end of a training session on those occasions when they had done well in the session is an empowering coaching strategy.

There are a number of reasons that item 15 may load most strongly onto the empowering factor in this study. First, doing something “we like to do at the end of training” is most likely an activity (or activities) that the players enjoy, and thus creates the perception of a positive coaching strategy. Second, empowering coaching strategies are considered key predictors of the satisfaction of athletes’ basic psychological needs (Duda, 2013), including feelings of competence. It may be that the young footballers perceive the reward emphasized by item 15 as recognition for demonstrating sufficient level of performance (i.e., doing well), which may subsequently contribute to them feeling more competent. Finally, the “done well during the session” aspect of the item may have been interpreted by the athletes in a task-involving manner, reflecting their personal development, skill mastery and application of effort. Based on the findings reported here (as well as Appleton et al., 2016), we recommend that additional studies utilizing the data from the PAPA project proceed without the inclusion of item 15 from the EDMCQ-C. We also recommend that future research employing the EDMCQ should examine whether item15 cross-loads (more strongly) onto the empowering (compared to the disempowering) factor using data provided by junior sport participants from alternative (non-PAPA) samples and countries.

Although we found partial scalar invariance across languages, there were also non-invariant thresholds of nine items in this particular analyses. As highlighted above, non-invariant thresholds do not prevent a researcher from making meaningful comparisons of latent mean scores. However, a closer inspection of the results for eight of the nine items with non-invariant thresholds revealed differences in the proportion of athletes from each country for the less extreme responses (i.e., “disagree,” “neutral,” and “agree”). For example, regarding item three (“My coach gave players choices and options”), the analyses revealed that a larger proportion of the athletes in France responded “strongly disagree” and “disagree,” compared to athletes speaking the four other languages. This particular finding suggests that French speaking young footballers are given (or perceived to be given) fewer choices and options by their coaches during training and competition. Other studies (e.g., Tessier et al., 2013; Smith et al., 2016) assessing empowering and disempowering coach-created motivational climates have also revealed that French-speaking coaches provide lower levels of autonomy-supportive coaching compared to English, Greek and Spanish speaking athletes. The current study helps clarify the findings reported by Tessier et al. (2013) and Smith et al. (2016) by suggesting the differences in autonomy-support may lie in the lower choice and options provided by French speaking coaches. Conversations between the researchers and the participants during the data collection in the PAPA project provide an anecdotal explanation for this finding. Specifically, some of the French-speaking coaches perceived that giving their players choices and options was a sign of weakness and lack of authority, and the players reported that they did not understand choices their coaches could offer. Given that offering meaningful choices and attractive options is a key facet of an empowering climate, future intervention work with French-speaking coaches may wish to target this particular empowering strategy.

Regarding item 31 (“My coach tried to interfere in aspect of players’ lives outside of football”), the analyses revealed that Spanish speaking players’ responses were more optimistic (i.e., more players responding “strongly disagree”) compared to players from the other countries (albeit more athletes from the other countries disagreed than agreed with this statement). Why more Spanish speaking players responded strongly disagreed is not clear in the current study. It would be interesting to determine in future research whether this particular finding emerges when examining language invariance of the EDMCQ-C with athletes from other sports and competitive levels. Consistent support for this finding across other sports would suggest coaches in Spain are less disempowering with respect to interfering in their athletes’ lives outside of sport compared to coaches in other countries.

While the findings of this study are informative regarding the psychometric properties of the EDMCQ-C, we also recognize some limitations. For example, we were unable to test for gender invariance due to a gender imbalance in the sample and because in some cases, boys and girls were nested within the same team (i.e., they played for the same team). Future research may wish to test the gender invariance of the EDMCQ when completed by young athletes, and well as replicating the analyses reported in this study with a more diverse sample of athletes (e.g., older athletes, range of sports, more diverse range of languages). Consistent with recommendations (Muthén and Muthén, 1998–2015) for categorical-ordered data, we made the decision to constrain the factor loadings and thresholds in tandem, freeing the latter when testing partial (language) invariance. As a result, we did not test for factor loading invariance. We made this decision, as per previous studies (e.g., Myers et al., 2011; Viladrich et al., 2013), due to the complexities associated with managing factor loading non-invariance attributable to one language on one item in an ESEM. Moreover, with categorical data, the factor loadings and thresholds are not independent as both contribute to the item characteristic function. Based on the difficulties reported by Appleton et al. (2016) in modeling the hierarchical structure of the EDMCQ-C, we also decided to limit our analyses to a two-factor model that reflected the overall empowering and disempowering features of the coach-created motivational climate. However, if advances are made in refining the content of the EDMCQ-C to enable the successful modeling of the scale’s hierarchical nature, future research will be needed to re-examine the scale’s invariance. Finally, future research may wish to include interactional invariance tests (e.g., across countries, experimental conditions, time points, gender), which was not possible in this study due to the complexity of the analyses and the nature of our data (missing data, sample size and skewness).

Conclusion

Collectively, the results from this study provide further evidence on the psychometric properties of the EDMCQ-C as a measure of the empowering and disempowering features of the coach-created motivational climate in youth sport. Overall, the findings suggest the scale (minus item 15) may be used to provide meaningful latent mean comparisons (Marsh et al., 2013) across the five targeted languages, time, and experimental groups when utilizing the data from the PAPA project. This is especially important given the PAPA project attempted to help coaches create empowering (and reduced disempowering) motivational climates in youth sport. The findings reported in this study should also increase researchers’ confidence in using the EDMCQ-C as a measure of athletes’ perceptions of empowering and disempowering coach-created motivational climate in future research. Such research could use the EDMCQ-C to test theory-informed process models (see Duda and Appleton, 2016; Duda et al., 2018) that include the two climate factors proposed in this study, and such research would also contribute to the nomological validity of the EDMCQ-C.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

Ethical approval was granted by the Universities of each team of researchers working on the PAPA project. Prior to collecting data, parents had the opportunity to opt-out their son and/or daughter from the project. Participants were fully informed about the study prior to it taking place and a parental written opt-out form was used in accordance with national legislation and the institutional requirements.

Author contributions

PA and JD: conceived the study. PA, EQ, and JD: designed the original questionnaire. IB and LG-G: designed the Spanish version of the questionnaire. PS and J-PH: designed the French version of the questionnaire. AP: designed the Greek version of the questionnaire. YO: designed the Norwegian version of the questionnaire. JD, IB, AP, HH, PS, J-PH, YO, BW, and OS: funding acquisition. JD: supervised the manuscript. PA, EQ, LG-G, AP, HH, IB, YR, PS, J-PH, YO, BW, OS, and JD: collected the data. PA and CV: analyzed the data. PA, CV, and JD: interpreted the results of the research. CV: designed the figures and tables. PA and CV: wrote and edited the manuscript. All authors provided critical revisions and the formal analysis on the successive drafts and approved the manuscript in its final form.

Funding

This research received funding from the European Community’s Seventh Framework Programme FP7/2007e2013 (grant agreement no. 223600).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Ames, C. (1992). “Achievement goals, motivational climate, and motivational processes,” in Motivation in Sport and Exercise, ed. G. C. Roberts (Champaign, IL: Human Kinetics), 161–176.

Google Scholar

Appleton, P. R., and Duda, J. L. (2016). Examining the interactive effects of coach-created empowering and disempowering climate dimensions on athletes’ health and functioning. Psychol. Sport Exer. 26, 61–70. doi: 10.1016/j.psychsport.2016.06.007

CrossRef Full Text | Google Scholar

Appleton, P. R., Ntoumanis, N., Quested, E., Viladrich, C., and Duda, J. L. (2016). Initial validation of the coach-created Empowering and Disempowering Motivational Climate Questionnaire (EDMCQ-C). Psychol. Sport Exer. 22, 53–65. doi: 10.1016/j.psychsport.2015.05.008

CrossRef Full Text | Google Scholar

Asparouhov, T., and Muthén, B. O. (2009). Exploratory structural equation modeling. Struct. Equ. Modeling 16, 397–438. doi: 10.1080/10705510903008204

CrossRef Full Text | Google Scholar

Bartholomew, K., Ntoumanis, N., and Thøgersen-Ntoumani, C. (2010). The controlling interpersonal style in a coaching context: development and initial validation of a psychometric scale. J. Sport Exerc. Psychol. 32, 193–216. doi: 10.1123/jsep.32.2.193

PubMed Abstract | CrossRef Full Text | Google Scholar

Chen, F. F. (2007). Sensitivity of goodness of fit indices to lack of measurement invariance. Struct. Equ. Modeling. 14, 464–504. doi: 10.1080/10705510701301834

CrossRef Full Text | Google Scholar

Cheung, G. W., and Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Struct. Equ. Modeling 9, 233–255. doi: 10.1207/S15328007SEM0902_5

PubMed Abstract | CrossRef Full Text | Google Scholar

Deci, E. L., and Ryan, R. M. (1985). Intrinsic Motivation and Self-Determination in Human Behavior. New York, NY: Plenum.

Google Scholar

Deci, E. L., and Ryan, R. M. (2000). The “what” and “why” of goal pursuits: Human needs andthe self-determination of behavior. Psychol. Inquiry 11, 227–268. doi: 10.1207/S15327965PLI1104_01

CrossRef Full Text | Google Scholar

Duda, J. L. (2001). “Achievement goal research in sport: Pushing the boundaries and clarifying some misunderstandings,” in Advances in motivation in sport and exercise, ed. G. C. Roberts (Leeds: Human Kinetics), 129–182.

Google Scholar

Duda, J. L. (2013). The conceptual and empirical foundations of Empowering Coaching™: Setting the stage for the PAPA project. Int. J. Sport Exerc. Psychol. 11, 311–318. doi: 10.1080/1612197X.2013.839414

PubMed Abstract | CrossRef Full Text | Google Scholar

Duda, J. L., and Appleton, P. R. (2016). “Empowering and disempowering coaching behaviours: Conceptualisation, measurement considerations, and intervention implications,” in Sport and Exercise Psychology Research: From Theory to Practice, eds M. Raab, P. Wylleman, R. Seiler, A.-M. Elbe, and A. Harzigeorgiadis (Amsterdam: Elsevier), 373–388. doi: 10.1016/B978-0-12-803634-1.00017-0

CrossRef Full Text | Google Scholar

Duda, J. L., Appleton, P. R., Stebbings, J., and Balaguer, I. (2018). “Toward more empowering and less disempowering environments in youth sport: Theory to evidenced-based practice,” in Sport Psychology for Young Athletes, eds C. J. Knight, C. G. Harwood, and D. Gould (New York, NY: Routledge), 81–93.

Google Scholar

Duda, J. L., and Hayashi, C. T. (1998). “Measurement issues in cross-cultural research within sport and exercise psychology,” in Advances in Sport and Exercise Psychology Measurement, ed. J. L. Duda (Morgantown, WV: Fitness Information Technology), 471–483. doi: 10.1186/s12913-016-1423-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Duda, J. L., Quested, E., Haug, E., Samdal, E., Wold, B., and Balaguer, I. (2013). Promoting adolescent health through an intervention aimed at improving the quality of their participation in Physical Activity (‘PAPA’): Background to the project and main trial protocol. Int. J. Sport .Exerc. Psychol. 11, 319–327. doi: 10.1080/1612197X.2013.839413

CrossRef Full Text | Google Scholar

Fenton, S. A. M., Duda, J. L., Appleton, P. R., and Barrett, T. G. (2017). Empowering youth sport environments: implications for daily moderate-to-vigorous physical activity and adiposity. J. Sport Health Sci. 6, 423–433. doi: 10.1016/j.jshs.2016.03.006

PubMed Abstract | CrossRef Full Text | Google Scholar

Grouzet, F. M., Otis, N., and Pelletier, L. G. (2006). Longitudinal cross-gender factorial invariance of the Academic Motivation Scale. Struct. Equ. Modeling 13, 73–98. doi: 10.1207/s15328007sem1301_4

CrossRef Full Text | Google Scholar

Hambleton, R. K. (2005). “Issues, designs and technical guidelines for adapting tests into multiple languages and cultures,” in Adapting Psychological and Educational Tests for Cross-Cultural Assessment, eds R. K. Hambleton, P. F. Merenda, and C. D. Spielberger (Mahwah, NJ: Lawrence Erlbaum), 3–38.

Google Scholar

Harkness. (1999). In pursuit of quality: Issues for cross-national survey research. Int. J. Soc. Res. Methodol. 2, 124–140. doi: 10.1080/136455799295096

CrossRef Full Text | Google Scholar

Hu, L.-T., and Bentler, P. M. (1995). “Evaluating model fit,” in Structural equation modeling: Concepts, issues, and applications, ed. R. H. Hoyle (Thousand Oaks, CA: Sage Publications, Inc), 76–99.

Google Scholar

Mageau, G. A., and Vallerand, R. J. (2003). The coach-athlete relationship: A motivational model. J. Sports Sci. 21, 883–904. doi: 10.1080/0264041031000140374

PubMed Abstract | CrossRef Full Text | Google Scholar

Marsh, H. W. (2007). “Application of confirmatory factor analysis and structural equation modeling in sport and exercise psychology,” in Handbook of Sport Psychology, 3rd Edn, eds G. Tenenbaum and R. C. Eklund (Hoboken, NJ: John Wiley & Sons Inc), 774–798. doi: 10.1123/jsep.2014-0330

PubMed Abstract | CrossRef Full Text | Google Scholar

Marsh, H. W., Guo, J., Dicke, T., Parker, P. D., and Craven, R. G. (2020). Confirmatory Factor Analysis (CFA), Exploratory Structural Equation Modeling (ESEM), and Set-ESEM: Optimal Balance Between Goodness of Fit and Parsimony. Multivariate Behav. Res. 55, 102–119. doi: 10.1080/00273171.2019.1602503

PubMed Abstract | CrossRef Full Text | Google Scholar

Marsh, H. W., and Hau, K.-T. (1996). Assessing goodness of fit: is parsimony always desirable? J. Exp. Educ. 64, 364–390. doi: 10.1080/00220973.1996.10806604

CrossRef Full Text | Google Scholar

Marsh, H. W., Hau, K. T., and Wen, Z. (2004). In search of golden rules: Comment on hypothesis testing approaches to setting cutoff values for fit indexes and dangers in overgeneralising Hu & Bentler’s (1999) findings. Struct. Equ Model. 11, 320–341. doi: 10.1207/s15328007sem1103_2

CrossRef Full Text | Google Scholar

Marsh, H. W., Nagengast, B., and Morin, A. J. S. (2013). Measurement invariance of Big-Five factors over the life span: ESEM Tests of gender, age, plasticity, maturity, and la dolce vita effects. Dev. Psychol. 49, 1194–1218. doi: 10.1037/a0026913

PubMed Abstract | CrossRef Full Text | Google Scholar

Maydeu-Olivares, M., Fairchild, A. J., and Hall, A. G. (2017). Goodness of fit in item factor analysis: Effect of the number of response alternatives. Struct. Equ. Modeling 24, 495–505. doi: 10.1080/10705511.2017.1289816

CrossRef Full Text | Google Scholar

Millsap, R. E., and Yun-Tein, J. (2004). Assessing factorial invariance in ordered-categorical measures. Multivariate Behav. Res. 39, 479–515. doi: 10.1207/S15327906MBR3903_4

CrossRef Full Text | Google Scholar

Milton, D., Appleton, P. R., Bryant, A., and Duda, J. L. (2018). Initial validation of the teacher-created empowering and disempowering motivational climate questionnaire in physical education. J. Teach. Phys. Educ. 37, 340–351. doi: 10.1123/jtpe.2018-0119

CrossRef Full Text | Google Scholar

Muth, L. K., and Muth, B. O. (1998–2015). Mplus User’s Guide, 7th Edn. Los Angeles, CA: Muthén & Muthén.

Google Scholar

Myers, N. D., Chase, M. A., Pierce, S. W., and Martin, E. (2011). Coaching efficacy and exploratory structural equation modeling: A substantive-methodological synergy. J. Sport Exerc. Psychol. 33, 779–806. doi: 10.1123/jsep.33.6.779

PubMed Abstract | CrossRef Full Text | Google Scholar

Newton, M., Duda, J. L., and Yin, Z. N. (2000). Examination of the psychometric properties of the Perceived Motivational Climate in Sport Questionnaire-2 in a sample of female athletes. J. Sports Sci. 18, 275–290. doi: 10.1080/026404100365018

PubMed Abstract | CrossRef Full Text | Google Scholar

Nicholls, J. G. (1989). The Competitive Ethos and Democratic Education. Cambridge, MA: Harvard University Press.

Google Scholar

Quested, E., Ntoumanis, N., Viladrich, C., Haug, E., Ommundsen, Y., Van Hoye, A., et al. (2013). Intentions to drop-out of youth soccer: A test of the basic needs theory among European youth from five countries. Int. J. Sport Exerc. Psychol. 11, 395–407. doi: 10.1080/1612197X.2013.830431

CrossRef Full Text | Google Scholar

Raykov, T. (2012). “Scale construction and development using structural equation modeling,” in Handbook of structural equation modeling, ed. R. H. Hoyle (New York, NY: Guilford Press), 472–492.

Google Scholar

Reinboth, M., Duda, J. L., and Ntoumanis, N. (2004). Dimensions of coaching behavior, need satisfaction, and the psychological and physical welfare of young athletes. Motiv. Emot. 28, 297–313.

Google Scholar

Smith, N., Tessier, D., Tzioumakis, Y., Fabfra, P., Quested, E., and Appleton, P. (2016). The relationship between observed and perceived assessments of the coach-created motivational environment and links to athlete motivation. Psychol. Sport Exerc. 23, 51–63. doi: 10.1016/j.psychsport.2015.11.001

CrossRef Full Text | Google Scholar

Solstad, B. E., Stenling, A., Ommundsen, Y., Wold, B., Heuzé, J.-P., Sarrazin, P., et al. (2020). Initial psychometric testing of the coach-adapted version of the empowering and disempowering motivational climate questionnaire: A Bayesian approach. J. Sports Sci. 38, 626–643. doi: 10.1080/02640414.2020.1722575

PubMed Abstract | CrossRef Full Text | Google Scholar

Sukys, S., Kromerova-Dubinskiene, S. S. E., and Appleton, P. (2020). Validation of the lithuanian version of the coach-created empowering and disempowering motivational climate questionnaire (EDMCQ-C). Int. J. Environ. Res. Public Health 17:3487. doi: 10.3390/ijerph17103487

PubMed Abstract | CrossRef Full Text | Google Scholar

Tessier, D., Smith, N., Tzioumakis, Y., Quested, E., Sarrazin, P., and Papaioannou, A. (2013). Comparing the objective motivational climate created by grassroots soccer coaches in England, Greece and France. Int. J. Sport Exerc. Psychol. 11, 365–383. doi: 10.1080/1612197X.2013.831259

CrossRef Full Text | Google Scholar

Viladrich, C., Angulo-Brunet, A., and Doval, E. (2017). A journey around alpha and omega to estimate internal consistency reliability. Ann. Psychol. 33, 755–782. doi: 10.6018/analesps.33.3.268401

CrossRef Full Text | Google Scholar

Viladrich, C., Appleton, P. R., Quested, E., Duda, J. L., Alcaraz, S., Heuzé, J., et al. (2013). Measurement invariance of the Behavioural Regulation in Sport Questionnaire when completed by young athletes across five European countries. Int. J. Sport Exerc. Psychol. 11, 384–394. doi: 10.1080/1612197X.2013.830434

CrossRef Full Text | Google Scholar

Wang, Z., and Su, I. (2013). Longitudinal factor structure of general self-concept and locus of control among high school students. J. Psychoeduc. Assess. 31, 554–565. doi: 10.1177/0734282913481651

CrossRef Full Text | Google Scholar

Yu, C.-Y. (2002). Evaluating Cut-off Criteria of Model fit Indices for Latent Variable Models with Binary and Continuous Outcomes. [Ph.D. thesis]. California: University of California.

Google Scholar