Nationwide Subjective and Objective Assessments of Potential Talent Predictors in Elite Youth Soccer: An Investigation of Prognostic Validity in a Prospective Study

Höner, Oliver; Murr, Dennis; Larkin, Paul; Schreiner, Robert; Leyhr, Daniel

doi:10.3389/fspor.2021.638227

ORIGINAL RESEARCH article

Front. Sports Act. Living, 28 May 2021

Sec. Movement Science

Volume 3 - 2021 | https://doi.org/10.3389/fspor.2021.638227

This article is part of the Research Topic Talent Identification and Development in Sports Performance View all 29 articles

Nationwide Subjective and Objective Assessments of Potential Talent Predictors in Elite Youth Soccer: An Investigation of Prognostic Validity in a Prospective Study

$\nOliver Hner$ Oliver Höner¹^*

Dennis Murr¹

Paul Larkin²

Robert Schreiner¹

Daniel Leyhr^1,3

¹Institute of Sports Science, Eberhard Karls University of Tübingen, Tübingen, Germany
²Institute for Health and Sport, Victoria University, Melbourne, VIC, Australia
³Methods Center, Eberhard Karls University of Tübingen, Tübingen, Germany

Recent studies have provided empirical evidence on the prognostic relevance of objective performance diagnostics in the soccer talent identification and development process. However, little is known about the prognostic validity of coaches' subjective evaluations of performance. This study evaluated objective and subjective assessments within a nationwide talent development program and addressed motor, perceptual skill, and personality-related performance factors. Male players (N = 13,869; M_age = 12.59 ± 1.07 years) from the age groups U12 to U15 of the German soccer talent development program participated in this study. Participants completed an objective motor diagnostic (sprint, agility, dribbling, ball control, juggling) and were subjectively rated by their coaches (kicking skills, endurance, individual tactical skills, psychosocial skills). All nine predictors were assessed with sufficient psychometric properties (α ≥ 0.72; except dribbling and ball control: α ≥ 0.53). Players' success three seasons later was operationalized by achieving professional youth academy level or not (success rate, 9%). Independent-samples t-tests analyzed univariate mean group comparisons between future selected and non-selected players. Logistic regression models examined the multivariate prognostic validity of all assessments by predicting success with subjective (model 1), objective (model 2), and both groups of predictors (model 3). Confirming the univariate prognostic validity, future selected outperformed non-selected players regarding all predictors (each p < 0.001, except for agility in U15: p < 0.01). Tactical skills, kicking skills, and sprint were of highest predictive value (d ≥ 0.61 in each age group). Multivariate results provided empirical evidence for the subjective (7% ≤ Nagelkerke's R² ≤ 11%; each p < 0.001) and objective (8% ≤ Nagelkerke's R² ≤ 13%; each p < 0.001) assessments' prognostic validity. However, model 3 revealed the best statistical explanatory power in each age group (0.15 ≤ Nagelkerke's R² ≤ 0.20; p < 0.001). In this combined assessment model, sprint, tactical skills, and dribbling were found to be the most predictive variables. In conclusion, this study reinforces the call for multidimensional diagnostics integrating objective and subjective assessments. Future research is needed to address the demands for longitudinal analyses of subjective ratings, the integration of biological maturation, and empirical evidence for female soccer.

Introduction

Talent identification and development in soccer have been “vibrant research areas” for sport scientists in the last two decades (Williams et al., 2020, p. 1). Several prospective studies with multidimensional approaches have provided empirical evidence on the significant, yet also partly limited prognostic relevance of objective diagnostics that assess youth players' characteristics, abilities, and skills in soccer (Murr et al., 2018a,b; Sarmento et al., 2018; Ivarsson et al., 2020). However, little is known about the prognostic validity of subjective evaluations of such performance factors by coaches or scouts, and recent studies highlight the need to integrate both subjective and objective evaluations of potential talent predictors (Dugdale et al., 2020; Ford et al., 2020).

The identification of talent in soccer has been studied from a variety of theoretical and methodological approaches (Williams et al., 2020). Since soccer is a complex team sport, a number of performance factors must be considered when determining which youth athletes have the highest potential to develop into elite players (Lund and Söderström, 2017). For example, researchers have highlighted different qualities associated with performance, including physiological, technical, tactical, and psychological attributes (Hoare and Warr, 2000; Unnithan et al., 2012; Suppiah et al., 2015). With respect to physiological performance measurements, researchers indicate that selected youth soccer players are faster than non-selected players (Gil et al., 2014; Höner and Votteler, 2016). Furthermore, researchers found players who progressed to an elite level were more technically competent for skills such as juggling, dribbling, and passing accuracy (Höner et al., 2017; Bergkamp et al., 2019). In addition, skilled players have been found to possess superior perceptual–cognitive skills when compared to less skilled counterparts (Ward et al., 2013; O'Connor et al., 2016). Finally, psychological attributes such as self-confidence, motivation, mental toughness, commitment, and seeking social support have also been found to predict elite level soccer career success (Williams and Reilly, 2000; Toering et al., 2009; Van Yperen, 2009; Baláková et al., 2015; Höner and Feichtinger, 2016). Most of these findings were based on objective measurements and provide an indication of the factors that may predict future high performance. However, due to the considerable variation in study designs, findings across individual talent identification studies are inconsistent and difficult to compare. Therefore, there is no clear set of variables that uniformly predicts skill level (Breitbach et al., 2014; Höner and Feichtinger, 2016; Bergkamp et al., 2018; Johnston et al., 2018; Murr et al., 2018b).

Given the challenges associated with talent identification, standard talent identification procedures often rely on the evaluation of athletes' current performance by coaches and scouts (Williams et al., 2020). Within this process, coaches' experience and expertise in identifying potential talent comprise an important tool (Larkin et al., 2020). To this end, researchers have used interview techniques to determine what experienced soccer coaches look for when identifying potential talent. Christensen (2009) found Danish national team coaches valued game intelligence (i.e., ability to read and predict game play) and soccer-specific physiological and technical skills as the most important factors when assessing talent. Coaches also considered personal qualities (e.g., character, attitude, drive to succeed, and willingness to learn) as important. This finding is further supported by Larkin and O'Connor (2017), who identified a hierarchy of attributes perceived as important by coaches when identifying youth soccer players: technical (i.e., first touch, kicking skills, one-vs.-one ability, technical ability under pressure), tactical (i.e., decision-making skills), and psychological attributes (i.e., coachability, positive attitude).

Specifically related to the process of talent selection, Lund and Söderström (2017) found that Swedish soccer coaches made decisions based on previous experience, current elite players' qualities, and the values and belief system of the club. This suggests the decision to select or not to select an athlete is based on intuition and deliberation, grounded on an “overall impression,” factoring in subjective evaluations of technical skills, game understanding, and a variety of psychological characteristics (Meylan et al., 2010; MacMahon et al., 2018; Williams et al., 2020). However, based on knowledge from selection psychology, it is argued that individuals should be cautious when using such “clinical judgments” (Dawes et al., 1989), as various pieces of information have to be combined to make a decision, and as such, there is potential for different errors and biases. As a result, this may lead to less accurate decisions and inconsistencies between individual decision-makers (Den Hartigh et al., 2018). This issue has been identified with evidence to suggest the accuracy of subjective talent decisions by coaches and scouts is relatively low. Koz et al. (2011) found the accuracy of selection decisions for professional sports “entry drafts” suggests that even when these decisions are made late in development (i.e., early adulthood), the level of predictive accuracy is comparatively low.

Therefore, Ford et al. (2020) suggested researchers consider the integration of both subjective and objective evaluations of performance factors. However, according to the review of Williams et al. (2020), only two studies have explored the relationship between subjective evaluations and objective tests in this context. In a prospective design, Sieghartsleitner et al. (2019a) examined the isolated and combined prognostic relevance of several characteristics for the identification of future playing status, including the subjective evaluation of players' in-game performance in addition to objective diagnostics of technical and general motor performance factors. Results indicated that the use of subjective coach assessments and objective performance data was significantly better at predicting under 19 (U19) player status than objective performance data alone. More recently, Dugdale et al. (2020) compared levels of agreement between subjective and objective assessment of youth elite Scottish soccer players' (U11–U17 age group) physical performance. Athletes completed different physical performance assessments (e.g., endurance, acceleration, speed), with coaches providing a subjective evaluation of the physical variables. The findings indicated the coaches' subjective evaluation only corresponded with high and low performance on the objective physical assessments, suggesting that coaches' subjective evaluations may not be sensitive enough to discriminate between players whose performance level is rather similar. In conclusion, the two studies highlight the importance of combining both subjective evaluations and objective test results within the talent identification process. It should be noted, however, that Sieghartsleitner et al. (2019a) did not examine subjective performance factors, but general in-game performance, and Dugdale et al. (2020) did not address the prognostic validity of subjective rated performance factors with regards to players' future performance level. Thus, given the lack of data on coaches' and scouts' efficacy regarding selection decisions, there is a need to examine the prognostic validity of subjective and objective evaluations of youth players' performance factors in regard to their validity for talent selections over an extended period.

The Present Study

With an applied focus on the key process of talent selection, the present study was conducted within the talent development program of the German Soccer Association (Deutscher Fußball-Bund, DFB). In this program, two nationwide assessments of potential talent predictors are implemented to monitor the development of players' performance factors. First, a motor test battery was implemented as an objective diagnostic in 2004 and is conducted semiannually. Second, starting with the 2015/2016 season, coaches involved in the program rate players on subjective evaluation criteria in the spring of each season. This subjective evaluation supplements the objective motor diagnostic that currently addresses technical skills (i.e., ball control, dribbling, juggling) as well as linear and change of direction speed abilities (i.e., sprinting, agility¹).

Given these two existing nationwide assessments in the German talent development program, the aim of this prospective study was to investigate the objective diagnostic and subjective evaluation in the age groups U12–U15 in relation to their prognostic validity for future success. First, within a univariate perspective, we examined the prognostic validity of each single performance factor that was measured with an objective or subjective assessment. Second, within a multivariate perspective, we evaluated whether the (objective, subjective, and combined) assessments as a whole provide a meaningful prognostic model for players' future success and compared the contributions of each performance factor within these assessments.

Methods

Sample and Design

Within the German talent development program, nearly 1,300 part-time DFB coaches select about 14,000 early-adolescent players (i.e., age groups U12–U15) from amateur clubs all over the country to participate in one additional practice session per week at one of the 366 regional competence centers (CC). This quite homogeneous group of high-performing youth players belong to the top 4% of all German male youth players in their age group (Deutscher Fußball Bund, 2009). A central purpose of this program is the development of these CC players with the aim that they will be selected for one of the professional German soccer clubs' youth academies in middle-to-late adolescence (YA, top 1%).

The sample of this prospective cohort study comprised N = 13,869 male CC players (M_age = 12.59 ± 1.07 years; age groups U12–U15; birth cohorts: 2001–2005) who participated in the nationwide motor diagnostic (Höner et al., 2015). Furthermore, players were subjectively rated by their coaches regarding several performance factors either in spring of the 2015/2016 season (birth cohorts 2001–2004) or in spring of the 2016/2017 season (birth cohorts 2002–2005). The objectively and subjectively evaluated performance factors served as predictors for players' future success three seasons later.

Before entering the talent development program, players' legal guardian provided written informed consent for the recording and scientific use of the data. The CC coaches together with DFB staff members conducted the motor diagnostic and performed the subjective evaluations of the present study. The DFB provided the authors with players' data of the five birth cohorts (2001–2005). The university's ethics department approved the implementation of this study.

Measures

Predictors

The subjective performance evaluation was conducted by the 1,300 coaches of the German talent development program who possess at least the UEFA B-License. Coaches' subjective evaluations of motor (i.e., kicking skills, endurance), perceptual–cognitive (i.e., individual tactical skills), and personality-related domain (i.e., psychosocial skills) were assessed using a questionnaire consisting of 14 items (see Table 1). With respect to the motor domain, three items assessed kicking skills (i.e., kicking the ball with the dominant and non-dominant leg, heading the ball), and one item addressed endurance as a physiological performance factor. Regarding the perceptual–cognitive domain, individual tactical skills were subjectively assessed by the quality of individual tactical behavior in offensive and defensive situations (i.e., before, during, and after ball-related actions) as well as overall game intelligence (i.e., seven items in total). Finally, within the personality-related domain, the assessment of psychosocial skills comprised three items including motivational, volitional, and social skills (one item each). To enhance a nationwide common understanding of the items, the CC coaches were educated by DFB staff members, and a 16-page manual was distributed to the coaches before implementing the subjective evaluation. For each item, the manual included key points as well as detailed explanations and examples. Both the key points and examples guided the CC coaches what to look for in their subjective ratings.

TABLE 1

Table 1. Subjectively assessed youth players' performance factors serving as potential predictors for future success.

Table 2 provides an extract of the manual in regard to the three items addressing individual tactical behavior in offensive situations. For the item “individual tactical behavior in offensive situations before ball-related actions,” the CC coaches were directed by the manual to focus their subjective evaluation on the key points “preorientation” and “offering/creating space.” To get a more vivid and concrete understanding, the two key points were explained by typical examples: (1) CC players can orientate themselves in such a way that they make an appropriate decision (e.g., find an open position, look over their shoulders), and (2) CC players can make themselves available (e.g., separate themselves from a defender) in such a way that they can receive a pass or create space in which another players can receive a pass. The individual tactical behavior in offensive situations during ball-related actions was concretized by the key points “first touch”, “orientation on the ball”, and “situation-appropriate decision-making”, and regarding the behavior in offensive situations after ball-related actions, coaches should pay attention to the “reorientation” and “follow-up action” (these key points were also illustrated by typical examples, see Table 2; for corresponding information regarding the other performance factors, see Supplementary Table 1).

TABLE 2

Table 2. Key points and their explanations for rating players' individual tactical skills in offensive situations (before, during, and after ball-related actions).

Coaches were asked to judge each item from a holistic perspective (i.e., for their “overall impression” about the player in the respective season). For the motor and perceptual–cognitive performance factors, coaches rated their CC players' performance in comparison to the general CC player level and to the level of regional association team players (i.e., the next highest level within the talent development program). These reference levels were familiar to all CC coaches and thus implemented for the subjective assessment to ensure that CC coaches not only had a similar idea of the items (see Table 2) but also a reference norm that was as consistent as possible regarding the performance level. Thus, for each item, coaches evaluated their players on a 4-point rating scale as “below-average DFB competence center level” (0); “average DFB competence center level” (1); “level of the extended squad for regional association team” (2); or “level of core team for the regional association team” (3). As it was difficult for CC coaches to judge the psychosocial skills on these levels, no direct relation to the general CC or regional association team level was established. Thus, for the evaluation of the psychosocial skills items, the coaches rated their players as “below average level” (0), “average level” (1), “high level” (2), or “very high level” (3). Based on these ratings, a value for each respective performance factor was calculated by the average of the corresponding individual items (e.g., the mean of the three judgements for the kicking skills items represented the indicator for the subjective evaluation of the player's kicking skills).

The objective motor diagnostic included five individual tests to assess players' speed abilities and technical skills (for details, see Höner et al., 2015): sprint (i.e., time in a 20 m linear sprint); agility (i.e., time in a slalom course without a ball); dribbling (time in a slalom course with a ball); ball control (i.e., time needed to play six passes alternately against two opposing impact walls with at least two ball contacts); and ball juggling (i.e., juggle the ball alternately with the left and right foot through as many subsections of a figure eight-course without ground contact). Light barrier systems (Brower TC Timing, Draper, USA) were utilized to measure execution times for sprint, agility, and dribbling. Times for ball control were assessed by hand-stopped chronographs. Each test was performed twice with the best result recorded for analysis purposes. Between the attempts, the athletes were given enough time to recover. Whereas the time-based individual tests (i.e., sprint, agility, dribbling, and ball control) were negatively coded (i.e., a lower value indicated a better performance), a higher value in the juggling test represented higher performance.

As the nationwide assessments were more relevant for comparisons of players within age groups than over several age groups, the psychometric properties for the predictors in this study were investigated for each age group separately. The subjective performance scales were characterized by excellent reliability values in terms of internal consistency for tactical skills (0.89 ≤ α ≤ 0.91 for U12, U13, U14, and U15) and psychosocial skills (0.84 ≤ α ≤ 0.87), whereas kicking skills (0.73 ≤ α ≤ 0.77) showed at least satisfying values. Moreover, Höner et al. (2015) analyzed the age-specific motor test battery's psychometric properties for a sample of nearly 70,000 male CC players and found excellent internal consistency for sprint (0.92 ≤ α ≤ 0.93 for U12, U13, U14, and U15) and agility (0.90 ≤ α ≤ 0.90). Juggling revealed satisfying values (0.72 ≤ α ≤ 0.75), whereas those for dribbling (0.53 ≤ α ≤ 0.57) and ball control (0.61 ≤ α ≤ 0.64) were slightly lower.

Criterion

Three seasons after participating at the nationwide assessment in the competence centers (i.e., time of predictor data collection), players' future success was operationalized by achieving the German YA level or not (i.e., selected vs. non-selected). In general, the CC coaches who conducted the objective and subjective assessments were not involved in the future selection process for the YAs. Moreover, the approach ensured the same prognostic period (i.e., three seasons) for all players. The selected group comprised players who participated at the assessments in spring 2015 and were enrolled in a YA for the 2018/2019 season and those who completed the tests in spring 2016 and were enrolled in a YA for the 2019/2020 season. To identify the enrolled YA players, the squad lists for the respective age groups of all German YA were examined regarding the players' names and birth dates. Players who were—according to the squad lists in the respective seasons—identified as YA players were defined as selected and the others as non-selected players. Overall, n = 1,198 players were categorized as selected and n = 12,671 players as non-selected players (i.e., the success rate for achieving the youth academy level in this study was about 9%).

Statistical Analysis

Data were analyzed utilizing IBM SPSS version 26. To provide robust results regarding the predictors' prognostic relevance, the data from the two measurement points (i.e., season 2015/2016 and 2016/2017) was accumulated for each age groups (U12–U15). If a player participated in the assessment in both seasons, only the data from the first assessment was taken. However, the two seasons and the corresponding birth cohorts in the sample (2001–2004 and 2002–2005, respectively) may confound the analysis of performances' differences between selection levels (Elferink-Gemser et al., 2012). To control for this assumption, two-way ANOVAs for each performance factor were conducted testing whether there was an interaction effect between future success and the respective birth cohorts tested at the first and second season. As non-significant interactions [F_(1,13865) ≤ 3.36, p ≥ 0.07] were found for all variables, the cohort variable was not considered as a confounder in the following analysis. Similarly, the influence of relative age as a potential confounder within the analysis was investigated, and non-significant interactions [F_(1,13865) ≤ 3.12, p ≥ 0.08] between future success and relative age (player born in first or second half of the year) were found.

As coaches' subjective evaluations were ratings in relation to the respective age group (i.e., U12, U13, U14, and U15), the following statistical analyses were conducted for each age group separately. The variance of the investigated variables within each age group of the preselected samples and, accordingly, the expected statistical effect sizes may be limited in talent research studies (restriction of range of talent; Ackerman, 2014). Moreover, single predictors may represent only a small part of complex soccer performance. Thus, test power or sensitivity in prospective talent studies is a critical issue (Bergkamp et al., 2019). To determine the size of a possibly detected population effect for differences between two groups, sensitivity was calculated by post-hoc power analyses using G^*Power version 3.1.9.7 and predetermined parameters (α = 0.05, 1 – β = 0.85, two-tailed). For the age groups U12, U13, U14, and U15 and their corresponding sample sizes (see Table 3), the analyses were sensitive enough to detect small to medium effect sizes d ≥ 0.13, d ≥ 0.17, d ≥ 0.22, and d ≥ 0.40, respectively.

TABLE 3

Table 3. Test results for subjectively (printed in italics) and objectively evaluated performance factors of selected and non-selected players separated by age class (U12–U15, N = 13,869).

With respect to the univariate prognostic validity of the subjective evaluations as well as motor diagnostics of players' performance factors, mean group comparisons between future selected and non-selected players were computed for all assessed predictors. To identify significant differences between future successful and less-successful players, two independent samples t-tests were computed. Cohen's d (computed as the mean difference divided by the pooled standard deviation) served as effect size (including the respective 95% confidence intervals) and was classified into the categories small (0.2 ≤ d < 0.5), medium (0.5 ≤ d < 0.8), and large (d ≥ 0.8) in accordance with Cohen (1992). Regardless of whether the considered variable was negatively or positively coded, the provided d values were set positive when the selected players achieved better test results².

Second, a logistic regression approach was chosen to investigate the multivariate prognostic validity of the assessed performance factors. Within three different logistic regression models, the binary criterion variable (selected vs. non-selected for YA) was predicted by the subjectively (model 1), objectively (model 2), and the combination of subjectively and objectively (model 3) assessed performance factors. Independent variables for model 1 comprised the four subjectively judged performance factors (see Table 1), whereas model 2 consisted of the five single motor tests. Model 3 included both the four subjective and five objective performance factors. To evaluate the whole (subjective, objective, and combined) nationwide conducted assessments, the enter method was used for each regression analysis, and each overall model fit was analyzed with the likelihood ratio chi-squared test and Nagelkerke's R². Regression coefficients and the odds ratio coefficients e^b (including their 95% confidence intervals) were calculated to provide a clearer view of a player's relative chances to get selected, depending on the considered predictors. To facilitate comparisons for effect sizes of individual predictors, the odds ratio coefficients e^b were additionally adjusted to the standard deviations of the respective age group (Höner and Votteler, 2016). Thus, the resulting (e^b)^SD represents the relative change of the likelihood for being selected for a YA by a one standard deviation increase within the considered predictor. Thereby, the adjusted odds ratios were inverted and displayed as (e^b)^−SD for negatively coded predictors (i.e., the time-based tests sprint, agility, dribbling, and ball control) where lower values in time represent higher performance.

Regarding the chosen predictors within the models, considerable bivariate correlations were detected. Correlations among the four subjective performance factors in the respective age groups ranged from r = 0.51 (kicking skills with endurance in U15) to r = 0.79 (kicking skills with tactical skills in U12), while those for the five objective motor diagnostics ranged from r = 0.02 (sprint with juggling in U14) to r = 0.51 (agility with dribbling in U15). For the bivariate comparisons between subjective and objective performance factors, the highest relationship was found for endurance with sprint in U12 and U15 (i.e., each r = 0.22). Consequently, in order to control for a potential bias due to multicollinearity, the variance inflation factor (VIF) for the predictors within each logistic regression model was investigated. The VIF values for the predictors within the three models were all VIF ≤ 3.81, and because VIFs exceeding 10 are considered as an indicator for serious multicollinearity (Akinwande et al., 2015), no meaningful multicollinearity bias was expected for this study.

Results

Univariate Prognostic Validity of Subjective and Objective Performance Factors

Table 3 provides the results for the assessed predictors in the considered age groups separated by level of future success. Overall, the results for all predictors correspond to the later achieved performance level. That is, players who were selected for a YA after three seasons showed significantly superior performance factors compared to non-selected players (each p < 0.001, except for agility in the age group U15: p < 0.01).

Figure 1 illustrates the superior performances of the selected compared to the non-selected players in terms of Cohen's d. The results indicated no clear trend for a decline or incline of the effect sizes over the age groups, that is from U12 to U15. According to the median of the effect sizes, tactical skills [Mdn (d) = 0.75; 0.61 ≤ d ≤ 0.85 for U12, U13, U14, and U15]; kicking skills [Mdn (d) = 0.71; 0.61 ≤ d ≤ 0.82]; and sprint [Mdn (d) = 0.69; 0.40 ≤ d ≤ 0.73] proved the highest discriminative power between selected and non-selected players. Dribbling [Mdn (d) = 0.47; 0.41 ≤ d ≤ 0.54]; endurance [Mdn (d) = 0.43; 0.37 ≤ d ≤ 0.48]; psychosocial skills [Mdn (d) = 0.43; 0.38 ≤ d ≤ 0.50]; and juggling [Mdn (d) = 0.42; 0.35 ≤ d ≤ 0.52] revealed nearly medium effect sizes, whereas small effect sizes were found for ball control [Mdn (d) = 0.34; 0.22 ≤ d ≤ 0.41] and agility [Mdn (d) = 0.30; 0.25 ≤ d ≤ 0.36].

FIGURE 1

Figure 1. Superior performance of selected vs. non-selected players in terms of Cohen's d and respective 95% CI (the variables are ordered regarding the median of the effect sizes for the four age groups).

Overall, the predictive power of the four subjectively rated performance factors [Mdn (d) = 0.56; 0.38 ≤ d ≤ 0.85 for all subjective predictors over all age groups] were detected to be higher than those of the objectively assessed motor performance factors [Mdn (d) = 0.41; 0.22 ≤ d ≤ 0.73]. Except for sprinting, effect sizes for the objective diagnostics revealed slightly lower values ranging from d = 0.22 for ball control in U14 to d = 0.54 for dribbling in U15.

Multivariate Prognostic Validity of Nationwide Subjective and Objective Assessments

In each age group, the overall fits for the subjective [Model 1: 40.87 ≤ $χ_{(4)}^{2}$ ≤ 305.20, p < 0.001; 0.07 ≤ Nagelkerke's R² ≤ 0.11] and objective [Model 2: 40.87 ≤ $χ_{(5)}^{2}$ ≤ 388.02, p < 0.001; 0.08 ≤ Nagelkerke's R² ≤ 0.13] assessment models were significantly better compared to the null model. That is, both the subjective, as well as the objective assessment, significantly predicted the binary outcome (selected vs. non-selected player) in each age group (each p < 0.001). Except for the age group U15, where the explained variance by subjective evaluations (i.e., 10%) exceeded those by the objective assessment (i.e., 8%), model 2 comprising the objective diagnostics showed higher values of explained variance compared to model 1 (13, 14, and 13% vs. 10, 11, and 7% for the age groups U12, U13, and U14; for further details, see Supplementary Tables 3a,b).

Moreover, hierarchical regressions revealed that adding the subjective predictors of model 1 to the objective predictors of model 2 [for the age group U12–U14, 42.67 ≤ $Δ χ_{(4)}^{2}$ ≤ 137.29, each p < 0.001; 0.04 ≤ Δ Nagelkerke's R² ≤ 0.06] or adding the objective predictors of model 2 to the subjective predictors of model 1 [U15; $Δ χ_{(5)}^{2}$ = 20.42, p < 0.01; Δ Nagelkerke's R² = 0.05] led to a significant increase in the explained variance. Thus, the combination of the subjective and objective assessments (model 3) showed the best statistical explanatory power in every age group [Model 3: 61.29 ≤ $χ_{(9)}^{2}$ ≤ 525.31, p < 0.001; 0.15 ≤ Nagelkerke's R² ≤ 0.20].

The results of the logistic regressions for model 3 separated by age group are displayed in Table 4. Regarding the significance of the predictors, the subjectively evaluated tactical skills as well as the motor test results for sprinting and dribbling significantly contributed to the explanation of players' future success in each age group (each p < 0.05). Moreover, kicking skills was a significant predictor for all age groups except for U15 (each p < 0.05) and juggling for the younger age groups U12 and U13 (each p < 0.01), whereas ball control only showed a significance for the youngest age group U12 (p < 0.01). In contrast, the predictors endurance, psychosocial skills, and agility did not show any significant positive contribution to the models in three of the four investigated age groups. In 9 out of 12 cases, the regression coefficients for these three predictors in the four age groups were not significant (each p > 0.09). Interestingly, and presumably caused by the noticeable correlations to other explaining variables in the model, in the three remaining cases, psychosocial skills (U12, p = 0.02), endurance (U13, p < 0.01), and agility (U13, p = 0.01) contributed in the opposite, non-expected direction to the explanation of the criterion within the multivariate model.

TABLE 4

Table 4. Logistic regression results for the prediction of players future success (selected for a YA three seasons later) in dependence of the subjective (printed in italics) and objective assessments (model 3, separated by age group).

With regard to the adjusted odds ratios (e^b)^SD for significant predictors in the logistic regressions, it can generally be seen that sprint (except for U15) and tactical skills were characterized by the highest scores. For instance, a one standard deviation better result in the sprint would approximately double the chance of being selected for a YA three seasons later in the age groups U12 [(e^b)^SD = 1.75], U13 [(e^b)^SD = 2.05], and U14 [(e^b)^SD = 2.13]. The regression results revealed a similar importance for tactical skills in U15 [(e^b)^SD = 2.09], and this performance factor was still a relevant predictor in the younger age groups [1.52 ≤ (e^b)^SD ≤ 1.68]. Moreover, dribbling in all age groups [1.35 ≤ (e^b)^SD ≤ 1.53] and kicking skills in U13 [(e^b)^SD = 1.40] and U14 [(e^b)^SD = 1.36] proved their relevance in this regard, whereas all further significant predictors provided limited predictive relevance [(e^b)^SD ≤ 1.24].

Discussion

For a discussion of the study results, it is important first to identify the stage of talent development at which the study took place and second the parameters of the study design used to investigate the talent predictor's prognostic relevance. Regarding the first aspect, Williams et al. (2020) provided a conceptual framework for the talent identification and development process. They identified three key processes for players who are not involved in a structured talent program: detection (i.e., screening for players from outside soccer); participation (i.e., playing soccer but not in a structured development program); and identification (i.e., recognizing soccer players who have the potential to progress in a structured development program). Studies in this field are characterized by a heterogeneous sample (e.g., Hohmann et al., 2018). In contrast, the present study was conducted within the German soccer talent development program, where participants had already been selected as competence center (CC) players leading to a more homogeneous sample. Therefore, according to the conceptual framework, the present results refer to the stages of “development” (i.e., providing the players with a suitable learning environment), “selection” (i.e., on-going process of selecting players within the development program; e.g., for the next highest level or age group) and “deselection” (i.e., removing players from the program) of youth players (Williams et al., 2020).

The present study provides comprehensive insights into empirical evidence for the prognostic validity of nationwide assessments and thus of subjectively and objectively assessed performance factors in youth elite soccer. First, within a univariate perspective, the results confirmed the prognostic validity for all considered predictors addressing the motor, perceptual–cognitive, and personality-related domain. Thereby, the variables with the highest predictive value (i.e., tactical skills, kicking skills, and sprint) represent both assessment methods (subjective and objective) and depict different domains. However, effect sizes were limited, and thus, the sensitivity was not high enough to justify case-by-case selection decisions on the stand-alone basis of these assessments (Carling and Collins, 2014; Höner and Votteler, 2016). Second, within a multivariate perspective, the results provided empirical evidence for the subjective and objective assessments' prognostic validity. Overall, the objective assessment showed higher explanatory power when comparing both assessments. Most important, the combined model, integrating both assessments, revealed explained variances of 15% ≤ Nagelkerke's R² ≤ 20% over the four investigated age groups and proved superior to the models comprising either the subjective or objective assessment. Consequently, this addresses current gaps within the literature by linking subjective coach evaluations with objective measures (Williams et al., 2020). Within these multidisciplinary models, sprint, tactical skills, and dribbling were found to be the most predictive variables, although the order of the most significant predictors varied in parts among the age groups.

Prognostic Validity of the Performance Factors

The present study analyzed large-scale nationwide assessments based on a powerful sample size and assessed data from U12, U13, U14, and U15 players belonging to the top 4% of their age group in Germany. The prospective criterion was the achievement of youth academy status (i.e., top 1% of all players) three seasons later, with a success rate of 9%. For a more detailed interpretation, one must take into consideration the variety of study design features in prospective studies that influence the expectable amount of significant effects and their sizes (Hohmann et al., 2018; Bergkamp et al., 2019). For instance, players' age and soccer development stage (e.g., foundation, talent, or elite stage), the level and type of the criterion variable, and the applied assessments must be considered as moderator variables regarding the prognostic relevance of talent predictors (Murr et al., 2018b). Moreover, due to the sports-specific performance requirements, important information with respect to prognostic validity such as effect sizes are only comparable within a specific sport.

Therefore, the following discussion about the effect sizes presented in this study is focused on recent systematic reviews exploring prospective studies in youth soccer (such as Murr et al., 2018a,b; Sarmento et al., 2018; Williams et al., 2020) because reviews provide “overarching approaches” and are not dependent on each single study design feature. Moreover, individual studies with similar design features were considered for comparing effect sizes to the present study (and if these studies provided effect sizes other than Cohen's d, d was reanalyzed from the presented descriptive statistics). Studies from three nationwide talent development programs were considered. First, studies investigating male players from the German program at these age groups (Höner and Feichtinger, 2016; Höner and Votteler, 2016) seem to be most comparable to the present results (e.g., because of similar success rate). Second, two prospective studies conducted within the Swiss talent development program (Sieghartsleitner et al., 2019a,b) used similar diagnostic tools to assess motor performance factors (i.e., the same tests for agility, dribbling, juggling, and a modified version for ball control) and were characterized by further similar designs parameters. For example, the Swiss program selects the top 6% of the registered U12 players in their early selections for talent bases and about 1% for their elite youth development program in U15 (Romann and Fuchslocher, 2013). Grounding on this, the success rate in the prospective studies for a U13/U14 sample was 12%, and for a U14 sample, 17% (Sieghartsleitner et al., 2019a). However, these studies were characterized by less statistical power due to a smaller sample size (N ≤ 133), leading to a lower sensitivity to detect small effect sizes. Third, Saward et al. (2020) published a longitudinal study with data from male soccer players between 8 and 19 years from the English talent development system program. The data for the age groups U12–U15 was also considered suitable for a comparison with the present study results. In each age group, Saward et al. (2020) investigated about 1,000 players from 16 professional academies in England. As the prospective success rate of these players was between 17 and 30% for the age groups U12–U15, this might indicate a more elite sample compared to the present study.

Physiological Abilities (Motor Domain)

Regarding potential physiological talent predictors, this study investigated linear and change in direction speed abilities (i.e., sprint, agility) and endurance. According to the systematic review conducted by Murr et al. (2018b), mid-term prospective studies in early adolescence provided mixed results regarding the prognostic validity of agility [change of direction speed (CODS)] tests. For age groups U12–U15, there is a noticeable variation in the effect sizes across age groups ranging from non-significant effects (Deprez et al., 2015) to significant and large effect sizes of more than one standard deviation (Figueiredo et al., 2009). Presumably, due to its large sample size, the present study provided more robust results regarding the effect sizes. Here, agility varied only to a small extent around the median d = 0.30. This effect size is in line with the general conclusion in the review conducted by Williams et al. (2020, p. 3) that future successful players are “slightly more agile.” Similarly, a prospective study conducted by Höner and Votteler (2016) found not only significant but also small effect sizes for agility with CC players from earlier birth cohorts (i.e., 1993–1997). Saward et al. (2020) revealed significantly better agility performance in future professional players; however, the effect sizes were trivial to small in adolescence (d < 0.20 by the age 12.0) and got larger with further development (d = 0.50 by the age 18.0). In contrast, Sieghartsleitner et al. (2019a) descriptively presented agility test results of Swiss U14 players that did not indicate any differences (d < 0.10) between players who achieved a professional level 5 years later or those who did not.

In relation to sprint performance, Saward et al. (2020) found that future professional players in England were faster in a 20-m sprint than non-professional players but only reported a small effect size (d = 0.20 over all age groups, U9–U20). In the present study, the 20-m sprint was one of the most relevant tests and revealed the highest effect size of d = 0.73 for the age group U14, whereas Sieghartsleitner et al. (2019a) did not find prognostic relevance for a 40-m sprint test for the same age group. This inconsistency may be caused by the diverse sprinting distances in the study, although in the review by Murr et al. (2018b), sprinting tests with a distance of >20 m were more relevant than the tests with shorter distances ( ≤20 m). Going beyond, further inconsistencies related to study design, such as performance level of players and length of prognostic periods, may also impact the generalizability of the study results. For example, Sieghartsleitner et al. (2019a) as well as Saward et al. (2020) utilized longer prognostic periods (≥5 years). This, together with partly different performance levels of players at the time of predictors' assessment (i.e., a higher level of selection, Saward et al., 2020), might have led to lower effect sizes detected in these studies compared to the present study. Although recent reviews (Murr et al., 2018b; Williams et al., 2020) and the present study provide insight into the predictive power of sprint performance, future research is needed to clarify whether sprint results may be less relevant regarding predictive validity in older aged or higher selected groups. At least, there seems to be a trend in this direction since, in the present study, the effect size for U15 (d = 0.40; p < 0.001) was remarkably smaller than for the younger age groups (although still significant). Moreover, Murr et al. (2018b) found less significant empirical evidence for sprinting tests in the elite stage (U16–U19) compared to the talent stage (U12–U15).

With regard to the assessment of endurance, it should be noted that this performance factor was evaluated subjectively by coaches. Dugdale et al. (2020) indicate that coaches reach their limits in subjective evaluations when players show similar performances regarding endurance. Nevertheless, in the present study, the mean univariate effect size of d = 0.43 is within the range of the significant effect sizes presented in the review by Murr et al. (2018b) for objective endurance diagnostics (0.28 ≤ d ≤ 1.56). However, that review also revealed that nearly one-third of the effect sizes identified for endurance in the talent stage phase (U12–U15) were non-significant, and in a more recent study, Sieghartsleitner et al. (2019a) found the (objective) Yo-Yo intermittent recovery test in U14 age group not to be of predictive relevance for future success. Moreover, besides the different approaches of assessing endurance (i.e., subjective vs. objective), different statistical approaches may lead to a further inconsistency regarding the prognostic validity. For example, when utilizing a multivariate approach, Sieghartsleitner et al. (2019a) found a negative relationship with professional player status (i.e., Swiss U19 junior national players) for their comprehensive general motor performance variable (i.e., Yo-Yo intermittent recovery test, counter movement jump, 40 m linear sprint, and agility test). In a similar way, the multivariate analysis in the present study revealed in each of the considered age groups negative effects (i.e., e^b < 1; in one case significant: p < 0.01 for U13) for the subjectively rated endurance item when considered in combination with the other performance factors. Although multicollinearity between the predictors of the logistic regression models was acceptable in terms of VIF, the different results for endurance in the univariate and multivariate perspective are likely due to noticeable correlations with other (mainly subjective) predictors. Therefore, more research regarding the prognostic validity of this factor, predominantly in multidimensional approaches, is needed.

Technical Skills (Motor Domain)

In addition to the reviews for sport (Sarmento et al., 2018; Koopmann et al., 2020) or soccer skills (Murr et al., 2018a; Williams et al., 2020), the present study underlines the importance of technical skills. The median effect size for dribbling (d = 0.47) in this study was at the lower end of the range of effect sizes that Murr et al. (2018a) identified in 10 prospective soccer studies (0.47 ≤ d ≤ 1.24). Interestingly, ball control (d = 0.34) was below the range (0.57 ≤ d ≤ 1.28) detected by Murr et al. (2018a). Moreover, juggling, a skill not investigated as much as other technical skills, proved to be a significant predictor with a median effect size of d = 0.42 and was also found to be a relevant predictor in the younger age groups (i.e., d = 0.52 in U12 compared to d = 0.35 in U14). In both studies within the Swiss program, even higher effect sizes were detected regarding juggling compared to the present study. While Sieghartsleitner et al. (2019a) found a medium effect size (d = 0.62) for the U14 age group, in a second study with a comparable sample of U14 soccer players, a large effect size (d = 1.07) was found (Sieghartsleitner et al., 2019b). However, these inconsistencies regarding the detected effects in these studies could have been affected by the rather low sample size of players in the professional groups (17 ≤ n ≤ 20).

Of particular interest, especially when considering a direct comparison of objective and subjective assessments, are the results for kicking skills, as these results can be compared to the prognostic validity of a shooting test investigated by Höner and Votteler (2016). Although practitioners consider shooting skills to be very important for soccer performance, only small effect sizes were found in that study. However, a challenge for researchers is the ability to create a reliable and valid shooting skill test, and many tests have poor psychometric properties (Ali, 2011; Höner et al., 2015). Therefore, within the German talent development program, the shooting test was substituted by the subjective rating presented in this study. Indicated by an effect size of d = 0.71, the newly established subjective evaluation of kicking skills proved to have superior prognostic validity compared to the former shooting test. One reason for this may be that the subjective assessment of kicking skills was based on a broader concept compared to the objective shooting skill tests where only the precision and speed of the shots were registered. For example, coaches evaluate the ability of the CC players to use different kicking techniques, to pass or shoot at the goal with a high degree of variability, at a speed appropriate to the game situation or under pressure from the opponent (see Supplementary Table 1a).

Individual Tactical Skills (Perceptual–Cognitive Domain)

Tactical skills such as behavior in one vs. one offensive/defensive situations, anticipation, decision-making, or game intelligence have been found to be crucial for achieving top-level performance in football (e.g., Roca et al., 2012). In accordance to the current results, Murr et al. (2018a) identified in their systematic review medium to large effect sizes for the prognostic relevance of these skills. However, the effects sizes are based on only four prospective studies, indicating a lack of such studies and limiting the empirical evidence for perceptual–cognitive skills compared to other domains. Moreover, the review identified only one study in soccer to assess these skills based on an objective assessment. O'Connor et al. (2016) used video-based tests including different soccer-specific tasks of tactical skills (e.g., decision-making), which prompt a written response. Such video-based diagnostic instruments are established in the (cross-sectional) expertise research; however, they are considered to have limitations regarding the representativeness of the stimuli presentation as well as the limited motor response of the participants (e.g., Travassos et al., 2013). Besides critical arguments regarding the representativeness of video-based diagnostics, test economic reasons might cause the lack of prospective studies examining the predictive value of perceptual–cognitive factors. For example, three studies (i.e., Kannekens et al., 2011; Huijgen et al., 2014; Forsman et al., 2016) providing empirical evidence in the systematic review used the tactical skill inventory in which players rate their soccer performance in comparison to the top players in the same age category (TACSIS; Elferink-Gemser et al., 2004). However, besides general criticisms concerning the lack of soccer-specific performance response (Nortje et al., 2014), self-reported tactical skills might be biased by socially desired answers or unrealistic self-concept.

To avoid such biased assessments, Williams et al. (2020) suggested that researchers should consider measuring individual performance factors, via coach ratings, in matches or small-sided games. However, the authors acknowledged the challenge of developing rating tools with satisfactory psychometric properties. The present study revealed excellent reliabilities for the subjective evaluation tool. Going beyond, tactical skills were the strongest predictor in the univariate (0.61 ≤ d ≤ 0.85) and one of the strongest in the multivariate analyses. Therefore, the tactical skills evaluation tool (Table 1) proved to be an appropriate assessment method within a nationwide talent development program. While a positive outcome, future research is needed to explore the correlations between the coaches' ratings to other perceptual–cognitive diagnostics such as in-game performance assessments (e.g., small-sided games; Bergkamp et al., 2020), video-based decision-making skills tests with a soccer-specific response (Murr et al., 2021), or assessments of general cognitive functions (e.g., Beavan et al., 2020).

Psychosocial Skills (Personality-Related Domain)

Comprehensive models for talent development (e.g., DMGT; Gagné, 2010) provide a theoretical basis for examining the predictive value of psychosocial dispositions and skills from the personality-related domain, such as motivation or volitional self-regulation (e.g., Mills et al., 2012) that are mainly investigated with subjective self-report questionnaires. Several systematic reviews (e.g., Gledhill et al., 2017; Murr et al., 2018a; Ivarsson et al., 2020; Williams et al., 2020) highlight the importance of psychological predictors for (successful) talent development in soccer. Nevertheless, similar to the other investigated domains in this study, the effect sizes in these reviews do not support the use of sport psychological diagnostics as a selection tool in talent development programs.

This is also in line with the present results regarding the psychosocial skills, although the median effect size (d = 0.43) represents a recognizable prognostic relevance that was, to some extent, larger than the effects found in a comparable study within the German talent development program (Höner and Feichtinger, 2016). In that 4-year prospective study, U12 players (N = 2,677) completed 17 psychological scales of established sport psychological self-report questionnaires addressing motivational, volitional, self-referential cognitions and emotional characteristics. Whereas the majority of the psychological scales (10 out of 17) proved to be significant in predicting success (i.e., achieving the youth academy level at U16), the effect sizes were smaller (0.19 ≤ d ≤ 0.30) than in the present study. This indicates that coaches' subjective assessment of psychological characteristics may add explanatory power to self-report questionnaires (Musculus and Lobinger, 2018) because coaches can base their judgements on observations of a player's behavior in several “representative” situations and in reference to the other talented players. However, further research is needed to bridge the gap between the (scientifically sound) sport psychological self-report questionnaires and the (often unevaluated) scouting sheets occasionally used in youth soccer. Thereby, recommendations on how to improve the subjective assessments of personality-related factors and how to educate the youth soccer coaches in this regard should be considered (Musculus and Lobinger, 2018).

Limitations and Perspectives

Although the present study is characterized by several strong design features (i.e., homogeneous sample with talented players, high test power, midterm prognostic period, and multidimensional subjective and objective assessment), several limitations need to be considered. First, in accordance with the vast majority of studies in this research area, the current study focused on assessing performance at a single time point. However, based on the multidimensional and dynamic conceptualization of talent (Buekers et al., 2015) and, in some parts, the inconsistent results in longitudinal studies (e.g., Leyhr et al., 2018; Saward et al., 2020), further empirical studies are needed to examine the progress of players' performance factors longitudinally. However, it should be noted that the present study focused on the simultaneous examination of the validity of subjective and objective assessments in an attempt to address the gap in this field of research. Moreover, from a methodological perspective, there are some non-trivial difficulties when combining subjective and objective assessments in one longitudinal analysis. For example, in this study, the coaches were asked to evaluate the players' performance factors in regards to the corresponding reference groups (Ivarsson et al., 2020). Thus, in contrast to the motor tests, no absolute values were available from the implemented subjective assessment and an investigation of players' developmental process would have been restricted to the exploration of development only relative to other players.

Second, the operationalization of the criterion variable is a methodological issue critically debated in soccer talent identification research (Bergkamp et al., 2019). The prognostic period in this study ranged from early to middle adolescence, whereas the criterion was operationalized as performance level in age groups U15, U16, U17, and U18 respectively. As juvenile success as an appropriate predictor for success in adulthood might be questioned in sport (Güllich and Emrich, 2014), this prediction period may be regarded as a limitation. However, being selected for the youth academies (that is, the present study's criterion variable) seems to be strongly associated with success in adulthood in German soccer, as nearly 90% of German adult professional players played at least one season for a youth academy (Güllich, 2014). Furthermore, the criterion for the prediction was operationalized as performance level. Bergkamp et al. (2019, p. 1319), highlighted “an explicit measure of soccer performance is rarely used as a criterion,” and this may imply some problems. For instance, binary-coded performance levels, as utilized in this study, provide only limited information on the individual differences between players' soccer performance. However, such information is helpful as decisions on selecting or deselecting players frequently occur in sport talent pathways (Ford et al., 2020), and especially in the present study over a prolonged prognostic period of three seasons, different coaches and institutions (competence centers, regional associations, youth academies, youth national teams) were involved in these decisions concerning the players. Moreover, valid and test economic assessments for measuring individual soccer performance in large-scale studies are challenging. Consequently, an operationalization of soccer performance by players' performance level seems appropriate from a practical perspective (Bergkamp et al., 2019).

Third, the players' development and their performance factors are not only dynamic but also interactive with additional factors. Thereby, not only personal but also environmental factors, such as family support or training history, play a decisive role. However, one of the few studies investigating the combination of motor performance diagnostics and coaches' subjective evaluations with additional information about family support and training history led to the conclusion that including these environmental factors in existing data on motor performance tests and coach assessments did not provide significant additional explanatory power (Sieghartsleitner et al., 2019a). Therefore, the integration of environmental factors in prognostic models may be a demanding task for future studies (e.g., when considering the operationalization of the variables and the underlying assumptions about the association between constructs such as family support and successful player development). That is, probably an optimal but not maximal family support may be most supportive for a player, and going beyond, this optimum might be individualized and thus different for each player.

Moreover, further potential moderating or confounding variables were not in the scope of this study. For example, Sarmento et al. (2018) highlighted a complex relationship between youth players' performance factors according to relative age, maturity status, or specific playing positions. In particular, maturational status might have enabled interesting insights for the present study, as indicated by a recent study from Hill et al. (2020b), who observed positive associations between coaches' subjective match performance ratings and maturation in U14 and U15 academy soccer players. However, data on maturity status was not assessed in this large-scale study, and assessments such as the Mirwald method (Mirwald et al., 2002) or the Khamis–Roche method (Khamis and Roche, 1994) still need further investigation concerning their reliability and validity (Myburgh et al., 2019; Leyhr et al., 2020a). Regarding information about relative age, interactions between future success and relative age for each investigated predictor were found non-significant, and therefore, this was not considered as confounder for the present examination of prognostic validity. Nevertheless, future research should investigate relative age-related biases regarding objective (e.g., Votteler and Höner, 2017; Hill et al., 2020a) and subjective assessments (e.g., Furley and Memmert, 2016). In addition, talent predictors' prognostic relevance with respect to specific playing positions seems to be a promising perspective. As specific playing positions within players' talent development in the investigated age groups (U12–U15) may frequently change (e.g., especially considering their future playing position in professional soccer), data about specific playing positions were not assessed and considered in the present study. However, future research should address this when investigating predictors assessed in later stages of talent development and their predictive value for reaching a professional level in soccer.

A further limitation is that this study only investigated male players. With a few exceptions (Datson et al., 2020; Leyhr et al., 2020b), this is typical of talent research in sport, with an overrepresentation of male studies in general (Johnston et al., 2018; Koopmann et al., 2020) and in soccer (Murr et al., 2018b; Sarmento et al., 2018). However, gender must be considered as a potential moderator variable regarding the prognostic relevance of talent predictors, and thus, future studies should extend their focus to female athletes (Williams et al., 2020), in particular due to the increasing popularity of female soccer (Manson et al., 2014).

Conclusion and Implications for Practice

In conclusion, this study provides empirical evidence for the prognostic validity of performance factors from different domains and thus reinforces the call for multidimensional assessments and the use of objective as well as subjective assessments in talent development programs (Sieghartsleitner et al., 2019a; Williams et al., 2020). All nine investigated predictors proved to be significant, but integrating the univariate and the multivariate perspective, tactical skills, and sprint, and thereafter kicking skills and dribbling, were the most important predictors in this study.

Interestingly, the univariate predictive power of the four subjectively rated performance factors were detected to be slightly higher than those of the five objectively assessed motor performance factors. However, multivariate results indicated a higher explanatory power of the objective assessment compared to the subjective assessment. As a consequence, predictors such as kicking skills and psychosocial skills proved more relevant in the univariate than in the multivariate perspective. This may be due to the breadth and dominance of the tactical skills in the subjective assessments and also suggests that coaches' subjective ratings differentiate less between theoretical distinct constructs than objective assessments. At least, it is remarkable that the subjective assessments in this study addressed a broader spectrum of performance factors compared to the objective assessments but lost more explanatory power in the multivariate analyses. This may be interpreted as a weakness of the subjective assessment, and coaches should be educated regarding their analytic skills of performance factors. On the other hand, the subjective assessment also proved strengths in assessing more complex skills like tactical, kicking, or psychosocial skills that are difficult to assess via objective (and test economic) measures in nationwide programs. Thus, a challenge for future research is to explore strengths and weaknesses of both assessment approaches and the optimal method to integrate them.

In regard to practical implications, it is important to note that both subjective and objective assessments in the German talent development program mainly support the monitoring of players' development and are therefore not used as a “tool” for selection or deselection. Accordingly, CC coaches are advised to base their selection decisions on their personal experiential knowledge. Nevertheless, the assessments may provide an additional piece of information for making these decisions. More importantly, to know which factors are relevant for the development of successful players, the presented empirical evidence for the prognostic relevance of the investigated factors is crucial.

Moreover, from an organizational perspective, the study results may also inspire youth academies or soccer associations coach education programs to develop about the coaches “eye” for relevant talent predictors. First, by describing performance factors such as kicking skills, endurance ability, tactical skills, and psychosocial skills in a detailed manual provides a “common” understanding of these factors, and coaches can evaluate their players as uniformly as possible in the nationwide program. Second, by implementing the subjective assessments, CC coaches were consciously encouraged to deal with the described talent predictors and to discuss them with their peers or with the associations staff members responsible for CC coach education programs. Furthermore, CC coaches are able to evaluate and reflect on their diagnostic ratings and discuss this with peers or association staff members. Thereby, their documented views may be strengthened if the CC coaches' evaluations are confirmed by players' future success; however, if there is a difference, specific education may be offered. For this purpose, the manual may also provide the foundation and the described key points, and their explanations can be used to create instructional videos that allow CC coaches to convey a uniform idea of the individual talent characteristics.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by Ethics Committee of the Faculty of Social Sciences and Economics, University of Tübingen. Written informed consent to participate in this study was provided by the participants' legal guardian/next of kin.

Author Contributions

OH, DL, and DM: conceptualization and methodology. OH: data curation, supervision, funding acquisition, project administration, validation, visualization, and writing original draft. DL: formal analysis. OH, DL, DM, and RS: investigation. OH, DL, DM, RS, and PL: writing review and editing. All authors contributed to the article and approved the submitted version.

Funding

This study was part of the research project scientific support of the DFBs Talent Development Program, which was funded by the DFB and was granted by the German Football Association (Deutscher Fußball-Bund, DFB, http://www.dfb.de). DFB staff members conducted the data collection and the DFB provided the authors with players' data. We acknowledge support by Deutsche Forschungsgemeinschaft and Open Access Publishing Fund of University of Tübingen. This funding supported the payment of the publication fee.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We would like to thank the staff of the DFB's Department for Talent Development for the provision of data and productive discussions in several meetings.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fspor.2021.638227/full#supplementary-material

Footnotes

1. ^There are inconsistent definitions of agility in sport science literature (Sheppard et al., 2014). In the present study, the term “agility” is used for tests that assess a speed-related motor ability enabling preplanned changes of movement direction that does not include cognitive aspects such as anticipation or decision-making. Thus, this study investigated the physical dimension (or movement component) of agility. This perspective categorizes agility as a physical performance factor in accordance with the majority of prospective studies referred to in the present article (e.g., Hohmann et al., 2018; Sieghartsleitner et al., 2019b; Saward et al., 2020) as well as with the recent reviews on talent identification research in soccer (e.g., Bergkamp et al., 2019; Williams et al., 2020). However, following the work by Sheppard and Young (2006), agility is often used in a broader sense whereby agility is considered an open skill based on two factors, “change of direction speed” (CODS) and perceptual–cognitive facets such as anticipation and decision-making. To address the inconsistency in the literature, we use both terms in the figures and tables of the present study [i.e., “Agility (CODS)”] to indicate that we refer to the physical dimension of agility but not to the more complex reactive agility.

In a recent position paper, Young et al. (2021) encourage researchers to study (reactive) agility in more ecologically valid settings that include a sport-specific stimulus (e.g., video simulations, 1 vs. 1 situations, small-sided games). This call is also true for the other objective motor diagnostics investigated in the present study because they do not represent the contextual constraints of soccer games (for a discussion of the representativeness and fidelity of predictors in talent identification research in soccer, see also Bergkamp et al., 2019).

2. ^In addition, to have a pragmatic measure for the chance of players' future success, groups of low [percentile rank (PR) ≤ 33.33], medium (33.33 < PR ≤ 66.67), and high (PR > 66.67) performers were built for each considered predictor. Based on these categorizations, odds ratios being selected for a YA for players with high vs. medium and high vs. low performance outcomes were calculated (results for these analyses are presented in the Supplementary Table 2).

References

Ackerman, P. (2014). Nonsense, common sense, and science of expert performance: talent and individual differences. Intelligence 45, 6–17. doi: 10.1016/j.intell.2013.04.009