Implications of the choice of distance-based measures in assessing and investigating tumble turn performance

Although the tumble turn in swimming has been studied extensively, no consensus exists about which measure is best suited to capture its performance. The aim of this study was to better understand the implications of choosing a particular distance-based performance measure for assessing and investigating tumble turn performance in freestyle swimming. To this end, a large set of retrospective turn data consisting of 2,813 turns performed by 160 swimmers was analyzed statistically in three steps. First, a mixed-effects model was derived for the entire data set, which showed that both performance level and sex had clear effects on the distance-based performance measures and performance determining variables studied in the literature. Second, repeated measures correlations were calculated for the entire data set and four performance level- and sex-based subgroups to determine the level of association between the performance measures. This analysis revealed that the performance measures were strongly correlated (r > 0.84 and p < 0.05 for all possible pairs), largely independent of performance level and sex. This finding implies that the choice of performance measure is not very critical when one is interested solely in the overall performance. In the third and last step, mixed-effects models were derived for the performance measures of interest to establish the importance of different turn-related actions for each measure, again for both the entire data set and the four subgroups separately. The results of this analysis revealed that performance measures with short(er) distances are more sensitive to changes in the adaptation time and reflect the wall contact time better than performance measures with long(er) distances, which in contrast are more useful if the focus is on the approach speed prior to the turn. In this final analysis, various effects of performance level and sex were found on the technical execution of the tumble turn.

Introduction

Turning is an important component of competitive swimming, which determines a substantial portion of the overall swim performance. Research has shown that the accumulated duration of turns amounts to at least 19% of the total race time in long-course races and up to 44% in short-course races (Veiga et al., 2013; Morais et al., 2019a,b; Born et al., 2021). However, no consensus exists in the literature about the most suitable operational measure for capturing the time associated with performing the most common turn in swimming, the freestyle tumble turn.

In the literature, two types of definitions for turning performance are found: one based on the time that elapses between the submerging of the head at turn initiation and its resurfacing after the turn, thus covering a variable distance covered within this interval, and one based on the time that elapses between the head (or another body part) crossing predefined fixed distances toward and away from the wall. The limitation of the first definition is that a lower score on this performance measure (a shorter time) does not necessarily indicate a superior turn performance compared to a higher score (a longer time). This is due to the velocity profile of the swimmer after the push-off from the wall and the subsequent underwater phase. The velocity after the push-off from the wall is higher than the free-swimming speed and decreases over time due to drag until the speed is equal to the free-swimming speed, at which the swimmer ideally transfers to free swimming (Shimadzu et al., 2008). This moment can be postponed by extending the underwater phase [e.g., by making (more) undulatory kicks], however on the first, action-based definition, the performance score becomes lower (better) even if the swimmer resurfaces before the optimal point is reached. The sooner the swimmer resurfaces, the lower (better) the score on this performance measure will be. However, this lower score will typically not translate into a better overall swim performance.

Although the use of fixed distances overcomes this problem, there is quite some variation in the literature regarding the reference distances that are used to measure turning performance (Silveira et al., 2011). This variation exists because there are no distances that naturally limit the turn, except that, according to race regulations, the head must resurface before the 15-m mark after the turn for all swim strokes except the breaststroke. Obviously, the resulting variation in reference distances limits the comparison of results across studies. The performance measures that have been used in the literature are the 5 m round trip time (5mRTT), i.e., the time that elapses between the swimmer crossing the 5 m line away from the wall when approaching it (5 m-in) and crossing it again when moving out of the wall (5 m-out) (Blanksby et al., 1996, 1998; Cossor et al., 1999; Clothier, 2004; Mosavi et al., 2012; Pereira et al., 2015; Smithdorf, 2018; Nicol et al., 2019), the 2.5mRTT (Toshiaki et al., 2001), the 3mRTT (Puel et al., 2010a,b, 2012), the 7.5mRTT (Mason and Cossor, 2001; Tourny-Chollet et al., 2002; Shahbazi et al., 2007; Bahadoran et al., 2012; Veiga et al., 2013; Skyriene et al., 2017), the 5 m-in to 7.5 m-out time (Nicol et al., 2019), the 5 m-in to 10 m-out time (Webster et al., 2011; Nicol et al., 2019) and the 5 m-in to 15 m-out (McCabe et al., 2012; Suito et al., 2015; Morais et al., 2019a,b; Nicol et al., 2019; Marinho et al., 2020) time.

Based on empirical observations, Silveira concluded that the 5mRTT constitutes the best measure to describe turning performance in sub-elite swimmers because it included all turn-related actions and had the lowest percentage of free-swimming compared to other fixed-distance performance measures (Silveira et al., 2011). However, it remains unclear whether the turns of swimmers of different performance levels and/or sex, who might submerge or resurface earlier or later than 5 m from the wall, are also optimally reflected by the 5mRTT.

It is important to note in this context that the freestyle tumble turn is a highly complex skill, which involves many different actions that are performed sequentially in rapid succession. For instance, it takes about 1.1 s from turn initiation to the feet leaving the wall. Within this short time, the swimmer slows down, reverses position, and generates as much speed as possible in the opposite direction by applying a high push-off force to the wall. Immediately thereafter an underwater phase follows involving both passive gliding and active propulsion actions. Hence, in general, it is not realistic to assume that a performance measure that ignores speed and distance and is based on time alone can fully capture the freestyle tumble turn performance with all its performance determining factors.

Nonetheless, time-based performance measures can be used not only to assess the turn performance by itself but also to gain insight into the determinants of turn performance that might be improved through training. Based on previous analyses of the tumble turn, the technical variables that might be relevant in this regard include the approach speed toward the wall (Lyttle and Mason, 1997), the horizontal distance at which the turn is initiated (Blanksby et al., 1996; Lyttle and Mason, 1997), the adaptation time [from head submersion until the feet touching the wall (Lyttle and Mason, 1997; Maglischo, 2003)], the wall contact time (Blanksby et al., 2004; Pereira et al., 2006; Araujo et al., 2010), the push-off angle, the deepest point reached by the hips during underwater swimming (Lyttle et al., 1998), the horizontal distance from the wall at first downbeat (Lyttle et al., 2000), and the break-out distance (i.e., the horizontal distance from the wall at which the head breaks the water surface (Mason and Cossor, 2001).

To our knowledge, no other attempts besides Silveira (Silveira et al., 2011) have been made to examine the relative merits of the turning performance measures used in the literature, nor the information contained in them. The aim of the present study was to fill this lacuna by analyzing a large set of retrospective turn data that were collected from 160 elite and sub-elite swimmers of both sexes during regular training sessions. To this end, three statistical analyses were performed: (1) a foundational analysis to examine the impact of performance level and sex on the performance measures and performance determining variables that have been studied in the literature, (2) a coarse-grained analysis of the correlation strengths between the performance measures of interest, and (3) a fine-grained analysis of the degree to which the aforementioned technical variables account for the variation in each of the performance measures of interest. For all three analyses, we had specific expectations based on a combination of common sense and previous findings.

Considering the well-documented performance differences between male and female swimmers and between elite and sub-elite swimmers (Arellano et al., 1994; McCabe et al., 2012; Veiga and Roig, 2016; Morais et al., 2019b; Nicol et al., 2019; Marinho et al., 2020), we expected both performance level and sex to have a significant effect on the performance measures and performance determining variables of interest. We conducted this preliminary foundational analysis to determine how to treat the effects of performance level and sex in the subsequent statistical analyses (see section Materials and methods for details).

The coarse-grained analysis was motivated by the recognition that the information contained in the different performance measures is overlapping due to their overlapping fixed in and out distances. As a result, the performance measures will be correlated, which raises the question of how critical the choice of a particular performance measure is relative to other possible choices. After all, the higher the association between the performance measures, the less critical the choice of performance measure becomes. Based on logical grounds, we expected the association between measures to be high and largely independent of the performance level and sex of the participants. If so, this would render the choice of performance measure less critical from a statistical point of view.

The fine-grained analysis is based on the recognition that the tumble turn is a highly complex skill. As a result, the choice of a particular performance measure may reflect some turn-related actions more prominently and other turn-related actions less prominently, if at all. In other words, the information contained in a particular performance measure is likely to be a function of its definition. To select a suitable performance measure for answering a certain scientific or practical question, it is necessary to understand which information is contained in the set of (currently) available performance measures. This can be accomplished by deriving linear mixed-effects models to determine which specific turn-related actions account for (most of) the variance of each performance measure.

Based on logical grounds, we expected the technical variables associated with wall contact, notably the adaptation time and wall contact time, to account for more of the variance of the performance measures with a short(er) distance covered (that is, with fixed in and out distances close to the wall) than for performance measures with a long(er) distance covered (that is, with fixed in and out distances far from the wall). Conversely, we expected technical variables associated with the underwater phase, notably the push-off angle, the deepest point of the hip reached underwater and the break-out distance, to account for more of the variance of performance measures with a long(er) distance than performance measures with a short(er) distance covered.

Materials and methods

Participants

For this study, extensive retrospective freestyle turn data from 160 swimmers (85 male, 75 female) were analyzed. The participants, or their legal guardians in case they were 16 years of age or younger, signed an informed consent form stating that their turn data would be used anonymously for the purpose of scientific research. FINA points were determined to assess the performance level of the swimmers. To this end, the race times on any freestyle event (long course or short course) swum during the period from 2010 to 2021 in which the turn data were collected, were converted to FINA points based on the FINA point score of 2021 (FINA World Championships, 2022). The highest FINA point score of all these races was used to quantify the performance level of each swimmer. The 50 m long events were excluded from this procedure as they contain no turn.

The participants were categorized as either elite or sub-elite swimmers, depending on their FINA point score, which enabled making statistical comparisons based on performance level. Swimmers were considered elite swimmers if they had 860 FINA points or more. The 860 FINA points cut-off for elite swimmers was based on the average FINA point score corresponding to the FINA A qualification time for the freestyle events of the World Championships 2022 (Bouget, 2008).

Data acquisition

The tumble turn data for this study were derived from video footage recorded between 2010 and 2022 with a video system in the training pool of InnoSportLab de Tongelreep in Eindhoven. Before the measurements, a marker was placed on the swimmer's trochanter major to capture the position of the hip to derive the technical variables of interest (see below for further details).

The fixed video system in the training pool of De Tongelreep consists of four cameras (scA 1400-30gc, 50 Hz, Basler, Ahrensburg, Germany) that are embedded in the pool's lateral sidewall (at respectively, 2, 5, 10, and 15 m from the turn wall at a depth of 0.55 m below the water surface). The video data from the cameras were acquired using the software package Streampix 7 (Norpix, Montreal, Canada, 2016). The cameras were synchronized by means of an external trigger pulse generator (NI-DAQmx Pulse Generator). During the trials, the swimmers began swimming about 15 m from the wall (before the turn), sprinted toward the wall, turned and sprinted back until they were positioned well beyond the 15-m mark with their hip. All trials were recorded during regular training sessions and performed at maximal or race pace effort to mimic competitive races. Whether or not this was actually accomplished was not considered a major concern since having considerable variation in tumble turn performances is beneficial for the to-be-performed statistical analyses.

All video recordings were analyzed manually using a custom-made software package called Turnanalyzer (Escrito IT, Eindhoven, The Netherlands). The intrinsic parameters of the cameras were determined with the Camera Calibration Toolbox in Matlab (R Core Team, 2022) using image data of a checkerboard from various positions. The extrinsic calibration parameters were obtained by making use of control points at known positions in the pool. The intrinsic and extrinsic camera parameters were combined to reconstruct pixel data in the field of view of the cameras to 2D real-world coordinates on a sagittal plane positioned parallel to the sidewall at a distance of 3.6 m into the pool. The calibration error (root mean squared error) of the cameras was between 0.40 and 0.52 pixels for the cameras positioned at 2, 10, and 15 m, respectively, and 1.82 pixels for the camera positioned at 5 m, which was thus a factor less accurate than the other cameras. The technical variables were then manually extracted from the video and the resulting data were stored in a database.

Data preparation

All front crawl turn data of the included swimmers were extracted from the database and filtered for trials that contained all of the temporal performance measures depicted in Figure 1: from 5 m-in to 3 m-in to the wall (A), from 3 m-in to the wall until the moment the head is completely submerged before the turn (B), the time taken for the tumble turn and push-off [from head under the water surface until the feet losing contact with the wall (C)], the feet losing contact with the wall until hip at 5 m (D), from 5 m-out of the wall until 10 m-out of the wall (E) and 10 m-out of the wall to 15 m-out of the wall (F). The hip was used as the reference point for establishing the time at which the various spatial variables of interest were determined. In the protocol used in De Tongelreep for the measurement and analysis of swimming movements, the hip is routinely chosen for this purpose, because it is always in the field of view of the camera(s), regardless of the swimming stroke used (as opposed to the head, which is continuously visible in freestyle swimming, but not, for example, in the breaststroke and butterfly).

FIGURE 1

Figure 1. Graphical representation of the temporal variables and distances used for calculating the performance measures and the technical variables of interest. A: from 5 m-in until 3 m-in to the wall, B: from 3 m-in to the wall until head down before the turn, C: duration of the tumble turn and push-off (starting with the head underwater until the feet lose contact with the wall), D: the feet having lost contact with the wall until the hip crosses the 5 m mark, E: from 5 m-out of the wall until 10 m-out of the wall and F: 10 m-out of the wall until 15 m-out of the wall.

Based on the temporal variables used in the literature, the following temporal performance measures were determined for each turn: 5 m-in to 5 m-out, 3 m-in to 5 m-out, 3 m-in to 10 m-out, 5 m-in to 10 m-out, 3 m-in to 15 m-out and 5 m-in to 15 m-out.

Only swimmers with a minimum of five turns in the database were included in the statistical analyses to limit the impact of swimmers with only a small number of turns in the database satisfying the selection criteria. Furthermore, trials were excluded if they did not contain data on all of the eight technical variables of interest, as defined operationally in Table 1.

TABLE 1

Table 1. Operational definition of the variables of interest.

Statistical analysis

All statistical analyses were performed in R (Bates et al., 2014) using RStudio 4.2.0 (RStudio, Boston, Massachusetts, USA). In particular, the following modules were used: lme4 (Wickham et al., 2022), readxl (Barton, 2022), MuMIn (Long, 2022) and jtools (Bakdash and Marusich, 2022) and rmcorr (Bakdash and Marusich, 2017; Schweinberger, 2022). The statistical analysis of the data consisted of (1) a foundational analysis, followed by (2) a coarse-grained analysis, and (3) a fine-grained analysis (see Figure 2). For all statistical tests, an α-level of 0.05 was assumed.

FIGURE 2

Figure 2. Flow chart of the performed statistical analysis. 1) Foundational analysis to investigate the effect of performance level and sex on the performance measures and technical variables. 2) Coarse-grained analysis to determine the level of association between the performance measures. 3) Fine-grained analysis to examine the relationship between each performance measure and the selected technical variables.

Foundational analysis

First, the effects of performance level and sex on the performance measures and technical variables were assessed in a foundational analysis of the entire data set by deriving linear mixed-effect models for each of the six performance measures and the eight technical variables separately. In this analysis, the technical variables, performance level and sex were defined as fixed effects and athlete ID as a random effect (Figure 2).

As indicated, the main aim of this analysis was to determine how to treat the effects of performance level and sex in the subsequent statistical analyses. In particular, if a significant main effect of either performance level or sex would be found for (the majority of) the performance measures, the data set would be split into two subgroups (male and female, or elite and sub-elite, depending on the nature of the effect). Similarly, if main effects of both performance level and sex would be found for (the majority of) the performance measures, the data would be split into four subgroups (i.e., elite male, elite female, sub-elite male and sub-elite female, Figure 2). Splitting the data accordingly into subgroups may provide insight into possible group-specific differences in the performance measures and different turn strategies with regard to the technical, performance determining variables of interest. In this regard, the described statistical analysis was foundational for the subsequent coarse-grained and fine-grained analysis of the entire data set, and whether or not they would be complemented by group-specific analyses (Figure 2).

Coarse-grained analysis

To investigate the level of association between the different performance measures, repeated measures correlations were calculated for all possible pairs of performance measures, either for the complete data set alone or also for specific subgroups, depending on the results of the foundational analysis. This method is more suitable than Pearson's correlation analysis to deal with the repeated measures design of this data set and the unequal number of observations per individual. In particular, it is not biased by the mixing of intra- and inter-individual variance, where Pearson's correlation analysis would be (Schweinberger, 2022). The lower the repeated measures correlation coefficient, the more important the choice of one performance measure in favor of the other.

Fine-grained analysis

In the next, in-depth analysis, linear mixed-effect models with a bottom-up procedure (Lyttle et al., 1999) were constructed to determine the relationship between each performance measure and the eight derived technical variables as independent predictors (Figure 2). This was done to estimate the sensitivity of each performance measure with regard to the technical variables (Table 1) that were identified in previous research as potentially important determinants of turn performance (Blanksby et al., 1996, 2004; Lyttle and Mason, 1997; Welham and Thompson, 1997; Mason and Cossor, 2001; Maglischo, 2003; Pereira et al., 2006; Araujo et al., 2010). Again, this was either accomplished for the complete data set alone or also for (two or four) specific subgroups, depending on the result of the foundational analysis (Figure 2).

As a first pass on the data, boxplots and histograms were made and inspected for all independent variables to identify outliers amongst the independent predictors. Following this, collinearity was assessed by calculating the repeated measures correlation coefficients between all independent variables. In case there was a strong repeated measures correlation (r > 0.7) between two independent variables, the one with the highest absolute correlation coefficient with the performance measure was preserved as a predictor in the model while the other one was removed.

Finally, a bottom-up procedure (Lyttle et al., 1999) was used to determine which (combination of) independent variables provided the best fit for the model, starting with a model that only contained athlete ID as a random effect, due to the repeated measures design of the data. In each iteration round, a variable was added to the linear mixed-effects model and the new model was compared to the previous one using the likelihood ratio test (Nakagawa and Schielzeth, 2013). If the model was not significantly different from the previous one, the latest added variable was removed from the model. This procedure was continued until the model only contained variables that improved the model's fit. To be able to compare the contributions of the independent variables within and between the models, the same models were constructed with the standardized dependent and independent variables, thereby removing the effect of the scale at which they were measured. The following equation was used to standardize the variables:

$\begin{array}{l}{z}_i=\frac{x_i-\mu }{s{d}_X}\\ (1)\end{array}$

where z_i represents the standardized value, x_i represents the unstandardized value, μ represents the sample mean of the variable and sd is the sample standard deviation of the variable. Finally, the R² values of the linear mixed-effects models were determined using the method of Nakagawa and Schielzet (Lyttle et al., 1998).

Results

Foundational analysis

After application of the inclusion criterion to all turns available in the database, a total of 2,813 turns from 160 individual swimmers remained, out of which 1,154 turns were performed by 38 elite male and female swimmers combined (Table 2). The means and standard deviations of the performance measures and technical variables for both performance level and sex are displayed in Table 2. As expected, there was a significant main effect for both performance level and sex for all performance measures, in the absence of a significant sex by performance level interaction. The male swimmers were significantly faster than the female swimmers on all performance measures in both the elite and sub-elite subgroups. Furthermore, elite swimmers were significantly faster than the sub-elite swimmers on all performance measures in both the male and female subgroups. The results for the technical variables of interest are presented in Table 2. Since both performance level and sex had a significant main effect on all performance measures, both the course-grained and fine-grained analyses were performed on the entire data set and the elite male, elite female, sub-elite male and sub-elite female subgroups separately.

TABLE 2

Table 2. Descriptive statistics and means and standard deviations of performance measures and technical variables for the elite male and female and sub-elite male and female subgroups separately.

Coarse-grained analysis

As can be appreciated from the repeated measures correlation matrices presented in Table 3, all performance measures were strongly (r > 0.8) and significantly (p < 0.001) correlated. For the sake of clarity, the repeated measures correlations were presented for the entire data set only, since the results for the four subgroups, which are presented in Table 3, only showed marginal differences. The highest correlation of 0.99 was between the 3 m-in−15 m-out and 5 m-in−15 m-out and the lowest correlation of 0.84 was between the 3 m-in−5 m-out and 5 m-in−15 m-out. As expected, the greater the overlap in distance between the performance measures, the higher the degree of association.

TABLE 3

Table 3. The repeated measures correlation coefficient matrix between all possible combinations of performance measures for the entire data set and the elite male and female, and sub-elite male and female swimmers separately.

Fine-grained analysis

The boxplots and histograms of the technical variables revealed no outliers. According to the procedure described in the Materials and methods section, the horizontal initiation distance was excluded from the model because it was highly correlated with the adaptation time (r = 0.73), indicating collinearity between these variables. Additionally, the absolute repeated measures correlation coefficient with the performance measures was lower (e.g., r = 0.20 initiation distance and the 5 m-in−5 m-out time compared to r = 0.29 between adaptation time and 5 m-in−5 m-out time). For all six performance measures, a prediction model was derived with athlete ID as a random effect and the seven remaining technical variables as independent predictors.

All the models showed a high R² value ranging from 70 to 91% explained variance (see Appendix A). The prediction models of the performance measures that started 5 m before the wall had a higher R² than those starting 3 m before the wall. The bottom-up procedure resulted in the intercepts, estimates, and standardized estimates of the independent variables that are collated in Appendix A. All models for the different performance measures across all four subgroups included the speed-in, adaptation time, WCT and break-out distance. Although the break-out distance improved the models' fit significantly, the overall contribution to all models turned out minimal. The estimates of adaptation time and WCT had a positive sign in the model, indicating that an increase in these independent variables was associated with an increased turning time. The opposite was true for the estimates of speed-in and break-out distance, which had a negative sign, indicating that an increase was associated with a faster turn. The distance of the first kick, push-off angle and deepest point in the underwater trajectory were included only in some models. If included, the push-off angle and deepest point had a positive sign, implying slower turn times with greater values.

The estimates of the distance of the first kick had a negative sign for the sub-elite male swimmers, but a positive sign for the elite male and female swimmers and the sub-elite female swimmers. Compared across the different models, the contribution of the speed-in increased with the distance toward the wall covered in the performance measure (Figure 3, Appendix A). In contrast, the contribution of the adaptation time increased mainly for performance measures with an increased distance out-off the wall but decreased when the distance to the wall became longer.

FIGURE 3

Figure 3. (A–D) The absolute estimates of the seven contributors (speed-in, adaptation time, wall contact time (WCT), break-out distance, deepest point in the underwater phase, distance at first kick and push-off angle) to the elite male and female and sub-elite male and female models are presented graphically. The different models are presented on the x-axis.

The overall contribution of the WCT seemed higher for the elite swimmers than the sub-elite swimmers, for whom the speed-in was the predictor with the highest contribution to performance, especially for performance measures with long(er) distances. The adaptation time was a stronger performance predictor in the female swimmers than the male swimmers (Figure 3, Appendix A).

In the sub-elite female swimmers, all three main predictors (speed-in, adaptation time, and wall contact time) contributed to a similar degree, unlike the other three subgroups in which one main predictor clearly contributed more than the other two (notably WCT in the elite male and female swimmers and speed-in in the sub-elite male swimmers) (Figure 3, Appendix A).

Discussion

The present study examined the implications of selecting a particular distance-based performance measure for the tumble turn with regard to the overall turn performance and the underlying performance determining variables. To this end, three statistical analyses were performed: (1) a foundational analysis to examine the impact of performance level and sex on the selected performance measures and the performance determining variables, (2) a coarse-grained analysis of the correlation strengths between the performance measures of interest, and (3) a fine-grained analysis of the technical variables that account for the variation in each of those performance measures.

Foundational analysis

As expected, the foundational analysis revealed significant main effects of both performance level and sex on the performance measures and technical variables of interest. Based on these results the data set was divided into four subgroups: elite male, elite female, sub-elite male and sub-elite female. All subsequent analyses were done on the entire data set and on the four subgroups separately to gain insight into any subgroup-specific differences in performance and the adopted turn technique.

Coarse-grained analysis

The level of association between the fixed-distance performance measures of interest was assessed by calculating repeated measures correlation coefficients for all possible pairs of performance measures, both for the entire set of turn data and the four subgroups. This analysis revealed a high overall level of association with correlation coefficients ranging from 0.84 to 0.99. Although this result was expected on logical grounds since the performance measures covering longer distances overlap with the performance measures covering smaller distances (e.g., 5 m-in−5 m-out is part of the 5 m-in−10 m-out and 5 m-in−15 m-out), it has, to our knowledge, not been established or highlighted before in the literature. The strong correlation between the various performance measures implies that if one is interested solely in assessing the overall turn performance, the choice of performance measures is less critical. This finding was exacerbated by the marginal unsystematic differences in the results obtained between the four subgroups, which showed similar correlation patterns independent of performance level and sex. The small differences in the level of association between the various performance measures may have been due to differences in the turn-related actions and amount of free-swimming covered by the different performance measures (e.g., the moment of head resurfacing and stroke resumption occurring somewhere between 5 and 15 m).

However, even though all correlation coefficients were higher than 0.8, it appeared that the included distance after the turn had a stronger effect than the distance before the turn since the correlation coefficient decreased pair-wise with different out-off-the-wall distances, but not with different to-the-wall distances (see Table 3). This finding can be interpreted by realizing that the speed after the turn is largely determined by an effective push-off, an appropriate time spent on the wall and a properly streamlined position (Welham and Thompson, 1997; Lyttle et al., 1998; Mason and Cossor, 2001) and that some of these variables may be affected by the characteristics of the approach toward the wall. The longer the distance covered, the more variability may be generated by the interaction of all performance determining factors. This might explain why the level of association diminishes faster when correlating performance measures with different out-of-the-wall distances compared to different into-the-wall distances. Therefore, performance measures with longer out-of-the-wall distances seem more informative about the efficiency of turning actions and the transfer to swimming than performance measures with shorter out-of-the-wall distances.

Fine-grained analysis

The fine-grained sensitivity analysis revealed which turn-related actions are reflected predominantly in the performance measures of interest. This was achieved by finding the best fitting linear mixed-effects model for each performance measure using seven (eight minus one) technical variables as independent predictors and athlete ID as a random effect.

All the models showed a high R² value ranging from 70 to 91% explained variance. This is in contrast to the findings of Nicol et al., who did not find any of the technical variables to be a significant predictor of the overall turn time (Nicol et al., 2019). This discrepancy might be explained by the size of the included data set and the variability that was introduced in the present study by including swimmers of different performance levels. Interestingly, all models starting 5 m from the wall showed a higher R² than models starting 3 m before the wall. Speed-in, adaptation time, WCT, and break-out distance were significant predictors in all models. Of these four variables, speed-in was the only variable whose contribution differed between the 3 m-in and 5 m-in models. However, this is not surprising because the speed-in was measured between the 5 m and 3 m mark before the wall. The difference in contribution, therefore, seems to account for the higher explained variance of the 5 m-in models.

The estimates of speed-in had a negative sign, indicating that a higher swim speed toward the wall was associated with a better turn performance, as has been found in previous studies (Takahashi et al., 1982; Blanksby et al., 1996; Lyttle and Mason, 1997). The amount of variance explained by speed-in increased with increasing the distance covered by the performance measure. Taking an example from the elite male model (see Appendix A), an increase of speed-in of about 1 m/s would be associated with a turn time reduction of 0.44 s in the 3 m-in−5 m-out measure, while the same change would be associated with a turn time reduction of 2.32 s in the 5 m in−15 m out measure. This can be understood from the fact that the longer performance distances included more free-swimming; a higher swimming speed was, therefore, reflected more in the performance measures with longer distances. This could also explain why the contribution of the speed-in increased more strongly with distance in the sub-elite swimmers, who on average traveled a shorter distance underwater than the elite swimmers.

As expected, adaptation time and WCT turned out to be strong positive contributors to the overall turn performance. This finding is in agreement with the results of Blanksby and Puel (Blanksby et al., 1996; Puel et al., 2012), who also found that a reduction of the adaptation time and WCT resulted in faster turns. The adaptation time and WCT were reflected more strongly in the performance measures with shorter distances as indicated by their higher contribution to these models, which was expected as well. The model outcomes further revealed that the overall contribution of the WCT was higher for the elite compared to the sub-elite swimmers (Figure 3), which suggests that focusing the training on the optimization of WCT might have a beneficial effect on turn performance. However, as the WCT is very short (~0.3 s), easier improvements might be accomplished by shortening the adaptation time (~0.9 s). Another interesting finding of the fine-grained analysis was that the adaptation time is a stronger predictor in female compared to male swimmers, whereas the adaptation time itself was not significantly different between sexes. This suggests that a different turning technique was employed by male and female swimmers.

The contribution of the adaptation time and WCT decreased compared to the speed-in as the distance covered by the performance measures increased for all subgroups, except for the elite female swimmers. This decrease could be expected since these variables play a role early in the turn; therefore, even if their impact on turn time is large, it will be diluted by the impact of the actions later in the turn. This effect will be especially strong for the adaptation time. Also, the ratio of the WCT and the turn time decreases the longer the distance covered, which automatically reduces the impact. This result confirms our expectation that if the WCT is the focus of interest, a performance measure with short(er) distances should be chosen. The contradictory finding for the elite female swimmers could be explained by looking further into the characteristics of this subgroup, which contained a relatively small number of swimmers (N =15). This could have resulted in an overall higher variability of the included variables and an underestimation of their contribution.

The negative coefficient for the break-out distance is in line with the results of Mason et al., who concluded that a longer underwater phase is beneficial for the speed after the turn, resulting in a better turn performance (Mason and Cossor, 2001). The increase in the contribution of the break-out distance is also in line with our expectations because this variable plays out relatively late in the turn and thus was expected to contribute to the performance measures with longer distances. However, the contribution to the performance measures was generally much smaller than those of the speed-in and the WCT and also the relative impact declined with increasing distance when compared to the speed-in, which can be appreciated from the standardized estimates (see Appendix A).

By comparing the three main contributors (speed-in, adaptation time and WCT) between the subgroups, not one predictor appeared more dominant than the other two predictors in the sub-elite female swimmers. In contrast, WCT stood out as the main contributor in the elite male and female swimmers, while speed-in was the main contributor in the sub-elite male swimmers. This might suggest that the variation of turn strategies was markedly higher in the sub-elite female swimmers than in the other subgroups.

The remaining three technical variables, the push-off angle, the distance of the first kick and the deepest point in the underwater trajectory, were only minor contributors to some of the models. From the literature, a negative estimate was expected for the distance of the first kick (Lyttle et al., 2000). Surprisingly, this was only the case for the sub-elite male swimmers; in contrast, positive estimates of the distance of the first kick were found for the sub-elite male and female swimmers, while it did not even appear as a significant contributor in the elite female swimmers. Besides only playing a minor role to predict turn performance, the negative estimates of the distance of the first kick resulted from the relatively short distance to the wall of the sub-elite male swimmer (~1.82 m), while the elite male (~2.72 m) and sub-elite female swimmers (~2.55 m) might already had reached the optimal distance where a further increase would no longer be beneficial.

The deepest point in the underwater trajectory contributed to all elite male models, but only to the 10 m-out and longer sub-elite male models. Whereas the elite male swimmers reached a significantly greater depth during the underwater phase, their push-off angle was similar to that of their sub-elite counterparts. This suggests that the elite male swimmers held their downward line longer after push-off than the sub-elite male swimmers, which could also explain the later break-out distance. This is in line with the findings of Mason et al., who showed that swimmers with a longer underwater phase after the turn could benefit more than other swimmers from making quicker turns (Puel et al., 2010a).

Interestingly, the underwater phase seemed less important in the elite female swimmers, because none of the variables related to underwater swimming contributed to many of the derived models. This may be the case because the elite female swimmers within the data set adopted a wide variety of turn strategies, which could have obscured finding clear statistical relationships.

When comparing the subgroups with each other it became clear that there are some indications of different turning strategies between the subgroups. Furthermore, depending on the subgroup one is interested in, not all performance measures reflected the turn-related technical variables to the same degree. Therefore, it is important to choose a performance measure that reflects the variable of interest for the specific subgroup or question of interest. More research is needed to clarify these differences and their origins.

Although the standardized estimates enabled comparison, they should be interpreted with caution when comparing them between models. If the standard deviations between models are different, this could jeopardize drawing a valid conclusion based on their relative contributions. One example is the increase of standardized estimates for speed-in for the models for performance measures including a 3 m distance to the wall, while the standardized estimates decreased for models on the 5 m-in distances in elite swimmers. Using the non-standardized estimates showed that the effect of the speed-in variable increased with increasing distance from the wall, which is consistent with the idea that swimmers with a higher performance level have a higher speed-in and also will be faster after the turn.

One of the limitations of the present study is that the data set used contained no information about the underwater swimming ability of the swimmer, apart from the distance at which the head resurfaced after the turn. Although it was found in previous studies that underwater actions are important predictors of turn performance, this was not reflected in the present results (Clothier, 2004; Connaboy et al., 2016). It could be that the selected technical variables did not optimally reflect the underwater actions. The second limitation is that the data set only contained variables that were extractable from video footage. Adding biomechanical variables like the push-off force to the models would have rendered the analysis more comprehensive and should ideally be done in future research. However, most swimming facilities do not have the advanced measurement infrastructure required for this purpose, which precludes them from including other biomechanical variables besides kinematic variables.

The data set used in this study is unique as it contained over 2,800 turns from a broad variety of competitive swimmers, ranging from international Olympic finalists to regional swimmers. A sample size of this magnitude is unique in turn-related swimming research and allowed gaining a better understanding of the implications of choosing particular performance measures for both scientific and practical purposes. In particular, the present results clearly indicate that the various performance measures used in the literature are strongly correlated, they reflect different technical, performance determining variables, that play out differently in a subgroup-specific manner, depending on both performance level and sex. This implies that the choice for a particular performance measure has to be based on how well it is suited to answer the research or practical question of interest, and to which technical variables are reflected in the selected measure. The present results provide a wealth of information in this regard, which allows both researchers and coaches to make better-informed choices on which performance measures to use for their research or training purposes, and hopefully also inspires them to explore new leads in research and training.

Conclusion

The present results suggest that if one is interested solely in the overall turn performance, the choice of performance measure is less critical and can be adapted in accordance with the available facilities or personal preferences of the coach or researcher. Even though all correlation coefficients were higher than 0.8, it has to be taken into account that the included distance after the turn was a stronger contributor than the distance before the turn. When the aim is to identify performance determining variables that could be improved through training, the choice of performance measure matters. The present results showed that choosing a fixed-distance performance measure with short distances reflects the WCT better while performance measures with long distances are beneficial if the focus is on the approach speed prior to the turn. Also, performance measures with short distances before the turn are more sensitive to changes in the adaptation time.

Data availability statement

The data set presented in this article are not readily available because the data set contains elite athletes that can be identified from the data. Requests to access the data set should be directed to p.koster@fieldlabswimming.com.

Ethics statement

The studies involving human participants were reviewed and approved by the Scientific and Ethical Review Board (VCWE) of the Faculty of Behavioural and Movement Sciences of the Vrije Universiteit Amststerdam. Written informed consent to participate in this study was provided by the participants' legal guardian/next of kin.

Author contributions

PK, WA, SS, and PB conceptualized and designed the study. PK, WA, and SD performed the literature search and data processing. SS, PK, and WA were involved in data collection. PK, SD, SS, and PB drafted the current manuscript. All authors contributed substantially to the interpretation of the data, revised it critically and approved the final version of the manuscript before submission.

Acknowledgments

The data on which this study is based has been collected and reduced for further analysis over more than a decade. Many individuals have contributed to this process. The authors would like to thank the team of InnoSportLab De Tongelreep, in particular Roald van der Vliet and Carlo van der Heijden, and numerous research interns for their assistance in this regard.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fspor.2022.958548/full#supplementary-material

References

Araujo, L., Pereira, S., Gatti, R., Freitas, E., Jacomel, G., Roesler, H., et al. (2010). Analysis of the lateral push-off in the freestyle flip turn. J. Sports Sci. 28, 1175–1181. doi: 10.1080/02640414.2010.485207

PubMed Abstract | CrossRef Full Text | Google Scholar

Arellano, R., Brown, P., Cappaert, J., and Nelson, R. C. (1994). Analysis of 50-, 100-, and 200-m freestyle swimmers at the 1992 Olympic Games. J. Appl. Biomech. 10, 189–199. doi: 10.1123/jab.10.2.189

PubMed Abstract | CrossRef Full Text | Google Scholar

Bahadoran, M., Mosavi, S. H., Hasannejad, E., Moradlo, H., Education, P., and Branch, Z. (2012). “Investigating kinematic of the flip turn technique in front crawl swimming,” in 30 International Conference on Biomechanics in Sports (Melbourne), 76–79.

Bakdash, J., and Marusich, L. (2022). rmcorr: Repeated Measures Correlation. Available online at: https://cran.r-project.org/web/packages/rmcorr/index.html (accessed August 26, 2022).

Google Scholar

Bakdash, J. Z., and Marusich, L. R. (2017). Repeated measures correlation. Front. Psychol. 8, 456. doi: 10.3389/fpsyg.2017.00456

PubMed Abstract | CrossRef Full Text | Google Scholar

Barton, K. (2022). MuMIn: Multi-Model Inference. Available online at: https://cran.r-project.org/web/packages/MuMIn/index.html (accessed August 26, 2022).

Google Scholar

Bates, D., Maechler, M., Bolker, B., Walker, S., Haubo, R., Christensen, B., et al. (2014). lme4: Linear Mixed-Effects Models Using “Eigen” and S4. Available online at: https://cran.r-project.org/web/packages/lme4/index.html (accessed August 26, 2022).

Google Scholar

Blanksby, B., Skender, S., McElroy, K., Landers, G., and Elliott, B. (2004). Swimming: an analysis of the rollover backstroke turn by age-group swimmers. Sports Biomec. 3, 1–14. doi: 10.1080/14763140408522826

PubMed Abstract | CrossRef Full Text | Google Scholar

Blanksby, B. A., Hodgkinson, J. N., and Marshall, R. N. (1996). Force-time characteristics of freestyle tumble turn by elite swimmers. South Afr. J. Res. Sport Phys. Educ. Recreat. 19, 1–15.

Google Scholar

Blanksby, B. A., Simpson, J. R., Elliott, B. C., and McElroy, K. (1998). Biomechanical factors influencing breaststroke turns by age-group swimmers. J. Appl. Biomech. 14, 180–189. doi: 10.1123/jab.14.2.180

CrossRef Full Text | Google Scholar

Born, D. P., Kuger, J., Polach, M., and Romann, M. (2021). Start and turn performances of elite male swimmers: benchmarks and underlying mechanisms. Sports Biomech. 1–19. doi: 10.1080/14763141.2021.1872693

PubMed Abstract | CrossRef Full Text | Google Scholar

Bouget, J. Y. (2008). Camera Calibration Toolbox for Matlab. Available online at: http://robots.stanford.edu/cs223b04/JeanYvesCalib/download/index.html (accessed August 26, 2022).

Clothier, P. J. (2004). Underwater Kicking Following the Freestyle Tumble Turn, Vol. 43. University of Ballarat, 285. Available online at: http://www.ncbi.nlm.nih.gov/pubmed/21271636 (accessed August 26, 2022).

Google Scholar

Connaboy, C., Naemi, R., Brown, S., Psycharakis, S., McCabe, C., Coleman, S., et al. (2016). The key kinematic determinants of undulatory underwater swimming at maximal velocity. J. Sports Sci. 34, 1036–1043. doi: 10.1080/02640414.2015.1088162

PubMed Abstract | CrossRef Full Text | Google Scholar

Cossor, J. M., Blanksby, B. A., and Elliott, B. C. (1999). The influence of plyometric training on the freestyle tumble turn. J. Sci. Med. Sport 2, 106–116. doi: 10.1016/S1440-2440(99)80190-X

PubMed Abstract | CrossRef Full Text | Google Scholar

FINA World Championships (2022). Hungary Qualification and Entries of Competitors. Budapest. Available online at: https://swimswam.com/wp-content/uploads/2022/02/1_Budapest-2022_Qualification-System.pdf (accessed August 26, 2022).

Long, J. A. (2022). jtools: Analysis and Presentation of Social Scientific Data. Available online at: https://cran.r-project.org/web/packages/jtools/index.html (accessed August 26, 2022).

PubMed Abstract | Google Scholar

Lyttle, A. D., Blanksby, B. A., Elliott, B. C., and Lloyd, D. G. (1998). The effect of depth and velocity on drag during the streamlined guide. J. Swimming Res. 13, 15–22.

Google Scholar

Lyttle, A. D., Blanksby, B. A., Elliott, B. C., and Lloyd, D. G. (1999). Investigating kinetics in the freestyle flip turn push-off. J. Appl. Biomech. 15, 242–252. doi: 10.1123/jab.15.3.242

CrossRef Full Text | Google Scholar

Lyttle, A. D., Blanksby, B. A., Elliott, B. C., and Lloyd, D. G. (2000). Net forces during tethered simulation of underwater streamlined gliding and kicking techniques of the freestyle turn. J. Sports Sci. 18, 801–807. doi: 10.1080/026404100419856

PubMed Abstract | CrossRef Full Text | Google Scholar

Lyttle, A. D., and Mason, B. (1997). A kinematic and kinetic analysis of the freestyle and butterfly turns. J. Swimming Res. 12, 7–11.

PubMed Abstract | Google Scholar

Maglischo, E. W. (2003). Swimming Fastest. Champaign, IL: Human Kinetics.

Google Scholar

Marinho, D. A., Barbosa, T. M., Neiva, H. P., Silva, A. J., and Morais, J. E. (2020). Comparison of the start, turn and finish performance of elite swimmers in 100 m and 200 m races. J. Sports Sci. Med. 19, 397–407.

PubMed Abstract | Google Scholar

Mason, B., and Cossor, J. (2001). “Swim turn performances at Sydney 2000 Olympic Games,” in Biomechanics Symposia 2001 (San Francisco, CA: University of San Francisco), 65–69.

Google Scholar

McCabe, C., Mason, B., and Fowlie, J. (2012). “A temporal investigation into the butterfly kick placement following a breaststroke start and turn,” in 30th Annual Conference of Biomechanics in Sports - Melbourne. Conference Proceedings (Melbourne: New South Wales Institute of Sport), 200–203.

Google Scholar

Morais, J. E., Barbosa, T. M., Neiva, H. P., and Marinho, D. A. (2019a). Stability of pace and turn parameters of elite long-distance swimmers. Hum. Mov. Sci. 63, 108–119. doi: 10.1016/j.humov.2018.11.013

PubMed Abstract | CrossRef Full Text | Google Scholar

Morais, J. E., Marinho, D. A., Arellano, R., and Barbosa, T. M. (2019b). Start and turn performances of elite sprinters at the 2016 European Championships in swimming. Sports Biomech. 18, 100–114. doi: 10.1080/14763141.2018.1435713

PubMed Abstract | CrossRef Full Text | Google Scholar

Mosavi, S. H., Nikbakhsh, R., and Zafari, A. (2012). The effect of the combined training on the freestyle flip turn. Ann. Biol. Res. 3, 2078–2082.

Google Scholar

Nakagawa, S., and Schielzeth, H. (2013). A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods Ecol. Evolut. 4, 133–142. doi: 10.1111/j.2041-210x.2012.00261.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Nicol, E., Ball, K., and Tor, E. (2019). The biomechanics of freestyle and butterfly turn technique in elite swimmers. Sports Biomech. 20, 444–457. doi: 10.1080/14763141.2018.1561930

PubMed Abstract | CrossRef Full Text | Google Scholar

Pereira, S., Araujo, L., Freitas, E., Gatti, R., Silveira, G., and Roesler, H. (2006). Biomechanical analysis of the turn in front crawl swimming. Biomechan. Med. Swimm. 10(Suppl. 2), 77–79.

Pereira, S. M., Ruschel, C., Hubert, M., Machado, L., Roesler, H., Fernandes, R. J., et al. (2015). Kinematic, kinetic and EMG analysis of four front crawl flip turn techniques. J. Sports Sci. 33, 2006–2015. doi: 10.1080/02640414.2015.1026374

PubMed Abstract | CrossRef Full Text | Google Scholar

Puel, F., Morlier, J., Avalos, M., Mesnard, M., Cid, M., and Hellard, P. (2012). 3D kinematic and dynamic analysis of the front crawl tumble turn in elite male swimmers. J. Biomech. 45, 510–515. doi: 10.1016/j.jbiomech.2011.11.043

PubMed Abstract | CrossRef Full Text | Google Scholar

Puel, F., Morlier, J., Cid, M., Chollet, D., and Hellard, P. (2010a). Biomechanical factors influencing tumble turn performance of elite female swimmers. Biomech. Med. Swimm. 11, 155–157.

Google Scholar

Puel, F., Morlier, J., Mesnard, M., Cid, M., and Hellard, P. (2010b). Dynamics and kinematics in tumble turn: an analysis of performance. Comput. Methods Biomech. Biomed. Engin. 13, 109–111. doi: 10.1080/10255842.2010.495586

CrossRef Full Text | Google Scholar

R Core Team (2022). R: The R Project for Statistical Computing. Available online at: https://www.r-project.org/ (accessed August 26, 2022).

Google Scholar

Schweinberger, M. (2022). Fixed- and Mixed-Effects Regression Models in R. Brisbane, QLD: The University of Queensland. Available online at: https://slcladal.github.io/regression.html#Citation__Session_Info (accessed August 26, 2022).

Google Scholar

Shahbazi, M. M., Sanders, R. H., McCabe, C., and Adams, D. (2007). “Investigating correlations between swim pike turn kinematics variables in front crawl,” in XXV ISBS Symposium 2007 (Ouro Preto), 381–384.

Google Scholar

Shimadzu, H., Shibata, R., and Ohgi, Y. (2008). Modelling swimmers' speeds over the course of a race. J. Biomech. 41, 549–555. doi: 10.1016/j.jbiomech.2007.10.007

PubMed Abstract | CrossRef Full Text | Google Scholar

Silveira, G. A., Araujo, L. G., Freitas, E. D. S., Schütz, G. R., de Souza, T. G., Pereira, S. M., et al. (2011). Proposal for standardization of the distance for analysis of freestyle flip-turn performance. Revista Brasileira de Cineantropometria e Desempenho Humano 13, 177–182. doi: 10.5007/1980-0037.2011v13n3p177

CrossRef Full Text | Google Scholar

Skyriene, V., Dubosiene, M., Dubosas, M., and Eidukeviciute, M. (2017). The relationship between different age swimmers' flip turn temporal and kinematic characteristics. Mechanika 23, 604–609. doi: 10.5755/j01.mech.23.4.17852

CrossRef Full Text | Google Scholar

Smithdorf, G. (2018). Effect of Tumble Turns on Swimming Performance in Level 3 Swimmers. Cape Town: Faculty of Community and Health Sciences, University of the Western Cape, Magister Sport, Recreation and Exercise Science - MSRES.

Google Scholar

Suito, H., Nunome, H., and Ikegami, Y. (2015). “Relationship between 100 m race times and start, stroke, turn, finish phases at the freestyle Japanese swimmers”, in ISBS-Conference Proceedings Archive (Portiers).

Google Scholar

Takahashi, G., Yoshida, A., Tsubakimoto, S., and Miyashita, M. (1982). “Propulsive force generated by swimmers during a turning motion,” in Biomechanics and Medicine in Swimming: Proceedings of the Fourth International Symposium of Biomechanics in Swimming and the Fifth International Congress on Swimming Medicine Held in Amsterdam, The Netherlands, ed A. P. Hollander.

Toshiaki, G., Takagi, H., and Nomura, T. (2001). “The turning motion and forces involved in the backstroke flip turn,” in Biomechanics Symposia 2001 (San Francisco, CA: University of San Francisco), 87–89.

Google Scholar

Tourny-Chollet, C., Chollet, D., Hogie, S., and Papparodopoulos, C. (2002). Kinematic analysis of butterfly turns of international and national swimmers. J. Sports Sci. 20, 383–390. doi: 10.1080/026404102317366636

PubMed Abstract | CrossRef Full Text | Google Scholar

Veiga, S., Cala, A., Frutos, P. G., and Navarro, E. (2013). Kinematical comparison of the 200 m backstroke turns between national and regional level swimmers. J. Sports Sci. Med. 12, 730–737.

PubMed Abstract | Google Scholar

Veiga, S., and Roig, A. (2016). Underwater and surface strategies of 200 m world level swimmers. J. Sports Sci. 34, 766–771. doi: 10.1080/02640414.2015.1069382

PubMed Abstract | CrossRef Full Text | Google Scholar

Webster, J. M., West, A., Conway, P., and Cain, M. (2011). Development of a pressure sensor for swimming turns. Procedia Eng. 13, 126–132. doi: 10.1016/j.proeng.2011.05.062

CrossRef Full Text | Google Scholar

Welham, S. J., and Thompson, R. (1997). Likelihood ratio tests for fixed model terms using residual maximum likelihood. J. R. Statistical Soc. Ser. B Statistical Methodol. 59, 701–714. doi: 10.1111/1467-9868.00092

PubMed Abstract | CrossRef Full Text | Google Scholar

Wickham, H., Bryan, J., and R. Studio (2022). readxl: Read Excel Files. Available online at: https://cran.r-project.org/web/packages/readxl/index.html