Preliminary Checklist for Reporting Observational Studies in Sports Areas: Content Validity

Observational studies are based on systematic observation, understood as an organized recording and quantification of behavior in its natural context. Applied to the specific area of sports, observational studies present advantages when comparing studies based on other designs, such as the flexibility for adapting to different contexts and the possibility of using non-standardized instruments as well as a high degree of development in specific software and data analysis. Although the importance and usefulness of sports-related observational studies have been widely shown, there is no checklist to report these studies. Consequently, authors do not have a guide to follow in order to include all of the important elements in an observational study in sports areas, and reviewers do not have a reference tool for assessing this type of work. To resolve these issues, this article aims to develop a checklist to measure the quality of sports-related observational studies based on a content validity study. The participants were 22 judges with at least 3 years of experience in observational studies, sports areas, and methodology. They evaluated a list of 60 items systematically selected and classified into 12 dimensions. They were asked to score four aspects of each item on 5-point Likert scales to measure the following dimensions: representativeness, relevance, utility, and feasibility. The judges also had an open-format section for comments. The Osterlind index was calculated for each item and for each of the four aspects. Items were considered appropriate when obtaining a score of at least 0.5 in the four assessed aspects. After considering these inclusion criteria and all of the open-format comments, the resultant checklist consisted of 54 items grouped into the same initial 12 dimensions. Finally, we highlight the strengths of this work. We also present its main limitation: the need to apply the resultant checklist to obtain data and, thus, increase quality indicators of its psychometric properties. For this reason, as relevant actions for further development, we encourage expert readers to use it and provide feedback; we plan to apply it to different sport areas.

Introduction

Observational studies are mainly based on systematic observation, understood as an organized recording and quantification of behavior in its natural context (Anguera, 1979, 1996, 2003). These types of studies involve a low level of intervention (Chacón-Moscoso et al., 2013). Observational studies present the following important advantages compared to those with a medium or high level of intervention (Portell et al., 2015), such as: (1) they can be adapted to any situation in any setting and (2) they do not need standardized measurement instruments because the context of the study is prioritized and, as a consequence, the use of ad hoc instruments is accepted.

Observational studies are commonly used in many areas, such as social (Anguera and Redondo, 1991; Santoyo and Anguera, 1992), psycho-pedagogical (Moya et al., 2012; Herrero-Nivela et al., 2014), clinical (Roustan et al., 2013; Arias et al., 2015), or sport (Weigelt and Memmert, 2012; Anguera and Hernández-Mendo, 2014) studies. The concrete field of observational methodology, as applied to sports, currently benefits from the advanced development of statistical analyses and specific software to study men and women's sports-related behaviors in order to obtain indicators to improve their performance (Anguera and Hernández-Mendo, 2015; Anguera et al., 2017, 2018). For example, (1) sequential analysis of behaviors using SDIS-GSEQ software (Bakeman and Quera, 2011) has been developed to establish model sequences by high-level sportsmen and women (Castelão et al., 2015); (2) the use of polar coordinate analysis by means of HOISAN software (Hernández-Mendo et al., 2012) enables the study of interrelations between different categories of observational tools in different sports, such as tae kwon do (López-López et al., 2015), handball (Sousa et al., 2015), or soccer (Castellano and Hernández-Mendo, 2003; Castañer et al., 2016); and (3) T-pattern analysis using Theme software (Magnusson, 1996, 2000) can be applied to discover hidden structures in the observed behavior that are not directly visible in elite climbing (Arbulu et al., 2016), futsal (hard-court soccer, Sarmento et al., 2016), synchronized swimming (Iglesias et al., 2015), or bouts epee (fencing, Tarrag et al., 2015).

Although observational studies are frequently used and their utility in different contexts has been widely proven, a tool to measure the reporting quality of these types of studies does not exist, nor does a specific one for sports areas (Portell et al., 2015). This lack causes important consequences for observational studies in sports areas: (1) the author's report is the unique information we usually have about primary studies (Altman et al., 2001; Grimshaw et al., 2006; Cornelius et al., 2009). As authors do not have a checklist for reporting, transparency may be affected, and important information for assessing the quality of the study and, therefore, its degree of risk of bias, may be omitted (Portell et al., 2015). (2) Authors who want to publish these kinds of studies do not have a checklist to confirm that all the important elements were considered in the study and included in the report, and reviewers of these same studies lack a useful tool for determining the indicators to consider when accepting or rejecting their publication in a scientific journal (Chacón-Moscoso et al., 2016).

Checklists to measure the quality of the reporting of primary studies in general, without specifying the design type, have previously been published, e.g., by the Journal Article Reporting Standards (JARS) (American Psychological Association, 2010). In addition, as a consequence of the differences existing across designs, checklists with the same purpose but for specific study designs have been published (Portell et al., 2015; Chacón-Moscoso et al., 2016). For example, for high-intervention designs (randomized control trials), we have the Consolidated Standards of Reporting Trials (CONSORT) (Schulz et al., 2010); for epidemiological studies, such as cohort, case-control, and cross-sectional studies, we have the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement (von Elm et al., 2007); (3) for intensive repeated measurements in naturalistic settings, we have the Guidelines for Reporting Momentary Studies (Stone and Shiffman, 2002); (4) for qualitative studies, we have the Guidelines for Qualitative Research Methodologies (Blignault and Ritchie, 2009); and (5) for mixed methods, we have the Guidelines for Conducting and Reporting Mixed Research for Counselor Researchers (Leech and Onwuegbuzie, 2010). The current standard for low-intervention studies is the Guidelines for Reporting Evaluations Based on Observational Methodology (GREOM) (Portell et al., 2015; included in the EQUATOR library: http://www.equator-network.org/reporting-guidelines/guidelines-for-reporting-evaluations-based-on-observational-methodology/) which, combined with JARS, provides a general view of the structural characteristics of observational designs that must be considered for evaluation in low-intervention situations without specifying any concrete area.

The aim of this work is to develop a checklist to measure the quality of the reporting of sports-related observational studies. This checklist will further clarify the general guidelines presented in the GREOM; determine the main quality indicators of the reporting of sports-related observational studies; serve as a useful tool for authors conducting and publishing observational studies in this area, as well as for reviewers making decisions for publications; and present indicators of the representativeness (REP), relevance (REL), utility (U), and feasibility (F) of the developed tool to measure the quality of the reporting based on a content validity study.

Methods

Participants

Twenty-two of the 43 potential candidates who were contacted opted to participate in the study, resulting in a participation rate of 51.2%. The inclusion criteria for eligibility were a minimum of 3 years of experience in observational studies, sports areas, and methodology. The sample participants were between 31 and 70 years old [mean (M) = 45.9, standard deviation (SD) = 11.3], including 17 men (77.3%) and 5 women (22.7%). Their years of experience in methodology were between 3 and 44 (M = 17.1, SD = 11.7). Their years of experience in observational studies ranged from 3 to 35 (M = 13.6, SD = 9.3). Finally, their years of experience in sports areas ranged between 3 and 40 (M = 19.9, SD = 10).

The main dedication and sports interest of these experts was in the area of physical education (8 participants, 36.4% of the sample), understood as the set of disciplines that aim to develop the human body through sports participation and encourages psychomotor learning in a game-like setting or through movement exploration. This is a commonly introduced subject in primary and secondary education curricula (Woodward, 2016). The second most frequent area of interest was high-performance sports (five participants, 22.7%), referring to the type of activity (individual and team) in competition contexts (Harenberg et al., 2016). The third area of interest was sport initiation (4 participants, 18.2%), the process by which a person makes contact with new experiences in a physical activity or sport (Thomas et al., 2015). The areas of health, sports, and physical education (a set of educational, sporting, and organizational practices to promote well-being and health; Williams and Macdonald, 2015) and adapter sports (sports practices of people with some kind of physical and/or psychological disability; Park and Sinelnikov, 2016) were chosen by two participants each (9.1%). Finally, one participant (4.5%) chose technology, defined as the tools aimed to improve athletes' sports performance in order to set personal records and, thus, be more competitive (Hardcastle et al., 2015).

Instruments

Appendix I in Supplementary Material presents the instrument we designed to enable content validity experts to determine the main aspects of sports-related observational studies. It is composed of 60 items representing 12 dimensions from the GREOM (Portell et al., 2015): (1) Extrinsic characteristics (1 item); (2) Objectives delimitation (6 items); (3) Observational design (3 items); (4) Participants (9 items); (5) Context-setting (11 items); (6) Observational instrument (7 items); (7) Recording instrument (6 items); (8) Data (3 items); (9) Parameter specification (2 items); (10) Observational sampling (6 items); (11) Data quality control (5 items); and (12) Data analysis (1 item).

For the content validity study, four 5-point Likert scales (Sanduvete-Chaves et al., 2013) were associated with each item to be assessed by the experts referring to four different aspects with respect to its dimension: (1) REP referred to the degree to which each item represented the dimension to which it had been assigned; (2) REL was defined as the extent to which each item was important or highlighted something of the dimension in which it was included; (3) U referred to the extent to which each item was useful to evaluate the dimension to which it was assigned; and (4) F was defined as the possibility of recording information about each item. Additionally, an open-format item (comments) was available for experts who wished to propose something new, such as improving the writing of an item or exchanging it for something more appropriate.

This instrument was available in two formats: the Internet format using Google Drive Forms and a paper version. Microsoft Excel was used for the data analysis.

Procedure

Ethical approval and written informed consent were not needed for our study, as the participants were experts, a non-vulnerable group, and the information gathered was professional opinions about the adequacy of different items used to report observational studies, without medical, clinical, or personal implications.

Item Selection and Assignment to Dimensions

We delimited the main dimensions of observational studies and a list of items to measure those dimensions based on three information sources: (1) A systematic review (Chacón-Moscoso et al., 2016) was conducted of 12 databases that were of interest due to their content (Web of Science, Scopus, Springer, EBSCO Online, Medline, CINAHL, EconLit, MathSciNet, Current Contents, Humanities Index, ERIC, and PsycINFO). We found 548 different ways to measure methodological quality in primary studies. From this total, some of the tools were general reporting standards not specific to any particular research design (e.g., Zaza et al., 2000; American Educational Research Association, 2006; American Psychological Association, 2010; Möhler et al., 2012), while others were specific reporting standards for research designs with some similarities to observational designs (Stone and Shiffman, 2002; Tong et al., 2007; Blignault and Ritchie, 2009; Pluye et al., 2009; Leech and Onwuegbuzie, 2010). (2) The GREOM (Portell et al., 2015) represents the specific guidelines for developing observational studies. As an illustrative example of the GREOM's high degree of influence over the list of items gathered, apart from the common structure, we can see the direct correspondence in dimensions of the present checklist 6 Observational instrument, 7 Recording instrument, and 9 Parameters specification, according to section B2 Instruments of GREOM, including the guidelines 7 Observation instrument, 8 Primary recording parameters, and 9 Recording instruments. (3) The final dimension involves sports-related observational studies found in the previously cited databases (Anguera and Hernández-Mendo, 2015).

Two coders separately studied the degree of agreement in the items dimension assigned and intercoder reliability (Nimon et al., 2012; Stolarova et al., 2014) was studied by calculating Cohen's κ (Cohen, 1960). Any disagreements were resolved by consensus.

Content Validity Study

Once the 60 items were selected and assigned to one of the 12 dimensions chosen, the experts were asked, through 5-point rating scales, about the REP, REL, U, and F of each item with respect to its dimension.

The instrument was sent to experts in English (Appendix I in Supplementary Material) or Spanish (Supplementary Material), depending on their native language. We provided the access link to the instrument in Google Drive by sending an email to the potential experts that satisfied the participant's inclusion criteria. Fifteen days later, we reminded the participants that the instrument was available to be fulfilled in the same link. After another 15 days, we made the last call for answers in the same way. After a final 15 days, the application was definitively closed. As part of the final gathering stage, the same instrument was available in paper format for all of the participants (who worked in observational studies, methodology, and sports areas) at the VII European Congress of Methodology, held in Palma de Mallorca (Spain) in July 2016. Throughout the entire process, the information was gathered anonymously.

After gathering the information, the Osterlind index of congruence (Osterlind, 1998) was calculated for each item and each aspect measured (REP, REL, U, and F). The formula used was.

$\begin{array}{l}{I}_{ik}=\frac{(N-1){\displaystyle \sum _{j=1}^n}{X}_{ijk}+N{\displaystyle \sum _{j=1}^n}{X}_{ijk}-{\displaystyle \sum _{j=1}^n}{X}_{ijk}}{2(N-1)n}\end{array}$

where N = number of dimensions (12 in this case), X_ijk = score provided by each expert to each item referred to each aspect measured, and n = number of experts. The scores were provided in a 5-point Likert scale (−1 = strongly disagree, −0.5 = disagree, 0 = neither agree nor disagree, 0.5 = agree, and 1 = strongly agree) instead of the classical one with 3 points, to make the achievement of high results slightly difficult, as the 5-point version is more conservative (Revised Osterlind Index, Sanduvete-Chaves et al., 2013).

The results of the previous formula ranged from −1 to +1. Minus one implied a total agreement among the experts, indicating that all answered that they disagreed strongly; 1 meant a total agreement among the experts, positioning all in strong agreement; and 0 represented the highest possible disagreement among the experts.

Based on the criteria (Osterlind, 1998), items that obtained a score of 0.5 or higher on the four aspects measured were included in the final version of the checklist for reporting observational studies.

Results

The assignment of the 60 items selected to the 12 dimensions made by two independent researchers obtained a degree of consensus of κ = 0.76 (p < 0.001) and a 95% confidence interval (CI) of [0.646, 0.874]. This result can be considered appropriate (Landis and Koch, 1977).

Forty-three experts were contacted by email to fulfill the content validity questionnaire. A total of 14 experts answered via Google Drive. Two participants sent their responses after the first call for answers, five participants answered in the second round, and seven additional experts gave their opinions in the final round. Additionally, eight experts fulfilled the questionnaire in paper format during the VII European Congress of Methodology (July 2016). The total number of answers gathered was 22. According to Prieto and Muñiz (2000), a number of experts ranging from 10 to 30 through a systematic procedure can be considered a moderate sample size.

Table 1 presents the Osterlind indexes obtained for each item referring to REP, REL, U, and F. Fifty-three items met the criterion of having a result of 0.5 or higher in these four aspects. Only seven items were removed because they did not meet this criterion: those in dimension 4 referred to the participants, items 12 (cultural background), 13 (socio-economic level), 17 (differential exclusion of participants), and 18 (participants' allocation); those in dimension 5 referred to the context (setting), item 24 (number of non-observable periods); those in dimension 6 referred to the observational instrument, item 34 (criteria that lead to the catalogs and categories systems); and those in dimension 9 referred to the parameters specification, item 48 (parameters fitting). The removed items appear in bold text in Table 1.

TABLE 1

Table 1. Osterlind indexes obtained for each item in representativeness (REP), relevance (REL), utility (U), and feasibility (F).

Analyzing all of the items as a whole and taking into account that the possible results ranged from −1 to 1, we found that, in REP, Mdn = 0.71 (SD = 0.14), with the values ranging from 0.3 to 0.98; in REL, Mdn = 0.69, SD = 0.16, range = 0.21–1; in U, Mdn = 0.69, SD = 0.15, range = 0.33–0.98; finally, in F, Mdn = 0.73, SD = 0.14, range = 0.28–0.93.

Table 2 presents the open-format comments made by the experts and the actions developed in order to follow their advice. From a total of 22 different comments, all were followed with the exception of one (item 11), to which we made only a partial change. Four comments did not imply changes because they referred to items excluded by the Osterlind index results.

TABLE 2

Table 2. Open-format comments provided by experts and actions taken as a consequence.

All of the comments presented were provided by only one expert, except those referring to the graduation of the answers for some dichotomous items, which were proposed by five experts.

Appendix II in Supplementary Material presents the final version of the checklist for reporting sports-related observational studies after making the changes derived from the results of the Osterlind indexes and the experts' open-format comments. One proposal provided in open format was to add one more item. Originally, the instrument presented 60 items, and 7 were removed due to the Osterlind indexes, resulting in the inclusion of 54 items in the final version.

Discussion

We propose a 54-item and 12-dimension checklist to measure the reporting quality of observational studies in sports areas. Its use by authors and reviewers may contribute to the increased transparency of these studies, as it lists the main aspects to consider and delimit when designing, executing, or evaluating observational studies in sports areas. The importance of this checklist resides in its exclusivity, considering that no other tool with this same purpose exists in the literature. There are other checklists available with the same objective as our proposal, that is, to measure the quality of reporting, although to be applied in other contexts (e.g., in orthopedics, Mundi et al., 2008) and other kinds of designs (e.g., in orthopedics and randomized control trials, Chan and Bhandari, 2007). Additionally, there are checklists in sports (e.g., Arnold and Schilling, 2017), but in designs different from observational studies (Anguera et al., 2018; as guidelines created for this methodology, readers can see the GREOM included in the EQUATOR library: http://www.equator-network.org/reporting-guidelines/guidelines-for-reporting-evaluations-based-on-observational-methodology/). On other occasions, we find checklists applied to similar designs (e.g., STROBE for epidemiological studies, von Elm et al., 2007), although not exactly for observational studies understood as an organized recording and quantification of behavior in its natural context.

These checklists present some characteristics in common with our proposal, such as the format (closed-ended questions) or the capacity to detect relevant information that has not been reported. Nevertheless, they differ in content, not only due to the sport context [e.g., item 14, Sport modality: (1) Individual sport, (2) Team sport; or item 15, Professionalism: (1) Professionals, (2) Semi-professionals, (3) Sportsmen/women in training stage], but also due to the casuistic of the observational design [e.g., item 9, Justification of the observational design: (1) No, (2) Yes; or item 24, Number of non-observable periods].

The main strength of this work is that the content validity study was developed through a clear, careful, and explicit process, so it presented a high degree of reproducibility. In this way, we were able to define a list of items based on different sources of information: a systematic review, the GREOM as the theoretical framework and the basis for the 12 delimited dimensions and content in several dimensions (illustrated in the correspondence between dimensions 6 Observational instrument, 7 Recording instrument, and 9 Parameters specification of the presented checklist and guidelines 7 Observation instrument, 8 Primary recording parameters, and 9 Recording instruments, corresponding to section B2 Instruments of the GREOM) (Portell et al., 2015), and published observational studies in sports areas. We provided the full list of items assessed by the experts in English (Appendix I in Supplementary Material) and Spanish (Supplementary Material). We determined the inclusion criterion a priori; we reported the Osterlind index for all of the items in the four aspects measured (Table 1). We objectively applied the previously established inclusion criterion and transcribed all of the open-format comments provided by the experts and each action we executed in answer to each comment (Table 2). After considering the Osterlind indexes and open-format comments, we presented the final version of the checklist for reporting sports-related observational studies (Appendix II in Supplementary Material).

Additionally, we obtained adequate results for the fitness item dimension with respect to four aspects: REP, REL, U, and F, which provides a quality indicator of the content validity in favor of the use of the resulting tool as appropriate. The resultant checklist is expected to be extensively useful, as it can be applied to any sports area.

On the other hand, the main limitation we found in the checklist obtained is that it supposes a preliminary proposal in which further development is needed to increase the quality indicators of its psychometric properties. For this purpose, we encourage and urge expert readers to improve our final version checklist (Appendix II in Supplementary Material) with their comments or results regarding its application.

Additionally, we plan to apply the checklist to different sports areas in order to demonstrate that it is an adequate measurement instrument independent of the sport context and to develop an intercoder reliability study to locate discrepancies across the independent coding of a high number of studies (more than 40) by two different previously trained coders. We consider this proposal as open and in progress, as we will continue to consider additional comments for the improvement of the checklist that we might receive by experts.

Taking this work as the basis, we plan to develop a scale to measure methodological quality in sports-related observational studies. This checklist can serve as a guideline for measuring the reporting quality of these studies because it lists the main aspects to consider when designing, executing, and evaluating a sports-related observational study. We can also recommend concrete actions to increase the methodological quality of these studies.

Ethics Statement

This study was carried out in accordance with the recommendations of the Declaration on bioethics and human rights, UNESCO, 2005 with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Ethics Committee, Universidad Autónoma de Chile.

Author Contributions

All of the authors contributed to documenting, designing, drafting, and writing the manuscript, and revised it for important theoretical and intellectual content. Additionally, all of the authors provided final approval of the version to be published and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Funding

This research was funded by projects with the following reference numbers: 1150096 (Chilean National Fund of Scientific and Technological Development, FONDECYT); PSI2015-71947-REDT (Spain's Ministry of Economy and Competitiveness), DEP2015-66069-P (Spain's Ministry of Economy and Competitiveness, European Regional Development Fund), and PSI2011-29587 (Spain's Ministry of Science and Innovation). We gratefully acknowledge the support of the Generalitat de Catalunya Research Group (GRUP DE RECERCA E INNOVACIÓ EN DISSENYS [GRID]). Tecnología i aplicació multimedia i digital als dissenys observacionals, [Grant 2017 SGR 1405].

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors greatly appreciate all of the comments received from the reviewers and the English-language editor. We believe that the quality of this paper has been substantially enhanced as a result.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2018.00291/full#supplementary-material

References

Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., et al. (2001). The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann. Intern. Med. 134, 663–694. doi: 10.7326/0003-4819-134-8-200104170-00012

PubMed Abstract | CrossRef Full Text | Google Scholar

American Educational Research Association (2006). Standards for reporting on empirical social science research in AERA publications. Educ. Res. 35, 33–40. doi: 10.3102/0013189X035006033

CrossRef Full Text

American Psychological Association (2010). Publication Manual of the American Psychological Association, 6th Edn. Washington, DC: American Psychological Association.

Anguera, M. T. (1979). Observational typology. Qual. Quant. 13, 449–484.

Google Scholar

Anguera, M. T. (1996). Introduction. Eur. J. Psychol. Assess. 12, 87–88. doi: 10.1027/1015-5759.12.2.87

CrossRef Full Text | Google Scholar

Anguera, M. T. (2003). “Observational methods (General),” in Encyclopedia of Behavioral Assessment, Vol. 2, ed R. Fernández-Ballesteros (London: Sage), 632–637.

Anguera, M. T., and Hernández-Mendo, A. (2014). Metodología observacional y psicología del deporte: estado de la cuestión [Observational methodology and sport psychology: state of the art]. Rev. Psicol. Dep. 23, 103–109. Available online at: http://www.rpd-online.com/article/view/v23-n1-anguera-hernandez-mendo/pdf_es

Anguera, M. T., and Hernández-Mendo, A. (2015). Técnicas de análisis en estudios observacionales en ciencias del deporte [Analyses techniques in observartional studies in sport science]. Cuadern. Psicol. Dep. 15, 13–30. doi: 10.4321/S1578-84232015000100002

CrossRef Full Text | Google Scholar

Anguera, M. T., and Redondo, S. (1991). La evaluación de la intervención penitenciaria [The evaluation of the penitentiary intervention]. Delincuen. Delinq. 3, 245–289.

Anguera, M. T., Camerino, O., Castañer, M., Sánchez-Algarra, P., and Onwuegbuzie, A. J. (2017). The specificity of observational studies in physical activity and sports sciences: Moving forward in mixed methods research and proposals for achieving quantitative and qualitative symmetry. Front. Psychol. 8:2196. doi: 10.3389/fpsyg.2017.02196

PubMed Abstract | CrossRef Full Text | Google Scholar

Anguera, M. T., Portell, M., Chacón-Moscoso, S., and Sanduvete-Chaves, S. (2018). Indirect observation in everyday contexts: concepts and methodological guidelines within a mixed methods framework. Front. Psychol. 9:13. doi: 10.3389/fpsyg.2018.00013

PubMed Abstract | CrossRef Full Text | Google Scholar

Arbulu, A., Lapresa, D., Usabiaga, O., and Castellano, J. (2016). Detección y aplicación de T-patterns en la escalada de élite [Detection and application of T-patterns in elite climbing]. Cuadern. Psicol. Dep. 16, 95–102. Available online at: http://hdl.handle.net/10201/48609

Google Scholar

Arias, E., Fieschi, E., Miralbell, J., Castellò, C., Soldevila, A., Anguera, M. T., et al. (2015). Uso de la imitación en psicoterapia psicoanalítica con un niño con Trastorno del Espectro Autista grave: efectos en la interacción social recíproca [Use of imitation in psychoanalytic psychotherapy with a child with severe spectrum disorder autism: effects on reciprocal social interaction]. Rev. Psicopatol. Salud Mental Niño Adolesc. 25, 9–20.

Arnold, B. L., and Schilling, B. K. (2017). Evidence-Based Practice in Sport and Exercise: A Guide to Using Research. Philadelphia, PA: Davis Company.

Google Scholar

Bakeman, R. (1978). “Untangling streams of behavior: sequential analysis of observation data,” in Observing Behavior, Data Collection and Analysis Methods, Vol. 2, ed G. P. Sackett (Baltimore, MD: University of Park Press), 63–78.

Google Scholar

Bakeman, R. (1983). Computing lag sequential analysis statistics: the ELAG program. Behav. Res. Methods Instrum. 15, 530–535. doi: 10.3758/BF03203700

CrossRef Full Text | Google Scholar

Bakeman, R., and Quera, V. (2011). Sequential Analysis and Observational Methods for the Behavioral Sciences. Cambridge: Cambridge University Press.

Google Scholar

Blignault, I., and Ritchie, J. (2009). Revealing the wood and the trees: reporting qualitative research. Health Promot. J. Austr. 20, 140–145. doi: 10.1071/HE09140

PubMed Abstract | CrossRef Full Text | Google Scholar

Castañer, M., Barreira, D., Camerino, O., Anguera, M. T., Canton, A., and Hileno, R. (2016). Goal scoring in soccer: a polar coordinate analysis of motor skills used by Lionel Messi. Front. Psychol. 7:806. doi: 10.3389/fpsyg.2016.00806

PubMed Abstract | CrossRef Full Text | Google Scholar

Castelão, D. P., Garganta, J., Afonso, J., and da Costa, I. T. (2015). Análise seqüencial de comportamentos ofensivos desempenhados por seleções nacionais de futebol de alto rendimento [sequential analysis of attack behaviors carried out by high-performance national teams in soccer]. Rev. Brasil. Ciênc. Esp. 37, 230–236. doi: 10.1016/j.rbce.2015.05.001

CrossRef Full Text | Google Scholar

Castellano, J., and Hernández-Mendo, A. (2003). El análisis de coordenadas polares para la estimación de relaciones en la interacción motriz en fútbol [The analysis of polar coordenates to estimate the relationships in the motrix interaction in soccer]. Psicothema 15, 569–574. Available online at: http://www.psicothema.com/pdf/1109.pdf

Google Scholar

Chacón-Moscoso, S., Sanduvete-Chaves, S., and Sánchez-Martín, M. (2016). The development of a checklist to enhance methodological quality in intervention programs. Front. Psychol. 7:1811. doi: 10.3389/fpsyg.2016.01811

PubMed Abstract | CrossRef Full Text | Google Scholar

Chacón-Moscoso, S., Sanduvete-Chaves, S., Portell, M., and Anguera, M. T. (2013). Reporting a program evaluation: needs, program plan, intervention, and decisions. Int. J. Clin. Health Psychol. 13, 58–66. doi: 10.1016/S1697-2600(13)70008-5

CrossRef Full Text | Google Scholar

Chan, S., and Bhandari, M. (2007). The quality of reporting of orthopaedic randomized trials with use of a checklist for nonpharmacological therapies. J. Bone Joint Surg. Am. 89, 1970–1978. doi: 10.2106/jbjs.f.01591

PubMed Abstract | CrossRef Full Text | Google Scholar

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20, 37–46.

Google Scholar

Cornelius, V. R., Perrio, M. J., Shakir, S. A. W., and Smith, L. A. (2009). Systematic reviews of adverse effects of drug interventions: a survey of their conduct and reporting quality. Pharmacoepidemiol. Drug Saf. 18, 1223–1231. doi: 10.1002/pds.1844

PubMed Abstract | CrossRef Full Text | Google Scholar

Grimshaw, J., Eccles, M., Thomas, R., MacLennan, G., Ramsay, C. R., Fraser, C., et al. (2006). Toward evidence-based quality improvement. evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966-1998. J. Gen. Internal Med. 21(Suppl. 2), 14–20. doi: 10.1111/j.1525-1497.2006.00357.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Hardcastle, S. J., Tye, M., Glassey, R., and Hagger, M. S. (2015). Exploring the perceived effectiveness of a life skills development program for high-performance athletes. Psychol. Sport Exerc. 16, 139–149. doi: 10.1016/j.psychsport.2014.10.005

CrossRef Full Text | Google Scholar

Harenberg, S., Riemer, H. A., Jarraman, E., and Dorsch, K. D. (2016). Coaches' perspectives of intrateam competition in high performance sport teams. Int. Sport Coach. J. 3, 156–169. doi: 10.1123/iscj.2016-0056

CrossRef Full Text | Google Scholar

Hernández-Mendo, A., López-López, J. A., Castellano, J., Morales-Sánchez, V., and Pastrana, J. L. (2012). Hoisan 1.2: programa informático para uso en metodología observacional [Hoisan 1.2: software for observational methodology]. Cuadern. Psicol. Dep. 12, 55–78. doi: 10.4321/S1578-84232012000100006

CrossRef Full Text | Google Scholar

Herrero-Nivela, M. L., Escolano-Pérez, E., Anguera, M. T., and Acero-Ferrero, M. (2014). Evaluación mediante observación de la motricidad en niños de Educación Infantil [evaluation by means of observation of motor skills in children in early childhood education]. Int. J. Dev. Educ. Psychol. 5, 469–476. doi: 10.17060/ijodaep.2014.n1.v5.708

CrossRef Full Text | Google Scholar

Iglesias, X., Rodríguez-Zamora, L., Chaverri, D., Clapés, P., Rodríguez, F. A., and Anguera, M. T. (2015). Diversificación de patrones en rutinas de solo en natación sincronizada de alto nivel [Patterns of solo routines in high-performance synchronized swimming]. Cuadern. Psicol. Dep. 15, 89–98. doi: 10.4321/S1578-84232015000100009

CrossRef Full Text | Google Scholar

Landis, J. R., and Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33, 159–174. doi: 10.2307/2529310

PubMed Abstract | CrossRef Full Text | Google Scholar

Leech, N. L., and Onwuegbuzie, A. J. (2010). Guidelines for conducting and reporting mixed research in the field of counseling and beyond. J. Counsel. Dev. 88, 61–69. doi: 10.1002/j.1556-6678.2010.tb00151.x

CrossRef Full Text | Google Scholar

López-López, J. A., Menescardi, C., Estevan, I., Falcó, C., and Hernández-Mendo, A. (2015). Technical-tactical analysis in Taekwondo with polar coordinates through software HOISAN. Cuadern. Psicol. Dep. 15, 131–142. doi: 10.4321/S1578-84232015000100013

CrossRef Full Text | Google Scholar

Magnusson, M. S. (1996). Hidden real-time patterns in intra- and inter-individual behavior. Eur. J. Psychol. Assess. 12, 112–123. doi: 10.1027/1015-5759.12.2.112

CrossRef Full Text | Google Scholar

Magnusson, M. S. (2000). Discovering hidden time patterns in behavior: T-patterns and their detection. Behav. Res. Methods Instrum. Comp. 32, 93–110. doi: 10.3758/BF03200792

PubMed Abstract | CrossRef Full Text | Google Scholar

Möhler, R., Bartoszek, G., Köpke, S., and Meyer, G. (2012). Proposed criteria for reporting the development and evaluation of complex interventions in healthcare (CReDECI): guideline development. Int. J. Nurs. Stud. 49, 40–46. doi: 10.1016/j.ijnurstu.2011.08.003

PubMed Abstract | CrossRef Full Text | Google Scholar

Moya, J., de Armas, M., and Anguera, M. T. (2012). Els problemas de comportament en els nens i adolescents de Catalunya: estudi descriptiu del trastorn per déficit d'atenció / trastorn de conducta. Una reflexió epistemológica [The behavioral problems in children and adolescents in Catalonia: a descriptive study of attention deficit disorder / behavior disorder. an epistemological approach]. Àmb. Psicopedag. 35, 8–15. Available online at: http://www.psiara.cat/view_article.asp?id=4218

Google Scholar

Mundi, R., Chaudhry, H., Singh, I., and Bhandari, M. (2008). Checklists to improve the quality of the orthopaedic literature. Indian J. Orthop. 42, 150–164. doi: 10.4103/0019-5413.40251

PubMed Abstract | CrossRef Full Text | Google Scholar

Nimon, K., Zientek, L. R., and Henson, R. K. (2012). The assumption of a reliable instrument and other pitfalls to avoid when considering the reliability of data. Front. Psychol. 3:102. doi: 10.3389/fpsyg.2012.00102

PubMed Abstract | CrossRef Full Text | Google Scholar

Osterlind, S. J. (1998). Constructing Tests Items. Boston, MA: Kluwer Academic Publishers.

Google Scholar

Park, C. W., and Sinelnikov, O. A. (2016). Changing college students' perception on disability through adapted sports. Res. Q. Exerc. Sport 87, 1. doi: 10.1080/02701367.2016.1217725

CrossRef Full Text | Google Scholar

Pluye, P., Gagnon, M. P., Griffiths, F., and Johnson-Lafleur, J. (2009). A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int. J. Nurs. Stud. 46, 529–546. doi: 10.1016/j.ijnurstu.2009.01.009

PubMed Abstract | CrossRef Full Text | Google Scholar

Portell, M., Anguera, M. T., Chacón-Moscoso, S., and Sanduvete-Chaves, S. (2015). Guidelines for reporting evaluations based on observational methodology. Psicothema 27, 283–289. doi: 10.7334/psicothema2014.276

PubMed Abstract | CrossRef Full Text | Google Scholar

Prieto, G., and Muñiz, J. (2000). Un modelo para evaluar la calidad de los test utilizados en España. Pap. Psicól. 77, 65–75. Available online at: http://www.redalyc.org/articulo.oa?id=77807709

Google Scholar

Roustan, M., Izquierdo, C., and Anguera, M. T. (2013). Sequential analysis of an interactive peer support group. Psicothema 25, 396–401. doi: 10.7334/psicothema2012.93

PubMed Abstract | CrossRef Full Text | Google Scholar

Sánchez-Meca, J. (1997). “Methodological issues in the meta-evaluation of correctional treatment,” in Advances in Psychology and Law. International Contributions, eds S. Redondo, V. Garrido, J. Pérez, and R. Barberet (New York, NY: Walter de Gruyter), 486–498.

Google Scholar

Sánchez-Meca, J., and Ato, M. (1989). “Meta-análisis: una alternativa metodológica a las revisiones tradicionales de la investigación [Meta-analysis: a methodological alternative to traditional research reviews],” in Tratado de Psicología General, eds J. Arnau and H. Carpintero (Madrid: Alhambra), 617–669.

Google Scholar

Sanduvete-Chaves, S., Chacón-Moscoso, S., Sánchez-Martín, M., and Pérez-Gil, J. A. (2013). The revised osterlind index. a comparative analysis in content validity studies. Acc. Psicol. 10, 19–26. doi: 10.5944/ap.10.2.11821

CrossRef Full Text | Google Scholar

Santoyo, C., and Anguera, M. T. (1992). El hacinamiento como contexto: estrategias metodológicas para su análisis [Crowding as context: analytical and methodological strategies]. Psicothema 4, 551–569.

Google Scholar

Sarmento, H., Bradley, P., Anguera, M. T., Polido, T., Resende, R., and Campaniço, J. (2016). Quantifying the offensive sequences that result in goals in elite futsal matches. J. Sports Sci. 34, 621–629. doi: 10.1080/02640414.2015.1066024

PubMed Abstract | CrossRef Full Text | Google Scholar

Schulz, K. F., Altman, D. G., and Moher, D. (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. BMJ 340, 698–702. doi: 10.1136/bmj.c332

PubMed Abstract | CrossRef Full Text | Google Scholar

Sousa, D. J., Nunes-Prudente, J., Sequeira, P., López-López, J. A., and Hernández-Mendo, A. (2015). Analysis of 2vs. 2 situations in men's European handball championship 2012: application of polar coordinates technique. Cuadern. Psicol. Dep. 15, 181–193. Available online at: http://revistas.um.es/cpd/article/view/223391/173611

Google Scholar

Stolarova, M., Wolf, C., Rinker, T., and Brielmann, A. (2014). How to assess and compare inter-rater reliability, agreement and correlation of ratings: an exemplary analysis of mother–father and parent–teacher expressive vocabulary rating pairs. Front. Psychol. 5:509. doi: 10.3389/fpsyg.2014.00509

PubMed Abstract | CrossRef Full Text | Google Scholar

Stone, A. A., and Shiffman, S. (2002). Capturing momentary, self-report data: a proposal for reporting guidelines. Ann. Behav. Med. 24, 236–243. doi: 10.1207/S15324796ABM2403_09

PubMed Abstract | CrossRef Full Text | Google Scholar

Tarrag,ó, R., Iglesias, X., Michavila, J. J., Chaverri, D., Ruiz-Sanchis, L., and Anguera, M. T. (2015). Análisis de patrones en asaltos de espada de alto nivel [Patterns analysis in high-performance bouts epee]. Cuadern. Psicol. Dep. 15, 151–160. doi: 10.4321/S1578-84232015000100015

CrossRef Full Text | Google Scholar

Thomas, J. R., Silverman, S., and Nelson, J. (2015). Research Methods in Physical Activity. Champaign. IL: Human Kinetics.

Google Scholar

Tong, A., Sainsbury, P., and Craig, J. (2007). COnsolidated criteria for REporting Qualitative research (COREQ): a 32-item checklist for interviews and focus groups. J. Qual. Health Care 19, 349–357. doi: 10.1093/intqhc/mzm042

PubMed Abstract | CrossRef Full Text | Google Scholar

von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., and Vandenbroucke, J. P. (2007). The strengthening the reporting of observational studies in epidemiology (STROBE) statement. guidelines for reporting observational studies. Epidemiology 18, 800–804. doi: 10.1136/bmj.39335.541782.AD

PubMed Abstract | CrossRef Full Text | Google Scholar

Weigelt, M., and Memmert, D. (2012). Goal–side selection in soccer penalty kicking when viewing natural scenes. Front. Psychol. 3:312. doi: 10.3389/fpsyg.2012.00312

PubMed Abstract | CrossRef Full Text | Google Scholar

Williams, B. J., and Macdonald, D. (2015). Explaining outsourcing in health, sport and physical education. Sport Educ. Soc. 20, 57–72. doi: 10.1080/13573322.2014.914902

CrossRef Full Text | Google Scholar

Woodward, K. (2016). Embodied Sporting Practices: Regulating and Regulatory Bodies. London: Springer.

Google Scholar

Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., et al. (2000). Data collection instrument and procedure for systematic reviews in the guide to community preventive services. Am. J. Prev. Med. 18, 44–74. doi: 10.1016/S0749-3797(99)00122-1

PubMed Abstract | CrossRef Full Text | Google Scholar