Commentary: Effect evaluation of Sahtak bi Sahnak, a Lebanese secondary school-based nutrition intervention: a cluster randomised trial

A Commentary on
Effect evaluation of Sahtak bi Sahnak, a Lebanese secondary school-based nutrition intervention: a cluster randomised trial

by Said, L., Gubbels, J. S., and Kremers, S. P. J. (2022). Front. Nutr. 9:824020. doi: 10.3389/fnut.2022.824020

Said et al. (1) evaluated the effectiveness of an educational school-based nutrition intervention called Sahtak bi Sahnak (your health on your plate) on dietary knowledge and adherence to dietary guidelines among secondary school adolescents in Lebanon. The authors concluded that the intervention was effective. However, upon closer inspection, we identified issues with the authors' statistical analysis and description of the cluster randomized controlled trial (cRCT) process. Our main concern is that the authors ignored the clustered nature of their randomization for part of their statistical analysis, including the third model (effect on unhealthy items adherence score) in Table 2 and the entire results in Table 3 of the original paper. The authors stated, “Where the random intercept was non-significant in the multilevel analysis, a multivariate regression model was performed to test the intervention effect, adjusting for the same significant covariates.” However, this approach lacks statistical justification because a non-significant random intercept does not imply that clustering and nesting effects can be disregarded in cRCTs.

When a study aims to evaluate the effect of an intervention in which the treated unit was a group rather than an individual, the design and analysis must address the issue of clustering and nesting effects (2, 3). In Said et al. (1) study, the intervention unit was the school (a total of 16 schools, 8 assigned to the treatment group and 8 schools assigned to the control group); therefore, the statistical analysis should not be conducted at the single level such as student without accounting for clustering and nesting effects. By “clustering effects,” we mean accounting for the potential and usually positive correlation among students within the same school, which can be adjusted for by using a random effect for clusters. The authors should have also accounted for the nesting effect by adjusting the degrees of freedom or by using different approximations. Ignoring these positive correlations within clusters or the limited degrees of freedom may inflate the type I error rate (2).

The authors accounted for such clustering effects in only two places (the first and second models in Table 2) by applying multilevel hierarchical estimates on the dietary knowledge score and healthy items adherence score. But they ignored the clustering for all other estimates (the third model in Table 2 and all the estimates in Table 3). Nevertheless, in these two models, the authors considered only two-level (adolescents and school), whereas their description suggests potentially more clustering levels, such as location and grades. A single-level analysis that does not account for such clustering and nesting effects for these multilevel observations produces results which, by being based on unsound analyses, are inherently unsubstantiated (2).

We also noticed that the authors did not clearly describe the process of randomization of the 16 schools while categorizing them into private vs. public, rural vs. urban, and grade 10 vs. grade 11. For the analysis to be validated, readers need to understand whether these locations and grades were clusters and whether they were appropriately randomized (4). Given the limited description in the paper, in its current form, we are unable to differentiate between whether the intervention design was random or was stratified for convenience. In a cRCT, every clustering unit should be randomized. Moreover, Said et al. (1) did not take clustering and nesting into consideration when they conducted sampling power calculations. Owing to the various levels (e.g., school types, locations, grades), the sample size and power calculations must consider the number of clusters per condition, their sizes, and their intracorrelation (5). It is also important that we highlight that Said et al. applied inappropriate chi-square and t-test in Table 1 to check the baseline differences between control and treated students. Usually, it is not advisable to test for baseline differences between study arms (some journals forbid it), as stated in the Consolidated Standards of Reporting Trials Guidelines, because it only tests the hypothesis of whether the randomization had been performed correctly. However, in this case, there is enough doubt about the randomization that such tests seem wise. Therefore, we eschewed the inappropriate use of the chi-square and t-test in Table 1 and instead used linear mixed models for continuous variables and generalized linear mixed models for multinomial outcomes adjusting the degrees of freedom for number of schools and found no significant differences in any variable of Table 1.

TABLE 1

Table 1. Revised results and reproduced results from Table 2 of the original paper.

We thank Said et al. (1) for sharing their raw data and offering feedback to uphold the rigor of scientific research which have allowed us to reanalyze their work and present revised results in this letter (Tables 1, 2). While our reanalysis using their methods as explained in the paper generated some of their results that are close to original estimates in terms of both size of coefficients and statistical significance level, we have also failed to closely reproduce results in other cases. It is important to note that our analysis did not overturn Said et al. (1) conclusions or result in changes of formal statistical significance, and therefore no change in the paper's conclusion about the effects of the Sahtak bi Sahnak intervention Although the results were not qualitatively different, it is essential that we present the revised analysis using appropriate methods. Moreover, the confidence intervals were wider and the coefficient estimates differed between original and revised estimates, which has important implications particularly for research syntheses like meta-analyses. Our reanalysis assumed that there were only two levels—school and adolescents—in the intervention design. If there are more than two levels, which could not be confirmed because of the unclear description of the randomization process in the paper, we can neither support nor refute whether the Sahtak bi Sahnak intervention was effective in increasing dietary knowledge and adherence to dietary guidelines. Despite the qualitatively similar results, we present the corrected results because it is an ethical and professional scientific responsibility to correct any reported errors in published papers (6, 7). The statistical code used for reproducing the results and for analyzing the results with accounting for clustering and nesting (acknowledging that we could only account for the clustering and nesting that were explicitly described in the article) are available at the following link: https://doi.org/10.5281/zenodo.8030967.

TABLE 2

Table 2. Revised results and reproduced results from Table 3 of the original paper.

Author contributions

AS: conceptualization, data analysis, formal analysis, and writing—original draft preparation. AB, LG-A, and DA: conceptualization, review and editing, and approval of the final version. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by NIH grants R25DK099080, R25HL124208, R25GM099080, P20GM109096, and the Gordon and Betty Moore Foundation.

Conflict of interest

DA and his institution (Indiana University) have received payments for consultation, grants, contracts, in-kind donations, and contributions from multiple for-profit and not-for-profit entities interested in statistical design and analysis of experiments, but not directly related to the research questions addressed in the paper.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Author disclaimer

The opinions expressed are those of the authors and do not necessarily represent those of the NIH or any other organization.

References

1. Said L, Gubbels JS, Kremers SP. Effect evaluation of Sahtak bi Sahnak, a Lebanese secondary school-based nutrition intervention: a cluster randomised trial. Front Nutr. (2022) 428:824020. doi: 10.3389/fnut.2022.824020

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Murray DM, Murray C, of the D of EDM. Design and Analysis of Group-randomized Trials. Oxford: Oxford University Press (1998). 481 p.

Google Scholar

3. Parsons NR, Teare MD, Sitch AJ. Unit of analysis issues in laboratory-based research. eLife. (2018) 7:e32486. doi: 10.7554/eLife.32486

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: Updated guidelines for reporting parallel group randomised trials. Int J Surg. (2012) 10:28–55. doi: 10.1016/j.ijsu.2011.10.001

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Campbell MK, Piaggio G, Elbourne DR, Altman DG. Consort 2010 statement: extension to cluster randomised trials. BMJ. (2012) 345:e5661. doi: 10.1136/bmj.e5661

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Teixeira da Silva JA. An error is an error… is an erratum: The ethics of not correcting errors in the science literature. Publ Res Q. (2016) 32:220–6. doi: 10.1007/s12109-016-9469-0

CrossRef Full Text | Google Scholar

7. Barbour V, Kleinert S, Wager E, Yentis S. Guidelines for retracting articles. Committee on Publication Ethics (2009). Available online at: https://publicationethics.org/node/19896 (accessed June 30, 2022).

Google Scholar