AUTHOR=Mathur Maya B. , VanderWeele Tyler J. 

TITLE=New metrics for multiple testing with correlated outcomes

JOURNAL=Frontiers in Applied Mathematics and Statistics

VOLUME=9

YEAR=2023

URL=https://www.frontiersin.org/journals/applied-mathematics-and-statistics/articles/10.3389/fams.2023.1151314

DOI=10.3389/fams.2023.1151314

ISSN=2297-4687

ABSTRACT=<p>When investigators test multiple outcomes or fit different model specifications to the same dataset, as in multiverse analyses, the resulting test statistics may be correlated. We propose new multiple-testing metrics that compare the observed number of hypothesis test rejections (<inline-formula><mml:math id="M1" xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mover accent="true"><mml:mrow><mml:mi>θ</mml:mi></mml:mrow><mml:mo stretchy="false">^</mml:mo></mml:mover></mml:math></inline-formula>) at an unpenalized α-level to the distribution of rejections that would be expected if all tested null hypotheses held (the “global null”). Specifically, we propose reporting a “null interval” for the number of α-level rejections expected to occur in 95% of samples under the global null, the difference between <inline-formula><mml:math id="M2" xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mover accent="true"><mml:mrow><mml:mi>θ</mml:mi></mml:mrow><mml:mo stretchy="false">^</mml:mo></mml:mover></mml:math></inline-formula> and the upper limit of the null interval (the “excess hits”), and a one-sided joint test based on <inline-formula><mml:math id="M3" xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mover accent="true"><mml:mrow><mml:mi>θ</mml:mi></mml:mrow><mml:mo stretchy="false">^</mml:mo></mml:mover></mml:math></inline-formula> of the global null. For estimation, we describe resampling algorithms that asymptotically recover the sampling distribution under the global null. These methods accommodate arbitrarily correlated test statistics and do not require high-dimensional analyses, though they also accommodate such analyses. In a simulation study, we assess properties of the proposed metrics under varying correlation structures as well as their power for outcome-wide inference relative to existing methods for controlling familywise error rate. We recommend reporting our proposed metrics along with appropriate measures of effect size for all tests. We provide an <monospace>R</monospace> package, <monospace>NRejections</monospace>. Ultimately, existing procedures for multiple hypothesis testing typically penalize inference in each test, which is useful to temper interpretation of individual findings; yet on their own, these procedures do not fully characterize global evidence strength <italic>across</italic> the multiple tests. Our new metrics help remedy this limitation.</p>