Statistical Process Control Charts for Monitoring Next-Generation Sequencing and Bioinformatics Turnaround in Precision Medicine Initiatives

Purpose: Precision oncology, such as next generation sequencing (NGS) molecular analysis and bioinformatics are used to guide targeted therapies. The laboratory turnaround time (TAT) is a key performance indicator of laboratory performance. This study aims to formally apply statistical process control (SPC) methods such as CUSUM and EWMA to a precision medicine programme to analyze the learning curves of NGS and bioinformatics processes.

Patients and Methods: Trends in NGS and bioinformatics TAT were analyzed using simple regression models with TAT as the dependent variable and chronologically-ordered case number as the independent variable. The M-estimator “robust” regression and negative binomial regression were chosen to serve as sensitivity analyses to each other. Next, two popular statistical process control (SPC) approaches which are CUSUM and EWMA were utilized and the CUSUM log-likelihood ratio (LLR) charts were also generated. All statistical analyses were done in Stata version 16.0 (StataCorp), and nominal P < 0.05 was considered to be statistically significant.

Results: A total of 365 patients underwent successful molecular profiling. Both the robust linear model and negative binomial model showed statistically significant reductions in TAT with accumulating experience. The EWMA and CUSUM charts of overall TAT largely corresponded except that the EWMA chart consistently decreased while the CUSUM analyses indicated improvement only after a nadir at the 82^nd case. CUSUM analysis found that the bioinformatics team took a lower number of cases (54 cases) to overcome the learning curve compared to the NGS team (85 cases).

Conclusion: As NGS and bioinformatics lead precision oncology into the forefront of cancer management, characterizing the TAT of NGS and bioinformatics processes improves the timeliness of data output by potentially spotlighting problems early for rectification, thereby improving care delivery.

Introduction

Precision oncology refers to the use of therapeutics expected to benefit patients who harbor specific molecular or histopathological biomarkers (1). Due to the known complexity of cancers and the expanding body of knowledge of oncogenesis, molecular profiling, such as next generation sequencing (NGS) molecular analysis and bioinformatics, is being used to identify genetic mutations (2, 3), which may guide the deployment of targeted therapies (4, 5). As precision oncology remains a cornerstone in treating cancer patients and has been proven to improve outcomes (6, 7), its application is likely to be imperative in cancer research, clinical practice and patient care in the years to come. However, a critical element to this enterprise is the laboratory turnaround time (TAT), which refers to the time from sample receipt to the return of molecular results (8).

The Integrated Molecular Analysis of Cancers (IMAC) study was conducted to establish the prevalence and range of mutations in Asian patients with advanced cancers (9) to further develop and understand the value of targeted molecular profiling and gene sequencing in clinical decision making. TAT is a key performance indicator of laboratory performance (8) and a shorter TAT indicates that actionable information is being provided in a timely fashion (10), which expedites the decision-making process in management and treatment. Furthermore, a shorter TAT allows for the prudent use of resources in public health as patients can be discharged more quickly and require less hospital in-patient services (11). However, NGS labs are often hampered by various logistical and technical difficulties (12) and bioinformatics challenges (13).

However, the quantification of the learning curves for next generation sequencing labs has yet to been described in literature. As such, a first step in measuring the real-time efficiency and identifying bottlenecks in NGS and bioinformatics pipelines could be to adopt statistical process control (SPC) methods such as cumulative sum (CUSUM) (14), or exponentially weighted moving average (EWMA) (15), to monitor the temporal parameter stability and identify structural breakpoints in TAT. CUSUM is a sequential analysis technique which monitors change detection, particularly deviation from a performance standard (16–18), and has been widely used in a range of healthcare settings, from describing the learning curves of surgical or procedural skills (19–24), to clinical audits (25–27), and quality-assurance studies (22, 28–32). Similar to CUSUM, EWMA control charts have been widely used for measuring and monitoring healthcare outcomes (33, 34). EWMA analysis has been shown to detect smaller shift sizes that occur especially during the earlier period of monitoring (35).

To our knowledge, this study is the first to formally apply statistical process control (SPC) methods such as CUSUM and EWMA to a precision medicine programme to characterize the turnaround time for molecular report generation and analyze the learning curve of NGS and bioinformatics in the precision medicine era.

Methods

Patients

The characteristics and eligibility criteria of patients recruited to the Integrated Molecular Analysis of Cancers (IMAC) precision medicine platform has been described previously (9). Briefly, patients referred to the Developmental Therapeutics Unit (DTU) at the National University Cancer Institute, Singapore (NCIS) were offered participation in the IMAC programme if they were above 21 years of age, and had a histologically confirmed diagnosis of a solid malignancy or lymphoma, and had adequate tumor tissue for genome characterization. The IMAC protocol was approved by the National Healthcare Group-Domain Specific Review Board (NHG-DSRB) and the SingHealth Centralized Institutional Review Board and was undertaken in adherence with the Good Clinical Practice Guidelines. Written and informed consent was obtained from patients before study entry. Genomic profiles were characterized from a variety of pathological specimens (including tissue block or frozen specimens, slides, or cytological samples such as smears, and CT-guided or EUS guided, biopsy-imprint or FNA smear cytology, and DNA) retrieved during routine clinical procedures (e.g., surgery, biopsy, or ascites and pleural fluid drain).

Sequencing and Bioinformatics Analyses

As described previously (9), targeted next-generation sequencing was performed using the AmpliSeq Cancer Hotspot Panel v2 on an Ion Torrent/PGM system (Life Technologies, CA, USA) per the manufacturer’s instructions. We rejected samples with an average sequencing depth <500, unfiltered DNA variants >100, or uniformity of coverage <80% from bioinformatics analyses. The pathogenicity of exonic non-synonymous alterations were individually reported after reviewing published literature and public curated databases (i.e., ClinVar for pathogenicity; My Cancer Genome for variant frequency and exon number; Ensembl GRCh37 for the protein domain). When phenotypic information on rare variants were missing from ClinVar or Catalogue of Somatic Mutations in Cancer (COSMIC) databases, we interrogated Variant Call Format (VCF) files with Variant Effect Predictor (VEP) version 75 to infer their in silico pathogenicity (SIFT and POLYPHEN). As we did not perform paired normal tissue profiling, we excluded variants with minor allele frequency >5% among South East Asians in the 1000 Genomes Database. Since the evidence base for precision oncology biomarkers was in constant flux, we did not utilize any predetermined protocols to prioritize the actionability of reported variants.

Learning Curve Analyses

We began analyses of trends in TAT using simple regression models in which the dependent variable was the TAT and the independent variable was the chronologically-ordered case number. Two modelling approaches were chosen and serve as sensitivity analyses to each other: (i) M-estimator “robust” regression, in which a biweight loss function is minimized through an iteratively-weighted least squares algorithm, thus minimizing the impact of influential outliers; and (ii) negative binomial regression, since the turnaround time (in days) could constitute a form of ‘count data’ and therefore may be assumed to follow a Poisson-related family distribution.

Next, to illustrate the application of statistical process control (SPC) to the precision medicine paradigm, we exploited two popular SPC approaches: CUSUM and EWMA. We did not use risk-adjusted models as no predictors of TAT could be identified despite using multivariable linear, quantile, or negative binomial models, and evaluating a range of potential predictors (e.g., type of specimen [tissue block vs other types of specimens such as pap smears or fine needle aspiration], tumor content of specimen, etc.). The CUSUM statistic was calculated as the running sum of residuals (expected minus observed [O_j] outcome values) over chronologically-ordered patients, as per the following equation:

$$C_j={\displaystyle \sum _{k=1}^j(E_j-O_j)}$$

in which the expected value (Ej) refers to the mean TAT. On the basis of this equation, downward trends indicate that the observed values exceed the expected value (i.e., turnaround time for specimen greater than expected), whereas upward trends indicate an improvement in performance.

CUSUM log-likelihood ratio (LLR) charts were also generated, using clinically-selected thresholds of >6 weeks for overall TAT, and >3 weeks each for the wet-lab and dry-lab components of NGS respectively. To identify clinically-meaningful signals, we chose an odds ratios of 1.20 (e.g., a 20% increase in the odds of overall turnaround time > 42 days) and 1/1.20 = 0.83 (i.e. a 17% decrease in the odds of overall turnaround time > 42 days) as the alternative hypotheses. However, the CUSUM chart was not reset when the control limits were exceeded. The exponentially weighted moving average (EWMA) chart has been proposed to provide an ongoing local estimate of the average score that is purportedly easier for clinical staff to interpret and understand (19, 20). As such, we also illustrated the application of EWMA control chart for monitoring trends in TAT. The mean of the first ten observations were used to obtain the initial value for recursion, and the EWMA was calculated by assigning lesser weight to observations earlier in the series in a geometrically-declining fashion (decay factor: 0.1), with a smoothing parameter of alpha = 0.03. All statistical analyses were done in Stata version 16.0 (StataCorp), and nominal P < 0.05 were regarded to indicate statistical significance. All scripts were uploaded at the Github website (https://github.com/nicholassyn/Stata-codes-for-IMAC-CUSUM-and-EWMA-analysis/tree/main).

Results

Patient Information

A total of 365 patients with a variety of advanced malignancies underwent successful molecular profiling using the AmpliSeq Cancer Hotspot Panel v2 as part of the IMAC molecular screening initiative (9). The molecular landscape and histopathological characteristics of these patients have been reported previously (9).

Overall Learning Curve

Overall, there was a median of 27 days (IQR: 19-43) between the receipt of the sample to the generation of the molecular report. As indicated in Figure 1A, both the robust linear model (Robust β= -0.0346; 95% CI: -0.0479 to -0.0212, p<0.0001) and negative binomial model (IRR=0.9987; 95% CI: 0.9982 to 0.9992, p<0.0001) showed statistically significant reductions in TAT with accumulating experience. The EWMA chart of overall TAT (Figure 1B) showed that TAT consistently decreased from 0 to 365 cases, although interjected by a transient increase in TAT between 250 to 310 cases. CUSUM analyses of overall TAT indicated that from 0 to 82 cases, there was a worsening trend and the chart reached its nadir at 82. Subsequently, the CUSUM chart showed an improving trend from 82 to 365 cases with the exception of a deteriorating trend from 265 to 310 cases. The deteriorating phase corresponds largely with the findings in the EWMA chart which showed an increase in TAT from 250 to 300 cases (Figures 1C, D).

FIGURE 1

Figure 1 Overall Turnaround Time. (A) Scatter and regression fit of overall turnaround vs case load. (B) Exponential weighted moving average of overall turnaround time. (C) CUSUM of overall turnaround time. (D) CUSUM log-likelihood ratio chart for overall turnaround time > 3 weeks.

Learning Curve of NGS Assays (Wet-Lab Component)

There was a median of 13 days (IQR: 9-21) from the receipt of the sample to the NGS assay result. Interestingly, robust regression of NGS assay TAT versus case numbers did not reveal any statistically-significant temporal trend; in fact, negative binomial regression unexpectedly indicated a possible increase in TAT over time (Figure 2A). EWMA analyses also indicated that the moving average TAT estimate was higher at almost all timepoints as compared to the initial ~20 cases (Figure 2B). CUSUM analyses corroborated these findings, and demarcated three time periods: an initial increase in TAT up to 85 cases (nadir at 82 cases), followed by a decrease in TAT up to 265 cases, and then a subsequent increase in TAT up till the 365th case. Notably, the increase in NGS turnaround time at the 262nd case approximately corresponds with a period of transition and changes in key lab personnel (Figures 2C, D).

FIGURE 2

Figure 2 Turnaround Time for NGS Assays. (A) Scatter and regression fit of NGS turnaround vs case load. (B) Exponential weighted moving average of NGS turnaround time. (C) CUSUM of NGS turnaround time. (D) CUSUM log-likelihood ratio chart for nGS turaround time > 3 weeks.

Learning Curve of Bioinformatics Analyses (Dry-Lab Component)

There was a median of 10 days (IQR: 8-13) from NGS result to generation of the bioinformatics report. As demonstrated in Figure 3A, the TAT for bioinformatics analyses decreased significantly over time. This was supported by the EMWA analysis, which shows a rapid decrease in TAT up to the 110th case, followed by a more gradual pace of decrease subsequently (Figure 3B). The CUSUM analyses, however, revealed that between the 0th to 54th case, there was an initial increase in TAT, only after which did the turnaround time for bioinformatics analyses begin to improve (Figures 3C, D).

FIGURE 3

Figure 3 Turnaround Time for Bioinformatics Analyses. (A) Scatter and regression fit of bioinformatics turnaround vs case load. (B) Exponential weighted moving average of bioinformatics turnaround. (C) CUSUM of bioinformatics turnaround time. (D) CUSUM log-likelihood ratio chart for bioinformatics turnaround > 3 weeks.

Discussion

Learning curve analysis has been widely used in surgery (29, 36–41), due to its usefulness in presenting the performance of surgeons using various outcome measures relative to their cumulative case loads (18, 42–44). This enables the objective quantification of caseloads required for surgeons to surmount their learning curves and spotlights performance deviation that necessitates intervention (18, 36, 45). In oncology, precision medicine has been shown to achieve improved therapeutic outcomes and is becoming increasingly prevalent (46). NGS allows for sequencing of the whole genome to identify molecular aberrations instead of a singular biomarker, coupled with bioinformatics, targeted approaches could be identified to treat cancers and advance the effectiveness of personalized medicine in the years to come (47). However, the complex nature of NGS and bioinformatics are barriers to their ubiquitous use due to logistical and technical difficulties (9). In order to improve lab performances, the characterization of learning curve in NGS and bioinformatics could prove useful in facilitating greater adoption of precision oncology and accelerate the move towards comprehensive genomic profiling (48). To our knowledge, this is the first paper to characterize the learning curves of both NGS and bioinformatics, two key components in precision oncology.

The RA-CUSUM analysis found that the bioinformatics team took a lower number of cases (54 cases) to overcome the learning curve compared to the NGS team (85 cases) and this difference could be attributed to more experienced personnel and smoother workflows with bioinformatics analyses. The CUSUM showed an overall decrease in the TAT as caseloads increased for both NGS and bioinformatics (Figure 2C and Figure 3C). For bioinformatics, the improving trend also corresponded with the decrease in overall turnaround time in the EWMA chart for bioinformatics (Figure 3B), likely due to the increased efficiency of the bioinformatics team as they gained experience. However, for NGS, there was an overall increasing trend in TAT reflected by the EWMA (Figure 2B). This overall increase was attributable to the change of main pathologist operating the NGS equipment (262^nd case) and the number of cases analyzed subsequently did not reflect the full learning curve of the NGS team.

An important observation, of a second dip in performance (after 262 cases), was made in the NGS CUSUM chart (Figure 2C). This dip was also observed in the EWMA chart (Figure 2B), as well as the overall CUSUM and EWMA in Figure 1. In this period, two key personnel left the NGS laboratory and there was a handover of work to new manpower. The individual learning curves of transition to new personnel could explain this increase in TAT after 262 cases. The NGS laboratory comprised two to three pathologists involved in tissue quality control, and four to six in the genomics team involved in accessioning, sample preparation, NGS and report preparation. Meanwhile, the bioinformatics laboratory comprised four to six members. While there were changes in personnel in both laboratories, at any given point in time, there was a team focused on IMAC formed by two to three pathologists, two members in the genomics team, and one to two members in the bioinformatics team.

Interestingly, clustering of cases in both NGS (Figure 2A) and bioinformatics laboratories (Figure 3A) were observed. This habitual clustering of consecutive cases suggests that researchers tended to accumulate samples even after receiving them, before analyzing them as a single batch. Batch processing is a common practice to reduce cost and wastage or reagents, but will consequently impact on TAT if the required number of samples per batch does not materialize within a specified time-frame. Batching samples can also lead to batch-to-batch irregularities in results due to different laboratory conditions, sample degradation and personnel changes (49). Some of these inaccuracies can possibly be reduced by strict standardization of sequencing protocols and training of lab staff.

Within healthcare contexts, the application of control charts has largely been limited to procedural, particularly surgical disciplines (36, 50). The IDEAL Framework for surgical innovation describes five stages of evolution for new surgical therapeutic interventions - Idea, Development, Exploration, Assessment, and Long-term Study (51). This idea can be applied to the field of precision oncology to better the process of finding gene sequences that cause cancers, interpreting these results and finding targeted therapies for these patients. CUSUM and EWMA scrutinizes the NGS and bioinformatics process and assists in mapping the TAT and finding areas of improvement in the evolving process of precision oncology. While the findings of this paper are not necessarily generalizable, it highlights the potential use of statistical process monitoring for precision medicine. As more centres use targeted therapies, analyses should be routinely performed to monitor learning curves and find potential areas of improvement in the process. In doing so, it is important to keep in mind the regional differences which have the potential to influence precision medicine including disparities in the clinical trial landscape and infrastructure, and the sociocultural attitudes towards genetic testing.

Limitations

This study is limited by the retrospective nature of the data and the fact that the NGS was done in an academic rather than commercial laboratory where changes in a small number of lab personnel and the need to batch cases together can impact on the learning curve of the team and lead to an increase in TAT.

Conclusions

As NGS and bioinformatics lead precision oncology into the forefront of cancer management, it is vital that the implementation and management of NGS and bioinformatics processes are monitored closely to improve the timeliness of data output and care delivery. By characterizing the TAT using SPC methods, including CUSUM and EWMA, we were able to objectively evaluate how differences in workflows and changes in manpower impacts the performance of precision oncology laboratories. As precision oncology is increasingly adopted, applying SPC prospectively can potentially help spotlight problems early for rectification in NGS and bioinformatics processes.

Data Availability Statement

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found below: https://www.ncbi.nlm.nih.gov/, https://pubmed.ncbi.nlm.nih.gov/28994108//ijc.31091.

Author Contributions

Concept and design: NLS, RS, and DT. Acquisition, analysis, or interpretation of data: NLS, SJ, WS, CNg, YC, WL, VH, NK, MG, XL, NSS, XK, ZI, JL, DL, BP, AW, RAS, WY, CC, S-CL, B-CG, RS, and DT. Drafting of the manuscript: SJ, WS, CNg, CH, WL, NLS, NK, MG, and DT. Critical revision of the manuscript for important intellectual content: SJ, WS, CNg, YC, WL, NLS, NK, MG, DT and RS. Statistical analysis: NLS and DT. Study supervision: DT, B-CG, and RS. All authors contributed to the article and approved the submitted version.

Funding

This investigator-initiated study was funded by the Singapore Ministry of Health’s National Medical Research Council (NMRC) under the Transitional Award Scheme to DT, and its Centre Grant scheme to the National University Cancer Institute, Singapore (NCIS), and the NCIS Yong Siew Yoon (YSY) Cancer Drug Development fellowship to VH. The research is also supported by the National Research Foundation Singapore and the Singapore Ministry of Education under their Research Centres of Excellence (RCE) Initiative. The NMRC had no involvement in the design and conduct of the study; collection, management, analysis and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Conflict of Interest

DT consults on the advisory board for Astra Zeneca and has received research funding from Karyopharm Therapeutics.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Yates LR, Seoane J, Le Tourneau C, Siu LL, Marais R, Michiels S, et al. The European Society for Medical Oncology (ESMO) Precision Medicine Glossary. Ann Oncol (2018) 29(1):30–5. doi: 10.1093/annonc/mdx707

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Schwartzberg L, Kim ES, Liu D, Schrag D. Precision Oncology: Who, How, What, When, and When Not? Am Soc Clin Oncol Educ Book (2017) 37):160–9. doi: 10.1200/EDBK_174176

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Kruglyak KM, Lin E, Ong FS. Next-Generation Sequencing in Precision Oncology: Challenges and Opportunities. Expert Rev Mol Diagn (2014) 14(6):635–7. doi: 10.1586/14737159.2014.916213

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Bode AM, Dong Z. Recent Advances in Precision Oncology Research. NPJ Precis Oncol (2018) 2:11–. doi: 10.1038/s41698-018-0055-0

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Dancey JE, Bedard PL, Onetto N, Hudson TJ. The Genetic Basis for Cancer Treatment Decisions. Cell (2012) 148(3):409–20. doi: 10.1016/j.cell.2012.01.014

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Warner JL. Giving Up on Precision Oncology? Not So Fast! Clin Transl Sci (2017) 10(3):128–9. doi: 10.1111/cts.12457

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Haslem DS, Chakravarty I, Fulde G, Gilbert H, Tudor BP, Lin K, et al. Precision Oncology in Advanced Cancer Patients Improves Overall Survival With Lower Weekly Healthcare Costs. Oncotarget (2018) 9(15):12316–22. doi: 10.18632/oncotarget.24384

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Hawkins RC. Laboratory Turnaround Time. Clin Biochem Rev (2007) 28(4):179–94.

PubMed Abstract | Google Scholar

9. Heong V, Syn NL, Lee XW, Sapari NS, Koh XQ, Adam Isa ZF, et al. Value of a Molecular Screening Program to Support Clinical Trial Enrollment in Asian Cancer Patients: The Integrated Molecular Analysis of Cancer (IMAC) Study. Int J Cancer (2018) 142(9):1890–900. doi: 10.1002/ijc.31091

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Masamba LPL, Mtonga PE, Kalilani Phiri L, Bychkovsky BL. Cancer Pathology Turnaround Time at Queen Elizabeth Central Hospital, the Largest Referral Center in Malawi for Oncology Patients. J Glob Oncol (2017) 3(6):734–9. doi: 10.1200/JGO.2015.000257

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Goswami B, Singh B, Chawla R, Gupta VK, Mallika V. Turn Around Time (TAT) as a Benchmark of Laboratory Performance. Indian J Clin Biochem (2010) 25(4):376–9. doi: 10.1007/s12291-010-0056-4

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Ulahannan D, Kovac MB, Mulholland PJ, Cazier JB, Tomlinson I. Technical and Implementation Issues in Using Next-Generation Sequencing of Cancers in Clinical Practice. Br J Cancer (2013) 109(4):827–35. doi: 10.1038/bjc.2013.416

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Singer J, Irmisch A, Ruscheweyh H-J, Singer F, Toussaint NC, Levesque MP, et al. Bioinformatics for Precision Oncology. Briefings Bioinf (2019) 20(3):778–88. doi: 10.1093/bib/bbx143

CrossRef Full Text | Google Scholar

14. Crosier RB. A New Two-Sided Cumulative Sum Quality Control Scheme. Technometrics (1986) 28(3):187–94. doi: 10.1080/00401706.1986.10488126

CrossRef Full Text | Google Scholar

15. Hunter JS. The Exponentially Weighted Moving Average. J Qual Technol (1986) 18(4):203–10. doi: 10.1080/00224065.1986.11979014

CrossRef Full Text | Google Scholar

16. Grigg OA, Farewell VT, Spiegelhalter DJ. Use of Risk-Adjusted CUSUM and RSPRT Charts for Monitoring in Medical Contexts. Stat Methods Med Res (2003) 12(2):147–70. doi: 10.1177/096228020301200205

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Holzhey DM, Jacobs S, Walther T, Mochalski M, Mohr FW, Falk V. Cumulative Sum Failure Analysis for Eight Surgeons Performing Minimally Invasive Direct Coronary Artery Bypass. J Thorac Cardiovasc Surg (2007) 134(3):663–9. doi: 10.1016/j.jtcvs.2007.03.029

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Forbes TL, DeRose G, Kribs SW, Harris KA. Cumulative Sum Failure Analysis of the Learning Curve With Endovascular Abdominal Aortic Aneurysm Repair. J Vasc Surg (2004) 39(1):102–8. doi: 10.1016/S0741-5214(03)00922-4

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Waller HM, Connor SJ. Cumulative Sum (Cusum) Analysis Provides an Objective Measure of Competency During Training in Endoscopic Retrograde Cholangio-Pancreatography (ERCP). HPB (2009) 11(7):565–9. doi: 10.1111/j.1477-2574.2009.00091.x

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Kim CW, Kim WR, Kim HY, Kang J, Hur H, Min BS, et al. Learning Curve for Single-Incision Laparoscopic Anterior Resection for Sigmoid Colon Cancer. J Am Coll Surgeons (2015) 221(2):397–403. doi: 10.1016/j.jamcollsurg.2015.02.016

CrossRef Full Text | Google Scholar

21. Je S, Cho Y, Choi HJ, Kang B, Lim T, Kang H. An Application of the Learning Curve-Cumulative Summation Test to Evaluate Training for Endotracheal Intubation in Emergency Medicine. Emerg Med J (2015) 32(4):291–4. doi: 10.1136/emermed-2013-202470

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Sood A, Ghani KR, Ahlawat R, Modi P, Abaza R, Jeong W, et al. Application of the Statistical Process Control Method for Prospective Patient Safety Monitoring During the Learning Phase: Robotic Kidney Transplantation With Regional Hypothermia (IDEAL Phase 2a-B). Eur Urol (2014) 66(2):371–8. doi: 10.1016/j.eururo.2014.02.055

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Tekkis PP, Senagore AJ, Delaney CP, Fazio VW. Evaluation of the Learning Curve in Laparoscopic Colorectal Surgery: Comparison of Right-Sided and Left-Sided Resections. Ann Surg (2005) 242(1):83–91. doi: 10.1097/01.sla.0000167857.14690.68

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Vigano L, Laurent A, Tayar C, Tomatis M, Ponti A, Cherqui D. The Learning Curve in Laparoscopic Liver Resection: Improved Feasibility and Reproducibility. Ann Surg (2009) 250(5):772–82. doi: 10.1097/SLA.0b013e3181bd93b2

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Calsina L, Clará A, Vidal-Barraquer F. The Use of the CUSUM Chart Method for Surveillance of Learning Effects and Quality of Care in Endovascular Procedures. Eur J Vasc Endovascular Surg (2011) 41(5):679–84. doi: 10.1016/j.ejvs.2011.01.003

CrossRef Full Text | Google Scholar

26. Cockings JG, Cook DA, Iqbal RK. Process Monitoring in Intensive Care With the Use of Cumulative Expected Minus Observed Mortality and Risk-Adjusted P Charts. Crit Care (2006) 10(1):R28. doi: 10.1186/cc3996

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Lim TO, Soraya A, Ding LM, Morad Z. Assessing Doctors’ Competence: Application of CUSUM Technique in Monitoring Doctors’ Performance. Int J Qual Health Care (2002) 14(3):251–8. doi: 10.1093/oxfordjournals.intqhc.a002616

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Novick RJ, Fox SA, Stitt LW, Forbes TL, Steiner S. Direct Comparison of Risk-Adjusted and Non-Risk-Adjusted CUSUM Analyses of Coronary Artery Bypass Surgery Outcomes. J Thorac Cardiovasc Surg (2006) 132(2):386–91. doi: 10.1016/j.jtcvs.2006.02.053

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Novick RJ, Fox SA, Stitt LW, Kiaii BB, Swinamer SA, Rayman R, et al. Assessing the Learning Curve in Off-Pump Coronary Artery Surgery via CUSUM Failure Analysis. Ann Thorac Surg (2002) 73(1):S358–62. doi: 10.1016/S0003-4975(01)03399-9

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Noyez L. Control Charts, Cusum Techniques and Funnel Plots. A Review of Methods for Monitoring Performance in Healthcare. Interact Cardiovasc Thorac Surg (2009) 9(3):494–9. doi: 10.1510/icvts.2009.204768

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Klompmaker S, van der Vliet WJ, Thoolen SJ, Ore AS, Verkoulen K, Solis-Velasco M, et al. Procedure-Specific Training for Robot-Assisted Distal Pancreatectomy. Ann Surg (2019) 274(1):e18–27. doi: 10.1097/SLA.0000000000003291

CrossRef Full Text | Google Scholar

32. Hasegawa Y, Nitta H, Takahara T, Katagiri H, Baba S, Takeda D, et al. Safely Extending the Indications of Laparoscopic Liver Resection: When Should We Start Laparoscopic Major Hepatectomy? Surg Endosc (2017) 31(1):309–16. doi: 10.1007/s00464-016-4973-z

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Smith IR, Gardner MA, Garlick B, Brighouse RD, Cameron J, Lavercombe PS, et al. Performance Monitoring in Cardiac Surgery: Application of Statistical Process Control to a Single-Site Database. Heart Lung Circ (2013) 22(8):634–41. doi: 10.1016/j.hlc.2013.01.011

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Gomes IC, Mingoti SA, Oliveira CDL. A Novel Experience in the Use of Control Charts for the Detection of Nosocomial Infection Outbreaks. Clinics (2011) 66:1681–9. doi: 10.1590/S1807-59322011001000004

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Han SW, Tsui K-L, Ariyajunya B, Kim SB. A Comparison of CUSUM, EWMA, and Temporal Scan Statistics for Detection of Increases in Poisson Rates. Qual Reliability Eng Int (2010) 26(3):279–89. doi: 10.1002/qre.1056

CrossRef Full Text | Google Scholar

36. Valsamis EM, Chouari T, Dowd-Booth C, Rogers B, Ricketts D. Learning Curves in Surgery: Variables, Analysis and Applications. Postgraduate Med J (2018) 94(1115):525. doi: 10.1136/postgradmedj-2018-135880

CrossRef Full Text | Google Scholar

37. Liao HJ, Dong C, Kong FJ, Zhang ZP, Huang P, Chang S. The CUSUM Analysis of the Learning Curve for Endoscopic Thyroidectomy by the Breast Approach. Surg Innov (2014) 21(2):221–8. doi: 10.1177%2F1553350613500722

PubMed Abstract | Google Scholar

38. Bokhari MB, Patel CB, Ramos-Valadez DI, Ragupathi M, Haas EM. Learning Curve for Robotic-Assisted Laparoscopic Colorectal Surgery. Surg Endosc (2011) 25(3):855–60. doi: 10.1007/s00464-010-1281-x

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Liu Z, Zhang X, Zhang W, Zhang Y, Chen W, Qin W, et al. Comprehensive Evaluation of the Learning Curve for Peroral Endoscopic Myotomy. Clin Gastroenterol Hepatol (2018) 16(9):1420–6.e2. doi: 10.1016/j.cgh.2017.11.048

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Chua D, Syn N, Koh YX, Goh BKP. Learning Curves in Minimally Invasive Hepatectomy: Systematic Review and Meta-Regression Analysis. Br J Surg (2021) 108(4):351–8. doi: 10.1093/bjs/znaa118

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Chan KS, Wang ZK, Syn N, Goh BKP. Learning Curve of Laparoscopic and Robotic Pancreas Resections: A Systematic Review. Surgery (2021) 170(1):194–206. doi: 10.1016/j.surg.2020.11.046

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Goh BKP, Prieto M, Syn N, Koh Y-X, Lim K-I. Critical Appraisal of the Learning Curve of Minimally Invasive Hepatectomy: Experience With the First 200 Cases of a Southeast Asian Early Adopter. ANZ J Surg (2020) 90(6):1092–8. doi: 10.1111/ans.15683

PubMed Abstract | CrossRef Full Text | Google Scholar

43. van der Poel MJ, Besselink MG, Cipriani F, Armstrong T, Takhar AS, van Dieren S, et al. Outcome and Learning Curve in 159 Consecutive Patients Undergoing Total Laparoscopic Hemihepatectomy. JAMA Surg (2016) 151(10):923–8. doi: 10.1001/jamasurg.2016.1655

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Halls MC, Alseidi A, Berardi G, Cipriani F, van der Poel M, Davila D, et al. A Comparison of the Learning Curves of Laparoscopic Liver Surgeons in Differing Stages of the IDEAL Paradigm of Surgical Innovation: Standing on the Shoulders of Pioneers. Ann Surg (2019) 269(2):221–8. doi: 10.1097/SLA.0000000000002996

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Duclos A, Carty MJ, Peix JL, Colin C, Lipsitz SR, Kraimps JL, et al. Development of a Charting Method to Monitor the Individual Performance of Surgeons at the Beginning of Their Career. PloS One (2012) 7(7):e41944. doi: 10.1371/journal.pone.0041944

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Schwaederle M, Zhao M, Lee JJ, Lazar V, Leyland-Jones B, Schilsky RL, et al. Association of Biomarker-Based Treatment Strategies With Response Rates and Progression-Free Survival in Refractory Malignant Neoplasms: A Meta-Analysis. JAMA Oncol (2016) 2(11):1452–9. doi: 10.1001/jamaoncol.2016.2129

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Ewalt MD, West H, Aisner DL. Next Generation Sequencing—Testing Multiple Genetic Markers at Once. JAMA Oncol (2019) 5(7):1076–. doi: 10.1001/jamaoncol.2019.0453

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Malone ER, Oliva M, Sabatini PJB, Stockley TL, Siu LL. Molecular Profiling for Precision Cancer Therapies. Genome Med (2020) 12(1):8. doi: 10.1186/s13073-019-0703-1

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Leek JT, Scharpf RB, Bravo HC, Simcha D, Langmead B, Johnson WE, et al. Tackling the Widespread and Critical Impact of Batch Effects in High-Throughput Data. Nat Rev Genet (2010) 11(10):733–9. doi: 10.1038/nrg2825

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Salowi MA, Choong YF, Goh PP, Ismail M, Lim TO. CUSUM: A Dynamic Tool for Monitoring Competency in Cataract Surgery Performance. Br J Ophthalmol (2010) 94(4):445–9. doi: 10.1136/bjo.2009.163063

PubMed Abstract | CrossRef Full Text | Google Scholar

51. McCulloch P, Altman DG, Campbell WB, Flum DR, Glasziou P, Marshall JC, et al. No Surgical Innovation Without Evaluation: The IDEAL Recommendations. Lancet (2009) 374(9695):1105–12. doi: 10.1016/S0140-6736(09)61116-8

PubMed Abstract | CrossRef Full Text | Google Scholar