How Sensitive is Sensitivity Analysis?: Evaluation of Pharmacoeconomic Submissions in Korea

Purpose: We aimed to describe the types of uncertainties examined in the economic evaluations submitted for reimbursement in Korea and their impact on the incremental cost-effectiveness ratio (ICER).

Method: Fifty dossiers were submitted by pharmaceutical companies to the economic subcommittee of the Pharmaceutical Benefit Coverage Advisory Committee (PBCAC) from January 2014 to December 2018. The types of uncertainties were categorized as structural and parametric, and the frequencies of the sensitivity analysis per variables were analyzed. The impact of uncertainties was measured by the percent variance of the ICER relative to that of the base case analysis.

Results: Of the 50 submissions, varying discount rate (44 submissions), followed by time horizon (38 submissions) and model assumptions (29 submissions), were most frequently used to examine structural uncertainty, while utility (42 submissions), resource use (41 submissions), and relative effectiveness (26 submissions) were used to examine parametric uncertainty. A total of 1,236 scenarios (a scenario corresponds to a case where a single variable is varied by a single range) were presented in the one-way sensitivity analyses, where parametric and structural sensitivity analyses comprised 679 and 557 scenarios, respectively. Varying drug prices had the highest impact on ICER (median variance 19.9%), followed by discount rate (12.2%), model assumptions (11.9%), extrapolation (11.8%), and time horizon (10.0%).

Conclusions: Variables related to long-term assumptions, such as model assumptions, time horizon, extrapolation, and discounting rate, were related to a high level of uncertainty. Caution should be exercised when using immature data.

Introduction

Model-based analysis synthesizes clinical, economical, and epidemiological evidence from various sources and extrapolates the expected value over the long term (Briggs et al., 2012). Since the evidence directly related to the research question is frequently missing, or multiple sources are available with conflicting results, model-based analysis almost always suffers from various forms of uncertainties (Bilcke et al., 2011).

Uncertainties can be classified as stochastic uncertainty (first-order), parametric uncertainty (second-order), structural uncertainty, and heterogeneity (Briggs et al., 2012). Stochastic uncertainty implies random variability and is intrinsically unavoidable, whereas parametric uncertainties imply the uncertainties in parameter estimation and can be examined via deterministic sensitivity analysis (DSA), where parameter values are varied based on defensible ranges of values to examine the robustness of the results, or via probabilistic sensitivity analysis (PSA), where parameter values are sampled from a predefined distribution and vary simultaneously (Doubilet et al., 1985). Structural uncertainty, which is inherent in the assumptions of the decision model, is a difficult type of uncertainty to define and can be examined by scenario analysis. Heterogeneity deals with the variability of patients of interest and is usually explored through a subgroup analysis. Interestingly, recommendations on how to tackle structural uncertainties or heterogeneity are largely vague in many international pharmacoeconomic guidelines, including Korea, while several provide specific details on parametric uncertainties (Ghabri et al., 2018).

Structural uncertainties are associated with a wide variation in the incremental cost-effectiveness ratio (ICER) (Le, 2016), and given that clinical trials usually last shorter than expected in the economic evaluation, extrapolating beyond the time horizon is frequently required, which introduces additional uncertainties (Kearns et al., 2020). When policymakers set priorities among competing demands based on model-based analysis, uncertainties related to the point estimates, how those uncertainties were examined or reported, and variance of the ICERs are deeply considered (Jackson et al., 2011; Bae et al., 2013), few studies have described how sensitivity analyses in the dossiers submitted for the reimbursement decision are handled, including parametric and structural uncertainties, much less the variance of ICER relative to the base case.

Many HTA organizations review DSA as well as PSA due to the advantage of being able to transparently check the effect of uncertainty of each variable on the results (Australian Government: Department of Health, 2016; The Canadian Agency for Drugs, 2017; The National Institute for Health and Care Excellence (NICE), 2013), Yet DSA, especially one-way SA, has a problem of underestimating the overall uncertainty because it examines only the effects of variations that one variable can have while other variables are fixed (Claxton, 2008). In addition, there is a limitation that the nonlinearity of the model is not reflected, and if there is a correlation between variables, it cannot be considered appropriately (Briggs et al., 2012; McCabe et al., 2020; Vreman et al., 2021). Moreover, despite many guidelines stipulated that that “clinically and statistically feasible ranges” are recommended, the ranges of the DSA are often chosen arbitrarily (Vreman et al., 2021).

This study examines how uncertainty is explored in economic evaluations submitted by pharmaceutical companies in Korea, which is a necessary part of the reimbursement decision-making process. Specifically, we examine how variables related to long-term effects are analyzed. Additionally, the impact of uncertain variables on ICER was explored through variation in the ICER.

Methods

Economic evaluation dossiers submitted by the pharmaceutical industry to the economic subcommittee of the Pharmaceutical Benefit Coverage Advisory Committee from January 2014 to December 2018 were evaluated by two independent reviewers (SB and EB).

Uncertainties were categorized as DSA and PSA, and DSA was further categorized into structural and parametric uncertainties. Stochastic uncertainty, which is intrinsically unavoidable, and heterogeneity, which is known variability, are not included in the analysis. The number of sensitivity analyses per submission is used to identify the frequently tested parameters or structural assumptions. The parametric uncertainties considered in our study are drug prices, resource use (unit cost or resource utilization other than drug), utility weights, relative effectiveness of the intervention (including odds ratio, relative risk, or hazard ratio), baseline risk (natural history of the disease not related to specific treatment), and others (parameters relevant to specific treatment, such as incidence of the adverse event). The plausible range of values used in the sensitivity analysis is categorized as a 95% confidence interval (CI) of a specific parameter (statistically obtained from clinical studies), arbitrarily selected values (± 20%), or sourced from other studies.

The structure of a model varies by disease type, yet we refer to variables that are universally applicable and can be clearly defined, such as time horizon, discount rate [variation from the recommended 5% (Bae et al., 2013)], extrapolation method used (i.e., Weibull vs. lognormal), model assumptions [i.e., treatment duration, duration of the effectiveness, selection of comparator(s)], and patient characteristics (i.e., age, disease severity, weight, or race).

The frequency of the sensitivity analysis in this study is estimated on a submission or scenario basis. When presenting the proportion of submissions with sensitivity analysis for each category of variables, it is analyzed per submission, and the frequency of sensitivity analysis for each variable is analyzed for each scenario. To examine the ranges of the values used and their ICERs relative to the base case, we count scenarios; a single scenario for the sensitivity analysis corresponds to a case where a single parameter is varied by a single plausible range. A paired case (i.e., ± 20%) is also defined as a single scenario.

The variance of the ICER related to the sensitivity analyses is measured in percentage,

$\frac{\left|{ICER}_{sensitivityanalysis}{-ICER}_{basecase}\right|}{{ICER}_{basecase}}\text{×}100\text{,}$

where paired (±95% CI) values are estimated as follows:

$\frac{{\text{ICER}}_{\text{max}}{-\text{ICER}}_{\text{min}}}{{2\text{×ICER}}_{\text{base}\text{case}}}\text{×}100\text{.}$

Results

Of the 50 dossiers submitted to the economic subcommittee, 26 (52%) fall under antineoplastic and immunomodulating agents, and 24 (48%) are injection formulations and were submitted evenly across the observation period (Table 1). 46 submissions (92%) employed cost-utility analysis, and DSA was conducted in 49, all of which conducted one-way sensitivity analysis (Table 2), and only two of them conducted multivariate (2-way) sensitivity analysis (data not shown). Regarding structural uncertainties, the discount rate was the most frequently examined (44 submissions), followed by the time horizon (38 submissions), and model assumptions (29 submissions) (Table 2). For parametric uncertainty, utility was most frequently used (42 submissions), followed by resource use (41 submissions), and relative effectiveness (26 submissions) (Table 2). PSA was conducted in 18 submissions and cost was most frequently examined (17 submissions), followed by utility (16 submissions) and relative effectiveness (10 submissions).

TABLE 1

TABLE 1. Basic characteristics of the 50 dossiers submitted to the Economic Sub-Committee for listing at the Korean National Health Insurance.

TABLE 2

TABLE 2. Types of sensitivity analysis of the 50 dossiers examined.

A total of 1,236 scenarios were presented in the one-way sensitivity analyses, where structural and parametric sensitivity analyses comprised 557 and 679 scenarios, respectively (Table 3). The ranges of parametric uncertainties were arbitrarily selected in 48% of them (326 scenarios), followed by alternative sources (256 scenarios, 38%) and 95% CI (97 scenarios, 14%). The 95% CI was more likely to be employed in the relative effectiveness (36%), yet arbitrary values were frequently used in resource use (78%) and drug price (70%).

TABLE 3

TABLE 3. The ranges of values used in the parametric sensitivity analysis of the 50 dossiers submitted to the Economic Sub-Committee for listing at the Korean National Health Insurance.

Regarding structural uncertainties, the discounting rate (202 scenarios) was most frequently examined, followed by time horizon (130 scenarios), model assumptions (i.e., treatment duration, duration of the effectiveness, comparator, adjusting for cross-over design; 94 scenarios), extrapolation (95 scenarios), and patient characteristics (age, disease severity, weight, or race; 36 scenarios).

The relative variance of ICER for each variable is presented in Figure 1 as box plots, where the distributions of each variable are skewed. In general, structural uncertainties showed wider interquartile ranges, compared with parametric uncertainties. The median of the relative variances in terms of percentage indicates that drug price has the highest impact (19.9%), followed by discount rate (12.2%), model assumptions (11.9%), extrapolation (11.8%), and time horizon (10.0%), suggesting that the most frequently examined variables do not always have the highest level of uncertainty. As shown in Figure 1, the median value of the percentage change is within 10% for most variables, excluding drug price, discount rate, model assumption, and extrapolation.

FIGURE 1

FIGURE 1. Boxplot comparing the variance of ICER (Incremental cost effectiveness ratio) for each scenario with reference to that of the base case. The “Other” implies parameters relevant with specific treatment, such as the incidence of the adverse event, or hospitalization rate.

Discussion

This study examined how uncertainty was explored in economic evaluations submitted for coverage decision-making in South Korea. The second version of the PE guidelines required that DSA be performed on all uncertain variables and encouraged submitters to conduct PSA for parametric uncertainty (Bae et al., 2013). When analyzing the submissions, 49 out of 50 cases, including a case of CMA, presented the results of DSA, and 18 of them additionally presented PSA.

Although most cases involved DSA, the assessment of uncertainty was somewhat limited. Relative effectiveness is one of the most critical parameters in cost-effectiveness analysis, yet only 52% of submissions conducted sensitivity analysis. Even though most of the submitted cases (48 out of 50) were analyzed using the model, only 58% of them performed sensitivity analysis on the model assumptions, and 4 cases did not perform DSA for utility among 46 cases that performed a cost-utility analysis (data not shown). In addition, 8 cases did not submit the DSA for the extrapolated model, even though survival analysis was performed.

The selection of the structural aspect of the model is an important decision that determines the model’s predictability (Afzali and Karnon, 2015). According to previous studies, the impact of structural uncertainty is similar to that of parametric uncertainty (Kim and Thompson, 2010; Frederix et al., 2014). However, it is known to be insufficiently reviewed compared to parametric uncertainty (Afzali and Karnon, 2015). Ghabri et al. (2016) reported that, as a result of reviewing manufacturers’ submissions to the French National Authority for Health (HAS), structural uncertainty was less frequently explored than methodological or parametric uncertainty, consistent with our assumptions (Ghabri et al., 2016). According to the analysis results of this study, however, there is no basis for concluding that structural uncertainty is more overlooked than parametric uncertainty, even though the term “structural uncertainty” is more widely defined in this study as including both methodological and structural uncertainty.

As shown in Figure 1, which shows the impact of each variable on the ICER, the median value of the percentage change is within 10%, excluding drug price, discount rate, model assumption, and extrapolation, which is smaller than the variances estimated in previous studies (Frederix et al., 2014; Kearns et al., 2020). Frederix et al. (2014) found that the ICER varied by 2–3 times depending on the difference in the structural aspects of the model and its parameterization (Frederix et al., 2014). In Kearns et al. (2000), it was confirmed that the ICER changed by 46.2% when different extrapolation methods were used (Kearns et al., 2020). Among the applications reviewed in this study, however, the median percentage change of ICER was 11.8%, and the upper quartile was only 24.9% in the cases where the sensitivity analysis was performed for the extrapolation method. Even considering that Kearns' study used a hypothetical dataset, it is questionable whether pharmaceutical companies have performed sensitivity analysis over a sufficient range.

A clear criterion such as 95% CI is used for only 14% of the sensitivity analyses, most of which are for relative efficacy. Arbitrary values or values cited from other studies are used in most cases. Even when published sources were cited, it is not easy to assess whether DSA was performed within a plausible range unless these sources were searched systematically. According to Ghabri et al. (2016), 43% of the submissions to HAS also lacked justification for the plausible range surrounding the point estimate of the parameter (Ghabri et al., 2016), which is similar to what we have observed in our analysis (48%).

Generally, high uncertainty has a negative impact on the reimbursement recommendation. Although our data do not provide any information about the association between the uncertainty and reimbursement decision, the authors’ experience of participating in the economic subcommittee of PBCAC suggested that when the uncertainty has a significant impact on the results, additional data is requested or negative appraisals are made. In this case, pharmaceutical companies are likely to be tempted to report with reduced uncertainty.

Therefore, when performing or reviewing sensitivity analysis, it is necessary to check the plausibility of the range used for sensitivity analysis. It is most desirable to determine the range through a systematic approach such as 95% CI. When such information is not available, systematically reviewing the existing literature is generally recommended to obtain a plausible range (Australian Government: Department of Health, 2016; The Canadian Agency for Drugs and Technologies in Health, 2017; The National Institute for Health and Care Excellence (NICE), 2013). If there is no proper prior research, it is necessary to seek expert opinions in a systematic way and set the range based on this.

From Figure 1, it is apparent that the influence of the variables related to the long-term effect is relatively large, except for the drug price, which pharmaceutical companies can strategically select. The discount rate, time horizon, and extrapolation are all in this case. In estimating the long-term effect based on short-term observations, the results vary greatly depending on the model assumptions, particularly the assumptions about the effect after the observation period. Accordingly, each country’s guidelines focus on the uncertainty that long-term extrapolation may have. Korea also emphasizes this point, as it revised the guidelines in 2021.

Similarly, 57% of submissions to the French HAS had the problem of unfounded extrapolation beyond the clinical trial (Ghabri et al., 2016). Masucci et al. (2017) also reported that the time horizon (56%) and model structure (36%) were frequently discussed by the economic reviewers of the pan-Canadian Oncology Drug Review (Masucci et al., 2017).

Recently, as drugs claiming long-term effects such as immune therapy and advanced therapy medicinal products have appeared, it is becoming more critical to evaluate the uncertainty in estimating long-term effects (Jönsson et al., 2019; Huygens et al., 2021) Due to insufficient patients or ethical reasons, new drugs used for rare severe diseases are often authorized based on a single-arm study rather than a randomized controlled trial. Additionally, the evidence for long-term effects is often uncertain because survival data are immature, along with other reasons. However, due to social pressure for early access, approvals or reimbursement decisions for these drugs are often made with very high uncertainty about clinical benefits (Grimm et al., 2020; Huygens et al., 2021). According to Kim and Prasad (2015), who followed up on the survival improvement of drugs approved based on the surrogate endpoint at the time of FDA approval (median follow-up 4.4 years), only 5 out of 36 cases demonstrated survival gain (Kim and Prasad, 2015). However, few efforts have been made to assess the validity of survival predictions compared to actual data (Latimner, 2013; Vickers, 2019).

In previous studies, several methods for exploring and managing uncertainty regarding long-term effects have been proposed, such as developing more specific guidance on exploring uncertainty surrounding extrapolation, requiring to follow up data after entry, using mature external data, or combining observed survival data with expert opinion in estimating long-term survival (Frederix et al., 2014; Cope et al., 2019; Huygens et al., 2021).

This study has several limitations. By reviewing the sensitivity analysis included in the first report submitted by the pharmaceutical company, we analyzed which variables were subjected to sensitivity analysis and their impact on ICER. However, no qualitative evaluation was performed to determine whether the range of values subjected to sensitivity analysis for each variable was appropriate. Moreover, the values identified in this study are those included in the first report and may differ from those used in the committee’s final deliberation. PSA was not reviewed in detail in this study because it was not a mandatory requirement for the study period, and the intention of this study is to confirm which variables were reviewed for structural and parametric uncertainties and the impact of each uncertain variable.

This study is the first attempt to explore the uncertainty in economic evaluations submitted for reimbursement decision-making in South Korea. Several studies explored the impact of uncertainty in economic evaluations, but only a few examined actual documents submitted to the HTA agencies. Given the growing importance of uncertainty, by reviewing how pharmaceutical companies are handling uncertainty in submissions to relevant authorities, we can find implications for what points should be emphasized to better address the uncertainty in cost-effectiveness. In particular, the fact that the ICER variation in this study was smaller than what was reported in the previous studies suggests that more prescribed guidance is necessary. Further study is necessary to assess the real impact of uncertainty in terms of the difference between what was predicted at the time of listing and how they actually performed in the follow-up studies.

Conclusion

Most dossiers submitted to the committee for reimbursement decisions presented DSA results as suggested in the guidelines. However, considering the variance of ICER, in terms of the impact of each uncertainty, variability was not significant in most scenarios, which raises doubts as to whether the uncertainty evaluation was carried out within a sufficiently plausible range for each variable. Specific guidance regarding the ranges of the sensitivity analysis is necessary. Long-term benefits are often modeled based on uncertain short-term clinical data; therefore, the evaluation and management of uncertainties become more critical than before.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Author Contributions

SB and EB contributed to the study design and prepared the first draft of the manuscript. SB and EB extracted and analyzed data. JL visualized data and reviewed literature. All the authors reviewed and commented on the manuscript at all stages.

Funding

This study was financially supported by the Health Insurance Review and Assessment service, Korea.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

This study was carried out as part of Pharmacoeconomic guideline revision project, which was commissioned by Health Insurance Review and Assessment Service (HIRA). The authors would like to thank project team members who provided valuable comments on the related issues, and HIRA staff, who supported the team in collecting the relevant data. The views expressed in this paper are those of the authors and do not necessarily represent those of HIRA. We are also grateful to two anonymous referees for their valuable comments.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fphar.2022.884769/full#supplementary-material

References

Afzali, H. H., and Karnon, J. (2015). Exploring Structural Uncertainty in Model-Based Economic Evaluations. Pharmacoeconomics 33 (5), 435–443. doi:10.1007/s40273-015-0256-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Australian Government: Department of Health, (2016). Guidelines for Preparing a Submission to the Pharmaceutical Benefits Advisory Committee, Canberra, Australia: Australian Government.

Google Scholar

Bae, S., Lee, S., Bae, E. Y., and Jang, S. (2013). Korean Guidelines for Pharmacoeconomic Evaluation (Second and Updated Version): Consensus and Compromise. PharmacoEconomics 31 (4), 257–267. doi:10.1007/s40273-012-0021-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Bilcke, J., Beutels, P., Brisson, M., and Jit, M. (2011). Accounting for Methodological, Structural, and Parameter Uncertainty in Decision-Analytic Models: A Practical Guide. Med. Decis. Mak. 31 (4), 675–692. doi:10.1177/0272989X11409240

CrossRef Full Text | Google Scholar

Briggs, A. H., Weinstein, M. C., Fenwick, E. A., Karnon, J., Sculpher, M. J., and Paltiel, A. D. (2012). Model Parameter Estimation and Uncertainty Analysis: a Report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6. Med. Decis. Mak. 32 (5), 722–732. doi:10.1177/0272989X12458348

PubMed Abstract | CrossRef Full Text | Google Scholar

Claxton, K. (2008). Exploring Uncertainty in Cost-Effectiveness Analysis. PharmacoEconomics 26 (9), 781–798. doi:10.2165/00019053-200826090-00008

PubMed Abstract | CrossRef Full Text | Google Scholar

Cope, S., Ayers, D., Zhang, J., Batt, K., and Jansen, J. P. (2019). Integrating Expert Opinion with Clinical Trial Data to Extrapolate Long-Term Survival: a Case Study of CAR-T Therapy for Children and Young Adults with Relapsed or Refractory Acute Lymphoblastic Leukemia. BMC Med. Res. Methodol. 19 (1), 182. doi:10.1186/s12874-019-0823-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Doubilet, P., Begg, C. B., Weinstein, M. C., Braun, P., and McNeil, B. J. (1985). Probabilistic Sensitivity Analysis Using Monte Carlo Simulation. A Practical Approach. Med. Decis. Mak. 5 (2), 157–177. doi:10.1177/0272989X8500500205

PubMed Abstract | CrossRef Full Text | Google Scholar

Frederix, G. W., van Hasselt, J. G., Schellens, J. H., Hövels, A. M., Raaijmakers, J. A., Huitema, A. D., et al. (2014). The Impact of Structural Uncertainty on Cost-Effectiveness Models for Adjuvant Endocrine Breast Cancer Treatments: the Need for Disease-specific Model Standardization and Improved Guidance. PharmacoEconomics 32 (1), 47–61. doi:10.1007/s40273-013-0106-x

PubMed Abstract | CrossRef Full Text | Google Scholar

Ghabri, S., Cleemput, I., and Josselin, J. M. (2018). Towards a New Framework for Addressing Structural Uncertainty in Health Technology Assessment Guidelines. PharmacoEconomics 36 (2), 127–130. doi:10.1007/s40273-017-0603-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Ghabri, S., Hamers, F. F., and Josselin, J. M. (2016). Exploring Uncertainty in Economic Evaluations of Drugs and Medical Devices: Lessons from the First Review of Manufacturers' Submissions to the French National Authority for Health. PharmacoEconomics 34 (6), 617–624. doi:10.1007/s40273-016-0381-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Grimm, S. E., Pouwels, X., Ramaekers, B. L. T., Wijnen, B., Knies, S., Grutters, J., et al. (2020). Development and Validation of the TRansparent Uncertainty ASsessmenT (TRUST) Tool for Assessing Uncertainties in Health Economic Decision Models. PharmacoEconomics 38 (2), 205–216. doi:10.1007/s40273-019-00855-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Huygens, S. A., Versteegh, M. M., Vegter, S., Schouten, L. J., and Kanters, T. A. (2021). Methodological Challenges in the Economic Evaluation of a Gene Therapy for RPE65-Mediated Inherited Retinal Disease: The Value of Vision. PharmacoEconomics 39 (4), 383–397. doi:10.1007/s40273-021-01003-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Jackson, C. H., Bojke, L., Thompson, S. G., Claxton, K., and Sharples, L. D. (2011). A Framework for Addressing Structural Uncertainty in Decision Models. Med. Decis. Mak. 31 (4), 662–674. doi:10.1177/0272989X11406986

CrossRef Full Text | Google Scholar

Jönsson, B., Hampson, G., Michaels, J., Towse, A., von der Schulenburg, J. G., and Wong, O. (2019). Advanced Therapy Medicinal Products and Health Technology Assessment Principles and Practices for Value-Based and Sustainable Healthcare. Eur. J. Health Econ. 20 (3), 427–438.

PubMed Abstract | Google Scholar

Kearns, B., Stevens, J., Ren, S., and Brennan, A. (2020). How Uncertain Is the Survival Extrapolation? A Study of the Impact of Different Parametric Survival Models on Extrapolated Uncertainty about Hazard Functions, Lifetime Mean Survival and Cost Effectiveness. PharmacoEconomics 38 (2), 193–204. doi:10.1007/s40273-019-00853-x

PubMed Abstract | CrossRef Full Text | Google Scholar

Kim, C., and Prasad, V. (2015). Cancer Drugs Approved on the Basis of a Surrogate End Point and Subsequent Overall Survival: An Analysis of 5 Years of US Food and Drug Administration Approvals. JAMA Intern Med. 175 (12), 1992–1994. doi:10.1001/jamainternmed.2015.5868

PubMed Abstract | CrossRef Full Text | Google Scholar

Kim, L. G., and Thompson, S. G. (2010). Uncertainty and Validation of Health Economic Decision Models. Health Econ. 19 (1), 43–55. doi:10.1002/hec.1444

PubMed Abstract | CrossRef Full Text | Google Scholar

Latimner, N. (2013). NICE DSU Technical Support Document 14: Undertaking Survival Analysis for Economic Evaluations Alongside Clinical Trials—Extrapolation with Patient-Level Data. London, UK: National Institute for Health and Care Excellence.

Google Scholar

Le, Q. A. (2016). Structural Uncertainty of Markov Models for Advanced Breast Cancer: A Simulation Study of Lapatinib. Med. Decis. Mak. 36 (5), 629–640. doi:10.1177/0272989X15622643

PubMed Abstract | CrossRef Full Text | Google Scholar

Masucci, L., Beca, J., Sabharwal, M., and Hoch, J. S. (2017). Methodological Issues in Economic Evaluations Submitted to the Pan-Canadian Oncology Drug Review (pCODR). Pharmacoecon Open 1 (4), 255–263. doi:10.1007/s41669-017-0018-3

PubMed Abstract | CrossRef Full Text | Google Scholar

McCabe, C., Paulden, M., Awotwe, I., Sutton, A., and Hall, P. (2020). One-Way Sensitivity Analysis for Probabilistic Cost-Effectiveness Analysis: Conditional Expected Incremental Net Benefit. PharmacoEconomics 38 (2), 135–141. doi:10.1007/s40273-019-00869-3

PubMed Abstract | CrossRef Full Text | Google Scholar

The Canadian Agency for Drugs and Technologies in Health (CADTH) (2017). Guidelines for the Economic Evaluation of Health Technologies (Ottawa, Canada: The Canadian Agency for Drugs and Technologies in Health).

Google Scholar

The National Institute for Health and Care Excellence (NICE) (2013). Guide to the Methods of Technology Appraisal. London, UK: National Institute for Health and Care Excellence.

Google Scholar

Vickers, A. (2019). An Evaluation of Survival Curve Extrapolation Techniques Using Long-Term Observational Cancer Data. Med. Decis. Mak. 39 (8), 926–938. doi:10.1177/0272989X19875950

PubMed Abstract | CrossRef Full Text | Google Scholar

Vreman, R. A., Geenen, J. W., Knies, S., Mantel-Teeuwisse, A. K., Leufkens, H. G. M., and Goettsch, W. G. (2021). The Application and Implications of Novel Deterministic Sensitivity Analysis Methods. PharmacoEconomics 39 (1), 1–17. doi:10.1007/s40273-020-00979-3

PubMed Abstract | CrossRef Full Text | Google Scholar