Solving the Evidence Interpretability Crisis in Health Technology Assessment: A Role for Mechanistic Models?

Health technology assessment (HTA) aims to be a systematic, transparent, unbiased synthesis of clinical efficacy, safety, and value of medical products (MPs) to help policymakers, payers, clinicians, and industry to make informed decisions. The evidence available for HTA has gaps—impeding timely prediction of the individual long-term effect in real clinical practice. Also, appraisal of an MP needs cross-stakeholder communication and engagement. Both aspects may benefit from extended use of modeling and simulation. Modeling is used in HTA for data-synthesis and health-economic projections. In parallel, regulatory consideration of model informed drug development (MIDD) has brought attention to mechanistic modeling techniques that could in fact be relevant for HTA. The ability to extrapolate and generate personalized predictions renders the mechanistic MIDD approaches suitable to support translation between clinical trial data into real-world evidence. In this perspective, we therefore discuss concrete examples of how mechanistic models could address HTA-related questions. We shed light on different stakeholder's contributions and needs in the appraisal phase and suggest how mechanistic modeling strategies and reporting can contribute to this effort. There are still barriers dissecting the HTA space and the clinical development space with regard to modeling: lack of an adapted model validation framework for decision-making process, inconsistent and unclear support by stakeholders, limited generalizable use cases, and absence of appropriate incentives. To address this challenge, we suggest to intensify the collaboration between competent authorities, drug developers and modelers with the aim to implement mechanistic models central in the evidence generation, synthesis, and appraisal of HTA so that the totality of mechanistic and clinical evidence can be leveraged by all relevant stakeholders.

Introduction

Health technology assessment (HTA) is a systematic and multidisciplinary process that summarizes medical evidence, social and economic impact, and ethical issues related to the use of health technology. HTA addresses both the direct and intended effects of this technology, as well as its indirect and unintended consequences—with the goal of informing decision making. A major feature of the collective output of a HTA process is the reimbursement by the health insurance system of the medical product (MP).

In general, two levels of decision-making regarding health care should be informed by HTA: (1) for the community—is the MP worth giving to the population, and could it be more or less beneficial for a group in the population? (2) for an individual: will a particular patient benefit from the MP, and if yes to what extent?

HTA seeks to couple the available evidence on the MP and the disease with the decision-making process itself, and thus has similarities to evidence-based health care and evidence-based policymaking (1). By evidence, one should understand a comprehensive record of knowledge and data collected in clinical trials (of which randomized, placebo-controlled studies are the gold standard), observational studies and from various sources relating to patient health status and/or the routine delivery of health care (often referred to as “real-world data,” RWD). One could say that HTA interprets clinical data from a real-world perspective by considering the realistic epidemiology of the disease and the full range of standard of care options (available to the population of interest). For a given MP (we focus on new drugs in this Perspective), the first round of assessment occurs during the review of the market authorization (MA) application by the regulators, e.g., FDA (Food and Drug Administration) or EMA (European Medicines Agency) for safety and efficacy. Given that the evidence included in these applications is generated throughout several years of development, key stakeholders could and should synergize and could streamline evidence generation and assessment from the beginning (2). Non-RCT data such as observational study data or RWD might bear relevant and additional information about safety, effectiveness, and cost effectiveness of MPs at potentially a larger scale. However, issues with identification, access, quality, representativeness, and heterogeneity of such data are limiting their practical applicability in HTA (3, 4).

The COVID-19 pandemic has disrupted global healthcare systems and created significant challenges for the HTA and payer communities (5). The COVID-19 pandemic has clearly shown where evidence generation, synthesis, assessment, and decision making are limited: (a) the typical bench-to-bedside timeframes of several years are simply unacceptable in a pandemic context; (b) clinical trial evidence collected in “emergency mode” suffers from increased uncertainty regarding the expected treatment effect, outcomes and costs (6); (c) the diversity of national policies and their frequent changes make it hard to come to conclusions on ethical and societal issues and raise barriers for patients to fully capture and understand the impact of a new MP on their life.

Especially in light of the COVID-19 pandemic, the importance of the assessment for the individual cannot be underestimated. While some patients do not suffer from any symptoms, others do not survive, or are affected on long timescales. Clinical data on COVID-19 prophylaxis and treatment currently under-represents the individual course of the disease due to the diversity and time dependency of the interactions between the virus and the patients' bodies. Here, the inherent limitation of HTA—being centered around population-based approaches—is aggravated. Issues related to better guiding economic evaluation of personalized medicine interventions—e.g., how study questions are developed, how populations are characterized, how comparators are defined, how effectiveness is evaluated, how outcomes are valued and how resources are measured (7)—need urgently to be addressed for the assessment of MPs related to COVID-19.

The COVID-19 pandemic has also raised the bar for communication around HTA. There has been divergence of opinion among international HTA agencies on how to deal with evidence for early COVID-19 treatments (8). This divergence and lack of transparency about the reasoning behind the assessments during this unsettling period have triggered public unease and skepticism with HTA as a whole.

As a response to the urgency to address these challenges, we wish to advocate using mechanistic models to bridge clinical MP development and HTA thanks to their capability for evidence generation, synthesis, and stakeholder communication alike.

Challenges for Health Technology Assessment

A key issue for HTA of a new MP is the number of limitations regarding the representativeness and validity of the evidence that is available. For conclusions useful for patients and public health, more quantitative knowledge and valid answers to questions need to be found (Table 1).

TABLE 1

Table 1. Examples of how published (mechanistic) models rooted in the clinical development space (model informed drug development, MIDD) could address uncertainties in new medicinal product assessment reports.

Randomized Clinical Trials (RCTs) Deliver a Binary Answer to a Binary Question

A first reason for the limited use of data generated during development lies in the results provided by randomized clinical trials (RCT) which are the gold standard for clinical evidence. An RCT is an instrument built to determine if the new treatment is effective or not by statistical testing. The frequentist inference paradigm (26) is still today's standard method in RCT despite the advent of innovative trial designs and analysis techniques [i.e., Bayesian (27)] but can limit drastically the interpretation of the efficacy tested in the trial (26, 28). In addition, the fact that statistical models are not designed to look for causality—but only to identify correlations available in the data—prevents a quantitative appraisal of the MP efficacy tailored to patient profile (20).

RCT Data Reflects Benefit of the Population and Not of the Individual

A second limitation in HTA is the fact that currently population (and sometimes stratified) medicine is pursued during clinical development while for HTA, the benefit for individual patient (groups) becomes important. RCTs, done either separately for different strata for the population or analyzed for different subgroups of one larger study population are currently the only tool available to “individualize” a MP efficacy estimate. As it is the central focus of an RCT to robustly estimate the average effect in a given population, cannot be obtained easily and hence, detailed information at patient level and the mean estimated effect is “applied equally” to each patient. Frequently, patients enrolled during clinical development are not entirely representative of the future target population because of the way they are selected to enter the trials. And they are furthermore limited in number and diversity. Reliably quantifying the effect for individuals from this evidence is therefore limited as well.

The advent of personalized medicine puts the “mean efficacy” approach in question (7) and calls for a paradigm shift of how efficacy should be considered for market authorization (MA)—and market access.

High Quality Data Exceeding the Scope of Market Authorization Is Scarce

For sponsors, there are increased barriers to conducting randomized trials after registration. Availability of a treatment with proven efficacy may pose ethical problems for placebo-controlled trials. Additional information about the effect of a treatment often needs to rely on observational studies and RWD (for example registers, patient records). The fact that RWD contains routinely collected information and low accessibility but high heterogeneity of data (29, 30) does not easily reveal the detailed and true epidemiological status of a disease or the effect of an intervention in the population. Even with the additional use of RWD, it remains difficult to derive an overview of the long-term and real-life impact in the clinical practice necessary for the HTA exercise.

In summary, gold-standard evidence for HTA (RCTs) can be regarded as more qualitative than quantitative, it has a domain of validity restricted to the context tested in clinical trials during clinical development and does not answer a number of important questions (see Table 1). It is not always possible to collect enough high-quality observational data and RWD to fill the gaps. In view of these challenges, and even more so when there is a strong, urgent, unmet therapeutic need (as today—facing the COVID-19 pandemic), HTA agencies are faced with a difficult dilemma: They can assess and position themselves on the basis of uncertain evidence (risk of misjudgement) or wait for more solid evidence (risk of delaying the access to a potentially effective product for patients with progressive disease or in treatment failure). This situation advocates to make better use of the “totality of evidence” generated during development.

Advent of the Mechanistic Approach in Model Informed Drug Development

Model-informed drug development (MIDD) applies drug exposure-based, (systems) biological and statistical models derived from preclinical and clinical data sources to inform drug development and decision-making (31). It integrates information from diverse data sources to decrease uncertainty and lower failure rates, and to develop information that cannot or would not be generated experimentally. The most widespread fields of application within MIDD are pharmacokinetics and pharmacodynamics and dose-response relationship modeling for dosing-regimen explorations as well as trial simulation for design optimization.

Within MIDD and regulatory decision making, a new set of models is emerging (32, 33). These models are based on knowledge with theoretical rules describing known mechanisms (called mechanistic models¹). Within the family of mechanistic models physiologically based pharmacokinetic modeling (PBPK) adopts a mechanistic approach to describe what the body does to the drug and quantitative systems pharmacology (QSP) models aspire to capture what the drug does to the marker, organ, or clinical outcome. As opposed to data driven models, mechanistic ones describe known or hypothesized mechanisms at a smaller scale so that the higher scale behavior emerges (34). In most mechanistic models the equations describe functional relationships between molecules, cells, or organs. The choice of the used equations and their parameters is informed through systematically reviewing and curating the available biomedical knowledge about the process of interest, and in turn, each component of the model (variable state, parameter, and equation) can be unequivocally justified by a corresponding piece of knowledge in the literature (or other considered source of knowledge)¹. The equations often come in the form of systems of ordinary differential equations (ODEs) that can describe coupled dynamics of the entities in the biological system of interest (but also other approaches such as partial differential equation systems or agent-based models exist). The covered composition of biological entities and scale of the description such as molecules, cells, organs, or the whole organism can vary depending on the context (35–37) and thereby define the specific scope and limitations of the model. Annotation and metadata for this knowledge can comprise additional information, for example a collaboratively curated or consensus strength of evidence and ontologies. These features can provide biological plausibility to those models by design and thus be used to rationalize, explain, and translate representative or individual clinical findings based on the (often large) body of mechanistic knowledge used in the model. Where parameters cannot informed by knowledge and remain unknown, heterogenous (in vitro, preclinical, omics, clinical) data can be used for (algorithmic) calibration (38).

The adoption and use of mechanistic models in model informed drug development and especially in regulatory decision making requires to establish their credibility through verification, validation and uncertainty quantification for which existing guidelines need to be adopted by modelers and more specific guidance issued by regulators (34).

In response to the COVID-19 pandemic, mechanistic models have been put forward to guide antiviral drug repurposing (39) and vaccine development (40), showing that such models can synthesize and translate the body of biological knowledge into a clinically relevant setting in a short time frame.

Mechanistic models are associated with a Virtual Population (VPop) to introduce interpatient variability. A VPop is a set of virtual patients, each one being characterized by its own set of descriptors (model parameters values) that follow pre-defined joint distributions (41–43). Simulations can be conducted in varying scenarios (such as different treatment regimens) according to a simulation protocol that defines the entire in silico clinical trial. These in silico trials produce digital evidence to explain, complement or partially replace in vivo clinical trials for drug development (44, 45). Running mechanistic model based in silico trials with a theoretically infinite number of patients can support evidence in rare settings and place population-level results in relation to individual simulated patients. The mechanistic and individual nature of the underlying model further allows one to allocate “clones” of the same patients in different arms and simulation scenarios corresponding to idealized clinical trial settings. In this way, effectiveness can be rationalized through tracing it to impacting and confounding factors.

Mechanistic models thus can bring biological plausibility, equity of clinical and mechanistic evidence as well as individual predictions (similar to idealized RCT settings) to the table of evidence synthesis and generation.

Modeling in Health Technology Assessment

Modeling in HTA is conducted during (1) the evidence synthesis phase and (2) economic impact assessment, mostly through data-driven modeling approaches. Mechanistic models are still underrepresented in this field but coming of age.

For evidence synthesis, different data-driven modeling approaches are commonly used. Pairwise and network meta-analyses (NMA) (46) using fixed effect and random effects models are tools to synthesize evidence from randomized controlled trials. NMA allows for comparisons that have not been directly obtained in head-to-head trials but comes with methodological challenges. NMA relies on the assumption that the analyzed studies are similar in all factors affecting the relative effects, which can lead to biased results. Moreover, these types of models are often limited in their data source scope. To address this issue, a technique combining NMA with quantitative modeling of effect modifiers (e.g., doses) has become available—utilizing the “totality of evidence” (47). Such “model-based” NMA can mimic randomization and allows estimation and predictions for multiple agents and a range of doses, using plausible physiological dose-response models (48). Additional to data from RCTs, data from observational studies is increasingly used in the evidence synthesis, which, however, lacks an unbiased control arm and techniques for reducing biases need to be applied (49).

For extrapolating a clinical effect into longer-term economic impact there exists quite a variety of methods, which are used for HTA and can be classified as cost-benefit analysis, cost-effectiveness analysis, and cost-utility analysis (50, 51). Simple graph-based decision trees, Markov models [suited for diseases that involve an ongoing risk (52)] or more involved discrete event simulation (DES) (53) and agent-based models (54) are frequently used for data analysis, classification and interpolation and extrapolation in time. The data fed into these models, however, is incomplete due to the limited evidence generated in clinical development (see open questions in Table 1).

Mechanistic models can bridge the gap between development and HTA. Given that validation can establish the credibility of a model for regulatory decision making, exploration of a much larger number of situations than in RCTs (with different patient subgroups, treatment compliance or comparators for instance) might be feasible. Such digital evidence supporting RCT data alleviates several difficulties such as power, representativeness, costs to run the trials, and ethical issues. For the consideration of such evidence in HTA one should consider the following unique benefits of mechanistic model that statistical ones cannot provide. First, mechanistic models possess biological plausibility by design—using biological, chemical, and physical processes as “blueprint”—and are therefore well suited for extrapolations. Second, the VPop can be set up to assess the very same patient under various conditions and scenarios (such as treatment arms) which corresponds to an idealized crossover design and allows to assess clinical benefit for every individual.

A concrete list of examples of how mechanistic models can address unanswered questions left in the MA dossier is given in Table 1. In summary an individual estimate of the (real and long-term) benefit-risk ratio using mechanistic models and adequate metrics (21, 22) feed a precise estimate of the costs of treatment for better health economic projections.

Modeling for Stakeholder Engagement

HTA is a multi-stakeholder activity that should shed light on more facets of an MP than just a technical analysis. Especially in the appraisal phase, “complex calculations, arbitrary assumptions, debatable choices of whose perspectives to pursue, difficult-to-understand methods, research designs and underlying philosophy/concepts, and time-consuming processes” are at risk of narrowing the HTA findings (55). It has therefore become clear that a diverse set of views need to be captured, consulted, and considered. At the same time, different stakeholders have unique needs that must be addressed before these stakeholders can position themselves. It is to note that recently the importance of engaging patients and patient groups in HTA has been emphasized (56) and there are examples of such engagement in several countries. Nevertheless, systematic involvement from beginning to end of the HTA process [not only during the appraisal stage as currently often the practice (57)] is still an ongoing effort (58). Apart from the need to include different stakeholder groups, there is no consensus what role each stakeholder group should assume in overall decision-making process ranging from information, consultation, participation in the debate, co-decision, as sole decision maker (59). Despite this ongoing debate on the exact role, better mutual understanding, communication, and engagement are sought, all centered around the available evidence. Modeling and simulation and especially mechanistic models may be used as a tool for stakeholder engagement apart from their capability to create (digital) evidence and synthesize data. There is an example from the literature underlining that participation can be achieved by applying an adapted conceptual framework for the modeling and simulation process [see for example (60)]. For this reason, we attempt a mapping of the differences between roles and contributions of stakeholders with specific needs and a suggested use of mechanistic models in Table 2.

TABLE 2

Table 2. List of different Stakeholder groups increasingly involved in the appraisal stage of HTA with dedicated contribution, special needs (to understand and capture a drug's mechanism, effect, role, or impact) and example of how mechanistic modeling can help to address this need and fill persistent gaps.

There are still barriers dissecting modeling in the HTA space and modeling in the clinical development space. These barriers are conceptually similar to the known barriers to bring HTA to policy making (65). Specific barriers delaying the use of mechanistic models in HTA are (i) the lack of an adapted model validation framework for decision-making process in both contexts (MA and HTA), (ii) inconsistent and unclear support of mechanistic models by the involved stakeholders (competent authorities, and stakeholders involved in HTA likewise), (iii) limited use cases with relevance to clinical development and HTA alike, and (iv) absence of appropriate incentives to use mechanistic modeling throughout the MP development lifecycle. The earlier a dedicated modeling strategy will be put in place the greater will be the demonstrated ability to predict a drug's impact, robustness, and credibility. Bringing mechanistic modeling to HTA, and thus the availability of this tool for the stakeholders requires, however, that drug developers, competent authorities and modelers anticipate the use in HTA.

While drug developers could generate more HTA-relevant data during Phase III, the resulting pivotal trials would be more complex and risk missing the statistical target. Drug developers should therefore consider mechanistic models to bridge this gap and report HTA-relevant modeling outcomes, validated with Phase III results.

Competent authorities will have a special role in facilitating model-based stakeholder engagement. They should issue more precise and dedicated guidance so that more modeling is included in MA. They should intensify the reporting of mechanistic modeling studies in benefit-risk assessment reports.

The modeler needs to embrace the fact that non-experts will also be exposed to the (potentially complex) model and its results. There is a lot of work being done concerning the communication and reporting of clinical trial results to patients and the public which are also applicable for simulated trials. There are EU Commission recommendations on the content of a lay summary (wording and layout) its development and dissemination—Good Lay Summary Practice (66). Communication of complex modeling results could profit from adopting such good practice.

Conclusion

The immediate and urgent unmet need for interventions and prophylaxis during the COVID-19 pandemic has suggested that drugs backed up by little empirical evidence (compared to the non-pandemic context), but a strong mechanistic background can be approved. The implications of this paradigm shift for HTA still need to be fully understood. In this article, we have advocated that mechanistic models can be used to reproduce, support and extrapolate clinical trials and could constitute a new type of evidence. Mechanistic models can provide causal and quantitative links between patient characteristics, personalized/realistic drug regimen or other extrinsic factors and individual benefit—under consideration of alternative treatment scenarios. They can therefore help to overcome barriers for a more quantitative appraisal of clinical data in HTA and they should also be considered to inform and educate special populations and individuals from a bottom-up perspective. Generation and uptake of in silico evidence will need more work of modelers, drug developers, and regulators, who will need to endorse and guide the use of mechanistic models early and consequently in the development process. Likewise, special attention will have to be paid to convey the totality of evidence to different stakeholder groups for empowering them to judge and formulate their specific viewpoint on the MP.

Data Availability Statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Author Contributions

EC, J-PB, AK, EP, JM, and RK wrote the manuscript. All authors discussed and reviewed the manuscript.

Conflict of Interest

EC, J-PB, RK, MH, EP, and AK are employed by Novadiscovery.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^Please note that, depending on the availability of knowledge, mechanistic modeling approaches may combine a fully mechanistic design for well-known processes with simplifications and assumptions or more phenomenological approaches where knowledge gaps exist. Therefore, the more general term Knowledge-Based Models, (KBM) might be more accurate than “mechanistic models,” but the latter is more widely used.

References

1. Banta D. The development of health technology assessment. Health Policy. (2003) 63:121–32. doi: 10.1016/S0168-8510(02)00059-3

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Horgan D, Lal JA. Making the most of innovation in personalised medicine: an EU strategy for a faster bench to bedside and beyond process. Public Health Genomics. (2018) 21:101–20. doi: 10.1159/000497809

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Asche CV, Seal B, Kahler KH, Oehrlein EM, Baumgartner MG. Evaluation of healthcare interventions and big data: review of associated data issues. Pharmacoeconomics. (2017) 35:759–65. doi: 10.1007/s40273-017-0513-5

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Franklin JM, Glynn RJ, Martin D, Schneeweiss S. Evaluating the use of nonrandomized real-world data analyses for regulatory decision making. Clin Pharmacol Ther. (2019) 105:867–77. doi: 10.1002/cpt.1351

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Lorgelly PK, Adler A. Impact of a global pandemic on health technology assessment. Appl Health Econ Health Policy. (2020) 18, 339–43. doi: 10.1007/s40258-020-00590-9

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Leahy J, Hickey C, McConnell D, Cassidy O, Trela-Larsen L, Barry M, et al. Coronavirus disease 2019: considerations for health technology assessment from the national centre for pharmacoeconomics review group. Value Health. (2020) 23:1423–6. doi: 10.1016/j.jval.2020.09.003

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Husereau D, Marshall DA, Levy AR, Peacock S, Hoch JS. Health technology assessment and personalized medicine: are economic evaluation guidelines sufficient to support decision making? Int J Technol Assess Health Care. (2014) 30:179–87. doi: 10.1017/S0266462314000142

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Lee Z, Rayner CR, Forrest JI, Nachega JB, Senchaudhuri E, Mills EJ. The rise and fall of hydroxychloroquine for the treatment and prevention of COVID-19. Am J Trop Med Hyg. (2021) 104:35–8. doi: 10.4269/ajtmh.20-1320

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Yoshida K, Budha N, Jin JY. Impact of physiologically based pharmacokinetic models on regulatory reviews and product labels: frequent utilization in the field of oncology. Clin Pharmacol Ther. (2017) 101:597–602. doi: 10.1002/cpt.622

PubMed Abstract | CrossRef Full Text | Google Scholar

10. de Winter W, DeJongh J, Post T, Ploeger B, Urquhart R, Moules I, et al. A mechanism-based disease progression model for comparison of long-term effects of pioglitazone, metformin and gliclazide on disease processes underlying Type 2 Diabetes Mellitus. J Pharmacokinet Pharmacodyn. (2006) 33:313–43. doi: 10.1007/s10928-006-9008-2

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Zheng J, van Schaick E, Wu LS, Jacqmin P, Perez Ruixo JJ. Using early biomarker data to predict long-term bone mineral density: application of semi-mechanistic bone cycle model on denosumab data. J Pharmacokinet Pharmacodyn. (2015) 42:333–47. doi: 10.1007/s10928-015-9422-4

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Mueller-Schoell A, Puebla-Osorio N, Michelet R, Green MR, Künkele A, Huisinga W, et al. Early survival prediction framework in CD19-specific CAR-T cell immunotherapy using a quantitative systems pharmacology model. Cancers. (2021) 13:2782. doi: 10.3390/cancers13112782

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Schmidt BJ, Papin JA, Musante CJ. Mechanistic systems modeling to guide drug discovery and development. Drug Discov Today. (2013) 18:116–27. doi: 10.1016/j.drudis.2012.09.003

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Morrissey KM, Yuraszeck TM, Li C-C, Zhang Y, Kasichayanula S. Immunotherapy and novel combinations in oncology: current landscape, challenges, and opportunities. Clin Transl Sci. (2016) 9:89–104. doi: 10.1111/cts.12391

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Mandema JW, Cox E, Alderman J. Therapeutic benefit of eletriptan compared to sumatriptan for the acute relief of migraine pain - Results of a model-based meta-analysis that accounts for encapsulation. Cephalalgia. (2005) 25:715–25. doi: 10.1111/j.1468-2982.2004.00939.x

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Thorlund K, Dron L, Park JJH, Mills EJ. Synthetic and external controls in clinical trials - A primer for researchers. Clin Epidemiol. (2020) 12:457–67. doi: 10.2147/CLEP.S242097

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Kenna LA, Labbé L, Barrett JS, Pfister M. Modeling and simulation of adherence: approaches and applications in therapeutics. AAPS J. (2005) 7:E390–E407. doi: 10.1208/aapsj070240

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Knights J, Heidary Z, Cochran JM. Detection of behavioral anomalies in medication adherence patterns among patients with serious mental illness engaged with a digital medicine system. JMIR Mental Health. (2020) 7:e21378. doi: 10.2196/21378

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Fröhlich F, Kessler T, Weindl D, Shadrin A, Schmiester L, Hache H, et al. Efficient parameter estimation enables the prediction of drug response using a mechanistic pan-cancer pathway model. Cell Syst. (2018) 7:567–79.e6. doi: 10.1016/j.cels.2018.10.013

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Boissel J-P, Pérol D, Décousus H, Klingmann I, Hommel M. Using numerical modeling and simulation to assess the ethical burden in clinical trials and how it relates to the proportion of responders in a trial sample. PLoS One. (2021) 16:e0258093. doi: 10.1371/journal.pone.0258093

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Kahoul R, Gueyffier F, Amsallem E, Haugh M, Marchant I, Boissel FH, et al. Comparison of an effect-model-law-based method versus traditional clinical practice guidelines for optimal treatment decision-making: application to statin treatment in the French population. J R Soc Interface. (2014) 11:20140867. doi: 10.1098/rsif.2014.0867

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Boissel JP, Cogny F, Marko N, Boissel FH. From clinical trial efficacy to real-life effectiveness: why conventional metrics do not work. Drugs Real World Outcomes. (2019) 6:125–32. doi: 10.1007/s40801-019-0159-z

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Ferreira S, Fisher C, Furlong LI, Laplanche L, Park BK, Pin C, et al. Quantitative systems toxicology modeling to address key safety questions in drug development: a focus of the TransQST Consortium. Chem Res Toxicol. (2020) 33:7–9. doi: 10.1021/acs.chemrestox.9b00499

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Yu H, Basu S, Hallow KM. Cardiac and renal function interactions in heart failure with reduced ejection fraction: a mathematical modeling analysis. PLoS Comput Biol. (2020) 16:e1008074. doi: 10.1371/journal.pcbi.1008074

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Cook SF, Bies RR. Disease progression modeling: key concepts and recent developments. Curr Pharmacol Rep. (2016) 2:221–30. doi: 10.1007/s40495-016-0066-x

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Wagenmakers E-J, Lee M, Lodewyckx T, Iverson GJ. Bayesian versus frequentist inference. In: Bayesian Evaluation of Informative Hypotheses. New York, NY: Springer (2008). p. 181–207.

Google Scholar

27. Bittl JA, He Y. Bayesian analysis: a practical approach to interpret clinical trials and create clinical practice guidelines. circulation: cardiovascular quality and outcomes. (2017) 10:e003563. doi: 10.1161/CIRCOUTCOMES.117.003563

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Kostis JB, Dobrzynski JM. Limitations of randomized clinical trials. Am J Cardiol. (2020) 129:109–15. doi: 10.1016/j.amjcard.2020.05.011

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Forstag EH, Kahn B, Gee AW, Shore C. (editors). Examining the Impact of Real-World Evidence on Medical Product Development. Washington, DC: National Academies Press (2019).

PubMed Abstract | Google Scholar

30. Pongiglione B, Torbica A, Blommestein H, de Groot S, Ciani O, Walker S, et al. Do existing real-world data sources generate suitable evidence for the HTA of medical devices in Europe? Mapping and critical appraisal. Int J Technol Assess Health Care. (2021) 37:e62. doi: 10.1017/S0266462321000301

PubMed Abstract | CrossRef Full Text | Google Scholar

31. FDA. CDER Conversation: Model Informed Drug Development. (2018). Available online at: https://www.fda.gov/drugs/news-events-human-drugs/cder-conversation-model-informed-drug-development (accessed October 19, 2021).

32. Wang Y, Zhu H, Madabushi R, Liu Q, Huang SM, Zineh I. Model-informed drug development: current US regulatory practice and future considerations. Clin Pharmacol Ther. (2019) 105:899–911. doi: 10.1002/cpt.1363

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Bai JPF, Schmidt BJ, Gadkar KG, Damian V, Earp JC, Friedrich C, et al. FDA-Industry Scientific Exchange on assessing quantitative systems pharmacology models in clinical drug development: a meeting report, summary of challenges/gaps, future perspective. AAPS J. (2021) 23:60. doi: 10.1208/s12248-021-00585-x

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Musuamba FT, Skottheim Rusten I, Lesage R, Russo G, Bursi R, Emili L, et al. Scientific and regulatory evaluation of mechanistic in silico drug and disease models in drug development: building model credibility. CPT Pharmacometrics Syst Pharmacol. (2021) 10:804–25. doi: 10.1002/psp4.12669

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Peterson MC, Riggs MM. A physiologically based mathematical model of integrated calcium homeostasis and bone remodeling. Bone. (2010) 46:49–63. doi: 10.1016/j.bone.2009.08.053

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Gauthier LD, Greenstein JL, O'Rourke B, Winslow RL. An integrated mitochondrial ROS production and scavenging model: implications for heart failure. Biophys J. (2013) 105:2832–42. doi: 10.1016/j.bpj.2013.11.007

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Hallow KM, Gebremichael Y. A quantitative systems physiology model of renal function and blood pressure regulation: model description. CPT Pharmacometrics Syst Pharmacol. (2017) 6:383–92. doi: 10.1002/psp4.12178

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Palgen J-L, Perrillat-Mercerot A, Ceres N, Peyronnet E, Coudron M, Tixier E, et al. Integration of heterogeneous biological data in multiscale mechanistic model calibration: application to lung adenocarcinoma. bioRxiv [Preprint]. doi: 10.1101/2022.01.17.476676

CrossRef Full Text | Google Scholar

39. Dodds MG, Krishna R, Goncalves A, Rayner CR. Model-informed drug repurposing: viral kinetic modelling to prioritize rational drug combinations for COVID-19. Br J Clin Pharmacol. (2021) 87:3439–50. doi: 10.1111/bcp.14486

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Giorgi M, Desikan R, Graaf PH, Kierzek AM. Application of quantitative systems pharmacology to guide the optimal dosing of COVID-19 vaccines. CPT Pharmacometrics Syst Pharmacol. (2021) 10:1130–3. doi: 10.1002/psp4.12700

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Chabaud S Girard P Nony P Boissel J-P and HERapeutic MOdeling and Simulation Group. Clinical trial simulation using therapeutic effect modeling: application to ivabradine efficacy in patients with angina pectoris. J Pharmacokinet Pharmacodyn. (2002) 29:339–63. doi: 10.1023/A:1020953107162

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Kim Y, Hatley O, Rhee S, Yi S, Lee HA, Yoon S, et al. Development of a Korean-specific virtual population for physiologically based pharmacokinetic modelling and simulation. Biopharmaceutics Drug Dispos. (2019) 135–50. doi: 10.1002/bdd.2178

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Henscheid N. Generating patient-specific virtual tumor populations with reaction-diffusion models and molecular imaging data. Math Biosci Eng. (2020) 17:6531–56. doi: 10.3934/mbe.2020341

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Viceconti M, Henney A, Morley-Fletcher E. In silico Clinical Trials: how computer simulation will transform the biomedical industry. Int J Clin Trials. (2016) 3:37. doi: 10.18203/2349-3259.ijct20161408

CrossRef Full Text | Google Scholar

45. Viceconti M, Emili L, Afshari P, Courcelles E, Curreli C, Famaey N, et al. Possible contexts of use for in silico trials methodologies: a consensus-based review. IEEE J Biomed Health Informatics. (2021) 25:3977–82. doi: 10.1109/JBHI.2021.3090469

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Jansen JP, Naci H. Is network meta-analysis as valid as standard pairwise meta-analysis? It all depends on the distribution of effect modifiers. BMC Med. (2013) 11:159. doi: 10.1186/1741-7015-11-159

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Upreti VV, Venkatakrishnan K. Model-based meta-analysis: optimizing research, development, and utilization of therapeutics using the totality of evidence. Clin Pharmacol Ther. (2019) 106:981–92. doi: 10.1002/cpt.1462

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Mawdsley D, Bennetts M, Dias S, Boucher M, Welton NJ. Model-based network meta-analysis: a framework for evidence synthesis of clinical trial data. CPT Pharmacometrics Syst Pharmacol. (2016) 5:393–401. doi: 10.1002/psp4.12091

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Reeve BB, Smith AW, Arora NK, Hays RD. Reducing bias in cancer research: application of propensity score matching. Health Care Financ Rev. (2008) 29:69–80.

PubMed Abstract | Google Scholar

50. Ademi Z, Kim H, Zomer E, Reid CM, Hollingsworth B, Liew D. Overview of pharmacoeconomic modelling methods. Br J Clin Pharmacol. (2013) 75:944–50. doi: 10.1111/j.1365-2125.2012.04421.x

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Thomas D, Hiligsmann M, John D, Al Ahdab OG, Li H. Pharmacoeconomic analyses and modeling. In: Thomas D, editor. Clinical Pharmacy Education, Practice and Research: Clinical Pharmacy, Drug Information, Pharmacovigilance, Pharmacoeconomics and Clinical Research. Amsterdam; Oxford; Cambridge, MA: Elsevier (2018). p. 261–75. doi: 10.1016/B978-0-12-814276-9.00018-0. Available online at: https://www.sciencedirect.com/book/9780128142769/clinical-pharmacy-education-practice-and-research#book-info

CrossRef Full Text | Google Scholar

52. Carta A, Conversano C. On the use of markov models in pharmacoeconomics: pros and cons and implications for policy makers. Front Public Health. (2020) 8:569500. doi: 10.3389/fpubh.2020.569500

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Caro JJ, Möller J. Advantages and disadvantages of discrete-event simulation for health economic analyses. Expert Rev Pharmacoecon Outcomes Res. (2016) 16:327–9. doi: 10.1586/14737167.2016.1165608

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Chhatwal J, He T. Economic evaluations with agent-based modelling: an introduction. Pharmacoeconomics. (2015) 33:423–33. doi: 10.1007/s40273-015-0254-2

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Tantivess S. Policy making and roles of health technology assessment. J Med Assoc Thailand. (2008) 91(Suppl. 2):S88–S99. Available online at: http://ihppthaigov.net/DB/publication/attachinter/178/Full-text.pdf (accessed February 7, 2022).

PubMed Abstract

56. HTAi (Health Technology Assessment International) Interest Groups. Patient and Citizen Involvement. Available online at: https://htai.org/interest-groups/pcig/ (accessed October 29, 2021).

57. Scott AM, Wale JL. Patient advocate perspectives on involvement in HTA: an international snapshot. Res Involv Engag. (2017) 3:2. doi: 10.1186/s40900-016-0052-9

PubMed Abstract | CrossRef Full Text | Google Scholar

58. O'Rourke B, Werkö SS, Merlin T, Huang LY, Schuller T. The ‘Top 10' challenges for health technology assessment: INAHTA viewpoint. Int J Technol Assess Health Care. (2020) 36:1–4. doi: 10.1017/S0266462319000825

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Christiaens W, Kohn, L, Léonard, C, Denis, A, Daue, F, Cleemput, I,. KCE REPORT 195C, Models for Citizen Patient Involvement in Health Care Policy. Part I: Exploration of Their Feasibility Acceptability. (2012). Available online at: https://kce.fgov.be/sites/default/files/atoms/files/KCE_195C_citizen_and_patient_participation.pdf (accessed February 05, 2022).

60. Tako AA, Kotiadis K. Participative simulation (Partisim): a facilitated simulation approach for stakeholder engagement. In: 2018 Winter Simulation Conference (WSE). Gothenburg:IEEE (2019). p. 192–206.

Google Scholar

61. Abelson J, Giacomini M, Lehoux P, Gauvin FP. Bringing ‘the public' into health technology assessment and coverage policy decisions: from principles to practice. Health Policy. (2007) 82:37–50. doi: 10.1016/j.healthpol.2006.07.009

PubMed Abstract | CrossRef Full Text | Google Scholar

62. European Network for Health Technology Assessment. Healthcare Professional Involvement in Relative Effectiveness Assessments. (2020). Available online at: https://www.eunethta.eu/wp-content/uploads/2020/04/Final_HCP-Involvement-in-EUnetHTA-assessments.pdf (accessed February 05, 2022).

63. Bergmann L, Enzmann H, Broich K, Hebborn A, Marsoni S, Goh L, et al. Actual developments in European regulatory and health technology assessment of new cancer drugs: what does this mean for oncology in Europe? Ann Oncol. (2014) 25:303–6. doi: 10.1093/annonc/mdt488

PubMed Abstract | CrossRef Full Text | Google Scholar

64. European Federation of Pharmaceutical Industries Associations. Joint Healthcare Industry Paper - The Value of Industry Involvement in HTA. (2011). Available online at: https://www.efpia.eu/media/25171/the-value-of-industry-involvement-in-hta-december-2011.pdf (accessed February 05, 2022).

65. Cheung KL, Evers SMAA, de Vries H, Hiligsmann M. Most important barriers and facilitators regarding the use of health technology assessment. Int J Technol Assess Health Care. (2017) 33:183–91. doi: 10.1017/S0266462317000290

PubMed Abstract | CrossRef Full Text | Google Scholar

66. Clinical Trials Expert Group,. Good Lay Summary Practice. (2021). Available online at: https://ec.europa.eu/health/system/files/2021-10/glsp_en_0.pdf (accessed February 5, 2022).