Milou A. Hogervorst
Johan Pontén3
Rick A. Vreman
Aukje K. Mantel-Teeuwisse
Wim G. Goettsch- 1Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences (UIPS), Utrecht University, Utrecht, Netherlands
- 2National Health Care Institute (ZIN), Diemen, Netherlands
- 3The Dental and Pharmaceutical Benefits Agency (TLV), Stockholm, Sweden
The available evidence on relative effectiveness and risks of new health technologies is often limited at the time of health technology assessment (HTA). Additionally, a wide variety in real-world data (RWD) policies exist among HTA organizations. This study assessed which challenges, related to the increasingly complex nature of new health technologies, make the acceptance of RWD most likely. A questionnaire was disseminated among 33 EUnetHTA member HTA organizations. The questions focused on accepted data sources, circumstances that allowed for RWD acceptance and barriers to acceptance. The questionnaire was validated and tested for reliability by an expert panel, and pilot-tested before dissemination via LimeSurvey. Twenty-two HTA organizations completed the questionnaire (67%). All reported accepting randomized clinical trials. The most accepted RWD source were patient registries (19/22, 86%), the least accepted were editorials and expert opinions (8/22, 36%). With orphan treatments or companion diagnostics, organizations tended to be most likely to accept RWD sources, 4.3–3.2 on a 5-point Likert scale, respectively. Additional circumstances were reported to accept RWD (e.g., a high disease burden). The two most important barriers to accepting RWD were lacking necessary RWD sources and existing policy structures. European HTA organizations seem positive toward the (wider) use of RWD in HTA of complex therapies. Expanding the use of patient registries could be potentially useful, as a large share of the organizations already accepts this source. However, many barriers still exist to the widespread use of RWD. Our results can be used to prioritize circumstances in which RWD might be accepted.
Introduction
Sufficient amounts and quality of data on a treatment’s effects, safety and costs is of crucial importance in order to minimize uncertainty in decision-making on reimbursement (Vreman et al., 2020a; Hogervorst et al., 2021). However, the available evidence for these evaluations can be limited (Hogervorst et al., 2021). Novel, complex treatment strategies such as health technologies with concomitant genetic testing, advanced therapy medicinal products (ATMPs) or the assessment of sequences of treatments with disease-modifying capacities may be especially prone to these limitations (Garrison et al., 2015; Lipska et al., 2015; Berm et al., 2016; Vreman et al., 2019; Ghabri et al., 2020; Lloyd-Williams and Hughes, 2020). A survey among European health technology assessment (HTA) organizations assessed challenges associated with such complex health technologies. HTA assessors reported various challenges occurring at all steps throughout the HTA process, but regardless of step in the process, most of the reported challenges rooted in data insufficiencies at the time of HTA assessment (Hogervorst et al., 2021).
Some of these complex health technologies inherently cause data insufficiencies, making them more challenging for HTA (Ribeiro et al., 2020). Often there is uncertainty around long-term claims for gene-therapies and other ATMPs as clinical trials do not cover life-long effects (Fruergaard Jorgensen, 2017; Jönsson et al., 2019; Coyle et al., 2020). There is an increasing amount of precision medicines and orphan medicines that may limit the possibility of performing well-controlled, large trials (Margaret, 2013; Moloney et al., 2015; Griffiths et al., 2017). Additionally, patients’ access to new health technologies may be requested more early in the authorization and reimbursement process because of high unmet medical needs, which has led to the development of expedited approval processes (Kesselheim et al., 2015; Leyens and Brand, 2016). This trend also results in decreasing amounts and quality of data available at the time of the HTA process. For example, interim data or effectiveness data only based on surrogate endpoints is all that is available upon approval of a newly developed intervention (Larson et al., 2017; Pruce et al., 2017; Blagden et al., 2020; Ribeiro et al., 2020).
For both reasons -increased complexity of health technologies and expedited approval processes-there is an increased necessity of real-world data (RWD) as an addition to data from traditional randomized controlled trials (RCT) (Makady et al., 2017a; IMI GetReal, 2021). Simultaneously, there is a strong voice supporting the use of “real-world” effectiveness in addition to the RCT-tested efficacy for reimbursement decision-making, due to another limitation of RCTs referred to as the “efficacy-effectiveness gap”. (Eichler et al., 2011; Ankarfeldt et al., 2017).
Published economic evaluations based on RWD, on the other hand, have trouble meeting quality criteria, as guided by the CHEERS checklist, for properly accounting for biases (Guertin et al., 2017; Parody-Rúa et al., 2020). In general, the quality in terms of internal validity is a concern with RWD, which has been extensively discussed (Collins et al., 2020; Eichler et al., 2020; Ramagopalan et al., 2020). The lack of external validity of RCTs and current quality issues with RWD make reimbursement decisions difficult.
As the implementation of RWD in HTA is challenging but nevertheless increasing, organizations such as the Professional Society for Health Economics and Outcomes Research (ISPOR), the International Society for Pharmacoepidemiology (ISPE), European Network for HTA (EUnetHTA, or EUnetHTA 21), the Food and Drug Administration (FDA) and European Medicines Agency (EMA) have published their positions on inclusion of RWD in regulation and HTA decision-making (Berger et al., 2017; Public Policy Committee and International Society of Pharmacoepidemiology, 2016; Research C for DE, 2020; Anonymous, 2018; EUnetHTA Publication of a JA2 methodological guideline, 2015; ENCePP Home Page, 2021). These statements generally agree that communication, for example on study protocols, outcomes selection, measurement, reporting and interpretation is important as well as stakeholder collaboration or alignment regarding the design, conduct and interpretation of RWD studies. All statements describe that careful consideration is needed for each individual situation to assess whether the use of RWD is appropriate, focusing on internal validity (various forms of bias) or external validity (applicability to practice), and some even indicate the methods that should be used.
Makady et al. demonstrated, with large differences among HTA organizations and between all the domains of the HTA process, that current policies and guidelines from HTA agencies in six European countries did in general not actively encourage the use of RWD (Makady et al., 2017a). Certainly this was the case in relative effectiveness assessments (REAs), whereas the interest in RWD in the case of cost-effectiveness assessments (CEAs) was wider, sometimes even requested. In HTA practice of oncological drugs for the treatment of melanoma, another study by Makady et al. showed that in five European countries, the actual use of RWD was indeed higher for CEA parameters than for the REA, although many differences among the countries were still observed (Makady et al., 2018). Alignment on policies for RWD use could increase the adaptation of RWD in HTA processes in Europe.
Literature is not conclusive on the types of RWD that are accepted, nor on the transferability of these results to other countries. Additionally, there is a lack of knowledge about the circumstances in which RWD is more likely to be used, such as specific types of complex health technologies or procedural circumstances. Besides the general concern on quality and robustness of RWD, there is no insight in barriers to the use of RWD in HTA. In an attempt to guide future research on method development and alignment of policies for expanding use of RWD in HTA, this study assesses which challenges in HTA, related to the increasingly complex nature of new health technologies, make the acceptance of RWD most likely, based on practical experiences by European HTA organizations.
Materials and Methods
Data were collected through dissemination of a questionnaire. This strategy allowed us to gain insight in daily practice in HTA organizations. The questions on RWD were combined with questions informing additional deliverables in the HTx project, all focusing on complex treatments. The HTx project is a Horizon 2020 project supported by the European Union lasting for 5 years from January 2019, with the aim to create a framework for the Next Generation HTA to support patient-centered, societally oriented, real-time decision-making on access to and reimbursement for health technologies throughout Europe (HTx, 2020). The broad definition for RWD that we used was ‘all routinely collected data on patients that are not RCTs’ (Makady et al., 2017b). A full description of the methods on the questionnaire’s development, validation and dissemination is published in an earlier study (Hogervorst et al., 2021).
Both national and regional European HTA organizations were invited using the EUnetHTA member database, 33 in total. This approach ensured the representation of a balanced mixture of European countries. All organizations were directly involved in decision-making or advised decision-making parties. The targeted representatives in these organizations were experienced HTA assessors (at least 3–5 years of experience), to ensure sufficient knowledge and experience. Appendix 1 shows a list with all selected and responding HTA organizations.
Questionnaire Structure
Three topics were addressed in the part of the questionnaire that focused on RWD: the general willingness to use and accept RWD, likelihood to accept RWD in particular challenging circumstances and the barriers to accept RWD. First, respondents could indicate whether they experienced the need for wider systematic use of RWD and whether they experienced a willingness among assessors, among decision-makers or among both groups, as both affect the ultimate reimbursement decision. Second, respondents selected in a binary way (yes/no) the types of data sources that were accepted for assessment by their organization. Third, respondents indicated their likelihood for accepting RWD sources in predefined challenging circumstances on a 5-point Likert scale. The selection of these predefined treatments has been described in an earlier study (Hogervorst et al., 2021). In the fourth and only open question, respondents could describe additional circumstances that they encountered that would allow for the acceptance of RWD in HTA, ensuring the sensitivity of the questionnaire. Last, the respondents ranked a list of barriers to accepting RWD for HTA. This list was based on literature and practical experience of authors (JP, RV, WG) affiliated at HTA organizations.
Questionnaire Validation and Testing
The questionnaire was thoroughly tested for validation and reliability before dissemination (Kimberlin and Winterstein, 2008; Bolarinwa, 2020; Venkitachalam, 2021). An expert panel with two representatives from academia and five from three HTA organizations (i.e. the National Institute for Health and Care Excellence [NICE], the Dental and Pharmaceutical Benefits Agency [TLV], the Dutch National Health Care Institute [ZIN]), tested the questionnaire for content and face validity, as well as reliability. Subsequently, a pilot test was performed to test the feasibility of completion and correct interpretation of the full set of questions. The pilot test was not considered in the presented results and performed by other representatives in the specific HTA organizations than the final participants. The more detailed development and validation approach has also been described in earlier published work (Hogervorst et al., 2021).
Dissemination and Analysis
The questionnaire was built in LimeSurvey (LimeSurvey GmBH, Hamburg, Germany) and was disseminated between January and February 2020 (Schmitz, 2020). To increase the response rate, an announcement as well as 3 reminders were sent to targeted participants. The analyses were performed in Microsoft Excel (Microsoft, Redmond, WA) and GraphPad Prism v9.0.0 for Windows (PraphPad Software, San Diego, CA) (Microsoft Excel and Spreadsheet Software, 2020; Home - GraphPad [Internet], 2021). Descriptive statistics were used to visualize the questionnaire results, using averages, bar charts, boxplots and tables for the open question.
Results
Out of 33 invited HTA organizations, 22 organizations from 21 different countries completed the questionnaire (response rate 67%), see appendix. There was a relatively balanced spread of organizations throughout Europe, with a slight overrepresentation from the Nordic countries. Twenty-one responding organizations (95%) were responsible for assessing pharmaceuticals, of which nine (41%) were assessing solely pharmaceuticals. Thirteen organizations (59%) were responsible for assessment of non-pharmaceuticals, of which one (5%) solely assessed non-pharmaceuticals. Consequently, twelve organizations (55%) were responsible for assessing both pharmaceuticals and non-pharmaceuticals.
Willingness to Use and Accept RWD Sources in HTA
Out of 22 representatives, 18 (82%) indicated to see a need for wider systematic use of RWD in HTA decisions than is currently in practice. Sixteen representatives indicated that they experience a willingness among both assessors and decision-makers, two representatives indicated they experience a willingness only among the assessors and one only among decision-makers. The remainder indicated to have no knowledge about the willingness or to experience no willingness at all. When looking at the accepted data sources, the traditional sources, i.e., meta-analyses, systematic reviews and randomized controlled trials (RCTs), embodied the top three accepted data sources for HTA in Europe. All three were accepted by all participating organization. This was followed by patient registries, which were accepted by 19 HTA organizations (86%). Case reports, unpublished data and editorial and expert opinions were among the least accepted RWD sources, each accepted by one third of the organizations (Figure 1).
FIGURE 1. Different data sources accepted by HTA organizations (N = 22).
Likelihood to Accept RWD in Challenging Circumstances
The assessment of orphan drugs or other treatments with small patient populations was presented as the most likely situation to accept RWD sources, scoring 4.3 out of the 5-point Likert scale. This was closely followed by (companion) diagnostic procedures and surgical interventions, scoring 4.2 and 4.1, respectively. Organizations would be least likely to accept RWD in HTA if this data came from countries in regions outside their own region, despite it being the only available data source (3.2), see Figure 2. The scores ranged between 3.2 and 4.3, creating a substantial gap in likelihood of accepting RWD between the first and last ranked situation (between orphan drugs and using RWD from outside your country’s region). Generally, the results indicate that among these circumstances the attitude towards RWD acceptance leans more toward positive than to negative.
FIGURE 2. Average Likert scores (1–5) with standard deviation of the likelihood to accept (additional) RWD in various challenging circumstances in HTA. ATMP, advanced therapy medicinal product.
Seven organizations reported additional circumstances where RWD may be accepted. Some reported that RWD would not be accepted as the sole source of evidence, though could be supplementary to traditional RCT evidence. In case of no available RCT data, single-arm studies could be accepted. However, a high level of uncertainty would still be a concern in this case. Table 1 shows all reported circumstances in which organizations would be willing to accept RWD, as derived from the open question.
TABLE 1. Additional circumstances in which the HTA organizations would be willing to accept RWD for their assessments. RWD, real world data; RCT, randomized clinical trial; CEA, cost-effectiveness analysis; HT, health technologies.
Barriers to Accept RWD
There was considerable variation in responses of HTA agencies. On average, the organizations ranked “lacking necessary RWD sources” as the most important barrier to being able to accept RWD in HTA with a mean rank of 3.3, see Figure 3. This was followed by “existing policy structures or information governance” (mean rank 3.5) that complicated accepting RWD, and third, that there was “no possibility to interpret or verify data, or that it was challenging to do so” (mean rank 3.9). “Financial reasons” and “lack of statisticians or other relevant analysts” ranked last. When considering the median in the boxplots instead of the mean ranks listed on the left side of the figure, the order of ranking is almost equal, except for “lack methods to use RWD”, due to a skewed spread of rankings (Figure 3). “No possibility to interpret or verify data, or that it was challenging to do so” showed the most consistent ranking among all reasons (smallest interquartile range). Additionally, the medians of “necessary data sources are lacking’ (2.5) and “existing policy structures or information governance” (3.0) show that, despite the wide range of ranks, more than half the HTA organizations ranked these two reasons in the top three.
FIGURE 3. Boxplots of the ranking of the barriers to accepting RWD in HTA or decision-making. The barriers are placed in order of their median ranked score and include the 25th and 75th percentile as well as the minimum and maximum rank. The mean scores are listed between brackets on the left side.
Discussion
As input for their HTA decision-making process, all HTA organizations reported the acceptance of meta-analyses, systematic reviews and traditional RCTs. As expected, data sources not based on RCTs were less accepted. Patient registries were the most accepted RWD source. The least accepted sources–accepted by only one third—included case series or reports, unpublished data sources and editorials or expert opinions. A large share of respondents indicated to experience the need and willingness among both assessors and decision-makers for wider systematic use of RWD in HTA. HTA assessors also indicated a likelihood to accept RWD sources in all of our pre-defined “complex” situations, as well as in a number of additionally reported situations. Among predefined barriers to using RWD, lacking sources and existing policy structures ranked highest. Despite the great spread in answers among HTA organizations, lacking resources (available methods, sufficient finances or skilled employees) seemed to be least hindering the use of RWD.
Makady et al. found in RWD policies among six European HTA organizations, that all six accepted any source of data, including RWD, albeit with a certain hierarchy (22). Based on our results from 21 countries, it seems that other European countries may be more reluctant to accept RWD. The hierarchy among data sources is, however, also reflected in our results. Makady’s finding that registries are widely used seem to be transferable to other European countries based on our results (Makady et al., 2018). Moreover, their study showed that RWD was uncommonly accepted in the initial assessment of effectiveness, however for an estimation for long-term effectiveness (stretching further than RCT length) RWD seemed to become more relevant (Makady et al., 2018). As long-term effect claims based on short RCTs are a common feature in ATMP assessment, this explains why ATMPs ranked in the top 5 circumstances to accept RWD in the present study.
According to our questionnaire, difficulty with verifying and interpreting RWD seemed to be a less important barrier to the use of RWD than the lack of data and existing policies. However, an inquiry by Facey et al., indicated that a lack of clarity on methods to assess real-world evidence (RWE) is seen by HTA agencies as a major obstacle to adequate use of RWE in HTA (Facey et al., 2020). Our ranking method solely visualized the order of importance. Combined with these previous findings we may conclude that all of our listed barriers, including the lowest ranked, should be considered as important. Therefore, initiatives to help overcome barriers to use RWD in HTA are an important next step. These initiatives could first include methodological work on RWD quality assessment, alternative trial designs or combining RCT with RWD sources as well as implementing methods into the decision making process (Selker et al., 2014; Makady et al., 2017c; Efthimiou et al., 2017; Selker et al., 2019; Chalkou et al., 2021). Second, the focus could be on consensus building on when and how to use RWD (methods) (Berger et al., 2017; Vreman et al., 2020b; Facey et al., 2020; Orsini et al., 2020). A third suggestion could be to invest more in generating and aligning the required data, such as patient registries, at an early stage, as is aimed by the European IMI EHDEN project as well as by national initiatives across the continent (Ministerie van VolksgezondheidWen, 2019; IMI Innovative Medicines Initiative EHDEN, 2021; The National Patient Register, 2021).
Our results, supported by existing initiatives, could be used to guide prioritization. Reasoning from the most likely situations where RWD may be acceptable based on our findings, focus could first be put on RWD for orphan drugs, ATMPs, data for assessments of sequences of treatments, promising treatments for high burden diseases or other treatments that generally come with high uncertainty (Makady et al., 2017a; Didden et al., 2018; Moseley et al., 2020; Hogervorst et al., 2021). Non-pharmaceuticals, including surgical interventions, diagnostics, wearables or other devices, gene sequencing techniques and digital interventions, need special attention as these do not generally require the same data quality standards as compared to pharmaceuticals (FITTH, 2018). Unfortunately, due to a low number of agencies assessing non-pharmaceuticals, we could not compare responses between agencies that assess medicines only versus those that also or only assess non-pharmaceuticals in the present study.
Due to new European HTA legislation, a new consortium of HTA partners called EUnetHTA21, will provide an updated methodological framework (Health Technology Assessment, 2021; History of EUnetHTA, 2021). At this moment it is still unclear how questions on collection, use and assessment of RWD will be addressed in the activities of EUnetHTA21. However, the current divergence in HTA policies and RWD acceptance found in literature (Makady et al., 2017a; Makady et al., 2018; Health Technology Assessment, 2021), as well as our results indicating that policies still complicate the use of RWD, suggest that at least European coordination and consensus-building is necessary to ensure adequate, consistent and reliable use of RWD in HTA practice.
Strengths and Limitations
In this study we used the term RWD as referring to all routinely collected information about people outside a randomized controlled trial setting, and RWE as referring to the evidence derived from the analysis of RWD (Makady et al., 2017b; Gonzalez, 2020; Commissioner O of the. Real-World Evidence, 2021). We acknowledge the discussion surrounding the terminology, as many new, flexible, adaptive and pragmatic trial designs emerge, even in registries, making it more difficult to distinguish between a controlled setting and uncontrolled setting (Efthimiou et al., 2017; Didden et al., 2018; Schünemann, 2019; Ford and Norrie, 2016; Pallmann et al., 2018; Li et al., 2016; Hamers et al., 20215). RWD, however, is a widely recognized term, specifically among our targeted participants.
Our results were informed by responses from 22 HTA organizations representing 21 European countries. Only a few larger countries such as Portugal, France and Italy did not respond. We would, however, not expect large differences since we included a large cohort of countries, small and large, from all European regions. Additionally, the disseminated questionnaire was well tested for validity and reliability by an expert panel, including experienced representatives from HTA practice. This ensures our questions and thus results are well aligned with practice. The inclusion of multiple middle/lower income countries in Europe allows for some transferability of results to similar countries in other regions that perform HTA and wish to accept (more) RWD. (Justo et al., 2019; Lou et al., 2020).
We did not include the newer advanced (pragmatic or flexible) trial designs in our questionnaire, so we do not know yet how these relate to observational designs. However, as these designs are new and not yet widely used, it is likely that HTA organizations have not encountered them often. Inquiring after the differences between initial assessments an reassessments and additional barriers may be useful, but were outside the scope of this study.
Conclusion
Despite the wide variety in acceptance of RWD sources and in the concomitant policies, assessors from European HTA organizations do seem to be positive toward the (wider) use of RWD for their assessments. Expanding use of patient registries seems potentially useful as a large share of organizations already accept this data source. Our results can be used to prioritize health technologies and circumstances in which accepting RWD could be desired. Initial focus could be placed on orphan therapies and diagnostic tools, innovative treatments for high burden diseases or on health technologies that generally come with high uncertainty. However, many barriers to the use of RWD still exist. Future research and policies should focus on strategies to build and maintain high quality patient registries as well as on consensus building and implementation of trustworthy evidence generation and assessment methods based on these registries.
Data Availability Statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics Statement
Ethical review and approval was not required for the study of human participants in accordance with the local legislation and institutional requirements. Written informed consent from the participants was not required to participate in this study in accordance with the national legislation and the institutional requirements.
Author Contributions
All authors contributed to the conceptualization and design of the study, as well as to drafting the manuscript and a critical and intellectual revision of the content. Data acquisition, analysis and initial interpretation was performed by MH.
Funding
This research was performed as part of the HTx project. The project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 825162. This dissemination reflects only the author’s view and the Commission is not responsible for any use that may be made of the information it contains.
Conflict of Interest
RV and WG declare that during the conduct of this study they worked part-time for the Dutch National Health Care Institute, unrelated to this study.
The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Acknowledgments
The authors wish to express their gratitude towards members of the HTx consortium, engaged in this project, for their feedback on the manuscript. In particular we would like to thank Dalia Dawoud, Richard Ofori-Asenso and prof. Marie Louise De Bruin for their involvement in the development of the questionnaire and Ayla Lokhorst for dissemination of the questionnaire.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fphar.2022.837302/full#supplementary-material
Footnotes
1Well-established use: this is the case if the active compound in a pharmaceutical has been used for more than 10 years and the efficacy and safety are thus “well-established”. WEU product dossiers need to fulfil legislative requirements of Directive 2001/83/EC by showing that the product applying for market access is safe and efficacious and of high quality
References
Ankarfeldt, M. Z., Adalsteinsson, E., Groenwold, R. H., Ali, M. S., and Klungel, O. H. (2017). A Systematic Literature Review on the Efficacy-Effectiveness gap: Comparison of Randomized Controlled Trials and Observational Studies of Glucose-Lowering Drugs. Clin. Epidemiol. 9, 41–51. doi:10.2147/CLEP.S121991
Anonymous, . (2018). Good Pharmacovigilance Practices [Internet]. Amsterdam: European Medicines Agency. Available at: https://www.ema.europa.eu/en/human-regulatory/post-authorisation/pharmacovigilance/good-pharmacovigilance-practices (Accessed on May 12, 2021)
Berger, M. L., Sox, H., Willke, R. J., Brixner, D. L., Eichler, H. G., Goettsch, W., et al. (2017). Good Practices for Real-World Data Studies of Treatment And/or Comparative Effectiveness: Recommendations from the Joint ISPOR-ISPE Special Task Force on Real-World Evidence in Health Care Decision Making. Pharmacoepidemiol. Drug Saf. 26 (9), 1033–1039. doi:10.1002/pds.4297
Berm, E. J., Looff, Md., Wilffert, B., Boersma, C., Annemans, L., Vegter, S., et al. (2016). Economic Evaluations of Pharmacogenetic and Pharmacogenomic Screening Tests: A Systematic Review. Second Update of the Literature. PLoS One 11 (1), e0146262. doi:10.1371/journal.pone.0146262
Blagden, S. P., Billingham, L., Brown, L. C., Buckland, S. W., Cooper, A. M., Ellis, S., et al. (2020). Effective Delivery of Complex Innovative Design (CID) Cancer Trials-A Consensus Statement. Br. J. Cancer 122 (4), 473–482. doi:10.1038/s41416-019-0653-9
Bolarinwa, O. A. Principles and Methods of Validity and Reliability Testing of Questionnaires Used in Social and Health Science Researches. [Internet]. [cited 2020 Jul 16]. Available at: http://www.npmj.org/article.asp?issn=1117-1936.
Chalkou, K., Steyerberg, E., Egger, M., Manca, A., Pellegrini, F., and Salanti, G. (2021). A Two-Stage Prediction Model for Heterogeneous Effects of many Treatment Options: Application to Drugs for Multiple Sclerosis. ArXiv200413464 Stat [Internet]Available at: http://arxiv.org/abs/2004.13464 (Accessed on May 12, 2021).
Collins, R., Bowman, L., Landray, M., and Peto, R. (2020). The Magic of Randomization versus the Myth of Real-World Evidence. N. Engl. J. Med. 382 (7), 674–678. doi:10.1056/NEJMsb1901642
Commissioner O of the. Real-World Evidence, (2021). FDA. Available at: https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence (Accessed on Nov 29, 2021).
Coyle, D., Durand-Zaleski, I., Farrington, J., Garrison, L., Graf von der Schulenburg, J.-M., Greiner, W., et al. (2020). HTA Methodology and Value Frameworks for Evaluation and Policy Making for Cell and Gene Therapies. Eur. J. Health Econ. 21, 1421–1437. doi:10.1007/s10198-020-01212-w
Didden, E. M., Ruffieux, Y., Hummel, N., Efthimiou, O., Reichenbach, S., Gsteiger, S., et al. (2018). Prediction of Real-World Drug Effectiveness Prelaunch: Case Study in Rheumatoid Arthritis. Med. Decis. Making 38 (6), 719–729. doi:10.1177/0272989X18775975
Efthimiou, O., Mavridis, D., Debray, T. P., Samara, M., Belger, M., Siontis, G. C., et al. (2017). Combining Randomized and Non-randomized Evidence in Network Meta-Analysis. Stat. Med. 36 (8), 1210–1226. doi:10.1002/sim.7223
Eichler, H. G., Abadie, E., Breckenridge, A., Flamion, B., Gustafsson, L. L., Leufkens, H., et al. (2011). Bridging the Efficacy-Effectiveness gap: a Regulator's Perspective on Addressing Variability of Drug Response. Nat. Rev. Drug Discov. 10 (7), 495–506. doi:10.1038/nrd3501
Eichler, H. G., Koenig, F., Arlett, P., Enzmann, H., Humphreys, A., Pétavy, F., et al. (2020). Are Novel, Nonrandomized Analytic Methods Fit for Decision Making? the Need for Prospective, Controlled, and Transparent Validation. Clin. Pharmacol. Ther. 107 (4), 773–779. doi:10.1002/cpt.1638
ENCePP Home Page, (2021). Available at: http://www.encepp.eu/encepp_studies/indexRegister.shtml (Accessed on Aug 20, 2021).
EUnetHTA Publication of a Ja2 methodological guideline, (2015). Internal Validity of Non-randomised Studies (NRS) on Interventions [Internet]. Available at: https://eunethta.eu/publication-of-a-ja2-methodological-guideline-internal-validity-of-non-randomised-studies-nrs-on-interventions/(Accessed on Nov 29, 2021).
Facey, K. M., Rannanheimo, P., Batchelor, L., Borchardt, M., and de Cock, J. (2020). Real-world Evidence to Support Payer/HTA Decisions about Highly Innovative Technologies in the EU-Actions for Stakeholders. Int. J. Technol. Assess. Health Care, 1–10. doi:10.1017/S026646232000063X
Fitth, (2018). Medical Devices [Internet]. European Medicines AgencyAvailable from: https://www.ema.europa.eu/en/human-regulatory/overview/medical-devices (Accessed on Dec 1, 2021).
Ford, I., and Norrie, J. (2016). Pragmatic Trials. N. Engl. J. Med. 375 (5), 454–463. doi:10.1056/NEJMra1510059
Fruergaard Jorgensen, L. (2017). Winning with GLP-1 [Internet]. Capital Markets Day Presented at: Capital Markets Day Novo Nordisk. Malov, Denmark: R&D campus Novo Nordisk. Available at: https://www.novonordisk.com/content/dam/Denmark/HQ/investors/irmaterial/cmd/2017/04_Winning%20with%20GLP-1.pdf (Accessed on Aug 19, 2021).
Garrison, L. P., Carlson, J. J., Bajaj, P. S., Towse, A., Neumann, P. J., Sullivan, S. D., et al. (2015). Private Sector Risk-Sharing Agreements in the United States: Trends, Barriers, and Prospects. Am. J. Manag. Care 21 (9), 632–640.
Ghabri, S., Lam, L., Bocquet, F., and Spath, H. M. (2020). Systematic Literature Review of Economic Evaluations of Biological Treatment Sequences for Patients with Moderate to Severe Rheumatoid Arthritis Previously Treated with Disease-Modifying Anti-rheumatic Drugs. PharmacoEconomics 38 (5), 459–471. doi:10.1007/s40273-020-00887-6
Gonzalez, J. C. (2020). Definining Real-World Evidence and Real-World Data [Internet]. Available at: https://konplik.health/a-definition-of-real-world-evidence/(Accessed on Nov 26, 2021).
Griffiths, E. A., Macaulay, R., Vadlamudi, N. K., Uddin, J., and Samuels, E. R. (2017). The Role of Noncomparative Evidence in Health Technology Assessment Decisions. Value Health 20 (10), 1245–1251. doi:10.1016/j.jval.2017.06.015
Guertin, J. R., Bowen, J. M., De Rose, G., O'Reilly, D. J., and Tarride, J. E. (2017). Illustration of the Impact of Unmeasured Confounding within an Economic Evaluation Based on Nonrandomized Data. MDM Pol. Pract 2 (1), 2381468317697711. doi:10.1177/2381468317697711
Hamers, P. A. H., Elferink, M. A. G., Stellato, R. K., Punt, C. J. A., May, A. M., Koopman, M., et al. (2021). Informing Metastatic Colorectal Cancer Patients by Quantifying Multiple Scenarios for Survival Time Based on Real-Life Data. Int. J. Cancer 148 (2), 296–306. doi:10.1002/ijc.33200
Health Technology Assessment, Nformal deal between Council and European Parliament [Internet]. Available at: https://www.consilium.europa.eu/en/press/press-releases/2021/06/22/health-technology-assessment-informal-deal-between-council-and-european-parliament/(Accessed on Nov 29, 2021).
History of EUnetHTA, (2021). EUnetHTA [Internet]. Available at: https://www.eunethta.eu/about-eunethta/history-of-eunethta/(Accessed on Dec 1, 2021).
Hogervorst, M. A., Vreman, R. A., Mantel-Teeuwisse, A. K., and Goettsch, W. G. (2021). Reported Challenges in Health Technology Assessment of Complex Health Technologies. Value Health [Internet]. Available at: https://www.sciencedirect.com/science/article/pii/S1098301521031879 (Accessed on Jan 19, 2021).
Home - GraphPad [Internet], . Available at: https://www.graphpad.com/(Accessed on Nov 30, 2021).
HTx, (2020). Project | Next Generation Health Technology Assessment [Internet]. Available at: https://www.htx-h2020.eu/(Accessed on Nov 29, 2021).
Imi GetReal, (2021). IMI GetReal - Real-Life Data in Drug Development [Internet]. IMI GetReal Initiative. Available from: https://www.imi-getreal.eu/(Accessed on Sep 19, 2021).
Imi Innovative Medicines Initiative Ehden, (2021). European Health Data and Evidence Network [Internet]. MI Innovative Medicines Initiative. Available at: http://www.imi.europa.eu/projects-results/project-factsheets/ehden (Accessed on Nov 29, 2021).
Jönsson, B., Hampson, G., Michaels, J., Towse, A., von der Schulenburg, J. G., and Wong, O. (2019). Advanced Therapy Medicinal Products and Health Technology Assessment Principles and Practices for Value-Based and Sustainable Healthcare. Eur. J. Health Econ. 20 (3), 427–438. doi:10.1007/s10198-018-1007-x
Justo, N., Espinoza, M. A., Ratto, B., Nicholson, M., Rosselli, D., Ovcinnikova, O., et al. (2019). Real-World Evidence in Healthcare Decision Making: Global Trends and Case Studies from Latin America. Value Health 22 (6), 739–749. doi:10.1016/j.jval.2019.01.014
Kesselheim, A. S., Wang, B., Franklin, J. M., and Darrow, J. J. (2015). Trends in Utilization of FDA Expedited Drug Development and Approval Programs, 1987-2014: Cohort Study. BMJ [Internet]. Available at: http://www.bmj.com/content/351/bmj.h4633 (Accessed on Aug 19, 2021).
Kimberlin, C. L., and Winterstein, A. G. (2008). Validity and Reliability of Measurement Instruments Used in Research. Am. J. Health Syst. Pharm. 65 (23), 2276–2284. doi:10.2146/ajhp070364
Larson, L., Walker, A., and Parow, D. (2017). Innovative Clinical Trial Designs Welcomed by Regulators but what about the Payers. [Internet]. Glasgow: ISPOR. [cited 2020 Apr 17]; Glasgow. Available at: https://www.ispor.org/docs/default-source/presentations/1330.pdf?sfvrsn=56ed4398_1.
Leyens, L., and Brand, A. (2016). Early Patient Access to Medicines: Health Technology Assessment Bodies Need to Catch up with New Marketing Authorization Methods. Public Health Genomics 19 (3), 187–191. doi:10.1159/000446537
Li, G., Sajobi, T. T., Menon, B. K., Korngut, L., Lowerison, M., James, M., et al. (2016). Registry-based Randomized Controlled Trials- what Are the Advantages, Challenges, and Areas for Future Research. J. Clin. Epidemiol. 80, 16–24. doi:10.1016/j.jclinepi.2016.08.003
Lipska, I., Hoekman, J., McAuslane, N., Leufkens, H. G., and Hövels, A. M. (2015). Does Conditional Approval for New Oncology Drugs in Europe lead to Differences in Health Technology Assessment Decisions. Clin. Pharmacol. Ther. 98 (5), 489–491. doi:10.1002/cpt.198
Lloyd-Williams, H., and Hughes, D. A. (2020). A Systematic Review of Economic Evaluations of Advanced Therapy Medicinal Products. Br. J. Clin. Pharmacol. 87 (6), 2428–2443. doi:10.1111/bcp.14275
Lou, J., Kc, S., Toh, K. Y., Dabak, S., Adler, A., Ahn, J., et al. (2020). Real-world Data for Health Technology Assessment for Reimbursement Decisions in Asia: Current Landscape and a Way Forward. Int. J. Technol. Assess. Health Care 36 (5), 474–480. doi:10.1017/S0266462320000628
Makady, A., de Boer, A., Hillege, H., Klungel, O., and Goettsch, W. (2017). What Is Real-World Data? A Review of Definitions Based on Literature and Stakeholder Interviews. Value Health 20 (7), 858–865. doi:10.1016/j.jval.2017.03.008
Makady, A., Ham, R. T., de Boer, A., Hillege, H., Klungel, O., Goettsch, W., et al. (2017). Policies for Use of Real-World Data in Health Technology Assessment (HTA): A Comparative Study of Six HTA Agencies. Value Health 20 (4), 520–532. doi:10.1016/j.jval.2016.12.003
Makady, A., Stegenga, H., Ciaglia, A., Debray, T. P., Lees, M., Happich, M., et al. (2017). Practical Implications of Using Real-World Evidence (RWE) in Comparative Effectiveness Research: Learnings from IMI-GetReal. J. Comp. Eff. Res. 6 (6), 485–490. doi:10.2217/cer-2017-0044
Makady, A., van Veelen, A., Jonsson, P., Moseley, O., D'Andon, A., de Boer, A., et al. (2018). Using Real-World Data in Health Technology Assessment (HTA) Practice: A Comparative Study of Five HTA Agencies. PharmacoEconomics 36 (3), 359–368. doi:10.1007/s40273-017-0596-z
Margaret, A., and Hamburg, M. D. (2013). Paving the Way for Personalized Medicine: FDA’s Role in a New Era of Medical Product Development. Silver Spring, Maryland: FDA (U.S. Food an Drug Administration). Available from: https://www.fdanews.com/ext/resources/files/10/10-28-13-Personalized-Medicine.pdf (Accessed October 2013).
Microsoft Excel, Spreadsheet Software, (2020). Excel Free Trial [Internet]. Available at: https://www.microsoft.com/nl-nl/microsoft-365/excel (Accessed on May 12, 2020).
Ministerie van VolksgezondheidWen, Regie Op Registers Voor Dure Geneesmiddelen (ROR DGM) - Werkagenda - Zorginstituut Nederland [Internet]. Ministerie van Volksgezondheid, Welzijn en Sport; 2019Available at: https://www.zorginstituutnederland.nl/werkagenda/regie-op-registers-voor-dure-geneesmiddelen (Accessed on Nov 29, 2021).
Moloney, R., Mohr, P., Hawe, E., Shah, K., Garau, M., and Towse, A. (2015). Payer Perspectives on Future Acceptability of Comparative Effectiveness and Relative Effectiveness Research. Int. J. Technol. Assess. Health Care 31 (1–2), 90–98. doi:10.1017/S0266462315000203
Moseley, J., Vamvakas, S., Berntgen, M., Cave, A., Kurz, X., Arlett, P., et al. (2020). Regulatory and Health Technology Assessment Advice on Postlicensing and Postlaunch Evidence Generation Is a Foundation for Lifecycle Data Collection for Medicines. Br. J. Clin. Pharmacol. 86 (6), 1034–1051. doi:10.1111/bcp.14279
The National Patient Register, (2021). Socialstyrelsen. Accessed on 2021 Nov 29]. Available at: https://www.socialstyrelsen.se/en/statistics-and-data/registers/register-information/the-national-patient-register/
Orsini, L. S., Berger, M., Crown, W., Daniel, G., Eichler, H. G., Goettsch, W., et al. (2020). Improving Transparency to Build Trust in Real-World Secondary Data Studies for Hypothesis Testing-Why, what, and How: Recommendations and a Road Map from the Real-World Evidence Transparency Initiative. Value Health 23 (9), 1128–1136. doi:10.1016/j.jval.2020.04.002
Pallmann, P., Bedding, A. W., Choodari-Oskooei, B., Dimairo, M., Flight, L., Hampson, L. V., et al. (2018). Adaptive Designs in Clinical Trials: Why Use Them, and How to Run and Report Them. BMC Med. 16 (1), 29. doi:10.1186/s12916-018-1017-7
Parody-Rúa, E., Rubio-Valera, M., Guevara-Cuellar, C., Gómez-Lumbreras, A., Casajuana-Closas, M., Carbonell-Duacastella, C., et al. (2020). Economic Evaluations Informed Exclusively by Real World Data: A Systematic Review. Int. J. Environ. Res. Public Health 17 (4).
Pruce, D., Michel, S., and Theocharous, P. (2017). Do Payers Find Value in Innovative Trial Designs, 3.
Public Policy Committee, International Society of Pharmacoepidemiology, (2016). Guidelines for Good Pharmacoepidemiology Practice (GPP). Pharmacoepidemiol. Drug Saf. 25 (1), 2–10. doi:10.1002/pds.3891
Ramagopalan, S. V., Simpson, A., and Sammon, C. (2020). Can Real-World Data Really Replace Randomised Clinical Trials. BMC Med. 18 (1), 13. doi:10.1186/s12916-019-1481-8
Research C for De, (2020). Best Practices for Conducting and Reporting Pharmacoepidemiologic Safety Studies Using Electronic Healthcare Data Sets. Silver Spring, Maryland: U.S. Food and Drug Administration. FDA. [Internet]Available from: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/best-practices-conducting-and-reporting-pharmacoepidemiologic-safety-studies-using-electronic (Accessed on Aug 20, 2021).
Ribeiro, T. B., Buss, L., Wayant, C., and Nobre, M. R. C. (2020). Comparison of FDA Accelerated vs Regular Pathway Approvals for Lung Cancer Treatments between 2006 and 2018. PLoS One 15 (7), e0236345. doi:10.1371/journal.pone.0236345
Schmitz, C. (2020). LimeSurvey [Internet]. LimeSurvey - the Free & Open Source Survey Software Tool. Available at: https://www.limesurvey.org/(Accessed on May 12, 2021).
Schünemann, H. J. (2019). All Evidence Is Real World Evidence [Internet]. The BMJ[Available at: https://blogs.bmj.com/bmj/2019/03/29/holger-j-schunemann-all-evidence-is-real-world-evidence/(Accessed on Nov 26, 2021).
Selker, H. P., Eichler, H. G., Stockbridge, N. L., McElwee, N. E., Dere, W. H., Cohen, T., et al. (2019). Efficacy and Effectiveness Too Trials: Clinical Trial Designs to Generate Evidence on Efficacy and on Effectiveness in Wide Practice. Clin. Pharmacol. Ther. 105 (4), 857–866. doi:10.1002/cpt.1347
Selker, H. P., Oye, K. A., Eichler, H. G., Stockbridge, N. L., Mehta, C. R., Kaitin, K. I., et al. (2014). A Proposal for Integrated Efficacy-To-Effectiveness (E2E) Clinical Trials. Clin. Pharmacol. Ther. 95 (2), 147–153. doi:10.1038/clpt.2013.177
Venkitachalam, R. (2021). Validity and Reliability of Questionnaires [Internet]. Healthcare presented at; 04:23:11 UTCAvailable from: https://www.slideshare.net/Venkitachalam/validity-and-reliability-of-questionnaires (Accessed on Nov 29, 2021).
Vreman, R. A., Bouvy, J. C., Bloem, L. T., Hövels, A. M., Mantel-Teeuwisse, A. K., Leufkens, H. G. M., et al. (2019). Weighing of Evidence by Health Technology Assessment Bodies: Retrospective Study of Reimbursement Recommendations for Conditionally Approved Drugs. Clin. Pharmacol. Ther. 105 (3), 684–691. doi:10.1002/cpt.1251
Vreman, R. A., Leufkens, H. G. M., and Kesselheim, A. S. (2020). Getting the Right Evidence after Drug Approval. Front. Pharmacol. Available at: https://www.frontiersin.org/articles/10.3389/fphar.2020.569535/full. doi:10.3389/fphar.2020.569535
Vreman, R. A., Naci, H., Goettsch, W. G., Mantel-Teeuwisse, A. K., Schneeweiss, S. G., Leufkens, H. G. M., et al. (2020). Decision Making under Uncertainty: Comparing Regulatory and Health Technology Assessment Reviews of Medicines in the United States and Europe. Clin. Pharmacol. Ther. 108 (2), 350–357. doi:10.1002/cpt.1835
Keywords: real-world data, health technology assessment, complex health technologies, Europe, questionnaire
Citation: Hogervorst MA, Pontén J, Vreman RA, Mantel-Teeuwisse AK and Goettsch WG (2022) Real World Data in Health Technology Assessment of Complex Health Technologies. Front. Pharmacol. 13:837302. doi: 10.3389/fphar.2022.837302
Received: 17 December 2021; Accepted: 24 January 2022;
Published: 10 February 2022.
Edited by:
Mina Tadrous, Women’s College Hospital, CanadaReviewed by:
Linda Mundy, University of Adelaide, AustraliaDaniela Oliveira De Melo, Federal University of São Paulo, Brazil
Copyright © 2022 Hogervorst, Pontén, Vreman, Mantel-Teeuwisse and Goettsch. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Wim G. Goettsch, w.g.goettsch@uu.nl