Ivett Jakab1*
Bertalan Németh1
Baher Elezbawy2Melis Almula Karadayı3
Hakan Tozan3Sabahattin Aydın3Jie Shen4
Zoltán Kaló1,5- 1Syreon Research Institute, Budapest, Hungary
- 2Syreon Middle East, Alexandria, Egypt
- 3İstanbul Medipol University, İstanbul, Turkey
- 4Novartis International AG, Basel, Switzerland
- 5Center for Health Technology Assessment, Semmelweis University, Budapest, Hungary
Background: Multicriteria Decision Analysis (MCDA), a formal decision support framework, has been growing in popularity recently in the field of health care. MCDA can support pricing and reimbursement decisions on the macro level, which is of great importance especially in countries with more limited resources.
Objectives: The aim of this systematic review was to facilitate the development of future MCDA frameworks, by proposing a set of criteria focusing on the purchasing decisions of single-source innovative pharmaceuticals in upper middle-income countries.
Methods: A systematic literature review was conducted on the decision criteria included in value frameworks (VFs) or MCDA tools. Scopus, Medline, databases of universities, websites of Health Technology Assessment Agencies, and other relevant organizations were included in the search. Double title-abstract screening and double full-text review were conducted, and all extracted data were double-checked. A team of researchers performed the merging and selection process of the extracted criteria.
Results: A total of 1,878 articles entered the title and abstract screening. From these, 341 were eligible to the full-text review, and 36 were included in the final data extraction phase. From these articles 394 criteria were extracted in total. After deduplication and clustering, 26 different criteria were identified. After the merging and selection process, a set of 16 general criteria was proposed.
Conclusion: Based on the results of the systematic literature review, a pool of 16 criteria was selected. This can serve as a starting point for constructing MCDA frameworks in upper middle-income countries after careful adaptation to the local context.
Introduction
The Need for Complex Value Assessment in HTA
Resource allocation in health care is a complex process, economic constraints in all countries necessitate a rational priority setting and a transparent decision-making framework (Baltussen and Niessen, 2006). This need called Health Technology Assessment (HTA) into being, a multidisciplinary field of policy analysis using explicit analytical frameworks, with the main purpose to inform technology-related policymaking in health care (Facey et al., 2006). Essentially, HTA aims to provide the best available information regarding various health technologies in a variety of settings (Velasco-Garrido and Busse, 2005; Grutters et al., 2011).
Another key aspect of health care decision making, in line with the multidisciplinary nature of HTA, is that it is not based on a single criterion, but on a set of different criteria, like efficacy or effectiveness, safety, cost-effectiveness, equity or fairness, etc. (Guindo et al., 2012) that are often in conflict with one another (Keeney and Raiffa, 1993). Lack of awareness of the value elements considered across the decision-making continuum as well as lack of transparency in decision making can potentially create tension amongst stakeholders (Tanios et al., 2013). As a result, more and more organizations are proposing and implementing a “value assessment framework” (or “value framework” (VF) for short) for aiding health care decisions with a clear list of criteria that has to be taken into account (Neumann et al., 2018). It has also been a generally accepted principle that health care decision‐making should rely on strong scientific evidence (Rudmik and Drummond, 2013), which called the need for a scientifically justified method to evaluate more than one aspect of the decision-making problem at once. When criteria of a value framework are weighted as opposed to each other and given a specific measurement method (so-called scoring function), of which a summary of scores can be calculated, the framework is called a Multi-Criteria Decision Analysis (MCDA) or a Multiple-Criteria Decision Making (MCDM) tool.
What Is MCDA?
MCDA is the collective term for those formal decision support approaches, that take multiple criteria into account in an explicit way (Belton and Stewart, 2002). MCDA has already been used in various fields of science, and it has been gaining popularity recently in the field of health care decision making as well (Marsh et al., 2014; Antioch et al., 2017). Key elements of the MCDA structure are: 1) clearly defined and well-structured criteria to assess the alternatives that are being evaluated in the particular decision setting, 2) weights to express the differences in relative importance of various criteria, and 3) the aforementioned scoring functions for converting performance measurements of the analyzed alternatives into scores that can be aggregated to inform the decision-makers on the overall performance of the alternatives (Thokala et al., 2016).
Potential of MCDA Use in Health Care
MCDA can be used on all levels of decision making in health care. At micro-level decision making when choosing the treatment to be administered to a certain patient, MCDA can be especially helpful for involving the patient’s voice into his/her own treatment decision (Marsh et al., 2017) through shared decision making [e.g. American Society of Clinical Oncology (ASCO) framework (Cherny et al., 2018)]. The mezzo, clinical guideline or hospital-level decision making can also be guided by MCDAs, for example during the development of the hospital formulary list or a new clinical guideline. Last but not least MCDA can be of great help at macro level in structuring risk-benefit assessment to support market authorization decisions (Angelis and Phillips, 2020) or informing reimbursement decisions for different health technologies, health care programs or health priorities (e.g. EVIDEM [Evidence and Value: Impact on Decision Making) framework (Goetghebeur et al., 2008); The Advance Value Framework (Angelis and Kanavos, 2017)].
In general, an MCDA tool can have two potential roles in reimbursement decisions. If an explicit threshold of MCDA final scores is used as a hard decision rule, MCDAs can be used as independent decision-making tools. For example, if the total attainable score of the MCDA is 100 points, a threshold at 60 points can be set above which all assessed health technologies can receive reimbursed status. The other general approach is to use the MCDA framework in a supportive role, without being the single instrument that guides the final decision. This way, MCDA frameworks can be used as part of deliberate processes (Baltussen et al., 2017), or as a way of conducting so-called augmented cost-effectiveness analyses (Garrison et al., 2018).
The International Society of Pharmacoeconomics and Outcomes Research (ISPOR) had a task force dedicated to the topic of MCDA and published two comprehensive reports (Marsh et al., 2016; Thokala et al., 2016) guiding the development of such tools. Wide MCDA-related research has been conducted both regarding the theoretical side, like reviewing the various approaches adopted in health care MCDAs (Marsh et al., 2014), and the methodological side, for example on the selection of the optimal criteria weighting methods (Németh et al., 2019). Some examples have also been published on the practical issues of using MCDA in real-life decision settings, from applying the EVIDEM framework in Lombardy, Italy (Radaelli et al., 2014), through assessing the value of hospital technologies in Hungary (Endrei et al., 2014) or improving the current procurement framework for off-patent pharmaceuticals in Indonesia (Inotai et al., 2018a). Above all, the usefulness of incorporating MCDA in HTA to support transparent and systematic appraisal of health care interventions has been demonstrated in a proof-of-concept study (Goetghebeur et al., 2012). Also, it has been shown that MCDA integrated into HTA allows a consistent approach for appraising health care interventions, by promoting systematic consideration of all decision criteria and the underlying evidence (Tony et al., 2011).
HTA in Upper Middle-Income Countries
The level of HTA used in reimbursement decisions is uneven throughout countries, often being less developed in upper-middle-income countries compared to high-income ones. Nonetheless, countries with limited resources can spend a lower amount of monetary resources on health care compared to high-income countries, making their allocative decisions more influential for the whole system (Kaló et al., 2016). This situation is often troubled with a worse general health status of the population (Boncz et al., 2014), and insufficient capacity of HTA expertise (Kaló et al., 2013), therefore the need for and the challenges of implementing HTA for health care technologies are even greater in these countries (Sorenson et al., 2009; Inotai et al., 2012). Application of an MCDA tool adapted to local circumstances can be a good option for overcoming these challenges for upper-middle-income countries, especially for innovative, usually expensive pharmaceuticals.
In this study, we aimed to facilitate the development of such transparent decision-making processes. The main goal of our research was to provide a wide set of criteria based on a systematic literature review to assist the development of future MCDA tools in upper-middle-income countries for purchasing decisions of single-source innovative pharmaceuticals. Authors believe that the proposed criteria set can be a potential starting point for national adaptation in individual countries, which necessitates conclusion on how many criteria are applied at the national level and how the weight of each criterion is determined to reflect priorities of national health policies.
Methods
Sources of Information
A systematic literature review was conducted to identify value framework and MCDA articles listing criteria relevant for reimbursement level decision making of pharmaceuticals. The systematic literature review covered Scopus and Medline (via PubMed) databases. It was extended with a targeted literature review on specific databases of universities (Centre for Reviews and Dissemination (CRD) database; University of York, England; University of British Columbia, Canada) and websites of Health Technology Assessment Agencies (Independent Institute for Quality and Efficiency in Health Care (IQWIG), Germany; Agency for Health Quality and Assessment of Catalonia (AQUAS), Spain; Swedish Agency For Health Technology Assessment and Assessment Of Social Services (SBU), Sweden; National Institute for Health and Care Excellence (NICE), England; Scottish Medicines Consortium (SMC), Scotland; French National Authority for Health (HAS), France; Canadian Agency for Drugs and Technologies in Health (CADTH), Canada). Additionally, the following organizations’ websites were searched for eligible VFs and MCDAs: World Health Organisation (WHO) Health Evidence Network database; Institute for Clinical and Economic Review (ICER); European Union Projects (The Seventh Framework Programme (FP7), Horizon 2020 Programme, Innovative Medicines Initiative Joint Action Programme 2 (IMI2)); European Network for Health Technology Assessment (EUnetHTA); Health Technology Assessment International (HTAi); Innovation and Value Initiative (IVI); International Society for Pharmacoeconomics and Outcomes Research (ISPOR); European Patients’ Forum (EPF); European Federation of Pharmaceutical Industries and Associations (EFPIA); Pharmaceutical Research and Manufacturers of America (PhRMA); and the Society for Medical Decision Making. Registries of systematic literature reviews were also searched through PROSPERO and Cochrane Reviews databases.
Search Limitations and Terms
The literature search was performed on the 28th of March 2019 for all databases. All articles on MCDA tools or VFs relevant for reimbursement level decision making on pharmaceuticals, listing a clear set of criteria and published since January 2013 were included. A restriction of publication date was used as the field of value frameworks has evolved remarkably in the past 5 years, and we wished to focus on more recent approaches, to capture relevant value elements in the present and near future. The search was also limited to English-language papers.
The first part of the search term was a combination of domains relevant to MCDA or VFs, in addition to the second term related to health care or pharmaceuticals. Synonyms and MeSH Terms of these two domains were used to conduct a comprehensive search in each database. The detailed search syntax is included in Supplementary Figure 1.
Screening Phase
After deduplication, title and abstract screening and full-text screening were performed by two independent researchers. Disagreements between researchers were resolved by a principal researcher.
During the title and abstract screening phase, articles were excluded based on the following hierarchy: 1) Irrelevant title and no English abstract; 2) Not related to MCDA/VF OR MCDA/VF in other fields than human health care; 3) MCDA/VF in other fields than pharmaceuticals. Articles not fitting any of these exclusion criteria were eligible for full-text screening.
During full-text screening, articles were excluded if they fit in any of the following exclusion criteria: 1) Full text not found; 2) Abstract or commentary; 3) Duplicate; 4) Not English; 5) Non-pharmaceutical focus; 6) Not listing a clear set of criteria (e.g. purely methodological); 7) MCDA/VF supporting patient-level or shared decision making; 8) MCDA/VF supporting clinical guideline level/hospital level decision making; 9) Not original paper (e.g. framework already included in the review without major changes in criteria; literature review) or 10) Other specified reasons. Articles that did not fit any of the previous exclusion criteria were included in the data extraction phase. Other systematic literature reviews with a similar scope to this study were excluded by exclusion criterion 6, but were kept in a separate folder for snowball search.
Data Extraction Phase
To systematically assess the findings, a standardized data extraction form was developed. The data extraction table and categorization principles for criteria were revised after a pilot data extraction phase. Qualitative research synthesis was used for presenting information, per the type of data. All criteria were extracted and clustered into seven predefined categories (disease-related; treatment-related; economic; societal; uniqueness and complexity; patient experience; other). All criteria related to the severity of disease, size of affected population, public health priorities or unmet need were considered disease-related. Treatment-related criteria were categorized as relating efficacy and/or safety of the treatment, or the strength of evidence of those. Criteria were considered as economic if they related to either the cost-effectiveness, budget impact, sustainability or potential for use outside of the reimbursed indication of the treatment. Additionally, organizational aspects of the assessed technology and the local investments of manufacturers were also considered as economic criteria. Societal criteria included equity aspects, prevention, productivity, caregiver burden, and legal aspects. Specific criteria for innovative technologies were categorized as uniqueness and complexity, including the innovative profile of the treatment and the complexity of manufacturing. Those criteria directly affecting patients’ experience other than those traditionally assessed in HTA (e.g. efficacy and safety) were categorized as patient experience. These criteria included patient convenience, adherence, and persistence, as well as value-added services.
The following information was extracted from each included study hence framework: 1) first author and year of publication; 2) country/region of (proposed) implementation; 3) organization developing the framework; 4) objective of the framework; 5) scope of use (e.g. general on all health technologies, oncologic drugs, orphan drugs); 6) name of criteria listed; 7) definition of each criterion, if available; 8) categorization of each criterion by the data extractor (into one of the 7 predefined categories); 9) scoring function of each criterion if available (which determines how a technology will be evaluated according to a particular criterion).
All extracted data were double-checked by a researcher other than the data extractor. Snowball search was performed to identify further relevant studies in included articles and in systematic literature reviews identified throughout the screening phases. When more than one article discussed the same framework, only the most recent article was included in the data extraction. Although, different adaptations of the same framework were included if criteria varied significantly.
Quality Assessment
Identified frameworks were not assessed for quality, as the aim of the research was to collect potential value elements from recently published frameworks not to evaluate the current practice of the field.
Merging and Selection Process of Criteria
When the data extraction phase was over, the merging and selection process of extracted criteria has started. First, criteria were deduplicated that of synonyms and substantially similar concepts. Then a multi-stakeholder team of experts went through all remaining criteria by the predefined categories in three rounds of workshops. Stakeholders involved were academics (2), HTA doers (2) and patient representatives (2) (Table 1).
Table 1 Detailed profiles stakeholders involved in the selection and merging process of criteria (n = 6).
The merging and selection process was conducted in compliance with the ISPOR MCDA Emerging Good Practices Task Force report’s principles: completeness, non-redundancy, nonoverlaps, and preference independence (Marsh et al., 2016).
Results
Results of the Systematic Literature Review
After the deduplication of results, a total of 1824 articles were identified through literature searches. The targeted literature search and the snowball method detected an additional 54 new records. Altogether, titles and abstracts of 1,878 records were screened from which 341 were eligible for the full-text screening phase. In total, 35 articles were considered to be eligible for the qualitative synthesis and were included in the data extraction phase (Figure 1).
Figure 1 PRISMA flow diagram.
A total of 37 different frameworks—MCDAs and VFs—were identified in the 35 articles. Based on the current World Bank Country and Lending rules for the 2020 fiscal year (World Bank Country and Lending Groups Country Classification, 2020), 9 were categorized as developed specifically for an upper-middle-income country. All others were meant to be implemented in a high-income country or were not specified other than continent or region (e.g. Europe, Latin-America, Central-Eastern Europe). Thirteen of the 37 had a general focus on all health technologies, 9 on medicines in general, while 6, 5 and 4 frameworks focused on orphan drugs, oncologic drugs, and off-patent drugs, respectively. The number of criteria listed ranged between 5 and 28, with an average of 11.3. About half of the frameworks (n=18) had explicit definitions of criteria listed, others published only the names of criteria. More than a third of the frameworks (n=15) had scoring functions, of which two had the same general scoring function proposed for all their criteria (e.g. 4-point scale (0–3)). About a third of the frameworks (n=14) had neither definitions nor scoring functions published for their criteria. Regarding the categories of criteria, all of them contained at least one treatment-related criterion. Economic and disease-related criteria were the second and third most prevalent, 86% (n = 32) and 81% (n = 30) of frameworks contained at least one of them, respectively. Fifty-nine percent (n=22) of value frameworks contained criteria categorized as societal, whereas 38% (n = 14) and 32% (n = 12) listed at least one criterion concerning the patient experience and the uniqueness and complexity of the treatment, respectively (Table 2).
Table 2 Summary table on the key results of the systematic literature review.
Results of the Merging and Selection Process of Criteria
In total 419 criteria were extracted. After deduplication and clustering of similar criteria by each category, 26 criteria remained (Figure 2). These were the following: 1) disease-related criteria: severity of disease; size of affected population; unmet need; public health priority; 2) treatment-related criteria: efficacy; safety; strength of evidence; 3) economic criteria: cost; cost-effectiveness; budget impact; capacity; supply track record; potential use outside of the reimbursed indication(s); Local investment; 4) societal criteria: equity; prevention; productivity; caregiver burden; 5) uniqueness and complexity of treatment criteria: innovative profile of the treatment; manufacturing complexity; number of indications; spill-over effect; formulation; 6) patient experience criteria: patient convenience; patient adherence; value-added services.
Figure 2 Flow of criteria through different phases of the foundation work and the recommended national adaption.
Throughout the expert workshops, these 26 criteria were further merged to reduce overlaps and minimize the number of criteria proposed as possible. Criterions “efficacy” and “safety” were merged into a new criterion named “health gain.” The criterion called “size of affected population” was merged into criterion “equity,” thereby rare disease patients got listed as one aspect of the neglected populations. Meanwhile, the geographical access aspect of criterion “equity” (e.g. access for patients living in rural areas) was merged into criterion “availability and accessibility of alternative therapies.” Criterion “prevention” was merged into criterion “public health priority” as a potential focus of local public health policies, and it is highlighted as an example in the description of the new criterion. Likewise, the criterion concerning “caregiver burden” was merged into criterion “productivity,” where thereby the productivity loss of households, including both patients and caregivers, is taken into account. Also, the criterion “patient convenience” was merged into criterion “patient adherence and persistence” to reduce overlaps of capturing both effects and potential causes. Criterion “cost” was excluded because overlaps with criterion “cost-effectiveness” and criterion “capacity” was excluded because it is more health system-related than treatment-related. Criterion “number of indications,” “spill-over effect,” and “formulation” were excluded based on expert suggestions, to keep the initial set of criteria as curtail and condensed as possible. The expert workshops also explored potential incentives for the use of each criterion and proposed descriptions and examples of scoring functions. The final set of 16 criteria proposed for local adaptation is shown in Table 3.
Table 3 The proposed set of criteria with descriptions, potential incentives for inclusion and proposed scoring functions.
Discussion
To the best of our knowledge, this is the first publication supporting the creation of MCDA for the reimbursement of innovative pharmaceuticals in upper-middle-income countries. Similar publications identified throughout our systematic literature review were a systematic review on the use of methodological frameworks to set health care priorities in low-income and lower middle-income countries (Wiseman et al., 2016), MCDA tools specific to one certain upper-middle-income country with either a specific or a general scope (Iskrov et al., 2013; Ramli et al., 2013; Youngkong, 2014; Hu et al., 2015; Iskrov and Stefanov, 2016; Jaramillo et al., 2016; Radu et al., 2016) and an MCDA tool for emerging countries focusing on the evaluation of off-patent pharmaceuticals (Brixner et al., 2017).
The proposed set of 16 criteria provides a comprehensive initial framework that can be adapted to upper-middle-income countries’ circumstances based on local priorities. As the literature review indicated, there is increasing attention to criteria concerning patient experience. To emphasize its significance, two independent criteria are proposed to capture this emerging aspect. Criteria concerning the uniqueness and complexity of the therapy were included as important aspects of innovative medicines, especially in the light of recently launched genetic therapies.
The merging and selection process aimed to minimize the number and magnitude of overlaps in the framework to prevent double-counting. However, some overlap is inevitable in any MCDA. For example, a potential remaining overlap might be between criteria “patient adherence and persistence” and “value added services”. Even though value-added services can potentially improve patient adherence and persistence, the importance of these services should be emphasized.
The core of an MCDA decision-supporting tool for repeated use is in its availability of increasing the transparency of decision making. When MCDAs are published, but without the detailed definitions and scoring functions of the criteria they may fail to fulfill this purpose. With almost half of the identified frameworks not publishing either definitions or scoring functions, we thought it was an absolute must to propose one for each criterion, even if they can be further modified during local adoption. This approach hopefully further supports the interpretation thereby the implementation of our work.
Previous Systematic Literature Reviews With a Similar Scope
Throughout the literature review, nine previous systematic literature reviews of MCDAs and VFs were identified. The most recent by Kolasa et al. present the then-current state of knowledge concerning the MCDA utilization in the value assessment of drug therapies. They identified 18 MCDAs in April 2017 concerning pricing and reimbursement level decision making of pharmaceuticals, similarly to the present review. Kolasa and colleagues revealed several shortcomings in the implementation of MCDA to health care decision–making, like the lack of consistency of approaches chosen toward the type and the number of stakeholders involved in the MCDA experiments. Similarly to our findings, they also noted that none of the reviewed studies have reported how MCDA results impacted the real-life settings, and that examples of MCDA’s successful launch in a real-life decision-making process are genuinely scarce (Kolasa et al., 2018). Another recent systematic review by Frazão and colleagues aimed to analyze and synthesize articles found in the literature on applying MCDA to health care while assessing general issues and methodological aspects. They analyzed the trends in MCDA publications in addition to the types of sources and methods used to develop particular parts of the MCDAs. Based on their review, they identified a growing trend in the application of MCDA in the field of health, with great emphasis from the year 2014 (Frazão et al., 2018). This finding is in line with our approach of restricting the timeframe of our research to 2013 and beyond. Another systematic review aimed to summarize the evaluation criteria of orphan medicines with a regional scope, particularly for Central and Eastern European countries (Zelei et al., 2016). The authors concluded that due to external price referencing of pharmaceuticals, the relative budget impact of orphan drugs is expected to be higher in CEE than in Western European (WE) countries unless accessibility of patients remains more limited in poorer European regions. This observation may apply for other innovative medicines and upper middle-income countries too, further strengthening the need for locally adopted MCDA tools.
Recommendations for Adapting the Study Results
The maximum number of criteria in an MCDA is recommended to set around 10, otherwise too many criteria might mean too little weight for each criterion (Inotai et al., 2018b). Hence, we suggest selecting approximately 10 of our 16 proposed criteria during the adaptation period, based on local priorities (Figure 2). Health care system structure, decision-making framework, the health status of the population, financial backgrounds, and several other key factors can differ significantly across countries, making transferability a key issue in health care decision-making (Goeree et al., 2011). Just like reimbursement recommendations of HTA bodies cannot be used for decision making in other countries without considering transferability (Kaló et al., 2012), the same applies for constructing MCDAs. It has been strongly recommended that MCDA frameworks should only contain locally relevant criteria, taking into close consideration local feasibility, as local experts are always more familiar with specific problems, health care priorities and processes of the particular country than international experts. Furthermore, the development of a new MCDA should be based on the decision-making criteria that are currently used in particular countries (Inotai et al., 2018b). Some steps of MCDA development identified by the second ISPOR Task Force report (Marsh et al., 2016) can be conducted by a small number of experts and policymakers (e.g. defining the decision problem), while others (e.g., selecting and structuring criteria) necessitate a wider, multi-stakeholder approach. Incorporating the preferences of various stakeholders can be conducted at specifically designed workshops (Inotai et al., 2018b), though the results of previously conducted desk research can be a good starting point in all cases. Our research aims to provide a background for such future implementations.
Potential Limitations of MCDA Use
Like all methodologies, MCDA has potential limitations. A scoping review of different priority setting approaches found that while the intention of developing such tools is for them to eventually be used to guide routine policymaking, not many have been integrated into routine practice. Cited limitations included the technical complexity of the approaches and resource demand (Kapiriri and Razavi, 2017). The lack of and need for publishing the real-word experience of using MCDAs has been highlighted by other researchers (Kapiriri and Razavi, 2017; Kolasa et al., 2018) and it is in line with our concerns as well. Such publicly available information can be crucial in supporting the implementation of MCDA in more countries and should be highly encouraged to publish. Another example of potential limitations of MCDA that has been discussed arises when incorporating cost-based criteria. Critics believe that the preferences of technology providers or users may not be the appropriate basis for evaluating the efficiency of technologies as the benefit that may be forgone is more prevalent in their thinking than the willingness to pay (WTP) of third-party payers. And even if the exercise aims to find the WTP with broad and imprecise definitions of criteria, its appropriateness can be questionable (Marsh et al., 2018). Another critique of the stakeholder involvement in the development of MCDAs is that when reviewed, there were no common patterns in the types and the size of the study population of involved stakeholders (Kolasa et al., 2018). Key issues of real-world feasibility may also emerge from the inappropriate selection of an analysis method, wrong interpretation of results, and using software solutions that are not widely available, as the results of the criteria ranking can depend on the various MCDA models that are being used (Kujawski, 2003). This is part of the reason why MCDA and its development method should be simple and easy to understand (Inotai et al., 2018b), and the trade-off between scientific accuracy and resource intensity and general participant burden should be taken into account in case of choosing between criteria weighting methods (Németh et al., 2019). To interpret the results of an MCDA evaluation easier, Angelis and Kanavos suggested a framework in which the aggregate value score (that does not include cost-related criteria) produced by the MCDA process would be the benefit component. The so-called incremental cost value ratio (ICVR), the purchasing cost per incremental MCDA value unit gained could be used similarly as the incremental cost-effectiveness ratio (ICER) is applied in several countries (Angelis and Kanavos, 2016).
Limitations of the Study
Limitations of the present research include, that only published sets of criteria were taken into account. Therefore, the criteria being used in confidential decision-making frameworks could not be incorporated into the analysis. In addition, the categorization of criteria was conducted based on personal judgments of researchers. We tried to minimize the personal variability by 1) creating an objective categorization guideline during the pilot data extraction phase and 2) by categorizing all criteria by two independent researchers and resolving conflicts by a principal researcher. A general limitation was, that a significant amount of frameworks (n=14) had nether descriptions, nor scoring functions published for their criteria listed as value elements. Therefore, the precise interpretation of certain criteria gathered from the literature was challenging and forms a source of uncertainty. Finally, as MCDA frameworks should always be adapted to the local environment and the particular decision problem, it is possible that some special cases may require additional specific criteria. Therefore, our set of 16 criteria should be used as a starting point only.
Future research can focus on MCDAs developed for other fields of health care and other decision problems. Since the theoretical side of MCDA has been relatively well-researched during the last few decades, more emphasis could be put on the practical side of MCDA implementation, based on real-life examples.
Conclusion
Based on the results of the systematic literature review, we established a pool of 16 criteria that can serve as a basis for constructing MCDAs to inform reimbursement decision making of innovative pharmaceuticals, especially in upper middle-income countries. The feasibility of implementation and adaptation to the local context are key aspects that should be taken into account in all cases. When adapting to the circumstances of a certain upper-middle-income country, we suggest setting up a group of local stakeholders to select and weigh not more than 10 of the 16 criteria based on local priorities. Well-defined descriptions and scoring functions of the criteria are inevitable for objective and transparent decision making. Further research is needed on the real-life examples of MCDAs as literature on the experiences after MCDA implementation is scarce and incomplete.
Author Contributions
Conception and design: ZK, BN, IJ, HT, JS. Analysis and interpretation of data: ZK, BN, IJ, BE, HT, MK. Drafting of the manuscript: IJ, BN, BE. Critical revision of the manuscript for important intellectual content: ZK, MK, HK, SA. Supervision: JS, ZK, SA.
Funding
The study was funded by Novartis International AG.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
This work was supported in part by a contract with Novartis International AG. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report. The content of the paper and the conclusions are those of each author and cannot be understood as reflecting those of the organization that employs them.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fphar.2020.01203/full#supplementary-material
References
Angelis, A., Kanavos, P. (2016). Value-based assessment of new medical technologies: towards a robust methodological framework for the application of multiple criteria decision analysis in the context of health technology assessment. Pharmacoeconomics 34 (5), 435–446. doi: 10.1007/s40273-015-0370-z
Angelis, A., Kanavos, P. (2017). Multiple criteria decision analysis (MCDA) for evaluating new medicines in health technology assessment and beyond: the Advance Value Framework. Soc. Sci. Med. 188, 137–156. doi: 10.1016/j.socscimed.2017.06.024
Angelis, A., Phillips, L. D. (2020). Advancing structured decision-making in drug regulation at the FDA and EMA. Br. J. Clin. Pharmacol. 2020, 1–11. doi: 10.1111/bcp.14425
Annemans, L., Aymé, S., Le Cam, Y., Facey, K., Gunther, P., Nicod, E., et al. (2017). Recommendations from the European working Group for Value Assessment and Funding Processes in rare diseases (ORPH-VAL). Orphanet. J. Rare Dis. 12 (1), 50. doi: 10.1186/s13023-017-0601-9
Antioch, K. M., Drummond, M. F., Niessen, L. W., Vondeling, H. (2017). International lessons in new methods for grading and integrating cost effectiveness evidence into clinical practice guidelines. Cost Effect. Res. Allocation 15 (1), 1. doi: 10.1186/s12962-017-0063-x
Baltussen, R., Niessen, L. (2006). Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Effect. Resource Allocation 4 (1), 14. doi: 10.1186/1478-7547-4-14
Baltussen, R., Jansen, M. P. M., Bijlmakers, L., Grutters, J., Kluytmans, A., Reuzel, R. P., et al. (2017). Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes. Value in Health 20 (2), 256–260. doi: 10.1016/j.jval.2016.11.019
Belton, V., Stewart, T. (2002). Multiple criteria decision analysis: an integrated approach (Norwell, Massachussets, US; Dordrecht, The Netherlands: Kluwer Academic Publishers).
Boncz, I., Vajda, R., Ágoston, I., Endrei, D., Sebestyén, A. (2014). Changes in the health status of the population of Central and Eastern European countries between 1990 and 2010. Eur. J. Health Econ. 15 (1), 137–141. doi: 10.1007/s10198-014-0602-8
Brixner, D., Maniadakis, N., Kaló, Z., Hu, S., Shen, J., Wijaya, K. (2017). Applying multi-criteria decision analysis (MCDA) simple scoring as an evidence-based HTA methodology for evaluating off-patent pharmaceuticals (OPPs) in emerging markets. Value Health Reg. Issues 13, 1–6. doi: 10.1016/j.vhri.2017.02.001
Canadian Agency for Drugs Technologies in Health (CADTH) (2016). pCODR Expert Review Committee Deliberative Framework 16 March 2019) (CADTH). Available from: https://www.cadth.ca/sites/default/files/pcodr/The%20pCODR%20Expert%20Review%20Committee%20%28pERC%29/pcodr_perc_deliberative_frame.pdf
Cherny, N. I., Sullivan, R., Dafni, U., Kerst, J. M., Sobrero, A., Zielinski, C., et al. (2015). A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS). Ann. Oncol. 26 (8), 1547–1573. doi: 10.1093/annonc/mdv249
Cherny, N. I., de Vries, E. G., Dafni, U., Garrett-Mayer, E., McKernin, S. E., Piccart, M., et al. (2018). Comparative assessment of clinical benefit using the ESMO-magnitude of clinical benefit scale version 1.1 and the ASCO value framework net health benefit score. J. Clin. Oncol. 37 (4), 336–349. doi: 10.1200/JCO.18.00729
Dankó, D., Molnár, M. P. (2017). Balanced assessment systems revisited. J. Market Access Health Policy 5 (1), 1355190. doi: 10.1080/20016689.2017.1355190
Dionne, F., Mitton, C., Dempster, B., Lynd, L. D. (2015). Developing a multi-criteria approach for drug reimbursement decision making: an initial step forward. J. Popul. Ther. Clin. Pharmacol. 22 (1), e68–e77.
Doyle, J. J., Hawryluk, E., Niemira, J., Wood, B. (2019). Evidence-based valuation in oncology: lessons learned from a case study. Ther. Innovation Regul. Sci. 53 (3), 403–411. doi: 10.1177/2168479018786701
Drake, J. I., de Hart, J. C. T., Monleón, C., Toro, W., Valentim, J. (2017). Utilization of multiple-criteria decision analysis (MCDA) to support healthcare decision-making FIFARMA, 2016. J. Market Access Health Policy 5 (1), 1360545. doi: 10.1080/20016689.2017.1360545
Endrei, D., Molics, B., Ágoston, I. (2014). Multicriteria decision analysis in the reimbursement of new medical technologies: real-world experiences from Hungary. Value Health 17 (4), 487–489. doi: 10.1016/j.jval.2014.01.011
Facey, K., Topfer, L.-A., Chan, L. (2006). Health technology assessment (HTA) glossary (Sweden: INAHTA Secretariat, c/o SBU Stockholm).
Frazão, T. D., Camilo, D. G., Cabral, E. L., Souza, R. P. (2018). Multicriteria decision analysis (MCDA) in health care: a systematic review of the main characteristics and methodological steps. BMC Med. Inf. Decision making 18 (1), 90. doi: 10.1186/s12911-018-0663-1
Garrison, L. P., Jr., Neumann, P. J., Willke, R. J., Basu, A., Danzon, P. M., Doshi, J. A., et al. (2018). A health economics approach to US value assessment frameworks—summary and recommendations of the ISPOR Special Task Force Report [7]. Value Health 21 (2), 161–165. doi: 10.1016/j.jval.2017.12.009
Gilabert-Perramon, A., Torrent-Farnell, J., Catalan, A., Prat, A., Fontanet, M., Puig-Peiró, R., et al. (2017). Drug evaluation and decision making in Catalonia: development and validation of a methodological framework based on multi-criteria decision analysis (MCDA) for orphan drugs. Int. J. Technol. Assess. Health Care 33 (1), 111–120. doi: 10.1017/S0266462317000149
Goeree, R., He, J., O’Reilly, D., Tarride, J.-E., Xie, F., Lim, M., et al. (2011). Transferability of health technology assessments and economic evaluations: a systematic review of approaches for assessment and application. Clinico. Econ. Outcomes Res.: CEOR 3, 89. doi: 10.2147/CEOR.S14404
Goetghebeur, M. M., Wagner, M., Khoury, H., Levitt, R. J., Erickson, L. J., Rindress, D. (2008). Evidence and Value: Impact on DEcisionMaking–the EVIDEM framework and potential applications. BMC Health Serv. Res. 8 (1), 270. doi: 10.1186/1472-6963-8-270
Goetghebeur, M. M., Wagner, M., Khoury, H., Levitt, R. J., Erickson, L. J., Rindress, D. (2012). Bridging health technology assessment (HTA) and efficient health care decision making with multicriteria decision analysis (MCDA) applying the EVIDEM framework to medicines appraisal. Med. Decision making 32 (2), 376–388. doi: 10.1177/0272989X11416870
Grutters, J. P., Seferina, S. C., Tjan-Heijnen, V. C., van Kampen, R. J., Goettsch, W. G., Joore, M. A. (2011). Bridging trial and decision: a checklist to frame health technology assessments for resource allocation decisions. Value Health 14 (5), 777–784. doi: 10.1016/j.jval.2011.01.005
Guindo, L. A., Wagner, M., Baltussen, R., Rindress, D., van Til, J., Kind, P., et al. (2012). From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Effect. Resource Allocation 10 (1), 9. doi: 10.1186/1478-7547-10-9
Henshall, C., Schuller, T. (2013). Health technology assessment, value-based decision making, and innovation. Int. J. Technol. Assess. Health Care 29 (4), 353–359. doi: 10.1017/S0266462313000378
Howard, S., Scott, I. A., Ju, H., McQueen, L., Scuffham, P. A. (2018). Multicriteria decision analysis (MCDA) for health technology assessment: the Queensland Health experience. Aust. Health Rev. 43 (5), 591–599. doi: 10.1071/AH18042
Hu, S., Zhang, Y., He, J., Du, L., Xu, M., Xie, C., et al. (2015). A case study of pharmaceutical pricing in China: setting the price for off-patent originators. Appl. Health Econ. Health Policy 13 (1), 13–20. doi: 10.1007/s40258-014-0150-5
Inotai, A., Pékli, M., Jóna, G., Nagy, O., Remák, E., Kaló, Z. (2012). Attempt to increase the transparency of fourth hurdle implementation in Central-Eastern European middle income countries: publication of the critical appraisal methodology. BMC Health Serv. Res. 12 (1), 332. doi: 10.1186/1472-6963-12-332
Inotai, A., Brixner, D., Maniadakis, N., Dwiprahasto, I., Kristin, E., Prabowo, A., et al. (2018a). Development of multi-criteria decision analysis (MCDA) framework for off-patent pharmaceuticals–an application on improving tender decision making in Indonesia. BMC Health Serv. Res. 18 (1), 1003. doi: 10.1186/s12913-018-3805-3
Inotai, A., Nguyen, H. T., Hidayat, B., Nurgozhin, T., Kiet, P. H. T., Campbell, J. D., et al. (2018b). Guidance toward the implementation of multicriteria decision analysis framework in developing countries. Expert Rev. Pharmaco. Outcomes Res. 18 (6), 585–592. doi: 10.1080/14737167.2018.1508345
Institute for Clinical Economic Review (ICER) (2017). Overview of the ICER value assessment framework and update for 2017–2019 14 March 2019). Available from: https://icer-review.org/wp-content/uploads/2018/03/ICER-value-assessment-framework-update-FINAL-062217.pdf
Iskrov, G., Stefanov, R. (2016). Criteria for drug reimbursement decision-making: an emerging public health challenge in Bulgaria. Balkan Med. J. 33 (1), 27–35. doi: 10.5152/balkanmedj.2015.15185
Iskrov, G. G., Raycheva, R. D., Stefanov, R. S. (2013). Insight into reimbursement decision-making criteria in Bulgaria: implications for orphan drugs. Folia Med. 55 (3–4), 80–86. doi: 10.2478/folmed-2013-0032
Iskrov, G., Miteva-Katrandzhieva, T., Stefanov, R. (2016). Multi-criteria decision analysis for assessment and appraisal of orphan drugs. Front. Public Health 4, 214. doi: 10.3389/fpubh.2016.00214
Jaramillo, H. E. C., Goetghebeur, M., Moreno-Mattar, O. (2016). Testing multi-criteria decision analysis for more transparent resource-allocation decision making in Colombia. Int. J. Technol. Assess. Health Care 32 (4), 307–314. doi: 10.1017/S0266462316000350
Kaló, Z., Landa, K., Doležal, T., Vokó, Z. (2012). Transferability of National Institute for Health and Clinical Excellence recommendations for pharmaceutical therapies in oncology to Central-Eastern European countries. Eur. J. Cancer Care 21 (4), 442–449. doi: 10.1111/j.1365-2354.2012.01351.x
Kaló, Z., Bodrogi, J., Boncz, I., Dózsa, C., Jóna, G., Kövi, R., et al. (2013). Capacity building for HTA implementation in middle-income countries: the case of Hungary. Value Health Reg. Issues 2 (2), 264–266. doi: 10.1016/j.vhri.2013.06.002
Kaló, Z., Gheorghe, A., Huic, M., Csanádi, M., Kristensen, F. B. (2016). HTA implementation roadmap in Central and Eastern European countries. Health Econ. 25, 179–192. doi: 10.1002/hec.3298
Kapiriri, L., Razavi, D. (2017). How have systematic priority setting approaches influenced policy making? A synthesis of the current literature. Health Policy 121 (9), 937–946. doi: 10.1016/j.healthpol.2017.07.003
Keeney, R. L., Raiffa, H. (1993). Decisions with multiple objectives: preferences and value trade-offs (Cambridge, United Kingdom: Cambridge University Press).
Kolasa, K., Zwolinski, K. M., Kalo, Z., Hermanowski, T. (2016). Potential impact of the implementation of multiple-criteria decision analysis (MCDA) on the Polish pricing and reimbursement process of orphan drugs. Orphanet. J. Rare Dis. 11 (1), 23. doi: 10.1186/s13023-016-0388-0
Kolasa, K., Zah, V., Kowalczyk, M. (2018). How can multi criteria decision analysis support value assessment of pharmaceuticals?-findings from a systematic literature review. Expert Rev. Pharmaco. Outcomes Res. 18 (4), 379–391. doi: 10.1080/14737167.2018.1467759
Kolasa, K. (2014). Are pricing and reimbursement decision-making criteria aligned with public preferences regarding allocation principles in the Polish healthcare sector? Expert Rev. Pharmaco. Outcomes Res. 14 (5), 751–762. doi: 10.1586/14737167.2014.940903
Kristensen, F. B., Lampe, K., Wild, C., Cerbo, M., Goettsch, W., Becla, L. (2017). The HTA Core Model®—10 years of developing an international framework to share multidimensional value assessment. Value Health 20 (2), 244–250. doi: 10.1016/j.jval.2016.12.010
Kujawski, E. (Ed.) (2003). “4.7. 3 Multi-Criteria Decision Analysis: Limitations, Pitfalls, and Practical Difficulties,” in INCOSE International Symposium. Vol. 13 no. 1. (Washington, DC: Wiley Online Library), 1169–1176. doi: 10.1002/j.2334-5837.2003.tb02692.x
Kwon, S.-H., Park, S.-K., Byun, J.-H., Lee, E.-K. (2017). Eliciting societal preferences of reimbursement decision criteria for anti cancer drugs in South Korea. Expert Rev. Pharmaco. Outcomes Res. 17 (4), 411–419. doi: 10.1080/14737167.2017.1277144
Lakdawalla, D. N., Doshi, J. A., Garrison, L. P., Jr., Phelps, C. E., Basu, A., Danzon, P. M. (2018). Defining elements of value in health care—a health economics approach: an ISPOR Special Task Force report [3]. Value Health 21 (2), 131–139. doi: 10.1016/j.jval.2017.12.007
Marsh, K., Lanitis, T., Neasham, D., Orfanos, P., Caro, J. (2014). Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature. Pharmacoeconomics 32 (4), 345–365. doi: 10.1007/s40273-014-0135-0
Marsh, K., IJzerman, M., Thokala, P., Baltussen, R., Boysen, M., Kaló, Z., et al. (2016). Multiple criteria decision analysis for health care decision making—emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value \Health 19 (2), 125–137. doi: 10.1016/j.jval.2015.12.016
Marsh, K., Caro, J. O., Hamed, A., Zaiser, E. (2017). Amplifying each patient’s voice: a systematic review of multi-criteria decision analyses involving patients. Appl. Health Econ. Health Policy 15 (2), 155–162. doi: 10.1007/s40258-016-0299-1
Marsh, K. D., Sculpher, M., Caro, J. J., Tervonen, T. (2018). The use of MCDA in HTA: great potential, but more effort needed. Value Health 21 (4), 394–397. doi: 10.1016/j.jval.2017.10.001
Memorial Sloan Kettering Cancer Center (MSKCC) (2015). DrugAbacus tool. DrugPricingLab (Downloaded: 16 March 2019). Available from: https://drugpricinglab.org/tools/drug-abacus/methods.
Németh, B., Molnár, A., Bozóki, S., Wijaya, K., Inotai, A., Campbell, J. D., et al. (2019). Comparison of weighting methods used in multicriteria decision analysis frameworks in healthcare with focus on low-and middle-income countries. J. Comp. Effect. Res. 8 (4), 195–204. doi: 10.2217/cer-2018-0102
Neumann, P. J., Willke, R. J., Garrison, L. P., Jr. (2018). A health economics approach to US value assessment frameworks—introduction: an ISPOR Special Task Force report [1]. Value Health 21 (2), 119–123. doi: 10.1016/j.jval.2017.12.012
Oortwijn, W., Sampietro-Colom, L., Habens, F. (2017). Developments in value frameworks to inform the allocation of healthcare resources. Int. J. Technol. Assess. Health Care 33 (2), 323–329. doi: 10.1017/S0266462317000502
Radaelli, G., Lettieri, E., Masella, C., Merlino, L., Strada, A., Tringali, M. (2014). Implementation of EUnetHTA core model® in Lombardia: the VTS framework. Int. J. Technol. Assess. Health Care 30 (1), 105–112. doi: 10.1017/S0266462313000639
Radu, C.-P., Chiriac, N. D., Pravat, A. M. (2016). The development of the Romanian scorecard HTA system. Value Health Reg. Issues 10, 41–47. doi: 10.1016/j.vhri.2016.07.006
Ramli, A., Aljunid, S. M., Sulong, S., Yusof, F. A. M. (2013). National Drug Formulary review of statin therapeutic group using the multiattribute scoring tool. Ther. Clin. Risk Manage. 9, 491. doi: 10.2147/TCRM.S52078
Rudmik, L., Drummond, M. (2013). Health economic evaluation: important principles and methodology. Laryngoscope 123 (6), 1341–1347. doi: 10.1002/lary.23943
Sorenson, C., Kanavos, P., Karamalis, M. (2009). HTA in central and Eastern Europe: current status, challenges and opportunities. J. Med. device Regul. 6 (1), 34–45.
Sussex, J., Rollet, P., Garau, M., Schmitt, C., Kent, A., Hutchings, A. (2013). A pilot study of multicriteria decision analysis for valuing orphan medicines. Value Health 16 (8), 1163–1169. doi: 10.1016/j.jval.2013.10.002
Tanios, N., Wagner, M., Tony, M., Baltussen, R., van Til, J., Rindress, D., et al. (2013). Which criteria are considered in healthcare decisions? Insights from an international survey of policy and clinical decision makers. Int. J. Technol. Assess. Health Care 29 (4), 456–465. doi: 10.1017/S0266462313000573
Thokala, P., Devlin, N., Marsh, K., Baltussen, R., Boysen, M., Kalo, Z., et al. (2016). Multiple criteria decision analysis for health care decision making—an introduction: report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health 19 (1), 1–13. doi: 10.1016/j.jval.2015.12.003
Tony, M., Wagner, M., Khoury, H., Rindress, D., Papastavros, T., Oh, P., et al. (2011). Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada. BMC Health Serv. Res. 11 (1), 329. doi: 10.1186/1472-6963-11-329
Toumi, M., Rémuzat, C. (2017). Value Added Medicines - Time to Adjust the HTA Decision Frameworks (17 March 2019). Available from: https://www.medicinesforeurope.com/wp-content/uploads/2017/09/White-paper_HTAonValueaddedmedicines_Final.pdf.
Velasco-Garrido, M., Busse, R. (2005). Health technology assessment: an introduction to objectives, role of evidence, and structure in Europe. Health Technol. Assess.: an introduction to objectives role evidence structure Europe. 24 p.
Williams, P., Mauskopf, J., Lebiecki, J., Kilburg, A. (2014). Using multicriteria decision analysis during drug development to predict reimbursement decisions. J. Market Access Health Policy 2 (1), 25270. doi: 10.3402/jmahp.v2.25270
Wiseman, V., Mitton, C., Doyle-Waters, M. M., Drake, T., Conteh, L., Newall, A. T., et al. (2016). Using economic evidence to set healthcare priorities in low-income and lower-middle-income countries: a systematic review of methodological frameworks. Health Econ. 25, 140–161. doi: 10.1002/hec.3299
Working Group on Mechanism of Coordinated Access to Orphan Medicinal Products (MoCA-OMP) (2014). Transparent Value Framework (17 March 2019). Available from: http://download2.eurordis.org.s3.amazonaws.com/moca/history/WG%20MoCA-OMP%20Transparent%20Value%20Framework.pdf
World Bank Country and Lending Groups Country Classification (2020). [cited 2019 17 Sept]. Available from. https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups.
Youngkong, S. (2014). Application of HTA research on policy decision-making. J. Med. Assoc. Thailand= Chotmaihet thangphaet 97, S119–S126.
Zah, V., Thompson, S., Berjan, M., Goeree, R. (2015). Zrx Mcdm: a fully flexible tool to support the local adaptation of multiple-criteria decision criteria in health care. Value Health 18 (7), A703. doi: 10.1016/j.jval.2015.09.2634
Keywords: health technology assessment, pharmaceutical, reimbursement, developing country, multiple criteria
Citation: Jakab I, Németh B, Elezbawy B, Karadayı MA, Tozan H, Aydın S, Shen J and Kaló Z (2020) Potential Criteria for Frameworks to Support the Evaluation of Innovative Medicines in Upper Middle-Income Countries—A Systematic Literature Review on Value Frameworks and Multi-Criteria Decision Analyses. Front. Pharmacol. 11:1203. doi: 10.3389/fphar.2020.01203
Received: 06 March 2020; Accepted: 23 July 2020;
Published: 14 August 2020.
Edited by:
Sam Salek, University of Hertfordshire, United KingdomReviewed by:
Mohamed Izham Mohamed Ibrahim, Qatar University, QatarLuis Laranjeira, Eli Lilly, Portugal
Copyright © 2020 Jakab, Németh, Elezbawy, Karadayı, Tozan, Aydın, Shen and Kaló. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Ivett Jakab, ivett.jakab@syreon.eu