Multi-Criteria Decision Analysis (MCDA) Models in Health Technology Assessment of Orphan Drugs—a Systematic Literature Review. Next Steps in Methodology Development?

Abstract

Background: Multi-criteria decision analysis (MCDA) is a decision-making tool that can take into account multidimensional factors and enables comparison of (medical) technologies by combining individual criteria into one overall appraisal. The MCDA approach has slowly gained traction within Health Technology Assessment (HTA) and its elements are gradually being incorporated into HTA across Europe. Several groups of scientists have proposed MCDA approaches targeted toward orphan drugs and rare diseases by including criteria specific to rare diseases. The goal of this article is to provide an overview of the current state of knowledge and latest developments in the field of MCDA in HTA for orphan drugs, to review existing models, their design characteristics, as well as to identify opportunities for further model improvement.

Methods: A systematic literature search was conducted in January 2018 using four databases: MEDLINE (Pubmed), EBSCO HOST, EMBASE, and Web of science to find publications related to use of MCDA in the rare disease field (keywords: MCDA/orphan drug/rare disease and synonyms). Identified MCDA models were analyzed, e.g., structure, criteria, scoring, and weighting methodology.

Results: Two hundred and eleven publications were identified, of which 29 were included after removal of duplicates. 9 authors developed own MCDA models, 7 of which based on literature reviews intended to identify the most important and relevant decision criteria in the model. In 13 publications (8 models) weights were assigned to criteria based on stakeholder input. The most commonly chosen criteria for creation of the MCDA models were: comparative effectiveness/efficacy, the need for intervention, and disease severity. Some models have overlapping criteria, especially in the treatment cost and effectiveness areas.

Conclusions: A range of MCDA models for HTA have been developed, each with a slightly different approach, focus, and complexity, including several that specifically target rare diseases and orphan drug appraisal. Models have slowly progressed over the years based on pilots, stakeholder input, sharing experiences and scientific publications. However, full consensus on model structure, criteria selection and weighting is still lacking. A simplification of the MCDA model approach may increase its acceptance. A multi-stakeholder discussion on fundamental design and implementation strategies for MCDA models would be beneficial to this end.

Introduction

Health technology assessment of orphan drugs

There is an ongoing debate if conventional health technology assessment (HTA) methods are still appropriate for orphan medicinal products (OMPs) and other highly specialized, innovative, expensive treatments (1–4). The one-size-fits-all approach of traditional cost-effectiveness analysis (expressed as cost per quality-adjusted life-year) no longer seems suitable for many new innovative interventions, leading to difficulties in the reimbursement process and delay in patient access to potentially valuable treatments. Two main characteristics of rare disease populations lead to structural methodological problems: 1. the rarity of these diseases which creates hurdles in setting up randomized clinical trials with enough statistical power to discern an overall treatment effect, and 2. disease heterogeneity, which causes problems with defining suitable endpoints and the generation of clinically relevant, quantifiable and reproducible treatment outcomes (5–7). The fact that the costs of these specialized interventions are often much higher than treatments for common diseases, makes difficulties during reimbursement assessment nearly inevitable. Drug manufacturers try to justify the high prices by the costly research and development path for these drugs and the small patient populations on which they try to generate a return of investment. Societies universally share the intention to effectively treat patients with a high medical need but at what cost? Healthcare budgets are limited, which is why HTA is used to assess the worth of various medical interventions and make coverage decisions.

Direct treatment costs and effectiveness are the main (and often only) criteria that are taken into account, as well as the impact on the healthcare budget (7). The effect of treatment interventions on the quality of life and indirect costs, such as patients' and/or caregivers' loss of productivity, are often not included (8–10). These factors are especially relevant for rare diseases, where effects beyond the direct treatment effect and indirect costs can be of relatively high importance (11) and this can lead to an unfair view of the actual cost-effectiveness of orphan drugs (ODs). Medical costs other than drug costs are often high for rare disorders, which usually start at an early age and require life-long special care, including many healthcare professional (HCP) visits, hospitalization and other treatments by specialists and caregivers (11, 12)¹; (13). Although some factors that are relevant to rare diseases may be considered by HTA agencies, such as “unmet medical need” and “disease severity” (14), HTA bodies generally have their own set of rules, which are often not well defined. Some agencies have lower requirements for reimbursement of orphan drugs, e.g., France and Turkey do not require a full cost-effectiveness analysis and Germany uses lower evidence thresholds for OMP's in the AMNOG process. UK's NICE accepts higher incremental cost-effectiveness ratios (ICER) for orphan and ultra-orphan drugs (15– 17)^2,3; (18)⁴. HTA and reimbursement decision-making processes frequently lack transparency and consistency, making it difficult for manufacturers, payers, patients, and society as a whole to discuss or justify reimbursement decisions or to compare HTA outcomes between countries and regions. MCDA could help in providing a structured, predictable, and transparent approach.

Multi-criteria decision analysis (MCDA)

Multi-criteria decision analysis is a decision-making tool that can take into account multidimensional factors and enables comparison of medical technologies by combining individual criteria into one overall appraisal (19, 20). MCDA can facilitate complex decision-making processes and has been used in a range of industries since the 1960's e.g., in financial decision-making, geographical information systems, and environmental impact studies (21–23). Having realized the issues with the healthcare reimbursement decision-making process, several groups of scientists adapted the traditional MCDA model toward HTA. One such initiative is the EVIDEM framework (Evidence and Value: Impact on DEcision Making), created by Goetghebeur and Wagner et al. (24) in 2008, which aims to be an open-source toolkit for healthcare reimbursement decision-making and which is continuously updated and improved (10th edition in 2017)⁵. In 2016 a first adaptation of EVIDEM specifically tailored to orphan drugs was created, trying to capture effects beyond costs and treatment efficacy, such as impact on quality-of-life of both patient and caregivers, impact on indirect costs such as productivity loss, as well as societal criteria which are traditionally not considered by payers/health insurers (25). A similar initiative is the Transparent Value Framework (TVF) created by Hughes-Wilson, which was tested within the EU Mechanism of Coordinated Access to orphan medicinal products project (MoCA) in 2013 (26)⁶. The main recommendation of MoCA was to develop a coordinated mechanism between the 12 participating Member States and OMP developers to evaluate the value of OMPs. This process would be based on a so-called transparent value framework, to support the exchange of information enabling informed pricing and reimbursement decisions at Member State level. This can be considered a first attempt to implement multi-criteria analysis into reimbursement processes across the EU (26)⁶.

The MCDA approach has slowly gained traction with HTA agencies and elements are gradually being incorporated into HTA across Europe. In Hungary MCDA was introduced by a ministerial decree in 2010 for the evaluation of new hospital medical technologies (27). In the Italian region of Lombardy MCDA has been used since 2011 as a supportive HTA tool for diagnostic and medical devices, interventional procedures and medicinal products (incl. OMPs), called the VTS model (28). The current VTS model is based on EUNetHTA's Core HTA model (v3.0), with an incorporation of MCDA elements from the EVIDEM MCDA framework (v3.0) (29). This pioneering work of the Lombardy region is now being explored by other regions in Italy as well as at the national level (30).

MCDA is also starting to be utilized by researchers in the rare disease field, which has resulted in a list of scientific publications. Palaska published an overview of existing OMP MCDA models in 2014 (poster presentation) (31). The group of experts collaborating in the ORPH-VAL working group also listed some of the OMP MCDA models and created a set of principles which should be taken into consideration during OMP reimbursement processes (30). After conducting the literature review in January 2018 the authors came across a review article with a similar concept (32). However, the identified article included fewer studies and focused mostly on explaining the methodology of MCDA.

The objective of the study

The goal of this article is to provide an overview of the current state of knowledge and latest developments in the field of MCDA in HTA for orphan drugs, to review the existing models, their design characteristics, as well as to identify opportunities for further model improvement. Given the speed at which the orphan drug HTA field is changing, the authors believe a publication with the latest information on the use of MCDA models in orphan drugs is warranted (e.g., 10 new studies concerning this topic were published in 2017). Focus was laid on studies that discuss the design and practical implementation of MCDA, e.g., the identification of relevant model criteria and weights, scoring methods, assessment, and comparison of drugs through MCDA appraisal.

Materials and methods

A systematic literature search was conducted using four databases: MEDLINE (Pubmed), EBSCO HOST, EMBASE and Web of Science in January 2018. The search strategy was focused on the use of MCDA in HTA and reimbursement of orphan drugs and methodological aspects of the different models. As many reimbursement/HTA aspects can be expressed in various ways, a high-level search was conducted with only the keywords “MCDA,” “orphan drugs,” and “rare diseases” (and synonyms) to capture all possible relevant publications.

The following search strategy was applied in all four databases:

[MCDA or (“multi-criteria-decision-analysis”) or (“multicriteria decision analysis”) or (“multi-criteria decision analysis”) or (“multi criteria decision analysis”) or (“multiple criteria decision analysis”) or (“multiple-criteria decision analysis”) or (“multiple-criteria-decision-analysis”) or (“multicriteria analysis”) or (“multi-criteria analysis”) or (“multi-criteria-analysis”) or (“multiple criteria analysis”) or (“multiple-criteria analysis”) or (“multiple-criteria-analysis”) or (“multi-criteria decision making”) or (“multi-criteria-decision-making”) or (“multi criteria decision making”) or (“multicriteria decision making”) or (“multiple-criteria decision making”) or (“multiple-criteria-decision-making”) or (“multiple criteria decision making”)] AND [(orphan drug^*) OR “OD” OR “OMP” OR “ODs” OR (OMPs) OR (orphan medicinal product^*) OR (rare disease^*) OR (rare disorder^*)].

No time limits were imposed in order to spot trends over time (if any). Only full-text articles and abstracts published by 1st January 2018 written in English were included. All steps of the literature review (identification, screening, eligibility, inclusion, and data extraction) were performed by 2 independent researchers.

Included in this review were original research publications addressing the following subjects within the area of MCDA for OMP/rare diseases: model creation and adjustment, identification and definition of model criteria, weight elicitation, model validation through MCDA OMP appraisal/testing, and articles on discussing the impact of MCDA application on decision making.

Publications that did not address MCDA and OMPs in-depth or as the main subject, as well as publications on the subject of MCDA outside the field of HTA and/or reimbursement for OMPs (e.g., MCDA in benefit-risk assessments or used in other medical conditions than rare diseases) were excluded.

Identification

The initial search resulted in 211 publications (from Pubmed n = 24; EBSCO HOST n = 98; EMBASE n = 55; web of science n = 34). No additional records were identified using other sources.

Screening

One hundred and four records remained after removing duplicates. They were individually screened by title and abstract. Fifty seven records were excluded as they were not relevant to orphan drugs/rare diseases and MCDA in reimbursement/HTA field (e.g., concerning the use of MCDA in benefit-risk assessment or other medical conditions than rare diseases). Forty seven records were screened by reading full-text articles, out of which 13 were excluded which did not have MCDA as the main subject or in case the publication had a more theoretical nature (not research-based) or contained only limited information on MCDA.

Eligibility

Thirty four publications were included in qualitative synthesis. The authors decided to remove an additional 5 abstracts which had overlapping results and outcomes with identified full-text articles.

Inclusion

Twenty nine publications were included in the systematic review, i.e., articles concerning MCDA in the field of reimbursement/HTA processes of rare diseases and orphan drugs: 13 full-text articles and 16 posters. Sixteen publications dealt with the creation and testing of MCDA models and 6 publications were of a more general nature describing MCDA principles (Figure 1). Data such as: title, author(s), year of publication, type of publication, goal of the article, methodology, involved stakeholders, methods used for criteria weighting, and scoring, a short description of results and information whether the model was tested/validated with a drug evaluation was extracted from the identified publications.

Figure 1

The systematic literature review was conducted and described in accordance with the PRISMA statement.

Results

Study characteristics

Of the 29 publications (8, 33–51), 18 were published in 2016 and onwards (the oldest being from 2011), which reflects the “novelty” of MCDA in OMP healthcare decision-making. All publications described the use of MCDA for assessment of OMPs in Europe, specifically in 10 European countries: Bulgaria, France, Germany, Italy, the Netherlands, Poland, Russia, Spain, Ukraine and the United Kingdom. Two publications discussed MCDA application in multiple European countries [Sussex et al. (37) and Wagner et al. (43)].

Sixteen publications focused mostly on defining the most appropriate model criteria [Wagner et al. (25) and Wagner et al. (43)], Hughes-Wilson et al. (33), Kolasa et al. (34), Iskrov et al. (35), Trip et al. (36), Sussex et al. (37), Schey et al. (42), Fedyaeva et al. (38), Paulden et al. (8), Palaska and Hutchings (31), and Piniazkho et al. (40)] but in 5 it did not lead to the creation of a defined MCDA model [Schlander et al. (52), Zhang et al. (53), Nemeth and Piniazhko (54), Korchagina et al. (55), and Hutching et al. (56)].

In 8 studies weights were assigned to criteria based on stakeholder input [Trip et al. (36), Sussex et al. (37), Gilabert-Perramon et al. (44), Garau et al. (45), Iskrov et al. (35), Fedyeva et al., Wagner et al. (43) Piniazkho et al. (40), and Piniazkho and Nemeth (41)] and in one publication weights were allocated by the authors [Wagner et al. (25)]. Two articles provided suggestions for MCDA score definition and calculation [Kolasa et al. (34) and Hughes-Wilson et al. (33)].

Trip et al. (36), Kolasa et al. (34), Schey et al. (42), Iskrov et al. (35), Paulden et al. (8), Sussex et al. (37), Wagner et al. (25), and Piniazkho et al. (40) developed their own MCDA models, primarily based on literature reviews that were intended to identify the most important and relevant decision criteria for the model. Hughes-Wilson (33), Fedyaeva et al. (38, 39), and Krysanova et al. (49) also built MCDA models, but information on the criteria definition and selection process is lacking. Wagner et al. (25) tailored the existing EVIDEM MCDA framework to orphan drugs, by adding sub-criteria specific for rare diseases into the model and by adjusting scoring scales.

Trip et al. (36), Iskrov et al. (35), Kolasa et al. (34), Sussex et al. (37), Schey et al. (46), Piniazkho et al. (40), Wagner et al (43), Schey et al. (46), Tony et al. (47), Badia et al. (50), Garau et al. (45), Jimenez et al. (48), Fedyaeva et al. and Krysanova et al. (49) performed a test of the MCDA model for orphan drug evaluation in their research. Hughes-Wilson et al. did not provide any information on testing the Transparent Value Framework MCDA model in the identified publication, however, the model was later tested within the EU MoCA project⁶. For more details refer to Table 1.

The EVIDEM framework

The EVIDEM framework was the most researched model in the identified publications: its applicability was assessed by: Jimenez et al. (48) in Spain (v 4.0), by Garau et al. (45) in Italy (v3.0), by Gilabert-Perramon et al. (44) in Catalonia (v3.0) and by Badia et al. (50), Tony et al. (47, 58) and Wagner et al. (43) in Italy, Spain and France (v2.4), respectively. In addition, Gilabert-Perramon et al. (44) compared the EVIDEM framework to the current HTA system in place in Catalonia (PASFTAC). The MCDA model created by Hughes-Wilson et al. (33) was tested by Schey et al. (46) and within the EU MoCA project.

Seven authors tried to identify criteria that should be taken into consideration during reimbursement and HTA processes of orphan drugs. Two of them [Gutierrez et al. (51), Annemans et al. (30)] emphasized the principles of MCDA development specific for orphan drug HTA in their articles. The rest were posters (52–56) without sufficient information to include them in this publication. Neither Gutierrez et al. (51) nor the authors of the ORPH-VAL group created a MCDA model, however, they selected and defined the main guiding principles for an OMP MCDA model that could be implemented in the EU. For the definition of the principles, Gutierrez et al. (51) took the perspective of a manufacturer of orphan drugs, whereas the ORPH-VAL group used a multi-stakeholder approach. Gutierrez et al. (51) defined a list of 10 principles that could help improve consistency, effectiveness and sustainability of orphan drug HTA models (no specific model was taken). Because the authors used the perspective of an OMP manufacturer, several principles were different from the other publications, e.g., the principle not to perform EMA's assessment of therapeutic benefit again during HTA on the national level. Goetghebeur et al. (57) conducted a questionnaire (based on EVIDEM) among health authority representatives from 8 countries to collect feedback on the mandates and values of HTA agencies, to examine ethical underpinnings of the HTA values and to explore trade-offs.

Detailed information on the identified studies can be found in Table 1 (57).

Weighting

In 13 publications (using 8 different models) the authors assigned weights to the individual criteria in their models, based on stakeholder interviews via questionnaires or in workshops but also based on the authors' own insights. The most commonly used methods for weighting and scoring in the reviewed publications were: simple additive weighting (five-point scale) and hierarchical point allocation. Three criteria were consistently rated as the most important (i.e., having highest weight scores): disease severity/burden (Iskrov et al. (35), Wagner et al. (43), Gilabert-Perramon et al. (44), and Sussex et al. (37)], effectiveness/comparative effectiveness/therapeutic effect [Iskrov et al. (35), Wagner et al. (24, 25, 43), Gilabert-Perramon et al. (44), and Sussex et al. (37)] and “unmet medical need” (i.e., lack of alternative treatments) [Gilabert-Perramon et al. (44) and Sussex et al. (37)]. Other criteria that were rated high were quality/strength of evidence (of efficacy) (Gilabert-Perramon et al. (44), Wagner et al. (24, 25, 43), and Iskrov et al. (35)] and the safety/tolerability of the treatment [Gilabert-Perramon et al. (44) and Wagner et al. (43)].

The comparison of criteria used in MCDA models

Just as in Gutierrez' publications, several articles considered criteria that are relevant for OMP manufacturers: 3 publications (2 models) included the complexity of the manufacturing process [Schey et al. (46), Hughes-Wilson et al. (33), and Kolasa et al. (34)] and in 6 (4 models) the “level of research undertaken/innovativeness” was defined as a criterion [Schey et al. (46), Kolasa et al. (34), Sussex et al. (37), Hughes-Wilson et al. (33), Piniazkho et al. (40), and Piniazkho and Nemeth (41)].

A detailed list of all criteria included into the respective MCDA models is presented in Table 2.

Discussion

Limitations of the study

One of the limitations of this review is its focus on publications specifically related to MCDA for orphan drugs, as this could have led to omission of publications with relevant and valuable content for this topic. A full review of MCDA in broader healthcare decision-making processes (e.g., for drugs for common diseases, medical devices or in hospital purchasing processes), in medical benefit-risk assessments or even the use of MCDA in other industries could be beneficial to get a complete picture of the methodology, its advantages and drawbacks. This could also give valuable insights on practical hurdles that exist for implementation of MCDA into decision-making processes, as well as relevant experiences and best practices.

The literature search strategy was designed to be broad with only 3 keywords, albeit designed to capture all possible variants of these keywords. A potential methodological shortcoming is that not all relevant publications may have been found due to articles missing the specific keyword of “rare disease” and/or “orphan drug,” e.g., research articles dealing with MCDA in a specific rare disease.

A systematic review format was used, no qualitative assessment was done of the different MCDA models, i.e., using a rating scale. Studies differ in design and scope and many models have both strengths and weaknesses. A valid comparison would require well-defined and objective metrics and valuation. Instead, the main focus has been laid on an assessment of the different criteria used in the MCDA models, general model structures, and possibilities for optimization of an MCDA model that is suitable, predictable and straightforward to use in practice. A more detailed analysis of model criteria and design considerations is given below.

Commentary on the most frequently used criteria

Evaluation of model criteria—general commentary

Recently, a number of authors have expressed critical views on the current MCDA frameworks and pointed out several flaws and drawbacks of the used design and scoring methodology, as well as proposing new models or alterations to correct these faults (8, 59, 60). These critiques and the fact that new MCDA models or improvements to existing ones are continuously being developed, shows that the MCDA approach in healthcare has not yet reached maturity or general consensus. This could (partly) explain the slow or partial uptake by national HTA bodies in the EU. It is expected that any advancement in orphan drug MCDA will follow improvement and implementation of MCDA methodology in the broader healthcare setting. Incorporating orphan drug and rare disease-related factors into the current MCDA developments and discussions would be beneficial to prevent unnecessary model corrections for orphan drugs later on.

Simplicity and overlapping criteria

While it may seem methodologically or ethically desirable to include as many rare-disease specific components as possible into the MCDA model for orphan drugs, careful attention must be paid to criteria selection and weighting, to ensure meaningfulness, appropriate representation and to prevent overweighting and double-counting. This is also underlined by the ISPOR MCDA guideline, which specifies that criteria should be complete, non-redundant, non-overlapping and independent of preferences, as well as being unambiguous, comprehensive, direct, operational, and understandable (59).

Some identified MCDA models include the aspect of treatment effectiveness or cost-related aspects expressed in various manners, e.g., Iskrov et al. (35) uses the criteria “clinical effectiveness,” “health-benefits,” and “cost-effectiveness” (which includes an “effectiveness” component), which can amplify the outcome into a specific direction when items related to effectiveness are counted twice or more. Trip's model contains drug effectiveness, cost-effectiveness and budget impact, as well as annual costs of the drug per patient, with overlap and therefore possible overweighting of costs and/or effectiveness in the scoring. Similarly, Kolasa et al. (34) took into account in their MCDA model the level of scientific evidence for “clinical efficiency” (level of uncertainty) and cost-effectiveness and budget impact, with clear overlap.

Criterion—benefits (EVIDEM, annemans)

Benefits is a broad “bucket” criterion that can contain a range of clinical advantages such as treatment convenience (as used in Annemans' model) and other contributions to patient care that can be relevant for disease management. The Benefits criterion can capture effects brought on by enhanced pharmaceutical formulations (e.g., improved patient-friendliness), less burdensome dosing schedules or other impact on treatment burden (e.g., fewer hospital visits needed), which can be relevant for certain rare disease treatments. The authors recommend a structured approach to the Benefits criterion with pre-specified sub-criteria. Including a field that captures “treatment compliance” would be beneficial, as this is an important factor for treatment success, and especially for costly orphan drugs the financial impact of non-compliance could be large. None of the identified models included compliance/adherence as a criterion.

Wagner et al's. (24, 25, 43) EVIDEM model contains the “Type of benefit” domain, which is used to quantify the “impact” of the preventive/therapeutic benefit for patients, e.g., a cure versus symptom relief. As discussed above, it can be argued that this field overlaps with the EVIDEM “Outcomes of intervention” domain that includes clinical effectiveness and safety. A similar situation exists for Patient-perceived health/PROs” (EVIDEM, Annemans) as this is a patient-reported measure of effectiveness that is additive and possibly overlapping. Models including PROs as a separate criterion should aim to prevent overweighting of clinical efficacy and other treatment benefits through PRO scores.

Criterion—disease population size (EVIDEM, hughes-wilson, kolasa, annemans)

While it is understood that rare diseases should be treated fairly and need a special approach, it is difficult to directly translate (a small) “population size” into a factor that is relevant to the value of a treatment and thus the MCDA outcome. Should the largest affected population receive priority treatment, which is a utilitarian view, or does the rarity of a disease justify a “compassionate” application of healthcare and therefore a higher priority above common diseases? This immediately complicates scoring and weighting: is less more or is more more?

The disease prevalence threshold in the EU for an orphan drug designation is well-defined at ≤ 5 per 10.000. Since it is expected that OMPs will most likely be compared directly with other similar (orphan) drugs, the relevance of population size on the MCDA outcome is limited. E.g. does a prevalence of 3 per 10.000 make a difference for the MCDA score over 2 per 10.000, even though it represents a 50% population size increase? Gilabert-Perramon et al. (44) also found that size of the population had the lowest relevance for decision-making (37).

Additionally, if the total budget impact of the treatment is included as a cost component in MCDA, it could be argued that the impact of “rarity” is amplified, e.g., possibly receiving a high score on “rarity” and a high score on “budget impact” (low budget impact, depending on a price of the product). While this can be the aim of a model, it should be considered during design/weighting and clearly expressed to assessors during scoring.

Since population size as such is difficult to score and might not have a relevant impact on the MCDA outcome anyway (especially when comparing orphan drugs), one can argue to not include this criterion, as Iskrov et al. (35), Sussex et al. (37), and Trip et al. (36) have done in their models. However, the authors consider that “Disease population size” could be a valuable factor in the model for certain stakeholders, as it could represent a direct incentive for manufacturers to develop drugs and other treatments for otherwise non-economical disease populations, which would otherwise be lacking.

Criterion—population priorities (EVIDEM, iskrov and annemans)

The argumentation for whether rare diseases should be treated differently by the population as a whole is essentially the same as described for “Disease Population Size.” It is difficult to generate an objective, meaningful value-score out of population priorities, and it should be considered that “Unmet need” or “Disease Severity” could also cover the “Population priority” aspect.

Criterion—unmet need (EVIDEM, trip, iskrov, hughes-wilson, kolasa, sussex, fedyaeva, annemans)

“Unmet (medical) need” is expressed differently in the various models, such as Availability of treatment (alternatives)/Unmet medical need or “Uniqueness of indication”, e.g., no strict definition exists. Unmet need can incorporate several defining criteria: patient population size, disease severity and lack of effective and/or approved treatments, which should be clearly defined, again to prevent overlap. Orphan drugs, by definition, target a disease population that has no effective and approved treatment; this is a requirement from regulatory agencies as EMA and FDA to be granted an orphan designation (exception: when a drug shows “significant benefit” over an existing orphan drug). In some cases, food products or nutritional supplements can be used to manage or alleviate symptoms of the disease (e.g., genistein for Sanfilippo syndrome) or food for special medical purposes (e.g., elimination diets) and OTC medication (e.g., pain killers). As food products, supplements and OTC products are usually out of scope of healthcare coverage and don't have established efficacy and safety profiles (i.e., frequent off-label use), one can argue to not take these products into consideration when assessing the unmet medical need for OMPs.

Criterion—comparative effectiveness and safety (EVIDEM)

Including the criterion “Comparative Effectiveness” in MCDA for orphan drugs can be questioned, as per definition there is no approved treatment available in orphan indications, and direct comparisons will rarely happen. Any (indirect) comparator would likely include off-label drugs (often older, generic drugs) and other non-approved treatments, which could limit the validity of such a comparison. It can be argued that the effectiveness or overall “value” of OMPs can be compared with drugs for similar orphan indications or drugs for indications with similar defining properties (e.g., disease type, patient population size). The same goes for comparative safety. Moreover, the efficacy and safety of the drug (and a benefit-risk outcome) have already been established during the marketing authorization application, and in case of an acceptable outcome, an approval (or conditional approval) would be granted. When assessing the efficacy/safety during MCDA one should take the scientific assessments of regulatory agencies into consideration, as clearly contradictory outcomes would be debatable.

Criterion—quality of evidence (EVIDEM, trip, kolasa, annemans, garau, paulden)

Generation of robust clinical evidence, endpoint validation and assessing clinical outcomes are inherently problematic in small population groups, which reduces the suitability of “Quality of Evidence” criterion in MCDA assessments for OMPs. Disease severity and progressive disease limits the use of placebo-controlled trials in many rare diseases and data sets will generally be small. Evidence quality by itself would not lead to meaningful differentiation between many (ultra) orphan drugs and could automatically disqualify many orphan drugs if they were to be directly compared to drugs for common diseases with a perceived “adequate” evidence quality. The authors think that a proper approach would be pragmatic and aim for certain flexibility, i.e., to allow for evidential uncertainty when this cannot be reasonably avoided, within pre-defined limits. Regulatory agencies are willing to accept a reduced burden of evidence in certain situations (i.e., for orphan drugs), but often under specific conditions that oblige manufacturers to generate additional evidence on efficacy, safety and practical use (post-approval). Accordingly, payers could approach HTA and reimbursement in case of evidential uncertainty in a similar fashion, i.e., allow early but “conditional reimbursement” for certain promising treatments, with the requirement to generate real-world evidence and perform an updated pharmacoeconomic assessment periodically. An example of a country where conditional reimbursement was used is the Netherlands, but due to the limited number and low quality of applications it has been replaced by a more general subsidy program focused on small and medium-sized enterprises (47).

Garau et al. (45) and Paulden et al. (8) have proposed to include Quality of Evidence as a “multiplier” (or weight) to the effectiveness/outcomes scores, e.g., diminish the effectiveness by a “Quality factor” depending on the certainty of the evidence. While this is an elegant solution, it would create a twice-weighting process for each of the treatment outcomes (e.g., preference weights to the fields and quality of evidence weights to the field scores), and thus complicate the model and interpretation of scores.

Criterion—innovativeness and incentives for manufacturers (hughes-wilson, kolasa, annemans, piniazkho)

A major hurdle for orphan drug developers is generating a return on investment, due to the small patient population that will use the final product. However, most HTA models take a societal perspective or a payer-centric view and do not put emphasis on factors that are important for developers of orphan drugs. In contrast, the EU regulatory framework specifically incentivizes orphan drug development by providing a range of regulatory and commercial benefits for OMP manufacturers. To align EU HTA processes with these incentives, MCDA criteria would be needed that represent a similar benefit/valuation for manufacturers of orphan drugs. Several authors have tried to capture “innovation” as a factor, e.g., Gutierrez et al. (51) provided a review of MCDA principles from the manufacturer's perspective and stated that “innovativeness” should be represented in models; Sussex et al. (37) defined the criterion “Treatment innovation: scientific advanced contribution to patient outcome”; Hughes-Wilson et al. (33) included “Level of research” and Annemans et al. (30) considered innovation as one of the key principles to improve consistency of orphan drug pricing and reimbursement assessment.

Kolasa et al. (34) presented a detailed scoring proposal for measuring manufacturing complexity via the cost of biotech processes, complexity of drug synthesis based on the number of “chemical transformations” and the necessity to use “separation techniques” for chemical intermediates. To measure “Advancement of technology”, Kolasa et al. (34) made a distinction between Advanced therapy medicinal products (ATMPs), complex drugs, small molecules (with at least one chiral center) and simple chemical entities. While the intent is appreciated, the described methodology to measure innovation has several drawbacks. Firstly, Kolasa et al's. (34) focus lies primarily on rewarding the investment in the manufacturing process, while the bulk of innovation and development costs would typically be related to the clinical trial program. Even for a new indication of a “simple” and well-known molecule, at least a Phase II or Phase III trial program is needed to generate data on safety and effectiveness. Trial costs are relatively high for orphan drugs since trials often need to be run globally over many centers to recruit sufficient numbers of eligible rare disease patients. A lack of clinical (development) guidelines, regulatory precedence, clinical experience, and expertise complicate orphan clinical programs further. Secondly, the authors believe that the chemical properties of the active ingredient do not necessarily represent the level of innovation, treatment success or research and development (R&D) investment. The exact monetary investments and pricing components by manufacturers are difficult to establish and verify externally, especially on the product level: project R&D costs often include broad overhead costs and company-wide write-offs. Alternatively, the authors propose to include the following two, more objectively quantifiable criteria to reward innovation: “Novelty of the active pharmaceutical ingredient” (e.g., defined as first in-class product, second in-class, or existing active ingredient) and “Novelty of the disease indication” (new target population or a subgroup of an existing target population, e.g., a disease subpopulation with certain mutations/biomarkers). One way to measure these would be through the ATC-code of the compound and ICD disease classification. If not considered elsewhere, innovation steps regarding pharmaceutical form improvements could also be considered here, e.g., those that result in increased ease of administration/dosing, but over-rewarding relatively simple R&D investments should be avoided. Together with Population size, these criteria would represent a clear incentive for drug developers.

The type of trial that has been performed (i.e., randomized controlled trial (RCT), open-label trial, use of historical control arms) also represents a type of innovativeness and R&D investment. However, trial design would typically be already taken into account through the Quality of Evidence criterion or weight-factor (where a double-blind RCT represents the golden standard), so double-counting would need to be carefully avoided if this were to be considered.

Economic consequences of intervention

Separating cost-components from the value-components of the intervention

A possible methodological flaw in several MCDA models is the incorporation of both costs and value (benefits) into one aggregated MCDA end-score, i.e., one additive score containing disease characteristics, treatment outcomes and cost-related criteria. Using such an aggregate score creates difficulties in assessing and interpreting the final MCDA outcome and complicates comparison of technologies. Since economic consequences are highly dependent on the local healthcare system and economic factors (e.g., GDP/healthcare budget/reimbursement structure/availability of care), separating criteria that define the “Value of the intervention” from those defining “Economic consequences” (e.g., costs) would allow for generating more objective, meaningful and understandable MCDA scores. Having a distinct “Value-Score” and “Cost-Score” of the treatment could simplify models, ease interpretation and create the flexibility to easily transfer outcomes to the national HTA level (transferability). For example, a treatment's value-score (for use in HTA) could be assessed centrally by the EMA or any other appointed scientific/regulatory and be transferred to the national HTA body for use in MCDA or other methods. The selection and definition of cost-related factors for HTA and the actual cost-calculation is performed nationally, where the cost- and value-scores can be weighed for decision-making on local healthcare coverage. Such an approach would also be aligned with the EU proposal for a new HTA Regulation (16) which describes a centralized “joint clinical assessment” (JCA). A JCA at the European level enables sharing of HTA workload between member states, similar to regulatory assessments in the EMA Centralized Procedure. Especially for orphan drugs, a centralized clinical (value) assessment by highly skilled medical and OMP HTA experts could improve the quality and speed of national health technology assessments. Objectively established value-scores could also be used for relatively quick and easy value-comparison of treatments, without bringing in cost-factors. Paulden et al. (8) used a similar approach in their model design.

Type of economic analysis

How to balance value vs. costs in healthcare budgets is one of the key considerations made by several researchers, especially whether the economic assessment should be based on opportunity-cost analysis or more traditional cost-effectiveness analysis. In situations where budgets are finite, as in healthcare systems, opportunity-cost analysis allows for the evaluation of resources that cannot be spent due to expenditure elsewhere, which stimulates the “replacement” of older interventions by newer ones with a lower opportunity cost. The opportunity cost approach plays a central role in Paulden et al's. (8) decision framework, in which the “Net Value” is the total treatment valuation minus opportunity cost (vs. relevant comparators). The Lombardian VTS framework also explicitly promotes a mandatory “delisting” of older, obsolete technologies via an opportunity-cost approach (28).

Other methodology considerations

Multiple mathematical and statistical methods can be used for scoring, weighting, analyzing and comparing alternatives in MCDA models. The theoretical basis for the models is rarely discussed in most of the reviewed studies. An ISPOR task force has issued 2 publications on best practices in MCDA to guide model developers in 2016 (19, 61). However, since most MCDA models are older than 2016 they are not following these guidelines yet, e.g., regarding non-overlapping criteria. Broekhuizen et al. (62) identified the 5 most common approaches for dealing with uncertainty in MCDA models (such as fuzzy set theory), but these are not discussed by any of the model creators (62). A study on the practical applicability of MCDA in Canada found that the quantification of evidence and interpretation of the aggregate MCDA score was “challenging” and that comparing/ranking interventions would require a better grasp of the underlying methodology (63). Gandjour (60) commented on the EVIDEM framework and expressed that the model in its current state is an “intermediate” solution and several improvements should be made, including an independent ethical justification and stronger theoretical foundation, especially for the use of individual preferences (60).

A recent EU survey (63) shows that a wide range of factors are taken into account by HTA agencies, e.g., 63% include social aspects such as “burden on care-givers.” A certain flexibility in data quality requirements is visible, as 80% of agencies are reported to accept prospective, non-randomized studies or other kinds of observational studies, traditionally viewed as being of lower quality. Similarly, 90% can accept surrogate endpoints for effectiveness or safety and around 80% acknowledge PROs, Health-Related Quality of Life measures and indirect comparisons if head-to-head comparator trials are lacking. Furthermore, 71% of agencies accept composite endpoints and network meta-analysis while 96% acknowledge subgroup analyses (63). These findings show that many HTA agencies assess non-traditional factors, which could serve as a basis for introducing MCDA into healthcare HTA.

Conclusions

Over the last 10 years a range of MCDA models for HTA have been created, each with a slightly different approach, focus and complexity, including several models that specifically target rare diseases and orphan drugs. These models have slowly progressed based on pilots, stakeholder input, sharing experiences and scientific publications. However, full consensus on model structure, criteria selection, and weighting is still lacking. As shown in this and other publications, a more fundamental discussion on the methodological and mathematical design aspects of MCDA models is ongoing and needed. Initiatives such as the EVIDEM framework and the MoCA project help to create momentum and standardization, but have not led to any significant EU-wide implementation. This shows that apparently hurdles still need to be overcome and model improvements are necessary. The planned EU HTA regulation could have some positive spill-over effects to stimulate the development and harmonization of MCDA in HTA. Given the differences in national healthcare and reimbursement systems, as well as local variations in economy and rare disease policies, HTA models that are flexible and adaptable would be required. MCDA can capture factors beyond standard cost-effectiveness analysis and offer a range of possibilities that would be suitable for rare disease HTA.

The authors have attempted to highlight some of the problems that exist with the design of current MCDA models, with a review of commonly used model criteria and how to generate more clear and meaningful results with them. As it stands, models are often unclear and results difficult to interpret, which warrants a simplification of current designs with fewer and well-defined domains and with less overlap between criteria. A strict separation of value from costs in MCDA models would be increase the flexibility, clarity of the model, and transferability of the results, which could aid implementation. Further research, model improvement and validation, practical application and multi-stakeholder discussion are necessary to bring about consensus and to fulfill the potential that MCDA promises in healthcare HTA.

References

• 1.

  Drummond MF . Challenges in the economic evaluation of orphan drugs. Eurohealth (2008) 14:16–17. Available online at: http://www.lse.ac.uk/lse-health/assets/documents/eurohealth/issues/eurohealth-v14n2.pdf?from_serp=1

  • Google Scholar

• 2.

  Simoens S . Health technologies for rare diseases: does conventional HTA still apply?Expert Rev Pharmacoecon Outcomes Res. (2014) 14:315–7. 10.1586/14737167.2014.906903

  • CrossRef
  • Google Scholar

• 3.

  Walker A . Challenges in using MCDA for reimbursement decisions on new medicines?Value Health (2016) 19:123–4. 10.1016/j.jval.2016.02.001

  • CrossRef
  • Google Scholar

• 4.

  Vorobiev P Holownia M Krasnova L . Multi-criteria decision analysis (MCDA) and its alternatives in health technology assessment, JHPOR (2015) 1:34–43. 10.7365/JHPOR.2015.1.4

  • CrossRef
  • Google Scholar

• 5.

  U.S. Food and Drug Administration, PariserA. Rare Disease and Clinical Trials (04.11.2014). Available online at: https://www.fda.gov/downloads/Drugs/NewsEvents/UCM440797.pdf (Accessed March 18, 2018).

  • Google Scholar

• 6.

  Hilgers RD König F Molenberghs G Senn S . Design and analysis of clinical trials for small rare disease populations. J Rare Dis Res Treat. (2016) 1:53–60. 10.29245/2572-9411/2016/3.1054

  • CrossRef
  • Google Scholar

• 7.

  Schuller Y Hollak CEM Biegstraaten M . The quality of economic evaluations of ultra-orphan drugs in Europe – a systematic review. Orphanet J Rare Dis. (2015) 10:92. 10.1186/s13023-015-0305-y

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8.

  Paulden M Stafinski T Menon D McCabe C . Value-based reimbursement decisions for orphan drugs: a scoping review and decision framework. Pharmacoeconomics (2014) 33:255–69. 10.1007/s40273-014-0235-x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 9.

  Kodra Y Cavazza M Schieppati A De Santis M Armeni P Arcieri R et al . The social burden and quality of life of patients with haemophilia in Italy. Blood Transfus. (2014) 12(Suppl. 3):s567–75. 10.2450/2014.0042-14s

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 10.

  Angelis A Kanavos P López-Bastida J Linertová R Juan Oliva-Moreno Pedro Serrano-Aguilar P et al . and BURQOL-RD Research Network. Social/economic costs and health-related quality of life in patients with epidermolysis bullosa in Europe. Eur J Health Econ. (2016) 17(Suppl. 1):31–42. 10.1007/s10198-016-0780-7

  • CrossRef
  • Google Scholar

• 11.

  Angelis A Tordrup D Kanavos P . Socio-economic burden of rare diseases. a systematic review of cost of illness evidence. Health Policy (2015) 119:964–79. 10.1016/j.healthpol.2014.12.016

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12.

  Baran A Czech M Hermanowski T Skoczynska K . Direct and indirect costs of Pompe disease. Farmacja Współczesna (2014) 7:149–55. Available online at: https://www.akademiamedycyny.pl/wp-content/uploads/2016/05/201404_Farmacja_001.pdf

  • Google Scholar

• 13.

  Blankart CR Koch T Linder R Verheyen F Schreyögg J Stargardt T . Cost of illness and economic burden of chronic lymphocytic leukemia. Orphanet J Rare Dis. (2013) 8:32. 10.1186/1750-1172-8-32

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 14.

  Gammie T Lu CY Babar ZU . Access to orphan drugs. a comprehensive review of legislations, regulations and policies in 35 countries. PLoS ONE10:e0140002. 10.1371/journal.pone.0140002

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 15.

  Libura M Władysiuk M Małowicka M Grabowska E Gałazka-Sobotka M Gryglewicz J . Rare Disease in Poland, Current Status and Perspectives; Choroby rzadkie w Polsce, Stan Obecny i Perspektywy. Uczelnia Łazarskiego (2016)

  • Google Scholar

• 16.

  Belgin G Macarthur D . Access to Orphan Drugs in Turkey. Jan 04, (2016)

  • Google Scholar

• 17.

  Kiliç P Koçkaya G Yemşen Ö Tan C Handan Öztunca F Aksungur P et al . Orphan drug regulation in Turkey. JPHSR (2013) 4:151–3. 10.1111/jphs.12018

  • CrossRef
  • Google Scholar

• 18.

  Weinstein N Martin M Campbell R . Orphan drugs in the Uk, do they meet the nice highly specialised technology threshold?Value Health (2017) 20:A660. 10.1016/j.jval.2017.08.1581

  • CrossRef
  • Google Scholar

• 19.

  Thokala P Devlin N Marsh K Baltussen R Boysen M Kalo Z et al . Multiple criteria decision analysis for health care decision making—an introduction:report 1 of the ISPOR MCDA emerging good practices task force. Value Health (2016) 19:1–13. 10.1016/j.jval.2015.12.003

  • CrossRef
  • Google Scholar

• 20.

  Keeney RL Raiffa H . Decisions with Multiple Objectives: Preferences and Value Trade-Offs. Cambridge: Cambridge University Press (1993).

  • Google Scholar

• 21.

  Carver SJ . Integrating multi-criteria evaluation with geographical information systems. Int J Geogr Inf Syst. (1991) 5:321–39. 10.1080/02693799108927858

  • CrossRef
  • Google Scholar

• 22.

  Hallerbach W Spronk J . The relevance of MCDM for financial decisions. J Multi-Criteria Decis Anal. (2003) 11:187–95. 10.1002/mcda.328

  • CrossRef
  • Google Scholar

• 23.

  Linkov I Moberg E . Multi-Criteria Decision Analysis: Environmental Applications and Case Studies. Boca Raton, FL: CRC Press, Taylor & Francis Group (2011).

  • Google Scholar

• 24.

  Wagner M Khoury H Levitt RJ Erickson LJ Rindress D . Evidence and value: impact on decision Making – the EVIDEM framework and potential applications. BMC Health Serv Res. (2008) 8:270. 10.1186/1472-6963-8-270

  • CrossRef
  • Google Scholar

• 25.

  Wagner M Khoury H Willet J Rindress D Goetghebeur M . Can the EVIDEM Framework tackle issues raised by evaluating treatments for rare diseases: analysis of issues and policies, and context-specific adaptation. PharmacoEconomics (2016) 34:285–301. 10.1007/s40273-015-0340-5

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 26.

  Hughes-Wilson W . MoCA Concept and Pilot Project, Feedback from the process around the first pilot project, ECRD Berlin (10.05.2014). Available online at: http://download2.eurordis.org.s3.amazonaws.com/moca/presentations/PRES-2014-05%20MoCA%20Concept%20and%20Pilot%20Project%20(Hughes-Wilson).pdf (Accessed March 18, 2018).

  • Google Scholar

• 27.

  Endrei D Molics B Ágoston I . Multicriteria decision analysis in the reimbursement of new medical technologies: real-world experiences from Hungary. Value Health (2014) 17:487–9. 10.1016/j.jval.2014.01.011

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 28.

  Radaelli G Lettieri E Masella C Merlino L Strada A Tringali M . Implementation of EUnetHTA core Model® in Lombardia: the VTS framework. Int J Technol Assess Health Care (2014) 30:105–12. 10.1017/S0266462313000639

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 29.

  Marsh K Wagner M Thokala P Baltussen R . Multi-Criteria Decision Analysis to Support Healthcare. Cham: Springer (2017). 10.1007/978-3-319-47540-0

  • CrossRef
  • Google Scholar

• 30.

  Annemans L Aymé S Le Cam Y Facey K Gunther P Nicod E et al . Recommendations from the European working group for value assessment and funding processes in rare diseases (ORPH-VAL)Orphanet J Rare Dis. (2017) 12:50. 10.1186/s13023-017-0601-9

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 31.

  Palaska C Hutchings A . Value assessment and pricing frameworks for rare disease treatments: new approaches from the literature. PSY113. Value Health (2015) 18:A678. 10.1016/j.jval.2015.09.2013

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 32.

  Friedmann C Levy P Hensel P Hiligsmann M . Using multi-criteria decision analysis to appraise orphan drugs: a systematic review. Expert Rev Pharmacoeconom Outcomes Res. (2018) 18:135–46. 10.1080/14737167.2018.1414603

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 33.

  Hughes-Wilson W Palma A Schuurman A Simoens S . Paying for the Orphan Drug System: break or bend? Is it time for a new evaluation system for payers in Europe to take account of new rare disease treatments?Orphanet J Rare Dis. (2012) 7:74. 10.1186/1750-1172-7-74

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 34.

  Kolasa K Zwolinski KM Kalo Z Hermanowski T . Potential impact of the implementation of multiple-criteria decision analysis (MCDA) on the Polish pricing and reimbursement process of orphan drugs OrphanetJ Rare Dis. (2016) 11:23. 10.1186/s13023-016-0388-0

  • CrossRef
  • Google Scholar

• 35.

  Iskrov G Miteva-Katrandzhieva T Stefanov R . Multi-criteria decision analysis for assessment and appraisal of Orphan Drugs. Front. Public Health (2016) 4:214. 10.3389/fpubh.2016.00214

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 36.

  Trip AM Tsiachristas A Koenders JM Kanters TA . Multi-criteria decision analysis for reimbursing orphan drugs: a Dutch demonstration study using the analytic hierarchy process method. PSY 114, Value Health (2014) 17:A541–2. 10.1016/j.jval.2014.08.1744

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 37.

  Sussex J Rollet P Garau M Schmitt C Kent A Hutchings A . BRIEF REPORTS A pilot study of multicriteria decision analysis for valuing orphan medicines. Value Health (2013) 16:1163–9. 10.1016/j.jval.2013.10.002

  • CrossRef
  • Google Scholar

• 38.

  Fedyaeva VK Omelyanovsky VV Rebrova O Khan N Petrovskaya EV . Mcda approach to ranking rare diseases in Russia: preliminary, PSY99. Value Health (2014) 17:A539. 10.1016/j.jval.2014.08.1729

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 39.

  Fedyaeva VK OmelyanovskiyVVRebrovaOYMarshK. PSY121 - Comparison of methods to assess the relative importance of criteria in multi-criteria decision analysis: an evaluation of orphan drugs in Russia. Value Health (2016) 19:A596–6. 10.1016/j.jval.2016.09.1438

  • CrossRef
  • Google Scholar

• 40.

  Piniazhko O Zalis'ka O Brezden O . Methodological issues in MCDA for training needed: eliciting stakeholders' value preferences in preferences in Ukraine. Value Health (2017) 20:A45.

  • Google Scholar

• 41.

  Piniazhko O Nemeth B . Practical issues of determining weights for criteria to be used in an MCDA framework-based on a case-studyValue Health (2017) 20:A51–2.

  • Google Scholar

• 42.

  Schey C Irwin J Teneishvili M Krabbe PFM Connolly M . Assessing the relationship between individual attributes identified in review of Multi-Criteria Decision Analysis (MCDA) of rare diseases and annual treatment costs in rare endocrine disorders, PRM108. Value Health (2014) 17:A323–686

  • Pubmed Abstract
  • Google Scholar

• 43.

  Wagner M Khoury H Bennetts L Berto P Ehreth J Badia X et al . Appraising the holistic value of lenvatinib for radio-iodine refractory differentiated thyroid cancer: a multi-country study applying pragmatic. BMC Cancer (2017) 17:272. 10.1186/s12885-017-3258-9

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 44.

  Gilabert-Perramon A Torrent-Farnell J Catalan A Prat A Fontanet M Puig-Peiró R et al . Drug evaluation and decision making in Catalonia: development and validation of a methodological framework based on multi-criteria decision analysis (MCDA) for orphan drugs. Int J Technol Assess Health Care (2017) 33:111–20. 10.1017/S0266462317000149

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 45.

  Garau M Marsden G Devlin N Amedeo Mazzanti N Profico A . Applying a multi-criteria decision analysis (MCDA) approach to elicit stakeholders' preferences in Italy. The case of obinutuzumab for rituximab-refractory indolent Non-Hodgkin Lymphoma (iNHL). Office of Health Economics Res. Research Paper 16/08, December (2016).

  • Google Scholar

• 46.

  Schey C Krabbe PFM Postma MJ Connolly MP . Multi-criteria decision analysis (MCDA):testing a proposed MCDA framework for orphan drugs. Orphanet J Rare Dis. (2017) 12:10. 10.1186/s13023-016-0555-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 47.

  Tony M Goetghebeur MM Khoury H Wagner M Deal CL Battista R . A common road map for rational clinical and policy decisionmaking: application of the mcda-based evidem framework to growth hormone use in patients with prader-willi syndrome. Value Health (2011) 14:A328. 10.1016/j.jval.2011.08.525

  • CrossRef
  • Google Scholar

• 48.

  Jimenez A Ais A Acuña L González ME Paco N Gil A . Determining the value of selexipag for the treatment of pulmonary arterial hypertension (PAH) in Spain by multi-criteria decision analysis (MCDA). Value Health (2017) 20:A570. 10.1016/j.jval.2017.08.971

  • CrossRef
  • Google Scholar

• 49.

  Krysanova V Krysanov I Ermakova V . The multicriteria decision analysis of using tetrabenazine for patients with hungtington's disease in Russia. Value Health (2017) 20:A565. 10.1016/j.jval.2017.08.945

  • CrossRef
  • Google Scholar

• 50.

  Badia X Pontes C Fontanet M Obach M Vallano A Torrent J et al . PHP129 - development of an specific evaluation framework for orphandrugs based on Multi-Criteria Decision Analysis (MCDA) for health care decision making in Catalonia. Value Health (2017) 20:A674–4. 10.1016/j.jval.2017.08.1662

  • CrossRef
  • Google Scholar

• 51.

  Gutierrez L Patris J Hutchings A Cowell W . Principles for consistent value assessment and sustainable funding of orphan drugs in Europe. Orphanet J Rare Dis. (2015) 10:53. 10.1186/s13023-015-0269-y

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 52.

  Schlander M Garattini S Holm S Kolominsky-Rabas P Marshall DA Nord E et al . Interventions for Ultra-Rare Disorders (URDs) and the logic of cost effectiveness. Value Health (2015) 18:A6. 10.1016/j.jval.2015.03.043

  • CrossRef
  • Google Scholar

• 53.

  Zhang A Weisse S Chen X . Health Technology Assessment (HTA) for orphan drugs in cost-effectiveness (CE) markets: current development and future trends. Value Health (2016) 19:A601. 10.1016/j.jval.2016.09.1464

  • CrossRef
  • Google Scholar

• 54.

  Nemeth B ., Piniazhko O. Mcda application in central and eastern Europe: selection of the most important criteria based on examples. Value Health (2016) 19:A471. 10.1016/j.jval.2016.09.723

  • CrossRef
  • Google Scholar

• 55.

  Korchagina D Millier A Toumi M Falissard B . Elements of orphan drugs value. Value Health (2016) 19:A600–1. 10.1016/j.jval.2016.09.1463

  • CrossRef
  • Google Scholar

• 56.

  Hutchings A Ethgen O Schmitt C Rollet P . Defining elements of value for rare disease treatments. Value Health (2012) 15:A31.

  • Google Scholar

• 57.

  Goetghebeur MM Wagner M Samaha D O'Neil W Badgley D Castro-Jaramillo H et al . Exploring values of health technology assessment agencies using reflective multicriteria and rare disease case. Int J Technol Assess Health Care (2017) 33:504–20. 10.1017/S0266462317000915

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 58.

  Tony M Wagner M Khoury H Rindress D Papastavros T Oh P et al . Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada. BMC Health Serv Res. (2011) 11:329. 10.1186/1472-6963-11-329

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 59.

  Angelis A Kanavos P . Multiple Criteria Decision Analysis (MCDA) for evaluating new medicines in health technology assessment and beyond: the advance value framework. Soc Sci Med. (2017) 188:137–56. 10.1016/j.socscimed.2017.06.024

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 60.

  Gandjour A . Comment on: “Can the EVIDEM framework tackle issues raised by evaluating treatments for rare diseases: analysis of issues and policies, and context-specific adaptation”. Pharmacoeconomics (2017) 35:603–4. 10.1007/s40273-017-0493-5

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 61.

  Marsh K IJzerman M Thokala P Baltussen R Boysen M Kaló Z et al . Multiple criteria decision analysis for health care DECISION Making—emerging good practices: report 2 of the ISPOR MCDA emerging good practices task force. Value Health (2016) 19:125–37. 10.1016/j.jval.2015.12.016

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 62.

  Broekhuizen H Groothuis-Oudshoorn CG van Til JA Hummel JM IJzerman MJ . A review and classification of approaches for dealing with uncertainty in multi-criteria decision analysis for healthcare decisions. Pharmacoeconomics (2015) 33:445–5. 10.1007/s40273-014-0251-x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 63.

  European Commission Finn Børlum Kristensen . Mapping of HTA methodologies in EU and Norway (2017). Available online at: https://ec.europa.eu/health/sites/health/files/technology_assessment/docs/2018_mapping_methodologies_en.pdf (Accessed March 18, 2018).

  • Google Scholar