Assessing the Preferences for Criteria in Multi-Criteria Decision Analysis in Treatments for Rare Diseases

Background: Increasingly, multi-criteria decision analysis has gained importance as a method by which to assess the value of orphan drugs. However, very little attention has been given to the weight (relative preferences) of the individual criteria used in a framework.

Aims: This study sought to gain an understanding of the preferential weights that should be allocated in a multi-criteria decision analysis framework for orphan drugs, from a multi-stakeholder perspective.

Method: Using key MCDA criteria for orphan drugs reported in the literature, we developed an interactive web-based survey tool to capture preferences for different criteria from a general stakeholder sample who were requested to assign weights from a reimbursement perspective. Each criterion could be assigned a weight on a sliding scale from 0 to 100% as long as the sum of all the criteria was 100%. We subsequently used the interactive tool with an expert focus group, followed up with a group discussion regarding each criterion and their perspectives on the weight that each criterion should be allocated when assessing an orphan drug. The expert focus group participants were then able to adjust their weights, if the group discussion had changed their perspectives.

Results: The interactive tool was completed by 120 general stakeholder sample from a wide range of countries and professional backgrounds and an expert focus group of ten members. The results showed the differences in perspectives on the importance of criteria. Both groups considered Treatment efficacy to be the most important criterion. The general stakeholder sample weighted Treatment safety at 12.03% compared to the expert focus group's average of 20%. The results also demonstrated the value of the group discussion, which provided additional insights into the perspectives on the importance of criteria in assessing orphan drugs.

Conclusion: This study aimed to contribute to the important aspect of preferences for different criteria in MCDA. This study sheds light on the important aspect of the preferences of the different criteria. All respondents agreed on the relative importance of Treatment efficacy and Treatment safety, criteria that are captured in conventional cost-effectiveness studies, but they also expressed the view that in addition to those, several disease-related and drug-related criteria should be included in MCDA frameworks for assessing orphan drugs.

Background

Decision-making in healthcare is usually a process whereby different alternatives are identified and compared to find the best solution based on multiple factors that address the decision-makers' and the organization's expectations (1). With the launching of new and in some cases, innovative treatments such as gene therapy for rare diseases, it's likely that the increasing demands for the reimbursement of such treatments will raise added concerns on the long-term affordability, especially of expensive treatments (2, 3). In order to mitigate the uncertainty of the health benefits of treatments that have been trialed for relatively short periods of time, healthcare systems might have to adopt novel ways of reimbursement decision-making, in addition to cost-effectiveness studies, which is the most commonly used way of assessing the value-add of a new treatment (4, 5). Several decision-making approaches have been developed in healthcare, such as the three-talk model (6) that focuses on shared decision-making in clinical practice, and the use of health outcomes and economic analyses (7). In particular, since the introduction of the Orphan Drug Regulation in Europe in 2000 (8), healthcare systems and reimbursement bodies have had to consider new ways of assessing the value-add of rare disease treatments. Methodologies have ranged from cost-effectiveness models (5, 9), to budget impact models (10) and other tools such as multi-criteria decision analysis (MCDA) (11).

While the European Medicines Agency (EMA) grants the marketing authorization for treatments in Europe (12), the reimbursement decisions are made a local country level (13, 14). Despite the introduction of the Mechanism of Coordinated Access to orphan medicinal products (MoCA) (15), and the commitment by member states to collaborate on improving access to treatments for rare diseases (16), access to orphan drugs across Europe remains variable due to different reimbursement decisions in each country (14, 17). In England, for example, less than 50% of centrally authorized orphan drugs are funded by the National Health Service, with one-third of these recommended by the National Institute for Health and Care Excellence (NICE) (14). The issue of access to orphan drugs might further be complicated by the difference in pricing (18) and pricing negotiations from one country to another, and the metrics by which payers adjudicate the added value of the orphan drug (19). In light of these complex issues, and the increase in expensive orphan drugs (20, 21) different models are being proposed to increase the confidence in the evidence for reimbursement from pre-launch all the way through to post-launch activities. Furthermore, some alternative reimbursement models include patient access schemes (22), reference pricing in pricing negotiations, “cost plus” pricing imposed by the health technology assessment body (23), discount pricing (24) managed entry agreements or managed access agreements (25) and rebate schemes (26), (27). Ultimately, payers are faced with an increasing need to use robust measures with which to assess if new orphan drugs demonstrate value for money, while considering the real clinical benefits and risk of adverse events for the patients, while not wasting medical resources (28).

Recently, MCDA has gained increasing attention in reimbursement decisions for orphan drugs (29, 30) as an alternative to cost-effectiveness. The practice of MCDA in decision-making processes has been used extensively in many industries such as the aviation industry (31) and energy and environmental industries (32) for many years. This particular interest in MCDA is due to the belief that the traditional cost-effectiveness approach used to assess the value of orphan drugs is not robust enough to capture all the multi-dimensional factors that inform on the real benefits of the treatment under review (33, 34). Some of the earliest use of MCDA in healthcare include a study in 1989 to review options in pyelonephritis (35) and for haemoglobinopathies (36). Subsequent use of MCDA in healthcare has been intermittent. To support the development of MCDA for orphan drugs and decision-making by health technology assessment (HTA) bodies and payers, The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Taskforce in multi-criteria decision analysis published recommendations on MCDA (37, 38). However, the ISPOR Taskforce reports are not payer-centric and don't provide insights on how to address the criterion of cost of orphan drugs.

The principle of MCDA is that it provides a matrix or framework whereby multiple factors that describe the disease and treatment under review can be arranged and assessed. The MCDA framework includes the different drug options, the criteria which measure the outcomes of the drugs being assessed, the scaling system by which criteria are measured (allocated a score) and the weights ascribed to the different criteria (38–40). The choice of criteria is open to interpretation depending on the decision question, although there seems to be a common list of criteria that are considered appropriate for assessing orphan drugs using MCDA, as described in the literature (34, 41, 42). Preferences or weights, that describe the relative importance of the criteria against which the different treatments are compared, can then be ascribed to each criterion. Finally, a total “drug score” can be aggregated and used comparatively against other drugs in decision-making and resource allocation (37, 43). Thereby, the MCDA framework provides the opportunity to compare qualitative and quantitative outcomes (44). There are various methods in which to apply MCDA, with the most commonly used ones being the multi-attribute utility theory (MAUT), Analytic Hierarchy Process (AHP), and Outranking (45).

MCDA is being used in some healthcare systems to support the allocation of resources. In England, the Advisory Group for National Specialized Services (AGNSS) was the first health technology assessment body to adopt MCDA to assess orphan drugs (46) using a framework based on efficacy, societal value of the new treatment, the cost of the new treatment, and impact of the new treatment on service delivery (47). AGNSS was subsequently taken over by the Highly Specialized Technologies (HST) group for NICE, who also use the MCDA approach (48). In Spain, the Spanish Agency for Medicines and Health Products developed a MCDA framework based on the EVIDEM framework (49) and similarly in Catalonia (50). The region of Lombardia in Italy has adopted a MCDA approach to regulate the introduction of new health technologies. Their MCDA is based on the EVIDEM framework. The introduction of this formal MCDA model stemmed from the desire to balance goals of continuous innovation with the needs of steady cost containment, and to instill uniformity and transparency in a process that may be highly complex. While subjectivity cannot be completely removed, the framework seeks to minimize discretion in decision making and to produce decisions perceived as legitimate by all the stakeholders (51).

Interestingly, the very earliest reference to MCDA seems to be in a letter that Benjamin Franklin (one of the Founding Fathers of the USA) wrote to an acquaintance in 1772. Benjamin Franklin explained that when faced with multiple factors that influence a decision, it is worth assessing the problem by dividing a page into two columns labeled Pros and Cons under which the decision-maker can list all the necessary factors. He further described his method by saying “When I have thus got them all together in one view, I endeavor to estimate their respective weights; and where I find two, one on each side, that seem equal, I strike them both out. If I find a reason pro equal to two reasons con, I strike out the three. If I judge some two reasons con, equal to some three reasons pro, I strike out the five; and thus proceeding I find at length where the balance lies; and if, after a day or two of further consideration, nothing new that is of importance occurs on either side, I come to a determination accordingly” (52).

Benjamin Franklin's letter alludes to a key point in the use of MCDA, and particularly in assessing orphan drugs—namely the use of weights, or preferences of the different criteria, thereby implying that not all criteria should carry the same importance and that the MCDA model should reflect the weighted preferences. In assessing orphan drugs using MCDA, it could therefore be postulated that Disease severity might be more important than Disease rarity, for example. The weighting is used when aggregating all the criteria to establish the overall “value” (score) of the drug, and the weighting score allocated to a criterion is independent of individual score given to each criterion. For example, the Treatment convenience of a drug might be scored as “high” (meaning that the drug offers easy administration with low associated costs), but the criterion is not considered important by the adjudicating panel, and is therefore given a low weight (preference) in aggregating all the criteria scores for each drug.

While ample research has been performed in recent years on the suitable criteria for inclusion in a MCDA model for orphan drugs (42, 50, 53), less research seems to have been done on the weights or preferences that should be applied to the different criteria. The weighting scales used in research range from 5-point scales (50, 54), to 10-point scales (55) and several studies mention using a 100-point scale (34, 41, 56). A pilot study by Reddy et al. included a group of eight participants (plus the facilitator) to select the criteria that should be considered for a MCDA framework for several public health preventative programmes. They adopted an AHP approach to weight each criterion, thereby obtaining a total score for each under discussion (57). Although this pilot study focused on the choice of topics with which to assess which criteria for public health programmes rather than for rare disease treatments, it highlights the benefits and some of the potential pitfalls of using AHP and MCDA.

The aim of this study was to ascertain the preferences for different criteria from many participants, by comparing the results gathered in a large respondent pool with those from a focus group of experts. The target audience is anyone interested in assessing if MCDA has a role to play in reimbursement decisions for orphan drugs. The authors hope that the research would help further increase the dialogue on the use of alternative methods to assess the value of orphan drugs in addition to cost-effectiveness. Intentionally, the authors did not include a discussion on orphan drug prices, as the key aim was to better understand the weight preferences for all other criteria that might be included in a MCDA framework. In the follow-up study that is planned using a far larger group of respondents from many different countries, the orphan drug prices will be prioritized for debate.

Method

Using key MCDA criteria for orphan drugs reported in the literature (34, 41, 42, 58), we developed a web-based interactive tool (59) to capture preferences for different criteria from a general stakeholder sample attending medical, healthcare or health economic conferences. The criteria were divided into 4 major categories: Disease Burden, Product Development, Clinical Impact and Economic Impact. The tool was designed so that respondents could allocate any weight to the criteria by means of a sliding scale up to a total of 100% and allowing for the inclusion of as many of the criteria as the respondent wanted. i.e., not all criteria had to be allocated a weight. Respondents were asked to use the tool based on what they deemed important in reimbursement decisions. The tool invited respondents to select their field of expertise (e.g., academia, industry) or affiliation (patient representative organization) and the country in which they worked. Once the respondent had completed their weighting, a comparative output showed by how much their inputs deviated from the existing sample average, both as an overall average as well as a diagrammatic representation for each criterion.

We subsequently conducted an expert focus group to gather additional insights from different perspectives on the weights that could be allocated to different criteria when MCDA is used to assess orphan drugs. To this end, we recruited a panel of ten people, all based in Switzerland, comprised of two payers (one each from the state and private insurance sectors), two clinicians specialized in rare diseases (different diseases), two clinical Pharmacists (both working in large teaching hospitals), two health economists (one each from academia and private health insurance), and two patient representatives from different rare disease organizations. Panel recruitment invitations were distributed by personal contact with each participant, with explanations of the study outline, objectives and detailed descriptive information about each criterion included in the tool (summarized in Table 1).

TABLE 1

Table 1. Summarized descriptions of criteria provided to the expert focus group.

We conducted the focus group session according to published focus group methodology (60) in a face-to-face group session held at the hospital in Lausanne, Switzerland. The session was run in English and all participants were fluent English speakers. We initiated the session by describing each criterion and demonstrating how the interactive tool worked. The focus group participants were then invited to each use the interactive tool and to complete the weighting exercise from the perspective of their employment role rather than from a societal perspective. In contrast to the larger study, the focus group participants were asked to provide reasoning for their choices. Following a group discussion of each criterion, the participants had the opportunity to change their weighting if they so desired.

Results

General Stakeholder Sample Weight Preference Data Collection

The interactive tool was completed by 120 respondents from a wide range of professional backgrounds and multiple countries. The country representation and professional affiliations of the large pool respondents are listed in Tables 2, 3, respectively. The largest representation of respondents were from the USA (13%), the Netherlands (12%), Belgium, and Germany (both 10%). The highest number of professional affiliations were Academia and Industry (Pharmaceutical/Medical devices/Diagnostics) at 27% and 23%, respectively.

TABLE 2

Table 2. Countries of origin of the general stakeholder sample.

TABLE 3

Table 3. Professional affiliations of the general stakeholder sample.

Figure 1 shows the average weights of the criteria for all respondents in the general stakeholder sample, which highlights that Treatment efficacy was considered the criterion that deserved the highest weight, followed closely by Disease severity and then Unmet need. The respondents scored Dynamic efficiency (R&D spillover) as the criterion with the least importance.

FIGURE 1

Figure 1. The average weight scores for the general stakeholder sample.

Focus Group Weight Preference Data Collection and Discussion

Table 4 lists the weight preferences for each of the participants of the expert focus group session after the group discussion and having had the opportunity to change their choices if they so wished.

TABLE 4

Table 4. Weight allocations after the focus group discussion for the expert focus group.

The numbers in red cells are those that were decreased and those in green cells were increased. Both the medical doctors and the patient representatives were pleased with their initial choices, while all the other respondents made some changes.

The discussion provided some useful insights herewith summarized according to each criterion. The general consensus was that Disease severity can have a significant negative impact on the health-related quality of life of patients, especially for those with progressive rare, genetic disease. Following the group discussion, a Pharmacist and a Health Economist each decreased their weight allocation from 20 and 18%, respectively, to 10%, while the Payers increased their respective weights from 0 to 5% and 10%. By contrast, Disease rarity was considered to not be a significant criterion, and further downplayed following discussion, as seen in the changes made by 2 members (Pharmacist & Health Economist). It was felt that in light of the Orphan Drug Regulation in Europe, the rarity of disease was already accounted for in the incentives made available to drug manufacturers. As with the general stakeholder sample, the focus group felt that Treatment efficacy was the criterion that deserved the most weight, with Treatment safety in very close second.

Across the group, 70% of the members agreed that Unmet need, which related to the availability of suitable treatment(s), was an important criterion scoring it a weight of an average of 10.57% whereas 30% of the group believed that the criterion should somehow be linked to the drug's efficacy and safety, implying that Unmet need alone was not important enough to warrant a high weight.

The initial weight bestowed upon Level of research undertaken ranged from 5% (Health Economist) to 21% (Pharmacist). In three cases, the weight was increased following the discussion (pharmacist, health economist & public sector payer) and the private-sector payer (private sector) reduced the weight slightly (14–13%). Innovation is a criterion that was not considered at all important and on average was given a weight of only 2%.

Dynamic efficiency (R&D spillover), where the development of a drug for one indication leads to the drug being effective in a second indication or the advent of added benefits from the research and development of the drug, was viewed as an unimportant criterion even though a subsequent indication could influence the price, sales volume and reimbursement negotiations.

Treatment convenience, which relates to the route of administration and frequency of administration, was awarded a relatively low weight. Similarly, Treatment follow-up measures was considered a relatively unimportant criterion, although one of the clinicians pointed out the potential cost implications of a treatment that would require very regular costly monitoring.

The economic criteria of Budget impact/Affordability, Broader economic consequences and Value for money (cost-effectiveness) were awarded a wide range of weights with Budget impact/Affordability being perceived as the most important of the economic criteria. These criteria experienced only minor changes in weights after the discussion. Overall, the average weights were 7, 6, and 5% for Budget impact/Affordability, Broader economic consequences and Value for money (cost-effectiveness), respectively. The average results are shown comparatively for both sets of respondents in Table 5, which highlights significant differences in the perception of the 2 groups (the large group vs the expert group), most notably for Treatment efficacy and Treatment safety that are both favoured by the expert focus group. The variations between the groups with respect to the economic criteria are not significant.

TABLE 5

Table 5. Comparative weights between the two study groups.

Discussion

This study identified the perspective of a large group of people (120 respondents) and an expert focus group via an interactive tool, on the suggested weights of commonly cited criteria suggested for use in assessing the value of orphan drugs by means of MCDA. Although both groups applied the highest weight to Treatment efficacy, the results show that the expert group favored the Clinical Impact criteria (Treatment efficacy, Treatment safety, Treatment convenience, Treatment follow-up measures), and the large group, who did not have the benefit of any discussions, weighted the Disease Burden criteria (Disease severity, Unmet need, Disease rarity) the most.

With each group of respondents basing their choices from a reimbursement perspective as indicated in the instructions, the results suggest that having the opportunity to discuss the criteria and their impact within a small, expert group gives additional insights of the importance, and therefore the weights, of some key criteria. As a result of the group discussion, three members increased the weight for Level of research undertaken on the basis that the more extensive the clinical research, the less the uncertainty of outcomes in the real clinical setting, and therefore the more the criterion should be weighted with a scale that describes different levels of research. These changes made to Level of research undertaken by the expert group, and difference between the average weights allocated to the criterion by both groups suggests that the interactive forum assisted weight allocation by respondents.

One of the main advantages of the group discussion is that it provides a confidential platform for participants to openly comment, explain, disagree, and share their views, perspectives and knowledge (61). Although the focus group session enabled the members to respond in their own words and to share opinions in a non-judgmental way, it was also a structured session aimed at allowing us to obtain information pertinent to the study. In the pilot performed by Sussex et al., they included stakeholders who worked in a pharmaceutical company (GlaxoSmithKline), EU clinical and health economics experts, and representatives of rare diseases patient groups in the EU. In our study, we excluded representation from the pharmaceutical industry sector to ensure that the members felt free to share their opinions. We included payers from the state-funded and private insurance sectors to identify potential differences in perspectives.

One of the points that raised debate among the participants was what they referred to as “double counting,” especially for Unmet need, which initially several members felt was already captured in Disease severity. However, as one of the clinicians described, a patient could present with a very severe disease but for which there are several treatments, such as pulmonary arterial hypertension, in which case the unmet need is not substantial compared to an equally severe disease for which there is no treatment. The participants suggested that another area at risk of “double counting” are the economic criteria, and that clear differentiation needs to be made between Budget impact/Affordability, Broader economic consequences, and Value for money/cost-effectiveness. In particular, the sub-categories for Broader economic consequences would need to be clearly defined.

In exploring the weights for Treatment efficacy and Treatment safety, two members of the focus group commented that at the point that they consider drugs from a reimbursement perspective, the drug would have been approved by the European Medicines Agency or an equivalent organization (Federal Office of Public Health, Switzerland), and therefore it can be assumed that that both the efficacy and safety of the new product meet at least the expected minimum standards, and therefore these 2 criteria should not necessarily be weighted as much as the trend seemed to be.

One of the limitations of this study relates to the nature of the 2 groups. Under ideal conditions, it would be advisable to have multi-country representation in the expert focus group rather than only representation from Switzerland, thereby giving breadth to the discussion and outcomes. Nonetheless, this provided a valuable opportunity to test the concept and approach, which can now be refined and rolled out to a larger, multi-national, multi-stakeholder expert focus group. Similarly, the representation in the large group was not spread evenly by country nor profession. Since the tool is web-based, we plan to further test it by sharing it with a far wider audience, although we believe that group sessions and the discussions they elicit are far more beneficial than individual responses.

A further limitation was that the focus group included only one payer from the public sector, the sector. However, in a country where most medicines used in the community are funded through private insurance or out-of-pocket for patients, it was deemed appropriate to include a payer from the private insurance sector.

A clear omission in this study is the lack of attention given to the pricing of orphan drugs. The ever-increasing cost of orphan drugs is clearly a major cause for concern for payers and healthcare systems. The rationale for this omission was so that the focus would be only on reviewing the weighting preferences for all other criteria, and to avoid the discussion leading to criticism of orphan drug pricing, an issue that is frequently addressed in the media (62–64).

While the Level of research undertaken seemed to be of significant importance to the expert focus group with an average weight of 13%, the general stakeholder sample scored it an average of 7.28%. Other Product Development criteria (Dynamic efficiency (R&D spillover), Innovation) were considered less important by both groups. Dynamic efficiency (R&D spillover) was viewed as a criterion that depended on changes in the future that could only be judged retrospectively, most likely after a substantial lapse of time. The expert focus group discussion led to a heated debate on Innovation. The participants voiced concern for how to judge a drug's level of innovation, with some members referring to the emergence of gene therapy as potentially clouding the understanding of the criterion. The scoring system adopted in France by the Haute autorité de santé (HAS), l'amélioration du service médical rendu (ASMR; the improvement in actual benefit) (65) was considered a “good starting point” although the difference between the levels was deemed unclear. Furthermore, currently there is much debate about real innovation in orphan drugs because if a new treatment comes to market for a disease for which there is no treatment, then purely by comparison, the new treatment might be deemed “innovative,” whilst not necessarily providing as innovative an outcome as necessary to justify some of the high prices demanded (66). It has also been argued that the use of “salami slicing” and re-purposing of drugs merely serve to generate treatments for more indications, but without generating added innovation (67). Payers are increasingly scrutinizing the cost-benefit ratio of new orphan drugs (68).

It is of interest that Treatment convenience was perceived as an unimportant criterion given that if a drug has to be administered frequently in a clinical setting, it is likely to incur far more costs than a treatment that is self-administered by the patient (oral, self-inject). Furthermore, one could argue that a drug that is difficult to administer or that has to be taken multiple times daily is likely to result in poor treatment adherence (69, 70), which has a ripple of effect of suboptimal treatment outcomes, an increased risk of hospitalization (70) and wasted resources, all with negative economic outcomes.

Conclusion

MCDA frameworks are complex models with increasing complexity as more criteria are added to the decision-making process and additional evidence generation is required (71). Some people might consider that replacing a cost-effectiveness analysis with MCDA results in added uncertainty of what the results really imply in the context of reimbursement decisions. On the one hand, HTA bodies have gained many years of experience in interpreting cost-effectiveness model outcomes. However, there is concern that cost-effectiveness does not offer a realistic method to consider an orphan drug's value-add because it does not consider the multi-dimensional factors relating to the rare disease and its treatment. Conversely, MCDA offers added insights in value assessments of orphan drugs, but at this point, there is limited experience using MCDA. This study sheds light on the important aspect of the preferences of the different criteria. All respondents agreed on the relative importance of Treatment efficacy and Treatment safety, which are captured in conventional cost-effectiveness studies, but they also expressed the view that Disease severity, Level of research undertaken (which could contribute to addressing uncertainty) and Unmet need should definitely be included in MCDA frameworks for assessing orphan drugs.

Data Availability Statement

The datasets generated for this study are available on request to the corresponding author.

Author Contributions

CS and MC jointly determined the overall structure of the study and aligned it with the inputs from MP and PK. OT and AV developed, tested and refined the web-based interactive MCDA tool. CS, MC, OT, and AV conducted the initial tool data capture. CS developed the protocol and questions for the interviews, recruited and managed the focus group and analyzed the data and interpretation of the findings. MP and PK reviewed analyses, interpretation and reporting for critical content. All the authors read and approved the final manuscript.

Conflict of Interest

MP has received grants and honoraria from various pharmaceutical companies, but all unrelated to this manuscript. AV and OT were employed by the company Digital Health Outcomes for which no remuneration was made for this work.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

Thank you to OT and AV for developing the interactive MCDA tool. CS would like to thank all the focus group members for their commitment to the project and their valuable insights that will hopefully be applied to this challenging field of defining the criteria and their relative importance in assessing the value-add of orphan drugs and would also like to thank the staff at Centre Hospitalier Universitaire Vaudois for their support with this research.

Abbreviations

AHP, Analytical hierarchy process; ASMR, Amélioration du service médical rendu; HAS, Haute autorité de santé; HTA, Health technology assessment; ISPOR, International Society for Pharmacoeconomics Outcomes Research; MAUT, Multi-attribute utility theory; MCDA, Multi-criteria decision analysis; NICE, National Institute for Health and Care Excellence; R&D, Research and development.

References

1. Guo KL. DECIDE: a decision-making model for more effective decision making by health care managers. Health Care Manag (Frederick). (2008) 27:118–27. doi: 10.1097/01.HCM.0000285046.27290.90

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Jorgensen J, Kefalas P. Reimbursement of licensed cell and gene therapies across the major European healthcare markets. J Mark Access Health Policy. (2015) 3:29321. doi: 10.3402/jmahp.v3.29321

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Salzman R, Cook F, Hunt T, Malech HL, Reilly P, Foss-Campbell B, et al. Addressing the value of gene therapy and enhancing patient access to transformative treatments. Mol Ther. (2018) 26:2717–26. doi: 10.1016/j.ymthe.2018.10.017

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Bonis PAL, Wong JB. A Short Primer on Cost-Effectiveness Analysis. UpToDate. (2011).

Google Scholar

5. Drummond MF. Challenges in the economic evaluation of orphan drugs. Eurohealth. (2015) 14:16–18.

Google Scholar

6. Elwyn G, Frosch D, Thomson R, Joseph-Williams N, Lloyd A, Kinnersley P, et al. Shared decision making: a model for clinical practice. J Gen Intern Med. (2012) 27:1361–7. doi: 10.1007/s11606-012-2077-6

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Levine M, Taylor R, Ryan M, Sculpher M. Decision-making by healthcare payers. Respir Med. (2002) 96(Suppl C):S31–8. doi: 10.1016/S0954-6111(02)80032-4

PubMed Abstract | CrossRef Full Text | Google Scholar

8. European Medicines Agency. Regulation (EC) No 141/2000 of the European Parliament and of the Council of 16 December 1999 on orphan medicinal products (OJ L 18, 22.1.2000, p.1), last amended by Regulation (EC) No 596/2009 (OJ L 188, 18.07.2009, p. 14). (2000). Available online at: https://www.ema.europa.eu/en/human-regulatory/overview/orphan-designation/legal-framework-orphan-designation

9. Schuller Y, Hollak CE, Biegstraaten M. The quality of economic evaluations of ultra-orphan drugs in Europe - a systematic review. Orphanet J Rare Dis. (2015) 10:92. doi: 10.1186/s13023-015-0305-y

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Schlander M, Dintsios CM, Gandjour A. Budgetary impact and cost drivers of drugs for rare and ultrarare diseases. Value Health. (2018) 21:525–31. doi: 10.1016/j.jval.2017.10.015

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Ministry of Health. National Medicines Policy. 2018-2022. (2018). Available online at: https://www.gov.pl/web/zdrowie/rada-ministrow-przyjela-dokumentpolityka-lekowa-panstwa-20182022 (accessed March, 2019).

12. European Medicines Agency. Marketing Authorisation. (2020). Available online at: https://www.ema.europa.eu/en/human-regulatory/marketing-authorisation (accessed March, 2019).

13. Czech M, Baran-Kooiker A, Atikeler K, Demirtshyan M, Gaitova K, Holownia-Voloskova M, et al. A review of rare disease policies and orphan drug reimbursement systems in 12 eurasian countries. Front Public Health. (2019) 7:416. doi: 10.3389/fpubh.2019.00416

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Zamora B, Maignen F, O'Neill P, Mestre-Ferrandiz J, Garau M. Comparing access to orphan medicinal products in Europe. Orphanet J Rare Dis. (2019) 14:95. doi: 10.1186/s13023-019-1078-5

PubMed Abstract | CrossRef Full Text | Google Scholar

15. EURORDIS. Mechanism of Coordinated Access to Orphan Medicinal Products (MoCA). (2017). Available online at: https://www.eurordis.org/content/moca.

PubMed Abstract | Google Scholar

16. MoCA-OMP. Process on Corporate Social Responsibility in the Field of Pharmaceuticals Platform on Access to Medicines in Europe Working Group on Mechanism of Coordinated Access to Orphan Medicinal Products (MoCA-OMP). (2014). Available online at: http://download2.eurordis.org.s3.amazonaws.com/moca/history/WG%20MoCA-OMP%20Transparent%20Value%20Framework.pdf (accesssed March, 2020).

17. DeRidder R, Adriaens C, Mortier M, Quanten A, F A. Mechanism of coordinated access to orphan drugs. Orphanet Journal of Rare Diseases. (2012) 7:A24. doi: 10.1186/1750-1172-7-S2-A24

CrossRef Full Text | Google Scholar

18. Young KE, Soussi I, Hemels M, Toumi M. A comparative study of orphan drug prices in Europe. J Mark Access Health Policy. (2017) 5:1297886. doi: 10.1080/20016689.2017.1297886

PubMed Abstract | CrossRef Full Text | Google Scholar

19. World Health Organization. Negotiating Prices of Drugs For Rare Diseases. (2016) Available online at: https://www.who.int/bulletin/volumes/94/10/15-163519/en/ (accessed March, 2020).

20. Kwon D. How Orphan Drugs Became a Highly Profitable Industry. The Scientist. (2018).

21. CMS Law. Orphan Drugs and Reasons Behind Their Exorbitant Prices. (2020). Available online at: https://www.cms-lawnow.com/ealerts/2020/02/orphan-drugs-and-reasons-behind-their-exorbitant-prices?cc_lang=en (accessed April, 2020).

Google Scholar

22. National Institute for Health and Care Excellence. Patient Access Schemes Liaison Unit. (2020). Available online at: https://www.nice.org.uk/about/what-we-do/patient-access-schemes-liaison-unit (accessed March, 2020).

23. Gammie T, Lu CY, Babar ZU. Access to orphan drugs: a comprehensive review of legislations, regulations and policies in 35 countries. PLoS ONE. (2015) 10:e0140002. doi: 10.1371/journal.pone.0140002

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Cohen D. Paying Twice: the “charitable” Drug With A High Price Tag. (2014) Available online at: https://www.bmj.com/bmj/section-pdf/752706?path=/bmj/348/7945/Feature.full.pdf (accessed February, 2018).

Google Scholar

25. National Institute for Health and Care Excellence. Spinraza Access Agreement Extended. (2019) Available online at: https://www.nice.org.uk/news/article/nice-recommends-first-ever-treatment-for-children-with-rare-muscle-wasting-condition

26. Vogler S, Paris V, and Dimitra P. Ensuring Access to Medicines: How to Redesign Pricing Reimbursement and Procurement? European Observatory on Health Systems and Policies (2018). Available online at: http://www.euro.who.int/__data/assets/pdf_file/0009/379710/PolicyBrief_AUSTRIA_PB30_web_13082018.pdf (accessed March 30, 2020).

Google Scholar

27. Godman B, Malmstrom RE, Diogene E, Gray A, Jayathissa S, Timoney A, et al. Are new models needed to optimize the utilization of new medicines to sustain healthcare systems? Expert Rev Clin Pharmacol. (2015) 8:77–94.

PubMed Abstract | Google Scholar

28. Drugs in 2017: a brief review. Prescrire Int. (2018) 27:110-1.

29. Hughes-Wilson W, Palma A, Schuurman A, Simoens S. Paying for the orphan drug system: break or bend? Is it time for a new evaluation system for payers in Europe to take account of new rare disease treatments? Orphanet J Rare Dis. (2012) 7:74. doi: 10.1186/1750-1172-7-74

CrossRef Full Text | Google Scholar

30. Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Bridging health technology assessment (HTA) and efficient health care decision making with multicriteria decision analysis (MCDA): applying the EVIDEM framework to medicines appraisal. Med Decis Making. (2012) 32:376–88. doi: 10.1177/0272989X11416870

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Janic M, Reggiani A. An Application of the Multiple Criteria Decision Making (MCDM) Analysis to the Selection of a New Hub Airport. Eur J Trans Infrastruct Res. (2002) 2:113–39. doi: 10.18757/ejtir.2002.2.2.3692

CrossRef Full Text | Google Scholar

32. Huang JP, Poh KL, Ang BW. Decision analysis in energy and environmental modeling. Energy. (1995) 20:843–55. doi: 10.1016/0360-5442(95)00036-G

CrossRef Full Text | Google Scholar

33. Campillo-Artero C, Puig-Junoy J, Culyer AJ. Does MCDA trump CEA? Appl Health Econ Health Policy. (2018) 16(Suppl 5):147–151. doi: 10.1007/s40258-018-0373-y

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Iskrov G, Miteva-Katrandzhieva T, Stefanov R. Multi-criteria decision analysis for assessment and appraisal of orphan drugs. Front Public Health. (2016) 4:214. doi: 10.3389/fpubh.2016.00214

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Dolan JG. Medical decision making using the analytic hierarchy process: choice of initial antimicrobial therapy for acute pyelonephritis. Med Decis Making. (1989) 9:51–6. doi: 10.1177/0272989X8900900109

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Le Gales C, Moatti JP. Searching for consensus through multi-criteria decision analysis. Assessment of screening strategies for hemoglobinopathies in southeastern France. Int J Technol Assess Health Care. (1990) 6:430–49. doi: 10.1017/S0266462300001033

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Thokala P, Devlin N, Marsh K, Baltussen R, Boysen M, Kalo Z, et al. Multiple criteria decision analysis for health care decision making–an introduction: report 1 of the ISPOR MCDA emerging good practices task force. Value Health. (2016) 19:1–13. doi: 10.1016/j.jval.2015.12.003

CrossRef Full Text | Google Scholar

38. Marsh K, IJzerman M, Thokala P, Baltussen R, Boysen M, Kalo Z, et al. Multiple criteria decision analysis for health care decision making–emerging good practices: report 2 of the ISPOR MCDA emerging good practices task force. Value Health. (2016) 19:125–37. doi: 10.1016/j.jval.2015.12.016

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Dolan JG. Multi-criteria clinical decision support: A primer on the use of multiple criteria decision making methods to promote evidence-based, patient-centered healthcare. Patient. (2010) 3:229–48. doi: 10.2165/11539470-000000000-00000

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Diaby V, Goeree R. How to use multi-criteria decision analysis methods for reimbursement decision-making in healthcare: a step-by-step guide. Expert Rev Pharmacoecon Outcomes Res. (2014) 14:81–99. doi: 10.1586/14737167.2014.859525

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Sussex J, Rollet P, Garau M, Schmitt C, Kent A, Hutchings A. A pilot study of multicriteria decision analysis for valuing orphan medicines. Value Health. (2013) 16:1163–9. doi: 10.1016/j.jval.2013.10.002

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Friedmann C, Levy P, Hensel P, Hiligsmann M. Using multi-criteria decision analysis to appraise orphan drugs: a systematic review. Expert Rev Pharmacoecon Outcomes Res. (2018) 18:135–46. doi: 10.1080/14737167.2018.1414603

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Department for Communities and Local Government. Multi-Criteria Analysis: A Manual. (2009). Available online at: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/191506/Mult-crisis_analysis_a_manual.pdf.

Google Scholar

44. Drake JI, de Hart JCT, Monleon C, Toro W, Valentim J. Utilization of multiple-criteria decision analysis (MCDA) to support healthcare decision-making FIFARMA 2016. J Mark Access Health Policy. (2017) 5:1360545. doi: 10.1080/20016689.2017.1360545

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Velasquez M, Hester PT. An analysis of multi-criteria decision making methods. Int J Oper Res. (2013) 10:56–66.

Google Scholar

46. Simoens S. Health technologies for rare diseases: does conventional HTA still apply? Expert Rev Pharmacoecon Outcomes Res. (2014) 14:315–7. doi: 10.1586/14737167.2014.906903

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Barham L. Orphan Medicines: Special Treatment Required? (2012). Available online at: http://www.2020health.org/2020health/Publications/publications-2012/Orphan—Drugs.html (accessed November, 2019).

Google Scholar

48. Excellence NIfHaC. Interim Process and Methods of the Highly Specialised Technologies Programme. (2013) Available online at: https://www.nice.org.uk/Media/Default/About/what-we-do/NICE-guidance/NICE-highly-specialised-technologies-guidance/Highly-Specialised-Technologies-Interim-methods-and-process-statements.pdf.

Google Scholar

49. Hernández C, Blázquez A, Gil A. Value of Reflective multi-Criteria Decision Analysis (MCDA) drug evaluation among therapeutic positioning report evaluators from the Spanish Agency of medicines (AEMPS). Value in Health. (2017) 20:A699. doi: 10.1016/j.jval.2017.08.1806

CrossRef Full Text | Google Scholar

50. Guarga L, Badia X, Obach M, Fontanet M, Prat A, Vallano A, et al. Implementing reflective multicriteria decision analysis (MCDA) to assess orphan drugs value in the Catalan Health Service (CatSalut). Orphanet J Rare Dis. (2019) 14:157. doi: 10.1186/s13023-019-1121-6

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Radaelli G, Lettieri E, Masella C, Merlino L, Strada A, Tringali M. Implementation of EUnetHTA core Model(R) in Lombardia: the VTS framework. Int J Technol Assess Health Care. (2014) 30:105–12. doi: 10.1017/S0266462313000639

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Carper TH. Moral or Prudential Algebra. (2017) Available online at: http://www.historycarper.com/1772/09/19/moral-or-prudential-algebra/ (accessed April, 2019).

53. Baran-Kooiker A, Czech M, Kooiker C. Multi-criteria decision analysis (MCDA) models in health technology assessment of orphan drugs-a systematic literature review. Next steps in methodology development? Front Public Health. (2018) 6:287. doi: 10.3389/fpubh.2018.00287

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Gilabert-Perramon A, Torrent-Farnell J, Catalan A, Prat A, Fontanet M, Puig-Peiro R, et al. Drug evaluation and decision making in catalonia: development and validation of a methodological framework based on multi-criteria decision analysis (Mcda) for orphan drugs. Int J Technol Assess Health Care. (2017) 33:111–20. doi: 10.1017/S0266462317000149

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Fedyaeva VK, Omelyanovsky VV, Rebrova O, Khan N, Petrovskaya EV. Mcda approach to ranking rare diseases in russia: preliminary results. Value Health. (2014) 17:A539. doi: 10.1016/j.jval.2014.08.1729

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Piniazhko O, Zalis'ka O, Brezden O. Methodological issues in MCDA for training needed: eliciting stakeholders' value preferences in preferences in Ukraine. Value in Health. (2017) 20:A45.

57. Reddy BP, Kelly MP, Thokala P, Walters SJ, Duenas A. Prioritising public health guidance topics in the National Institute for Health and Care Excellence using the Analytic Hierarchy Process. Public Health. (2014) 128:896–903. doi: 10.1016/j.puhe.2014.07.003

PubMed Abstract | CrossRef Full Text | Google Scholar

58. Zah V. MCDA in the Real World: case studies in Europe and Americas. (2017). Available online at: http://www.pp.u-tokyo.ac.jp/HPTA/events/data/171211/2_Zah-for-web.pdf (accessed March, 2019).

59. Digital Health Outcomes. Web-based MCDA Survey Tool. (2017) Available online at: https://dho-gmastool.firebaseapp.com/tool.

60. Krueger RA, Casey MA. Focus Groups: A Practical Guide for Applied Research. Handbook of Practical Program Evaluation. 4th Edition. Thousand Oaks, CA: Sage Publications Inc (2009).

Google Scholar

61. Leung FH, Savithiri R. Spotlight on focus groups. Can Fam Physician. (2009) 55:218–9.

PubMed Abstract | Google Scholar

62. AHIP. The Rise of Orphan Drugs. (2019) Available online at: https://www.ahip.org/wp-content/uploads/IB_OrphanDrugs-1004.pdf (accessed March, 2020).

Google Scholar

63. Burns J. Orphan Drugs: Way Too Many, Way Too Expensive. (2017) Available online at: https://www.managedcaremag.com/archives/2017/6/orphan-drugs-way-too-many-way-too-expensive (accessed March, 2020).

PubMed Abstract | Google Scholar

64. BBC. A GP's view: Orphan Drug Costs 'prohibitive'. (2013) Available online at: https://www.bbc.com/news/uk-scotland-21032355 (accessed March, 2020).

65. Haute autorité de santé (HAS). Le Service Médical Rendu (SMR) et l'amélioration du Service Médical Rendu (ASMR). (2013) Available online at: https://www.has-sante.fr/jcms/r_1506267/fr/le-service-medical-rendu-smr-et-l-amelioration-du-service-medical-rendu-asmr.

66. Michalopoulos S. Patients: 'Think carefully' Before Questioning Orphan Drugs' Incentives. (2017). Available online at: https://www.euractiv.com/section/health-consumers/news/patients-think-carefully-before-questioning-the-orphan-drugs-incentives/.

67. Lee GL. Orphan Drug Act: Fostering Innovation or Abuse?, in The Source on Healthcare Price and Competition. (2017) Available online at: https://sourceonhealthcare.org/orphan-drug-act-fostering-innovation-or-abuse/.

68. Hall AK, Carlson MR. The current status of orphan drug development in Europe and the US. Intractable Rare Dis Res. (2014) 3:1–7. doi: 10.5582/irdr.3.1

PubMed Abstract | CrossRef Full Text | Google Scholar

69. Srivastava K, Arora A, Kataria A, Cappelleri JC, Sadosky A, Peterson AM. Impact of reducing dosing frequency on adherence to oral therapies: a literature review and meta-analysis. Patient Prefer Adherence. (2013) 7:419–34. doi: 10.2147/PPA.S44646

PubMed Abstract | CrossRef Full Text | Google Scholar

70. Jimmy B, Jose J. Patient medication adherence: measures in daily practice. Oman Med J. (2011) 26:155–9. doi: 10.5001/omj.2011.38

PubMed Abstract | CrossRef Full Text | Google Scholar

71. Oliveira MD, Mataloto I, Kanavos P. Multi-criteria decision analysis for health technology assessment: addressing methodological challenges to improve the state of the art. Eur J Health Econ. (2019) 20:891–918. doi: 10.1007/s10198-019-01052-3

PubMed Abstract | CrossRef Full Text | Google Scholar