Application of Multi-Criteria Decision Analysis Techniques for Informing Select Agent Designation and Decision Making

The Centers for Disease Control and Prevention (CDC) Select Agent Program establishes a list of biological agents and toxins that potentially threaten public health and safety, the procedures governing the possession, utilization, and transfer of those agents, and training requirements for entities working with them. Every 2 years the Program reviews the select agent list, utilizing subject matter expert (SME) assessments to rank the agents. In this study, we explore the applicability of multi-criteria decision analysis (MCDA) techniques and logic tree analysis to support the CDC Select Agent Program biennial review process, applying the approach broadly to include non-select agents to evaluate its generality. We conducted a literature search for over 70 pathogens against 15 criteria for assessing public health and bioterrorism risk and documented the findings for archiving. The most prominent data gaps were found for aerosol stability and human infectious dose by inhalation and ingestion routes. Technical review of published data and associated scoring recommendations by pathogen-specific SMEs was found to be critical for accuracy, particularly for pathogens with very few known cases, or where proxy data (e.g., from animal models or similar organisms) were used to address data gaps. Analysis of results obtained from a two-dimensional plot of weighted scores for difficulty of attack (i.e., exposure and production criteria) vs. consequences of an attack (i.e., consequence and mitigation criteria) provided greater fidelity for understanding agent placement compared to a 1-to-n ranking and was used to define a region in the upper right-hand quadrant for identifying pathogens for consideration as select agents. A sensitivity analysis varied the numerical weights attributed to various properties of the pathogens to identify potential quantitative (x and y) thresholds for classifying select agents. The results indicate while there is some clustering of agent scores to suggest thresholds, there are still pathogens that score close to any threshold, suggesting that thresholding “by eye” may not be sufficient. The sensitivity analysis indicates quantitative thresholds are plausible, and there is good agreement of the analytical results with select agent designations. A second analytical approach that applied the data using a logic tree format to rule out pathogens for consideration as select agents arrived at similar conclusions.

Introduction

In the 1990s, Congress recognized the need for regulations to ensure the biosafety and biosecurity of activities that involved the use of hazardous biological agents and toxins, and the facilities in which these activities occur. In 1996 Congress passed the Antiterrorism and Effective Death Penalty Act (Public Law 104-132, 1996); Section 511 of this Act addressed the heightened concern about the ease with which disease-causing agents could be obtained. The legislation directed the Department of Health and Human Services (HHS) to establish a list of biological agents and toxins (i.e., HHS select agents) that could potentially threaten public health and safety, establish procedures governing the transfer of those agents, and training requirements for entities working with them. This led to the promulgation of the Select Agent Regulation. HHS delegated the authority to administer this regulation to CDC (i.e., CDC Select Agent Program).

After the intentional release of anthrax spores through the U.S. mail in the fall of 2001, regulations to control these agents were enhanced (Morse, 2015). Congress significantly strengthened oversight of select agents with the passage of the Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism Act of 2001 (USA PATRIOT Act) (Public Law 107-56, 2001) and the Public Health Security and Bioterrorism Preparedness and Response Act of 2002 (Bioterrorism Act) (Public Law 107-188, 2002). The USA PATRIOT Act restricted who could have access to select agents, and the Bioterrorism Act created increased safeguards and security measures for select agents. The Bioterrorism Act required that the list of HHS select agents and toxins be reviewed and republished at least biennially, and that the HHS Secretary consider the following criteria in determining whether to include a biological agent or toxin on the list: the effect on human health of exposure to an agent or toxin; the degree of contagiousness of the agent and the methods by which the agent or toxin is transferred to humans; the availability and effectiveness of pharmacotherapies and immunizations to treat and prevent illnesses resulting from an agent or toxin; and any other criteria including the needs of children and other vulnerable populations that the HHS Secretary deems relevant (Public Law 107-188, 2002). The CDC Select Agent Program utilizes these and additional criteria (CDC, HHS, 2020) together with SME guidance to support the biennial review process.

To support the CDC Select Agent Program biennial review process, we developed and evaluated two risk-based analytical approaches for classifying bacteria and viruses as HHS select agents: a multi-criteria decision analysis (MCDA) framework and a decision support framework (DSF) in a logic tree format with a focus on agent properties and public health. Factors such as financial or opportunity cost were not considered. Previous efforts by the CDC Select Agent Program relied on SME assessments to rank the agents and have not applied the approach broadly to include non-select agent pathogens due to the additional burden placed on the SMEs. The analytical approaches presented here seek to provide a system engineering approach and decision analysis techniques for assessing bioterrorism risk, and to reduce the burden on SMEs by documenting the supporting data from the peer-reviewed literature in archivable data sheets. We applied the methodology broadly to evaluate the general applicability of the approach by including a variety of non-select agents in the assessment, such as Risk Group 3 and 4 pathogens (NIH, 2019), emerging pathogens, and former select agents that had been removed from the list by the Select Agent Program. The results of this initial assessment of such an approach for classifying select agents is presented here.

MCDA is a sub-discipline of operations research and is comprised of a set of methodological approaches that are well described (Keeney and Raiffa, 1993; Greco et al., 2005; Hwang and Yoon, 2012) and well-documented in the literature for conducting structured risk assessments (Linkov et al., 2006). MCDA offers a transparent method for conducting risk assessments that serve to quantify and communicate risk and support a risk management-based approach to decision making. This method allows for data uncertainty, can combine multiple information sources including those based upon expert judgement, and is simple in concept and amenable to a user-friendly software tool. Disadvantages of MCDA include those cited for qualitative methodologies, i.e., the lack of absolute measurements and potential for rank reversal (i.e., the possibility that adding a new option to, or removing an option from, a set of ranked options and redoing the analysis might change, or even reverse, the relative ordering of the other options, even though none of their attributes had been changed) (Cox et al., 2005). The use of MCDA for risk-based decision making has been described for environmental applications (Kiker et al., 2005; Steele et al., 2009), healthcare (Velasquez and Hester, 2013; Thokala et al., 2016), as well as emerging threats to animal and plant health (Cook and Proctor, 2007) and foodborne pathogens (Ruzante et al., 2010).

Methods

Analytical Framework

The three basic elements of an MCDA approach are (1) key values or criteria that form the basis for decision making, (2) the analytical framework that relates the key values and measures to the decision outputs, and (3) the underlying data and inputs that inform the scoring. To evaluate options against the key values, every criterion has a scoring scale based on linear ordinal values (e.g., 0–10), definitions for each numerical score, and an assigned weight. Options are scored on the basis of underlying data or SME input using the provided scoring definitions; weights are typically assigned through SME elicitation. The analytical framework describes the hierarchy for how the overall scores for each option are tallied based on the underlying scores; a common approach is to combine all the criteria and weights into a single score (A) by summing all the weighted numerical values (a_ij,w_i), where a_ij represents a criteria score and w_j is the criteria weighting value:

$A= {\displaystyle \sum }_{j=1}^{n}aij•wj$

Calculation of overall scores enables comparison of considered options through means such as rank order, where the options are ranked 1–n. Common applications of the MCDA technique are to select a single preferred outcome, which is typically the highest-ranking option; however, other output options are possible (Greco et al., 2005).

The criteria and hierarchy used are shown in Figure 1. This hierarchy includes the criteria set forth by the Bioterrorism Act and Select Agent Regulation (Public Law 107-188, 2002; CDC, HHS, 2020) and breaks down the agent score into key elements of bioterrorism risk: i.e., difficulty of a successful attack and consequence. These elements are further broken down into factors that measure the relative difficulty of agent production and population exposure for inhalation and ingestion scenarios, and the impact on human consequence and potential mitigation measures. Four criteria—Ease of Production, Degree of Pathogenicity, Burden on Health Care Systems, and Decon and Restoration—incorporate sub-criteria. For these criteria, scores are calculated as the average of the sub-scores. For example, the final score for the Ease of Production criteria is the average of the scores for Production Skill Required, Growth Conditions, Growth Time, Production Yield, and Storage Stability. Calculating the scores in this way assigns all the sub criteria of any given criterion equal weight.

FIGURE 1

FIGURE 1. Summary of the criteria and hierarchy captured in the MCDA framework and fact sheets.

For each criterion, a scoring scale of 0–10 was adopted. The scoring scale reflects relative concern as it pertains to the agent designation as a select agent, with 0 corresponding to lowest concern and 10 corresponding to highest concern. For simplicity, a linear scale was chosen for the initial evaluation; however, the scale can be non-linear. Definitions are provided for each of the criteria shown in Figure 1 and the even-numbered scoring options (0, 2, 4, 6, 8, and 10) (Supplementary Table S1). Although odd-numbered scores are allowed, no specific definitions are provided for them; however, odd numbered scores could account for options intermediate in concern between the two adjacent defined scores, based on SME judgement.

Two views of the data were explored for assessing the relative ranking of the agents: 1) a one-dimensional “risk ranking” whereby the total weighted sum for each agent is tallied and the agents are ranked from lowest to highest; and 2) a two-dimensional “risk style” plot whereby the weighted sum of the sub-scores for the “production” plus “exposure” branches of the hierarchy are plotted against the weighted sum of the sub-scores for the “consequences” plus “mitigation” branches of the hierarchy (see Figure 1). This plot is similar to standard risk plots of likelihood versus consequence.

To enable comparison of results using different weighting values, we utilized normalized scores, whereby the total or sub-total scores are normalized to those of a hypothetical “maximal agent” that would receive 10s for all the criteria scores.

Agent Fact Sheets

To document the data used for scoring pathogens against the criteria shown in Figure 1, we developed fact sheets for 73 pathogens (Table 1); the list includes 36 HHS select agents, nine former HHS select agents, 21 Risk Group 3 and 4 pathogens, and four emerging pathogens. Five HHS select agents identified for removal from the current select agent list (B. anthracis Pasteur strain, Brucella abortus, Brucella suis, Coxiella burnetii, and Rickettsia prowazekii) were included as current select agents as their removal was not formally approved at the time of this study (CDC, HHS, 2020).

TABLE 1

TABLE 1. HHS Select and non-select agents evaluated in this study.

Development of the fact sheets utilized peer-reviewed open literature such as Medline, PubMed, Google Scholar and other unclassified data followed by extensive review by several SMEs who work with the specific pathogen. In cases where there were data gaps, data for similar organisms (e.g., same virus family) or appropriate animal models were captured to support scoring. When available, case fatality rates for the U. S. or countries with comparable levels of health care were used for scoring. In all cases, SME review was relied upon to provide expert judgement and concurrence on the best available data or basis for scoring. All agent fact sheets were reviewed by one or more SMEs identified by the CDC Select Agent Program. SMEs were asked to review the data provided on the fact sheets for accuracy and relevance, as well as the scores assigned to each data category. Comments received from the SMEs were incorporated into the sheets and scoring adjusted, as necessary.

Criteria Weighting

To determine appropriate weighting of criteria, SMEs were asked to rank order the 15 criteria, where one describes the most important criterion and 15 describes the least important criterion for determining whether an agent should be a select agent. This approach was chosen to gauge the level of agreement among the SMEs; the average of the results obtained from 13 SMEs for each of the criteria is shown in Table 2. The results showed good agreement among SMEs in some areas; typically, at the two ends of the ranking spectrum. The SD ranged between 1.2 and 3.8. The highest agreement in ranking occurred for Degree of Pathogenicity, Availability of Medical Countermeasures, Rate of Transmission, and Dissemination Efficacy. Ease of Production and Ability to Genetically Manipulate also showed good agreement. The greatest disagreement in ranking occurred for Long-term Effects, Matrix Stability and Aerosol Stability. Case Fatality Rate and Severity of Illness also showed higher SD; however, 10 of 13 SMEs ranked Case Fatality Rate as number 1 and 11 of 13 SMEs ranked Severity of Illness in the top 3. The reasons for the disagreements were not explored further but may reflect differences in available information or exemplars used to inform individual ranking choices, among others (Hollingshead, 1996).

TABLE 2

TABLE 2. Criteria rank ordering by SMEs and assignment to one of three weighting groups (A, B or C).

Because there was a lack of consensus on rank order, SMEs agreed to group similarly ranked criteria together and assign equal weighting values to the group. The results of the ranking exercise suggested that the criteria could be reasonably grouped into three categories for the assignment of weights, based on where the breaks occur in the average rankings, and for simplicity (Table 2). The average ranking results were used to group the top five criteria into Group A, the next four criteria into Group B, and the last six criteria into Group C (5/4/6 grouping). An alternative grouping was also evaluated whereby the top three criteria were placed in Group A, the next three in Group B, and the last nine into Group C (3/3/9 grouping). Both weighting options were explored to test the sensitivity of the results to how the groupings are specified. A baseline case of weighting values was tested where the ratio of the weights for Group A: Group B: Group C were set to be 3:2:1; specifically, a weighted value of three was defined for all criteria in group A, a weighted value of 2 for all criteria in group B, and a weighted value of 1 for all criteria in group C.

Sensitivity Analysis

To study the sensitivity of the two-dimensional results to the choice of weights—both the numerical value assigned to the weights and the way the weight groups are defined (e.g., 5/4/6 and 3/3/9 groupings)—a sensitivity analysis was conducted where the numerical weighting values were randomly varied over a broad range (up to 100:1) while the agent scoring values were kept constant, creating thousands of weighting scenarios. For this analysis, the relative weights of Groups A, B and C are constrained as weight (Group A) ≥ weight (Group B) ≥ weight (Group C). Weights for Groups A and B were allowed to vary between one and 100, while the weight for Group C was fixed at one for all scenarios.

A secondary objective of the analysis was to investigate a quantitative basis for providing minimum scoring thresholds needed for designating a Select Agent. To evaluate this, a “select agent scoring region” was designated as the upper right quadrant in the two-dimensional plot formed by graphing difficulty of attack (weighted sum of exposure and production criteria) vs. consequences (weighted sum of consequence and mitigation criteria). This is the region where agents with the highest combined scores for Production, Exposure, Consequence and Mitigation would fall and thus they would represent the highest bioterrorism risk. To ensure the weighting scenario library does not include scenarios that allow clearly non-Select Agent scores to fall in the Select Agent region, we defined four simulated test agents that should not classify as select agents, and constrained the scenario set to only those scenarios where the test agents’ scores fall outside the select agent scoring region.

The four simulated test agents were assigned scores consistent with the following properties:

• Agent 1 is not a human pathogen

• Agent 2 does not cause severe disease and infection is non-fatal to humans

• Agent 3 does not cause infection in humans by inhalation or ingestion routes

• Agent 4 has characteristics of both Agents 2 and 3 (i.e., not infectious to humans by inhalation or ingestion, and does not cause severe disease nor mortality)

For the test agents, the targeted traits were scored low (typically 0) in the tool, while all other traits were scored as high as possible (10). In this way, the test agents serve as extreme cases of non-select agents; even though they may be perfect in all other attributes (e.g., very easy to produce in large quantities), they cannot be select agents because they don’t meet critical requirements. The scoring basis used to generate the test agents is shown in Table 3.

TABLE 3

TABLE 3. Scoring basis used to generate four simulated test agents.

The sensitivity analysis was run using scores for all 73 pathogens plus the four simulated test agents, with the select agent scoring region in the 2-D plot format initially defined by the lines x = 0.50 and y = 0.50 and then varied to optimize the size of the select agent scoring region to be as large as possible without artificially constraining the number of viable scenarios (i.e., test agent scores fall outside the Select Agent scoring region). We found that the thresholds of x = 0.36 and y = 0.56 for both weighting schemes studied (5/4/6 and 3/3/9) provided the optimal trade-off between permitting the largest number of viable scenarios while also allowing for a potentially permissive select agent window.

Decision Support Framework

The DSF approach applies key criteria using a logic tree format to identify pathogens which may be of sufficiently low concern that they can be ruled out from consideration as a select agent, which is different from the MCDA criteria. The DSF is complementary and critical as a precursor to the analytical framework in order to avoid the possible unintended numerical equivalences from weighted sums. Knowledgeable adversaries will look at an entire proposed scenario to identify any “show-stoppers” that cannot be overcome, and toss out those that are infeasible, while seeking ways to overcome any undesirable characteristics of the remaining potentially feasible candidate scenarios. Frequently, the scenario selected for exploitation is not the one with the most advantages, but rather the one with the least disadvantages (in number and/or severity) so long as it meets the adversary’s basic criteria for desired outcome (often consequences or publicity). As a first step, the DSF considers the potential impact associated with regulating an agent versus the public health implications and practices. This avoids not regulating SARS-CoV-2 during the current pandemic to enable public health, health care practitioners, medical countermeasures and diagnostics developers, and scientists to perform their mission effectively. Using this approach, if a pathogen does not meet a threshold value for any one of the criteria set, it is deemed of low concern and thus is not considered a select agent. Those pathogens that exceed all criteria thresholds are considered for select agent status. Criteria include both risk-based (e.g., severity of illness and rate of transmission) and non-risk-based criteria such as clinical prevalence. The logic tree is shown schematically in Figure 2. SME judgment based on data captured in the agent fact sheets provided the basis for scoring. In general, criteria which received a score of zero or two typically served as a basis for a “low concern” qualitative assessment. In contrast to the MCDA approach, which uses a graded scoring system for ranking agents, the DSF approach can rule out an agent from select agent consideration using a single (low scoring) criterion. Many of the criteria overlap between the two approaches; however, there are key differences, such as the inclusion of clinical prevalence in the DSF approach.

FIGURE 2

FIGURE 2. Decision Support Framework for assignments of select and non-select agents.

Results

Data Gaps and Quality

When considering such a large number of micro-organisms across a broad range of attributes, data gaps and variability in data quality are inevitable. Data availability in the open literature tends to parallel scientific inquiry for the organism; for example, aerosol studies are more prevalent for pathogens known or suspected to be infectious by the aerosol route, and surface stability data are often more available for pathogens where fomite transmission is a concern. Overall, we found the most prominent data gaps were in aerosol stability and human infectious dose by inhalation and ingestion routes. For aerosol stability data, we typically used data for similar organisms (e.g., same virus family) as proxies, and infectious dose data for animal models was leveraged where available and needed to address human data gaps.

In the case of rare diseases where there is not a lot of human case data available, the data that are published tend to be for the severe cases reported by hospitals. Out of the 73 select and non-select agents examined (Table 1), six had fewer than 10 known cases: B. cereus biovar anthracis (zero cases), Chapare virus (one case) (Delgado et al., 2008), Flexal virus (two cases) (Archer and Rico Hesse, 2002), Lujo virus (5 cases) (Paweska et al., 2009), Sabia virus (four cases) (Barry et al., 1995; Lisieux et al., 1994) and Semliki Forest virus (one case) (Krauss, 2003; PHAC 2011). For four of these agents, the known cases were severe and the extent of mild or asymptomatic cases is largely unknown. Because it is difficult to extrapolate with confidence from a few cases to a potential large population exposure that may result from a bioterror attack, severity of illness and case fatality rates (CFRs) are poorly predicted from these data, and extrapolation tends to overestimate these parameters. To account for these factors, the Severity of Illness scoring definition was modified for a score of four to include “or <10 known cases” and the Case Fatality Rate definition for a score of two to include “or <10 known cases.” The changes in scoring definition had no effect on agents associated with mild disease (i.e., Flexal and Semliki Forest viruses), and significantly lowered scores for agents associated with rare severe disease (i.e., B. cereus biovar anthracis, Chapare virus, Lujo virus, Sabia virus). This approach was reviewed by the SMEs and felt to be consistent with current understanding of these organisms, given the gaps in data.

An additional challenge of applying the severity of illness and CFR criteria occurred for agents with a high proportion of asymptomatic or mild cases. CFRs for these agents may appear disproportionately high if they do not consider unreported asymptomatic and mild cases. To assess whether the CFR scores for agents with large proportions of asymptomatic or mild cases were too high, we reviewed the CFR scores for 15 agents and adjusted the scores for those with large asymptomatic or mild cases by multiplying the CFR by the percentage of severe cases (Table 4).

TABLE 4

TABLE 4. Agents causing large numbers of asymptomatic or mild cases.

Technical review of published data by SMEs expert in the specific pathogen was found to be important in general and in particular for pathogens with many data gaps, and where the use of proxy data for similar viruses led to high apparent total scores. Technical reviewers of limited published data available for White Water Arroyo virus, for example, concluded that it was questionable whether the virus causes human disease (Reignier et al., 2008; Zong et al., 2014). Similarly, scores for MERS-CoV based on similarity to other coronaviruses were adjusted upon review and updated to include recent data showing limited aerosol transmission likelihood (Kim et al., 2016; Arabi et al., 2017). While hantavirus is virulent when found in nature, technical reviewers indicated much lower virulence occurs when the virus is propagated in the laboratory (Lundkvist et al., 1997; McElroy et al., 2002; Nemirov et al., 2003; Safronetz et al., 2014; Prescott et al., 2017), suggesting much lower scoring is appropriate under public health consequences for this family of viruses.

Unweighted Rankings

Initial inspection of the risk-ranking results (data not shown) indicated that, in general, the Tier 1 select agents were found at the top of the rank-ordered list (unweighted score 0.64–0.77), other select agents were located in the middle section (unweighted score 0.45–0.70), and non-select agents at the bottom (unweighted score 0.09–0.56). However, there were a few exceptions. For example, R. prowazekii (unweighted score 0.35) and B. anthracis Pasteur strain (unweighted score 0.31) fall near the bottom of the list. C. botulinum (unweighted score 0.44), a Tier 1 select agent, falls in the middle of the list; although an outlier among the Tier 1 agents, this is consistent with low scores in many consequence categories.

Analysis of the two-dimensional plot of the unweighted data (Figure 3A) shows similar groupings, where in this case the select agents generally fall into the upper right-hand quadrant of the plot, except for R. prowazekii, B. anthracis Pasteur strain and C. botulinum, which appear as outliers, and fall into other quadrants. When considering non-select agents, both the one- and two-dimensional plots indicated that, although there were general trends in the data that were consistent with current classifications, there are no sharp breaks in scoring that could serve as a basis or threshold for classifying select agents. Instead, the plots represent a continuum of scores. Any designation of a minimal score – whether as the total score in the one-dimensional plot, or as sub-scores corresponding to the x- and y-axes in the two-dimensional plot—will result in some exceptions to current classifications. For example, in the two-dimensional plot, if the lowest x-axis score for a select agent is designated as x = 0.48 and the y-axis score as y = 0.35 (Figure 3A), all select agents score in the select agent scoring region except B. abortus, B. melitensis, B. suis, C. botulinum, C. burnetii, R. prowazekii, B. anthracis Pasteur strain, and VEEV (1AB and 1C). All non-select agents score as non-select agents.

FIGURE 3

FIGURE 3. Two-dimensional plots of unweighted (A), 5/4/6 grouping (B) and 3/3/9 grouping (C) normalized scoring results.

Because the two-dimensional plotted results provided greater fidelity for understanding agent placement compared to the one-dimensional risk ranking results, the one-dimensional approach was dropped, and the two-dimensional approach was pursued and is described in more detail below.

Weighted Rankings

To evaluate the impact of applying criteria weights on the scoring results, initial numerical assignments for weights were chosen to be A:B:C = 3:2:1. Subsequent analysis of the impact of numerical weight assignments on the scoring assignments were evaluated more broadly (see below). Figure 3B shows the results of the weighted data in the two-dimensional format for the 5/4/6 grouping of criteria. Comparison of the results with the unweighted data indicate that in addition to the three select agents falling as outliers outside the upper right-hand quadrant (R. prowazekii, B. anthracis Pasteur strain and C. botulinum), several additional select agents (B. abortus, B. melitensis, B. suis, C. burnetii and VEEV (1AB and 1C) have pulled away slightly from the others. Also, a wider gap between select agents and non-select agents in the vertical dimension is observed with the weighted data, allowing for a potential threshold to fall between CCHFV and MERS-CoV. However, there was less of a gap in the data in the horizontal dimension, making the classification results for the borderline agents sensitive to the threshold choice. For example, if the lowest x-axis score for a select agent is designated as 0.41 and the y-axis score as 0.42 (dotted lines shown in Figure 3B), all select agents fall in the select agent scoring region except B. abortus, B. melitensis, B. suis, C. botulinum, C. burnetii, R. prowazekii, B. anthracis Pasteur strain, and VEEV (1AB and 1C). All non-select agents score as non-select agents.

The alternate 3/3/9 grouping for the criteria using A:B:C = 3:2:1 was analyzed to explore its impact on the results. The two-dimensional plot (Figure 3C) depicts the results of this analysis. If the lowest x-axis score for a select agent was designated as 0.43 and y-axis score as 0.40 (dotted lines in Figure 3C), all select agents reclassified as select agents except for B. abortus, B. melitensis, B. suis, C. botulinum, C. burnetii, R. prowazekii, B. anthracis Pasteur strain and VEEV (1AB and 1C). All non-select agents reclassified as non-select agents. These results were similar to those obtained with the 5/4/6 grouping.

The results from the weighted analysis using two different weighting schemes indicated that the designation of a “select agent region” in the two-dimensional plots, for the purposes of classifying select agents and distinguishing them from non-select agents, is possible but is sensitive to how the thresholds are set. Some select and non-select agent scores are quite close to the thresholding boundaries. To explore the sensitivity of the two-dimensional results to the choice of weights, and to investigate a more rigorous basis for defining the boundaries of the select agent region in the two-dimensional plot, we conducted a sensitivity analysis where the numerical weighting values were randomly varied over a broad range (up to 100:1) while the agent scoring values were kept constant, creating thousands of weighting scenarios. Similar Monte Carlo approaches for evaluating the sensitivity of weighting choices using rank order, additive sum models and multi-criteria decision models have been described previously and found helpful as an aid for supporting decision making (Butler et al., 1997).

Sensitivity Analysis

Using the threshold optimized to permit the largest number of scenarios while also allowing for a potentially permissive select agent window (x ≥ 0.36 and y ≥ 0.56) to define the select agent scoring region, we found that 97% of scenarios were viable for the 5/4/6 ABC grouping, and 96% of scenarios were viable for the 3/3/9 grouping. In this test, viable scenarios are those that exclude the four Test Agents from the select agent scoring region. The rejected scenarios differed slightly between the two groupings, but typically encompassed the lower scoring ratios (data not shown). Two-dimensional plots of the median scores with error bars indicating the 5th and 95th percentiles are shown for the 5/4/6 and 3/3/9 groupings in Figures 4A,B, respectively. The percentage of scenarios where each agent scored in the select agent scoring region for the 5/4/6 and 3/3/9 groupings are shown in Table 5. For the 5/4/6 grouping, the median scores for 20 agents landed within the select agent region for 100% of the scenarios and another six agents landed in the select agent region for >45% of the scenarios (Table 5). The drop-off was quite sharp after this, as only two more agents (B. cereus biovar anthracis [9.6%] and TBEV CE [0.6%]) spent any time at all in the select agent region.

FIGURE 4

FIGURE 4. Two-dimensional plot of normalized median scores with error bars (5th and 95th percentiles) for (A) 5/4/6 grouping results and (B) 3/3/9 grouping results.

TABLE 5

TABLE 5. Percentage of scenarios in the sensitivity analysis where the agent score was within the select agent region as defined by thresholds x ≥ 0.36 and y ≥ 0.56.

The sharp drop-off suggested a possible quantitative threshold for classifying select agents. A lenient threshold could be >0%; a more restrictive threshold could be >45%. If we consider the break point to be >5% of scenarios in the select agent region, then the classification results strongly parallel the two-dimensional 5/4/6 classification results using the 3:2:1 weighting.

Similar results were observed for the 3/3/9 grouping (Figure 4B). In this case, the median scores for 10 agents landed within the select agent region for 100% of the scenarios, and another 16 agents landed in the select agent region for >56% of scenarios (Table 5). The drop-off was quite sharp after this, as only five more agents spent any time at all in the select agent region (ranges between 0.1–6.9%). If we use a break point of >5% of scenarios in the select agent region, the classification results are similar to the two-dimensional 3/3/9 classification results using the 3:2:1 weighting except now B. cereus biovar anthracis is also excluded from the select agent region.

The results of the sensitivity analysis indicated that the reclassification of select and non-select agents using the two-dimensional method was robust over a broad range of weighting scenarios. In particular, the results supported the observations made using the 3:2:1 weighting assignment for both the 5/4/6 and 3/3/9 criteria groupings.

Comparison of the MCDA and DSF approaches

To compare the results from the MCDA method with those using a different approach (logic tree) to identify select agents, the 73 select and non-select agents were evaluated using the DSF approach. The results are summarized in Figure 2. A comparison of results using this method with those obtained using the weighted two-dimensional MCDA approach indicated that both approaches classified B. anthracis Pasteur strain, B. abortus, B. melitensis, B. suis, C. botulinum, C. burnetii, R. prowazekii and VEEV (1AB and 1C) as non-select agents.

Discussion

There are several reports in the open literature wherein authors describe approaches for evaluating pathogens for bioterrorism preparedness and other considerations. Rotz et al., 2002 utilized four high-level categories and 11 factors to evaluate 16 pathogens and toxins for designation as Category A, B or C agents. Cieslak et al., 2018 described a method using 12 attributes scored on a scale of zero to three for assessing the weapon potential of 33 pathogens and toxins. Casadevall and Pirofski, 2004, Casadevall and Pirofski, 2006 proposed using five criteria for assessing weapon potential, while MacIntyre et al. (2006), utilized 10 criteria for determining a risk priority score for six agents. Menrath et al., 2014 compared these and other approaches for evaluating and ranking biological agents. The aforementioned approaches faced many of the same challenges we did when we began this effort, including:

1) Lack of data to support measurements of critical criteria. Casadevall and Pirofski, 2004 noted this problem was a major limitation of their approach. As much as possible, we chose criteria and defined scoring options so that open-source data could be used for scoring. The areas where we found it most difficult to find data for all agents included infectious dose in humans and aerosol stability.

2) Reliance on SMEs to score agents. Several approaches relied heavily on SMEs for scoring. For example, Cieslak et al., 2018 relied on SMEs affiliated with the biodefense establishment of member nations of NATO’s Biological Medical Advisory Panel to score all the parameters for each organism. This likely contributed to the observed spread in scores, as it is challenging to find SMEs with expertise in all the factors assessed for an agent. For our approach, we initially scored agents using data from literature searches, then vetted the data and scoring with SMEs. We had more than one SME review each fact sheet, enabling each SME to review those areas where they have the most expertise.

3) Limited assessment scale (typically 0–3) with generalized scoring definitions (e.g., low/med/high). This factor, combined with a heavy reliance on SMEs, can result in widespread scores, as was observed by Cieslak et al., 2018. Our approach provided specific definitions for each scoring choice, literature data to inform scoring and SME verification of the resulting scores.

4) Use of criteria which are very difficult to accurately measure. Examples from the literature include public perception (Rotz et al., 2002) and a “terror factor” (Casadevall and Pirofski, 2004). We did not include these as criteria, as they are hard to predict, difficult to quantify and will vary over time. Any pathogen, if used to intentionally expose a population as part of a bioterrorism plot, is likely to generate some level of terror, disruption or public perceptions of fear.

In addition, the Select Agent biennial review process has often become highly contentious, as reflected in the often highly adverse reactions to efforts to remove certain pathogens from the list of regulated agents (Franz, 2015). A systematic approach as described herein offers the opportunity to provide substantial justification for such requests. The criteria we employed in this analysis are based on those identified in the Federal Registry Notice of Proposed Rule Making that the HHS Secretary should consider in determining whether to include a biological agent or toxin on the list (Public Law 107-188, 2002; CDC, HHS, 2020). Our goal for this study is to develop a robust and reproducible methodology for evaluating pathogens of concern for the establishment of a select agent list. As such we considered additional criterial to ensure it was a robust and to add additional fidelity to the process. Comparison of these criteria with other published methods shows that many of them overlap, such as morbidity and mortality, route of exposure, environmental stability, transmissibility, ease of production, availability of MCMs, etc. We included the status of immunity, an important host consideration pointed out by Casadevall and Relman, 2010.

In addition to the choice of criteria, the focus on bioterrorism scenarios (i.e., aerosol or food-based attacks affecting a large segment of the general population) is embodied in the scoring scales—for example, sexually transmitted pathogens rank low because aerosol and ingestion transmission routes are scored higher than those that are sexually transmitted. Also, pathogens that cause severe disease only in the presence of co-morbidities would rank low for Severity of Illness in the general population. Common pathogens causing mild illness and where there are treatments readily available may be unlikely to necessitate calling up a large-scale public health response.

Criteria we did not consider include public perception or terror factor, availability of agent and ease of detection, surveillance and laboratory diagnostic factors. Although we did not include incubation period as a specific criterion, we did take this factor into account as part of scoring Severity of Illness; i.e., diseases with long incubation periods (e.g., months to years) were considered chronic diseases and thus were scored lower than acute diseases. The fraction of symptomatic to non-symptomatic infections was considered as part of scoring CFR and severity of illness.

The results obtained using SME derived weights and the two-dimensional MCDA approach were tested against the current Select Agent Program designation and recent program decisions, and showed a high level of agreement, providing confidence that as new agents are considered, their terrorism risk can be assessed quickly and effectively using this method. The MCDA methodology reproduced all current select agent designations and all delisted select agents (Table 1) except for VEEV (1AB and 1C), which is currently a select agent but has been proposed for removal (CDC, HHS, 2020). In particular, the results are consistent with the 2020 proposal by the CDC Select Agent Program to remove B. abortus, B. melitensis, B. suis, B. anthracis Pasteur strain, C. burnetii, R. prowazekii (CDC, HHS, 2020) and confirms earlier decisions to remove C. immitis, C. posadasii, R. rickettsii, Flexal virus, Herpes B virus, Japanese encephalitis virus, Madariga virus, Monkeypox virus (Congo Basin clade), TBEV (Central European subtype) and VEEV (non 1AB and 1C subtypes) (CDC, HHS, 2012). The DSF also reproduces all current select agent designations, all delisted select agents (Table 1) and proposed select agents for removal including – VEEV (1AB and 1C). Both approaches confirm the exclusion of all non-select agents evaluated.

Two additional agents deserve specific mention. Highly Pathogenic Avian Influenza H5N1 (HPAI), a non-select HHS agent, received a score using the 3/3/9 criteria grouping with 3:2:1 weighting that placed it close to the threshold. However, it was classified as a non-select agent using the 5/4/6 weighting and the DSF approach. Although not an HHS select agent, HPAI remains a US Department of Agriculture select agent. C. botulinum is classified as a Tier 1 select agent (Bhattacharjee, 2011) because of the lethality of botulinum toxins; however, neither method evaluated here classified the organism as a select agent. C. botulinum spores are ubiquitous (Smith, 1978) and infections with this bacterium are rare (Werner et al., 2000). Analogous to the situations with ricin and castor beans and abrin and jequirity seeds, there appears to be little justification for keeping C. botulinum as a Tier 1 select agent.

Application of the methodology across a large and diverse pathogen set, while helping to demonstrate the robustness of the approach, highlighted the challenge of how to handle data gaps for many pathogens. The use of proxies and other assumptions at times artificially elevated some pathogens, requiring SME review of the data and discussions on how to account for the uncertainties in the data. Thus, we found the methodology also useful for identifying those parameters and pathogens where more data are needed, to help with prioritizing future research studies.

Conclusion

The goal of this effort was to explore the use of MCDA and logic tree approaches for supporting Select Agent Program decision making. We found that the two methods show strong promise. The two-dimensional MCDA approach provided a risk-informed assessment that implements the Select Agent Program’s decision criteria and its focus on bioterrorism scenarios with the potential for large-scale public health consequences. The DSF is a complementary approach to classifying select agents and provides additional insight into the factors that influence decision making. The two methods represent different ends of a spectrum for using criteria thresholding to identify select agents: the MCDA approach applies thresholds to two sub-scores after considering the criteria shown in Figure 1, while the DSF approach applies thresholds at the single criterion level for nine criteria. Applying weights using the MCDA approach can be used to fine-tune the effective number of criteria used to identify a threshold.

Although the two-dimensional MCDA data did not present clear data breaks for establishing thresholds, we demonstrated an analytical approach for designating quantitative thresholds through the use of simulated test agents over a broad range of weighting scenarios. In general, we found that the Tier 1 agents (plus Nipah virus and minus C. botulinum) consistently ranked above the other agents. One might argue that it is logical to assume that these agents represented the highest risk of public health consequences and thus constituted a special group of pathogens that should be regulated. Below the Tier 1 agents is a second group of agents that pose a lower risk; this group included mostly select agents, although there are some non-select agents as well. Below that is a third group of agents that poses the lowest risk; this group is mostly non-select agents and includes some delisted select agents. One can argue that this lowest group should not require controls such as those required by the select agent regulations. The need for controls for the middle group of agents is then an area for further investigation. Casadevall and Relman, 2010 contended that the Select Agent Program should restrict the number of select agents to a very few, as that enables the others to be used for research. Exclusion of this middle group of agents from the regulations would be consistent with that viewpoint. It should be noted that these results represent the data at the time of scoring, but the methodology, developed to support the biennial review, allows for updates to the scores as new data emerges.

Author Contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

Funding

This work was funded by Department of Homeland Security Science and Technology Directorate Contract HSHQPM-15-X-00071.

Conflict of Interest

SM was a CDC employee when this study was conducted.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors would like to thank the following individuals for their input and expertise: Donna Edwards, Sandia National Laboratory, Livermore, CA; Alan Barrett, Chuck Fulhorst, Tom Ksiazek, James LeDuc, Kent Tseng and Scott Weaver, University of Texas Medical Branch, Galveston, TX; Chris Broder, Uniformed Services University; Lark Coffey and Amy Morrison, University of California, Davis; Brian Hjelle, University of New Mexico School of Medicine; Molly Hughes, University of Virginia School of Medicine; Colleen Johnson, University of Tennessee; Fatah Kashanchi, George Mason University; Laura Kramer, University of Albany; Erich Mackow, Stony Brook University; David Sanders, Perdue University; Tony Schountz, Colorado State University; Stephen St. Jeor, University of Nevada-Reno; Antti Vaheri, University of Helsinki; William Wilson, Harvard University; Richard Yanagihara, University of Hawaii; Sina Bavari, Jay Hooper, Anthony Macaluso, Mark Poli and Connie Schmaljohn, U.S. Army Medical Research Institute for Infectious Disease; Armen Donabedian, Mary Lambert, Theresa Lawrence, ASPR, DHHS; John Barr, Darin Carrol, Dean Erdman, Robert Massung, Ann Powers, Maria E. Negron Sureda, Rebekah Tiller, Bryan Shelby, Alex Hoffmaster, Andrea McCollum, Brett Petersen, Carolina Luquez, Gilbert Kersh, Gregory Dasch, Grishma Kharod, Jay Gee, Jannine Petersen, John Towner, Kiersten Kugeler, Larry Schonberger, Michael Kosoy, Natalie Thornburg, Pierre Rollin, Stuart Nichol, Tyler Sharp, Vicky Olsen, Cinsia Bristol, Mark Pallansch, Michael Bowen, and Susan Gerber, CDC; David Asher and Drusilla Burns, FDA; Eric Freed and Rachelle Salomon, NCA, NIH; Deborah Wilson, NIEHS,NIH; Rodolfo Alarcon, Mark Challberg, Dennis Dixon, Heinz Feldman, Andrew Ford, Michael Holbrook, Peter Jharling, Jane Knisely, Ed Nuzum, Ryan Ranallo, Patricia Repik, NIAID, NIH; Kathryn Harris, Motad Hassani, Robert Jambou, Marina O’Reilly, Jeff Potts, Gene Rosenthal and Jessica Tucker, OSP, NIH; Frederick Doddy, Cyrill Gay, Joe Kozlovac, Kenneth Linthicum, Roxann Motroni, Eric Nicholson, Steven Olsen and David Swayne, ARS, USDA; Joyce Bowling-Heyward, Randal Capsel, Adis Dijab, Jodie Kulpa-Eddy, Mark Pagala, Suelee Robbe-Austerman, Abu Sayed, Linda Schlater, Janet Warg and Keith Wiggins, APHIS, USDA. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fbioe.2022.756586/full#supplementary-material

Supplementary Table S1 | Criteria scoring definitions.

References

Arabi, Y. M., Balkhy, H. H., Hayden, F. G., Bouchama, A., Luke, T., Baillie, J. K., et al. (2017). Middle East Respiratory Syndrome. N. Engl. J. Med. 376, 584–594. doi:10.1056/nejmsr1408795

PubMed Abstract | CrossRef Full Text | Google Scholar

Archer, A. M., and Rico Hesse, R. (2002). High Genetic Divergence and Recombination in Arenaviruses from the Americas. Virology 304, 274–281. doi:10.1006/viro.2002.1695

PubMed Abstract | CrossRef Full Text | Google Scholar

Barry, M., Russi, M., Armstrong, L., Geller, D., Tesh, R., Dembry, L., et al. (1995). Treatment of a Laboratory-Acquired Sabiá Virus Infection. N. Engl. J. Med. 333, 294–296. doi:10.1056/nejm199508033330505

PubMed Abstract | CrossRef Full Text | Google Scholar

Bhattacharjee, Y. (2011). Panel Selects Most Dangerous Select Agents. Science 332, 1491–1492. doi:10.1126/science.332.6037.1491

PubMed Abstract | CrossRef Full Text | Google Scholar

Butler, J., Jia, J., and Dyer, J. (1997). Simulation Techniques for the Sensitivity Analysis of Multi-Criteria Decision Models. Eur. J. Oper. Res. 103 (3), 531–546. doi:10.1016/s0377-2217(96)00307-4

CrossRef Full Text | Google Scholar

Buzgan, T., Karahocagil, M. K., Irmak, H., Baran, A. I., Karsen, H., Evirgen, O., et al. (2010). Clinical Manifestations and Complications in 1028 Cases of Brucellosis: a Retrospective Evaluation and Review of the Literature. Int. J. Infect. Dis. 14, e469–e478. doi:10.1016/j.ijid.2009.06.031

PubMed Abstract | CrossRef Full Text | Google Scholar

Casadevall, A., and Pirofski, L.-A. (2004). The Weapon Potential of a Microbe. Trends Microbiol. 12, 259–263. doi:10.1016/j.tim.2004.04.007

PubMed Abstract | CrossRef Full Text | Google Scholar

Casadevall, A., and Pirofski, L.-A. (2006). The Weapon Potential of Human Pathogenic Fungi. Med. Mycol. 44, 689–696. doi:10.1080/13693780600928503

PubMed Abstract | CrossRef Full Text | Google Scholar

Casadevall, A., and Relman, D. A. (2010). Microbial Threat Lists: Obstacles in the Quest for Biosecurity? Nat. Rev. Microbiol. 8, 149–154. doi:10.1038/nrmicro2299

PubMed Abstract | CrossRef Full Text | Google Scholar

Centers for Disease Control (2018). Cholera-Vibrio cholerae Infection. Available at: http://www.cdc.gov/cholera/index.html.

Google Scholar

Centers for Disease Control (2019a). Eastern Equine Encephalitis. Available at: http://www.cdc.gov/EasternEquineEncephalitis/index.html.

Google Scholar

Centers for Disease Control (2019b). Japanese Encephalitis. Available at: http://www.cdc.gov/japaneseencephalitis/index.html.

Google Scholar

Centers for Disease Control (2019c). Lassa Fever. Available at: http://www.cdc.gov/vhf/lassa/index.html.

Google Scholar

Centers for Disease Control, Health and Human Services (2012). Possession, Use, and Transfer of Select Agents and Toxins; Biennial Review. Fed. Reg 77 (194), 61084–611115. Available at: https://www.govinfo.gov/content/pkg/FR-2012-10- 05/pdf/2012-24389.pdf.

Google Scholar

Centers for Disease Control, Health and Human Services (2020). Possession, Use, and Transfer of Select Agents and Toxins; Biennial Review. Fed. Reg. 85 (52), 15087–15092. Available at: https://www.federalregister.gov/documents/2020/03/17/2020-05477/possession-use-and-transfer-of-select-agents-and-toxins-biennial-review.

Google Scholar

Chu, J. H., Feudtner, C., Heydon, K., Walsh, T. J., and Zaoutis, T. E. (2006). Hospitalizations for Endemic Mycoses: A Population-Based National Study. Clin. Infect. Dis. 42, 822–825. doi:10.1086/500405

PubMed Abstract | CrossRef Full Text | Google Scholar

Cieslak, T. J., Kortepeter, M. G., Wojtyk, R. J., Jansen, H.-J., Reyes, R. A., and Smith, J. O. The NATO Biological Medical Advisory Panel (2018). Beyond the Dirty Dozen: A Proposed Methodology for Assessing Future Bioweapon Threats. Mil. Med. 183, e59–e65. doi:10.1093/milmed/usx004

PubMed Abstract | CrossRef Full Text | Google Scholar

Cook, D., and Proctor, W. (2007). Assessing the Threat of Exotic Plant Pests. Ecol. Econ. 63, 594–604. doi:10.1016/j.ecolecon.2006.12.021

CrossRef Full Text | Google Scholar

Cox, L. A., Babayev, D., and Huber, W. (2005). Some Limitations of Qualitative Risk Rating Systems. Risk Anal. 25, 651–662. doi:10.1111/j.1539-6924.2005.00615.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Delgado, S., Erickson, B. R., Agudo, R., Blair, P. J., Vallejo, E., Albariño, C. G., et al. (2008). Chapare Virus, a Newly Discovered Arenavirus Isolated from a Fatal Hemorrhagic Fever Case in Bolivia. Plos Pathog. 4, e1000047. doi:10.1371/journal.ppat.1000047

PubMed Abstract | CrossRef Full Text | Google Scholar

DiCaudo, D. J. (2006). Coccidioidomycosis: a Review and Update. J. Am. Acad. Dermatol. 55, 929–942. doi:10.1016/j.jaad.2006.04.039

PubMed Abstract | CrossRef Full Text | Google Scholar

Franz, D. R. (2015). Implementing the Select Agent Legislation: Perfect Record or Wrong Metric? Health Security 13 (4), 290–294. doi:10.1089/hs.2015.0029

PubMed Abstract | CrossRef Full Text | Google Scholar

Gritsun, T. S., LashkevichGould, V. A., and Gould, E. A. (2003). Tick-Borne Encephalitis. Antiviral Res. 57, 129–146. doi:10.1016/s0166-3542(02)00206-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Hollingshead, A. B. (1996). The Rank-Order Effect in Group Decision Making. Organizat. Behav. Human Decis. Proc. 68 (3), 181–193. doi:10.1006/obhd.1996.0098

CrossRef Full Text | Google Scholar

Huang, J. Y., Bristow, B., Shafir, S., and Sorvillo, F. (2012). Coccidioidomycosis-Associated Deaths, United States, 1990-2008. Emerg. Infect. Dis. 18, 1723–1728. doi:10.3201/eid1811.120752

PubMed Abstract | CrossRef Full Text | Google Scholar

Hwang, C. L., and Yoon, K. (2012). Lecture Notes in Economics and Mathematical Systems, 186. NY: Springer-Verlag. Multiple Attribute Decision Making: Methods and Applications a State-Of-The Art Survey.

Google Scholar

Keeney, R. L., and Raiffa, H. (1993). Decisions with Multiple Objectives: Preferences and Value Trade-Offs. U.K: Cambridge University Press.

Google Scholar

Kiker, G. A., Bridges, T. S., Varghese, A., Seager, T. P., and Linkov, I. (2005). Application of Multicriteria Decision Analysis in Environmental Decision Making. Integr. Environ. Assess. Manag. 1, 95–108. doi:10.1897/ieam_2004a-015.1

PubMed Abstract | CrossRef Full Text | Google Scholar

Kim, S.-H., Chang, S. Y., Sung, M., Park, J. H., Bin Kim, H., Lee, H., et al. (2016). Extensive Viable Middle East Respiratory Syndrome (MERS) Coronavirus Contamination in Air and Surrounding Environment in MERS Isolation Wards. Clin. Infect. Dis. 63, 363–369. doi:10.1093/cid/ciw239

PubMed Abstract | CrossRef Full Text | Google Scholar

Krauss, H. (2003). Infectious Diseases Transmissible from Animals to Humans. 3rd ed. Washington: ASM Press.

Google Scholar

Laughlin, D. C., Meegan, J. M., Strausbaugh, L. J., Morens, D. M., and Watten, R. H. (1979). Epidemic Rift Valley Fever in Egypt: Observations of the Spectrum of Human Illness. Epidemic Med. Hyg. 73, 630–633. doi:10.1016/0035-9203(79)90006-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Linkov, I., Satterstrom, F. K., Kiker, G., Batchelor, C., Bridges, T., and Ferguson, E. (2006). From Comparative Risk Assessment to Multi-Criteria Decision Analysis and Adaptive Management: Recent Developments and Applications. Environ. Int. 32, 1072–1093. doi:10.1016/j.envint.2006.06.013

PubMed Abstract | CrossRef Full Text | Google Scholar

Lisieux, T., Coimbra, M., Nassar, E. S., Burattini, M. N., de Souza, L. T., Ferreira, I., et al. (1994). New Arenavirus Isolated in Brazil. Lancet 343, 391–392.

PubMed Abstract | CrossRef Full Text | Google Scholar

Lundkvist, A., Cheng, Y., Sjölander, K. B., Niklasson, B., Vaheri, A., and Plyusnin, A. (1997). Cell Culture Adaptation of Puumala Hantavirus Changes the Infectivity for its Natural Reservoir, Clethrionomys glareolus, and Leads to Accumulation of Mutants with Altered Genomic RNA S Segment. J. Virol. 71, 9515–9523. doi:10.1128/jvi.71.12.9515-9523.1997

PubMed Abstract | CrossRef Full Text | Google Scholar

MacIntyre, C. R., Seccull, A., Lane, J. M., and Plant, A. (2006). Development of a Risk-Priority Score for Category A Bioterrorism Agents as an Aid for Public Health Policy. Mil. Med. 171, 589–594. doi:10.7205/milmed.171.7.589

PubMed Abstract | CrossRef Full Text | Google Scholar

Majoor, C. J., Magis-Escurra, C., van Ingen, J., Boeree, M. J., and van Soolingen, D. (2011). Epidemiology of Mycobacterium Bovis Disease in Humans, The Netherlands, 1993-2007. Emerg. Infect. Dis. 17, 457–463. doi:10.3201/eid1703.101111

PubMed Abstract | CrossRef Full Text | Google Scholar

McElroy, A. K., Bray, M., Reed, D. S., and Schmaljohn, C. S. (2002). Andes Virus Infection of Cynomolgus Macaques. J. Infect. Dis. 186, 1706–1712. doi:10.1086/345768

PubMed Abstract | CrossRef Full Text | Google Scholar

Menrath, A., Tomuzia, K., Frentzel, H., Braeunig, J., and Appel, B. (2014). Survey of Systems for Comparative Ranking of Agents that Pose a Bioterroristic Threat. Zoonoses Public Health 61, 157–166. doi:10.1111/zph.12065

PubMed Abstract | CrossRef Full Text | Google Scholar

Monath, T. P. (2001). Yellow Fever: An Update. Lancet Infect. Dis. 1, 11–20. doi:10.1016/s1473-3099(01)00016-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Morse, S. A. (2015). Pathogen Security-Help or Hindrance? Front. Bioeng. Biotechnol. 2, 83. doi:10.3389/fbioe.2014.00083

PubMed Abstract | CrossRef Full Text | Google Scholar

Nemirov, K., Lundkvist, A., Vaheri, A., and Plyusnin, A. (2003). Adaptation of Puumala Hantavirus to Cell Culture Is Associated with Point Mutations in the Coding Region of the L Segment and in the Noncoding Regions of the S Segment. J. Virol. 77, 8793–8800. doi:10.1128/jvi.77.16.8793-8800.2003

PubMed Abstract | CrossRef Full Text | Google Scholar

NIH (2019). Classification of Human Etiological Agents on the Basis of Hazard. Available at: https://www.uidaho.edu/-/media/UIdaho-Responsive/Files/research/Faculty/research-assurances/IBC/resources/nih-guidelines-appendix-b-risk-group-classification.pdf.

Google Scholar

Paweska, J. T., Sewlall, N. H., Ksiazek, T. G., Blumberg, L. H., Hale, M. J., Lipkin, W. I., et al. (2009). Nosocomial Outbreak of Novel Arenavirus Infection, Southern Africa. Emerg. Infect. Dis. 15, 1598–1602. doi:10.3201/eid1510.090211

PubMed Abstract | CrossRef Full Text | Google Scholar

Peters, C. J. (2006). Lymphocytic Choriomeningitis Virus - An Old Enemy up to New Tricks. N. Engl. J. Med. 354, 2208–2211. doi:10.1056/nejmp068021

PubMed Abstract | CrossRef Full Text | Google Scholar

Petersen, L. R., and Marfin, A. A. (2002). West Nile Virus: A Primer for the Clinician. Ann. Intern. Med. 137, 173–179. doi:10.7326/0003-4819-137-3-200208060-00009

PubMed Abstract | CrossRef Full Text | Google Scholar

Prescott, J., Feldmann, H., and Safronetz, D. (2017). Amending Koch's Postulates for Viral Disease: When “growth in Pure Culture” Leads to a Loss of Virulence. Antiviral Res. 137, 1–5. doi:10.1016/j.antiviral.2016.11.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Public Health Agency of Canada (2011). Semliki Forest Virus. Available at: http://www.phac-aspc.gc.ca/lab-bio/res/psds-ftss/semliki-eng.php (February 18, 2011).

Google Scholar

Public Law 104-132 (1996). Antiterrorism and Effective Death Penalty Act of 1996. Available at: www.gpo.gov/fdsys/pkg/PLAW-104publ132/PLAW-104/publ132.pdf.

Google Scholar

Public Law 107-188 (2002). Public Health Security and Bioterrorism Preparedness and Response Act of 2002. Available at: www.gpo.gov/fdsys/pkg/PLAW-107publ188/pdfPLAW-107/publ188.pdf.

Google Scholar

Public Law 107-56 (2001). Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism. Available at: www.gpo.gov/fdsys/pkg/PLAW-107publ56/pdf/PLAW-107publ56.pdf.

Google Scholar

Reed, D. S., Lackemeyer, M. G., Garza, N. L., Norris, S., Gamble, S., Sullivan, L. J., et al. (2007). Severe Encephalitis in Cynomolgus Macaques Exposed to Aerosolized Eastern Equine Encephalitis Virus. J. Infect. Dis. 196, 441–450. doi:10.1086/519391

PubMed Abstract | CrossRef Full Text | Google Scholar

Reignier, T., Oldenburg, J., Flanagan, M. L., Hamilton, G. A., Martin, V. K., and Cannon, P. M. (2008). Receptor Use by the Whitewater Arroyo Virus Glycoprotein. Virology 371, 439–446. doi:10.1016/j.virol.2007.10.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Rotz, L. D., Khan, A. S., Lillibridge, S. R., Ostroff, S. M., and Hughes, J. M. (2002). Public Health Assessment of Potential Biological Terrorism Agents. Emerg. Infect. Dis. 8, 225–230. doi:10.3201/eid0802.010164

PubMed Abstract | CrossRef Full Text | Google Scholar

Ruzante, J. M., Davidson, V. J., Caswell, J., Fazil, A., Cranfield, J. A. L., Henson, S. J., et al. (2010). A Multifactorial Risk Prioritization Framework for Foodborne Pathogens. Risk Anal. 30, 724–742. doi:10.1111/j.1539-6924.2009.01278.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Růžek, D., Yakimenko, V. V., Karan, L. S., and Tkachev, S. E. (2010). Omsk Haemorrhagic Fever. Lancet 376, 2104–2113. doi:10.1016/S0140-6736(10)61120-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Safronetz, D., Prescott, J., Feldmann, F., Haddock, E., Rosenke, R., Okumura, A., et al. (2014). Pathophysiology of Hantavirus Pulmonary Syndrome in Rhesus Macaques. Proc. Natl. Acad. Sci. U.S.A. 111, 7114–7119. doi:10.1073/pnas.1401998111

PubMed Abstract | CrossRef Full Text | Google Scholar

Greco, S., Figueira, J., and Ehrgott, M. (Editors) (2005). Multiple Criteria Decision Analysis (NY. Analysis: Springer's International Series, Springer).

Smith, L. D. (1978). The Occurrence of Clostridium Botulinum and Clostridium Tetani in the Soil of the United States. Health Lab. Sci. 15, 74–80.

PubMed Abstract | Google Scholar

Steele, K., Carmel, Y., Cross, J., and Wilcox, C. (2009). Uses and Misuses of Multicriteria Decision Analysis (MCDA) in Environmental Decision Making. Risk Anal. 29, 26–33. doi:10.1111/j.1539-6924.2008.01130.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Thokala, P., Devlin, N., Marsh, K., Baltussen, R., Boysen, M., Kalo, Z., et al. (2016). Multiple Criteria Decision Analysis for Health Care Decision Making-An Introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value in Health 19, 1–13. doi:10.1016/j.jval.2015.12.003

PubMed Abstract | CrossRef Full Text | Google Scholar

Velasquez, M. H., and Hester, P. T. (2013). An Analysis of Multi-Criteria Decision Making Methods. Int. J. Op. Res. 10, 56–66.

Google Scholar

Werner, S. B., Passaro, D., McGee, J., Schechter, R., and Vugia, D. J. (2000). Wound Botulism in California, 1951-1998: Recent Epidemic in Heroin Injectors. Clin. Infect. Dis. 31, 1018–1024. doi:10.1086/318134

PubMed Abstract | CrossRef Full Text | Google Scholar

World Health Organization (2017). Dengue and Severe Dengue. Available at: http://who.int/mediacentre/factsheets/fs117/en/.

Google Scholar

Zong, M., Fofana, I., and Choe, H. (2014). Human and Host Species Transferrin Receptor 1 Use by North American Arenaviruses. J. Virol. 88, 9418–9428. doi:10.1128/jvi.01112-14

PubMed Abstract | CrossRef Full Text | Google Scholar