94% of researchers rate our articles as excellent or good
Learn more about the work of our research integrity team to safeguard the quality of each article we publish.
Find out more
Anna L. Hotton1*
Pedro Nascimento de Lima2
Arindam Fadikar3
Nicholson T. Collier3
Aditya S. Khanna4Darnell N. Motley1Eric Tatara3Sara Rimer3Ellen Almirol1
Harold A. Pollack5John A. Schneider1,5,6
Robert J. Lempert7,8
Jonathan Ozik3
- 1Department of Medicine, University of Chicago, Chicago, IL, United States
- 2Engineering & Applied Sciences, RAND, Arlington, VA, United States
- 3Consortium for Advanced Science and Engineering, University of Chicago, Chicago, IL, United States
- 4Department of Behavioral and Social Sciences, Brown University School of Public Health, Providence, RI, United States
- 5Crown Family School of Social Work, Policy, and Practice, University of Chicago, Chicago, IL, United States
- 6Department of Public Health Sciences, University of Chicago, Chicago, IL, United States
- 7Pardee RAND Graduate School, RAND, Santa Monica, CA, United States
- 8Frederick S. Pardee Center for Longer Range Global Policy and the Future Human Condition, RAND, Santa Monica, CA, United States
There is much focus in the field of HIV prevention research on understanding the impact of social determinants of health (e.g., housing, employment, incarceration) on HIV transmission and developing interventions to address underlying structural drivers of HIV risk. However, such interventions are resource-intensive and logistically challenging, and their evaluation is often limited by small sample sizes and short duration of follow-up. Because they allow for both detailed and large-scale simulations of counterfactual experiments, agent-based models (ABMs) can demonstrate the potential impact of combinations of interventions that may otherwise be infeasible to evaluate in empirical settings and help plan for efficient use of public health resources. There is a need for computational models that are sufficiently realistic to allow for evaluation of interventions that address socio-structural drivers of HIV transmission, though most HIV models to date have focused on more proximal influences on transmission dynamics. Modeling the complex social causes of infectious diseases is particularly challenging due to the complexity of the relationships and limitations in the measurement and quantification of causal relationships linking social determinants of health to HIV risk. Uncertainty exists in the magnitude and direction of associations among the variables used to parameterize the models, the representation of sexual transmission networks, and the model structure (i.e. the causal pathways representing the system of HIV transmission) itself. This paper will review the state of the literature on incorporating social determinants of health into epidemiological models of HIV transmission. Using examples from our ongoing work, we will discuss Uncertainty Quantification and Robust Decision Making methods to address some of the above-mentioned challenges and suggest directions for future methodological work in this area.
1 Introduction
In the United States, HIV disproportionately impacts sexual and gender minorities and other marginalized communities, with the greatest impact on those with fewer resources and limited access to prevention and treatment services (1, 2). It is now well recognized that the elimination of HIV will require a greater focus on communities that experience complex and co-occurring socio-structural barriers to HIV prevention and care (3, 4). Thus, plans to eliminate HIV have increasingly focused on factors that indirectly impact HIV transmission, including housing, employment, and incarceration, which also disproportionately affect people vulnerable to HIV (3, 5, 6). However, interventions to address social determinants of health (SDOH) can be resource intensive, logistically challenging, and require long-term follow-up, making it difficult to assess their impact using standard study designs. Although not necessarily unique to interventions addressing SDOH, it may be unethical or impossible to randomly assign people to a control condition. In addition, those with the greatest need for intensified interventions may have limited interaction with traditional research settings due to the same life challenges that make it difficult to engage in HIV prevention and care. Thus, while housing and employment interventions have been developed, (7–10), they are difficult to conduct and evaluate on a larger scale.
Simulation approaches can be useful for elucidating mechanisms or evaluating interventions that are difficult to test empirically. Computational models have long been used to understand the spread of infectious diseases and to make predictions about the course of epidemics. Commonly used methods for modeling HIV transmission dynamics include compartmental models (11), such as the classic susceptible-infected-recovered (SIR) model (12) and its extensions (13), individual-level microsimulation models, including agent-based models (ABMs), and network models (14–16), which can be used in combination with ABMs (17). These models differ in their complexity, granularity, representation of mixing between populations, and behavioral impacts on disease transmission (18). Compartmental models use differential equations to model the rate of transition between compartments, assuming homogeneous populations within compartments and homogeneous mixing within and between compartments (19). ABMs, in contrast, simulate individual interactions, decisions, and behaviors using a bottom-up approach in which decision rules are applied such that the interactions between agents and their behaviors can give rise to population-level patterns not explicitly programmed into the model (20, 21). One of the benefits of ABMs as compared to compartmental models for modeling HIV transmission dynamics is the ability of ABMs to explicitly model heterogeneous individual agent characteristics, behaviors, and interactions among agents that result in HIV transmission. Network models focus on detailed representation of networks that connect individuals and through which information or disease can spread (14, 16).
Whereas classic SIR models do not incorporate behavioral dynamics, recent work has demonstrated the importance of behavioral responses in epidemic modeling (18, 22, 23), where people change their behaviors in response disease dynamics, which in turn affects transmission, resulting in behavioral-epidemiological feedback (22, 24). Decisions about whether to adhere to pharmaceutical or non-pharmaceutical interventions (NPI) are likely to be shaped by the prevalence and severity of the disease, perceived risk, and the cost of adhering to the interventions, including social or economic costs (24). These decisions can change over time in response to the epidemic and can be shaped by social or contextual factors that affect the ease of adherence or access to NPI or vaccinations.
ABMs simulate individual agent decision making, making them well suited to explicitly model behavioral-epidemiological feedback and to observe emergent patterns resulting from this feedback over time (21). For example, individuals may change their adherence or use of PrEP in response to changes in the HIV epidemic and perceived risk, and SDOH can affect the ease with which individuals change their behavior in response to disease dynamics. ABMs can also be used to conduct counterfactual experiments in which candidate interventions are systematically evaluated and compared. This can facilitate identification of effects that would be difficult to isolate using traditional statistical approaches (25, 26) and more efficient and focused intervention development (27, 28). In addition to allowing for behavioral feedback, ABMs are well suited to incorporate the dependency structures and feedback loops common among co-occurring social determinants of health (21, 26, 29). Thus, ABMs can demonstrate the potential impact of combinations of interventions that are not typically feasible to evaluate in empirical settings. Applications of ABMs for studying the impact of complex social phenomena on health outcomes have been described, most notably in the context of chronic diseases, where ABMs have elucidated network and system level drivers of obesity (30), tobacco use (31) and violence and PTSD (32–34). However, most ABMs of HIV transmission to date have focused on more proximal influences on transmission dynamics (e.g., sexual behaviors, networks) (17, 35, 36), and do not include distal factors that may impact transmission indirectly. This is in part due to the complexity of the relationships among SDOH and HIV and limitations in the measurement and quantification of causal relationships among SDOH.
ABMs are typically based on the inherent assumption that associations among the variables used to parameterize the model reflect causal relationships, but causality often cannot be inferred from empirical data. Although one of the benefits of ABMs is their ability to combine and triangulate information from multiple sources, it can be difficult to reconcile variation in estimates across different data sources. Effect sizes often vary widely, obtained from observational studies conducted across a wide range of populations and time periods. Uncertainty exists in terms of the magnitude and direction of associations in the variables used to parameterize the model, as well as in the model structure itself (i.e., the causal relationships among the variables being modeled). Thus, while a primary motivation to use ABMs is often that they offer the opportunity to study aspects of complex systems that are difficult or impossible to study empirically, ABMs require detailed empirical data or strong assumptions to draw conclusions about the systems they represent. These issues are not unique to modeling SDOH, but they are likely to be particularly relevant since data on SDOH and interventions to address them are often limited and complex social constructs are difficult to accurately measure even when data are available. Understanding how uncertainty and bias in empirical data impact ABM outputs, and how ABMs can be used to guide public health decision-making in the presence of this uncertainty, are active and fruitful areas for future work.
Recent reviews have described various approaches to modeling social determinants of health in the context of HIV, including statistical and compartmental models and ABMs. Hogan et al. (37) discuss uses of mechanistic and statistical models to answer different types of questions related to SDOH in HIV. De Oliveira et al. (38) describe the process of developing a conceptual framework for incorporating SDOH into a compartmental model of HIV. ABMs are valuable for studying the impact of human behavior on HIV because they allow explicit modeling of heterogeneous agent behaviors and interactions, but with a significant increase in complexity and computational resources compared to compartmental and statistical models. The sections below focus on ABMs because of their utility for modeling heterogeneous individual behaviors and transmission networks, and the impact of SDOH on agent behaviors, interactions, and disease dynamics. We also discuss challenges unique to ABMs compared to other types of models.
The remainder of the manuscript is organized as follows. First, we describe current examples of how SDOH can be incorporated into agent-based models of HIV transmission. We then describe approaches to addressing uncertainty and their limitations. Finally, we discuss ongoing applied work in the use of ABMs to support public health decision making in the presence of deep uncertainty.
2 Approaches to incorporating SDOH into agent-based models
2.1 Developing a conceptual model
The first step of incorporating SDOH into the ABM is to develop a conceptual framework for describing how SDOH impact HIV transmission. Embedded in this framework are a number of causal hypotheses about how various factors relate to one another. These hypotheses are then translated into the ABM. Based on our previous research, review of the literature, feedback from community members, and analysis of empirical data, we hypothesized that SDOH impact HIV transmission through one or more of the following mechanisms: (1) changes in individual psychosocial factors (e.g., substance use, mental health) that impact transmission related behaviors, prevention and care engagement, and medication adherence; (2) changes in sexual networks that affect the probability of serodiscordant sexual partnerships; (3) changes in health care access or engagement (e.g., as a result of employment and insurance changes, economic hardship, or housing instability) that impact the probability that individuals are retained in HIV care or on PrEP. From our conceptual model, we focus on modifiable or actionable intervention levers. For example, we do not model racism explicitly, but rather the model incorporates the effects of racism, discrimination, and other larger social constructs on factors that impact HIV transmission; i.e., housing, employment, and incarceration can affect use of ART or PrEP, which impact the probability of HIV transmission.
2.2 Incorporating relevant pathways and processes to model interventions to address SDOH
To evaluate the potential impact of interventions to address SDOH on HIV transmission, we developed the Integrated Framework for Modeling Social Determinants of HIV (INFORM-HIV) ABM. The model extends and expands an agent-based network model (17) of HIV that incorporates sexual behaviors and networks, HIV testing, diagnosis, linkage to care, and ART and PrEP use (Figure 1). The original model was modified by (1) specifying hypothesized mechanisms by which SDOH impact HIV prevention and care engagement directly and indirectly via intermediate factors and assigning rules by which these factors impact model processes, and (2) incorporating agent characteristics that impact the probability of experiencing incarceration, housing instability, and unemployment while accounting for correlations among agent attributes. A simplified model diagram is shown in Figure 2. Agent characteristics are assigned probabilistically at the beginning of the simulation to match the empirical distribution of SDOH in the population represented in the model and the correlations among SDOH. SDOH can produce changes in one or more domains, and relationships among factors within and between domains can be cyclical and bidirectional. As the simulation progresses, agents make decisions at various steps in the HIV prevention and care continuum (e.g., whether to get tested for HIV, take medication, adhere to PrEP/ART). Decision probabilities reflect agents’ past and current experiences of housing instability, employment, and other structural factors, which can change in response to internal or external factors, including substance use. Agent decisions are also influenced by individual psychosocial factors, such as mental health. Structural and psychosocial factors can influence each other and can vary over time.
Figure 1
Figure 1. Conceptual model of the pathways by which social determinants of health impact HIV transmission.
Figure 2
Figure 2. Components of the integrated framework for modeling social determinants of HIV (INFORM-HIV).
2.3 State transition models
In order to allow for dynamic (i.e., time-varying) agent behaviors, states, and characteristics, we apply a state chart approach within the ABM. Statecharts are a type of state diagram (39), or visual depiction of sequential control procedures or algorithms. Early examples of state diagrams include Moore machines (40) and Mealy machines (41). Our implementation is based the Repast Simphony implementation of Harel’s statecharts (42) described in (43).
Each agent has multiple state charts, each tracking separate attributes, such as housing or employment, with distinct states (e.g., housed/unhoused, or employed/unemployed). Transition probabilities determine when an agent moves from one state to another. The transitions incorporate dependency among different agent attributes, such that rates of transition between states in one statechart can be based on the occupied states of another statechart, as well as on any other agent attribute. Transitioning between two states is determined by a probability where transitions are more or less likely depending on agent characteristics and previous states. We applied a multiplier to base transition probability to indicate the increased or decreased probability of transitioning into or out of a state based on other agent characteristics. For example, an agent is more likely to transition from housed to unhoused at a subsequent time step if they become unemployed than if they remain employed.
3 Modeling transmission networks
Network models provide a statistically principled approach to representing concurrency (i.e., having one or more sexual partnerships that overlap in time as opposed to serial or sequential partnerships that do not overlap), which is thought to be an important contributor to HIV transmission dynamics in some settings due to efficiency of transmission during the acute phase of infection (44–46). Extensive methodological and software development has been undertaken in modeling networks in the context of infectious diseases, and we refer the readers to the published literature for in-depth reviews and example applications (47–49). Below is a brief overview of how HIV transmission networks are modeled within the ABM using exponential-family random graph models (ERGMs).
3.1 Exponential-family random graph models (ERGMs)
ERGMs are a class of statistical models used to represent the structure of social or sexual networks through the presence or absence of ties, which are assigned to approximate target statistics describing the observed network (50). Target statistics reflect the likelihood that agents form partnerships based on various individual (nodal) or dyadic (edge) characteristics. For example, SDOH can impact the likelihood of partnership formation and dissolution, this impacting the network structure. Extensions of ERGMs, separable temporal exponential-family random graph models (STERGMs), are used to represent dynamic networks over time (51). These models are said to be “separable” because the factors affecting partnership formation and dissolution are assumed to be independent conditional on the existing network structure (16). Thus, STERGMs have separate formulas for partnership formation and dissolution. While the formation and dissolution models are conditionally independent at a given time step, they are modeled jointly over time, allowing the structure of the tie formation model to be identified in the context of the existing persistence model (16). To optimize the computational cost of fitting ERGMs, we use the recommended “statnet” approximation for cases where the networks are relatively sparse (such as in sexual networks) and the relationships are of at least moderate duration (relative to the time units of the model) (52).
Main and casual partnership networks are estimated using the “ergm” package in R (47), and the coefficients for the ERGM parameters are estimated using Markov Chain Monte Carlo methods (47, 48, 53), using Carnegie et al.’s approximation method (52). These estimated adjusted coefficients are used to simulate main and casual networks, which are read into the ABM. The “rpy2” package (54) is used to connect the Python code to the ERG model in R.
3.2 Uncertainty in network structure
ERGMs are typically developed to approximate target statistics, and the formation and dissolution coefficients are used to simulate networks dynamically over time. ERGMs seek to maintain equilibrium with respect to the target statistics, and the amount of variation across stochastic realizations of a network given a set of formation and dissolution parameters is often not characterized. Because HIV is transmitted via sexual networks, the extent to which the network structure (e.g., in terms of mixing patterns, partnership duration, concurrency, etc.) accurately reflects the underlying network in the population represented in the model is essential for accurately modeling HIV transmission. Computational demands of modeling networks has previously limited the extent to which sensitivity analyses involving varying network structures could be conducted. However, with increases in computational resources, it is possible to experiment with different network configurations to determine the impact of the network structure on model outputs. Such analyses are important given the challenges involved in measuring networks using empirical surveys (55–57). Additionally, there may be limited data on how SDOH impact network structure and dynamics.
4 Methodological considerations for empirical data: obtaining causal effect estimates
Associations have been demonstrated between various SDOH (e.g., housing instability, incarceration, economic hardship) and HIV risk (58, 59), mental health and substance use (60, 61), and HIV prevention and care engagement (61–63) across multiple geographic contexts and populations. However, much of the existing evidence base has come from cross-sectional studies or lacked power to test for interactions or mediation effects. Further, many of these relationships are likely cyclical (e.g., housing instability creates employment barriers which leads to ongoing lack of resources and reduced ability to obtain stable housing). This makes it challenging to determine directionality in terms of the causal structures underlying the model, which is necessary to understand which interventions or combinations of interventions would have the greatest public health impact.
Ideally, for each factor of interest (X) and outcome (Y), it would be possible to determine (1) whether X affects Y (with the assumption that the effect is causal), (2) how X affects Y (i.e., what are the mechanisms, or pathways, by which X affects Y, including direct and indirect effects), and (3) whether the effect of X on Y is the same across the whole population being modeled, or whether it varies across subgroups of the population (i.e., is there effect heterogeneity). If all of the above statements are known, we can use the model to answer questions such as “What would be the impact of intervening on X on the outcome Y?” (in terms of absolute and relative effects, cost, or any other relevant effect measure). We can also answer questions such as “Would interventions that are focused on a subgroup or subgroups of the population be more efficient (in terms of cost, infections averted, etc.) than more broadly targeted interventions?”
Many traditional applications of ABMs have focused on answering the two questions above, assuming that the preceding items (1–3 above) are known and quantified. There is also typically the assumption that model parameters and calibration targets are unbiased and measured without error (this is discussed further below in the context of sensitivity analysis and calibration). However, these assumptions often cannot be easily tested or verified for most SDOH or other factors that are not amenable to testing in randomized controlled trials. Thus, conclusions are highly dependent on the availability and quality of the data used to parameterize the model. There is thus a need to account for and address uncertainty in models seeking to understand the potential impact of interventions on SDOH for reducing HIV transmission. This is discussed in further detail in the sections that follow.
ABMs’ potential to aid epidemiological causal inference has been recognized (20, 25, 26) but applications in the context of complex social phenomena have been limited (34). Recently published work has demonstrated how uncertain or biased input parameter estimates can result in biased results from ABMs (64, 65). These illuminating examples have focused on clinical data and endpoints, and the potential for bias is likely amplified in the case of social determinants of health. Debate remains about the circumstances under which necessary assumptions can be tested and verified (66), and estimates obtained from one population can be transported to another (64, 67). More work is needed to determine to what extent uncertainties can be addressed using computational approaches and how ABMs can be used to study highly complex systems in the presence of uncertainty.
5 Approaches to dealing with uncertainty: uncertainty quantification, sensitivity analysis, and calibration
Understanding the magnitude and potential impact of various forms of uncertainty is essential in ABM research because it directly impacts conclusions that can be drawn from models, particularly when comparing interventions in order to make public health decisions about use of limited resources. Broadly, uncertainty is categorized into two types: aleatory and epistemic. Aleatory uncertainty captures the inherent randomness in the stochastic processes that drive agent interactions within an ABM and the empirical data being observed. Epistemic uncertainty, in contrast, arises from a lack of knowledge about the process or input to the computational model. Generally, epistemic uncertainty is considered “reducible” by collecting more data, refining model parameters, enhancing model structure, or conducting further experiments, while aleatory uncertainty is “irreducible.” The objective is to identify and account for all quantifiable uncertainties within projections created by the ABMs.
HIV transmission models that integrate the influences of SDOH are inherently complex, with many relationships that are indirect, bi-directional, or non-explicit. Such complexity introduces structural uncertainty due to the incomplete understanding of causal relationships among SDOH and between SDOH and HIV. This complexity necessitates the inclusion of a large number of parameters to control the various aspects of the model. However, estimating these parameters is challenging, often relying on incomplete data that may include significant measurement errors, which underscores the need for large-scale sensitivity analysis to assess the robustness of the model. Thus, the overall task becomes computationally demanding due to the model’s intricacy. Although sensitvity analysis (SA) has been applied in ABM research previously [e.g., (68)], complex models integrating SDOH entail far larger parameter spaces and, hence, more uncertainties to address.
In the following sections, we provide a brief outline of approaches for examining the impact of various forms of uncertainty on model outcomes using uncertainty quantification and sensitivity analysis tools (we refer readers to comprehensive resources such as (69) and other cited works for a more in-depth understanding). We also review how complex models with uncertainties, including ABMs that seek to understand how SDOH factors impact HIV transmission, can be calibrated.
5.1 Uncertainty quantification and sensitivity analysis
Uncertainty quantification (UQ) and sensitivity analysis (SA) are complementary approaches that are often performed together to understand the impact of uncertainty on model outcomes. UQ involves characterizing the uncertainty in a model’s input parameters that arise from measurement error or natural variation, often in the form of distributions or ranges (sometimes referred to as uncertainty characterization) and subsequently evaluating the impact of this uncertainty on model outcomes (sometimes referred to as uncertainty propagation) (70, 71). In other words, UQ is used to link uncertainty in the input parameters to uncertainty in the model outputs. Sensitivity analysis is then performed to understand which input parameters have the most impact on model outcomes. In an ABM, some parameter values are derived empirically from observed data, while others, which are not directly observable, are estimated through calibration and data assimilation. Uncertainty in the data can lead to varying degrees of influence on model parameter ranges and distributions. SA aids in simplifying the model by identifying parameters with minimal impact on outcomes and facilitating dimensional reduction. It also highlights parameters with large impacts, where improving on empirical precision would be beneficial to reduce model output uncertainty.
The first step in SA is to identify model parameters and define their ranges. Parameter ranges may be informed by empirical data, previous studies, expert opinion, or, when data is sparse, assigned a broad range. When there is a large parameter space, an initial screening is useful to identify parameters with minimal influence, allowing these to be fixed at specific values and thereby reducing the problem’s dimensionality. The Morris method (72) is effective for this screening, requiring only simulation runs, where is the number of settings tested per parameter , and is the number of replicates when a model is stochastic. This approach provides an efficient way to identify influential parameters before performing a full-scale SA. However, as model complexity, and in turn, the number of parameters, increases, comprehensive UQ/SA can become computationally infeasible (70). Surrogate models, discussed next, and high-performance computing (HPC) workflows become essential for global SA in large computational models (73–75).
5.2 Surrogate models
Surrogate models, also referred to as emulators or reduced-order models, serve as efficient approximations to actual models, in the current context serving as surrogates for ABMs. These are mathematical representations of the relationships between inputs and outputs of a model, and are constructed to predict outputs of a model based on a limited number of simulations. Trained surrogates can be used to quickly reproduce predictions from a model without having to run the model (76–79). Surrogates can significantly reduce the computational costs in ABM analyses, thus facilitating large-scale UQ/SA. Building a surrogate model involves selecting an appropriate functional form, such as Gaussian processes (80), polynomial chaos expansions (81), or neural networks (82), and fitting it to a set of model input/output pairs, using static or adaptive experimental designs (80). Once fitted, the surrogate provides fast approximations of ABM outputs across the input space, enabling extensive sampling needed for UQ/SA and other critical analyses such as ABM calibration (83), discussed next.
For SA applications, surrogates can be used to estimate Sobol sensitivity indices (84). To train a Gaussian process (GP) surrogate, an initial simulation campaign is performed, e.g., using a Latin hypercube space-filling design (85). If the parameter space is high-dimensional, a single round of GP training can require a significant number of simulations to produce acceptable prediction accuracy. However, by integrating the GP with an active learning strategy, such as the one developed in (86), the total simulation budget can be reduced. This approach starts with a small number of simulations and iteratively adds more, prioritizing those that enhance the estimation of Sobol sensitivity indices. The active learning framework can be combined with various GP constructs to tailor the surrogate to the specific needs of the underlying model. In addition, approximate GPs can be used to accelerate the computation of sensitivity indices by constructing surrogates that are local to specific parameter subspaces (87).
5.3 Calibration
Prior to scenario analyses or predictions, ABMs need to be calibrated to ensure that the outputs from the model are consistent with empirical data targets. Calibration involves running the model with different sets of parameters and comparing the output to empirical data with the goal of minimizing model error (88). Calibration targets can be evaluated based on point estimates or ranges, but the greater the uncertainty in the calibration targets the more challenging and computationally expensive the calibration becomes.
Uncertainty in calibration targets is an important issue in HIV modeling, particularly when modeling sub-populations in which the denominators defining the population size estimates used in incidence and prevalence calculations have to be estimated from multiple sources (89–91). This results in additional uncertainty in surveillance estimates above the standard limitations of surveillance data which typically suffer from some degree of under reporting and other reporting biases. Thus, calibration approaches that address uncertainty in both the input parameters and the calibration targets are needed.
5.3.1 Approaches to calibration
History matching History matching is a common calibration technique which works by systematically ruling out parameter values that are inconsistent with the observed data. Unlike other calibration methods that seek to optimize parameters directly, history matching iteratively eliminates implausible regions of the parameter space, zooming in on parameter combinations that yield predictions aligned with empirical data (92, 93). History matching has seen its use in healthcare applications, particularly for calibrating high-dimensional models. For computationally expensive models such ABMs, it can be paired with surrogate models, enabling efficient exploration of the parameter space without exhaustive simulations (94). The iterative refinement process is especially applicable to healthcare applications, where robust model calibration is necessary for forecasting and evaluating interventions under uncertainty. By focusing computational resources on the most plausible parameter sets, history matching provides a rigorous yet computationally feasible approach to calibration, enhancing model reliability in critical healthcare applications (95).
Approximate Bayesian computation Approximate Bayesian computation involves sampling a set of parameters from a prior distribution and testing if simulated model outputs (simulated targets) are within pre-specified thresholds of empirical values (empirical targets). A parameter set is accepted if the error is smaller than the threshold and otherwise rejected. Testing a large number and range of parameters can be used to approximate the posterior distribution, i.e., the probability of the parameters given the data. Sampling the parameter space and evaluating model outputs can become computationally expensive so efficient sampling methods are needed. Different ABC algorithms have been applied in the literature, ranging from simple rejection sampling to sequential Monte Carlo approaches (96, 97).
Bayesian Calibration Bayesian calibration using GPs has become a widely adopted approach in various modeling domains, providing a principled way to calibrate complex models by accounting for both parameter uncertainty and model discrepancy. This method combines Bayesian inference with a GP-based surrogate model approximating the computationally expensive model across the input space, enabling efficient estimation of posterior distributions for model parameters (98). The Bayesian approach allows for incorporating prior knowledge and updating beliefs about model parameters based on observed data, yielding a posterior distribution that represents uncertainty about parameter values and the empirical data. This is particularly relevant in healthcare applications, such as modeling disease progression or patient outcomes, where data is often limited, error-prone, and computational resources are constrained (99). Bayesian GP calibration also facilitates robust UQ by capturing both aleatoric and epistemic uncertainty. As a result, the model predictions are accompanied by realistic uncertainty estimates, (100), which, as discussed in the following section, is important in the context of decision-making in critical healthcare scenarios.
Optimization Optimization-based calibration with active learning builds upon GPs to efficiently refine parameter estimates in complex simulation models. In this approach, the GP surrogate enables rapid evaluations of an otherwise computationally expensive objective function (i.e., the expensive simulation), guiding the search for optimal parameter values. Active learning further aids in this process by adaptively selecting data points in regions of high model uncertainty or near potential optima, thus prioritizing areas where new information would most improve model fidelity (101). The combination of GPs and adaptive sampling reduces the need for exhaustive simulations, focusing computational resources on the most informative parts of the parameter space (102).
The INFORM-HIV ABM contains complexities that require further considerations for applying the calibration approaches presented above. These include the need to address potential issues of non-identifiability (103). To improve feasibility there is also the need for dimension reduction techniques such as active subspaces (104) and variational autoencoders (105). These methods aim to capture lower-dimensional representations of the input data, facilitating more efficient exploration of the parameter space while accounting for the complex interdependent relationships represented in INFORM-HIV. Another challenge is addressing multiple (potentially dependent) calibration targets. A GP can be extended to a multi-output framework, where a single set of input parameters affects several outputs, and the correlation between these outputs is modeled via a covariance function (106). Alternatively, multi-objective optimization provides a method for optimizing multiple, often conflicting, outputs. This approach aims to identify a set of optimal solutions, known as the Pareto front, where improving one objective would degrade another. GP surrogates can be integrated into this optimization framework to enable efficient solution exploration (107).
6 From models to decision making
Despite the uncertainties described above, public health stakeholders must make decisions in the presence of limited information. Developing solutions to complex public health problems like HIV involves collaboration among many stakeholders from different sectors, with different and often competing priorities, and differences in focus in terms of policies, intervention levers, and measures of success. Incorporation of the extensive knowledge and applied public health experience among stakeholders is essential for developing meaningful models for public health decision making. However, there is a challenge in translating this knowledge into models that can inform decision making in contexts where there is a high degree of uncertainty from a data perspective as discussed in Section 5, but also characterized by deep uncertainty. In other words, in addition to uncertainties in the data used to parameterize models, there are aspects of the current and future states of the world that are unknowable. Tools to help stakeholders make informed decisions in the presence of uncertainties in data, causal structures, transmission networks, and future states are needed to develop effective tools for modeling interventions to address SDOH in complex systems such as HIV.
Robust Decision Making (RDM) is a multi-scenario, multi-objective decision-analytic approach designed to inform policy decisions when decision-makers and scientists lack confidence or agreement in either (i) the appropriate models to describe interactions among a system’s variables, (ii) the probability distributions to represent uncertainty about key parameters in a model, or (iii) how to value the desirability of alternative outcomes (108). Under these conditions, often labeled as deep uncertainty (109), traditional decision-analytical approaches can lead to gridlock over contested assumptions or unwarranted overconfidence in model predictions, both of which can prove counterproductive (108).
RDM offers a rigorous approach to address the deep uncertainties inherent in modeling SDOH and their impacts on HIV transmission described in this paper. The RDM process typically begins with stakeholder engagement to develop a conceptual framework, identify potential interventions, set policy objectives, and recognize key uncertainties. These elements are then programmed into a simulation model, such as an agent-based model. Experiments are conducted to test how different interventions perform across a wide range of future scenarios and assumptions. RDM studies then focus on assessing the robustness of policy decisions to uncertainty and on clarifying the conditions under which policies fail to meet policymakers’ goals. Analysis of the results focuses on identifying robust strategies that perform well across many plausible futures rather than optimizing for a single set of assumptions (110, 111). RDM’s focus on clarifying the conditions under which policymakers’ goals can be achieved is particularly relevant for HIV elimination efforts as policymakers seek combinations of policy interventions that can achieve HIV elimination goals.
Existing RDM applications in the life sciences provide examples of how RDM can help address deep uncertainty in HIV modeling. For instance, (112) used a scenario discovery approach to identify conditions under which blood-based tests could be superior to established tests for colorectal cancer screening. This approach identified regions in the parameter space where blood tests could be more cost-effective than existing tests. Similarly, RDM was used to inform the decision to reopen the economy following the vaccination campaigns in the COVID-19 pandemic, addressing uncertainties about the transmissibility of future variant strains and the population’s willingness to accept a new vaccine (113, 114). Following the COVID-19 pandemic, RDM was also used to demonstrate how public health decision-makers can weigh health and economic outcomes while accounting for uncertainty when introducing nonpharmaceutical interventions (115).
RDM can support HIV elimination decision-making in multiple ways. First, RDM can be used to stress-test the pathway to HIV elimination that current HIV elimination plans aim to achieve. Such stress-tests can be helpful to increase the confidence that existing HIV elimination plans can achieve HIV elimination goals, or may reveal that current levels of funding and effort are not sufficient to achieve aspirational goals. Although advocates and policymakers have set goals to mobilize resources, it is unclear whether those goals are either overambitious or are not sufficient to achieve HIV elimination. Second, RDM can be used to stress-test the robustness of multiple alternative policy portfolios to help decision-makers understand how to “dose” different policy interventions. Finally, in an environment where funding for HIV elimination interventions cannot be taken for granted, RDM can help decision-makers understand the critical policies without which HIV elimination cannot be achieved.
INFORM-HIV could be further extended to include endogenous adaptation in agent behaviors in response to epidemic trends. Currently, rules by which agent characteristics and interactions impact epidemic dynamics are specified as exogenous inputs. However, in the context of HIV and other infectious diseases, people often adapt their behavior in response to changes in the epidemiology of the disease, in addition to factors such as the availability of effective treatments or vaccines and perceived risk (22–24). Adaptive responses could also be modified by SDOH. RDM tools could play a critical role given the additional deep uncertainties associated with modeling adaptive behavioral responses.
7 Conclusions and future directions
When addressing complex public health issues such as HIV, policymakers are often required to make decisions in the presence of deep uncertainty in multiple domains. ABMs can help to understand and quantify the population-level impacts of combinations of biomedical, psychosocial, and structural interventions. However, to quantify the impact of these interventions, ABMs require parameter estimates that reflect causal relationships and structures among many interconnected factors, which are hard to quantify using standard statistical techniques. There is a need for further development of computational methods to address uncertainty in highly complex ABMs, as well as tools to aid in public health decision making in order to translate ABM results into actionable interventions. Integrating methods from statistics, agent-based modeling, and robust decision-making could offer new tools for understanding and addressing complex public health issues.
Author contributions
AH: Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Supervision, Writing – original draft, Writing – review & editing. PN: Conceptualization, Methodology, Writing – original draft, Writing – review & editing, Investigation. AF: Conceptualization, Methodology, Writing – original draft, Writing – review & editing, Investigation. NC: Methodology, Software, Writing – original draft, Conceptualization, Investigation. AK: Conceptualization, Methodology, Writing – review & editing, Investigation. DM: Conceptualization, Writing – review & editing, Investigation. ET: Writing – review & editing, Methodology, Software. SR: Methodology, Software, Writing – review & editing. EA: Writing – review & editing, Project administration, Visualization. HP: Writing – review & editing, Conceptualization, Funding acquisition, Investigation. JS: Conceptualization, Funding acquisition, Writing – review & editing, Investigation. RL: Conceptualization, Funding acquisition, Writing – review & editing, Investigation, Methodology. JO: Conceptualization, Funding acquisition, Methodology, Writing – review & editing, Investigation, Software, Supervision, Writing – original draft.
Funding
The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This work was supported by National Institutes of Health grants R21MH128116 and R01DA057350 and by the National Science Foundation under Grant No. 2200234. AK was additionally supported by the Providence/Boston CFAR (P30AI042853).
Acknowledgments
The authors would like to thank Dr. Raffaele Vardavas for helpful discussions and feedback on the manuscript.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The author(s) declare that no Generative AI was used in the creation of this manuscript.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1. Mayer KH, Nelson L, Hightow-Weidman L, Mimiaga MJ, Mena L, Reisner S, et al. The persistent and evolving HIV epidemic in American men who have sex with men. Lancet. (2021) 397:1116–26. doi: 10.1016/S0140-6736(21)00321-4
2. Sullivan PS, Johnson AS, Pembleton ES, Stephenson R, Justice AC, Althoff KN, et al. Epidemiology of HIV in the USA: epidemic burden, inequities, contexts, and responses. Lancet. (2021) 397:1095–106. doi: 10.1016/S0140-6736(21)00395-0
3. Beyrer C, Adimora AA, Hodder SL, Hopkins E, Millett G, Mon SHH, et al. Call to action: how can the US ending the HIV epidemic initiative succeed? Lancet. (2021) 397:1151–6. doi: 10.1016/S0140-6736(21)00390-1
4. Phillips G, McCuskey D, Ruprecht MM, Curry CW, Felt D. Structural interventions for HIV prevention and care among US men who have sex with men: a systematic review of evidence, gaps, and future priorities. AIDS Behav. (2021) 25:2907–19. doi: 10.1007/s10461-021-03167-2
7. Hill BJ, Motley DN, Rosentel K, VandeVusse A, Fuller C, Bowers SME, et al. Employment as HIV prevention: an employment support intervention for adolescent men who have sex with men and adolescent transgender women of color. J Acquir Immune Defic Syndr. (2022) 91:31–8. doi: 10.1097/QAI.0000000000003020
8. Jennings Mayo-Wilson L, Coleman J, Timbo F, Ssewamala FM, Linnemayr S, Yi GT, et al. Microenterprise intervention to reduce sexual risk behaviors and increase employment and HIV preventive practices among economically-vulnerable African-American young adults (EMERGE): a feasibility randomized clinical trial. AIDS Behav. (2020) 24:3545–61. doi: 10.1007/s10461-020-02931-0
9. Lightner JS, Barnhart T, Shank J, Adams D, Valleroy E, Chesnut S, et al. Outcomes of the KC life 360 intervention: improving employment and housing for persons living with HIV. PLoS One. (2022) 17:e0274923. doi: 10.1371/journal.pone.0274923
10. Towe VL, Wiewel EW, Zhong Y, Linnemayr S, Johnson R, Rojas J. A randomized controlled trial of a rapid re-housing intervention for homeless persons living with HIV/AIDS: impact on housing and HIV medical outcomes. AIDS Behav. (2019) 23:2315–25. doi: 10.1007/s10461-019-02461-4
11. Cassels S, Clark SJ, Morris M. Mathematical models for HIV transmission dynamics: tools for social and behavioral science research. J Acquir Immune Defic Syndr. (2008) 47:S34. doi: 10.1097/QAI.0b013e3181605da3
12. Kermack WO, McKendrick AG. A contribution to the mathematical theory of epidemics. Proc R Soc Lond Ser A. (1927) 115:700–21. doi: 10.1098/rspa.1927.0118
13. Bjørnstad ON, Shea K, Krzywinski M, Altman N. The SEIRS model for infectious disease dynamics. Nat Methods. (2020) 17:557–8. doi: 10.1038/s41592-020-0856-2
14. Johnson LF, White PJ. A review of mathematical models of hiv/aids interventions and their implications for policy. Sex Transm Infect. (2011) 87:629–34. doi: 10.1136/sti.2010.045500
15. Khanna AS, Dimitrov DT, Goodreau SM. What can mathematical models tell us about the relationship between circular migrations and HIV transmission dynamics? Math Biosci Eng. (2014) 11:1065–90. doi: 10.3934/mbe.2014.11.1065
16. Krivitsky PN, Handcock MS. A separable model for dynamic networks. J R Stat Soc Ser B Stat Methodol. (2014) 76:29–46. doi: 10.1111/rssb.12014
17. Khanna AS, Schneider JA, Collier N, Ozik J, Issema R, Paola AD, et al. A modeling framework to inform preexposure prophylaxis initiation and retention scale-up in the context of “Getting to Zero” initiatives. AIDS. (2019) 33:1911–22. doi: 10.1097/qad.0000000000002290
18. Hamilton A, Haghpanah F, Tulchinsky A, Kipshidze N, Poleon S, Lin G, et al. Incorporating endogenous human behavior in models of COVID-19 transmission: a systematic scoping review. Dialogues Health. (2024) 4:100179. doi: 10.1016/j.dialog.2024.100179
19. Hunter E, Kelleher JD. Understanding the assumptions of an seir compartmental model using agentization and a complexity hierarchy. J Comput Math Data Sci. (2022) 4:100056. doi: 10.1016/j.jcmds.2022.100056
20. Murray EJ, Robins JM, Seage GR, Freedberg KA, Hernán MA. A comparison of agent-based models and the parametric G-formula for causal inference. Am J Epidemiol. (2017) 186:131–42. doi: 10.1093/aje/kwx091
21. Tracy M, Cerdá M, Keyes KM. Agent-based modeling in public health: current applications and future directions. Annu Rev Public Health. (2024) 39:77–94. doi: 10.1146/annurev-publhealth-040617-014317
22. Bergstrom CT, Hanage WP. Human behavior and disease dynamics. Proc Natl Acad Sci. (2024) 121:e2317211120. doi: 10.1073/pnas.2317211120
23. Fenichel EP, Castillo-Chavez C, Ceddia MG, Chowell G, Parra PAG, Hickling GJ, et al. Adaptive human behavior in epidemiological models. Proc Natl Acad Sci. (2011) 108:6306–11. doi: 10.1073/pnas.1011250108
24. Saad-Roy CM, Traulsen A. Dynamics in a behavioral-epidemiological model for individual adherence to a nonpharmaceutical intervention. Proc Natl Acad Sci USA. (2023) 120:e2311584120. doi: 10.1073/pnas.2311584120
25. Hernán MA. Invited commentary: agent-based models for causal inference–reweighting data and theory in epidemiology. Am J Epidemiol. (2015) 181:103–5. doi: 10.1093/aje/kwu272
26. Marshall BDL, Galea S. Formalizing the role of agent-based modeling in causal inference and epidemiology. Am J Epidemiol. (2015) 181:92–9. doi: 10.1093/aje/kwu274
27. Atkinson J-A, Page A, Wells R, Milat A, Wilson A. A modelling tool for policy analysis to support the design of efficient and effective policy responses for complex public health problems. Implement Sci. (2015) 10:26. doi: 10.1186/s13012-015-0221-5
28. Northridge ME, Metcalf SS. Enhancing implementation science by applying best principles of systems science. Health Res Policy Syst. (2016) 14:74. doi: 10.1186/s12961-016-0146-8
29. Cerdá M, Keyes KM. Systems modeling to advance the promise of data science in epidemiology. Am J Epidemiol. (2019) 188:862–5. doi: 10.1093/aje/kwy262
30. Yang Y, Auchincloss AH, Rodriguez DA, Brown DG, Riolo R, Diez-Roux AV. Modeling spatial segregation and travel cost influences on utilitarian walking: towards policy intervention. Comput Environ Urban Syst. (2015) 51:59–69. doi: 10.1016/j.compenvurbsys.2015.01.007
31. Huang W, Chang C-H, Stuart EA, Daumit GL, Wang N-Y, McGinty EE, et al. Agent-based modeling for implementation research: an application to tobacco smoking cessation for persons with serious mental illness. Implement Res Pract. (2021) 2:26334895211010664. doi: 10.1177/26334895211010664
32. Cerdá M, Tracy M, Keyes KM. Reducing urban violence: a contrast of public health and criminal justice approaches. Epidemiology. (2018) 29:142–50. doi: 10.1097/EDE.0000000000000756
33. Cerdá M, Tracy M, Keyes KM, Galea S. To treat or to prevent?: reducing the population burden of violence-related post-traumatic stress disorder. Epidemiology. (2015) 26:681–9. doi: 10.1097/EDE.0000000000000350
34. Mooney SJ, Shev AB, Keyes KM, Tracy M, Cerdá M. G-computation and agent-based modeling for social epidemiology: can population interventions prevent posttraumatic stress disorder?. Am J Epidemiol. (2022) 191:188–97. doi: 10.1093/aje/kwab219
35. Gantenberg JR, King M, Montgomery MC, Galárraga O, Prosperi M, Chan PA, et al. Improving the impact of HIV pre-exposure prophylaxis implementation in small urban centers among men who have sex with men: an agent-based modelling study. PLoS One. (2018) 13:e0199915. doi: 10.1371/journal.pone.0199915
36. Jenness S, Goodreau S, Rosenberg E, Beylerian E, Hoover K, Smith D, et al. Impact of the centers for disease control’s HIV preexposure prophylaxis guidelines for men who have sex with men. Men in the United States. J Infect Dis. (2016) 214:1800–7. doi: 10.1093/infdis/jiw223
37. Hogan JW, Galai N, Davis WW. Modeling the impact of social determinants of health on HIV. AIDS Behav. (2021) 25:215–24. doi: 10.1007/s10461-021-03399-2
38. de Oliveira RB, Rubio FA, Anderle R, Sanchez M, de Souza LE, Macinko J, et al. Incorporating social determinants of health into the mathematical modeling of HIV/AIDS. Sci Rep. (2022) 12:20541. doi: 10.1038/s41598-022-24459-0
39. Shannon CE. A mathematical theory of communication. Bell Syst Tech J. (1948) 27:623–56. doi: 10.1002/j.1538-7305.1948.tb00917.x
40. Church A. Edward F. Moore. Gedanken-experiments on sequential machines. Automata studies, edited by C. E. Shannon and J. Mccarthy, Annals of Mathematics Studies No. 34, litho-printed, Princeton University Press, Princeton 1956, pp. 129–153. J Symb Logic. (1958) 23:60–. doi: 10.2307/2964500
41. Mealy GH. A method for synthesizing sequential circuits. Bell Syst Tech J. (1955) 34:1045–79. doi: 10.1002/j.1538-7305.1955.tb03788.x
42. Harel D. Statecharts: a visual formalism for complex systems. Sci Comput Program. (1987) 8:231–74. doi: 10.1016/0167-6423(87)90035-9
43. Ozik J, Collier N, Combs T, Macal CM, North M. Repast simphony statecharts. J Artif Soc Soc Simul. (2015) 18:11. doi: 10.18564/jasss.2840
44. Goodreau SM. A decade of modelling research yields considerable evidence for the importance of concurrency: a response to Sawers and Stillwaggon. Afr J Reprod Gynaecol Endosc. (2011) 14:12. doi: 10.1186/1758-2652-14-12
45. Goodreau SM, Cassels S, Kasprzyk D, Montaño DE, Greek A, Morris M. Concurrent partnerships, acute infection and hiv epidemic dynamics among young adults in zimbabwe. AIDS Behav. (2012) 16:312–22. doi: 10.1007/s10461-010-9858-x
46. Mah TL, Halperin DT. Concurrent sexual partnerships and the HIV epidemics in Africa: evidence to move forward. AIDS Behav. (2010) 14:11–6. doi: 10.1007/s10461-008-9433-x
47. Handcock MS, Hunter DR, Butts CT, Goodreau SM, Krivitsky PN, Bender-deMoll S, et al. statnet: software tools for the statistical analysis of network data (2016). The Statnet Project. Available online at: http://www.statnet.org.
48. Hunter D, Handcock M, Butts C, Goodreau S, Morris M. ergm: a package to fit, simulate and diagnose exponential-family models for networks. J Stat Softw. (2008) 24:nihpa54860. doi: 10.18637/jss.v024.i03
49. Robins G, Snijders T, Wang P, Handcock M, Pattison P. Recent developments in exponential random graph (p*) models for social networks. Soc Networks. (2007) 29:192–215. doi: 10.1016/j.socnet.2006.08.003
50. Hunter DR, Goodreau SM, Handcock MS. Goodness of fit of social network models. J Am Stat Assoc. (2008) 103:248–58. doi: 10.1198/016214507000000446
51. Krivitsky PN, Handcock MS, Morris M. Adjusting for network size and composition effects in exponential-family random graph models. Stat Methodol. (2011) 8:319–39. doi: 10.1016/j.stamet.2011.01.005
52. Carnegie NB, Krivitsky PN, Hunter DR, Goodreau SM. An approximation method for improving dynamic network model fitting. J Comput Graph Stat. (2015) 24:502–19. doi: 10.1080/10618600.2014.903087
53. Goodreau S, Handcock M, Hunter D, Butts C, Morris M. A statnet tutorial. J Stat Softw. (2008) 24:1–27. doi: 10.18637/jss.v024.i09
55. Chandra C, Morris M, Van Meter C, Goodreau SM, Sanchez T, Janulis P, et al. Comparing sexual network mean active degree measurement metrics among men who have sex with men. Sex Transm Dis. (2022) 49:808–14. doi: 10.1097/OLQ.0000000000001708
56. Krivitsky PN, Morris M, Bojanowski M. Impact of survey design on estimation of exponential-family random graph models from egocentrically-sampled data. Soc Networks. (2022) 69:22–34. doi: 10.1016/j.socnet.2020.10.001
57. Uong S, Rosenberg ES, Goodreau SM, Luisi N, Sullivan P, Jenness SM. Assessment of bias in estimates of sexual network degree using prospective cohort data. Epidemiology. (2020) 31:229–37. doi: 10.1097/EDE.0000000000001151
58. Aidala AA, Wilson MG, Shubert V, Gogolishvili D, Globerman J, Rueda S, et al. Housing status, medical care, and health outcomes among people living with {HIV}/{AIDS}: a systematic review. Am J Public Health. (2016) 106:1–23. doi: 10.2105/ajph.2015.302905
59. Nelson LE, Wilton L, Moineddin R, Zhang N, Siddiqi A, Sa T, et al. Economic, legal, and social hardships associated with HIV risk among black men who have sex with men in six US cities. J Urban Health. (2016) 93:170–88. doi: 10.1007/s11524-015-0020-y
60. Hussen SA, Doraivelu K, Camp DM, Moore SJ, Kalokhe AS, Wade R, et al. Burden and correlates of mental health symptoms among young black gay, bisexual, and other men who have sex with men engaged in HIV care in Atlanta. AIDS Behav. (2022) 26:2844–54. doi: 10.1007/s10461-022-03629-1
61. Liu Y, Brown L, Przybyla S, Bleasdale J, Mitchell J, Zhang C. Characterizing racial differences of mental health burdens, psychosocial determinants, and impacts on hiv prevention outcomes among young men who have sex with men: a community-based study in two U.S. cities. J Racial Ethn Health Disparities. (2021) 9:1114–24. doi: 10.1007/s40615-021-01052-1
62. Holloway IW, Beltran R, Shah SV, Cordero L, Garth G, Smith T, et al. Structural syndemics and antiretroviral medication adherence among black sexual minority men living with HIV. J Acquir Immune Defic Syndr. (2021) 88:S12–9. doi: 10.1097/QAI.0000000000002806
63. Miller SJ, Harrison SE, Sanasi-Bhola K. A scoping review investigating relationships between depression, anxiety, and the PrEP care continuum in the United States. Int J Environ Res Public Health. (2021) 18:11431. doi: 10.3390/ijerph182111431
64. Murray EJ, Robins JM, Seage GR, Freedberg KA, Hernán MA. The challenges of parameterizing direct effects in individual-level simulation models. Med Decis Making. (2020) 40:106–11. doi: 10.1177/0272989X19894940
65. Murray EJ, Robins JM, Seage GR, Lodi S, Hyle EP, Reddy KP, et al. Using observational data to calibrate simulation models. Med Decis Making. (2018) 38:212–24. doi: 10.1177/0272989X17738753
66. Naimi AI. Commentary: integrating complex systems thinking into epidemiologic research. Epidemiology. (2016) 27:843–7. doi: 10.1097/EDE.0000000000000538
67. Hernán MA, VanderWeele TJ. Compound treatments and transportability of causal inference. Epidemiology. (2011) 22:368–77. doi: 10.1097/EDE.0b013e3182109296
68. Thiele JC, Kurth W, Grimm V. Facilitating parameter estimation and sensitivity analysis of agent-based models: a cookbook using netlogo and R. J Artif Soc Soc Simul. (2014) 17:11. doi: 10.18564/jasss.2503
69. Ghanem R, Higdon D, Owhadi H (editors). Handbook of Uncertainty Quantification. New York: Springer (2017). Vol. 6.
70. Pathmanathan P, Cordeiro JM, Gray RA. Comprehensive uncertainty quantification and sensitivity analysis for cardiac action potential models. Front Physiol. (2019) 10:721. doi: 10.3389/fphys.2019.00721
71. Saltelli A, Tarantola S, Chan K, Campolongo F. Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index. Comput Phys Commun. (2008) 180:2584–96. doi: 10.1016/j.cpc.2009.09.018
72. Morris MD. Factorial sampling plans for preliminary computational experiments. Technometrics. (1991) 33:161–74. doi: 10.1080/00401706.1991.10484804
73. Hotton AL, Ozik J, Kaligotla C, Collier N, Stevens A, Khanna AS, et al. Impact of changes in protective behaviors and out-of-household activities by age on COVID-19 transmission and hospitalization in Chicago, Illinois. Ann Epidemiol. (2022) 76:165–73. doi: 10.1016/j.annepidem.2022.06.005
74. Ozik J, Collier NT, Wozniak JM, Spagnuolo C. From desktop to large-scale model exploration with swift/T. In: 2016 Winter Simulation Conference (WSC). (2016). p. 206–20.
75. Ozik J, Wozniak JM, Collier N, Macal CM, Binois M. A population data-driven workflow for COVID-19 modeling and learning. Int J High Perform Comput Appl. (2021) 35:483–99. doi: 10.1177/10943420211035164
76. Binois M, Gramacy RB, Ludkovski M. Practical heteroscedastic gaussian process modeling for large simulation experiments. J Comput Graph Stat. (2018) 27:808–21. doi: 10.1080/10618600.2018.1458625
77. Fadikar A, Collier N, Stevens A, Ozik J, Binois M, Toh KB. Trajectory-oriented optimization of stochastic epidemiological models. In: 2023 Winter Simulation Conference (WSC). IEEE (2023). p. 1244–55.
78. Kimpton L, Challenor P, Salter J. Data from: Uncertainty quantification for agent based models: a tutorial. arXiv [Preprint]. arXiv:2409.16776 (2024).
79. ten Broeke G, van Voorn G, Ligtenberg A, Molenaar J. The use of surrogate models to analyse agent-based models. J Artif Soc Soc Simul. (2021) 24:3. doi: 10.18564/jasss.4530
80. Gramacy RB. Surrogates: Gaussian Process Modeling, Design and Optimization for the Applied Sciences. Boca Raton, FL: Chapman Hall/CRC (2020).
81. Lüthen N, Marelli S, Sudret B. Automatic selection of basis-adaptive sparse polynomial chaos expansions for engineering applications. Int J Uncertain Quantif. (2022) 12:49–74. doi: 10.1615/Int.J.UncertaintyQuantification.2021036153
82. Cockrell C, Ozik J, Collier N, An G. Nested active learning for efficient model contextualization and parameterization: pathway to generating simulated populations using multi-scale computational models. Simulation. (2021) 97:287–96. doi: 10.1177/0037549720975075
83. Fadikar A, Higdon D, Chen J, Lewis B, Venkatramanan S, Marathe M. Calibrating a stochastic, agent-based model using quantile-based emulation. SIAM/ASA J Uncertain Quantif. (2018) 6:1685–706. doi: 10.1137/17M1161233
84. Sobol’ IM. Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Math Comput Simul. (2001) 55:271–80. doi: 10.1016/S0378-4754(00)00270-6
85. McKay MD, Beckman RJ, Conover WJ. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics. (1979) 21:239–45. Avilable online at: https://www.jstor.org/stable/1268522
86. Chauhan MS, Ojeda-Tuz M, Catarelli RA, Gurley KR, Tsapetis D, Shields MD. On active learning for gaussian process-based global sensitivity analysis. Reliab Eng Syst Saf. (2024) 245:109945. doi: 10.1016/j.ress.2024.109945
87. Gramacy RB, Apley DW. Local approximate gaussian process regression. J Comput Graph Stat. (2015) 24:561–78. doi: 10.1080/10618600.2014.914442
88. McCulloch J, Ge J, Ward JA, Heppenstall A, Polhill JG, Malleson N. Calibrating agent-based models using uncertainty quantification methods. J Artif Soc Soc Simul. (2022) 25:1. doi: 10.18564/jasss.4791
89. Grey JA, Bernstein KT, Sullivan PS, Purcell DW, Chesson HW, Gift TL, et al. Estimating the population sizes of men who have sex with men in US states and counties using data from the American community survey. JMIR Public Health Surveill. (2016) 2:e14. doi: 10.2196/publichealth.5365
90. Meerwijk EL, Sevelius JM. Transgender population size in the United States: a meta-regression of population-based probability samples. Am J Public Health. (2017) 107:e1–8. doi: 10.2105/AJPH.2016.303578
91. Purcell DW, Johnson CH, Lansky A, Prejean J, Stein R, Denning P, et al. Estimating the population size of men who have sex with men in the United States to obtain HIV and syphilis rates. Open AIDS J. (2012) 6:98–107. doi: 10.2174/1874613601206010098
92. Craig PS, Goldstein M, Seheult AH, Smith JA. Pressure matching for hydrocarbon reservoirs: a case study in the use of bayes linear strategies for large computer experiments. In: Case Studies in Bayesian Statistics. Springer (1997). Vol. 3. p. 37–93.
93. Vernon I, Goldstein M, Bower RG. Galaxy formation: a bayesian uncertainty analysis. Bayesian Anal. (2010) 5:619–69. doi: 10.1214/10-ba524
94. Andrianakis I, Vernon I, McCreesh N, McKinley TJ, Oakley JE, Nsubuga RN, et al. History matching of a complex epidemiological model of human immunodeficiency virus transmission by using variance emulation. J R Stat Soc Ser C Appl Stat. (2015) 64:517–36. doi: 10.1111/rssc.12198
95. Goldstein M, Rougier JC. Reified bayesian modelling and inference for physical systems. J Stat Plan Inference. (2009) 139:1221–39. doi: 10.1016/j.jspi.2008.07.019
96. Beaumont MA. Approximate bayesian computation. Annu Rev Stat Appl. (2019) 6:379–403. doi: 10.1146/annurev-statistics-030718-105212
97. Rutter CM, Ozik J, DeYoreo M, Collier N. Microsimulation model calibration using incremental mixture approximate Bayesian computation. Ann Appl Stat. (2019) 13:2189–212. doi: 10.1214/19-AOAS1279
98. Kennedy MC, O’Hagan A. Bayesian calibration of computer models. J R Stat Soc Ser B Stat Methodol. (2001) 63:425–64. doi: 10.1111/1467-9868.00294
99. Gramacy RB, Lee HKH. Bayesian treed gaussian process models with an application to computer modeling. J Am Stat Assoc. (2008) 103:1119–30. doi: 10.1198/016214508000000689
100. Higdon D, Gattiker J, Williams B, Rightley M. Computer model calibration using high-dimensional output. J Am Stat Assoc. (2004) 99:570–83. doi: 10.1198/016214507000000888
101. Jones DR, Schonlau M, Welch WJ. Efficient global optimization of expensive black-box functions. J Glob Optim. (1998) 13:455–92. doi: 10.1023/A:1008306431147
102. Gramacy RB, Lee HK. Adaptive design and analysis of supercomputer experiments. Technometrics. (2009) 51:130–45. doi: 10.1198/TECH.2009.0015
103. Tarantola A. Inverse Problem Theory and Methods for Model Parameter Estimation. Philadelphia, PA: SIAM (2005).
104. Constantine PG. Active Subspaces: Emerging Ideas for Dimension Reduction in Parameter Studies. Philadelphia, PA: Society for Industrial and Applied Mathematics (2015). http://epubs.siam.org/doi/book/10.1137/1.9781611973860
105. Kingma DP, Welling M. Auto-encoding variational bayes. In: Proceedings of the 2nd International Conference on Learning Representations (ICLR) (2014).
106. O’Hagan A. Bayesian analysis of computer code outputs: a tutorial. Reliab Eng Syst Saf. (2006) 91:1290–300. doi: 10.1016/j.ress.2005.11.025
107. Knowles JD, Parpinelli RS. The use of gaussian processes in multi-objective optimization. In: Proceedings of the 2006 IEEE Congress on Evolutionary Computation (CEC 2006). (2006). p. 321–8.
108. Lempert RJ. Robust decision making (RDM). In: Marchau VAWJ, Walker WE, Bloemen PJTM, Popper SW, editors. Decision Making under Deep Uncertainty: From Theory to Practice. Cham: Springer International Publishing (2019). p. 23–51.
109. Walker WE, Lempert RJ, Kwakkel JH. Deep uncertainty. In: Gass SI, Fu MC, editors. Encyclopedia of Operations Research and Management Science. Boston, MA: Springer US (2013). p. 395–402.
110. Lempert RJ, Groves DG, Popper SW, Bankes SC. A general, analytic method for generating robust strategies and narrative scenarios. Manage Sci. (2006) 52:514–28. doi: 10.1287/mnsc.1050.0472
111. Lempert RJ, Popper SW, Bankes SC. Shaping the Next One Hundred Years: New Methods for Quantitative, Long-Term Policy Analysis. Santa Monica, CA: RAND Corporation (2003).
112. Nascimento de Lima P, Van Den Puttelaar R, Knudsen AB, Hahn AI, Kuntz KM, Ozik J, et al. Characteristics of a cost-effective blood test for colorectal cancer screening. J Natl Cancer Inst. (2024) 116:1612–20. doi: 10.1093/jnci/djae124
113. Nascimento de Lima P, Lempert R, Vardavas R, Baker L, Ringel J, Rutter CM, et al. Reopening California: seeking robust, non-dominated COVID-19 exit strategies. PLoS One. (2021) 16:e0259166. doi: 10.1371/journal.pone.0259166
114. Nascimento de Lima P, Vardavas R, Baker L, Ringel JS, Lempert RJ, Rutter CM, et al. Reopening Under Uncertainty: Stress-Testing California’s COVID-19 Exit Strategy. Santa Monica, CA: RAND Corporation (2021).
Keywords: social determinants of health, agent-based models, HIV, infectious disease modeling, network models, uncertainty quantification, surrogate models, robust decision-making
Citation: Hotton AL, Nascimento de Lima P, Fadikar A, Collier NT, Khanna AS, Motley DN, Tatara E, Rimer S, Almirol E, Pollack HA, Schneider JA, Lempert RJ and Ozik J (2025) Incorporating social determinants of health into agent-based models of HIV transmission: methodological challenges and future directions. Front. Epidemiol. 5:1533119. doi: 10.3389/fepid.2025.1533119
Received: 23 November 2024; Accepted: 6 February 2025;
Published: 27 February 2025.
Edited by:
Romulus Breban, Institut Pasteur, FranceReviewed by:
Erin Bradley, Agnes Scott College, United StatesRodrigo Volmir Anderle, Federal University of Bahia, Brazil
Copyright: © 2025 Hotton, Nascimento de Lima, Fadikar, Collier, Khanna, Motley, Tatara, Rimer, Almirol, Pollack, Schneider, Lempert and Ozik. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Anna L. Hotton, YWhvdHRvbkBic2QudWNoaWNhZ28uZWR1