Challenges and Benefits of Approaches Used to Integrate Regional Monitoring Programs

Although challenging to develop and operate, some degree of integrated monitoring is often necessary, especially at regional scales, to address the complex questions of environmental management and regulation. The concept of integration is well-understood, but its practice across programs and studies can be diverse suggesting a broader examination of the existing general approaches is needed. From the literature, we suggest integration of monitoring can occur across three study components: interpretation, analysis, and design. Design can be further subdivided into partial and full integration. Respectively combining information, data, and designs, we further define these types of integration and describe their general benefits and challenges, such as strength of inference. We further use the Oil Sands Monitoring program in northern Alberta as an example to clarify the practices common among integrated monitoring programs. The goal of the discussion paper is to familiarize readers with the diverse practices of integrated monitoring to further clarify the various configurations used to achieve the wider goals of a program.

Introduction

Human activities influence the environment in many ways and monitoring over time is intended to provide timely, clear, systematic information on their status (Yoccoz et al., 2001). While needed to inform the larger strategic goals of reducing, eliminating, or controlling the impacts of those activities, multiple approaches are available for measuring environmental attributes and assessing status. However, monitoring may not always be straightforward and may not always provide widely-accepted and certain evidence showing the presence of impacts (leading to effective interventions), or their absence (and reassurance of safety; Munkittrick et al., 2019). For example, monitoring programs must often translate high-level management questions, such as “Are there effects?” into focused and testable scientific hypotheses which may not wholly reflect the original objectives. Monitoring must also contend with other challenges. Risk tolerances among stakeholders may vary and “safe”, or acceptable conditions may not be objectively characterized. There can also be unknown long-term consequences of low levels of environmental change (Soulé, 1985; Biber, 2011; Addison et al., 2016; Suter, 2016), obscure or unknown exposure of organisms (Tetreault et al., 2020), and various interactions of both anthropogenic and natural environmental stressors (Gieswein et al., 2017; Arciszewski et al., 2018).

The well-known challenges of monitoring have motivated the further development of approaches and tools (e.g., Lindenmayer and Likens, 2010). One common solution to improve the utility of information provided by a monitoring program is expanding its spatial, temporal, and technical scope (Dowdeswell et al., 2010; Gosselin et al., 2010; Lott and Jones, 2010; AEMP, 2011). However, as these scopes increase, new challenges can emerge. While it may be desirable, monitoring at regional scales (such as those larger than 10³ km²) can extend across different biomes (Hunsaker et al., 1990) and can incorporate the influence of multiple human activities and multiple stakeholders with diverse values. Regional monitoring initiatives may also cover areas comprised of multiple political jurisdictions with different environmental regional planning objectives and legal requirements, such as monitoring programs examining status and trends, environmental effects monitoring, compliance monitoring, cumulative effects assessment, pre-development baseline studies, and research programs.

Further solutions are often suggested and implemented to overcome the challenges of monitoring programs with wide temporal, spatial, and technical scopes. Most notable among them are the repeated calls for integration (e.g., Dowdeswell et al., 2010; Liu et al., 2012; Hopke et al., 2016). Integration of programs is seen as a solution to enhance efficiency and efficacy of programs (Lindenmayer and Likens 2010; Liu et al., 2012; Norton et al., 2014; Cooke and Hilton 2015; Arciszewski et al., 2017b), to provide increased sensitivity to surprises (Munn, 1988), and to foster scientific contributions to environmental management (Thackeray and Hampton, 2020).

While integration is popular and can be beneficial, it is often a strategic and conceptual objective (e.g., Bagstad et al., 2013; Dubé et al., 2013; Noble 2015; Arciszewski et al., 2017b; Cronmiller and Noble, 2018b) may refer to diverse practices. Integration can refer to multiple, but related actions including the combination of approaches (Burton et al., 2002a; Ankley et al., 2010), of several related measurements within a given study (Löfgren et al., 2011; Scrimgeour et al., 2013; Vethaak et al., 2017), or of measuring and modeling (Haeberli et al., 2007). Integration can also refer to the combination of information and data from a variety of spatial and temporal scales using techniques such as Weight-of-Evidence and development of causal criteria (Lowell et al., 2000; Cash et al., 2003). Finally, integration may also refer to the combination of multiple studies, including multiple field studies, or laboratory and field studies (Wolska et al., 2007; Barra et al., 2020), parallel and mutually supporting monitoring and research (Wrona and Cash 1996; Bunn et al., 2010), or monitoring based on tiers and triggers (Arciszewski and Munkittrick, 2015; Somers et al., 2018).

While all examples above satisfy the conceptual demands of integration, the diversity of its practices may not be recognized when initial recommendations for a program′s direction and objectives are made. This can be especially problematic if the conceptual and initial vision for an integrated program is also not explicitly and practically defined. In this discussion paper, our specific goal is to propose a framework to organize the types of integrated monitoring programs and describe the practical methods currently used to combine information, data, and designs. We also discuss their benefits and the potential challenges with each approach and frame these practices using the Oil Sands Monitoring (OSM) program; a multi-year, multi-indicator, multi-theme, multi-million dollar program in northern Alberta. Finally, we conclude with a general discussion of how to overcome the challenges of different forms of integration and how they can occur simultaneously within a large regional monitoring program. Our general goal with this discussion is to clarify the various practices of integration and provide a common language for use among participants of large regional monitoring programs.

Oil Sands Monitoring

The Oil Sands Region (OSR) of northeastern Alberta covers a surface area of 142,200 km² containing three deposits of bitumen, together comprising the third largest known reserve of hydrocarbons in the world (Figure 1). Where the bitumen is within ∼75 m of the surface, it is extracted through conventional truck and shovel mining. Where the reserves are deeper, in situ methods, often using steam-assisted gravity drainage (SAGD), liquefy the bitumen in the reservoir and transport it to the surface. The extraction of bitumen occurs throughout the region. Within the 4,800 km² minable region north of Fort McMurray, roughly 1,264 km² is disturbed (ABMI, 2018). As of 2018, ∼22,252 km² of the total OSR is physically disturbed (ABMI, 2018), but this is most intense within the Athabasca Oil Sands (sub) region. While physically located in northern Alberta, gases and particles emitted to the atmosphere can be deposited in downwind areas outside the OSR, including northern Saskatchewan (Makar et al., 2018). The activity is also the subject of intense local, national, and global environmental concern and scrutiny (e.g., Miall, 2013).

FIGURE 1

FIGURE 1. Locations of monitoring areas and activities done under OSM in the OSR of north eastern Alberta; (A): Key maps showing location in North America; (B): Map of the surface minable area (white polygon with dashed outline; (C): Map of the Athabasca OS Region (D): Map of the Peace OS Region; (E): Map of the Cold Lake OS Region; Environmental monitoring sites are shown as colored points (green = land; blue = water; red = air; orange = groundwater; purple = wetlands), with shapes representing different monitoring partner organizations.

Regional monitoring initiatives have been common in northern Alberta. Historically, there were many monitoring programs examining potential changes in the ambient environment surrounding oil sands development, including regional cooperative programs, industry-driven programs, industry-funded programs, provincial and federal monitoring programs, and joint government initiatives (Figure 2). However, in 2009 and 2010 independent research showing the accumulation of contaminants of concern in snow near mines was published (Kelly et al., 2009; Kelly et al., 2010). This work highlighted gaps in monitoring and initiated reviews of the existing regional programs (e.g., Dowdeswell et al., 2010; AEMP, 2011). The reviews of monitoring consistently identified the need to achieve a more fully integrated monitoring system in the OSR (e.g., Gosselin et al., 2010; Lott and Jones, 2010). For example, the lack of data compatibility, including lack of consistency in chemical measurements and biological indicators, truncated sampling schedules, incompatible analytical methods, and fragmented data sets have affected the attempts at integrated interpretation and analyses conducted prior to 2012 (Gosselin et al., 2010).

FIGURE 2

FIGURE 2. Status of historical and current monitoring or data collection entities operating in the Oil Sands region; AOSERP = Alberta Oil Sands Environmental Research Program; NRBS = Northern River Basins Study; NREI = Northern Rivers Ecosystem Initiative; RAMP = Regional Aquatics Monitoring Program; WBEA = Wood Buffalo Environmental Association; TEEM = Terrestrial Environmental Effects Monitoring; ABMI = Alberta Biodiversity Monitoring Institute; CEMA = Cumulative Effects Management Association; AENV = Alberta Environment; OSLI = Oil Sands Leadership Initiative; EMCLA = Ecological Monitoring Committee for the Lower Athabasca; PAZA = Peace Airshed Zone Association; LICA = Lakeland Industry and Community Association; SEWG = Sustainable Ecosystems Working Group; TEK = Traditional Ecological Knowledge; GWWG = Groundwater working group; SWWG = Surfacewater working group; RWG = Regional Working Group.

In 2012 the existing programs were replaced by the Joint Oil Sands Monitoring Program (JOSM). JOSM was intended to be an integrated program (Environment Canada, 2011a) and was implemented in phases for each target medium (water, air, terrestrial). First the surface water program was released (Baird et al., 2011) followed later by separate program designs for terrestrial biodiversity (Environment Canada, 2011b), air (Environment Canada, 2011c), and an expanded geographical area (Wrona et al., 2011). In 2016, JOSM was rebranded as the OSM program (Dubé et al., 2018) after changes to administratively streamline and simultaneously broaden the governance of the program (Figure 3). OSM is currently administered and operated jointly by the Governments of Alberta and Canada and funded through a $50 M per year industry levy. The total funding allocated for the OSM/JOSM program is now approaching $500 M CAD.

FIGURE 3

FIGURE 3. Current administrative structure of the Oil Sands Monitoring program including central (red) administrative and governance committees (Indigenous Community Based Monitoring Advisory Committee; Oversight Committee; Science and Indigenous Knowledge Integration Committee), peripheral Technical Advisory Committees (in blue; e.g., Groundwater), and delivery partners (green) identified from the most recent publicly available workplans available (2017–2018) from http://environmentalmonitoring.alberta.ca/activities/oil-sands-monitoring-projects/20172018-projects/; not shown are additional linkages with Environment and Climate Change Canada, Alberta Environment and Parks, or industry organizations; PRAMP = Peace River Area Monitoring Program; See Figure 2 for additional acronym definitions.

The technical scope of OSM is broad. It examines multiple indicators across multiple media, including air, water, land, wetlands, and groundwater (Environment Canada, 2011a; Figure 1). OSM also partitions programs into “focused studies” and “core monitoring”. Focused studies are diverse and include many different types of activities, including research and methods development. Core monitoring programs include long-term routine programs such as water quality (Swanson, 2019a; Swanson, 2019b). A current emphasis of OSM is also expanding the role of community-based monitoring (e.g., Brunet et al., 2020) and inclusion of Indigenous Knowledge and Wisdom throughout the program, including establishing Indigenous representation within the program governance (Dubé et al., 2018).

Another objective of reconfiguring the regional monitoring in the OSR was increasing the quality of the work through external peer-review. While many papers have been published by researchers funded by this program (e.g., Summers et al., 2016; Makar et al., 2018; Landis et al., 2019), additional work also occurs in the region outside of the OSM funding envelope, including industry-funded research, compliance monitoring, provincial monitoring, and independent research (e.g., Shotyk et al., 2017; Redman et al., 2018; Fennell and Arciszewski, 2019). Since 2012 these research efforts have collectively produced hundreds of research papers.

Despite the initial aspiration for OSM to be a fully integrated program, much of the research remains isolated in individual research papers. A recent review of the regional ambient monitoring in the OSR concluded that more effort is needed to achieve the original objective of an integrated program (Hopke et al., 2016). While the initial efforts of integration were undermined by challenges with the timing of initial sampling and program roll-out, inertia from legacy monitoring programs, and the absence of a clear and coordinated strategic plan, we also suggest those early and continuing efforts are affected by the potentially unrecognized diversity of practices used to satisfy the delivery of an integrated program.

Models of Integration

Within any monitoring application, integration refers to the straightforward act of combination. While the specifics, preferences, and traditions of integration may differ by discipline (e.g., Linkov et al., 2009), we suggest four hierarchical and general models are typically used to integrate monitoring programs: integrated interpretation, integrated analysis, and partially and fully integrated designs (Table 1). Based on this framework, monitoring studies can be differentiated by the timing (when in a study cycle) and approach (what: data, conclusions, or designs) to integrate. The approaches to integration can also be differentiated by their attributes, including flexibility, strength of inference, sensitivity to errors, and administrative burden (Table 2).

TABLE 1

TABLE 1. Summary of when types of integration typically occur, how they are undertaken, their benefits, disadvantages, and potential outcomes.

TABLE 2

TABLE 2. Attributes to consider in the integration of monitoring “early” (e.g., during design, a priori) or “later” (e.g., during interpretation, retrospectively, a posteriori) phases of the program and typical qualities (e.g., high, low) of applying integration during those study phases.

A Posteriori Integration

Integration of monitoring can occur in multiple ways, but when the combination comprises existing data, conclusions or information, this is considered here as a posteriori integration. A posteriori integration can, however, be further subdivided into integrated interpretation and integrated analysis.

Integrated Interpretation

Of the four general types of integration, the most common, recognizable, and straightforward is the contextualization of conclusions or information when reporting the results of a study. Here we call this form Integrated Interpretation. Integrated interpretation involves the combination of information simultaneously considering conclusions from several independent, but related studies, each potentially addressing a specific hypothesis or set of questions. Typically, integrated interpretation occurs as the discussion of results in scientific publications. Methods, such as Weight-of-Evidence have also been developed and use quantitative, qualitative, or semi-quantitative techniques including scoring, ranking, and weighting to combine information (Burton et al., 2002b; Chapman et al., 2002). Broad approaches such as synthesis reports or general review papers (e.g., Mupepele et al., 2015) are common examples.

The benefits of integrated interpretation include quickly summarizing information and that a priori planning and administrative oversight are not necessary. Consequently, integrated interpretation can be done retrospectively with ease and the approach is flexible. The more closely the information aligns in space, time, or methodological approach, the stronger the combined interpretation, but even loosely associated information can be combined to generate novel and testable hypotheses.

Integrated interpretation is common within the OSM, reflected in hundreds of published research papers, each including, at minimum, the contextualization of results with previous findings (e.g., Landis et al., 2019) and synthesis of results (Davidson et al., 2020). OSM researchers have also published some review papers (e.g., Harner et al., 2018).

Integrated Analysis

Integration also commonly occurs during analysis. In integrated analysis, data from disparate projects may also be combined to address new questions or hypotheses, or to decrease uncertainty around an established question or set of questions to generate novel results. Meta-analyses are a potent example of this type of integration (Lipsey and Wilson, 2001).

Benefits of integrated analysis include more accurate information or results (Bustamante et al., 2016) and the ability to draw inferences at multiple spatiotemporal scales (Bricker and Ruggiero, 1998; Haeberli et al., 2007). Combining data, rather than conclusions often allows evaluation of hypotheses that the original studies were not designed to directly test. Integrated analysis can be done retrospectively with ease if information is available, accessible, and relevant to the study questions of the investigator. Publicly available data can facilitate integrated analyses. Similar to integrated interpretation, integrated analysis does not necessarily require a priori administrative oversight, systematic planning, or coordinated collection of data. Integrated interpretations are also nested within integrated analyses. The relative simplicity of integrated analysis and its associated adaptability contributes to its popularity.

Within OSM, integrated analyses are facilitated both by direct researcher collaboration, but also by providing data on a publicly-accessible databases. Integrated analyses in OSM have included combining individual studies undertaken in overlapping periods to enhance interpretation (e.g., Fisher and Burton 2018; Eccles et al., 2019; Kilgour et al., 2019). Integrated analyses have also been conducted using closely related data collected by few researchers examining similar questions (Wasiuta et al., 2019) and may include digested data summarized for study areas in a weight-of-evidence approach (Culp et al., 2018; Culp et al., 2020), or the combination of several raw datasets for a originally collected for different purposes, such as discharge and water quality (Arciszewski et al., 2018), discharge, air temperature, and health of fishes (Kilgour et al., 2019; Arciszewski and McMaster, 2021), and land use and water quality (Schwalb et al., 2015; Alexander and Chambers, 2016). Studies have also conducted formal geospatial analyses and modeling (Eccles et al., 2019) and compared novel measurement techniques with conventional emissions reporting (Li et al., 2017). Evaluations of the environmental effects of oil sands operations have benefited from some efforts to complete integrated regional analyses and interpretations (Alexander and Chambers, 2016; Culp et al., 2020). These analyses improved understanding of environmental stressors from oil sands operations, such as deposition models (Kirk et al., 2014; Wasiuta et al., 2019). The use of data products, such as standardized effect sizes, meta-analyses comparing strength of results and other methodological parameters (Lipsey and Wilson, 2001) has not yet been fully explored within OSM.

Challenges and Limitations of a Posteriori Integration

While common and straightforward, there can be significant challenges with a posteriori integration. More specifically, the challenges include the misalignment of data sets and limitations in the information and data available. Within OSM, there have been challenges with timely and synchronous availability of related data sets, resulting in significant delays in releasing results and undertaking integrated analyses by third parties. A posteriori types of integration prohibit logistical and economical efficiencies associated with integrated designs and sacrifice the greater statistical power of aligning sampling dates and methodologies. These deficiencies are not limited to OSM, and previous studies have noted limitations of data availability (Parr et al., 2002; Nilssen et al., 2015), inaccessibility (Liu et al., 2012; Cronmiller and Noble, 2018a), fragmentation (Liu et al., 2012), and incompatibility (Janousek et al., 2019). Incompatability can occur for many reasons, including differences in data types, resolution, or scales (Paul et al., 2000; Teatini et al., 2005; Liu et al., 2012; Bustamante et al., 2016; Cronmiller and Noble, 2018b). Limitations of data quality and accuracy (Teatini et al., 2005; Liu et al., 2012), or lack of familiarity with datasets among third-party users can also affect interoperability.

Challenges with a posteriori interpretation and analyses from the OSR highlight vulnerabilities of using data and information from multiple sources. In the OSR, studies have included, either directly or indirectly, inaccurate details of industrial sites, such as the placement of emission sources like upgrading complexes and the location of facilities and their operational status in the analysis and interpretation (Kurek et al., 2013; Kirk et al., 2014; Alexander and Chambers, 2016; Liggio et al., 2016; Summers et al., 2016; Mundy et al., 2019; Tetreault et al., 2020). Unfamiliarity with other features beyond industrial sources in the study area, such as potentially confounding influences of outboard motors on the accumulation of polyaromatic compounds in the sediments of remote lakes with a fly-in fishing lodge (e.g., Kurek et al., 2013) has also occurred. While the consequences of drawing conclusions based on incorrect information may be innocuous, it can inflate the Type I and Type II error rates and over- or under-report effect sizes which affect priorities in follow-up research.

Addressing the challenges of existing datasets and information requires effort. Resolving interoperability issues may require collapse, reformatting, stretching, compression, interpolation, extrapolation, or other transformation or summarization to reconcile datasets (Posthuma et al., 2016). Indices are popular choices across multiple fields for combining and decomposing data (Tarassenko et al., 2006), but are also an imperfect solution (Green and Chapman, 2011). These approaches to integration may also lead to decision fatigue when combining data, especially where assessment criteria are not available (Vethaak et al., 2017) and complex analyses and unique workflows emerge (Botvinik-Nezer et al., 2020). Some questions, such as change relative to an unknown baseline cannot be addressed with integrated interpretation and analysis. While interpretation and analysis are common and straightforward approaches, these tools may also be cumbersome when addressing interdisciplinary questions, such as cumulative effects.

While not unique, programs relying on a posteriori integration can be more susceptible to scope creep than a priori approaches. While expansion is not necessarily disadvantageous, there can be temptation to continually add measurements or pursue potentially unnecessary precision (Lindenmayer and Likens, 2010) which may strain or otherwise re-prioritize funding. Subdivision, branching, and extension of study designs can create challenges for consistency of datasets and for the eventual use of the data in decision-making (Sarkar et al., 2020). Intentional management of routine and novel studies are required in monitoring (Wintle et al., 2010) even in cases where the intent is to transition the program from a posteriori to a priori integration.

Although it may be tempting to derive strong inferences and conclusions from integrated interpretation or integrated analysis, the information generated in a retrospective and integrated analysis adopts the limitations and assumptions embedded in the original papers or datasets (Liu et al., 2012; Arciszewski and McMaster 2021). The results of a posteriori integration are necessarily limited to hypothesis generation (Lindenmayer and Likens 2010; Arciszewski et al., 2017a; Arciszewski et al., 2017b) since there may be challenges with multi-causality of issues (Liu et al., 2012) and over-simplification of complex (multi-covariate) analyses (Nilssen et al., 2015). Consequently, retrospective analyses are associated with weak inferences (Yoccoz et al., 2001; Mupepele et al., 2015).

A Priori Integration

While complete overlap is not required for a functional program, some a priori co-ordination of data collection, potentially involving multiple sub-projects, each with different investigators with differing expertize is commonly desired. The occurrence of some coordination and planning across groups, questions, approaches, or tools is considered here as a priori integration. A priori integration includes either the complete or partial design of a program prior to the collection of any data (e.g., Petitjean et al., 2020). Additionally, fully integrated studies may be part of a partially integrated program.

Partially Integrated Design (Integrated Data Collection)

A partially integrated design examines some fraction of the complete set of questions, indicators, or environments a program is designed to address. Partially integrated designs are common in the scientific literature and include programs such as the collection of data from biological indicators along with information for some environmental contaminants (Cruz-Martinez et al., 2015).

Partially integrated designs have several advantages over a posteriori integration. Partially integrated designs can provide stronger inferences than a posteriori approaches and can also be done opportunistically; in practice some measurements can be added in an integrated design until field crews vacate a study area. Integrated collection can enhance the suitability of data for broader and more complex applications and analyses, such as the combination of reach-scale data in streams to watershed-level questions (Wrona et al., 2000). Integrated data collection can also improve the ability to validate or compare multiple overlapping datasets (Teatini et al., 2005; Vethaak et al., 2017) through the a priori adoption of data quality objectives. Similarly, integrated collection may reduce overall costs and avoid unnecessary duplication of effort (Nilssen et al., 2015).

Integrated collection of data can facilitate integrated analysis and interpretation. For example, use of remote cameras and automated recording units are not species-specific and data are subject to the detection parameters of the devices (Zwart et al., 2014). Additionally, while an investigator may be interested in one particular species, they may collect data on many more (e.g., Zwart et al., 2014). Researchers are, however, not required to include these additional data in their analyses, but may often choose to do so. Another benefit of collecting secondary data is in adaptive monitoring constructs (Arciszewski et al., 2017b); partially integrated design ensures that some relevant data may be available to more deeply probe any emerging patterns or hypotheses (Arciszewski et al., 2017a).

Maturation of the OSM program has been accompanied with the desire for, and in some cases delivery of, a transition of integration through interpretation and analysis toward data collection. While not available in the public domain yet, the Terrestrial Biological Monitoring program within OSM has begun this process with the Before-After-Dose-Response (BADR) design. These efforts were supported by establishing technical advisory committees whose mandate includes standardizing data collection methodologies at the regional scale. These committees augmented early commitments for integrated monitoring of related studies in the initial design of the program (e.g., Environment Canada 2011a). Committees also supported efforts to optimize sampling designs for water quality (Cooke et al., 2018) and air monitoring (Soares et al., 2018; Wentworth and Zhang 2018). Among OSM studies, integrated data collection is relatively common in the aquatic theme (e.g., Culp et al., 2020). Measuring multiple indicators is also common in terrestrial programs (Mahon et al., 2019).

In contrast to within-themes, few examples of partially integrated designs are available which cross themes in OSM, but forest health is a prime example (e.g., Davidson et al., 2020). Expanding this approach to other studies within OSM has long been a goal (Dowdeswell et al., 2010). Importantly, however, the Forest Health example predates JOSM and is managed by a single administrative entity. Linkages of the forest health program to other study areas of OSM may be present, but these require deliberate and dedicated effort.

Fully Integrated Design

In contrast to a partially integrated design, a fully integrated design is considered the complete overlap of all study parts to address a single scientific objective (e.g., Cash et al., 2003). It accomplishes this not only by coordinating experimental designs and/or analytical units (as occurs in integrated collection), but also by asking more complex questions spanning multiple disciplines (Parr et al., 2002; Wolska et al., 2007; Vethaak et al., 2017) to enhance the understanding of causality (Chapman et al., 2010; Liu et al., 2012; Sperling et al., 2014; Cronmiller and Noble 2018b). Integrated designs underlie effective monitoring of entire systems (e.g., Wolska et al., 2007; Liu et al., 2012; Vethaak et al., 2017) and promote more efficient use of funds and can enhance credibility (Cronmiller and Noble, 2018a; Cronmiller and Noble, 2018b). Consequently, integrated designs can provide data for high-impact work.

There are some examples of integrated designs for individual studies in OSM, including the benthic macro-invertebrate program (Culp et al., 2020) and the Enhanced Monitoring Program designed to inform mine water return (Hicks and Scrimgeour, 2019). Studies within theme areas may also include measurements linking themes, but are currently rare. One example is the Representative Sub-basins Study (REPS) used to conduct necessary focused integrated monitoring, process studies, and modeling to assess and predict casual linkages of oil sands operations to observed effects (e.g., Cardoso et al., 2020). Studies within the same environmental medium tend to be internally consistent and more amenable to a priori integration.

While there are examples of isolated integrated designs in OSM, the program has taken many steps, with varying level of success, to work toward a fully integrated monitoring design. Efforts to develop a fully integrated design consist of four main actions: 1) identification of a framework for integration and efforts to impress the importance of it to program participants 2) enabling program participants to achieve integration through program design, 3) publicly accessible databases, and 4) public reporting to increase transparency and accountability (e.g., annual public reporting of the program and annual work plans, public data access). An important step was recognizing the importance of both western science and local and Indigenous knowledge, and that braiding of the two information systems will be foundational to achieving an integrated design.

Program participants were supported to achieve the overall integrated design by: 1) establishing technical advisory committees whose mandate included project integration within and among media 2) posting annual work plans on open internet portals, 3) producing synthesis reports within media (e.g., Culp et al., 2020), 4) releasing technical reports describing efforts to optimize the spatial and temporal design of water quality (Cooke et al., 2018) and air sampling (e.g., Soares et al., 2018; Wentworth and Zhang 2018), and 5) hosting technical workshops (Swanson, 2019a; Swanson, 2019b). While these efforts have been beneficial for program design and the understanding of ambient impacts, they require unexpectedly large initial and on-going time investments by scientists and managers, further underscoring the potentially high (and often front-loaded) investments of resources to reap the benefits of integrated design as a program matures.

Challenges and Limitations of a Priori Integration

While collection of multiple datasets at sites of interest using fully or partially integrated design facilitates integrated analysis and interpretation, there are also associated challenges, limitations, and considerations. Within each OSM theme, such as water and land, each study tends to address closely related questions with strong historical linkages to techniques dominant in the study of these media. For instance, ecotoxicological studies tend to dominate the water theme and landscape and wildlife ecology tend to dominate terrestrial studies. While some studies which do not follow this tendency have been published (e.g., Mundy et al., 2019), typically the practices used within a theme area can limit opportunities for integeration. A unifying approach within a theme may increase the likelihood of a broader integrated design, but approaches across specific media or indicators may be subject to potentially conflicting experimental designs and scales of analysis associated with the traditions of various disciplines requiring deliberate effort to reconcile.

Within a theme, there can still be challenges to overcome to develop an integrated design. For example, within OSM chemical measurements, such as water (Wasiuta et al., 2019) or air quality (Makar et al., 2018) may be instantaneous, near-continuous, or continuous, whereas physical measurements, such as flow may be summarized daily (Arciszewski et al., 2018), and measurements of biota may reflect annual, seasonal, weekly, or daily conditions (McMaster et al., 2018). Some relevant information, such as precipitation may also be interpolated (Alexander et al., 2020). Further, measurements of water quality may be done at more locations than where biological monitoring occurs (Glozier et al., 2018; McMaster et al., 2018) and may be automated and transmitted remotely (Fang et al., 2014). In these scenarios, data collection might also include the use of instruments at remote locations where deployment and retrieval are comparatively less expensive and time-consuming than more frequent visits (Zwart et al., 2014).

While simultaneously establishing multiple monitoring projects at the same locations may economize logistical effort and cost, resulting issues of sampling design, particularly with respect to spatial scales, may negate these efficiencies. Integrating these datasets post hoc may be onerous and burdensome (Burton et al., 2015). For example, collecting and analyzing all data available made available through remote cameras or automated recording units also increases the risks of the Sharpshooter Fallacy (Arciszewski et al., 2017b). There may also be temptation to collect data which could or might be useful and strain funding envelopes. Furthermore, not all measurements are possible with field probes, such as total alkalinity of a water sample.

While there has been more recent effort expended by the governing committees within OSM to integrate the programs, historically the individual project leads were responsible for integrating their work. A key emerging challenge to developing an integrated program in the OSR is communication across the organizational structure. OSM functions under a semi-distributed model with geographically dispersed groups of individuals from the Provincial and Federal governments, local community members and representatives, private contractors, and non-governmental organizations. Difficulty of communicating a clear direction on what constitutes integrated monitoring and how it can be achieved, unclear roles and responsibilities, or a lack familiarity among participants on how to effectively operate within an integrated program, a lack of investment, travel restrictions, or all of the above can influence the development of an integrated program. The semi-distributed model across multiple monitoring partners or subcontractors may also not automatically or effectively address any potential competing financial self-interests for the limited funding pool. More deliberate effort may be required by all participants to facilitate the development of an integrated design.

Administratively, all types of integrated monitoring are best served by a strategic planning framework (Liu et al., 2012). While an integrated design requires deep foresight (Culp et al., 2000), it can also be onerous and difficult to initially plan, especially in a multi-stakeholder environment (Munkittrick et al., 2019). Failing to plan properly prior to the initiation of monitoring may lead to increased management and operational costs and loss of institutional or sectoral support (Reynolds et al., 2011). The administrative challenges are also compounded at large spatial scales (Culp et al., 2000) and iterative planning cycles are likely necessary to eventually build toward a rigorous integrated design.

Monitoring programs, irrespective of their approach for integration, are sensitive to changes in political whim and shifting economic priorities (Cronmiller and Noble 2018b). There may also be differences among participants regarding the goals, operation, tools, and approaches of the monitoring (Parr et al., 2002; Liu et al., 2012; Nilssen et al., 2015; Cronmiller and Noble 2018b). Highly focused a priori designs examining esoteric measurements may be more sensitive to these changing priorities than generic a priori programs or a posteriori programs. Highly focused programs can also create inertia. Through inertia associated with previous monitoring efforts in a region there may also be attempts to salvage existing designs and a reluctance to adopt alternative approaches (Arciszewski et al., 2017b; Jones et al., 2017). Consequently, where a priori designs are narrowly defined to determine the extent of change in a single indicator, they are at greater risk to these forces compared to programs based on integrated interpretation and analyses.

Flexibility of studies may also be reduced when integration occurs during design, but strength of inference is reinforced and expanded for known questions (Table 2). The capacity to successfully address large questions requiring integrated data may lead to compromises and difficulty addressing more nuanced questions with a single design (Lindenmayer and Likens, 2010). This is especially salient in programs like OSM where researchers and scientists conduct massive annual sampling campaigns (Figure 1). A likely outcome of a broad design is the loss of specificity, but a gain in generality (e.g., Hughes et al., 1990). Technical challenges of integration are also compounded when studies examine many indicators in a large area; the number of potential linkages among studies grows multiplicitively (Figure 4). Adequately addressing the exponential linkages within a single integrated design can be daunting when the means do not match the ends.

FIGURE 4

FIGURE 4. The number of potential links between individual studies as the number of studies increases within an integrated program. Potential study links are calculated as Ns * (Ns - 1)/2 where Ns is the number of studies; If a program consists of 10 component studies, there are theoretically 45 possible linkages between the studies, each of which needs to be considered. If there are 15 studies, the number of potential linkages increases to 105.

Other challenges could affect a priori integration. Although designed perfectly, an integrated study may also not be executed to the same degree due to logistical or weather-related issues, equipment loss, failure, or vandalism, and other reasons (including regional health orders) leading to unintended gaps in data and ultimately incorporating some of the challenges of integrated analyses. Additionally, if integration is improperly applied during program or study design, results can hinder or constrain future scientific treatment of the program objectives. For example, prematurely integrating studies in early stages may unnecessarily compromise optimal sampling locations or conditions for a particular parameter by accommodating collection of accessory samples (Parr et al., 2002) or a study design optimized for one parameter may be applied to another (Emmerton et al., 2018). This may lead to over-powering analyses of some parameters while under-powering others (Arciszewski et al., 2018), or limit potential future analyses of the data and utility of conclusions (Ferretti and Chiarucci, 2003).

While planning is necessary, over-planning is also possible. Imposing a rigid (integrated) design, which is simple to administer, risks entrenchment (Arciszewski et al., 2017b) and alienation and disengagement of potential participants (Lindenmayer and Likens, 2010). There may also be a desire by some participants to integrate all existing long-term datasets, sites, and projects (Parr et al., 2002) which may or may not serve the goals of the program.

Benefits of Not Integrating

In addition to the difficulties associated with integrated monitoring, which may discourage researchers from participating in such programs, there are also conceptual and operational advantages of isolating some or many studies. Where the criteria to clearly indicate degradation or meaningful changes are absent, interpreting the results from monitoring systems and making regulatory decisions can be challenging (Munkittrick et al., 2019). Having separate data collections and/or analyses examining similar program components or questions can be beneficial for making stronger inference via multiple lines of independent evidence. Independent and overlapping study components can reveal consistent information if the system is sufficiently understood, and this “monitoring of the monitoring” can be a wise addition to any program (Chapman et al., 2002; Arciszewski et al., 2017b). Such approaches can reveal weak understanding, and while uncommon early in the design stages of regional monitoring programs, can foster long-term credibility. However, this approach also suggests all monitoring may be immature and impossible to perfectly design to anticipate all possible data needs in the future. However, some tools and approaches are better suited for future needs than others (Lindenmayer et al., 2015).

Overcoming Challenges to Achieve Integrated Environmental Monitoring

Regional monitoring programs conducted across large spatial scales, with multiple participants, would likely benefit from integration. Developing and operating an integrated monitoring program requires careful consideration of several general aspects, including the need for clear objectives, a strategic plan, and commitment to follow the intended (strategic) course (e.g., Dowdeswell et al., 2010). These general guidelines, however, will also be accompanied by tactical, technical, social, and administrative aspects specific to the scientists, stakeholders, and governors of the particular program in question requiring resolution to develop a suitable program.

The first aspects requiring resolution is when and what to integrate. Integration of a design early in monitoring is best used where clear technical questions, a common understanding of objectives (Fancy et al., 2008), and rigorous statistical design (Ferretti and Chiarucci, 2003) are defined. If these attributes are unknown, uncertain, questionable, or tenuous, a fully integrated design, though promoted by program managers, may not be possible or desirable and may be counter-productive (Vos et al., 2000). What type of integration to use also requires consideration. The various trade-offs involved in integrating monitoring (Table 2) affect the desired or most appropriate type of integration.

While when and what to integrate may influence the monitoring and management decisions, integration may also adapt to the evolving needs of the program or its technical sophistication. The type of integration desired or used within a given monitoring program may be affected by the desires of individual stakeholders and communities, changes in governance, or institutional inertia. Creating a hierarchy of monitoring and management decisions (Arciszewski et al., 2017b; Munkittrick and Arciszewski 2017) can be incorporated into an adaptive framework (Somers et al., 2018) to provide the needed flexibility, while retaining consistency and integration in key components and sites. Tiering of a program also provides rationale for some necessary changes in monitoring activities.

Updating the approaches to integration may naturally follow from a posteriori to a priori approaches as a program matures. As integration potentially moves from a posteriori forms to a priori, the needs will change and costs for administration, coordination, and data warehousing will typically increase. The focus of the questions (and the ability to adopt interdisciplinary questions) will also likely increase as can efficiency and ease of operation for individual participants and the program as a whole reflecting the natural progression towards routinized monitoring. While routinization of data collection and measurement techniques can be massively beneficial, its major disadvantage is that a highly focused and fully integrated design can also become entrenched, inflexible, and eventually irrelevant (Arciszewski et al., 2017b). As we have discussed already, there are strategies to mitigate these potential problems.

While an initial integrated design may emerge, forced periodic updates and additional measurements or approaches that can be integrated later in the study process, such as interpretation (Lindenmayer and Likens, 2010). Regular updates over a fixed period, such as five years may be a reasonable balance between a program which changes every year and a permanently static program (Cooke et al., 2018). The duration of the fixed update period may also be initially short when information is sparse and uncertain, but may alter as knowledge deepens and experience matures into wisdom. However, other administrative constraints, such as finding and hiring suitable candidates may also inform the appropriate planning cycle. Accounting for periodic updates in the planning and design of a program suggests that an optimal design may initially be based on integrated collection, which is also partially-integrated design. Similar to mathematical modeling (Timoshevskii et al., 2003), an algorithm to select optimal complexity of the monitoring program may be useful. Developing a model to determine the optimal complexity of an integrated monitoring program may be beneficial, as has been demonstrated in other aspects of monitoring (e.g., Wintle et al., 2010), but also requires its own effort and will likely be challenging in its own right.

Other factors will also affect the optimal integration for a given program, including the complexity, specificity, and number of the monitoring questions. Complex and specific questions may require complex and specific designs, but they may not necessarily be used to constrain the remainder of the program. Keeping some questions and studies separate from the larger design may improve the overall quality of the program, including OSM’s practice of dividing core and focused studies. In addition, limiting integration to a manageable unit, such as “surface water” is likely advantageous, but partitioning collections further into watersheds, sub-watersheds, or another meaningful spatial unit may also be necessary (Vos et al., 2000; Vaughan et al., 2009). These mixed designs may be a powerful compromise between the technical and administrative demands. For example, some degree of integrated interpretation may always be involved in study systems where a single governing body does not control all monitoring and research activities and as knowledge of a particular topic becomes more nuanced and the questions become more focused (i.e., Kelly et al., 2009; Zhang et al., 2016; Chibwe et al., 2019; Ahad et al., 2021).

Many potential solutions are related to organization and administration. The administration of the program defines how decisions are made, the deliverables and reporting schedules, how (and to whom) funding is allocated, and how and if changes can be made. The challenges of administration can, however sometimes match or exceed the technical challenges of monitoring. OSM underwent four major governance changes in its first eight years and has become an administratively integrated program, but retains some pre-existing initiatives and relies on both Governments, external independent contractors, Universities, and communities to deliver monitoring results (Figure 3). There are also requirements for the results to appear in the peer-reviewed literature. Given the distribution of partners among organization of the need to publish in the peer-reviewed literature, much of the results reported remain isolated to a given theme or to a single indicator suite, such as water chemistry (Figure 5). Recognizing and understanding the practical challenges limiting the integration of a program when it is a stated goal is the key to successfully enhancing the performance of the monitoring program to detect relevant change and provide robust answers to decision-makers. These attributes include the desired strength of inference, the sensitivity of the program to errors, the flexibility to adapt, an acceptable administrative burden, and how clearly understood the questions must be (Table 2).

FIGURE 5

FIGURE 5. Linkages between studies appearing in the peer-reviewed literature from 2009–2020; blue circles indicate a study reporting the results of a given oval; arrows from blue dots showing linkages, either explicit or implicit, within a study to another relevant result; shows few linkages of water programs to other theme areas within the OSM program; List of source papers included as Supplemental Information.

Planning is necessary to develop an optimal monitoring program (Reynolds et al., 2011). Planning for integration can be facilitated by conceptual effect models (Bunn et al., 2010) to focus attention. Conceptual models appear in many guises, including in Environmental Impact Assessments describing the expected interactions between human industrial developments and natural environments to guide decisions. While describing linkages between chemical, physical, and biological processes and components (Born and Sonzogni, 1995), conceptual models of environmental effects are analogous to systems-based monitoring (Fancy et al., 2008). Conceptual stressor-pathway-effect models, while likely incomplete (Munkittrick et al., 2000), are also useful tools for organizing monitoring, studies within monitoring programs, and the linkages between them (Bunn et al., 2010). Conceptual models are broadly applicable as tools of integration (e.g., Liu et al., 2012), which have been recommended by OSM (Swanson, 2019a; Swanson, 2019b) and have been used to explore the environmental issues affecting the oil sands region in Alberta, Canada, including seepage from tailings ponds (Fennell and Arciszewski, 2019). Conceptual models can also form the basis of empirical models. Planning of integrated monitoring may also require detailed design documents, but can also include later “plans to plan” when regional programs are in their infancy and after pilot programs are complete (Environment Canada, 2011a). While they may be useful, CMs can also be unwieldy (Suter, 2016).

Among the many solutions to improve organization and administration, better communication and collaboration among various participants is likely a disproportionately beneficial addition (Gibbons et al., 2008). Soft-skills, such as mutual understanding, open communication, incentives, power sharing, and mutual trust as important factors in the collaboration of policy-makers and researchers (Briggs, 2006). Other researchers also suggest fostering relationships, active and regular communications, contact mapping, secondments and joint appointments, and annual science-policy conferences will improve programs involving researchers and policy-makers (e.g., Gibbons et al., 2008) and researcher-to-researcher interactions (Cooke and Hilton, 2015). There, however, also needs to be recognition that monitoring decisions, like many others, are often based on incomplete information and the discretion of multiple participants including the stakeholders and the principal investigators. Hindsight will invariably reveal incorrect decisions, but these should be viewed as unavoidable and opportunities to improve the future monitoring, not as abject failures (Lindenmayer et al., 2015).

Conclusion

Integration is a natural feature of scientific practice and is very well understood conceptually when applied as interpretation within a single publication. However, as the spatial, temporal, and technical scope and demands of a monitoring program increases, so do the challenges of integration. In this paper, we explored the concept and practice of integration for environmental monitoring programs to facilitate understanding, including the benefits, limitations, challenges, and potential solutions.

The optimal form of integration depends on the goals of the program and its desired attributes, including the desired strength of inference. A fully integrated design may, or may not be desirable in many situations, but is the ideal choice when monitoring questions are clearly understood and technical approaches are widely accepted. In these cases, measurements of the environment will, by default, become part of a larger analytical framework and design providing strong inferences and answers to management questions.

Inconsistency in approaches and deliberate integration of multiple regional studies can be both beneficial in some respects, and harmful or limiting in others. Consequently, the choice of the optimal form of integration for a given program also depends on other factors. While the simplest to design and operate, integrated interpretation may not adequately address the questions a monitoring program is charged with answering. Where questions and approaches are not widely understood, practices of scientific disciplines are not aligned, or another source of uncertainty affects the monitoring, a partially integrated design may be most desirable and optimally complex to address questions. Given the discussion above, there is likely no single ideal way to integrate and decisions on how and when to do so must be considered carefully and with explicit purpose. The degree of integration may also change over time and designs need to anticipate and accommodate changing priorities, understanding, tools, and approaches. In summary, greater recognition of the diversity of practices and approaches possible to fulfill the general demands for an “integrated monitoring program” will benefit all involved.

Author Contributions

TA: Conceived work; wrote initial draft of manuscript; revised manuscript; DR: wrote initial draft; revised and edited subsequent drafts; KM: edited and revised draft manuscript; GS: edited and revised draft manuscript.

Funding

This work was funded through the Oil Sands Monitoring program.

Conflict of Interest

Authors TA and DR were employed by the Oil Sands Monitoring Program.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors thank Fred Wrona for his comments on an earlier draft of this manuscript. The authors also thank internal reviewers, members of the Science and Indigenous Knowledge Integration Committee, other the participants of the Oil Sands Monitoring Program, and the reviewers for their comments and contributions to this work. The work was indirectly funded by OSM but does not necessarily reflect an official position of OSM or of the authors’ institutions. TA and DR are salaried through the Oil Sands Monitoring program.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fenvs.2021.666698/full#supplementary-material

References

Addison, P. F. E., Cook, C. N., and de Bie, K. (2016). Conservation Practitioners' Perspectives on Decision Triggers for Evidence-Based Management. J. Appl. Ecol. 53, 1351–1357. doi:10.1111/1365-2664.12734

CrossRef Full Text | Google Scholar

AEMP (Alberta Environmental Monitoring Panel) (2011). A World Class Environ- Mental Monitoring, Evaluation and Reporting System for Alberta. Edmonton, Alta:Report of the Alberta Environmental Monitoring Panel.

Google Scholar

Alberta Biodiversity Monitoring Institute (ABMI). (2018). Oil Sands Regions Overview. Available at: https://www.abmi.ca/home/reports/2020/human-footprint/Oil-Sands-Regions (Accessed June 28, 2021).

Ahad, J. M. E., Pakdel, H., Labarre, T., Cooke, C. A., Gammon, P. R., and Savard, M. M. (2021). Isotopic Analyses Fingerprint Sources of Polycyclic Aromatic Compound-Bearing Dust in Athabasca Oil Sands Region Snowpack. Environ. Sci. Technol. 55 (9), 5887–5897. doi:10.1021/acs.est.0c08339

PubMed Abstract | CrossRef Full Text | Google Scholar

Alexander, A. C., and Chambers, P. A. (2016). Assessment of Seven Canadian Rivers in Relation to Stages in Oil Sands Industrial Development, 1972-2010. Environ. Rev. 24, 484–494. doi:10.1139/er-2016-0033

CrossRef Full Text | Google Scholar

Alexander, A. C., Levenstein, B., Sanderson, L. A., Blukacz-Richards, E. A., and Chambers, P. A. (2020). How Does Climate Variability Affect Water Quality Dynamics in Canada's Oil Sands Region? Sci. Total Environ. 732, 139062. doi:10.1016/j.scitotenv.2020.139062

PubMed Abstract | CrossRef Full Text | Google Scholar

Ankley, G. T., Bennett, R. S., Erickson, R. J., Hoff, D. J., Hornung, M. W., Johnson, R. D., et al. (2010). Adverse Outcome Pathways: a Conceptual Framework to Support Ecotoxicology Research and Risk Assessment. Environ. Toxicol. Chem. 29, 730–741. doi:10.1002/etc.34

PubMed Abstract | CrossRef Full Text | Google Scholar

Arciszewski, T. J., Hazewinkel, R. R., Munkittrick, K. R., and Kilgour, B. W. (2018). Developing and Applying Control Charts to Detect Changes in Water Chemistry Parameters Measured in the Athabasca River Near the Oil Sands: A Tool for Surveillance Monitoring. Environ. Toxicol. Chem. 37, 2296–2311. doi:10.1002/etc.4168

PubMed Abstract | CrossRef Full Text | Google Scholar

Arciszewski, T. J., and McMaster, M. E. (2021). Potential Influence of Sewage Phosphorus and Wet and Dry Deposition Detected in Fish Collected in the Athabasca River north of Fort McMurray. Environments 8, 14–21. doi:10.3390/environments8020014

CrossRef Full Text | Google Scholar

Arciszewski, T. J., and Munkittrick, K. R. (2015). Development of an Adaptive Monitoring Framework for Long-Term Programs: An Example Using Indicators of Fish Health. Integr. Environ. Assess. Manag. 11, 701–718. doi:10.1002/ieam.1636

PubMed Abstract | CrossRef Full Text | Google Scholar

Arciszewski, T. J., Munkittrick, K. R., Kilgour, B. W., Keith, H. M., Linehan, J. E., and McMaster, M. E. (2017a). Increased Size and Relative Abundance of Migratory Fishes Observed Near the Athabasca Oil Sands. Facets 2, 833–858. doi:10.1139/facets-2017-0028

CrossRef Full Text | Google Scholar

Arciszewski, T. J., Munkittrick, K. R., Scrimgeour, G. J., Dubé, M. G., Wrona, F. J., and Hazewinkel, R. R. (2017b). Using Adaptive Processes and Adverse Outcome Pathways to Develop Meaningful, Robust, and Actionable Environmental Monitoring Programs. Integr. Environ. Assess. Manag. 13, 877–891. doi:10.1002/ieam.1938

PubMed Abstract | CrossRef Full Text | Google Scholar

Bagstad, K. J., Semmens, D. J., Waage, S., and Winthrop, R. (2013). A Comparative Assessment of Decision-Support Tools for Ecosystem Services Quantification and Valuation. Ecosystem Serv. 5, 27–39. doi:10.1016/j.ecoser.2013.07.004

CrossRef Full Text | Google Scholar

Baird, D., Banic, C., Bickerton, G., Burn, D., Dillon, P., Droppo, I., et al. (2011). Lower Athabasca Water Quality Monitoring Program. PHASE 1□ Athabasca River Mainstem and Major Tributaries, Ottawa, Ontario, Canada: Energy Resour. Conserv. Board.90

Google Scholar

Barra, R. O., Cardenas-Soraca, D., Campos-Garagay, M., McMaster, M. E., and Hewitt, L. M. (2020). Integrated Approaches for Detecting the Occurrence and Effects of Endocrine Disrupting Substances in Surface Waters. Curr. Opin. Environ. Sci. Health 18, 20–25. doi:10.1016/j.coesh.2020.06.002

CrossRef Full Text | Google Scholar

Biber, E. (2011). The Problem of Environmental Monitoring. U. Colo. L. Rev. 83, 1. doi:10.1080/00221346908981147

Google Scholar

Born, S. M., and Sonzogni, W. C. (1995). Integrated Environmental Management: Strengthening the Conceptualization. Environ. Manage. 19, 167–181. doi:10.1007/BF02471988

CrossRef Full Text | Google Scholar

Botvinik-Nezer, R., Holzmeister, F., Camerer, C. F., Dreber, A., Huber, J., Johannesson, M., et al. (2020). Variability in the Analysis of a Single Neuroimaging Dataset by many Teams. Nature 582, 84–88. doi:10.1038/s41586-020-2314-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Bricker, O. P., and Ruggiero, M. A. (1998). Toward a National Program for Monitoring Environmental Resources. Ecol. Appl. 8, 326–329. doi:10.1890/1051-0761(1998)008[0326:tanpfm]2.0.co;2

CrossRef Full Text | Google Scholar

Briggs, S. V. (2006). Integrating Policy and Science in Natural Resources: Why So Difficult? Ecol. Manage. Restor 7, 37–39. doi:10.1111/j.1442-8903.2006.00245.x

CrossRef Full Text | Google Scholar

Brunet, N. D., Jardine, T. D., Jones, P. D., Macdermid, F., Reed, G., Bogdan, A.-M., et al. (2020). Towards Indigenous Community-Led Monitoring of Fish in the Oil Sands Region of Canada: Lessons at the Intersection of Cultural Consensus and Fish Science. Extractive Industries Soc. 7 (4), 1319–1329. doi:10.1016/j.exis.2020.06.014

CrossRef Full Text | Google Scholar

Bunn, S. E., Abal, E. G., Smith, M. J., Choy, S. C., Fellows, C. S., Harch, B. D., et al. (2010). Integration of Science and Monitoring of River Ecosystem Health to Guide Investments in Catchment protection and Rehabilitation. Freshw. Biol. 55, 223–240. doi:10.1111/j.1365-2427.2009.02375.x

CrossRef Full Text | Google Scholar

Burton, A. C., Neilson, E., Moreira, D., Ladle, A., Steenweg, R., Fisher, J. T., et al. (2015). REVIEW: Wildlife Camera Trapping: a Review and Recommendations for Linking Surveys to Ecological Processes. J. Appl. Ecol. 52, 675–685. doi:10.1111/1365-2664.12432

CrossRef Full Text | Google Scholar

Burton, G. A., Batley, G. E., Chapman, P. M., Forbes, V. E., Smith, E. P., Reynoldson, T., et al. (2002a). A Weight-Of-Evidence Framework for Assessing Sediment (Or Other) Contamination: Improving Certainty in the Decision-Making Process. Hum. Ecol. Risk Assess. Int. J. 8, 1675–1696. doi:10.1080/20028091056854

CrossRef Full Text | Google Scholar

Burton, G. A., Chapman, P. M., and Smith, E. P. (2002b). Weight-of-evidence Approaches for Assessing Ecosystem Impairment. Hum. Ecol. Risk Assess. Int. J. 8, 1657–1673. doi:10.1080/20028091057547

CrossRef Full Text | Google Scholar

Bustamante, M. M. C., Roitman, I., Aide, T. M., Alencar, A., Anderson, L. O., Aragão, L., et al. (2016). Toward an Integrated Monitoring Framework to Assess the Effects of Tropical forest Degradation and Recovery on Carbon Stocks and Biodiversity. Glob. Change Biol. 22, 92–109. doi:10.1111/gcb.13087

CrossRef Full Text | Google Scholar

Cardoso, D. N., Soares, A. M. V. M., Wrona, F. J., and Loureiro, S. (2020). Assessing the Acute and Chronic Toxicity of Exposure to Naturally Occurring Oil Sands Deposits to Aquatic Organisms Using Daphnia magna. Sci. Total Environ. 729, 138805. doi:10.1016/j.scitotenv.2020.138805

PubMed Abstract | CrossRef Full Text | Google Scholar

Cash, K. J., Culp, J. M., Dubé, M. G., Lowell, R. B., Glozier, N. E., and Brua, R. B. (2003). Integrating Mesocosm Experiments with Field and Laboratory Studies to Generate Weight-Of-Evidence Risk Assessments for Ecosystem Health. Aquat. Ecosystem Health Manage. 6, 177–183. doi:10.1080/14634980301464

CrossRef Full Text | Google Scholar

Chapman, P. M., McDonald, B. G., and Lawrence, G. S. (2002). Weight-of-Evidence Issues and Frameworks for Sediment Quality (And Other) Assessments. Hum. Ecol. Risk Assess. Int. J. 8, 1489–1515. doi:10.1080/20028091057457

CrossRef Full Text | Google Scholar

Chapman, P. M., Mcdonald, B. G., Lawrence, G. S., Chapman, P. M., Mcdonald, B. G., and Weight-, G. S. L. (2010, Weight-of-Evidence Issues and Frameworks for Sediment Quality (And Other) Assessments, Hum. Ecol. Risk Assess. 8, 1489, 1515. doi:10.1080/20028091057457

CrossRef Full Text | Google Scholar

Chibwe, L., Manzano, C. A., Muir, D., Atkinson, B., Kirk, J. L., Marvin, C. H., et al. (2019). Deposition and Source Identification of Nitrogen Heterocyclic Polycyclic Aromatic Compounds in Snow, Sediment, and Air Samples from the Athabasca Oil Sands Region. Environ. Sci. Technol. 53, 10–1021.

CrossRef Full Text | Google Scholar

Cooke, C., Droppo, I., di Cenzo, P., Glozier, N., Chambers, P., Conly, M., et al. (2018). Rationalizing And Optimizing The Water Quality Monitoring Network In the Oil Sands. Oil Sands Monitoring Program Technical Report Series No. 2. Available at: https://open.alberta.ca/publications/9781460140192.

Google Scholar

Cooke, N. J., and Hilton, M. L. (2015). Enhancing the Effectiveness of Team Science. Washington, DC: National Academies Press.

Cronmiller, J. G., and Noble, B. F. (2018b). Integrating Environmental Monitoring with Cumulative Effects Management and Decision Making. Integr. Environ. Assess. Manag. 14, 407–417. doi:10.1002/ieam.4034

PubMed Abstract | CrossRef Full Text | Google Scholar

Cronmiller, J. G., and Noble, B. F. (2018a). The Discontinuity of Environmental Effects Monitoring in the Lower Athabasca Region of Alberta, Canada: Institutional Challenges to Long-Term Monitoring and Cumulative Effects Management. Environ. Rev. 26, 169–180. doi:10.1139/er-2017-0083

CrossRef Full Text | Google Scholar

Cruz-Martinez, L., Fernie, K. J., Soos, C., Harner, T., Getachew, F., and Smits, J. E. G. (2015). Detoxification, Endocrine, and Immune Responses of Tree Swallow Nestlings Naturally Exposed to Air Contaminants from the Alberta Oil Sands. Sci. Total Environ. 502, 8–15. doi:10.1016/j.scitotenv.2014.09.008

PubMed Abstract | CrossRef Full Text | Google Scholar

Culp, J. M., Brua, R. B., Luiker, E., and Glozier, N. E. (2020). Ecological Causal Assessment of Benthic Condition in the Oil Sands Region, Athabasca River, Canada. Sci. Total Environ. 749, 141393. doi:10.1016/j.scitotenv.2020.141393

PubMed Abstract | CrossRef Full Text | Google Scholar

Culp, J. M., Cash, K. J., and Wrona, F. J. (2000). Integrated Assessment of Ecosystem Integrity of Large Northern Rivers: the Northern Rivers Basins Study Example. J. Aquat. Ecosyst. Stress Recover. 8, 1–5. doi:10.1023/A:1011486119870

CrossRef Full Text | Google Scholar

Culp, J. M., Droppo, I. G., Di Cenzo, P. D., Alexander-Trusiak, A., Baird, D. J., Bickerton, G., et al. (2018). Synthesis Report for the Water Component, Canada-Alberta Joint Oil Sands Monitoring: Key Findings and Recommendations. Government of Alberta Available at: https://open.alberta.ca/publications/9781460140253.

Google Scholar

Davidson, C. J., Foster, K. R., and Tanna, R. N. (2020). Forest Health Effects Due to Atmospheric Deposition: Findings from Long-Term forest Health Monitoring in the Athabasca Oil Sands Region. Sci. Total Environ. 699, 134277. doi:10.1016/j.scitotenv.2019.134277

PubMed Abstract | CrossRef Full Text | Google Scholar

Dowdeswell, L., Dillon, P., Ghoshal, S., Miall, A., Rasmussen, J., and Smol, J. P. (2010). A Foundation for the Future: Building an Environmental Monitoring System for the Oil Sands [Report of the federal advisory panel on oilsands monitoring].

Google Scholar

Dubé, M., Cash, K., Wrona, F., Enei, G., Cronmiller, J., Abel, R., et al. (2018). Oil Sands Monitoring Program Letter of Agreement and Operational Framework. Edmonton, AB: Alberta Environment and ParksAvailable at: https://open.alberta.ca/publications/9781460142363.

Dubé, M. G., Duinker, P., Greig, L., Carver, M., Servos, M., McMaster, M., et al. (2013). A Framework for Assessing Cumulative Effects in Watersheds: an Introduction to Canadian Case Studies. Integr. Environ. Assess. Manag. 9, 363–369. doi:10.1002/ieam.1418

PubMed Abstract | CrossRef Full Text | Google Scholar

Eccles, K. M., Pauli, B. D., and Chan, H. M. (2019). The Use of Geographic Information Systems for Spatial Ecological Risk Assessments: An Example from the Athabasca Oil Sands Area in Canada. Environ. Toxicol. Chem. 38, 2797–2810. doi:10.1002/etc.4577

PubMed Abstract | CrossRef Full Text | Google Scholar

Emmerton, C. A., Cooke, C. A., Wentworth, G. R., Graydon, J. A., Ryjkov, A., and Dastoor, A. (2018). Total Mercury and Methylmercury in Lake Water of Canada's Oil Sands Region. Environ. Sci. Technol. 52, 10946–10955. doi:10.1021/acs.est.8b01680

PubMed Abstract | CrossRef Full Text | Google Scholar

Environment Canada. 2011a. An Integrated Oil Sands Environment Monitoring Plan. Available from: http://publications.gc.ca/collections/collection_2011/ec/En14-47-2011-eng.pdf.

Google Scholar

Environment Canada 2011c. Integrated Monitoring Plan for the Oil Sands: Air Quality Component. Available from: http://publications.gc.ca/collections/collection_2011/ec/En14-45-2011-eng.pdf.

Google Scholar

Environment Canada 2011b. Integrated Monitoring Plan for the Oil Sands: Terrestrial Biodiversity Component. Available from: http://publications.gc.ca/collections/collection_2011/ec/En14-48-2011-eng.pdf.

Google Scholar

Fancy, S. G., Gross, J. E., and Carter, S. L. (2008). Monitoring the Condition of Natural Resources in US National parks. Environ. Monit. Assess. 151, 161–174. doi:10.1007/s10661-008-0257-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Fang, S., Da Xu, L., Zhu, Y., Ahati, J., Pei, H., Yan, J., et al. (2014). An Integrated System for Regional Environmental Monitoring and Management Based on Internet of Things. IEEE Trans. Ind. Inform. 10, 1596–1605. doi:10.1109/TII.2014.230263810.1109/tii.2013.2257807

CrossRef Full Text | Google Scholar

Fennell, J., and Arciszewski, T. J. (2019). Current Knowledge of Seepage from Oil Sands Tailings Ponds and its Environmental Influence in Northeastern Alberta. Sci. Total Environ. 686, 968–985. doi:10.1016/j.scitotenv.2019.05.407

PubMed Abstract | CrossRef Full Text | Google Scholar

Ferretti, M., and Chiarucci, A. (2003). Design Concepts Adopted in Long-Term forest Monitoring Programs in Europe-Problems for the Future? Sci. Total Environ. 310, 171–178. doi:10.1016/S0048-9697(02)00637-X

PubMed Abstract | CrossRef Full Text | Google Scholar

Fisher, J. T., and Burton, A. C. (2018). Wildlife Winners and Losers in an Oil Sands Landscape. Front. Ecol. Environ. 16, 323–328. doi:10.1002/fee.1807

CrossRef Full Text | Google Scholar

Gibbons, P., Zammit, C., Youngentob, K., Possingham, H. P., Lindenmayer, D. B., Bekessy, S., et al. (2008). Some Practical Suggestions for Improving Engagement between Researchers and Policy-Makers in Natural Resource Management. Ecol. Manag. Restor. 9, 182–186. doi:10.1111/j.1442-8903.2008.00416.x

CrossRef Full Text | Google Scholar

Gieswein, A., Hering, D., and Feld, C. K. (2017). Additive Effects Prevail: the Response of Biota to Multiple Stressors in an Intensively Monitored Watershed. Sci. Total Environ. 593-594, 27–35. doi:10.1016/j.scitotenv.2017.03.116

PubMed Abstract | CrossRef Full Text | Google Scholar

Glozier, N. E., Pippy, K., Levesque, L., Ritcey, A., Armstrong, B., Tobin, O., et al. (2018). Surface Water Quality Of the Athabasca, Peace And Slave Rivers And Riverine Waterbodies Within the Peace-Athabasca Delta. Oil Sands Monitoring Program Technical Report Series No. 1.4 Available at: https://open.alberta.ca/publications/9781460140284.

Google Scholar

Gosselin, P., Hrudey, S. E., Naeth, M. A., Plourde, A., Therrien, R., Van Der Kraak, G., et al. (2010). Environmental and Health Impacts of Canada’s Oil Sands Industry. Ottawa, Canada: R. Soc, 438.

Green, R., and Chapman, P. M. (2011). The Problem with Indices. Mar. Pollut. Bull. 62, 1377–1380. doi:10.1016/j.marpolbul.2011.02.016

PubMed Abstract | CrossRef Full Text | Google Scholar

Haeberli, W., Hoelzle, M., Paul, F., Zemp, M., and Zu, C. (2007). Integrated Monitoring of Mountain Glaciers as Key Indicators of Global Climate Change: the European Alps. Ann. Glaciol. 46, 150–160. doi:10.3189/172756407782871512

CrossRef Full Text | Google Scholar

Harner, T., Rauert, C., Muir, D., Schuster, J. K., Hsu, Y.-M., Zhang, L., et al. (2018). Air Synthesis Review: Polycyclic Aromatic Compounds in the Oil Sands Region. Environ. Rev. 26, 430–468. doi:10.1139/er-2018-0039

CrossRef Full Text | Google Scholar

Hicks, K., and Scrimgeour, G. (2019). A Study Design for Enhanced Environmental Monitoring of the Lower Athabasca River. Edmonton, Alberta: Office of the Chief Scientist, Ministry of Environment and ParksAvailable at: https://open.alberta.ca/publications/9781460145364.

Hopke, P. K., Alan, J., Johnson, D. H., Jana, K., Le, C., and Niemi, G. J. (2016). Assessing the Scientific Integrity of the Canada-Alberta Joint Oil Sands Monitoring (2012–2015): Expert Panel Review. Available at: https://open.alberta.ca/publications/assessing-the-scientific-integrity-of-the-canada-alberta-josm-2012-2015-expert-panel-review.

Google Scholar

Hughes, R. M., Whittier, T. R., Rohm, C. M., and Larsen, D. P. (1990). A Regional Framework for Establishing Recovery Criteria. Environ. Manage. 14, 673–683. doi:10.1007/bf02394717

CrossRef Full Text | Google Scholar

Hunsaker, C. T., Graham, R. L., Suter, G. W., O'Neill, R. V., Barnthouse, L. W., and Gardner, R. H. (1990). Assessing Ecological Risk on a Regional Scale. Environ. Manage. 14, 325–332. doi:10.1007/bf02394200

CrossRef Full Text | Google Scholar

Janousek, W. M., Hahn, B. A., and Dreitz, V. J. (2019). Disentangling Monitoring Programs: Design, Analysis, and Application Considerations. Ecol. Appl. 29, e01922. doi:10.1002/eap.1922

PubMed Abstract | CrossRef Full Text | Google Scholar

Jones, F. C., Plewes, R., Murison, L., MacDougall, M. J., Sinclair, S., Davies, C., et al. (2017). Random Forests as Cumulative Effects Models: A Case Study of Lakes and Rivers in Muskoka, Canada. J. Environ. Manage. 201, 407–424. doi:10.1016/j.jenvman.2017.06.011

PubMed Abstract | CrossRef Full Text | Google Scholar

Kelly, E. N., Schindler, D. W., Hodson, P. V., Short, J. W., Radmanovich, R., and Nielsen, C. C. (2010). Oil Sands Development Contributes Elements Toxic at Low Concentrations to the Athabasca River and its Tributaries. Proc. Natl. Acad. Sci. 107, 16178–16183. doi:10.1073/pnas.1008754107

PubMed Abstract | CrossRef Full Text | Google Scholar

Kelly, E. N., Short, J. W., Schindler, D. W., Hodson, P. V., Ma, M., Kwan, A. K., et al. (2009). Oil Sands Development Contributes Polycyclic Aromatic Compounds to the Athabasca River and its Tributaries. Pnas 106, 22346–22351. doi:10.1073/pnas.0912050106

PubMed Abstract | CrossRef Full Text | Google Scholar

Kilgour, B. W., Munkittrick, K. R., Hamilton, L., Proulx, C. L., Somers, K. M., Arciszewski, T., et al. (2019). Developing Triggers for Environmental Effects Monitoring Programs for Trout‐Perch in the Lower Athabasca River (Canada). Environ. Toxicol. Chem. 38, 1890–1901. doi:10.1002/etc.4469

PubMed Abstract | CrossRef Full Text | Google Scholar

Kirk, J. L., Muir, D. C. G., Gleason, A., Wang, X., Lawson, G., Frank, R. A., et al. (2014). Atmospheric Deposition of Mercury and Methylmercury to Landscapes and Waterbodies of the Athabasca Oil Sands Region. Environ. Sci. Technol. 48, 7374–7383. doi:10.1021/es500986r

PubMed Abstract | CrossRef Full Text | Google Scholar

Kurek, J., Kirk, J. L., Muir, D. C. G., Wang, X., Evans, M. S., and Smol, J. P. (2013). Legacy of a Half century of Athabasca Oil Sands Development Recorded by lake Ecosystems. Proc. Natl. Acad. Sci. 110, 1761–1766. doi:10.1073/pnas.1217675110

PubMed Abstract | CrossRef Full Text | Google Scholar

Landis, M. S., Studabaker, W. B., Patrick Pancras, J., Graney, J. R., Puckett, K., White, E. M., et al. (2019). Source Apportionment of an Epiphytic Lichen Biomonitor to Elucidate the Sources and Spatial Distribution of Polycyclic Aromatic Hydrocarbons in the Athabasca Oil Sands Region, Alberta, Canada. Sci. Total Environ. 654, 1241–1257. doi:10.1016/j.scitotenv.2018.11.131

PubMed Abstract | CrossRef Full Text | Google Scholar

Li, S.-M., Leithead, A., Moussa, S. G., Liggio, J., Moran, M. D., Wang, D., et al. (2017). Differences between Measured and Reported Volatile Organic Compound Emissions from Oil Sands Facilities in Alberta, Canada. Proc. Natl. Acad. Sci. USA 114, E3756–E3765. doi:10.1073/pnas.1617862114

PubMed Abstract | CrossRef Full Text | Google Scholar

Liggio, J., Li, S.-M., Hayden, K., Taha, Y. M., Stroud, C., Darlington, A., et al. (2016). Oil Sands Operations as a Large Source of Secondary Organic Aerosols. Nature 534, 91–94. doi:10.1038/nature17646

PubMed Abstract | CrossRef Full Text | Google Scholar

Lindenmayer, D. B., Burns, E. L., Tennant, P., Dickman, C. R., Green, P. T., Keith, D. A., et al. (2015). Contemplating the Future: Acting Now on Long-Term Monitoring to Answer 2050's Questions. Austral Ecol. 40, 213–224. doi:10.1111/aec.12207

CrossRef Full Text | Google Scholar

Lindenmayer, D. B., and Likens, G. E. (2010). Improving Ecological Monitoring. Trends Ecol. Evol. 25, 200–201. doi:10.1016/j.tree.2009.11.006

CrossRef Full Text | Google Scholar

Linkov, I., Loney, D., Cormier, S., Satterstrom, F. K., and Bridges, T. (2009). Weight-of-evidence Evaluation in Environmental Assessment: Review of Qualitative and Quantitative Approaches. Sci. Total Environ. 407, 5199–5205. doi:10.1016/j.scitotenv.2009.05.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Lipsey, Mark. W., and Wilson, D. B. (2001). Practical Meta-Analysis. Thousand Oaks, California: Sage Publications, Inc. doi:10.3386/w8084

CrossRef Full Text

Liu, H.-Y., Bartonova, A., Pascal, M., Smolders, R., Skjetne, E., and Dusinska, M. (2012). Approaches to Integrated Monitoring for Environmental Health Impact Assessment. Environ. Health 11. doi:10.1186/1476-069X-11-88

CrossRef Full Text | Google Scholar

Löfgren, S., Aastrup, M., Bringmark, L., Hultberg, H., Lewin-Pihlblad, L., Lundin, L., et al. (2011). Recovery of Soil Water, Groundwater, and Streamwater from Acidification at the Swedish Integrated Monitoring Catchments. Ambio 40, 836–856. doi:10.1007/s13280-011-0207-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Lott, E., and Jones, R. (2010). Review of Four Major Environmental Effects Monitoring Programs in the Oil Sands Region.

Google Scholar

Lowell, R. B., Culp, J. M., and Dubé, M. G. (2000). A Weight-Of-Evidence Approach for Northern River Risk Assessment: Integrating the Effects of Multiple Stressors. Environ. Toxicol. Chem. 19, 1182–1190. doi:10.1002/etc.5620190452

CrossRef Full Text | Google Scholar

Mahon, C. L., Holloway, G. L., Bayne, E. M., and Toms, J. D. (2019). Additive and Interactive Cumulative Effects on Boreal Landbirds: Winners and Losers in a Multi‐stressor Landscape. Ecol. Appl. 29, e01895. doi:10.1002/eap.1895

PubMed Abstract | CrossRef Full Text | Google Scholar

Makar, P. A., Akingunola, A., Aherne, J., Cole, A. S., Aklilu, Y.-a., Zhang, J., et al. (2018). Estimates of Exceedances of Critical Loads for Acidifying Deposition in Alberta and Saskatchewan. Atmos. Chem. Phys. 18, 9897–9927. doi:10.5194/acp-18-9897-2018

CrossRef Full Text | Google Scholar

McMaster, M. E., Parrott, J. L., Bartlett, A. J., Gagne, F., Evans, M. S., Tetreatult, G. R., et al. (2018). Aquatic Ecosystem Health Assessment of the Athabasca River Mainstem and Tributaries Using Fish Health and Fish and Invertebrate Toxicological Testing. Oil Sands Monitoring Program Technical Report Series No. 1.8.doi:10.2495/eid180371

CrossRef Full Text | Google Scholar

Miall, A. D. (2013). Geoscience of Climate and Energy 10. The Alberta Oil Sands: Developing a New Regime of Environmental Management, 2010–2013, 40. Canada: Geosci, 174–181. doi:10.12789/geocanj.2013.40.015

CrossRef Full Text

Mundy, L. J., Bilodeau, J. C., Schock, D. M., Thomas, P. J., Blais, J. M., and Pauli, B. D. (2019). Using wood Frog (Lithobates Sylvaticus) Tadpoles and Semipermeable Membrane Devices to Monitor Polycyclic Aromatic Compounds in Boreal Wetlands in the Oil Sands Region of Northern Alberta, Canada. Chemosphere 214, 148–157. doi:10.1016/j.chemosphere.2018.09.034

PubMed Abstract | CrossRef Full Text | Google Scholar

Munkittrick, K. R., Arciszewski, T. J., and Gray, M. A. (2019). Principles and Challenges for Multi-Stakeholder Development of Focused, Tiered, and Triggered, Adaptive Monitoring Programs for Aquatic Environments. Diversity 11, 155. doi:10.3390/d11090155

CrossRef Full Text | Google Scholar

Munkittrick, K. R., and Arciszewski, T. J. (2017). Using normal ranges for interpreting results of monitoring and tiering to guide future work: A case study of increasing polycyclic aromatic compounds in lake sediments from the Cold Lake oil sands (Alberta, Canada) described in Korosi et al. (2016). Environ. Pollut. 231, 1215–1222. doi:10.1016/j.envpol.2017.07.070

PubMed Abstract | CrossRef Full Text | Google Scholar

Munkittrick, K. R., McMaster, M. E., Portt, C., Gibbons, W. N., Farwell, A., Ruemper, L., et al. (2000). “The Development of Cumulative Effects Assessment Tools Using Fish Populations,” in In Integrated Assessment Of Ecosystem Health. Editors K. M. Scow, G. E. Fogg, D. E. Hinton, and M. L. Johnson (Boca Raton, FL: Lewis Publishers), 149–174.

Google Scholar

Munn, R. E. (1988). The Design of Integrated Monitoring Systems to Provide Early Indications of Environmental/ecological Changes. Environ. Monit. Assess. 11, 203–217. doi:10.1007/BF00394670

PubMed Abstract | CrossRef Full Text | Google Scholar

Mupepele, A.-C., Walsh, J. C., Sutherland, W. J., and Dormann, C. F. (2015). An Evidence Assessment Tool for Ecosystem Services and Conservation Studies. Ecol. Appl. 26, 1295–1301. doi:10.1890/15-0595.1

CrossRef Full Text | Google Scholar

Nilssen, I., Ødegård, Ø., Sørensen, A. J., Johnsen, G., Moline, M. A., and Berge, J. (2015). Integrated Environmental Mapping and Monitoring, a Methodological Approach to Optimise Knowledge Gathering and Sampling Strategy. Mar. Pollut. Bull. 96, 374–383. doi:10.1016/j.marpolbul.2015.04.045

PubMed Abstract | CrossRef Full Text | Google Scholar

Noble, B. F. (2015). Introduction to Environmental Impact Assessment: A Guide to Principles and Practice. 3rd ed. Don Mills, ON, Canada: Oxford University Press.

Norton, S. B., Cormier, S. M., and Suter, G. W. (2014). Ecological Causal Assessment. Boca Raton, Florida: CRC Press. doi:10.1002/9781118445112.stat07669

CrossRef Full Text

Parr, T. W., Ferretti, M., Simpson, I. C., Forsius, M., and Kovács-Láng, E. (2002). Towards a Long-Term Integrated Monitoring Programme in Europe: Network Design in Theory and Practice. Environ. Monit. Assess. 78, 253–290. doi:10.1023/A:1019934919140

PubMed Abstract | CrossRef Full Text | Google Scholar

Paul, J. F., Kiddon, J. A., Strobel, C. J., Melzian, B. D., Latimer, J. S., Cobb, D. J., et al. (2000). Condition of the Mid-Atlantic Estuaries: Production of a State of the Environment Report. Environ. Monit. Assess. 63, 115–129. doi:10.1023/A:1006450627164

CrossRef Full Text | Google Scholar

Petitjean, Q., Jean, S., Côte, J., Larcher, T., Angelier, F., Ribout, C., et al. (2020). Direct and Indirect Effects of Multiple Environmental Stressors on Fish Health in Human-Altered Rivers. Sci. Total Environ. 742, 140657. doi:10.1016/j.scitotenv.2020.140657

PubMed Abstract | CrossRef Full Text | Google Scholar

Posthuma, L., Dyer, S. D., de Zwart, D., Kapo, K., Holmes, C. M., and Burton, G. A. (2016). Eco-epidemiology of Aquatic Ecosystems: Separating Chemicals from Multiple Stressors. Sci. Total Environ. 573, 1303–1319. doi:10.1016/j.scitotenv.2016.06.242

PubMed Abstract | CrossRef Full Text | Google Scholar

Redman, A. D., Parkerton, T. F., Butler, J. D., Letinski, D. J., Frank, R. A., Hewitt, L. M., et al. (2018). Application of the Target Lipid Model and Passive Samplers to Characterize the Toxicity of Bioavailable Organics in Oil Sands Process-Affected Water. Environ. Sci. Technol. 52, 8039–8049. doi:10.1021/acs.est.8b00614

PubMed Abstract | CrossRef Full Text | Google Scholar

Reynolds, J. H., Thompson, W. L., and Russell, B. (2011). Planning for success: Identifying Effective and Efficient Survey Designs for Monitoring. Biol. Conservation 144, 1278–1284. doi:10.1016/j.biocon.2010.12.002

CrossRef Full Text | Google Scholar

Sarkar, S., Ghosh, K., Ghosh, K., and Petter, S. (2020). Using Secondary Data to Tell a New story: A Cautionary Tale in Health Information Technology Research. Cais 47, 95–112. doi:10.17705/1CAIS.04705

CrossRef Full Text | Google Scholar

Schwalb, A. N., Alexander, A. C., Paul, A. J., Cottenie, K., and Rasmussen, J. B. (2015). Changes in Migratory Fish Communities and Their Health, Hydrology, and Water Chemistry in Rivers of the Athabasca Oil Sands Region: a Review of Historical and Current Data. Environ. Rev. 23, 133–150. doi:10.1139/er-2014-0065

CrossRef Full Text | Google Scholar

Scrimgeour, G., Jones, N., and Tonn, W. M. (2013). Benthic Macroinvertebrate Response to Habitat Restoration in a Constructed Arctic Stream. River Res. Applic. 29, 352–365. doi:10.1002/rra.1602

CrossRef Full Text | Google Scholar

Shotyk, W., Appleby, P. G., Bicalho, B., Davies, L. J., Froese, D., Grant-Weaver, I., et al. (2017). Peat Bogs Document Decades of Declining Atmospheric Contamination by Trace Metals in the Athabasca Bituminous Sands Region. Environ. Sci. Technol. 51, 6237–6249. doi:10.1021/acs.est.6b04909

PubMed Abstract | CrossRef Full Text | Google Scholar

Soares, J., Makar, P., Aklilu, Y., and Akingunola, A. (2018). Hierarchical Clustering Network Analysis of Ambient Air Monitoring in Alberta: Phases 1 and 2. Oil Sands Monitoring Program. Technical Report Series No. 5.0.

Google Scholar

Somers, K. M., Kilgour, B. W., Munkittrick, K. R., and Arciszewski, T. J. (2018). An Adaptive Environmental Effects Monitoring Framework for Assessing the Influences of Liquid Effluents on Benthos, Water, and Sediments in Aquatic Receiving Environments. Integr. Environ. Assess. Manag. 14, 552–566. doi:10.1002/ieam.4060

PubMed Abstract | CrossRef Full Text | Google Scholar

Soulé, M. E. (1985). What Is Conservation Biology? Bioscience 35, 727–734.

CrossRef Full Text | Google Scholar

Sterling, S. M., Garroway, K., Guan, Y., Ambrose, S. M., Horne, P., and Kennedy, G. W. (2014). A New Watershed Assessment Framework for Nova Scotia: A High-Level, Integrated Approach for Regions without a Dense Network of Monitoring Stations. J. Hydrol. 519, 2596–2612. doi:10.1016/j.jhydrol.2014.07.063

CrossRef Full Text | Google Scholar

Summers, J. C., Kurek, J., Kirk, J. L., Muir, D. C. G., Wang, X., Wiklund, J. A., et al. (2016). Recent Warming, rather Than Industrial Emissions of Bioavailable Nutrients, Is the Dominant Driver of Lake Primary Production Shifts across the Athabasca Oil Sands Region. PLoS One 11, 1–20. doi:10.5061/dryad.k37q710.1371/journal.pone.0153987

CrossRef Full Text | Google Scholar

Suter, G. W. (2016). Ecological Risk Assessment. London, New York: CRC Press.

Swanson, S. (2019a). Oil Sands Monitoring Program: Integration Workshop Reports (Part 1 of 2). Available at: https://open.alberta.ca/publications/9781460144947.

Google Scholar

Swanson, S. (2019b). Oil Sands Monitoring Program: Recommendation Report (Part 2 of 2). Available at: https://open.alberta.ca/publications/9781460144954.

Google Scholar

Tarassenko, L., Hann, A., Young, D., Hospital, J. R., Way, H., and Ox, O. (2006). Integrated Monitoring and Analysis for Early Warning of Patient Deterioration. Br. J. Anaesth. 97, 64–68. doi:10.1093/bja/ael113

PubMed Abstract | CrossRef Full Text | Google Scholar

Teatini, P., Tosi, L., Strozzi, T., Carbognin, L., Wegmüller, U., and Rizzetto, F. (2005). Mapping Regional Land Displacements in the Venice Coastland by an Integrated Monitoring System. Remote Sensing Environ. 98, 403–413. doi:10.1016/j.rse.2005.08.002

CrossRef Full Text | Google Scholar

Tetreault, G. R., Bennett, C. J., Clark, T. W., Keith, H., Parrott, J. L., and McMaster, M. E. (2020). Fish Performance Indicators Adjacent to Oil Sands Activity: Response in Performance Indicators of Slimy Sculpin in the Steepbank River, Alberta, Adjacent to Oil Sands Mining Activity. Environ. Toxicol. Chem. 39, 396–409. doi:10.1002/etc.4625

PubMed Abstract | CrossRef Full Text | Google Scholar

Thackeray, S. J., and Hampton, S. E. (2020). The Case for Research Integration, from Genomics to Remote Sensing, to Understand Biodiversity Change and Functional Dynamics in the World's Lakes. Glob. Change Biol. 26, 3230–3240. doi:10.1111/gcb.15045

PubMed Abstract | CrossRef Full Text | Google Scholar

Timoshevskii, A., Yeremin, V., and Kalkuta, S. (2003). New Method for Ecological Monitoring Based on the Method of Self-Organising Mathematical Models. Ecol. Model. 162, 1–13. doi:10.1016/S0304-3800(02)00403-9

CrossRef Full Text | Google Scholar

Vaughan, I. P., Diamond, M., Gurnell, A. M., Hall, K. A., Jenkins, A., Milner, N. J., et al. (2009). Integrating Ecology with Hydromorphology: a Priority for River Science and Management. Aquat. Conserv: Mar. Freshw. Ecosyst. 19, 113–125. doi:10.1002/aqc10.1002/aqc.895

CrossRef Full Text | Google Scholar

Vethaak, A. D., Davies, I. M., Thain, J. E., Gubbins, M. J., Martínez-Gómez, C., Robinson, C. D., et al. (2017). Integrated Indicator Framework and Methodology for Monitoring and Assessment of Hazardous Substances and Their Effects in the marine Environment. Mar. Environ. Res. 124, 11–20. doi:10.1016/j.marenvres.2015.09.010

PubMed Abstract | CrossRef Full Text | Google Scholar

Vos, P., Meelis, E., and Ter Keurs, W. J. (2000). A Framework for the Design of Ecological Monitoring Programs as a Tool for Environmental and Nature Management. Environ. Monit. Assess. 61, 317–344. doi:10.1023/a:1006139412372

CrossRef Full Text | Google Scholar

Wasiuta, V., Kirk, J. L., Chambers, P. A., Alexander, A. C., Wyatt, F. R., Rooney, R. C., et al. (2019). Accumulating Mercury and Methylmercury Burdens in Watersheds Impacted by Oil Sands Pollution. Environ. Sci. Technol. 53, 12856–12864. doi:10.1021/acs.est.9b02373

PubMed Abstract | CrossRef Full Text | Google Scholar

Wentworth, G. R., and Zhang, L. (2018). Summary , Evaluation and Integration of Atmospheric Deposition Monitoring in the Athabasca Oil Sands Region. Oil Sands Monitoring Program Technical Report Series No. 3.0.

Google Scholar

Wintle, B. A., Runge, M. C., and Bekessy, S. A. (2010). Allocating Monitoring Effort in the Face of Unknown Unknowns. Ecol. Lett. 13, 1325–1337. doi:10.1111/j.1461-0248.2010.01514.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Wolska, L., Sagajdakow, A., Kuczyńska, A., and Namieśnik, J. (2007). Application of Ecotoxicological Studies in Integrated Environmental Monitoring: Possibilities and Problems. Trac Trends Anal. Chem. 26, 332–344. doi:10.1016/j.trac.2006.11.012

CrossRef Full Text | Google Scholar

Wrona, F., di Cenzo, P., and Schaefer, K. (2011). Integrated Monitoring Plan for the Oil Sands: Expanded Geographic Extent for Water Quality and Quantity, Aquatic Biodiversity and Effects, and Acid Sensitive lake Component. Ottawa - Ontario, Environ. Canada, 93.

Wrona, F. J., Carey, J., Brownlee, B., and McCauley, E. (2000). Contaminant Sources, Distribution and Fate in the Athabasca, Peace and Slave River Basins, Canada. J. Aquat. Ecosyst. Stress Recover. 8, 39–51. doi:10.1023/A:1011487605757

CrossRef Full Text | Google Scholar

Wrona, F. J., and Cash, K. J. (1996). The Ecosystem Approach to Environmental Assessment: Moving from Theory to Practice. J. Aquat. Ecosystem Health 5, 89–97. doi:10.1007/bf00662797

CrossRef Full Text | Google Scholar

Yoccoz, N. G., Nichols, J. D., and Boulinier, T. (2001). Monitoring of Biological Diversity in Space and Time. Trends Ecol. Evol. 16, 446–453. doi:10.1016/S0169-5347(01)02205-4

CrossRef Full Text | Google Scholar

Zhang, Y., Shotyk, W., Zaccone, C., Noernberg, T., Pelletier, R., Bicalho, B., et al. (2016). Airborne Petcoke Dust Is a Major Source of Polycyclic Aromatic Hydrocarbons in the Athabasca Oil Sands Region. Environ. Sci. Technol. 50, 1711–1720. doi:10.1021/acs.est.5b05092

PubMed Abstract | CrossRef Full Text | Google Scholar

Zwart, M. C., Baker, A., McGowan, P. J. K., and Whittingham, M. J. (2014). The Use of Automated Bioacoustic Recorders to Replace Human Wildlife Surveys: an Example Using Nightjars. PLoS One 9. doi:10.1371/journal.pone.0102770

CrossRef Full Text | Google Scholar