A Multi-Center Structural Equation Modeling Approach to Investigate Interpersonal Violence Screening for Public Health Promotion

Background: Interpersonal violence is a significant public health issue. Routine health screening is a cost-effective strategy that may reduce harmful physical and mental consequences. However, existing research finds consistently low rates of violence screening offered by healthcare providers, e.g., nurses, nurse practitioners, physicians. There is a critical need for research that helps understand how providers' screening behaviors are impacted by individual-level and organizational-level factors to promote the uptake of routine screening for interpersonal violence. Two recent studies, i.e., The Health Care Providers study and Nurse Practitioners Violence Screening study, involved quantitative data collected to measure providers' screening behavior and multi-level factors impacting violence screening.

Methods: The current analysis includes a combination of multi-center data collected from The Health Care Providers and Nurse Practitioners Violence Screening studies, respectively. The total sample is 389 providers across the United States. The proposed research develops a system-level multi-center structural equation model framework to rigorously integrate data from the two studies and examine providers' screening behavior for interpersonal violence based upon Theory of Planned Behavior from a quantitative perspective.

Results & Conclusions: We successfully examine the efficacy of the Theory of Planned Behavior proposed by Ajzen to predict healthcare providers' screening behavior for interpersonal violence. Organizational factors, e.g., availability of policy for interpersonal violence screening, organizational priority given to violence screening relative to other priorities, and if providers within the health center are interested in improving care quality, were significantly associated with providers' screening behavior. The knowledge and insights generated from our study may facilitate the design and optimization of health professional training and practice environment, and lead to improved women's health and quality of care.

Introduction

Healthcare screenings are vitally important and cost-effective prevention strategies to promote health for people of all ages. Preventative health screenings can identify disease or other health problems at an early stage so proactive treatments can be offered to lower the risk for the disease, inhibit its progression, or reduce other harmful health consequences (1, 2). There are various types of health screening tests available in today's healthcare systems. For instance, the American Diabetes Association recommends all adults be screened for diabetes starting at age 45 by evaluating the blood sugar levels through blood tests (3). Screening that uses mammography shows a great success in reducing breast cancer mortality rates among female patients (4). Osteoporosis is a disease that reduces the density and quality of bone and the United States Preventive Services Taskforce recommends screening for osteoporosis with bone measurement testing such as a dual energy X-ray absorptiometry scan to prevent osteoporotic fractures in women 65 years and older (5).

Among a variety of diseases and health issues, interpersonal violence, which includes intimate partner violence (IPV) and sexual violence (SV), has been increasingly recognized as a major public health problem. Intimate partner violence is defined as violence or aggression that occurs in a close relationship with intimate partners, who include current or former spouses and dating partners (6). It is estimated that 1 in 4 women will experience sexual violence, physical violence, and/or stalking by an intimate partner during their lifetime (6). Additionally, 20 percent of women are sexually assaulted in their lifetime and 1.3 to 5 million women experience interpersonal violence each year (6). According to a report from the World Health Organization, interpersonal violence results in acute and long-term effects on women's health, including injuries and death (7). Due to the prevalence and substantial health effects, screening for interpersonal violence has received a significant amount of attention as a secondary prevention strategy that may reduce the harmful consequences of violence and/or prevent the reoccurrence of further acts of violence (8). Different from prior examples of health screenings, i.e., diabetes screening, breast cancer screening, and osteoporotic screening, violence screening is primarily considered as a “patient care behavior” carried out by healthcare providers, such as nurses, nurse practitioners, and physicians, and is not dependent on timely and costly medical tests or examinations. Therefore, violence screening is a cost-effective prevention strategy for women's health promotion. However, existing studies report low rates of screening for interpersonal violence across various healthcare settings (9–11). Barriers to violence screening previously identified in literature include provider level factors (e.g., knowledge of interpersonal violence, provider comfort, self-efficacy) and clinic-level factors (e.g., protocols for screening) (12).

The prevalence and consequences of interpersonal violence make it imperative to further investigate healthcare providers' violence screening behaviors and identify potential factors that promote the uptake of routine screening. Findings may lead to improved women's health and quality of life. Studies have examined the potential influential factors associated with providers' interpersonal screening behavior (13–18), but many of these studies were qualitative. A few quantitative studies were completed but were limited by small and relatively homogeneous samples. Therefore, there is a need for large, more comprehensive data sets that involve larger samples of health care providers and their screening behaviors. Recognizing this need, the authors integrated two datasets from two separate studies related to healthcare providers' violence screening behaviors. The data were collected through the Health Care Providers (HCP) study (19–21) and the Nurse Practitioners Violence Screening (NPVS) study (22, 23). The scope of the HCP study was to study the violence screening behavior of providers at college health centers in a sample of 170 participants. The NPVS focused on healthcare providers in primary care with a final sample of 219 participants.

Both HCP and NPVS datasets were designed following the constructs of Theory of Planned Behavior (TPB) (24). TPB is a validated social psychological model to study human behavior and has been successfully applied to predict a wide range of human behaviors and intentions in various fields such as healthcare, public relations, and management (25–28). According to TPB, healthcare providers' screening behaviors should be determined by their behavioral intention to screen, which is further influenced by three constructs, i.e., the attitude toward behavior for violence screening, subjective norm of screening, and perceived behavioral control over screening. Following the Guidelines to Construct Questionnaires based on TPB (29), each construct can be further measured by a collection of questions carefully designed by researchers. Therefore, the two TPB-based survey datasets provide an opportunity to examine the relationship between providers' screening behavior and other TPB constructs. Furthermore, such complex relations among multiple constructs can be quantitively analyzed by structural equation modeling (SEM), which is a well-established statistical approach (30, 31).

Motivated by the recently available HCP and NPVS data, this paper proposes a cross-dataset multi-group SEM-based approach to integrate both datasets and investigate healthcare providers' violence screening behavior based on TPB. From the perspective of healthcare research, the inclusion of the wider population of participants in the study enables a more comprehensive understanding of providers' screening behaviors in various healthcare settings. From the perspective of data science research, combining multiple datasets could overcome challenges in data shortage and limited samples and increase the power of statistical models. There have been quite a few studies that focus on integrating/combining multiple datasets for a joint analysis (32–38). However, a number of challenges are posed when integrating multiple datasets for a comprehensive study. We take the integration of HCP and NPVS datasets as an example. First, simply pooling all observations from HCP and NPVS studies into a combined dataset is likely to increase the heterogeneity among participants, which results from potential differences in participant selection criteria adopted by different research teams. Secondly, because multi-center datasets are typically collected by different research teams at different time periods, it is common for each team to adopt their own measurements for the same variable of interests; another challenge when integrating multi-center datasets. A detailed discussion on the variable discrepany between HCP and NPVS studies can be found in sections Data Description and Building TPB Constructs by Integrating Variables from Two Datasets. An important consideration when combining multiple datasets is how to identify common data elements shared by all the datasets and integrate the common variables into the combined dataset.

The objective of this analysis is to examine healthcare providers' screening behaviors and related attitudes, beliefs, and practice environment to optimize providers' education, training, and workplace environment. A widely used and validated behavioral model, the Theory of Planned Behavior (24), was the theoretical model used to examine providers' screening behaviors and their attitudes, beliefs and practice environment characteristics from a qualitative perspective, while structural equation modeling was used to model and analyze the complex relationships specified by TPB from a quantitative perspective. The proposed study is a secondary analysis of two real-world datasets collected in HCP and NPVS studies. To address the two aforementioned challenges in multi-center data fusion, we propose a multi-group SEM that: (1) fits a separate model for each of two groups and offers the flexibility to enable joint estimation by imposing constraints to selected parameters between the two groups, instead of fitting a single SEM for the combined dataset; and (2) allows the same TPB constructs to be measured by similar but not identical items for each group of data to mitigate the potential discrepancy in cross-study variables. More details can be found in sections Building TPB Constructs by Integrating Variables from Two Datasets and A Multi-group Structural Equation Model (multi-group SEM). The proposed method contributes on/to both nursing research and data science research. From the data analysis perspective, combining multiple datasets could overcome challenges in data shortage and limited samples and increase the power of statistical models. For nursing research, a wider population of participants could be more representative of healthcare providers in general, and therefore help understand and examine the violence screening behavior of providers who work at various healthcare settings. The findings of this research can facilitate the design and optimization of education and training of healthcare providers to promote the uptake of routine screening for interpersonal violence, and may lead to improved patient outcomes and women's health.

Methods

Data Description

This study integrates two datasets collected from the Health Care Providers (HCP) study (19–21) and the Nurse Practitioners Violence Screening (NPVS) study (22, 23). Both studies were reviewed and approved by Boston College with the IRB Protocol Number 16.042.01 (for HCP) and IRB Protocol Number 18.067.01e (for NPVS). Online consent was obtained for both studies. The principal investigators of both studies designed the TPB-based survey questions and administered the questionnaire to healthcare providers across multiple healthcare centers in the U.S. In the questionnaire, violence screening behavior was measured by the screening percentage reported by each individual healthcare provider. Intention to perform screening was also measured. Providers' attitudes toward screening, subjective norm, and perceived behavior control were each measured by a collection of questions designed by two research groups based on literature review and investigators' clinical experience. The sample for this analysis includes 219 participants from the HCP study and 170 participants from the NPVS study, a total of 389 healthcare providers. Due to the limited space, a more detailed discussion on data collection procedure is skipped in this article and can be found in papers published by the HCP (19–21) and NPVS (22, 23) researchers.

Combining the datasets gives us the opportunity to perform this secondary data analysis, investigate violence screening behavior using the TPB on a relatively heterogeneous population, and enhances the power of the statistical analysis. However, integrating multi-center datasets typically poses a challenge in collecting common data elements across different datasets and studies. In the following section, more details are discussed in regard to identifying common variables/survey questions shared by both datasets and leveraging similar but not identical variables (questions) in this cross-dataset analysis.

The aim of the analysis was to use the data from two independent datasets to examine healthcare providers' interpersonal violence screening behavior from the perspective of TPB, and therefore only questions that describe providers' screening behavior, intention, and the other three TPB constructs, which include their attitudes toward screening, subjective norm, and perceived behavioral control, are included in the combined datasets. A detailed comparison of the list of questions included from both HCP and NPVS datasets is shown in Table 1, of which two components are treated as observed and the other three are latent and need to be measured by a few items. Specifically, the two observed components are “screening behavior” and “screening intention,” which are defined as “ the percentage of female patients (college students) you screen for IPV/SV” and “how likely you would screen all female patients (college students) for IPV/SV,” respectively, while “attitude toward behavior,” “subjective norm,” and “perceived behavioral control” are latent variables. Note that all items used to measure latent variables are ordinal. For instance, one of the items to measure the “attitude toward behavior” is “In your opinion, is it a bad or good idea to routinely screen for IPV with every female patient?” with options that include “very bad idea,” “bad idea,” “neutral,” “good idea,” and “very good idea,” which results in a five-point Likert score.

TABLE 1

Table 1. Summary of HCP and NPVS studies' questions included in TPB analyses.

Building TPB Constructs by Integrating Variables From Two Datasets

The Theory of Planned Behavior is a social psychological model of human behavior developed by Ajzen (24). As aforementioned, TPB predicts providers' screening behavior by their intention to screen, while their intentions are associated with three constructs, i.e., their attitudes toward screening, subjective norm, and perceived behavioral control toward screening. Table 1 shows how each construct in TPB is measured by a collection of questions. Besides the TPB constructs, items focused on organizational factors were collected as part of the study, that could potentailly impact the healthcare provider's behavior as well.

Two major differences between the dataset (HCP and NPVS) should be noted. The HCP study focused on investigating interpersonal violence screening for college women, and therefore recruited healthcare providers from college health centers. In the NPVS study, all participates worked in primary care and were recruited through the mailing list of American Association of Nurse Practitioners (https://www.aanp.org). Therefore, a difference between the two sets of survey questions is that NPVS focused violence screening for all female patients while HCP focused on screening behavior for college female students. Another difference was that the recall of behavior and intention were different for HCP and NPVS studies. For example, HCP dataset asked about screening percentages for the past 2 months, while NPVS dataset measured “screening behavior” that used providers' self-reported screening percentages for the past 3 months. The combination of two datasets creates a relatively heterogeneous population of healthcare providers with more subjects for the investigation of interpersonal violence screening behavior under a variety of clinical care settings.

To provide a composite measure for each construct, Guidelines on Constructing TPB Questionnaire (29) has different options for questions' design. Depending on which set of questions has been chosen to be measured for the construct, the guideline could make a specific recommendation to produce an effective composite measure from these questions. Therefore, the flexibility of the guidelines in producing composite scores allows us to first identify a set of common questions shared by both datasets and then choose the measurement that is most suitable to produce/compute a composite score based on the common questions set we identify from HCP and NPVS datasets. More details of each of the TPB constructs along with a list of organizational factors are described as follows:

Screening Behavior and Intention

Violence screening behavior was measured by providers' self-reported screening percentage on a scale from 0 to 100%. The screening intention was evaluated by asking providers “how likely is it that you would routinely screen all female patients/students for IPV and SV,” rated on a five-point Likert scale from “Very Unlikely” to “Very Likely.” It is noteworthy that the HCP study used a 2-month recall of screening behavior and intention and NPSV used a 3-month recall.

Attitude Toward Behavior

As shown in Table 1, attitude toward screening behavior was directly measured by two questions, i.e., “in your opinion, is it a bad or good idea to screen for IPV with every female patient/student who visits your healthcare setting?” and “In your opinion, is it a bad or good idea to screen for SV with every female patient/student who visits your healthcare setting?” According to the guidelines (29), the two items were considered as direct measurements of screening attitudes to evaluate the overall attitude toward screening.

Subjective Norm

According to Guidelines on Constructing TPB Questionnaire (29), both HCP and NPVS adopt indirect measurement questions to evaluate the subjective norm. That is, the subjective norm is evaluated indirectly by a belief-based measurement, which is obtained by computing the product between “normative beliefs” and “motivation to comply.” In Table 1, “normative beliefs” are measured by the questions such as “would any individual or group disapprove or approve if you routinely screened female students/patients for IPV/SV in your healthcare center?,” while the “motivation to comply” is measured by the corresponding question of “How important is it to you that this individual or group in the healthcare center approve of what you are doing?” For instance, the NPVS study collected providers' normative beliefs and motivation to comply with the beliefs about nurses, Nurse Practitioners, physicians, and health center directors, which are listed as questions (a.2), (b.2), (c.2) and (d.2) in Table 1, while HCP grouped the four roles into two categories that consist of colleagues who work with you and your health center directors. This resulted in two questions (a.1) and (b.1) under normative beliefs. By combining the questions for “normative beliefs” and “motivation to comply,” the overall subjective norm for HCP subjects is measured using two items, i.e., (a.1) × (c.1) and (b.1) × (d.1), while the subjective norm for NPVS subjects is measured using four items, i.e., (a.2) × (e.2) , (b.2) × (f.2) , (c.2) × (g.2), and (d.2) × (h.2).

Perceived Behavioral Control

Providers' perceived behavioral control toward screening was directly measured by evaluating how much the provider has control over interpersonal violence screening-related behavior. For example, “I am confident that I could screen for IPV and SV” is a question that measures how much control the provider has over screening. In Table 1 under “Perceived Behavioral Control,” there are four items that measure how confident the providers are at performing different screening-related assessments and their perceived behavioral control.

Organizational Factors

Recognizing that provider behavior is influenced by the organizational factors, both HCP and NPVS studies collected data on providers' perceptions of organizational characteristics in their practice settings [56]. This study included five organizational items: (1) Does your health center routinely screen all patients for IPV and SV? (2) How strongly do you agree that “IPV and SV screening is a top priority in this health center?” (3) How strongly do you agree that “There is a big push to screen all female patients for IPV and SV?” (4) How strongly do you agree that “People here put a lot of effort to ensure adherence to IPV and SV screening guidelines;” (5) How strongly do you agree that “Providers are very interested in improving care quality.” The five organizational items were self-reported by providers based on their individual perceptions of the health settings where they practiced.

A Multi-Group Structural Equation Model

SEM is a well-established statistical approach to represent, estimate, and test complex relations among variables, among which the relationship is usually specified according to domain knowledge (30, 31). As an extension of the traditional SEM, the mutli-group SEM examines structural relations in two or or more groups, in which the structural relations can be investigated across groups to determine whether underlying constructs differ or any particular items perform poorly in one group or another. Both SEM and multi-group SEM have been implemented in a number of statistical software. In particular, this study borrows strength from a build-in package “lavaan” (39) in an open-source statistical software R (40) for multi-group SEM esimtation. Aided by R, users only need to specifiy the relations among constructs for each group and the R software automatically provides the model estimation.

Next, we introduce the multi-group SEM following a five-step approach (41, 42) that consists of model identification, specification, estimation, testing, and re-specification. The model identification and specification for the multi-group SEM is based on the TPB theory, which specified a theoretical framework to describe the relationship between attitude toward behavior, subjective norm, and perceived behavioral control and their impact on screening intentions and behaviors. A graphical representation of the SEM specification can be found in Figure 1, in which Figure 1A presents the specification for HCP dataset and Figure 1B presents the specification for NPVS dataset. We take Figure 1A as an example. Observed variables are indicated by rectangles and latent variables are indicated by ellipses. Parameters of SEM include regression coefficients, covariance, and variance. The regression coefficients are represented along the single-headed arrows. The double-headed, curved arrows between two variables indicates the covariance, a non-causal association relationship. The variance can be represented by the double-headed, curved arrows, both heads of which point to the same variables. Note that variances are not shown in this paper's figures to simplify the SEM diagrams but are available upon request. As discussed in section Building TPB Constructs by Integrating Variables from Two Datasets, the two datasets use different items to measure the construct of “subjective norm;” therefore a multi-group SEM is adopted to integrate the two groups of data while allowing for similar but not identical items to be used to measure the same construct. Specifically, the structural models have the same forms for two datasets based on the TPB theory, while the measurement models are slightly different depending on which individual items are used to measure “subjective norm.” Considering the total estimable parameters and total number of parameters to be estimated, a degree of freedom is calculated to determine if the proposed SEM is identifiable or not. For identifiable models, different parameter estimation approaches are available for structural equation modeling, which depends on the type of variables. In this study, the robust maximum likelihood estimation is selected due to its reliable performance in handling ordinal variables (43). The overall model fitness is evaluated by multiple criteria such as likelihood ratio test (LRT), comparative fit index (CFI), Tucker-Lewis Index (TLI), and Root Mean Square Error of Approximation (RMSEA) (44, 45).

FIGURE 1

Figure 1. Model specification for the multi-group SEM in the HCP study (A) and the NPVS study (B).

In the initial model specification, multi-group SEM does not specify any constraints on the parameters between the two groups, whose results are introduced in section An Unconstrained Multi-group SEM Applied to TPB-based Violence Screening Behavior. Later, this study explores imposing constraints to ensure equal coefficients in the structural models to re-specify the SEM for comparison. The re-specified model can be found in section A Constrained Multi-group SEM Applied to TPB-based Violence Screening Behavior. Both sections specify the structural models and measurement models based on the traditional TPB theory without organizational influence. In recent years, organizational factors have shown increasing influences on providers' behaviors but their relationship with the TPB constructs remains unknown. To explore organizational expansion of the TPB theory, section Organizational Expansion of TPB-based Violence Screening Behavior investigates organizational factors' relationship with the TPB constructs through model specification and re-specification. The optimal model is recommended based on the goodness-of-fit, which facilitates the understanding of how organizational characteristics are related to providers' attitudes, beliefs, intentions, and behaviors for interpersonal violence screening for the optimization of providers' training, education, and workplace environment.

Results

The survey data from HCP and NPVS were first cleaned and then coded or reversely coded as necessary. Before combining the data of the two datasets, Chi-squared tests were used to compare demographics of participants from two different studies, which indicated no significant difference between the two groups of participants in age (p-value = 0.06), gender (p-value = 0.08), and race (p-value = 0.93). No statistically significant demographics difference was noted between the two groups; this provides support to justify the combination of the two data sources.

Prior to the structural equation modeling, the HCP and NPVS data are tested for validity, consistency and reliability. First, to examine if the data is suitable for factor analysis, Bartlett's Tests of Sphericity (46) and Kaiser-Meyer-Olkin Tests (47) are performed on two groups of data, respectively. Based on Bartlett's Tests of Sphericity, there is sufficient significant correlation in the two datasets for factor analysis with p-value <0.001. The Kaiser-Meyer-Olkin measures are 0.78 and 0.72 for HCP data and NPVS data, respectively, which indicate a strong correlation between factors. Moreover, Confirmatory Factor Analysis is conducted to establish the construct validity by testing the relationships between latent constructs and their individual items, which reports consistent results for both datasets. The likelihood ratio tests result in p-values <0.001. CFIs and TLIs are both higher than 0.95, while RMSEAs are <0.08 (44, 45). The loadings, i.e., coefficients of the measurement models, are all statistically significant with p-values <0.001. The results indicate a good fit between the datasets and the measurement model, thus establishing the construct validity.

The reliability and internal consistency among items within the same construct are examined by Cronbach's alpha. As shown in Table 2, Cronbach's alphas are calculated for HCP, NPVS, and the combined dataset. Cronbach's alphas >0.8 show strong internal consistency within “attitude toward behavior,” “subjective norm,” and “perceived behavior control,” except for “subjective norm” in HCP dataset, which has an alpha of 0.687. However, by combining HCP data with NPVS data, the lower consistency is mediated, and thus the combined dataset shows strong internal consistency for each of the three constructs. Specifically, as shown in Table 2, the construct that represents attitude toward behavior is directly measured by providers' attitude toward screening for IPV (item 1) and SV (item 2), which has a high Cronbach's alpha of 0.934. The construct of subjective norm is indirectly assessed by the products of normative beliefs and motivation to comply with the beliefs, which are labeled as items 11–14 in the table. The subjective norm construct in the combined dataset has a strong internal consistency with a Cronbach's alpha of 0.814. The construct for perceived behavioral control is directly evaluated from items 15–18, which measure providers' capability to preform interpersonal violence screening-related assessments.

TABLE 2

Table 2. Cronbach's alphas to examine internal consistency of each TPB construct.

An open-source statistical software R (40) was used for data processing and model development. Specifically, we take advantage of the existing R package “psy” (48) to examine internal consistency and reliability, R package “parameters” (49) to evaluate validity, and R package “lavaan” (39) for statistical modeling of multi-group SEMs.

An Unconstrained Multi-Group SEM Applied to TPB-Based Violence Screening Behavior

A multi-group Structural Equation Model was applied to the combined dataset to model, estimate, and test the relationship among interpersonal violence screening behavior, intentions, providers' attitudes toward behavior, subjective norm, and perceived behavior control. Structural models are the same for the two groups, while one of the measurement models, i.e., the model for measuring “subjective norm,” includes different individual items for different groups. No constraints on the parameters between the two groups are considered. Note that fitting an unconstrained two-group SEM is equivalent to fitting two SEMs for the two groups separately, since parameters for both groups are freely estimated without any constraints imposed. From Figure 2, we can see the total numbers of observed variables for the two groups are 10 and 12, respectively, and therefore the total number of estimable parameters is $\frac{10\times (10+1)}{2}+\frac{12\times (12+1)}{2}=133$. Considering the number of parameters to be estimated in Figure 2, the degree of freedom for the multi-group SEM is 79. Due to the presence of ordinal variables, a robust maximum likelihood estimation method is used for model estimation (43). The good fitness of the model demonstrates the efficacy of the Theory of Planned Behavior in understanding healthcare providers' violence screening behavior, with CFI = 0.96, TLI = 0.94, and RMSEA = 0.074.

FIGURE 2

Figure 2. An unconstrained multi-group SEM applied to TPB in the HCP study (A) and the NPVS study (B). (Model fit indices: LRT p-value < 0.001, CFI = 0.96, TLI = 0.94, RMSEA = 0.074; *** indicates p-value < 0.001, ** indicates p-value < 0.01, * indicates p-value < 0.05).

The path coefficients are reported in Table 3, which consists of both unstandardized coefficients (USCs) and standardized coefficients (SCs). USCs are the coefficients reported in the SEM diagram as shown in Figure 2. In path analysis, USCs are typically rescaled to SCs to place the coefficients in units of standard deviations of the mean and allow coefficients belonging to different paths to be compared. That is, the expected impact of one standard deviation difference in one variable can be compared to one standard deviation difference in another variable. As a result, we can observe from Table 3 that providers' perceived behavioral control is a stronger predictor to intention, compared with their attitude and subjective norm.

TABLE 3

Table 3. Path coefficients of the unconstrained multi-group SEM.

Figure 2 describes the quantitative relations estimated by the multi-group SEM. Except one, every other arrow that links two constructs is labeled with a p-value <0.05, which indicates a statistically significant association between the two constructs. Comparing the two groups, provider's intentions of screening are largely affected by their attitudes and perceived behavioral control rather than the subjective norms of screening. For both datasets, the coefficients for attitude toward behavior and perceived behavioral control constructs are significant, while the coefficient for subjective norm has a relatively smaller magnitude and is found to significant only in the HCP data. One of the possible reasons is that the sample size of NPSV data is smaller than that of HCP data, in which small effects, e.g., influence of subjective norms on intentions, is characterized by small coefficients that are difficult to be identified in analysis. As a result, the next section explores a constrained multi-group SEM with joint learning of structural models to leverage common variables shared by both HCP data and NPSV data to improve the statistical analysis with an increased sample size. Another possible reason may be due to the variations in violence screening behaviors among healthcare providers who work in different healthcare settings, which is a typical drawback for multi-center data fusion and analysis and is further elaborated in the discussion section.

Note that as a conventional statistical model, SEM typically cannot be used to identify any causal relationships between constructs/variables. However, the significant correlations discovered by SEM could partially support and verify the causations among constructs proposed by TPB from a quantitative perspective.

A Constrained Multi-Group SEM Applied to TPB-Based Violence Screening Behavior

Section An Unconstrained Multi-group SEM Applied to TPB-based Violence Screening Behavior discusses a free multi-group SEM model to analyze HCP data and NPVS data with no constraints imposed on the coefficients. To explore the similarity in relationships between the two groups, this section presents a constrained multi-group SEM by assuming the same coefficients in the structural models, i.e., ${\alpha }_1^H={\alpha }_1^N$, ${\alpha }_2^H={\alpha }_2^N$, ${\alpha }_3^H={\alpha }_3^N$, and β^H = β^N. As a result, the number of parameters to be estimated is reduced by four, and the degree of freedom is increased from 79 to 83.

Figure 3 describes the quantitative relations estimated by the multi-group SEM. The coefficients in structural models are equivalent between two groups and are all significant in terms of p-values. The goodness-of-fit testing indicates that the data fits well in the constrained multi-group SEM, with CFI = 0.96, TLI = 0.94, and RMSEA = 0.075. The estimated SEM diagram is shown in Figure 3. All the hypothesized paths in both datasets are found to be significant in terms of P-values as shown in Table 4, in which USCs are the same for the two groups due to the imposed constraints, but SCs are resulted from standardization of USCs based on group-specific standard deviations and thus different between the two groups of data. Path coefficients indicate providers' perceived behavioral control remains to be a stronger predictor to intention in the constrained model, compared with their attitude and subjective norm. From the perspective of model fitness and model estimation, the consistent results between the constrained model and unconstrained model justify the use of the multi-group modeling approach, which leverages cross-dataset information as well as mitigates the slight discrepancy that resulted from data collection. From the perspective of nursing research, the constrained model indicates the efficacy and stability of using TPB to model healthcare providers' behaviors toward violence screenings across various healthcare settings.

FIGURE 3

Figure 3. A constrained multi-group SEM applied to TPB in the HCP study (A) and the NPVS study (B). (Model fit indices: LRT p-value < 0.001, CFI = 0.96, TLI = 0.94, RMSEA = 0.075; *** indicates p-value < 0.001, ** indicates p-value < 0.01, * indicates p-value < 0.05).

TABLE 4

Table 4. Path coefficients of the constrained multi-group SEM.

Organizational Expansion of TPB-Based Violence Screening Behavior

Beyond the behavioral study of individual healthcare providers, there is a growing consensus that provider behavior is influenced by organizational characteristics of the healthcare settings where they work (50). Existing literature proposed a list of potential organizational factors that impact the healthcare provider's behavior, among which we identify three common factors collected by both HCP and NPVS to further verify the organizational expansion of TPB. Based on the literature review and researchers' experiences, the organizational construct was measured using five items related to availability of policy to screen all female patients at the healthcare center, organizational priorities given to violence screening relative to other priorities, and if providers within the health center are interested in improving care quality. Construct validity was examined based on confirmatory factor analysis with CFI = 0.98, TLT = 0.97, and RMSEA = 0.07.

To determine how TPB constructs are impacted by the organizational factors, we re-specify the multi-group SEM by adding the direct effect from the organizational factors to different TPB constructs. The final model is selected based on the model fitness, which is shown in Figure 4. The model fitness indices indicate a good fit between the HCP data and NPVS data and the multi-group SEM with LRT's p-value < 0.001, CFI = 0.93, TLT = 0.92, and RMSEA = 0.074. Path coefficients are summarized in Table 5, in which all hypothesize paths are shown to be significant with p-values <0.001. Comparing standardized coefficients of all three constructs impacting the intention, one standard deivation difference in the provider's perceived behavioral control leads to the greatest impact on his or her intention to screen for violence with the coefficients being 0.5 and 0.47 for HCP and NPVS studies, respectively. In addition to three newly added arrows, p-values of other arrows remain significantly small, which indicates the validity of adopting organizational expansion of TPB to predict providers' behavior for interpersonal violence screening.

FIGURE 4

Figure 4. Estimated SEM applied to Organizational Expansion of TPB in the HCP study (A) and the NPVS study (B). (Model fit indices: LRT p-value < 0.001, CFI = 0.93, TLI = 0.92, RMSEA = 0.074; *** indicates p-value < 0.001, ** indicates p-value < 0.01, * indicates p-value < 0.05).

TABLE 5

Table 5. Path coefficients of the multi-group SEM with organizational expansion.

Conclusions

This paper developed a multi-group Structural Equation Model integrated with Theory of Planned Behavior (29) to understand healthcare providers' interpersonal violence screening behavior and its relationship to providers' screening intention, attitude toward screening behavior, subjective norm, and perceived behavioral control. Two questionnaire-based studies, HCP and NPVS, are integrated into a combined dataset by identifying common or similar questions shared by both studies. The combined multicenter dataset results in a relatively larger and heterogeneous population in this study. Based upon Guidelines on Constructing TPB-based Questionnaires (29), each TPB construct that consists of a collection of survey questions is examined with Cronbach's alpha and showed a strong internal consistency with Cronbach's alpha >81%. To leverage information from both datasets and mitigate their discrepancy, a multi-group structural equation model was applied to the two datasets to investigate the TPB (24) constructs from a quantitative perspective. The good fitness of the structural equation model validated the utility of adopting the Theory of Planned Behavior (24) to study healthcare providers' behavior for violence screening. Furthermore, the organizational expansion of TPB was assessed by examining how TPB constructs were related to organizational items. These items included the availability of policy to screen all female patients at the healthcare center, the organizational priority given to violence screening relative to other priorities, and provider interest in improving care quality within the health center.

In the field of nursing research, this study is the first to examine the efficacy of the Theory of Planned Behavior (24) with and without organizational expansion, and to find adequate model fit and significant hypothesized paths. This study supports that providers' IPV and SV screening behavior is determined by their screening intentions, attitude toward screening, subjective norms, perceived behavioral control, as well as organizational factors. The knowledge discovered can be used to inform further interventions, such as educating and training healthcare providers and implementing screening policy at healthcare centers. Achieving this goal is important to today's healthcare system because routine screening is considered as a cost-effective strategy to help identify victims. The uptake of routine screening may reduce the harmful consequences of interpersonal violence and prevent the reoccurrence of further acts of violence, which leads to improved women's health and quality of life.

Discussions

This analysis demonstrates the efficacy of TPB in predicting healthcare providers' screening behaviors for interpersonal violence. Based on the SEM results, providers' behaviors are impacted by their intentions, which in turn are affected by their attitudes toward screening, subjective norms, and perceived behavior controls. The findings are consistent with existing literatures. The efficacy of TPB in predicting human behaviors from their intentions, attitudes, and beliefs has been demonstrated in various behavioral studies in healthcare. Guo et al. (51) found healthcare administrators' intentions to use evidence-based management were significantly associated with their attitudes and perceived behavioral control. In another study, perceived behavioral control has been identified as the strongest predictor for healthcare providers' sexual counseling behaviors in people with epilepsy (52). Lin et al. (53) adopted TPB to study the help-seeking behaviors for sexual problems among women, which reported self-stigma and perceived behavioral control had significant effects on help-seeking behaviors (23).

Furthermore, this study identified a few organizational variables as additional factors that impact providers' behaviors for violence screening, which are comparable with existing studies in healthcare practices. In an NIH-funded pilot study on violence screening among college female students, the availability of policy to screen all female students was also shown to be positively associated with providers' screening percentages for interpersonal violence (54). In addition, our study reported that organizational priorities focused on violence screening relative to other priorities and health center providers interested in improving care quality were factors that impact providers' violence screening behaviors. Similarly, Solberg (50) proposed a conceptual framework on identifying facilitators to improve medical practices, in which higher organizational priorities and a higher degree of involvement and engagement by personnel at all levels were shown to be positively associated with quality improvement in healthcare practices.

There are several limitations of this analysis that are common challenges inherent in a secondary multicenter cross-dataset analysis: (1) The analysis results are built upon the 389 providers collected from previous studies and therefore may suffer from bias of participant recruitment; (2) method biases that could arise from a variety of sources, e.g., the content of specific items, scale types, and response formats, may result in measurement errors and need to be examined in future studies; (3) the TPB constructs included in the analysis are produced from common questions shared by both datasets, and there are other approaches listed in the Guideline of TPB-based Survey, which could be used for question design and composite score computation; (4) the sample size of the HCP and NPVS data is sufficient for structural equation modeling, but more data at an individual level and organizational level are needed to examine the presence of ecological fallacy in providers' screening behaviors; and (5) the organizational factors used to test for organizational expansion of TPB are limited to common factors shared by both datasets, but future studies should examine additional factors that could potentially impact the behavior of providers who work at different healthcare centers.

Data Availability Statement

The data will not be deposited to any public repository but will be available upon request from the corresponding author. Requests for data access would be subject to review by researchers from the Health Care Providers (HCP) research steering group and by researchers from the Nurse Practitioners Violence Screening (NPVS) research steering group to ensure congruence with equity research goals. Requests to access the datasets should be directed to Bing Si ( YnNpJiN4MDAwNDA7YmluZ2hhbXRvbi5lZHU=).

Ethics Statement

This paper analyzed data collected from human subjects in the HCP and NPVS studies. Both studies were reviewed and approved by the Institutional Review Board at Boston College with the IRB Protocol Number 16.042.01 (for HCP) and IRB Protocol Number 18.067.01e (for NPVS). Online consent was obtained for both studies. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We would like to acknowledge the principal investigators of HCP and NPVS studies, other researchers who contributed to data collection of the studies, and the healthcare providers who participated.

References

1. InformedHealth.org [Internet]. Cologne, Germany: Institute for Quality and Efficiency in Health Care (IQWiG); 2006-. Benefits and Risks of Screening Tests. (2013). Available online at: https://www.ncbi.nlm.nih.gov/books/NBK279418/ (accessed May 26, 2021).

Google Scholar

2. Bell NR, Grad R, Dickinson JA, Singh H, Moore AE, Kasperavicius D, et al. Better decision making in preventive health screening: balancing benefits and harms. Canad Fam Phys. (2017) 63:521–4.

PubMed Abstract | Google Scholar

3. American Diabetes Association. Screening for diabetes. Diabet Care. (2002). 25(Suppl. 1):s21–4. doi: 10.2337/diacare.25.2007.S21

CrossRef Full Text | Google Scholar

4. Humphrey LL, Helfand M, Chan BK, Woolf SH. Breast cancer screening: a summary of the evidence for the US Preventive Services Task Force. Ann Int med. (2002) 137:347–60. doi: 10.7326/0003-4819-137-5_Part_1-200209030-00012

CrossRef Full Text | Google Scholar

5. Curry SJ, Krist AH, Owens DK, Barry MJ, Caughey AB, Davidson KW, et al. Screening for osteoporosis to prevent fractures: US preventive services task force recommendation statement. JAMA. (2018) 319:2521–31. doi: 10.1001/jama.2018.7498

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Centers for Disease Control and Prevention. Preventing Intimate Partner Violence Factsheet. CDC, National Center for Injury Prevention and Control (2018). Available online at: https://www.cdc.gov/violenceprevention/intimatepartnerviolence/fastfact.html

Google Scholar

7. World Health Organization. Understanding and Addressing Violence Against Women: Health Consequences. World Health Organization (2012). Available online at: https://www.who.int/reproductivehealth/topics/violence/vaw_series/en/

Google Scholar

8. Nelson HD, Bougatsos C, Blazina I. Screening women for intimate partner violence: a systematic review to update the US Preventive Services Task Force recommendation. Ann Int Med. (2012) 156:796–808. doi: 10.7326/0003-4819-156-11-201206050-00447

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Waalen J, Goodwin MM, Spitz AM, Petersen R, Saltzman LE. Screening for intimate partner violence by health care providers: barriers and interventions. Am J Prevent Med. (2000) 19:230–7. doi: 10.1016/S0749-3797(00)00229-4

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Stobo JD, Salmon ME, Cohn F editors. Confronting Chronic Neglect: the Education and Training of Health Professionals on Family Violence. National Academies Press (2002).

PubMed Abstract | Google Scholar

11. Kaur G, Herbert L. Recognizing and intervening in intimate partner violence. Cleveland Clin J Med. (2005) 72:407. doi: 10.3949/ccjm.72.5.406

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Alvarez C, Fedock G, Grace KT, Campbell J. Provider screening and counseling for intimate partner violence: a systematic review of practices and influencing factors. Trauma Violence Abuse. (2017) 18:479–95. doi: 10.1177/1524838016637080

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Tower LE. Barriers in screening women for domestic violence: a survey of social workers, family practitioners, and obstetrician–gynecologists. J Fam Violence. (2006) 21:245–57. doi: 10.1007/s10896-006-9024-4

CrossRef Full Text | Google Scholar

14. Campbell JC, Glass N, Sharps PW, Laughon K, Yragui N, Sutherland MA. Research on intimate partner violence and femicide, attempted femicide, and pregnancy-associated femicide. Fam Intimate Part Violence Q. (2009) 2.

Google Scholar

15. Sormanti M, Smith E. Intimate partner violence screening in the emergency department: US medical residents' perspectives. Int Q Commun Health Educ. (2010) 30:21–40. doi: 10.2190/IQ.30.1.c

CrossRef Full Text | Google Scholar

16. Colarossi LG, Breitbart V, Betancourt GS. Screening for intimate partner violence in reproductive health centers: an evaluation study. Women Health. (2010) 50:313–26. doi: 10.1080/03630242.2010.498751

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Jatoi A, Breitkopf CR. Is Spanish language a barrier to domestic violence assessment? J Womens Health. (2011) 20:1111–6. doi: 10.1089/jwh.2010.2536

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Todahl J, Walters E. Universal screening for intimate partner violence: a systematic review. J Marital Fam Ther. (2011) 37:355–69. doi: 10.1111/j.1752-0606.2009.00179.x

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Hutchinson MK, Sutherland MA. Conducting surveys with multidisciplinary health care providers: current challenges and creative approaches to sampling, recruitment, and data collection. Res Nurs Health. (2019) 42:458–66. doi: 10.1002/nur.21976

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Sutherland MA, Hutchinson MK. Organizational influences on the intimate partner violence and sexual violence screening practices of college health care providers. Res Nurs Health. (2019) 42:284–95. doi: 10.1002/nur.21950

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Sutherland MA, Fantasia HC, Hutchinson MK. Screening for intimate partner and sexual violence in college women: missed opportunities. Womens Health Issues. (2016) 26:217–24. doi: 10.1016/j.whi.2015.07.008

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Sutherland JL, Fantasia HC, Jiang L, Si B. US nurse practitioners' HIV screening behaviors and health center characteristics. J Am Assoc Nurse Pract. (2020). doi: 10.1097/JXX.0000000000000437

CrossRef Full Text | Google Scholar

23. Sutherland MA, Hutchinson MK, Fantasia HC, Si B, Jiang L. Are we asking? Nurse practitioners' interpersonal violence screening practices. J Nurse Pract. (2020) 16:683–8. doi: 10.1016/j.nurpra.2020.07.023

CrossRef Full Text | Google Scholar

24. Ajzen I. The theory of planned behavior. Organ Behav Hum Decis Process. (1991) 50:179–211. doi: 10.1016/0749-5978(91)90020-T

CrossRef Full Text | Google Scholar

25. Omondi DO, Walingo MK, Mbagaya GM, Othuon LO. Understanding physical activity behavior of type 2 diabetics using the theory of planned behavior and structural equation modeling. Int J Hum Soc Sci. (2010) 5:160–7.

Google Scholar

26. Teo TK, Lee CB. Examining the efficacy of the theory of planned behavior (TPB) to understand pre-service teachers' intention to use technology. In: ASCILITE-Australian Society for Computers in Learning in Tertiary Education Conference. (2010). p. 968–72.

Google Scholar

27. AL-Dossary SA. Why do college students cheat? A structural equation modeling validation of the theory of planned behavior. Int Educ Stud. (2017) 10:40–51. doi: 10.5539/ies.v10n8p40

CrossRef Full Text | Google Scholar

28. Zhang K, Guo H, Yao G, Li C, Zhang Y, Wang W. Modeling acceptance of electric vehicle sharing based on theory of planned behavior. Sustainability. (2018) 10:4686. doi: 10.3390/su10124686

CrossRef Full Text | Google Scholar

29. Francis J, Eccles MP, Johnston M, Walker AE, Grimshaw JM, Foy R, et al. Constructing questionnaires based on the theory of planned behaviour: a manual for health services researchers. Appl Health Sci. (2004).

Google Scholar

30. Fox J. Teacher's corner: structural equation modeling with the sem package in R. Struct Equat Model. (2006) 13:465–86. doi: 10.1207/s15328007sem1303_7

CrossRef Full Text | Google Scholar

31. Hu LT, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Struct Equat Model. (1999) 6:1–55. doi: 10.1080/10705519909540118

CrossRef Full Text | Google Scholar

32. Gupta MA, Gupta AK, Chen SJ, Johnson AM. Comorbidity of rosacea and depression: an analysis of the National Ambulatory Medical Care Survey and National Hospital Ambulatory Care Survey—outpatient department data collected by the US National Center for Health Statistics from 1995 to (2002). Br J Dermatol. (2005) 153:1176–81. doi: 10.1111/j.1365-2133.2005.06895.x

CrossRef Full Text | Google Scholar

33. Harding JJ, Egerton M, Van H, eyningen R, Harding RS. Diabetes, glaucoma, sex, and cataract: analysis of combined data from two case control studies. Br J Ophthalmol. (1993) 77:2–6. doi: 10.1136/bjo.77.1.2

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Miller NV, Currie SR. A Canadian population level analysis of the roles of irrational gambling cognitions and risky gambling practices as correlates of gambling intensity and pathological gambling. J Gambl Stud. (2008) 24:257–74. doi: 10.1007/s10899-008-9089-5

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Si B, Dumkrieger G, Wu T, Zafonte R, Dodick DW, Schwedt TJ, et al. A cross-study analysis for reproducible sub-classification of traumatic brain injury. Front Neurol. (2018) 9:606. doi: 10.3389/fneur.2018.00606

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Garrett-Mayer E, Parmigiani G, Zhong X, Cope L, Gabrielson E. Cross-study validation and combined analysis of gene expression microarray data. Biostatistics. (2007) 9:333–54. doi: 10.1093/biostatistics/kxm033

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Nir A, Lindinger A, Rauh M, Bar-Oz B, Laer S, Schwachtgen L, et al. NT-pro-B-type natriuretic peptide in infants and children: reference values based on combined data from four studies. Pediatr Cardiol. (2009) 30:3. doi: 10.1007/s00246-008-9258-4

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Warnat P, Oberthuer A, Fischer M, Westermann F, Eils R, Brors B. Cross-study analysis of gene expression data for intermediate neuroblastoma identifies two biological subtypes. BMC Cancer. (2007) 7:89. doi: 10.1186/1471-2407-7-89

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Rosseel Y. lavaan: an R package for structural equation modeling. J Stat Softw. (2012). 48:1–36. doi: 10.18637/jss.v048.i02

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Core R. Team. R: A language and environment for statistical computing. R foundation for statistical computing, Vienna, Austria. (2019). Available online at: http://www.R-project.org/ (accessed May 26, 2021).

Google Scholar

41. Beran TN, Violato C. Structural equation modeling in medical research: a primer. BMC Res Notes. (2010) 3:1–0. doi: 10.1186/1756-0500-3-267

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Kline RB. Principles and Practice of Structural Equation Modeling. Guilford Publications. (2015).

Google Scholar

43. Gana K, Broc G. Structural Equation Modeling With Lavaan. John Wiley & Sons. (2019). doi: 10.1002/9781119579038

CrossRef Full Text | Google Scholar

44. Hooper D, Coughlan J, Mullen MR. Equation modelling: guidelines for determining model fit. Electron J Bus Res Methods. (2008) 6:53–60.

Google Scholar

45. Bentler PM, Bonett DG. Significance tests and goodness of fit in the analysis of covariance structures. Psychol Bull. (1980) 88:588. doi: 10.1037/0033-2909.88.3.588

CrossRef Full Text | Google Scholar

46. Bartlett MS. The effect of standardization on a χ 2 approximation in factor analysis. Biometrika. (1951) 38:337–44. doi: 10.1093/biomet/38.3-4.337

CrossRef Full Text | Google Scholar

47. Kaiser HF, Rice J. Little jiffy, mark IV. Educ Psychol Meas. (1974) 34:111–7. doi: 10.1177/001316447403400115

CrossRef Full Text | Google Scholar

48. Falissard B. psy: Various Procedures Used in Psychometry. R Package Version 1.1. (2012). Available online at: https://CRAN.R-project.org/package=psy (accessed May 26, 2021).

Google Scholar

49. Lüdecke, Ben-Shachar & Makowski. Describe and Understand Your Model's Parameters. CRAN. (2020). Available online at: https://easystats.github.io/parameters (accessed May 26, 2021).

Google Scholar

50. Solberg LI. Improving medical practice: a conceptual framework. Ann Fam Med. (2007) 5:251–6. doi: 10.1370/afm.666

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Guo R, Berkshire SD, Fulton LV, Hermanson PM. Predicting intention to use evidence-based management among US healthcare administrators: application of the theory of planned behavior and structural equation modeling. Int J Healthc Manage. (2019) 12:25–32. doi: 10.1080/20479700.2017.1336856

CrossRef Full Text | Google Scholar

52. Lin CY, Fung XC, Nikoobakht M, Burri A, Pakpour AH. Using the Theory of Planned Behavior incorporated with perceived barriers to explore sexual counseling services delivered by healthcare professionals in individuals suffering from epilepsy. Epilepsy Behav. (2017) 74:124–9. doi: 10.1016/j.yebeh.2017.06.011

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Lin CY, Broström A, Årestedt K, Mårtensson J, Steinke EE, Pakpour AH. Using extended theory of planned behavior to determine factors associated with help-seeking behavior of sexual problems in women with heart failure: a longitudinal study. J Psychosom Obstet Gynecol. (2020) 41:54–61. doi: 10.1080/0167482X.2019.1572743

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Sutherland MA, Hutchinson MK. Intimate partner and sexual violence screening practices of college health care providers. Appl Nurs Res. (2018) 39:217–9. doi: 10.1016/j.apnr.2017.11.031

PubMed Abstract | CrossRef Full Text | Google Scholar