Sustaining and scaling a clinic-based approach to address health-related social needs

Objective: Scaling evidence-based interventions (EBIs) from pilot phase remains a pressing challenge in efforts to address health-related social needs (HRSN) and improve population health. This study describes an innovative approach to sustaining and further spreading DULCE (Developmental Understanding and Legal Collaboration for Everyone), a universal EBI that supports pediatric clinics to implement the American Academy of Pediatrics' Bright Futures™ guidelines for infants' well-child visits (WCVs) and introduces a new quality measure of families' HRSN resource use.

Methods: Between August 2018 and December 2019, seven teams in four communities in three states implemented DULCE: four teams that had been implementing DULCE since 2016 and three new teams. Teams received monthly data reports and individualized continuous quality improvement (CQI) coaching for six months, followed by lighter-touch support via quarterly group calls (peer-to-peer learning and coaching). Run charts were used to study outcome (percent of infants that received all WCVs on time) and process measures (percent of families screened for HRSN and connected to resources).

Results: Integrating three new sites was associated with an initial regression of outcome: 41% of infants received all WCVs on time, followed by improvement to 48%. Process performance was sustained or improved: among 989 participating families, 84% (831) received 1-month WCVs on time; 96% (946) were screened for seven HRSN, 54% (508) had HRSN, and 87% (444) used HRSN resources.

Conclusion: An innovative, lighter-touch CQI approach to a second phase of scale-up resulted in sustainment or improvements in most processes and outcomes. Outcomes-oriented CQI measures (family receipt of resources) are an important addition to more traditional process-oriented indicators.

1. Introduction

Evidence-based interventions (EBIs) delivered through the pediatric medical home can improve outcomes among families with young children (1). Integrating health-related social needs (HRSN) screening and support into primary care is a priority strategy to improve population health outcomes (2). The American Academy of Pediatrics (AAP) Bright Futures™ 4th Edition (BF4) recommends that pediatric clinics address HRSN during well-child visits (WCVs) (3).

Evidence for effectiveness of interventions addressing HRSN in pediatric settings comes mostly from pilot-sized studies (4). However, in the real world, many small-scale pilot studies demonstrating effectiveness are never implemented widely (5, 6). The American Academy of Pediatrics noted that most efforts to expand successful pilot interventions at scale result in disappointingly small effect sizes (7). Emerging literature describes frameworks for large-scale dissemination of EBIs, factors that facilitate uptake, and lessons learned from scale-up attempts (5, 8–13), but there is no established single best approach for scaling EBIs to improve population health (10).

The Institute for Healthcare Improvement's Breakthrough Series Collaborative model (BTS) has demonstrated potential to increase uptake of EBIs across diverse contexts (14–18), by combining continuous quality improvement methodologies and networked peer learning. Continuous quality improvement methods (CQI) engage the entire organization and its frontline providers in a series of ongoing observations, adjustments, and interventions to produce measurable improvements in outcomes (19, 20). Central to this approach are Plan-Do-Study-Act (PDSA) cycles, which allow cross-hierarchical teams of service providers to identify ideas they believe might improve outcomes; they plan a small test of that idea, do that test in real day-to-day practice, study how the test was executed and what resulted (using observations and data), and act on those results – i.e., abandoning ideas that did not work, adapting ideas that seemed promising but in need of optimization (then re-testing), or adopting into practice ideas that worked optimally in their contexts. By ensuring that change ideas are tested and adapted to the local context by frontline teams making real-time, data-based decisions, PDSA testing facilitates adaptive design that can accommodate different contexts as an intervention is scaled up (21).

The Breakthrough Series Collaborative model (BTS) combines CQI methodologies with networked peer learning (22). It recruits teams of direct service providers and stakeholders to pursue one shared, specific aim during a defined period of time, typically 9 to 18 months, and creates a structure wherein interested organizations can learn from each other and recognized experts. The model has three core elements: 1) learning sessions that bring teams together periodically for training and collaboration, separated by 2) “action periods” during which teams test what they have learned in practice, using 3) Plan-Do-Study-Act cycles – the structured approach for learning from rapid-cycle testing of innovations in practice. Figure 1 depicts this traditional BTS structure.

FIGURE 1

Figure 1. The Institute for Healthcare Improvement's Breakthrough Series Learning Collaborative.

Continuous quality improvement and BTS methods have been applied widely. The CQI approach emerged in the 1950s to overcome manufacturing deficiencies (23, 24) and has subsequently been applied in healthcare, public health, nonprofit and public management and, recently, education (25–28). Since 1995, the BTS model has increased uptake of EBIs and improved outcomes in public health (20, 27, 29), including improved rates and reduced disparities in immunizations (30), spread of trauma-informed care practices in child welfare (31, 32), and improved breastfeeding, developmental promotion, and caregiver depression outcomes in home visiting programs (33–35). In clinical medicine, BTS has been applied to several dozen topics involving over 2,000 teams from 1,000 healthcare organizations to achieve concrete results: reducing waiting times by 50%, worker absenteeism by 25%, ICU costs by 25%, hospitalizations for patients with congestive heart failure by 50%, and eliminating 100,000 deaths due to medical errors (15, 36). More recently, the Breakthrough Series has entered the education field as “Networked Improvement Communities.” (37–40). For example, two attendance-focused BTS collaboratives increased school attendance: one from 44.9% to 59.2% at seven early childhood education centers in New Zealand (41); another from 83.7% to 87.1% at five public preschools in rural Chile (42).

This study reports an effort to scale DULCE (Developmental Understanding and Legal Collaboration for Everyone) – an evidence-based, cross-sector intervention for addressing HRSN among families with infants that is delivered through pediatric primary care. A first effectiveness trial of DULCE, conducted at a single site with 330 families demonstrated that DULCE increased preventive care adherence and accelerated families' access to HRSN supports (43). A subsequent study of DULCE expansion to five sites serving 692 families used a BTS Collaborative model as its scale framework and replicated these findings - increased on-time WCVs and accelerated access to HRSN resource information (44). That application of the BTS model provided resource-intensive support including four in-person, group training sessions (12 days total), two or three coaching contacts each month (including monthly group implementation webinars and individual site CQI coaching), and two site visits.

Tailoring the type and intensity of support over time as an EBI spreads is crucial, since resources (e.g., time, technical, financial) are often limited (45), and different phases of scale may require different supports (46). Extant literature suggests that later phases of expansion may require less intensive (and less costly) supports because they benefit from people who participated in early phases of expansion championing the EBI and mentoring their peers. In addition, experience gained in the earliest phase of scale – experience testing the theory of change under a broad range of conditions, developing infrastructure and human capacity to support the method being used to scale up, etc.— may facilitate acceleration in the rate of EBI adoption (8). However, sustained implementation of EBIs in real-world settings is a considerable challenge, and many fail to continue once support is decreased or removed.

This study examined whether a lighter-touch application of the same BTS scale framework could be used to sustain DULCE practice improvements in four established sites and spread practice improvements to three new clinic sites. In addition, it added new effectiveness data by measuring families' HRSN resource use. Specifically, it answers two research questions:

1. Can a lighter-touch application of BTS sustain improvements achieved during a prior expansion by four established DULCE teams and spread improvements in on-time WCVs and identification and support for HRSN to three new clinics?

2. What proportion of families use resources after receiving resource information for identified HRSN?

2. Materials and methods

2.1. Intervention

DULCE (Developmental Understanding and Legal Collaboration for Everyone) is a universal, evidence-based pediatric primary care approach for families with infants from birth through 6 months of age. DULCE embeds a community health worker (“Family specialist,” FS) within a cross-sector team that includes an early childhood system representative, legal partner, clinic administrator, and pediatric and behavioral health clinicians. The team works together to link families to needed resources.

DULCE's theory of change for improving completion of preventive care is visualized in a driver diagram with four primary drivers – that is, key determinants – that contribute to reaching the goal of on-time well-child visit completion (see Figure 2). The first driver focuses on comprehensive care enriched by a Family Specialist (FS, i.e., a community health worker) who attends WCVs, reinforces protective factors, offers developmental guidance and is families' most frequent point of contact. All FS received Brazelton Touchpoints^TM training (47). The second driver concentrates on identification of families’ strengths and HRSN and family-led problem-solving across seven evidence-based HRSN domains: caregiver depression, intimate partner violence (IPV), housing conditions, housing instability, food insecurity, employment/financial needs, and utilities.

FIGURE 2

Figure 2. DULCE Key Driver Diagram.

The third driver emphasizes the cross-sector team that includes the FS, an early childhood system representative, legal partner, clinic administrator, and pediatric and behavioral health clinicians. This team conducts weekly case reviews; collaborates to support families' access to benefits, services and legal protections; and identifies opportunities to effect policy and systems improvements (48). The fourth driver prioritizes families as partners via diverse strategies, (e.g., via exit surveys, periodic celebrations with focus groups, and as DULCE CQI team members).

2.2. Implementation strategy

In previous work (hereafter referred to as “phase one”), a resource-intensive BTS collaborative succeeded in scaling DULCE to five clinical sites that increased on-time WCVs and accelerated access to HRSN resource information. This study (“phase two”) aimed to use a lighter-touch application of BTS to sustain improvements in four established DULCE teams (one site from phase one discontinued DULCE due to institutional leadership transitions) and spread improvements to three new clinics.

In both phases, DULCE's implementation was guided by the principles and core components of the BTS model. DULCE sites formed teams of direct service providers and stakeholders that included the Family Specialist, early childhood system representative, legal partner, clinic administrator, and pediatric and behavioral health clinicians. Teams committed to pursue one shared, specific aim (75% of 6-month-old infants and their families receive all five recommended WCVs on-time) during a defined period of time (19 months in phase one, 17 months in phase two).

In both phases, DULCE implementation was supported by the DULCE National team, comprised of staff from the lead organization, the Center for the Study of Social Policy (CSSP), together with DULCE model developers, a CQI expert, practicing pediatrician, and infant mental health specialists. CSSP staff had extensive experience and established relationships with Early Childhood Systems leaders nationally. DULCE model developers created the DULCE approach and conducted the original randomized controlled trial (RCT), then remained actively involved through phase one and phase two expansion not only as thought partners and strategists, but also as teachers, mentors and coaches to DULCE site teams and their role-specific counterparts (e.g., clinic leaders, providers, and legal partners). Similarly, the practicing pediatrician and infant mental health specialists each led portions of trainings and coached teams, clinicians, and Family Specialists. The CQI expert supported site teams in using data to inform practice and in applying CQI tools (e.g., PDSA cycles, process maps) to adapt DULCE interventions to work well in their local contexts.

In both phases, efforts were made to include DULCE team members who shared lived experience with the DULCE families. This involved hiring Family Specialists from similar racial, ethnic and linguistic backgrounds to the patients served by DULCE. In practice, this meant hiring Family Specialists with bachelor's or paraprofessional educational level, more consistent with a community health worker profile, in contrast with the original RCT that used masters-level Family Specialists. The decision to change the profile of the FS job description reflected two priorities: an effort to accommodate different cultures and languages to better serve local populations, as well as an effort to design for sustainability and scale, since requiring masters-level FS might limit DULCE's potential reach.

Like Family Specialists, DULCE clinics and teams reflect the populations they serve, often coming from the same communities as families. DULCE team members are culturally and linguistically equipped to care for their local populations, which maximizes the impact of their various areas of expertise (e.g., legal, behavioral health). Furthermore, DULCE print and multimedia materials are available in both English and Spanish. When a family cannot communicate in any of the languages spoken by their care team, clinic interpreters are available. DULCE teams also took advantage of trainings that were offered by cross-sector partners for their own employees in order to further develop their teams' capacities to serve their communities. For example, early childhood systems invited DULCE Family Specialists and behavioral health clinicians to participate in trainings on empathic inquiry, cultural humility, and other topics originally designed for public health home visitors. Legal partners invited clinical staff and early childhood partners to attend educational charlas they provided for immigrants and other patients on “Know Your Rights.”.

The DULCE National team created a structure wherein teams learned from each other and DULCE National using in-person learning sessions, virtual webinars, and individual team coaching. In addition, teams exchanged learning, identified gaps in process and outcome performance, and drafted PDSA cycles to test and adapt DULCE's intervention elements between Learning Network Calls, until they worked effectively in their own context. Throughout, teams shared data, lessons, and best practices to improve collectively.

DULCE teams' readiness and capacity for integrating DULCE practices and adapting them using CQI methods varied, which is expected and desirable within the BTS model. Rather than controlling for differences in organizational capacities, DULCE National facilitated activities at Learning Sessions for local site teams to identify gaps in process and outcome performance, then summarized teams' performance using a Balanced Scorecard (Figure 3). In April 2018 (four months prior to the start of this study period) and again in April 2019, sites compared their DULCE data (not inclusive of April data) against the Key Driver Diagram's aims, assessing how well they were implementing DULCE processes and meeting DULCE outcomes (needs improvement, partially in place, meeting aim). The Balanced Scorecard identified for DULCE National focused teaching topics and local site teams with strong performance in those areas who could teach alongside DULCE National to demonstrate how they successfully implemented that specific DULCE practice. Simultaneously, the Balanced Scorecard made it easy for each team to talk about their gaps and select priority areas for PDSA testing.

FIGURE 3

Figure 3. DULCE's balanced scorecard, which reflects its Key Driver Diagram measures, assessed sites’ implementation strengths and priority areas for improvement during two periods: January 2017–march 2018 (phase one; established sites only) and June 2018–march 2019 (phase two; new and established sites). These were self-assessments, and some sites did not report certain measures.

New and established DULCE teams designed PDSA cycles to address implementation challenges and drafted process maps with action plans for testing. Teams initiated PDSA cycles by generating predictions about how certain change ideas would impact implementation; then plan how to enact the change idea; do what they planned; study the data collected; and, finally, act on how to move forward with the change to achieve the desired results (adopt, adapt, abandon). Supplement 1 shows the PDSA form teams used to guide this process. In keeping with best practices, DULCE teams held monthly CQI meetings where they reviewed PDSAs and solicited feedback from all DULCE team members and clinic staff involved in testing the change idea, as well as DULCE families (e.g., exit surveys).

Figure 4 provides a comparison of support to sites during phases one and two. In phase one, between January 2017 and July 2018, the resource-intensive BTS support included four (4) two- or three-day, in-person Learning Sessions for all DULCE team members from all sites with the DULCE National team; bi-monthly group implementation webinars (total = 6); cross-site, role-alike calls (e.g., Family Specialists from all sites together (18 calls), legal partners from all sites together (12 calls), Early Childhood Leads all together [(18 calls), providers and clinic administrators all together (3 calls); (total = 51)]; monthly site-level CQI coaching with DULCE National's CQI Lead and data reports provided by DULCE National (13 per site = 52 total); and two site visits per site from the DULCE National team (8 total). Phase two's lighter-touch BTS support included two in-person convenings: an initial two-day in-person training where DULCE National provided training about DULCE and CQI methods and established DULCE teams presented illustrative examples (e.g., a role-play of cross-sector case review), and a second all-site convening halfway through (DULCE National Forum) with expert speakers and team presentations; monthly implementation webinars for new sites (6 total) followed by quarterly all-site webinars (4 total), two Family Specialist role-alike calls, monthly individual-level site-level CQI coaching with data compiled by the sites (6 per site = 42 total), and no site visits.

FIGURE 4

Figure 4. Comparison of CQI support during phases 1 & 2 of DULCE scale-up: established sites (“E”) and new sites (“N”) underwent different levels of CQI intensity when first implementing DULCE.

Thus, the phase two teams received fewer in-person trainings (2 v. 4) and site visits from the DULCE National team (0 v. 2) and fewer virtual supports (2 v. 51 role-alike group calls) for shorter duration (monthly cross-site implementation webinars for 6 months, followed by quarterly v. bimonthly webinars for 12 months; monthly CQI coaching for 6 months v. 13 months). Furthermore, phase two teams managed their own data locally, whereas in phase one, DULCE National managed all data in a centralized registry and provided monthly site-level data reports that included process and outcome measures.

2.3. Intervention study

To recruit participants for DULCE spread, the Center for the Study of Social Policy contacted its Early Childhood Learning and Innovation Network for Communities (EC-LINC), a national network of 14 communities that are early childhood systems innovators. In the first phase of expansion, three communities volunteered and recruited clinics serving predominantly Medicaid-insured patients and local public interest law organizations to form DULCE teams. In this second phase, the Los Angeles community added two additional clinics from healthcare systems that installed DULCE in phase one (Table 1). A third DULCE site, Palm Beach County, joined this CQI cohort during phase two; they began implementing DULCE during phase one but did not collect data nor participate in CQI until phase two and are thus considered part of this new cohort.

TABLE 1

Table 1. Cross-sector DULCE team members and participating communities.

2.3.1. Data and measures

Family Specialists entered individual-level data into an online, custom-built registry. Demographic characteristics included infant sex; caregiver role, age, marital status, race, and ethnicity; household composition (number of adults and children) and language(s) spoken in the home. Implementation measures included the proportion of families offered participation that enrolled in DULCE, the proportion of enrolled families that completed DULCE, duration of participation (in weeks), number of encounters, and total contact time between families and the Family Specialist. A measure of case review implementation was added for phase two (percent of families discussed in case review at least once within 2 months of DULCE enrollment) because experience from phase one taught us that weekly cross-sector case review was difficult for teams to implement, initially. It was new and logistically challenging to get all DULCE team members in the same room at the same time, especially during clinic operating hours when patient care is the priority. However, once teams experienced case review, it was highly valued and became self-sustaining. As one team leader shared at the October 2018 onboarding when established teams were asked to provide a piece of advice for new DULCE teams: “Commit yourself to weekly cross-sector case review; it's the heartbeat of DULCE.”.

Process measures aligned with the primary drivers: PD1) percent of WCVs attended by the FS, PD2) percent of families that were screened for seven HRSN using validated, standardized screening questions, and, among positive screens, the percent of families provided resource information PD3) percent of families with identified HRSN that used HRSN resources.

The main outcome was the percent of six-month-old infants who received all recommended WCVs on time. It includes infants who completed the intervention and received five WCVs on time and infants who dropped out and received all recommended WCVs on time up to the date of dropout. DULCE National defined “on time” based on precedent (44, 49).

Families' data was aggregated to the clinic site where they received care, except for the Children's Clinic–Los Angeles (LA) sites [Central Long Beach Family Health Center, S. Mark Taper Foundation Health Center (SMTF)], whose data was reported together. As a result, there are data from one mixed new-established site (Children's Clinic–LA), two new sites (Newhall–LA and Brumback–Palm Beach), and three established sites (Appleseed–Lamoille, Highland–Alameda, Northeast Valley–LA).

2.4. Definition of the sample

Families with newborns up to 8 weeks of age were enrolled at their first office visit, excluding newborns hospitalized for >7 days after birth because they may warrant specialized services. At sites with more newborns than one FS could serve, DULCE was offered to a randomly selected subset. Clinics introduced DULCE as part of routine care, included information about DULCE in welcome packets, and introduced the FS as a care team member at the first WCV. Families could opt out.

Newborn enrollment (up to 8 weeks of age) was ongoing and continued beyond the study period. This report includes babies born June 2018 through December 2019 and followed through their six-month WCV.

Descriptive statistics were calculated for patient demographic characteristics by site and for the complete analytic sample, which includes 989 families with infants born June 2018 through December 2019 (Table 2). To describe the reach of phase two expansion, Table 3 presents a comparison of the early childhood system's catchment population (i.e., the county) to DULCE-enrolled families (the “County” and “DULCE” columns, respectively). To examine how well-aligned DULCE team members' and DULCE families’ backgrounds were, the “Team” column of Table 3 presents the composition of each site's DULCE team, which largely reflected the racial, ethnic, and linguistic makeups of their communities and/or DULCE families. Besides the DULCE site in Vermont, which enrolled 127 of 346 newborns (37%) in the county, all other sites enrolled less than 1% of infants in the early childhood system's catchment area. To put in perspective how many families DULCE reached within each site's healthcare system, we also summarized the number of infants enrolled relative to the system-level and clinic-level newborn populations (Table 4). Similarly, the Vermont site reached a higher proportion of its system-level and clinic-level infants (49.0% and 89.4%, respectively), compared to the California sites' system-level and clinic-level reach, which ranged 8.3%–12.5% and 14.3%–50.0%, respectively.

TABLE 2

Table 2. Demographic characteristics of DULCE families by site.

TABLE 3

Table 3. Demographics of the early childhood system's county catchment area, DULCE families, and DULCE teams. .

TABLE 4

Table 4. DULCE's reach within each site's pediatric clinic(s) and affiliated healthcare system.

Table 3 also includes the Social Vulnerability Index (50) for each county: Lamoille County has low vulnerability, Alameda County has medium to high vulnerability, while Los Angeles and Palm Beach counties have a high level of vulnerability. In these latter three counties, Hispanic/Latinx and Black families are overrepresented among DULCE families, relative to county-level demographics. This overrepresentation reflects the intention to launch DULCE in clinics with high Medicaid-enrolled populations, which tend to have higher proportions of Hispanic/Latinx and Black patients.

2.5. Analysis

To answer the first research question – can a lighter-touch application of BTS sustain improvements achieved during a prior expansion by four established DULCE teams and spread improvements in on-time WCVs and identification and support for HRSN to three new clinics? – we first calculated descriptive statistics for measures of implementation fidelity for all sites together and for each clinic-based team separately: DULCE enrollment and completion rates, number of weeks enrolled, total number of encounters per family, percent of families discussed in case review at least once, and FS-family contact time. We compared these values to benchmark values from phase one, except for the case review measure which was collected for the first time during phase two.

To determine if improvements were sustained and spread, we calculated the process measures associated with DULCE's primary drivers and the outcome measure associated with its main aim for all sites together and for each clinic-based team separately. We then analyzed process and outcomes in time series as run charts, well-established methods that can identify changes that are unlikely due to chance alone and allow inferences to be drawn from the temporal relationships of interventions and results (51). Subjects were counted in the denominator of each measure once (i.e., denominators are independent of each other) and placed in the month they enrolled (all WCVs on time; all HRSN measures), in the month the data point occurred (FS WCV attendance), or, for the 1-month WCV timeliness measure, in the month containing the last day of the 1-month visit window.

Means were used because these were non-continuous data, and some measures' medians were extreme values due to high baseline performance (e.g., baseline median screening rates of 100%) and small site-level denominators (e.g., months with few or no identified IPV cases). Criteria for applying probability-based rules for identifying improvements were met: denominators were roughly equal over time, and at the aggregate level, data was appropriately dispersed (52).

Two probability-based rules were used to identify changes in the data that have less than 5% probability of occurring by chance: a “shift” of six or more points in a row above or below the mean, and a “trend” of five consecutive increasing or decreasing points (53). When a shift occurred, the average of the six shifted points became the new mean, from which subsequent shifts were identified.

To answer the second research question – what proportion of families use resources after receiving resource information for identified HRSN? – for families that screened positive for HRSN, we calculated the proportion of families that were provided resource information and the proportion that used HRSN resources for caregiver depression and/or IPV, for concrete supports, and for each HRSN separately.

Analyses were conducted using Stata 14.2.

2.6. Ethical considerations

The University of Chicago School of Social Administration's Institutional Review Board (IRB17-0414) approved this study.

3. Results

Table 2 describes the analytic sample. Families of 989 infants born June 1, 2018 through December 31, 2019 participated; 98.9% of primary caregivers were mothers whose median age was 28 years. Forty-three percent were single; 20.8% identified as White, 20.4% as Black, and 52.9% as Hispanic/Latinx. Sixty-one percent of families reported a second caregiver: 96.7% were fathers, whose median age was 30 years. Families mainly spoke English (63.4%) or Spanish (24.4%). Families represented the demographics of the clinics' populations and were similar to families in DULCE's phase one expansion (44). The level to which families reflected county-level demographics (i.e., the early childhood system's catchment area) varied by site (Table 3). For example, the demographic characteristics of DULCE families at Appleseed Pediatrics were similar to those of Lamoille County, but Highland Hospital in Oakland, CA (a safety-net hospital) served a much higher proportion of Hispanic/Latinx and Black families than are represented in Alameda County's demographic data.

To answer the first research question – can a lighter-touch application of BTS sustain improvements achieved during a prior expansion by four established DULCE teams and spread improvements in on-time WCVs and identification and support for HRSN to three new clinics? – we first consider the main outcome (i.e., on-time WCVs) then implementation and process measures. We compare them to the phase one outcome, implementation, and process measures.

Figure 5 shows the main outcome: at baseline, 41.4% of six-month-old infants completed all five recommended WCVs on time. A shift to 48.1% occurred at six months (February 2019). One clinic (Children's Clinic–LA) demonstrated an increase from 38.8% to 49.4% that correlated with improved 1-month WCVs timeliness (78.4% to 87.7%) when they tested standardized scheduling at their two clinics (one established and one new). Overall, this is comparable with phase one, during which sites demonstrated a shift from a baseline average of 45.8% to 65.4% that also was associated with improvements in on-time 1-month WCVs at three sites (Appleseed–Lamoille, Highland–Alameda, and Northeast Valley–LA). The more modest improvement might be explained, at least in part, by the decrease in preventive care by infants and their families during the COVID-19, which affected this measure for all infants born after August 2019. In addition, during phase two, one established site and one new site experienced downward shifts concurrent with the transition from individual site-level monthly coaching with data provided by DULCE National to quarterly Learning Network Calls and local data management: from 45.7% to 29.3% at Highland–Alameda (March 2019), and 39.1% to 23.9% at Brumback–Palm Beach (July 2019).

FIGURE 5

Figure 5. Percent of infants that received recommended well-child visits on time. Star symbols denote new DUCLE sites: Brumback (Palm Beach County), The Children's Clinic [Los Angeles (LA) County], which aggregated data for its established and new DULCE clinics, and Newhall, part of the Northeast Valley Health Corporation (LA county), a previously established DULCE site.

Overall, implementation measures were similar between phases one and two. All families offered DULCE enrolled, and 73.8% completed the six-month intervention, mirroring enrollment and completion rates observed in phase one (100% and 79%, respectively). Nearly two-thirds of families that left early moved away or changed clinics (Table 5), like in phase one. In phase two, enrolled families had a median intervention dose of 26 weeks [Interquartile Range (IQR), 20–28], nine encounters (IQR, 6–14), and 180 min of FS contact time (IQR, 120–295), compared to 24 weeks [confidence interval (CI), 23.1–24.2], 11 encounters (CI, 10.3–11.1), and 280 min (CI, 265–294) in phase one. During phase two, teams discussed 67% of families in case review within two months of DULCE enrollment; this measure was not collected in phase one.

TABLE 5

Table 5. DULCE enrollment, completion and reasons for leaving early.

Figure 6 shows that the process measure for FS present in WCVs (PD1) was high at baseline and remained stably high throughout phase two: in aggregate, FS attended 89.8% of WCVs (Figure 6). This exceeded phase one performance, which improved from 0% to 66% to 70%. Two phase two new sites immediately achieved and sustained high FS presence: Newhall–LA (92.3%) and Brumback–Palm Beach (92.0%). At the Children's Clinic–LA site, which reported data for its established site and new site together, FS presence in WCVs dipped to 88.2% when the new SMTF site joined, then shifted to 93.3% in October 2018, coincident with the two-day, in-person onboarding. All sites demonstrate a sharp decline in the first quarter of 2020 that corresponds to clinic safety protocol changes in response to the COVID-19 pandemic (Figure 6).

FIGURE 6

Figure 6. Percent of well-child visits (WCVs) attended by the family specialist. Star symbols denote new DUCLE sites: Brumback (Palm Beach County), The Children's Clinic [Los Angeles (LA) County], which aggregated data for its established and new DULCE clinics, and Newhall, part of the Northeast Valley Health Corporation (LA county), a previously established DULCE site.

Figure 7 shows process measures for HRSN screening (PD2). The phase two aggregate measure of families screened for all seven HRSN improved from a baseline of 75.7% to 89.9%, with three sites demonstrating shifts: one mixed new-established site [55.1% to 88.3% (Children's Clinic–LA)] and two established sites [70.4% to 84.7% (Highland–Alameda), 75.0% to 97.9% (Northeast Valley–LA)]. In phase one, sites screened 92% of families for all seven HRSN.

FIGURE 7

Figure 7. Percent of families screened for health-related social needs (HRSN). Star symbols denote new DUCLE sites: Brumback (Palm Beach County), The Children's Clinic [Los Angeles (LA) County], which aggregated data for its established and new DULCE clinics, and Newhall, part of the Northeast Valley Health Corporation (LA county), a previously established DULCE site.

For each HRSN, screening rates in phase two were 85.0%–93.4% at baseline and improved, with shifts at 6 months in caregiver depression (87.7% to 95.1%), IPV (90.0% to 94.8%), housing conditions (85.0% to 94.5%), housing instability (92.0% to 97.5%), food insecurity (92.7% to 97.5%), employment/financial needs (93.4% to 97.8%), and utilities (86.4% to 95.3%). Improved rates were similar with phase one screening performance: caregiver depression (95.9%), IPV (96.3%), housing conditions (shift from 94.5% to 95.8%), housing instability (97.2%), food insecurity (97.2%), employment/financial needs (98.6%), and utilities (96.8%).

In phase two, 54% of families had at least one positive HRSN screen: 20% had two and 11% had three or more. This differed from phase one, where 70% of families had at least one positive screen: 25% had two and 16% had three or more. Prevalence of individual HRSN varied as well between phase two and phase one: for food insecurity, 39.0% of phase two families vs. 46.1% of families in phase one; for employment/financial needs, 30.2% vs. 51.0%; for caregiver depression, 20% vs. 14.3%; for housing instability, 5.1% vs. 13.2%; for IPV, 4.3% vs. 5.1%; for unhealthy housing conditions, 2.1% vs. 3.5%, and for utility needs, 1.0% vs. 2.2% (Figure 8).

FIGURE 8

Figure 8. Identification and support for DULCE families’ health-related social needs (HRSN), by HRSN domain.

In phase two, 80% of families experiencing caregiver depression and/or IPV and 90% of families with concrete supports needs received resource information (PD2) (Figure 9), an increase from 70.7% of families with caregiver depression and/or IPV and 86.4% with concrete support needs at the end of phase one. In phase two, FS provided resource information to 92.1% of families with food insecurity, 55.3% with employment/financial needs, to 79.0% of depressed caregivers, 70.2% with housing instability, 61.5% of families with IPV, 94.7% with unhealthy housing conditions, and 55.6% with utility needs (Figure 8).

FIGURE 9

Figure 9. Percent of families with identified health-related social needs that received information about available resources. Star symbols denote new DUCLE sites: Brumback (Palm Beach County), The Children's Clinic [Los Angeles (LA) County], which aggregated data for its established and new DULCE clinics, and Newhall, part of the Northeast Valley Health Corporation (LA county), a previously established DULCE site.

The last process measure – the percent of families with HRSN that used resources (PD3) – responds to this study's second research question with new data that was not collected in phase one. Among families with at least one identified HRSN, 87.4% successfully used at least one related resource (Figure 8). The percent of families that used concrete supports resources increased from 87.1% to 92.9% of families (Figure 10). Resource use varied by HRSN (Figure 8): food insecurity (97.0%), employment/financial needs (58.2%), caregiver depression (54.7%), housing instability (12.8%), IPV (51.3%), housing conditions (78.9%), and utilities (44.4%).

FIGURE 10

Figure 10. Percent of families with identified health-related social needs using resources. Star symbols denote new DUCLE sites: Brumback (Palm Beach County), The Children's Clinic [Los Angeles (LA) County], which aggregated data for its established and new DULCE clinics, and Newhall, part of the Northeast Valley Health Corporation (LA county), a previously established DULCE site.

4. Discussion

This study examined the results of scaling a pediatric clinic-based EBI using a lighter-touch CQI approach. Scaling within existing healthcare and early childhood systems capitalized on existing relationships, infrastructure, and experience, allowing for lighter-touch CQI support that reduced implementation and maintenance costs. The main outcome (percent of infants who received all recommended WCVs on time) initially dropped to a similar baseline average (41.4%) seen in the first phase expansion, then demonstrated modest improvement to 48.1%, despite the lower use of preventive care during the initial phase of the COVID-19 pandemic (54).

Other process measures concerning families’ WCVs (1-month WCV timeliness, FS presence) and HRSN (rates of screening, resource discussion, and – newly – resource use) were maintained or improved. These results shed insight into the varying levels of CQI support needed to introduce or sustain different intervention components.

Many improvements from DULCE's phase one expansion spread quickly to new sites and further improved at established sites. DULCE practice change, namely FS presence in WCVs, increased from 0 to 66% over 10 months of intensive support during phase one, then reached 89.8% in the first six months of this second phase (until COVID-19 necessitated limits to in-person contact). The second phase benefitted from innovations in phase one, including providers acceptance of FS presence and CQI-driven changes in scheduling practices and clinic workflows (44). This measure's continued improvement suggests that once FSs are integrated into clinical teams, they become an essential, sustainable part of clinic function.

Similarly, HRSN screening and resource discussion processes were sustained by established sites and spread to new sites with lighter-touch CQI support. As is commonly observed in efforts to disseminate EBIs (8, 46, 52), integration of new sites at the beginning of phase two produced a dip in performance that recovered quickly – i.e., phase two's aggregate baseline screening rates were slightly lower than late phase-one levels, but reached very high rates within six months. For example, one site (The Children's Clinic–LA) took one year to improve screening for seven HRSN from 92% to 100% of families in phase one, then dropped to 50% when a second, much larger site (SMTF) was added, and recovered to 88% within six months.

Accelerated adoption by new sites and ongoing improvement of established sites reflects intentional design that included all actors in phase one, then drew on them to harvest tested implementation strategies and leverage the power of peer champions to introduce innovations (as described in the literature) (8, 14, 55, 56). In this case, two of the three new clinic sites were within the same healthcare delivery system as established sites, and clinical leadership benefitted from integrating into their “new” DULCE site teams the same legal partners and early childhood system leaders as the established sites. Such design allows for tapering the intensity of support while maintaining early adopters' outcomes and spreading improvements to additional sites (57).

However, not all phase-one improvements were sustained. The main outcome regressed to pre-phase one levels (∼41% of six-month-old infants received all WCVs on time at baseline in both phases) then improved to 48.1% but did not approach the 66% achieved during phase one. Those infants born after September 2019 may have missed visits due to pandemic-related disruptions; follow-on studies would determine whether performance improved as disruptions eased. In addition, none of the DULCE sites were able to monitor this measure during phase two, preventing data-driven CQI activities.

Most of phase one's improvement was associated with improvements in the 1-month WCV, fueled by learning that Medicaid covered this newly recommended visit (44, 58). In phase one, on-time 1-month WCVs improved from 62.5% to 79.5%. The healthcare systems spread the learning across all their sites – including the new DULCE sites – so that by the beginning of phase two, 84.3% of 1-month WCVs were on time. Like FS presence and HRSN process measures above, DULCE teams sustained and spread phase-one improvements in 1-month WCVs.

HRSN screening intends to identify and address family needs. In this second-phase study, we collected data on families’ HRSN resource use – addressing a limitation of the many studies that measure referrals but not whether families successfully use the resources (59–62). Connection rates varied by domain (13%–97%), and likely reflect clinics' relationships with different service providers and systemic barriers. Connection rates were highest for food (97%), housing conditions (79%), and employment/financial needs (58%). Over half of families with caregiver depression (55%), and IPV (51%) used resources, compared to four of nine families with utility needs (44%). Use of housing instability resources was lowest (13%), reflective of the affordable housing crisis.

These are promising results. Clinic-based HRSN interventions often struggle to link patients to resources: a CQI collaborative in 19 pediatric clinics increased HRSN screening from 19% to 73%, but did not increase HRSN referrals (63). A recent systematic review suggested that direct referrals or additional assistance with indirect referrals improved patient outcomes compared with indirect referrals only (64). Other pediatric clinic-based interventions that collected data on resource use demonstrated a wide range of connection rates (0.8%–75%), and most EBIs only address a single HRSN domain (61, 65–73).

Families with multiple HRSN often must navigate multiple programs that address single domains of need (74). Breaking down silos between healthcare, public health, legal, and early childhood systems serves families more equitably and efficiently. DULCE's FS and cross-sector team is designed to support families to navigate resources in a streamlined, cohesive manner, often utilizing warm handoffs to facilitate connections. This level of care coordination and family-led problem solving is not feasible for clinicians alone to execute; team-based, cross-sector care distributes this responsibility and strengthens relationships with families. The weekly cross-sector team meetings supported maintenance of this collaborative teamwork at each site.

Continuous quality improvement methods, including BTS, are designed to improve the original intervention and simultaneously promote sustainability in scale-up. Successful CQI efforts develop strong improvement teams that involve stakeholders, institutional leaders, frontline service providers, as well as patients; intentionally develop infrastructure and organizational capacity to support practice changes and a culture of CQI; and rely upon iterative use of data and feedback loops to inform practice changes. It flattens the hierarchy of decision-making within teams and balances power, elevating the voices of those who typically do not contribute to leadership decisions but whose perspectives are invaluable (e.g., frontline workers, families).

Continuous quality improvement also builds teams' capacities to solve their own problems and use data to identify areas of improvement. It transforms the way data is typically used (i.e., from a judgement to a learning opportunity). Furthermore, PDSAs require teams to start testing small (e.g., 1 patient interaction), empowering them to initiate rapid-cycle testing and learn from failure. While many implementation frameworks stress fidelity of implementation, CQI focuses on local adaptation informed by real-time data to retain fidelity to process and outcome measures. These cross-hierarchical, inquiry-driven tools strengthen not just the implementation of one EBI – in this case, DULCE – but these experiences build teams' capacity for learning through iterative trial-and-error, which benefits the entire clinical ecosystem, including parallel social determinants of health (SDOH) efforts and patients not enrolled in any interventions.

While offering many benefits to clinical teams and healthcare systems, CQI collaboratives such as BTS are resource intensive. Many scale-up frameworks intentionally design an early phase of expansion with intensive support to a core set of participants that represent all actors and relationships in a system. For example, McCannon et al. (46) and Barker et al. (8) both describe large-scale spread of EBIs that intentionally selected a small “wedge” or “scalable unit” — a microcosm of the entire system — to begin. By providing intensive support at a smaller scale, these efforts learn not only what needs to be adapted as the EBI spreads, but how to support teams in their adaptation (i.e., what infrastructure is needed). After this initial intensive learning phase, subsequent scale phase(s) is driven by activating induction-phase participants as peer mentors to help spread the EBI to additional “wedges.” (14, 56) In these subsequent phases, improvement is often accelerated – as it was in this DULCE expansion.

With respect to clinics' CQI-driven learnings, several themes emerged. First, identifying HRSN is not risk-free for families (75), particularly those with immigration issues during this study period when the Trump administration's public charge rule was in effect (76). FS built rapport and trust with families over their six-month enrollment, allowing families time to disclose sensitive or stigmatized needs (which typically occurred around the 4-month WCV), and were supported with accurate legal information from the legal partner at each site.

Family-centered care includes respecting families' desires around their HRSN. For example, some parents with infants tolerated overcrowded housing conditions rather than lose their proximity to extended family supports. Better understanding family agency amd priorities is a future area of research within DULCE. In such complex cases, the cross-sector team's diverse expertise generates creative problem solving that adapts solutions to families' circumstances and preferences. Cultivating a consistent, trusting relationship between the DULCE family and their entire care team makes family-led problem solving possible.

Continuous quality improvement also allows sites to develop culturally-aware approaches to some challenges. In some communities, stigma discourages parents from seeking support for postpartum depression. At two sites, the DULCE team collaborated to sidestep this stigma through careful messaging and the use of more acceptable resources. One clinic referred to a Fussy Baby clinic that included parent-child psychotherapy, and another one developed an infant massage class led by a mental health provider. Differences in local culture, resources, and team resourcefulness may have contributed to this cross-site variability.

This study contributes to the literature on CQI strategies for scaling and sustaining healthcare-based EBIs to improve population health. It also reports the results of families' rates of HRSN resource use, an outcomes-oriented indicator that is often omitted in SDOH interventions.

4.1. Limitations

The selection of volunteer sites for implementing DULCE limits generalizability. Our analyses identify improvements that are unlikely due to chance alone but lack causal inference. External events, notably the COVID-19 pandemic, may have affected some measures of reach and effectiveness. This study relied on data reported by Family Specialists for CQI purposes which did not include balancing measures; ongoing work will incorporate data about contextual factors and stakeholders' perspectives, E.H.R and claims data for participants and a comparison group (77).

5. Conclusion

An innovative, lighter-touch CQI approach to a second phase of scale-up resulted in maintenance or improvements in most processes and outcomes at four established clinics and three new clinics. Outcomes-oriented CQI measures (family HRSN resource use) are an important addition to more traditional process-oriented indicators.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by The University of Chicago School of Social Administration's Institutional Review Board (IRB17-0414). Written informed consent from the participants’ legal guardian/next of kin was not required to participate in this study in accordance with the national legislation and the institutional requirements.

Author contributions

MCA and PF conceptualized and designed the study, designed the data collection instruments, coordinated and supervised data collection, carried out the initial analyses, drafted the initial manuscript, and reviewed and revised the manuscript. BF and SM conceptualized and designed the study, designed the data collection instruments, and reviewed and revised the manuscript. SA carried out the initial analyses and reviewed and revised the manuscript. PH and JMS supported acquisition and interpretation of data and reviewed and revised the manuscript. RS conceptualized and designed the study and critically reviewed the manuscript for important intellectual content. All authors contributed to the article and approved the submitted version.

Funding

DULCE expansion was funded by The JPB Foundation through the Center for the Study of Social Policy.

Acknowledgments

We express our deep gratitude and appreciation for the DULCE families and teams.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at https://www.frontiersin.org/articles/10.3389/frhs.2023.1040992/full#supplementary-material.

References

1. Peacock-Chambers E, Ivy K, Bair-Merritt M. Primary care interventions for early childhood development: a systematic review. Pediatrics. (2017) 140(6):1–21. doi: 10.1542/peds.2017-1661

CrossRef Full Text | Google Scholar

2. Andermann A. Screening for social determinants of health in clinical care: moving from the margins to the mainstream. Public Health Rev. (2018) 39(1):19. doi: 10.1186/s40985-018-0094-7

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Hagan JF, Shaw JS, Duncan PM, Bright futures: Guidelines for health supervision of infants, children, and adolescents. Fourth edition. New York, NY: Bright Futures/American Academy of Pediatrics; 2017.

4. Parry J, Vanstone M, Grignon M, Dunn JR. Primary care-based interventions to address the financial needs of patients experiencing poverty: a scoping review of the literature. Int J Equity Health. (2021) 20(1):219. doi: 10.1186/s12939-021-01546-8

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Ben Charif A, Zomahoun HTV, LeBlanc A, Langlois L, Wolfenden L, Yoong SL, et al. Effective strategies for scaling up evidence-based practices in primary care: a systematic review. Implement Sci. (2017) 12(1):139. doi: 10.1186/s13012-017-0672-y

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Zomahoun HTV, Ben Charif A, Freitas A, Garvelink MM, Menear M, Dugas M, et al. The pitfalls of scaling up evidence-based interventions in health. Glob Health Action. (2019) 12(1):1–8. doi: 10.1080/16549716.2019.1670449

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Shonkoff JP. Rethinking the definition of evidence-based interventions to promote early childhood development. Pediatrics. (2017) 140(6):e20173136. doi: 10.1542/peds.2017-3136

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Barker PM, Reid A, Schall MW. A framework for scaling up health interventions: lessons from large-scale improvement initiatives in Africa. Implement Sci. (2016) 11(1):12. doi: 10.1186/s13012-016-0374-x

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Klein Walker D. Taking evidence-based interventions to scale in public health systems. Fairfax, VA: Society for Prevention Research (2019). https://www.preventionresearch.org/wp-content/uploads/2019/06/SPR-Public-Health-Brief_FINAL.pdf.

10. Yamey G. Scaling up global health interventions: a proposed framework for success. PLoS Med. (2011) 8(6):e1001049. doi: 10.1371/journal.pmed.1001049

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Radner JM, Ferrer MJS, McMahon D, Shankar AH, Silver KL. Practical considerations for transitioning early childhood interventions to scale: lessons from the saving brains portfolio. Ann N Y Acad Sci. (2018) 1419(1):230–48. doi: 10.1111/nyas.13684

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Côté-Boileau É, Denis JL, Callery B, Sabean M. The unpredictable journeys of spreading, sustaining and scaling healthcare innovations: a scoping review. Health Res Policy Syst. (2019) 17:1–26. doi: 10.1186/s12961-019-0482-6

CrossRef Full Text | Google Scholar

13. Yoshikawa H, Wuermli AJ, Raikes A, Kim S, Kabay SB. Toward High-Quality Early Childhood Development Programs and Policies at National Scale: Directions for Research in Global Contexts. Social Policy Report. Volume 31, Number 1. Society for Research in Child Development; 2018. Accessed July 27, 2022. https://eric.ed.gov/?id=ED595500

14. Singh K, Speizer I, Handa S, Boadu RO, Atinbire S, Barker PM, et al. Impact evaluation of a quality improvement intervention on maternal and child health outcomes in northern Ghana: early assessment of a national scale-up project. Int J Qual Health Care. (2013) 25(5):477–87. doi: 10.1093/intqhc/mzt054

PubMed Abstract | CrossRef Full Text | Google Scholar

15. McCannon CJ, Schall MW, Calkins DR, Nazem AG. Saving 100 000 lives in US hospitals. Br Med J. (2006) 332(7553):1328–30. doi: 10.1136/bmj.332.7553.1328

CrossRef Full Text | Google Scholar

16. Walsh MC, Crowley M, Wexelblatt S, Ford S, Kuhnell P, Kaplan HC, et al. Ohio Perinatal quality collaborative improves care of neonatal narcotic abstinence syndrome. Pediatrics. (2018) 141(4):1–10. doi: 10.1542/peds.2017-0900

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Kellams A, Parker MG, Geller NL, Moon RY, Colson ER, Drake E, et al. Todaysbaby quality improvement: safe sleep teaching and role modeling in 8 US maternity units. Pediatrics. (2017) 140(5):1–9. doi: 10.1542/peds.2017-1816

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Barker P, Barron P, Bhardwaj S, Pillay Y. The role of quality improvement in achieving effective large-scale prevention of mother-to-child transmission of HIV in South Africa. AIDS Lond Engl. (2015) 29(Suppl 2):S137–143. doi: 10.1097/QAD.0000000000000718

CrossRef Full Text | Google Scholar

19. Kritchevsky SB, Simmons BP. Continuous quality improvement: concepts and applications for physician care. JAMA. (1991) 266(13):1817–23. doi: 10.1001/jama.1991.03470130097036

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Riley WJ, Moran JW, Corso LC, Beitsch LM, Bialek R, Cofsky A. Defining quality improvement in public health. J Public Health Manag Pract. (2010) 16(1):5. doi: 10.1097/PHH.0b013e3181bedb49

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Paina L, Peters DH. Understanding pathways for scaling up health services through the lens of complex adaptive systems. Health Policy Plan. (2012) 27(5):365–73. doi: 10.1093/heapol/czr054

PubMed Abstract | CrossRef Full Text | Google Scholar

22. The Breakthrough Series. IHI's collaborative model for achieving breakthrough improvement. Boston, MA: Institute for Healthcare Improvement (2003). http://www.ihi.org/resources/Pages/IHIWhitePapers/TheBreakthroughSeriesIHIsCollaborativeModelforAchievingBreakthroughImprovement.aspx.

23. Deming WE. Out of the crisis. Cambridge, MA: Massachusetts Institute of Technology, Center for Advanced Engineering Study (1986).

24. Bisgaard S. Quality management and Juran's Legacy. Qual Reliab Eng Int. (2007) 23(6):665–77. doi: 10.1002/qre.860

CrossRef Full Text | Google Scholar

25. Øvretveit J. Public service quality improvement. In: Ferlie E, Lynn LE, Pollitt C, editors. The Oxford handbook of public management. Oxford, England: Oxford University Press (2009). p. 537–62. doi: 10.1093/oxfordhb/9780199226443.003.0024

26. Bryk AS. Improvement research carried out through networked communities: accelerating learning about practices that support more productive student mindsets. Stanford, CA: The Carnegie Foundation (2013). 32. Accessed December 22, 2022. https://www.carnegiefoundation.org/wp-content/uploads/2014/09/improvement_research_NICs_bryk-yeager.pdf.

27. Dilley JA, Bekemeier B, Harris JR. Quality improvement interventions in public health systems: a systematic review. Am J Prev Med. (2012) 42(5, Supplement 1):S58–71. doi: 10.1016/j.amepre.2012.01.022

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Nicolay CR, Purkayastha S, Greenhalgh A, Benn J, Chaturvedi S, Phillips N, et al. Systematic review of the application of quality improvement methodologies from the manufacturing industry to surgical healthcare. Br J Surg. (2012) 99(3):324–35. doi: 10.1002/bjs.7803

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Joly BM, Booth M, Shaler G, Conway A. Quality improvement learning collaboratives in public health: findings from a multisite case study. J Public Health Manag Pract. (2012) 18(1):87. doi: 10.1097/PHH.0b013e3182367db1

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Livingood WC, Sabbagh R, Spitzfaden S, Hicks A, Wells L, Puigdomenech S, et al. A quality improvement evaluation case study: impact on public health outcomes and agency culture. Am J Prev Med. (2013) 44(5):445–52. doi: 10.1016/j.amepre.2013.01.011

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Douglass A, Chickerella R, Maroney M. Becoming trauma-informed: a case study of early educator professional development and organizational change. J Early Child Teach Educ. (2021) 42(2):182–202. doi: 10.1080/10901027.2021.1918296

CrossRef Full Text | Google Scholar

32. Murray A, Campfield T, Dougherty S, Sweet K. Breakthrough Series Collaborative: timely Permanency through Reuinficiation. Casey Family Programs. (2011) 128:1–128, Accessed December 22, 2022. https://www.casey.org/media/TimelyPermanency.pdf.

Google Scholar

33. Tandon D, Mackrain M, Beeber L, Topping-Tailby N, Raska M, Arbour M. Addressing maternal depression in home visiting: findings from the home visiting collaborative improvement and innovation network. PLOS ONE. (2020) 15(4):e0230211. doi: 10.1371/journal.pone.0230211

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Arbour M, Mackrain M, Fitzgerald E, Atwood S. National quality improvement initiative in home visiting services improves breastfeeding initiation and duration. Acad Pediatr. (2019) 19(2):236–44. doi: 10.1016/j.acap.2018.11.005

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Arbour M, Mackrain M, Cano C, Atwood S, Dworkin P. National home visiting collaborative improves developmental risk detection and service linkage. Acad Pediatr. (2021) 21(5):809–17. doi: 10.1016/j.acap.2020.08.020

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Institute for Healthcare Improvement. The breakthrough series: iHI's Collaborative model for achieving breakthrough improvement. Diabetes Spectr. (2004) 17(2):97–101. doi: 10.2337/diaspect.17.2.97

CrossRef Full Text | Google Scholar

37. Bryk AS, Gomez LM, Grunow A. Getting ideas into action: building networked improvement communities in education. In: Hallinan MT, editors. Frontiers in sociology of education. Frontiers in sociology and social research. Dordrecht, Netherlands: Springer Netherlands (2011). p. 127–62. doi: 10.1007/978-94-007-1576-9_7

38. LeMahieu PG, Grunow A, Baker L, Nordstrum LE, Gomez LM. Networked improvement communities: the discipline of improvement science meets the power of networks. Qual Assur Educ Bradf. (2017) 25(1):5–25. doi: 10.1108/QAE-12-2016-0084

CrossRef Full Text | Google Scholar

39. Proger AR, Bhatt MP, Cirks V, Gurke D. Establishing and sustaining networked improvement communities: lessons from Michigan and Minnesota. Washington, D.C.: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Midwest (2017). 28.

40. Arbour M, Yoshikawa H, Atwood S, Duran FR, Godoy F, Trevino E, et al. Quasi-Experimental study of a learning collaborative to improve public preschool quality and Children's Language outcomes in Chile. BMJ Qual Saf. (2015) 24(11):727. doi: 10.1136/bmjqs-2015-IHIabstracts.11

CrossRef Full Text | Google Scholar

41. Tyler J, Davies M, Bennett B. Increasing early childhood education enrolment and attendance rates in south auckland, New Zealand. N Z Int Res Early Child Educ. (2018) 21(1):99–111. doi: 10.3316/informit.263276641913828

CrossRef Full Text | Google Scholar

42. Arbour M, Soto C, Aleé Y, Atwood S, Muñoz P, Marzolo M. Absenteeism prevention in preschools in Chile: impact from a quasi-experimental evaluation of 2011-2017 ministry of education data. Front Educ. (2022) 7:1–22. doi: 10.3389/feduc.2022.975092

CrossRef Full Text | Google Scholar

43. Sege R, Preer G, Morton SJ, Cabral H, Morakinyo O, Lee V, et al. Medical-Legal strategies to improve infant health care: a randomized trial. Pediatrics. (2015) 136(1):97–106. doi: 10.1542/peds.2014-2955

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Arbour MC, Floyd B, Morton S, Hampton P, Sims JM, Doyle S, et al. Cross-Sector approach expands screening and addresses health-related social needs in primary care. Pediatrics. (2021) 148(5):e2021050152. doi: 10.1542/peds.2021-050152

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Hodge LM, Turner KMT. Sustained implementation of evidence-based programs in disadvantaged communities: a conceptual framework of supporting factors. Am J Community Psychol. (2016) 58(1-2):192–210. doi: 10.1002/ajcp.12082

PubMed Abstract | CrossRef Full Text | Google Scholar

46. McCannon J, Massoud MR, Zier Alyesh A. Many ways to many: a brief compendium of networked learning methods. Stanf Soc Innov Rev:xx, 1–7. doi: 10.48558/ybqf-d263

Google Scholar

47. Nugent JK, Keefer CH, Minear S, Johnson LC, Blanchard Y. Understanding newborn behavior and early relationships: the newborn behavioral observations (NBO) system handbook. Baltimore, Maryland: Paul H Brookes Publishing (2007).

48. DULCE Family Specialist Profile. Los Angeles County, CA. Center for the Study of Social Policy. Accessed September 26, 2019. https://cssp.org/resource/dulce-family-specialist-los-angeles-county/.

49. Minkovitz CS, Hughart N, Strobino D, Scharfstein D, Grason H, Hou W, et al. A practice-based intervention to enhance quality of care in the first 3 years of life: the healthy steps for young children program. JAMA. (2003) 290(23):3081–91. doi: 10.1001/jama.290.23.3081

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Schmidtlein MC, Deutsch RC, Piegorsch WW, Cutter SL. A sensitivity analysis of the social vulnerability Index. Risk Anal. (2008) 28(4):1099–114. doi: 10.1111/j.1539-6924.2008.01072.x

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Provost LP. Analytical studies: a framework for quality improvement design and analysis. BMJ Qual Saf. (2011) 20(Suppl_1):i92–6. doi: 10.1136/bmjqs.2011.051557

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Provost LP, Murray S. The health care data guide: learning from data improvement. Hoboken, New Jersey: John Wiley & Sons (2011).

53. Perla RJ, Provost LP, Murray SK. The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ Qual Saf. (2011) 20(1):46–51. doi: 10.1136/bmjqs.2009.037895

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Schweiberger K, Patel SY, Mehrotra A, Ray KN. Trends in pediatric primary care visits during the coronavirus disease of 2019 pandemic. Acad Pediatr. (2021) 21(8):1426–33. doi: 10.1016/j.acap.2021.04.031

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol. (2006) 6(1):54. doi: 10.1186/1471-2288-6-54

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Institute for Healthcare Improvement. Lessons Learned from Ghana's Project Fives Alive! A Practical Guide for Designing and Executing Large-Scale Improvement Initiatives. Accessed September 26, 2019. http://www.ihi.org:80/resources/Pages/Publications/ProjectFivesAliveLessonsLearnedGuide.aspx.

57. Giovino JM. Holding the gains in quality improvement. Fam Pract Manag. (1999) 6(5):29–32. PMID: 10537794 10537794

PubMed Abstract | Google Scholar

58. Periodicity Schedules. California Department of Health Care Services. Accessed December 3, 2020. https://www.dhcs.ca.gov:443/services/chdp/Pages/Periodicity.aspx.

59. Hays-Grudo J, Slocum R, Root JD, Bosler C, Morris AS. Tulsa Children's Project: applying evidence-based interventions in early childhood settings. In: Morris AS, Williamson AC, editors. Building early social and emotional relationships with infants and toddlers: integrating research and practice. New York, NY: Springer International Publishing (2018). p. 277–303. doi: 10.1007/978-3-030-03110-7_12

60. Henize AW, Beck AF, Klein MD, Morehous J, Kahn RS. Transformation of a pediatric primary care waiting room: creating a bridge to community resources. Matern Child Health J. (2018) 22(6):779–85. doi: 10.1007/s10995-018-2508-z

PubMed Abstract | CrossRef Full Text | Google Scholar

61. Knowles M, Khan S, Palakshappa D, Cahill R, Kruger E, Poserina BG, et al. Successes, challenges, and considerations for integrating referral into food insecurity screening in pediatric settings. J Health Care Poor Underserved. (2018) 29(1):181–91. doi: 10.1353/hpu.2018.0012

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Zigler E, Pfannenstiel JC, Seitz V. The parents as teachers program and school success: a replication and extension. J Prim Prev. (2008) 29(2):103–20. doi: 10.1007/s10935-008-0132-1

PubMed Abstract | CrossRef Full Text | Google Scholar

63. Flower KB, Massie S, Janies K, Bassewitz JB, Coker TR, Gillespie RJ, et al. Increasing early childhood screening in primary care through a quality improvement collaborative. Pediatrics. (2020) 146(3):1–11. doi: 10.1542/peds.2019-2328

CrossRef Full Text | Google Scholar

64. Ruiz Escobar E, Pathak S, Blanchard CM. Screening and referral care delivery services and unmet health-related social needs: a systematic review. Prev Chronic Dis. (2021) 18:E78. doi: 10.5888/pcd18.200569

PubMed Abstract | CrossRef Full Text | Google Scholar

65. Garg A, Toy S, Tripodis Y, Silverstein M, Freeman E. Addressing social determinants of health at well child care visits: a cluster RCT. Pediatrics. (2015) 135(2):2014–888. doi: 10.1542/peds.2014-2888

CrossRef Full Text | Google Scholar

66. Beck AF, Klein MD, Schaffzin JK, Tallent V, Gillam M, Kahn RS. Identifying and treating a substandard housing cluster using a medical-legal partnership. Pediatrics. (2012) 130(5):831–8. doi: 10.1542/peds.2012-0769

PubMed Abstract | CrossRef Full Text | Google Scholar

67. Fleegler EW, Lieu TA, Wise PH, Muret-Wagstaff S. Families’ health-related social problems and missed referral opportunities. Pediatrics. (2007) 119(6):e1332–1341. doi: 10.1542/peds.2006-1505

PubMed Abstract | CrossRef Full Text | Google Scholar

68. Garg A, Marino M, Vikani AR, Solomon BS. Addressing Families’ unmet social needs within pediatric primary care: the health leads model. Clin Pediatr (Phila). (2012) 51(12):1191–3. doi: 10.1177/0009922812437930

PubMed Abstract | CrossRef Full Text | Google Scholar

69. Uwemedimo OT, May H. Disparities in utilization of social determinants of health referrals among children in immigrant families. Front Pediatr. (2018) 6:207. doi: 10.3389/fped.2018.00207

PubMed Abstract | CrossRef Full Text | Google Scholar

70. Polk S, Leifheit KM, Thornton R, Solomon BS, DeCamp LR. Addressing the social needs of spanish- and English-speaking families in pediatric primary care. Acad Pediatr. (2020) 20(8):1170–6. doi: 10.1016/j.acap.2020.03.004

PubMed Abstract | CrossRef Full Text | Google Scholar

71. Fiori KP, Rehm CD, Sanderson D, Braganza S, Parsons A, Chodon T, et al. Integrating social needs screening and community health workers in primary care: the community linkage to care program. Clin Pediatr (Phila). (2020) 59(6):547–56. doi: 10.1177/0009922820908589

PubMed Abstract | CrossRef Full Text | Google Scholar

72. Power-Hays A, Li S, Mensah A, Sobota A. Universal screening for social determinants of health in pediatric sickle cell disease: a quality-improvement initiative. Pediatr Blood Cancer. (2020) 67(1):e28006. doi: 10.1002/pbc.28006

PubMed Abstract | CrossRef Full Text | Google Scholar

73. Stenmark SH, Steiner JF, Marpadga S, Debor M, Underhill K, Seligman H. Lessons learned from implementation of the food insecurity screening and referral program at kaiser permanente Colorado. Perm J. (2018) 22:18–093. doi: 10.7812/TPP/18-093

PubMed Abstract | CrossRef Full Text | Google Scholar

74. Inkelas M, Bowie P, Guirguis L. Improvement for a community population: the Magnolia community initiative. New Dir Eval. (2017) 2017(153):51–64. doi: 10.1002/ev.20229

CrossRef Full Text | Google Scholar

75. Garg A, Boynton-Jarrett R, Dworkin PH. Avoiding the unintended consequences of screening for social determinants of health. JAMA. (2016) 316(8):813–4. doi: 10.1001/jama.2016.9282

PubMed Abstract | CrossRef Full Text | Google Scholar

76. The Trump Administration's New Public Charge Rule. Implications For Health Care & Public Health | Health Affairs. Accessed December 3, 2020. https://www.healthaffairs.org/do/10.1377/hblog20190813.84831/full/.

77. McCrae JS, Spain A, Byers K, Sander A, Axelrod J. Evaluating community approaches to preventing or mitigating toxic stress (research brief 1). Chicago, IL: Chapin Hall at the University of Chicago (2019. Accessed July 31, 2022. https://www.chapinhall.org/wp-content/uploads/Study-Overview-Brief-Final.pdf.