Design of the Focus on Restaurant Engagement to Strengthen Health (FRESH) study: leveraging systems science to work with independently-owned restaurants to increase access to and promotion of healthful foods

Background: High dietary quality can protect against diet-related chronic diseases. In the United States, racial and ethnic minorities and those with lower incomes consistently exhibit lower dietary quality. Independently-owned restaurants are a common prepared food source in minority low-income communities, but there are significant knowledge gaps on how to work with these restaurants to offer healthy food, due to underlying and dynamic complexities associated with providing healthy food options.

Methods: The Focus on Restaurant Engagement to Strengthen Health (FRESH) study addresses this complex problem by leveraging systems science approaches to work with independently-owned restaurants. FRESH has two interrelated objectives: (1) to test impact on regular customer dietary quality via a multisite cluster randomized controlled trial in two low-income urban areas (Baltimore and the Washington DC metropolitan area), and (2) to use systems science approaches to develop, parameterize, and calibrate a simulation model. The intervention is theory-and practice-based, comprising three phases: restaurant engagement, low-sugar beverages and healthy meals. The FRESH intervention will be implemented for 12 months in a total of 24 intervention and 24 comparison restaurants. The study is powered to detect a 5-point change in the Healthy Eating Index (HEI) score of regular customers, which would signify a meaningful shift toward healthier eating patterns.

Discussion: The FRESH study will test a novel, multilevel, multisite intervention that aims to improve access to healthier prepared food options among small, independently-owned restaurants located in under-resourced settings. The design of the FRESH intervention and its evaluation are described, as well as plans for the development of a system dynamics simulation model for policymakers and other stakeholders to virtually test future restaurant-based interventions.

Clinical trial registration: https://clinicaltrials.gov, identifier, NCT05869149.

Background

High dietary quality is a known protective factor against diet-related chronic diseases and all-cause mortality. For example, people with the highest dietary quality exhibit a 23% decreased risk of all-cause mortality compared to those with the lowest dietary quality (1). In the United States, non-Hispanic Black, Latinx, and other populations marginalized by inadequate policies bear an inequitable burden of chronic disease morbidity and mortality related to diet quality (2–4). Previous work to improve dietary quality sustainably has mainly focused on increasing access to healthful foods in retail store settings (5, 6). However, Americans spend about half of their food budget eating out (7, 8), representing at least a 15-fold inflation-adjusted increase since 1970 (9). This trend has contributed an additional 570 kcals to average daily energy intake, owing to both the increased frequency of eating out and larger portion sizes (10). Of the food consumed away from home, 76.8% is purchased at restaurants and fast-food establishments (11). Low-income and minority communities in urban settings are highly exposed to small, independently-owned prepared food sources in particular, most of which predominantly offer energy-dense, nutrient-poor options with few healthier options (12–14). These population groups spend a larger percentage of their food budget at these establishments (12), which are also not subject to policy efforts, such as menu-labeling (15). Working with independently-owned restaurants offers a unique venue to reach customers in low-income minority urban neighborhoods who may benefit most from improved dietary quality. Despite their prevalence in these under-resourced communities, little effort has been directed toward enhancing dietary quality in these establishments (11).

Improving access to and promotion of healthful foods in restaurants presents an inherently multilevel and potentially contradictory problem. Engagement in such efforts is affected by owners’ perceptions of customer demand for these foods, concerns with impact on revenues (e.g., ‘will offering healthy foods to my customers negatively affect sales?’), views concerning community health, and food sourcing challenges (16, 17). Our previous work (18–24) has demonstrated that it is feasible to promote existing healthful food options within independently-owned restaurants, improve their food preparation methods (5), and increase knowledge and acceptability of promoted healthful foods among customers (18, 21). However, community-based interventions tend to be resource-intensive and unsustainable if they do not consider the dynamic complexity across multiple levels of the food system.

The Focus on Restaurant Engagement to Strengthen Health (FRESH) study leverages systems science approaches to generate solutions to improve the food environment (25). Systems science addresses complex problems by understanding, identifying, and modeling multiple interacting factors—permitting the virtual testing of potential effects on dietary quality outcomes (26, 27), and saving resources that may otherwise be expended in trial and error (28). As an integral part of FRESH, systems science will inform the development of the trial, and resulting data from the trial will be used to parameterize and calibrate a system dynamics model that can predict the effects of similar strategies in other settings.

Therefore, this manuscript presents the protocol for the two inter-related objectives of this novel study: (1) to refine, implement and test—via a multisite randomized, controlled trial—the impact of the FRESH intervention in independently-owned restaurants located in two largely low-income minority urban sites, and (2) to develop, parameterize, and calibrate a simulation model using data generated from the trial.

Methods and design

The FRESH study is innovative in its combination of community-based interventions and systems science—the intervention trial will inform the systems science approaches used (Figure 1). As depicted in the Figure, the two main study objectives will be achieved across four phases. First, comprehensive formative work, including community-based systems science group model building workshops was and will continue to be conducted to refine intervention strategies and develop an initial causal loop diagram. The impact of FRESH will be evaluated via multisite group-randomized controlled trial (RCT) design in two sites—Baltimore, Maryland, and the greater Washington DC metropolitan (DC metro) area—where neighborhoods will be randomly assigned to intervention or comparison conditions. Second, baseline data collected on neighborhoods, restaurants, and their customers will be used to parameterize the initial simulation model and guide the implementation of the intervention. Third, process and impact data from the implementation of FRESH will be used to further calibrate the simulation model. Finally, an interactive, web-based, user-friendly dashboard (29) will be disseminated to policymakers allowing them to apply the system dynamics model to other urban settings.

Figure 1

Figure 1. Integration of FRESH intervention study phases with the systems dynamic model.

Formative and participatory research approaches

The FRESH study leans on academic-community expertise and partnerships with particular experience working with family-style, independent, Latinx-owned restaurants in the DC metro area and in Baltimore (18). This includes leveraging active Community Advisory Boards (CABs) per each site, and using community data collectors and interventionists who are knowledgeable and have deep lived experiences in the communities and neighborhoods in question.

In addition, the FRESH study utilizes extensive formative work to contextualize the trial’s intervention strategies for each site based on qualitative data from: (1) in-depth interviews with restaurant owners/managers and restaurant suppliers (n = 31), and (2) structured back-and front-of-house (n = 27) and back-of-house observations (n = 17).

In-depth interviews with restaurant owners/managers

Semi-structured interviews were conducted with the owners/managers of independently-owned restaurants located within predominantly low-income neighborhoods selected at each site. The interview guide was designed to deepen and contextualize our understanding of the restaurants’ and owners’ business history (e.g., length of time in business), business practices (e.g., processes related to suppliers, procurement, delivery, cooking settings, level of training of cooks and willingness to innovate, client-facing staff schedule, restaurant layout), reasons for offering the foods and beverages they do, perceptions of their customers’ reactions toward potential menu changes, and types of promotions they have tested and found to be more or less /successful with their customers.

Structured back-of-house observations

Direct observations were carried out in the kitchen or ‘back-of-house’ area (e.g., common areas where foods are prepared, and customers are not allowed). The observation protocol for the back-of-house was designed by the research team for the purpose of this intervention and will capture the number of kitchen staff, cooking equipment, cooking storage, cooking tools, cooking practices as observed (e.g., recipes, types of oil) and availability of specific food groups (e.g., whole grains, fruits without added sugars, vegetables).

Structured front-of-house observations

For the ‘front-of-house’, or customer area, observations, we used the NEMS-R (Nutrition Environment Measures Survey-Restaurants) tool (30). This instrument assesses facilitators and barriers to healthy eating, including the restaurants’ promotion of healthful foods that are available on the menu as well as nutrition information, portion options, and other promotional materials.

Group model-building workshops

Following completion of the formative research, we will conduct a series of group model-building workshops. Group model-building is a participatory approach that elicits participants’ insights into how the restaurants’ systems currently work (including supply procurement, food preparation, service, and interactions with customers), identifying feedback loops to maximize demand, profitability, and healthfulness (31, 32). The first step of the group model building process will be to revise and expand the original causal loop diagram (Figure 2) based on our research team’s experience conducting the formative work—and to produce a revised version (version 2). We will then hold two group model building workshops, one in each research site, where participants will further refine the causal loop diagram (versions 3 and 4). There will be one causal loop diagram for both research sites.

Figure 2

Figure 2. Early causal loop diagram of the restaurant food environment.

Group model building participants will engage in systems-thinking discussions about key factors that contribute to healthy food availability in restaurants, including the role of food access and procurement, food preparation, and the customer nutrition environment (e.g., the three core components of the FRESH intervention). They will add to and expand the existing CLD based on the formative research, CAB input and the experience of the FRESH team as we begin baseline data collection with restaurants and their owners.

Site selection, recruitment and baseline data collection

Site selection

The study sites are two urban areas: Baltimore, Maryland, and the greater Washington DC metropolitan (DC metro) area. The latter comprises the adjacent urban areas of the District of Columbia, Prince George’s, and Montgomery Counties in Maryland. In total, both sites contain hundreds of restaurants in densely populated areas, including carryout-type restaurants, restaurants with seating/dining areas, and carryouts located in food markets (11).

In Baltimore, neighborhoods (as defined by clustered census tracts) will be selected if they are predominantly (>80%) African American and low-income (with >15% of the population below the poverty level). There are about 1.5 small, prepared food sources per 1,000 residents within city limits, and the majority are independently-owned, carryout-type restaurants (80%) in low-income neighborhoods (11, 33). These restaurants emphasize deep-fried and high-fat foods on their menus, including fried chicken, French fries, pizza, and submarine-style sandwiches.

In the DC metro area, neighborhoods will be selected from clustered census tracts that are both low-income (with >15% of the population below the poverty level) and have a significant Latino population (>35%). These tracts often house smaller neighborhoods that are predominantly (>80%) Latinx, mostly from El Salvador and Guatemala. There are 1.24 restaurants per 1,000 residents in the DC metro area, and many offer primarily Central American foods. These restaurants offer a combination of traditional foods from the owners’ home country, as well as newer dishes, high in saturated fat and sugary beverages.

Inclusion and exclusion criteria

Table 1 presents criteria for inclusion in and exclusion from the FRESH trial.

Table 1

Table 1. Inclusion and exclusion criteria for the FRESH trial, by level.

Randomization of restaurants

Following completion of the baseline data collection, the 16 census tract clusters will be randomized to intervention or comparison, using a public community-engaged approach we have previously employed in a Native American community setting (34). Half of the restaurants (n = 24) will therefore receive the FRESH intervention. At the customer level, half of the customer sample (n = 288) will be regular consumers at restaurants receiving the FRESH intervention. The sampling design is presented in Figure 3.

Figure 3

Figure 3. Study design and sample structure of the multi-level FRESH study.

Development of the FRESH intervention

The FRESH Intervention Working Group will develop the FRESH intervention based on our previous experiences, the results of our formative research, and the input of the FRESH Community Advisory Boards (CABs). Two CABs will be formed, one for Baltimore and one for the DC metro area. CABs will help us stay grounded in community insights by helping us engage with individuals and organizations to maximize the impact of our project, increase project sustainability, and broaden the acceptance of our efforts with communities. Each CAB will be comprised largely of community members, particularly neighborhood association members and restaurant owners, as well as policymakers and food advocacy stakeholders. Members of the CABs will attend CAB meetings to provide feedback on: (1) eligibility of study neighborhoods, (2) identification of restaurants eligible for intervention in neighborhoods, (3) engagement and recruitment strategies, (4) design of data collection tools, (5) design of intervention strategies and materials, (6) how to manage implementation challenges, (7) the group model-building and simulation modeling components, and (8) how best to disseminate and support community initiatives that help improve project sustainability. CAB meetings will be ongoing for the duration of the project. CAB members will determine the frequency of meeting times based on the need to share study milestones and challenges, and when the study team would most benefit from CAB feedback. FRESH CAB members receive a $100 honorarium for their time sharing their input and insights.

The FRESH intervention

The FRESH intervention will be delivered by three to four trained interventionists over a 12-month period in three phases, each lasting approximately 4 months (Table 2): Engagement, Lower sugar beverages and Healthier meals. Each phase will include activities at the restaurant, consumer and supplier levels. Interventionists will share lived experiences similar to that of individuals working in the independently-owned restaurant businesses (e.g., having owned a restaurant before; working in sales; having a food/nutrition background or interest in restaurants) to prioritize the interventionists’ ability to build rapport with restaurant owners, which will be crucial to obtaining restaurant owner’s buy-in and follow-through on the intervention.

Table 2

Table 2. Overview of FRESH intervention strategies.

Phase 1 (engagement)

This phase is intended to build rapport and trust between restaurant owners, their workers and FRESH team members (FRESH Partners). In a series of planning meetings, FRESH interventionists will meet with each restaurant owner and collectively decide on the best strategies for implementing the FRESH intervention that will be uniquely suited to that restaurant. Discussions will center around identifying three new low-sugar beverages, and three new healthier meals to be offered by the restaurant. These new items will be in addition to any existing healthy choices. A team of Registered Dietitian Nutritionists (RDNs) and graduate-level dietetic students will complete a menu analysis for each restaurant. Menu analysts will work closely with interventionists to identify potential strategies for developing healthier menu items tailored to each restaurant. Specific strategies will be selected on a case-by-case basis by the FRESH Partners and may include collaboratively introducing new recipes or replacing, reducing, or rewriting existing recipes and adjusting cooking techniques toward healthier preparation methods (e.g., baking instead of deep frying). Changes to portion size will also be considered. Based on our formative research, restaurants already have experience using healthier preparation methods, even if these methods are not used for most menu items. Therefore, it was determined that providing restaurant staff with training on healthful cooking methods was unnecessary for the intervention. At the same time, the FRESH Partners will collectively decide on how to revise the restaurant’s menus to highlight and promote these new items (and other healthier items). This may involve professional production of a new menu board, preparation of laminated menus, and/or creation of menu QR code. Discussions during this phase will also cover selection of appropriate digital and print media to promote the restaurant in general. Possible digital media will include a series of social media posts (e.g., on the FRESH Instagram account, by a local influencer), development of a restaurant website (if they do not have one), or updating of their current website. Finally, a small initial subsidy ($500) will be provided to each restaurant to support the purchase of beverages/meal items to consider for promotion, from their local supplier. All costs will be covered by the FRESH project.

Phase 2 (reduced/no sugar beverages)

This phase will emphasize the stocking and promotion of three healthier beverages, that are low in added sugar (<5 g/serving) and priced at or below the cost of comparable beverages. Digital and print media will be used to promote beverages identified in Phase 1 to restaurant customers. Intervention restaurants will receive incentives in the form of supplier credit ($750/restaurant) to help offset the risk of stocking these beverages during the promotional period. This incentive will be received after the beverages have been stocked. Lists of targeted beverages will be provided to suppliers to ensure there are sufficient stocks.

Phase 3 (healthy meals)

This phase will emphasize the restaurant offering three new healthier meal options, that optimize dietary quality and taste following the DGA (35). A healthier meal is defined as 1 lean protein; AND 1 non-starchy vegetable; AND 1 vegetable (starchy or non-starchy) OR Fruit OR whole grain OR legumes. All components must be non-fried. As with beverages, entrees meeting FRESH criteria should be priced at or below the cost of comparable items. Existing menu items may be modified to meet FRESH criteria by simply adding a new side dish to an existing entrée. Specifically, a restaurant might already sell baked chicken that comes with two side dishes. If the restaurant already offers one side dish meeting FRESH criteria (e.g., collard greens), then the restaurant could develop one additional side dish, such as black-eyed peas, and create a menu item consisting of baked chicken, collard greens, and black-eyed peas to meet FRESH criteria. Digital and print media will be used to promote the three new healthy meals identified in Phase 1 to restaurant customers. Intervention restaurants will receive incentives in the form of supplier credit ($1,500/restaurant) to help offset the risk of providing these meals during the promotional period. This incentive will be received after the three new meals have begun to be sold by the restaurant.

In addition to supplier incentives described above, FRESH intervention staff will support restaurant owners/managers in sourcing the appropriate ingredients for proper FRESH implementation and will work directly with restaurant suppliers to allow them to anticipate stocking needs, a strategy we have used successfully in the past (36, 37). Lists of targeted meal ingredients will be provided to suppliers to ensure there are sufficient stocks.

Achieving sufficient intensity during implementation

A key challenge will be to ensure that the FRESH intervention is implemented with sufficient intensity to lead to changes in customer dietary quality. Two strategies will be used to ensure this outcome: (1) all participating restaurants will be required to meet specific minimal standards (e.g., the introduction of at least three low-or no-calorie beverages, and at least three healthy meals); and (2) the FRESH interventionists will meet with owners/managers at least once every 1–2 months to review progress and revise strategies as needed.

Process evaluation

We will assess reach, dose delivered, and fidelity (38) of implementation of the FRESH intervention (36). Process evaluation forms will be filled out on REDCap (Research Electronic Data Capture) (39, 40) periodically (e.g., weekly and monthly, depending on the phase of intervention implementation) by interventionists during their regular visits to the restaurants via pre-designed checklist/observation forms and activity logs. Data will be used to calculate indicators for fidelity (e.g., the percent of owners who agree or strongly agree with the beverage stocking option, the percent of items actually stocked), reach (e.g., the percent of customers who recall seeing a promotion, the percent of cook staff trained), and dose delivered (e.g., the number of promotions delivered, the number of training sessions delivered). For each FRESH intervention component, we will divide the resulting process data into tertiles according to low, moderate, or high implementation of each component. We will aim to achieve 100% for all process measure indicators while recognizing that high standards will vary somewhat between restaurants and possibly neighborhoods, and this variation will reflect recommendations for the system dynamics model. Process data for each component of FRESH will eventually be used to calibrate the system dynamics model to guide the intensity of the delivery of future FRESH interventions via machine learning that will be able to assess the ultimate impact of the intervention on final outcomes based on implementation intensity (Phase 4).

Impact assessment

Our primary study hypothesis is that regular customers of participating restaurants in FRESH intervention neighborhoods will demonstrate at least a five-point increase in dietary quality as assessed by the Healthy Eating Index 2015 (HEI) score, compared to restaurant customers living in FRESH comparison neighborhoods. We will assess this primary outcome in our consumer sample, as well as a variety of additional secondary outcome measures at the consumer, restaurant and supplier levels. All recruitment materials, consent forms and data collection instruments will be in English and Spanish. The language used will depend on the preferences of the respondent.

Selection and training of the evaluation team

Each site will staff a measurement and evaluation team composed of three to four members, separate from the intervention team. The evaluation team will be trained over a three to five-day period prior to collecting survey and anthropometric data to achieve acceptable (>0.90) interrater reliability. Data collectors will be blinded to the treatment group to mitigate risk of bias.

Recruitment of the restaurant sample

Forty-eight restaurants will be recruited at baseline according to the specified inclusion criteria. The baseline data collection team will approach the owners/managers of eligible restaurants at their business using a recruitment script that describes the intervention and presents the eligibility criteria.

Recruitment of the customer evaluation sample

In each sampled restaurant at baseline, all present customers will be approached by the evaluation team as they exit the restaurant, and, if willing, will be screened for eligibility. The evaluation team will set up a separate time to consent potential customer participants and to collect the survey and anthropometric data at a location convenient for the customer as we have done in other studies (20, 41). The evaluation team will continue to recruit and consent participants on a rolling basis until we have complete data from 12 customers for each participating restaurant (n = 576). The names and contact information of all consented participants will be recorded in order to follow them prospectively to conduct the same measures at the end of the intervention (12 months prospectively).

Potential for restaurant and customer drop out

It is important to recognize the possibility of low engagement and inadequate participation from restaurants owners, staff, and managers, increasing our risk for low recruitment and restaurant drop out as well as customer drop out. We lean on the experience of our team in working with restaurants and intervention trials in the past and we will employ participatory research approaches (e.g., establishing a CAB per site) as well as relying on support and commitment from community organizations, restaurant suppliers, and Departments of Health to support successful engagement of and transparent communication with independently-owned restaurants. In addition, we have allowed for loss of one restaurant in each neighborhood in our power calculation by having a power of greater than 80% if there are two restaurants. Similarly, anticipating customer drop out, we have oversampled our restaurant customer sample by 20% given experiences in previous trials.

Instruments and measures

At baseline, we will collect socio-demographic data and indicators of health status for customers (e.g., age, gender, self-reported health history, education level, socioeconomic status, and household composition), and acculturation (DC metro area only). We will also collect characteristics of restaurants at baseline (e.g., ethnicity of owner, cuisine served). See Table 3.

Table 3

Table 3. Overview of FRESH measures.

Customer-level impact

The primary outcome of dietary quality will be assessed via reported dietary intake from the past 24 h using the ASA24-Hour Dietary Assessment Tool (ASA24). Customers will complete the ASA24 twice at pre-intervention, two weeks apart and this protocol will occur again at post-intervention. The overall HEI score will be calculated according to the methods outlined by the National Cancer Institute (42). All impact measures will be assessed pre-and post-intervention to calculate a change score.

Secondary outcomes at the customer level will be assessed by the Veggie Meter®. This instrument noninvasively measures carotenoid status using a laser for blue light excitation of carotenoids in the skin (43) and has proven validity (44–46). Anthropometric measures will include body weight to the nearest 0.1 kg measured using the Tanita DC-13C scale and height measured to the nearest 1 mm using a stadiometer. Percent body fat will be measured using bioelectrical impedance (Tanita-BIA Model BF679W). Psychosocial constructs will focus on how to make healthy food choices in restaurant settings. Self-efficacy will assess the confidence that the respondent feels to perform restaurant-related healthy behaviors. Intentions will assess respondents’ intention to perform specified restaurant-related healthy behaviors. Customer Exposure to components of the intervention (dose received) will vary among respondents based on measures such as their use of community media and how frequently they eat at participating restaurants. At post-intervention, we will assess exposure using a standardized form and show representative materials from each component/phase of the intervention and then ask respondents if he/she has seen/heard the material or participated in the activity (47).

Restaurant owner/manager impact instruments

General Restaurant Information including restaurant hours, social media usage, age of the restaurant, and staffing information will be collected via short questionnaire from the restaurant owners. The Restaurant environment—Observations of food service style, menu format, as well as menu labeling, amongst other items, will be collected at baseline and post-intervention. Restaurant owners’ psychosocial impact will be measured through self-efficacy and intention questions related to food and carryout operations.

Restaurant tracking form

Collected nine times throughout the intervention, restaurant owners will be asked to recall the restaurant ordering of foods that are culturally-relevant and most likely to be ordered as part of the FRESH-approved meals from the last 7 days. Data collectors will ask restaurant owners to recall the restaurant sales of the promoted foods and beverages sold in the last week. They will also report on the number still in stock and the amount of each item that was wasted, as well as perceived customer demand for promoted items.

Sample size and power

Our power calculations are based on the primary hypothesis, which involves testing the change in mean Healthy Eating Index (HEI) of customers between study groups. Our chosen effect measure is the standardized mean effect of the intervention, representing the difference in mean change in HEI score between the intervention and comparison groups. This measure is standardized by the total variability in the outcome, denoted as the standardized main effect (ES).

$\begin{array}{l}\mathrm{The\; standardized\; main\; effect}\left(\mathrm{E}\mathrm{S}\right)\\ =\frac{{\mu }_I-{\mu }_C}{\sqrt{{\tau }_{\mathit{restaurent}}+{\tau }_{\mathit{neighborhoods}}+{\sigma }^2}}\end{array}$

Here, μ_I-μ_C is the mean difference of intervention and comparison groups, respectively. σ^2 corresponds to within cluster variation, τ_restaurant corresponds to between restaurant variation within neighborhoods, and τ_neighborhoods corresponds to between neighborhood variation within a site. For our calculations, we assumed a difference in means of 5 points in HEI score would be meaningful, leading to μI − μC = 5. Using data from one of our prior multilevel intervention studies (48), we estimated the standard deviation of the change in HEI outcome to be 9–10 points, giving us σ^{^2} = 9.52^{^2} ≈ 90. We also assumed the variance of change in HEI score between restaurants within a neighborhood, and between neighborhoods within a site, would be small relative to the variance within a restaurant, therefore assuming τ_restaurant = τ_neighborhood≈5_. Thus, we aimed to be powered to detect an effect size of ES = 0.5. This is equivalent to detecting a medium-sized effect.

Power calculations were conducted using the Optimal Design program (49) and following the documentation for “Multisite cluster randomized trials with treatment at level 3 randomization” section. For all calculations, we assumed a significance level of α = 0.05, an effect size of δ = 0.5, a total of L = 2 sites (for Baltimore and DC metro area), and an effect size variability of 0, corresponding to fixed effects by site rather than random effects by site.

We varied the number of neighborhoods per site (K = 4 to 12), the number of restaurants per neighborhood (J = 2,3,4), and the number of customers per restaurant (n = 10 to 20). We varied the proportion of variability in the outcome explained by site (Baltimore vs. DC metro area) from 0 to 20% (B = 0, 0.1, 0.2). We varied the intra-class correlation coefficients for both the level 2 variable (restaurants) and the level 3 variable (neighborhoods) from 0.01 to 0.1 (ρπ = ρβ = 0.01, 0.05, 0.1). Finally, we assumed a loss to follow-up of 20%. Power is calculated as a function of the number of neighborhoods per site, the number of restaurants per neighborhood, and the number of participating customers per restaurant at initial and follow-up visits. If 8 neighborhoods are randomly assigned to intervention and comparison groups at each site (8 neighborhoods per arm total), the power to detect an effect size a difference of 5 HEI points will exceed 80% if we recruit at least 12 customers per restaurant (N = 576) when ICC = 0.05. These calculations account for a potential 20% loss to follow-up.

Data analysis

The data analysis plan includes a visual review of completed surveys, data entry, data cleaning, exploratory data analysis using sociodemographics, acculturation, baseline health status variables, and multivariable analysis. The statistical plan is to: (1) examine intervention effects via calculating the change variable by subtracting the pretest score/intake from post-test score/intake (exploratory analysis), and then (2) conduct linear mixed-effect models for confirmatory testing of intervention effects across the two study groups.

Exploratory analysis will provide an assessment of the distributions of the outcomes, as well as reveal patterns of missingness and the need for additional data review and quality assurance. If any variable in the final dataset contains more than 5% missing values, we have prepared an imputation plan. To compare the intervention and comparison groups, our primary analysis will use a linear mixed effects model with an individual’s change in HEI score (post-intervention to pre-intervention) as the outcome variable, and including fixed effects for intervention and site, and random effects for restaurant and neighborhood. We will evaluate the effect of the intervention using a hypothesis test on the coefficient of the intervention group variable, which we will interpret as the difference in mean change in HEI score between treatment groups. We will also consider a fixed effect for an intervention-by-site interaction as a secondary analysis to see if the effect of the intervention is different by site (Baltimore vs. DC metro area). Finally, we will consider adjusting these mixed effect models to include customer-level (e.g., age, gender), restaurant-level (e.g., supplier impact questionnaire), and neighborhood-level (e.g., income, restaurant density) covariates.

Economic data analyses

We will compare weekly unit sales before and after the implementation using a difference-in-difference approach between intervention and comparison groups and implement an Interrupted Time Series Analysis to explore if changes in the trajectory of sales after the introduction of FRESH can be attributed to the implementation of the program (50). Additional analyses will include sales of promoted foods and ratio of healthy-to-unhealthy food in intervention restaurants.

Development, parametrization and calibration of the FRESH system dynamics model

We will develop a system dynamics (SD) simulation model that allows stakeholders to test restaurant intervention strategies and policies virtually. Using engineered systems science modeling methodology, the model will integrate the causal loop diagram developed during the group model building activities and parametrize it with FRESH study baseline data on each site, neighborhood, restaurant, and regular customer characteristics. Finally, it will be calibrated using process and impact data from the FRESH intervention RCT. The systems science team will develop a dynamic web-based dashboard for users to interface with the model, to test different intervention strategies and policies, and to assess expected and unexpected effects on customer dietary quality and health, and on restaurant offerings and sales.

Causal loop diagram

The initial version of the SD model will be based on the final causal loop diagram developed during the group model-building sessions. This diagram captures the relationships between various customer factors (flows), intervention components, and feedback loops (reverse flows) identified during the sessions. It provides an intuitive visual representation of how these factors interact within the system, including the flows and reverse flows between them.

Parameterization of the initial SD model

Publicly available data on parameters at the site and neighborhood levels will be used to parameterize an initial version of the SD model (Table 4). In addition, we will use baseline data from our study to set default values in terms of restaurant and customer characteristics. These parameters will eventually serve to “fit” the intervention to different urban minority sites and simulate the resulting effectiveness of the intervention.

Table 4

Table 4. Potential parameters and calibration factors to be included in the initial SD model.

Developing equations to represent stocks and flow relationships

The relationships between customer factors, intervention components, and feedback loops identified in the causal loop diagram will be translated into equations using Ordinary Differential Equations (ODEs) and rational equations. These equations will quantify the flows and accumulated stocks resulting from the FRESH intervention. Flows represent the movement of resources or quantities between different components of the system, such as the quantity of foods stocked, prepared, sold, and consumed. Accumulated stocks represent the capacity or level of certain components, such as the number of healthy foods in storage. By mathematically representing these flows and stocks, the model will provide a quantitative understanding of how the intervention components interact and influence customer behaviors, restaurant operations, and health outcomes over time.

Calibrating the SD model

The first round of calibration of the model will occur as we collect data from FRESH process evaluation to reflect the implementation variability that may occur between and within sites. Approximate Bayesian Computation and partial model validation will be used to calibrate and validate the SD model. Face validation will also be performed by demonstrating the model to community representatives and obtaining feedback on the model outputs.

As the data are being collected and cleaned, process indicators will be inputted into the SD model to determine the optimal intensity of future implementations of this intervention based on the following assumptions: (1) with increasing intensity of implementation, consumers will improve their healthy food choices, (2) there is a limit (threshold) after which increasing intensity will not yield substantial improvements in desired effects, and (3) implementing multiple intervention components will reinforce strategies and messages, leading to greater change. The intervention’s intensity-related variables will be calculated as outputs and will be used as references for initial recommendations.

The second round of calibration will use impact findings from the FRESH trial. By comparing initial model predictions with the actual FRESH trial outcomes, the systems science team will improve the model using machine learning techniques in which parameters are estimated (reflecting potential variability in site characteristics) within each component and updated iteratively until convergence is achieved (29). Given the expected multiple flows of the model associated with the different pathways of influence used by the FRESH intervention, we will employ a partial model validation approach to calibrate the system dynamics equations (29). Vensim software will be used to code the computational system dynamics models, including the underlying differential equations.

Sharing of results via a user-friendly interactive web-based dashboard

Once calibrated, the SD model will be coded using JavaScript and incorporated into a web-based dashboard. The systems science team used a similar approach in developing the web app for the Staple Foods Ordinance (29). In addition to JavaScript, the team will use NextJS and Tailwind (front-end stacks) to build an interactive web-based dashboard to display the simulation results as well as let users input their site parameters, change implementation intensities, and see the eventual effect on HEI and other outcomes of interest which in turn can be manipulated by the users to understand how specific FRESH intervention components would need to be implemented, under which intensity and site parameters, to achieve the specific level of the chosen outcome (e.g., “how often will a potential customer need to eat at an intervention restaurant in order to see an impact on their overall HEI?”). Figure 4 shows the input area and an example output of the dashboard, where intervention intensities and site/neighborhood characteristics can be adjusted in the simulation to observe changes in outcomes of interest at the restaurant and customer levels and create visualizations of these outcomes (e.g., consumer HEI).

Figure 4

Figure 4. Draft FRESH system dynamics model dashboard and sample output.

The dashboard will also feature multiple financial performance metrics, including predicted revenue growth from the sales of healthier menu items and cost evaluations that emphasize possible savings from sourcing and preparing healthier food. A future planned Return on Investment (ROI) calculator will permit restaurant proprietors to enter their particular expenses and anticipated sales growth, facilitating a tailored evaluation of the financial advantages linked to health-oriented initiatives.

The categories and variables in the input area will be adjusted based on the outcomes from the group model building sessions and the final causal loop diagram for the SD model. Information from the dashboard, as well as the FRESH intervention project materials, will be made freely available and downloadable from our project website. The final SD model will be shared with suppliers, restaurant owners, and policymakers to obtain reactions and usability feedback and to encourage them to share among other potential users, in addition to its purposeful dissemination to scientific audiences, policymakers and community users.

Discussion

Working with independently-owned restaurants is a unique venue to reach people in low-income minority urban neighborhoods who may benefit most from improved dietary quality. The FRESH trial will result in two main outputs: (1) a successful model intervention trial designed to engage independently-owned restaurants in minority neighborhoods, and (2) a user-friendly system dynamics simulation model for virtually testing restaurant interventions. Community-based interventions tend to be resource-intensive and unsustainable if they do not consider the dynamic complexity that characterizes public health problems. Systems science represents a methodological approach suitable for addressing complex problems by understanding, developing, and modeling multiple interacting factors, virtually testing potential effects on health outcomes (26, 27), and saving resources that may otherwise be expended in trial and error (28). This model, which will be made available to the public, will simulate the effects of FRESH on a variety of outcomes, providing the opportunity for others to understand the potential impact of such strategies as applied to their community contexts. FRESH is expected to contribute significantly to the currently limited evidence regarding the use of simulation models to test the impact of restaurant interventions (51), and will provide a tool to inform policymaking decisions for improving the food environment (28). To our knowledge, this is the first application of a systems science approach to develop and visualize the effects of implementing various strategies aimed at increasing access to and promoting healthy eating in small urban restaurants. This is both a significant and innovative step forward for public health intervention science.

Our study does have several limitations. We are recruiting 3 restaurants per neighborhood, leading to the possibility of selection bias. In addition, there is the possibility that working in just a few restaurants in each neighborhood may lead to insufficient intervention intensity. We hope to compensate in part for this challenge by selecting customers who are regular customers of that restaurant. An added limitation is the possibility that intervention and comparison customers may purchase food at the comparison and intervention restaurants at other neighborhoods. To address this, we will assess differences in exposure to the intervention by doing a customer assessment post-intervention of dose received and will examine impact of different levels of exposure on study outcomes as we have done in other studies (52).

Conclusion

We expect that the results and products from this study will support a shift in how we think about restaurant-based food system interventions, especially in predominantly low-income, minority neighborhoods, and how to engage existing establishments going forward. FRESH will be the first (1) multilevel, multisite restaurant intervention trial to inform a system dynamics model—the result of which will simulate the impact of different FRESH strategies on chronic disease preventive dietary behaviors, subsequently yielding large potential cost savings from avoided trial-and-error intervention work, and (2) intervention that combines previously successful components (e.g., food preparation training, food access improvements, consumer environment) and tests them in independently-owned restaurants.

The FRESH intervention is still in development. It will be implemented starting in late 2024. We anticipate that the trial will successfully demonstrate that an intervention in independently-owned, minority-serving restaurants can have a significant impact on neighborhood customers’ dietary quality. We also anticipate that the system dynamics model will prove to be an invaluable tool for policymakers and other stakeholders to use in their planning of interventions and policies to improve current and future prepared food source environments.

Ethics statement

This study was approved by the Johns Hopkins Bloomberg School of Public Health Institutional Review Board (IRB) (#IRB00024491). Informed consent will be obtained for all participants.

Author contributions

UC-R: Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing. EL: Conceptualization, Resources, Writing – original draft, Writing – review & editing. ACT: Conceptualization, Methodology, Writing – original draft. LP: Conceptualization, Data curation, Investigation, Methodology, Project administration, Supervision, Writing – original draft. CP: Conceptualization, Data curation, Methodology, Writing – review & editing. ME: Conceptualization, Methodology, Visualization, Writing – review & editing. TI: Conceptualization, Resources, Software, Visualization, Writing – review & editing. JW: Conceptualization, Methodology, Resources, Writing – review & editing. YM: Conceptualization, Methodology, Resources, Writing – review & editing. AET: Conceptualization, Investigation, Project administration, Resources, Supervision, Writing – review & editing. SH: Conceptualization, Investigation, Writing – review & editing. LC: Conceptualization, Investigation, Methodology, Resources, Supervision, Writing – review & editing. AJT: Conceptualization, Investigation, Methodology, Resources, Supervision, Writing – review & editing. AO: Software, Writing – review & editing. SW: Conceptualization, Project administration, Supervision, Writing – review & editing. RR: Investigation, Writing – review & editing. PS: Investigation, Writing – review & editing. JG: Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing. VV-B: Conceptualization, Investigation, Project administration, Resources, Supervision, Writing – review & editing, Conceptualization, Investigation, Writing – review & editing.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. R01MD018022 from the National Institutes of Health (NIH).

Acknowledgments

The authors would like to thank the research project staff, restaurant owners and managers, community members and stakeholders, and our funder for their ongoing support in developing and implementing this work.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2024.1427792/full#supplementary-material

Abbreviations

FRESH, Focus on Restaurant Engagement to Strengthen Health; HEI, Healthy Eating Index; RCT, Randomized Controlled Trial; DC metro, Washington DC metropolitan area; NEMS-R, Nutrition Environment Measures Survey-Restaurants; CAB, Community Advisory Board; DAG, Dietary Guidelines for Americans; REDCap, Research Electronic Data Capture; FFQ, Food Frequency Questionnaire; SD, Systems Dynamics; ODE, Ordinary Differential Equation.

References

1. Reedy, J, Lerman, JL, Krebs-Smith, SM, Kirkpatrick, SI, Pannucci, RE, Wilson, MM, et al. Evaluation of the healthy eating Index-2015. J Acad Nutr Diet. (2018) 118:1622–33. doi: 10.1016/j.jand.2018.05.019

Crossref Full Text | Google Scholar

2. Gu, X, and Tucker, KL. Dietary quality of the US child and adolescent population: trends from 1999 to 2012 and associations with the use of federal nutrition assistance programs. Am J Clin Nutr. (2017) 105:194–202. doi: 10.3945/ajcn.116.135095

Crossref Full Text | Google Scholar

3. Kirkpatrick, SI, Dodd, KW, Reedy, J, and Krebs-Smith, SM. Income and race/ethnicity are associated with adherence to food-based dietary guidance among US adults and children. J Acad Nutr Diet. (2012) 112:624–635.e6. doi: 10.1016/j.jand.2011.11.012

PubMed Abstract | Crossref Full Text | Google Scholar

4. Hiza, HA, Casavale, KO, Guenther, PM, and Davis, CA. Diet quality of Americans differs by age, sex, race/ethnicity, income, and education level. J Acad Nutr Diet. (2013) 113:297–306. doi: 10.1016/j.jand.2012.08.011

PubMed Abstract | Crossref Full Text | Google Scholar

5. Gittelsohn, J, Lee-Kwan, SH, and Batorsky, B. Community-based interventions in prepared-food sources: a systematic review. Prev Chronic Dis. (2013) 10:E180. doi: 10.5888/pcd10.130073

PubMed Abstract | Crossref Full Text | Google Scholar

6. Gittelsohn, J, Laska, MN, Karpyn, A, Klingler, K, and Ayala, GX. Lessons learned from small store programs to increase healthy food access. Am J Health Behav. (2014) 38:307–15. doi: 10.5993/AJHB.38.2.16

PubMed Abstract | Crossref Full Text | Google Scholar

7. Stewart, H, Blisard, N, and Joliffe, D. Let's eat out: Americans weigh taste, convenience, and nutrition. Economic information bulletin. (2006). Available at: http://www.ers.usda.gov/media/860870/eib19.pdf. (Accessed July 21, 2021).

Google Scholar

8. U.S. Department of Agriculture. ERS/USDA briefing room-food CPI and expenditures: Table 10 food away from as a share of food expenditures. (2011). Available at: http://www.ers.usda.gove/briefing/cpifoodandexpenditures/data/expenditures_tables/table10.htm

Google Scholar

9. Riehle, H, Grindy, B, and Altman, M. Restaurant industry forecast. National Restaurant Association. (2013). Available at: http://www.restaurant.org/News-Research/Research/Forecast-2013. (Accessed July 21, 2021).

Google Scholar

10. Duffey, KJ, and Popkin, BM. Energy density, portion size, and eating occasions: contributions to increased energy intake in the United States, 1977-2006. PLoS Med. (2011) 8:e1001050. doi: 10.1371/journal.pmed.1001050

PubMed Abstract | Crossref Full Text | Google Scholar

11. Lee, SH, Rowan, MT, Powell, LM, Newman, S, Klassen, AC, Frick, KD, et al. Characteristics of prepared food sources in low-income neighborhoods of Baltimore City. Ecol Food Nutr. (2010) 49:409–30. doi: 10.1080/03670244.2010.524102

PubMed Abstract | Crossref Full Text | Google Scholar

12. French, SA, Wall, M, and Mitchell, NR. Household income differences in food sources and food items purchased. Int J Behav Nutr Phys Act. (2010) 7:77. doi: 10.1186/1479-5868-7-77

PubMed Abstract | Crossref Full Text | Google Scholar

13. Block, JP, Scribner, RA, and DeSalvo, KB. Fast food, race/ethnicity, and income: a geographic analysis. Am J Prev Med. (2004) 27:211–7. doi: 10.1016/j.amepre.2004.06.007

PubMed Abstract | Crossref Full Text | Google Scholar

14. Cannuscio, CC, Weiss, EE, and Asch, DA. The contribution of urban foodways to health disparities. J Urban Health. (2010) 87:381–93. doi: 10.1007/s11524-010-9441-9

PubMed Abstract | Crossref Full Text | Google Scholar

15. Britt, JW, Frandsen, K, Leng, K, Evans, D, and Pulos, E. Feasibility of voluntary menu labeling among locally owned restaurants. Health Promot Pract. (2011) 12:18–24. doi: 10.1177/1524839910386182

PubMed Abstract | Crossref Full Text | Google Scholar

16. Fuster, M, Handley, MA, Alam, T, Fullington, LA, Elbel, B, Ray, K, et al. Facilitating healthier eating at restaurants: a multidisciplinary scoping review comparing strategies, barriers, motivators, and outcomes by restaurant type and initiator. Int J Environ Res Public Health. (2021) 18. doi: 10.3390/ijerph18041479

PubMed Abstract | Crossref Full Text | Google Scholar

17. Glanz, K, Resnicow, K, Seymour, J, Hoy, K, Stewart, H, Lyons, M, et al. How major restaurant chains plan their menus: the role of profit, demand, and health. Am J Prev Med. (2007) 32:383–8. doi: 10.1016/j.amepre.2007.01.003

PubMed Abstract | Crossref Full Text | Google Scholar

18. Poirier, L, Flores, L, Rivera, I, St. Pierre, C, Wolfson, JA, Fuster, M, et al. Feasibility of collaborating with independent Latino-owned restaurants to increase sales of a healthy combo meal. J Public Health Res. (2021) 11:2659. doi: 10.4081/jphr.2021.2659

Crossref Full Text | Google Scholar

19. Gittelsohn, J, Dennisuk, LA, Christiansen, K, Bhimani, R, Johnson, A, Alexander, E, et al. Development and implementation of Baltimore healthy eating zones: a youth-targeted intervention to improve the urban food environment. Health Educ Res. (2013) 28:732–44. doi: 10.1093/her/cyt066

PubMed Abstract | Crossref Full Text | Google Scholar

20. Gittelsohn, J, Anderson Steeves, E, Mui, Y, Kharmats, AY, Hopkins, LC, and Dennis, D. B'More healthy communities for kids: design of a multi-level intervention for obesity prevention for low-income African American children. BMC Public Health. (2014) 14:942. doi: 10.1186/1471-2458-14-942

PubMed Abstract | Crossref Full Text | Google Scholar

21. Lee-Kwan, SH, Goedkoop, S, Yong, R, Batorsky, B, Hoffman, V, Jeffries, J, et al. Development and implementation of the Baltimore healthy carry-outs feasibility trial: process evaluation results. BMC Public Health. (2013) 13:638. doi: 10.1186/1471-2458-13-638

PubMed Abstract | Crossref Full Text | Google Scholar

22. Lee-Kwan, SH, Bleich, SN, Kim, H, Colantuoni, E, and Gittelsohn, J. Environmental intervention in carryout restaurants increases sales of healthy menu items in a low-income urban setting. Am J Health Promot. (2015) 29:357–64. doi: 10.4278/ajhp.130805-QUAN-408

PubMed Abstract | Crossref Full Text | Google Scholar

23. Perepezko, K, Tingey, L, Sato, P, Rastatter, S, Ruggiero, C, and Gittelsohn, J. Partnering with carryouts: implementation of a food environment intervention targeting youth obesity. Health Educ Res. (2018) 33:4–13. doi: 10.1093/her/cyx078

Crossref Full Text | Google Scholar

24. Lee-Kwan, SH, Yong, R, Bleich, SN, Kwan, NH, Park, JH, Lawrence, R, et al. Carry-out restaurant intervention increases purchases of healthy food. J Hunger Environ Nutr. (2015) 10:456–66. doi: 10.1080/19320248.2015.1045673

Crossref Full Text | Google Scholar

25. Carey, G, Malbon, E, Carey, N, Joyce, A, Crammond, B, and Carey, A. Systems science and systems thinking for public health: a systematic review of the field. BMJ Open. (2015) 5:e009002. doi: 10.1136/bmjopen-2015-009002

PubMed Abstract | Crossref Full Text | Google Scholar

26. Peters, DH. The application of systems thinking in health: why use systems thinking? Health Res Policy Syst. (2014) 12:51. doi: 10.1186/1478-4505-12-51

PubMed Abstract | Crossref Full Text | Google Scholar

27. Homer, JB, and Hirsch, GB. System dynamics modeling for public health: background and opportunities. Am J Public Health. (2006) 96:452–8. doi: 10.2105/AJPH.2005.062059

PubMed Abstract | Crossref Full Text | Google Scholar

28. Seifu, L, Ruggiero, C, Ferguson, M, Mui, Y, Lee, BY, and Gittelsohn, J. Simulation modeling to assist with childhood obesity control: perceptions of Baltimore City policymakers. J Public Health Policy. (2018) 39:173–88. doi: 10.1057/s41271-018-0125-0

PubMed Abstract | Crossref Full Text | Google Scholar

29. Zhu, S, Mitsinikos, C, Poirier, L, Igusa, T, and Gittelsohn, J. Development of a system dynamics model to guide retail food store policies in Baltimore City. Nutrients. (2021) 13:3055. doi: 10.3390/nu13093055

PubMed Abstract | Crossref Full Text | Google Scholar

30. Saelens, BE, Glanz, K, Sallis, JF, and Frank, LD. Nutrition environment measures study in restaurants (NEMS-R): development and evaluation. Am J Prev Med. (2007) 32:273–81. doi: 10.1016/j.amepre.2006.12.022

PubMed Abstract | Crossref Full Text | Google Scholar

31. Group Model Building Tools. System dynamics society. Available at: https://www.systemdynamics.org/group-model-building-tools (Accessed November 3, 2020).

Google Scholar

32. Vennix, JAM. Group model building: Facilitating team learning using system dynamics. New York: John Wiley & Sons (2001). 297 p.

Google Scholar

33. Baltimore City's Food Environment Report. (2018). Center for a Livable Future Website. Available at: https://clf.jhsph.edu/sites/default/files/2019-01/baltimore-city-food-environment-2018-report.pdf (Accessed November, 2021).

Google Scholar

34. Gittelsohn, J, Jock, B, Redmond, L, Fleischhacker, S, Eckmann, T, Bleich, SN, et al. OPREVENT2: design of a multi-institutional intervention for obesity control and prevention for American Indian adults. BMC Public Health. (2017) 17:105. doi: 10.1186/s12889-017-4018-0

PubMed Abstract | Crossref Full Text | Google Scholar

35. U.S. Department of Agriculture and U.S. Department of Health and Human Services. Dietary guidelines for Americans, 2020–2025. 9th ed (2020).

Google Scholar

36. Schwendler, T, Shipley, C, Budd, N, et al. Development and implementation: B'More healthy communities for Kid's store and wholesaler intervention. Health Promot Pract. (2017) 18:822–32. doi: 10.1177/1524839917696716

Crossref Full Text | Google Scholar

37. Budd, N, Cuccia, A, Jeffries, JK, Prasad, D, Frick, KD, Powell, L, et al. B’More healthy: retail rewards - design of a multi-level communications and pricing intervention to improve the food environment in Baltimore City. BMC Public Health. (2015) 15:283. doi: 10.1186/s12889-015-1616-6

PubMed Abstract | Crossref Full Text | Google Scholar

38. Steckler, A, and Linnan, L In: A Steckler and L Linnan, editors. Process evaluation for public health interventions and research. San Francisco: Jossey-Bass (2002)

Google Scholar

39. Harris, PA, Taylor, R, Thielke, R, Payne, J, Gonzalez, N, and Conde, JG. Research electronic data capture (REDCap) – a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. (2009) 42:377–81. doi: 10.1016/j.jbi.2008.08.010

PubMed Abstract | Crossref Full Text | Google Scholar

40. Harris, PA, Taylor, R, Minor, BL, Elliott, V, Fernandez, M, O’Neal, L, et al. REDCap consortium, the REDCap consortium: building an international community of software partners. J Biomed Inform. (2019) 95:103208. doi: 10.1016/j.jbi.2019.103208

PubMed Abstract | Crossref Full Text | Google Scholar

41. McCarley, S, López-Ríos, M, Burgos Gil, R, Turner, MM, Cleary, SD, Edberg, M, et al. Using a community-based participatory mixed methods research approach to develop, evaluate, and refine a nutrition intervention to replace sugary drinks with filtered tap water among predominantly central-American immigrant families with infants and toddlers: the water up @home pilot evaluation study. Nutrients. (2021) 13. doi: 10.3390/nu13092942

PubMed Abstract | Crossref Full Text | Google Scholar

42. Krebs-Smith, SM, Pannucci, TE, Subar, AF, et al. Update of the healthy eating index: HEI-2015. J Acad Nutr Diet. (2018) 118:1591–602. doi: 10.1016/j.jand.2018.05.021

Crossref Full Text | Google Scholar

43. Jahns, L, Johnson, LK, Mayne, ST, Cartmel, B, Picklo, MJSr, Ermakov, IV, et al. Skin and plasma carotenoid response to a provided intervention diet high in vegetables and fruit: uptake and depletion kinetics. Am J Clin Nutr. (2014) 100:930–7. doi: 10.3945/ajcn.114.086900

PubMed Abstract | Crossref Full Text | Google Scholar

44. Radtke, MD, Pitts, SJ, Jahns, L, Firnhaber, GC, Loofbourrow, BM, Zeng, A, et al. Criterion-related validity of spectroscopy-based skin carotenoid measurements as a proxy for fruit and vegetable intake: a systematic review. Adv Nutr. (2020) 11:1282–99. doi: 10.1093/advances/nmaa054

Crossref Full Text | Google Scholar

45. Fultz, AK, Rex, SM, Mazelin, A, McGarry, C, Brewer, B, Patterson, F, et al. Examining fruit and vegetable intake in low-income older adults using the veggie meter®. Nutr Health. (2021) 28:13–7. doi: 10.1177/02601060211022275

PubMed Abstract | Crossref Full Text | Google Scholar

46. Jilcott Pitts, SB, Jahns, L, Wu, Q, et al. A non-invasive assessment of skin carotenoid status through reflection spectroscopy is a feasible, reliable and potentially valid measure of fruit and vegetable consumption in a diverse community sample. Public Health Nutr. (2018) 21:1664–70. doi: 10.1017/S136898001700430X

Crossref Full Text | Google Scholar

47. Nieuwenhuijsen, MJ. Design of exposure questionnaires for epidemiological studies. Occup Environ Med. (2005) 62:272–80. doi: 10.1136/oem.2004.015206

PubMed Abstract | Crossref Full Text | Google Scholar

48. Estradé, M, van Dongen, EJI, Trude, ACB, Poirier, L, Fleischhacker, S, Wensel, CR, et al. Exposure to a multilevel, multicomponent obesity prevention intervention (OPREVENT2) in rural native American communities: variability and association with change in diet quality. Int J Environ Res Public Health. (2021) 18. doi: 10.3390/ijerph182212128

Crossref Full Text | Google Scholar

49. Raudenbush, SW, et al. (2011) Optimal design software for multi-level and longitudinal research (Version 3.01).

Google Scholar

50. McDowall, D, McCleary, R, and Bartos, BJ. Interrupted time series analysis Oxford University Press (2019).

Google Scholar

51. Jalali, MS, Rahmandad, H, Bullock, SL, Lee-Kwan, SH, Gittelsohn, J, and Ammerman, A. Dynamics of intervention adoption, implementation, and maintenance inside organizations: the case of an obesity prevention initiative. Soc Sci Med. (2019) 224:67–76. doi: 10.1016/j.socscimed.2018.12.021

PubMed Abstract | Crossref Full Text | Google Scholar

52. Trude, ACB, Kharmats, AY, Jones-Smith, JC, and Gittelsohn, J. Exposure to a multi-level multi-component childhood obesity prevention community-randomized controlled trial: patterns, determinants, and implications. Trials. (2018) 19:287. doi: 10.1186/s13063-018-2663-y

PubMed Abstract | Crossref Full Text | Google Scholar