The Effects of Responsible Gambling Pop-Up Messages on Gambling Behaviors and Cognitions: A Systematic Review and Meta-Analysis

Pop-up messages utilized by gambling operators are normally presented to gamblers during gambling sessions in order to prevent excessive gambling and/or to help in the appraisal of maladaptive gambling cognitions. However, the effect of such messages on gambling behavior and gambling cognitions has not previously been synthesized quantitatively. Consequently, a meta-analysis estimating the efficacy of pop-up messages on gambling behavior and cognitions was conducted. A systematic literature search with no time constraints was performed on Web of Science, PsychInfo, Medline, PsychNET, and the Cochrane Library. Search terms included “gambling,” “pop-up,” “reminder,” “warning message,” and “dynamic message.” Studies based on randomized controlled trials, quasi-experimental designs and pre-post studies reporting both pre- and post-pop-up data were included. Two authors independently extracted data using pre-defined fields including quality assessment. A total of 18 studies were included and data were synthesized using a random effects model estimating Hedges' g. The effects of pop-ups were g = 0.413 for cognitive measures (95% CI = 0.115–0.707) and g = 0.505 for behavioral measures (95% CI = 0.256–0.746). For both outcomes there was significant between-study heterogeneity which could not be explained by setting (laboratory vs. naturalistic) or sample (gambler vs. non-gamblers). It is concluded that pop-up messages provide moderate effects on gambling behavior and cognitions in the short-term and that such messages play an important role in the gambling operators' portfolio of responsible gambling tools.

Introduction

Gambling can be defined as wagering money or other objects of value on an event of an uncertain outcome that is partly or completely determined by chance (1), and has become increasingly available to individuals due to such factors as increased accessibility via the internet and liberalization of gambling regulation. For most individuals, gambling represents a recreational activity. However, it is estimated that between 0.1 and 3.4% of the population in Europe and 0.1–5.8% of the population worldwide engage in problematic gambling behavior (2). Problem gambling is, according to Blaszczynski et al. [(3), p. 305], “a lay term that refers to a broader category of individuals exhibiting patterns of excessive gambling behavior that is associated with harmful effects” (p. 305). The terms “pathological gambling” and “gambling disorder” refer to a more specific pattern of problematic gambling, and has been classified as a mental disorder (3). Blaszczynski et al. [(3), p. 305] suggest that “problem gamblers may or may not suffer impaired control” and “conceptually, all pathological gamblers are problem gamblers, but not all problem gamblers are pathological gamblers.”

Gambling disorder (formerly pathological gambling) was the first non-chemical (i.e., behavioral) addiction to be recognized in formal diagnostic systems (4). According to the fifth edition of Diagnostic and Statistical Manual of Mental Disorders (DSM-5), gambling disorder is defined by nine criteria similar to those of substance abuse, such as lack of control, tolerance, withdrawal, and the maintaining of harmful behavior despite negative consequences (5). In the eleventh revision of the International Statistical Classification of Diseases (ICD-11), the World Health Organization (6) defines gambling disorder as “a pattern of persistent or recurrent gambling behavior, which may be online (i.e., over the internet) or offline.” The gambling behavior of individuals suffering from gambling disorder can significantly affect an individual's personal, family and social life, as well as their educational and/or occupational functioning (7–9).

Treatment of gambling disorder is associated with involvement of health services (e.g., counseling administered by specialists employed in public health programs). An important feature of treatment is the emphasis on gamblers who are already suffering from severe gambling problems (3). Different treatment approaches have been developed, including cognitive behavioral therapy incorporating motivational interviewing (10), mindfulness (11), and pharmacological treatment (12, 13). Overall, results show positive effect of these treatments. However, limitations across studies points to a lack of evidence of successful long-term effects and attrition (10–14). Furthermore, the treatment of problem gambling is usually costly and few of those affected seeks out treatment on their own (15, 16). For these reasons, the development and implementation of effective and cost efficient tools to reduce gambling-related problems seems warranted.

The past two decades' efforts to create, shape, and implement responsible gambling (RG) strategies and programs regarding the management of gambling-related activities, have been characterized by two main frameworks [(17), p. 1]: The Public Health Model and the Reno Model, with the latter historically and geographically being the more influential of the two (18). While it is true that they share some objectives (e.g., reliance on strong empirical data and collaborative efforts between stakeholders), they differ in the areas of focus and approaches (17, 19–21).

One of the central tenets of the Reno model, is informed choice—the making of a decision based on as much information as possible—“the ultimate decision to gamble or not lies with the gambler” [(17), p. 9]. Furthermore, the Reno Model states that measures to prevent problem gambling should be as non-intrusive as possible, so as to let recreational gamblers engage in “healthy gambling” (i.e., gambling without negative/adverse consequences). It also stresses the point that efforts from the medical community (i.e., treatments and prevention) ought to specifically target at-risk groups, without being intrusive toward the larger population (22). The Reno Model's emphasis on individual responsibility, informed choice, personal control, and prioritization of the recreational benefits of gambling, has been critiqued by Hancock and Smith (20), Delfabbro and King (19), and Young and Markham (21) for being ideological at its core with libertarianism as its central tenet. Hancock and Smith (18) further criticize the Reno Model for its minimal regard to effective RG safeguards following the last two decades' rapid increase of gambling.

The Public Health Model, casting a wider net, seeks the widespread use of epidemiological studies to map the impact of gambling-related harm (23). Public health officials should, in line with this perspective, view gambling as a population-based phenomenon and seek to identify the cultural, social, and economic factors that mediate gambling (17). Public policy should be guided in such a way so as to increase health in the population and to prevent gambling-related harm (23). The Public Health Model suggests implementing guidelines that promote healthy gambling, including large-scale public informational campaigns, similar to informational campaigns regarding alcohol and tobacco, about the possible adverse effects of gambling (23). The Public Health Model further urges for public policy and governmental legislation to regulate gamblers' behavior in such a way so as to reduce the likelihood of gambling-related harm (e.g., mandatory loss-limits, mandatory breaks in play, and reduced accessibility during specific hours) (23). Hancock and Smith (20), and Young and Markham (21) have called for a broad implementation of the Public Health Model on account of what they argue is the failure of the Reno Model to properly implement measures to reduce gambling-related harm.

“Responsible gambling” can be defined as the “policies and practices designed to prevent and reduce potential harms associated with gambling” and is emphasized by both frameworks [(3), p. 308]. More explicitly, RG refers to guidelines, strategies, or programs that attempt to avert possible harmful health outcomes, in contrast to the principles of treatment (3). RG may also imply “means to prevent gambling problems or to reduce the negative consequences of existing problems” [(24), p. 1,376]. “Means” may refer to “self-exclusion programs; behavioral tracking of play patterns; loss and deposit limit setting (both player and corporate); player pre-commitment to deposits, losses, wins, or gambling time; and warning messages” [(25), p. 225]. Most of these measures require registration and collection of player account gambling data, which are enabled by gambling via online gambling accounts or via loyalty card/player cards (26). An example of RG measures put into practice, is New Zealand's policy regarding EGMs in gambling venues, which are required by law to display pop-up messages to interrupt play at “irregular intervals not exceeding 30 min of continuous play” [(27), p. 1,116].

Hence, RG is about the necessity to sustain a safe environment for gamblers, and because the main objective of RG programs is to prevent gambling related harm, “[RG] programs should provide information that consumers use to make decisions” [(28), p. 570], which represents a perspective on RG tools congruent with the Reno Model. The use of pre-commitment to limit expenditure is an example of this. Such a measure permits gamblers to regulate how much money and/or time they can spend gambling, and allows gamblers to temporarily (or permanently) exclude themselves from gambling (24). Personal feedback interventions (PFIs) are another approach, where information about an individual's gambling behavior is compared to another individual's gambling behavior, and then presented to the gambler (29).

Pop-ups or dynamic messages, as understood in an RG context, comprise informative messages appearing on screen during gambling, halting play, with the overall aim of preventing and/or reducing gambling-related harm (30). The message subsequently either disappears after a set duration of time or requires some kind of action to be taken (e.g., pressing “OK” or “Press/click here to continue”) on the part of the gambler [(31), p. 933]. It can be used to present factually descriptive or normative information, such as information about time and/or monetary expenditure, either regarding the individual player solely, or in comparison with other gamblers (30). It can also be used to correct irrational or distorted beliefs about gambling (e.g., “Winning is not due to luck. It's random” or “The next spin has nothing to do with your previous spin”) (31), or as a reminder of progress toward a previously set limit (i.e., progression toward a monetary limit set before initiating a gambling session) (30). Furthermore, messages can be worded to encourage self-appraisal, so as to increase gamblers' awareness of their own gambling behavior (e.g., “Pause and think … Are you in control of your risk taking?”) (32, 33). Pop-up messages can vary in time of appearance, and usually appear after a set duration of time or a set monetary expenditure (34). In short, pop-up messages serve as a tool to deliver RG information to gamblers during play.

Several researchers have investigated both the efficacy of information delivery and the effect of various types of content on expenditure and time spent playing (35–38). Furthermore, Monaghan and Blaszczynski (39) conducted a study on recollection of message content, where the participants recalled dynamic messages (pop-up messages) more easily than static messages, and another study where pop-up messages where recalled more effectively than static messages, both immediately and at a 2-weeks follow-up (33). Studies on pop-up messages and limit setting—that is, a monetary limit set by the gambler before initiating a gambling session—suggest that individuals exposed to monetary limit reminders via pop-up messages, are more likely to adhere to the pre-set limits (40, 41).

As previously mentioned, gambling is more accessible to the public than ever before, which in turn may increase problem gambling. This, combined with the fact that there is support to suggest that pop-up messages can be an effective RG tool, and that (to date) no meta-analysis have been conducted on the effects of pop-up messaging (as far as the authors of the present meta-analysis are aware of), the goal of the present meta-analysis is to explore the effect of pop-up messages on gambling-related behaviors and cognitions.

Method

The present meta-analysis was conducted in accordance with the guidelines of the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) (42, 43). For complete checklist, see Table 1.

TABLE 1

Table 1. PRISMA checklist.

Eligibility Criteria: Participants, Interventions, Comparators

The meta-analysis included (i) randomized controlled trials, quasi-experimental studies as well as pre-post studies investigating the effect of (ii) RG pop-up messages on (iii) gambling behaviors and/or gambling cognitions in (iv) gamblers and non-gamblers of (v) all ages equal to or above the legal gambling age, and (vi) published in peer reviewed journals or as conference presentations.

A study was deemed to include a comparator/control group if the intervention group was compared to a group of any of the following kind: (i) no pop-up message control (including cases with other forms of warnings or pauses before, during, or after play), (ii) passive pop-up message controls (e.g., “click ok to continue”), (iii) irrelevant pop-up message intervention (e.g., “The roulette game was invented in 1720”), or (iv) active pop-up messages assumed to have significantly less of an effect than the experiment intervention (e.g., “You have now played 1,000 slot games. Do you want to continue? (YES/NO)” vs. “We would like to inform you, that you have just played 1,000 slot games. Only a few people play more than 1,000 slot games. The chance of winning does not increase with the duration of the session. Taking a break often helps, and you can choose the duration of the break”) [(36), p. 3]. In studies containing two or more control conditions, the control group selected for effect size calculations was chosen in line with the aforementioned order [e.g., (44)].

Studies with a quasi-experimental design (e.g., studies without randomization of participants to conditions) were included as this is often the only method of getting real life data from gambling providers. Furthermore, studies lacking comparison/control conditions were included if both pre- and post-intervention data were reported or obtained from the author(s).

RG pop-up messages were operationalized as dynamic messages intended to reduce gambling harm in some form or another, by interrupting play and that provided either (i) information regarding gambling behavior (e.g., time spent, trials played, money spent, progression toward a pre-set limit), (ii) general information about the nature of gambling machines (e.g., “You cannot predict anything in a game of chance”), or (iii) messages containing encouraging self-appraisal (e.g., “Stop and think… Are you in control of your risk-taking?”). Furthermore, RG pop-up messages had to appear on a gambling device containing a screen [e.g., electronic gaming machine [EGM] or personal computer]. This implied that studies where pop-ups either appeared prior to gambling or after gambling had ended, were administered via other forms of communications (e.g., email or SMS), or randomly over a longer period (i.e., weeks or months), were excluded [e.g., (45)]. In addition, this also meant that studies with encouraging messages (e.g., “You are a skillful player!”) were excluded, as were studies where RG pop-ups were not the primary independent variable (e.g., the message “The game is now paused,” followed by a forced pause in the game). Outcome measures of interest were pooled into two main categories: (i) behavioral (e.g., total number of bets, total amount spent, limit adherence), and (ii) cognitive (e.g., recall, arousal, dissociation).

The population in all included studies were classified as either gamblers or non-gamblers. This categorization was made by the authors of the individual included studies. Trials contained participants of both sexes and all ages, as long as they were equal to or over the legal age to gamble (this was 18 years in most cases, but could be 20 or 21, depending on area of jurisdiction). The trials took place in one of the following locations: (i) gambling venue, (ii) laboratory, and (iii) online. Studies were excluded if they (i) failed to meet the aforementioned inclusion criteria, (ii) were duplicates, (iii) written in a language other than Norwegian, Swedish, Danish, English, or Italian, (iv) did not contain sufficient information to calculate effect size, or (v) had self-report data on behavioral measures (e.g., time spent, amount spent, number of spins).

Search Strategy

Studies were identified by searching electronic databases, reference lists of relevant papers, and through contacting study authors, in cases where supplementary information was needed. No limits were set on language or time periods. The search was conducted on Web of Science (1945-Present), APA PsychINFO (Ovid) (1806-Present), Medline (Ovid) (1946-present), PubMed (1993-present), APA PsychNET (unable to retrieve information on time period), Cochrane Library on Wiley Online Library [including Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects (DARE)] and the Cochrane Central Register of Controlled Trials (CENTRAL) (1995-present). The last search was conducted by the first author on May 21, 2020.

The following search terms were used in all databases and trial registers searched: “gambling”; “pop up”; “pop ups”; “pop-up”^*; “reminder”; “warning message”; and “dynamic message.” The final search strategy was developed through identification and discussion of relevant keywords. Preliminary searches were conducted to identify further relevant keywords. The final search strategy was agreed upon through consensus. For detailed overview of search terms and search strategy, see Table 2.

TABLE 2

Table 2. Search terms and strategy.

Data Extraction

All studies were screened independently by two authors. Most studies were excluded based on screening of title and abstract (e.g., when it was apparent that the study did not report on the effects of pop-ups). Inclusion to the next review stage was determined by consensus, and by consultation with a third author. Full texts were subsequently screened by two authors independently, with disagreements being resolved through discussion and consultation with the third author. The reference lists of all included studies in the meta-analyses were also screened.

A data extraction sheet, based on the Cochrane Consumers and Communication Review Group's data extraction template, was developed by two authors and pilot-tested on eight randomly selected studies. The extraction sheet was then refined to code further aspects of studies included. Data were extracted from the included studies by the three first authors and the extracted data then checked by two authors independently. Disagreements regarding extracted data were resolved through discussion.

In cases without sufficient data to calculate effect sizes, authors listed with contact information were contacted via email. Means, standard deviations, and sample sizes on all measures were obtained from four sets of authors (46–49) as data had only been presented graphically or was not included in the original publication. Furthermore, four authors (50–53) responded, but were for various reasons unable to provide data. Additionally, three authors did not respond.

Data were extracted from each trial on: (i) characteristics of participants (age, gender, and categorization into gamblers or non-gamblers); (ii) study design (including between-participants design, pre-post measurement design, and repeated measures design); (iii) exclusion and inclusion criteria; (iv) type of intervention, including the type of pop-up intervention and the type of control condition (i.e., no intervention, passive intervention, or intervention assumed less effective than experiment intervention); (v) type of outcome measure (behavioral and/or cognitive), and (vi) follow-up interventions. Behavioral outcome measures were defined as any type of gambling action measured, while cognitive measures comprised any type of gambling cognition assessed.

Assessment of Risk of Bias

Risk of bias in individual studies was assessed using the Evidence Project Risk of Bias Tool developed by Kennedy et al. (54). Using this tool, two individual authors assessed the following areas of bias: (a) cohort, (b) control or comparison group, (c) pre-post intervention data, (d) random assignment of participants to the intervention, (e) random selection of participants for assessment, (f) follow-up rate of 80% or more, (g) comparison groups equivalent on socio-demographics, and (h) comparison groups equivalent on outcome measures at baseline. Items a, b, c, and e are dichotomous and have the response options “yes” and “no,” d is categorical and has “yes,” “no,” and “NA” (not applicable) as response options, whereas f, g, and h are categorical and have the response options “yes”, “no”, “NA” and “NR” (not reported) as response options. In cases of disagreement, a third author was consulted and disagreements resolved through discussion. Risk of bias was assessed at both the study and outcome level and Cohen's kappa was calculated to assess inter-rater reliability. No risk of bias scores were calculated, based on the recommendations of Kennedy et al. (54).

Meta-Analyses

Two meta-analyses were conducted: one investigating cognitive measures, and the other investigating behavioral measures. Cognitive measures comprised different non-behavioral outcomes such as player experience, keeping track of play, estimation of time and money spent, erroneous beliefs, dissociation, and recall of pop-up messages, whereas the behavioral outcomes typically consisted of measures such as amount of money spent, speed of gambling, number of games played, etc. For each study, Hedges' g was computed. The primary outcome measure of the meta-analyses was Hedges' g and Cochrane's Q. I² were calculated to assess heterogeneity, as it reflects the proportion of variation in observed effects that is due to variation in true effects (55). I² values of 0.25, 0.50, and 0.75 are regarded as small, medium, and large, respectively (56). Initially, additional meta-analyses were planned to assess the effect at follow-up. However, in the sample of studies, only two included follow-up data (33, 53), which rendered meta-analyses for follow-up data less meaningful. Therefore, no additional meta-analyses were conducted. Furthermore, in cases where studies had more than one pop-up condition not conceptually different from each other, the groups were collapsed, and standard error—and consequently the 95% confidence intervals—were adjusted accordingly, in order not to repeat the control group data, as per the recommendations of Giang et al. (57).

In cases with more than one pop-up condition conceptually different from the other, the control group was split into corresponding numbers of groups, as recommended in the Cochrane Handbook for Systematic Reviews of Interventions (58).

Two moderators were decided upon in case of significant heterogeneity: (i) laboratory setting vs. naturalistic setting, and (ii) gamblers vs. non-gamblers (a sample was deemed to consist of non-gamblers in cases where they accounted for 50% or more of the total sample). Many studies included multiple measures within the same category (category referring to either behavioral or cognitive). In these instances, the mean effect size and variance was calculated for the study as a whole. When combining results from more than one outcome within the same outcome category from the same study, setting the correlation coefficient between outcomes to the default (r = 1.00) used in most meta-analytic software overestimates the standard error (59). To correct for this, the correlation coefficient between the outcomes was set to 0.70.

In cases of significant heterogeneity, subgroup analyses were conducted with a focus on the two aforementioned a priori determined moderators. The moderator analyses comprised mixed effects models (random across subgroups pooling tau across studies, but combining subgroups using fixed effect models), as recommended by Borenstein et al. (59). The planned moderator analyses were conducted when there were four or more studies within each category, in line with the suggested minimum criteria for number of studies (60).

Publication Bias

Publication bias was examined by creating and inspecting funnel plots, and by using Duval and Tweedie's (61) “trim and fill” procedure, calculating a new and adjusted effect size which takes into account potential publication bias. In addition to this, Orwin's fail safe N was calculated, measuring the number of studies with zero effect needed to bring the observed effect size (Hedge's g) down to a pre-set trivial effect size (62), set to g = 0.20, which equals a small effect (63).

Results

Selection and Inclusion of Studies

A total of 18 papers involving 19 studies [(33) included two individual studies] were deemed eligible and were included in the present meta-analyses. The systematic searches conducted in Web of Science, PsychInfo, Medline, PubMed and Cochrane Library yielded a total of 436 hits. A total of 306 papers remained after the removal of duplicates. All of the remaining papers were systematically reviewed by the authors. Of the 306 papers, 263 were excluded based on title and abstract. This left 43 papers to be assessed for eligibility, of which 25 were excluded for the following reasons: seven did not provide sufficient data for calculation of effect size, six lacked a pop-up message condition altogether, three included measures deemed unreliable (e.g., self-report measures for spins per minute), three were review articles, two lacked a pop-up message condition during play, one had experiment and control condition deemed too similar, one contained a non RG-pop-up message (i.e., the message content was encouraging play), one was excluded due to language restrictions, and one was an abstract with no full text paper available. One unpublished study was identified through the database searches and found eligible. The screening process can be found in the PRISMA flow chart (see Figure 1).

FIGURE 1

Figure 1. PRISMA flow chart.

Characteristics of the Included Studies

In total 14 studies were conducted in laboratory settings, three were in situ (naturalistic) studies, and one study was conducted in a laboratory setting and subsequently replicated in a naturalistic setting. All studies were published in English. Furthermore, all studies were conducted within the past 15 years (from 2006 to 2019). Participants in seven out of the 18 studies were university students whereas participants in 12 out of the 18 studies consisted of non-university gamblers of all types. Nine of the included studies did not display pop-ups in their control condition, seven displayed some variation of a pop-up message in the control condition and two used pre-pop-up measures as the control condition. Only two studies included follow-up assessments, at 1 and 2 weeks, respectively. Eight studies contained outcome measures for behavior only and two for cognition only, whereas eight included outcomes for both behavior and cognition. The total number of participants included in the studies could not be assessed exactly because two studies listed approximate sample sizes of 50,000 [i.e., (35)] and 70,000 [i.e., (36)]. Studies lacking information on the socio-demographics of their samples were assumed to contain both genders and different ages, as the sheer size would have made gender and age homogeneity highly unlikely. Tables 3–5 provide a complete overview of the studies' key properties and characteristics.

TABLE 3

Table 3. Study characteristics, studies A-H.

TABLE 4

Table 4. Study characteristics, studies H-M.

TABLE 5

Table 5. Study characteristics, studies M-W.

Risk of Bias

Risk of bias of the included studies was assessed using the Evidence Project Risk of Bias Tool (54). This instrument was used by two authors independently, rating each of the 18 studies. Disagreements were resolved by consulting the third author. Of the 18 studies included, one met the criteria of cohort. Seventeen studies had a control or a comparison group, whereas one was based on a pre-post measurement design. In total, six studies reported data both pre- and post-pop-up intervention. All except two studies scored “yes” on “random assignment of participants to the intervention”; the two that did not were rated “not applicable” as their datasets were anonymous and provided by a real-world online gambling site. Two studies had random selection of participants of assessment, whereas the other 16 consisted of gamblers. Of the two studies containing follow up intervention, one met the criteria of retaining at least 80% of the participants.

All but two studies had comparison groups equivalent on socio-demographics (both rated “not applicable”). Lastly, two studies reported comparison groups equivalent on outcome measures at baseline, whereas the others were categorized as “not reported.” A calculation of inter-rater reliability of risk of bias yielded a Cohen's Kappa of 0.96 which according to Landis and Koch (68) is regarded as a perfect agreement. The risk of bias in the included studies is shown in Table 6.

TABLE 6

Table 6. Evaluation of risk of bias in the individual studies.

Synthesized Findings

Cognitive Measures

Results for cognitive measures after pop-up intervention during gambling showed an overall effect size of g = 0.413 (95% CI = 0.155–0.707), p < 0.01 (see Figure 2). Cochrane's Q was 48.63 (df = 13), p < 0.01) and the I² was 73.27. No subgroup analysis was performed due to lack of studies including non-gamblers (k = 2) and in situ trials (k = 1). In order to investigate whether the present findings were influenced by publication bias, a funnel plot was drawn. The funnel plot was not entirely symmetrical (see Figure 3), suggesting a lack of potential studies to the right of the distribution. The Duval and Tweedie s “trim and fill” procedure provided an adjusted effect size g = 0.578 (95% CI = 0.325–0.830, p < 0.01). Orwin's fail-safe N showed that 22 studies with zero effect would be needed to bring the overall effect size down to a trivial level (g = 0.20).

FIGURE 2

Figure 2. Forest plot showing the effect (Hedges' g) of pop-up messages on cognitive measures.

FIGURE 3

Figure 3. Funnel plot from the meta-analysis of cognitive measures, adjusted using Duval and Tweedie's trim and fill.

Behavioral Measures

The overall effect size for behavioral measures was g = 0.507 (95% CI = 0.267–0.747, p < 0.01; see forest plot, Figure 4). Cochrane's Q was 109.84 (df = 20), p < 0.5, and I² = 81.97, suggesting significant heterogeneity. Four studies (k = 4) were done in a naturalistic setting and 17 were performed in a laboratory setting, and thus, a subgroup analysis was performed. The effect size difference turned out not significant (Q_bet = 0.010, df = 1, p = 0.919). No subgroup analysis was performed due to the lack of studies containing non-gamblers (k = 1). To detect potential publication bias, a funnel plot was drawn. The plot was not symmetrical (see Figure 5), and indicated lack of potential studies to the right of the distribution. Hence, Duwal and Tweedie's “trim and fill” procedure was conducted, providing an adjusted effect size of 0.616 (95% CI = 0.359–0.872, p < 0.01). Orwin's fail-safe N showed that the number of missing studies with zero effect needed to bring Hedges' g below 0.20, was 12.

FIGURE 4

Figure 4. Forest plot showing the effect (Hedges' g) of pop-up messages on behavioral measures.

FIGURE 5

Figure 5. Funnel plot from the meta-analysis of behavioral measures, adjusted using Duval and Tweedie's trim and fill.

Discussion

The aim of the present meta-analysis was to investigate the effect of gambling pop-up message interventions on behavioral and cognitive outcomes at first exposure and at follow-up, as well as to investigate the potential moderating effect of study setting and sample characteristics. The meta-analysis of behavioral measures demonstrated a significant effect amounting to a medium effect size in accordance with Cohen (63), indicating that RG pop-up message interventions have a substantial impact on participants' gambling behavior. The heterogeneity was significant, reflecting true differences in effect size across studies (59). Consequently, a subgroup analysis was conducted. However, no significant difference between effects from laboratory settings and naturalistic settings was found. Orwin's fail-safe N demonstrated that the total number of studies needed to bring the effect size down to a trivial level (g = 0.20), was 12, which suggest some stability of the findings.

A total of five studies provided effect sizes above 1.00 (35, 37, 38, 38, 40, 67). When looking at the pop-ups from these studies, they all appeared to be personalized in terms of gambling behavior [e.g., “You have reached your preset limit”; (40)], whereas pop-ups of studies with no effect appeared to be more generic [e.g., “No matter how you play, you cannot influence the outcome of the game”; (64)]. Regarding effects in the three real world studies, the first study showed that when being informed of having played 1,000 slot games, 45 of 4,205 sessions ended compared to 5 of 4,220 sessions before the pop-up had been introduced by the gaming operator (35). The second study found that 169 of 11,878 sessions ended following an enhanced pop-up message compared to 75 of 11,232 sessions with a simple pop-up message (36). The third study reported a mean number of spins and amount wagered of 15.8 and $17.70 in the strong pop-up message condition, respectively, whereas the corresponding values in the control condition were 18.1 and $24.90 (66).

A significant and positive effect was also found for the cognitive measures, reflecting a medium effect size. Also here, the heterogeneity was significant. No subgroup analysis was conducted, as there were too few studies in the smallest of the subgroups. Orwin's fail-safe N demonstrated that the total number of studies with zero effect needed to bring the overall effect size down to a trivial level (g = 0.20) was 22, indicating that the findings are stable.

The gamblers vs. non-gambler moderator, was chosen as previous studies have shown that gamblers diverge from non-gamblers on some gambling-related behaviors and cognitions compared to non-gamblers (69). For instance, it has been found that gamblers discount probabilistic rewards less steeply than non-gamblers (70), that laboratory trials with student populations yield larger effects than with non-student populations (71), and that gamblers take larger monetary risks during roulette play than non-gamblers (72). The second moderator (laboratory vs. real gambling settings) was emphasized as there might be ecological challenges associated with laboratory studies such as the laboratory cubicle-casino ambiance variance and the absence of direct or personal monetary risk or loss (71, 73). The fact that this moderator turned out non-significant may be due to the relatively small amount of studies included in naturalistic settings. It may also reflect that other potential moderators, not identified by the authors, could explain significant proportions of the heterogeneity. The present analyses of heterogeneity could therefore have benefited from either different or simply more moderators.

Implications for RG Practices

The results of the present meta-analysis show that pop-up messaging appear to be an effective RG tool, as it seemed to reduce possible harmful gambling behaviors and cognitions. As previous studies have indicated, the intervention of a pop-up message reduces, amongst other things, the total amount wagered, spins per minute, and irrational beliefs. Within the framework of responsible gambling in both the Reno Model and the Public Health Model, these results are encouraging, as it complies with some of the models' central tenets: the emphasis on a scientific approach in the development of measures to promote responsible gambling and at the same time, reducing gambling-related harm. Furthermore, the non-intrusive and non-restrictive nature of pop-up messages coincides with three central Reno Model principles: (i) the opportunity to gamble without intrusion, (ii) the maintenance of personal control (as opposed to, for example, reduced casino opening hours), and (iii) the opportunity to gamble in an unrestricted, but informed manner. In addition to this, the result of the present meta-analyses provides further evidence of the suggestion that gamblers gamble more responsibly (e.g., fewer spins, less money wagered, less time spent playing, increased limit-adherence) when informed about the nature of the game they are playing (41, 44, 74). Finally, the relative non-intrusive nature of a pop-up message could help facilitate increased collaboration between different stakeholders involved in gambling, as emphasized in the Reno Model (17). This is further attested to, because major online gambling sites already have implemented pop-up messages as an RG feature (35, 36).

Although there is some support suggesting that RG pop-up messages influence behavior of a small number of people playing for a long time (1,000 or more spins on a virtual roulette game) (35), the failure to identify a sufficient number of studies containing problem gamblers, makes it difficult to assess the effect of pop-up messages on gamblers who already suffer from gambling-related problems, and therefore fails to shed any new light on its effectiveness in mitigating problematic gambling among problem gamblers. However, the effect it has on gambling behavior of casual/regular gamblers, contributes (at least short-term) to a reduction of gambling harm, which in turn promotes a public health perspective of safe gambling (17). As gambling becomes increasingly digitalized, pop-up messaging appears to represent an accessible and cost-effective way to attenuate excessive gambling behavior and to modify gambling-related cognitions.

Limitations and Future Directions

The present meta-analysis has several limitations that should be noted. One limitation concerns the risk of bias of the included studies, with several studies failing to pass some of the criteria of the assessment tool used (54). Most of the studies lacked one or more of the following: (i) a cohort (pre and post data on the same subjects), (ii) random selection of participants for assessment, (iii) follow-up assessments, and (iv) did not report whether there was equivalency on outcome measures at baseline for comparison groups. Only two studies (33, 53) included follow-up data. The lack of follow-up data in the included studies makes it difficult to draw conclusions about the long-term effects of pop-up interventions on gambling behavior and cognition. Furthermore, the absence of random sampling and equivalency on outcome measures at baseline for comparison groups, should be considered a limitation, as it can limit the generalizability of the results (59).

Only two of the included studies (35, 36) evaluated gamblers in their real-life gambling environments (i.e., actual gambling with players spending their own money). It is therefore important to be cognizant of the ecological validity of the findings from the majority of studies included in the present meta-analysis. Even though it has been shown that certain types of rewards (i.e., the possibility to win via raffle or lottery tickets) can be as effective as immediate monetary rewards (75–77), there is still reason to question the true ecological validity and generalizability of such trials, as, for example, the absence of risking one's own money in a gambling situation, has been shown to increase spending in gamblers (78). Other limitations related to studying gambling behavior in laboratory settings are that such settings often lack aspects present in real-life gambling such as variety of gambling motives, ability to choose between different games, playing games in different ways, and the distinct milieu of gambling venues (79). Therefore, more studies on the effects of pop-ups should be conducted in real world gambling contexts with real gamblers in real time [like those of Auer et al. (35) and Auer and Griffiths (36)]. Another limitation of the present meta-analysis is that outcomes subsumed as cognitive outcomes varied significantly. However, the limited number of studies prevented meta-analyses of more narrow outcomes and constructs.

One area for future research concerns the long-term effects of pop-ups. It can be argued that the main aim of pop-ups is to change behaviors and cognitions in the specific context of a gambling session. Consequently, it would be of interest to investigate whether repeated exposure to pop-ups during gambling can cause long-lasting and robust changes in gambling behaviors and cognitions or not. This should be addressed in future research. In addition to this, the authors of the present review echo other researchers' call for further investigation into the possible habituation of warning messages (31, 52). The effect of pop-ups in specific vulnerable populations (e.g., problem gamblers) should also be addressed in future studies.

The comparison condition of the included studies differed across studies. Some comprised a non-pop-up intervention, whereas other comprised a pause or neutral or irrelevant pop-up messages. This makes it difficult to conclude whether some of the effects are attributable to the presence of a pop-up in itself, or whether the effects were contingent on the specific form, placement, duration, or content of the pop-up. Given that the experimental interventions differed in message content (e.g., limit reminders, self-appraisal feedback, personalized feedback), future studies and meta-analyses are advised also to further investigate the effects of differences in message content. The present meta-analysis did not investigate the effects of pauses per se during gambling, and pure pauses comprised the control condition in several studies. Therefore, the present meta-analysis does not provide information about the effect of pauses on gambling cognitions and behavior. We thus recommend that future studies and reviews systematically investigate the effects of pauses, preferably by experimental designs in real-world settings.

Conclusions

The present meta-analysis examined the efficacy of RG pop-up messages on gambling behaviors and cognitions. The results showed that RG pop-up messages had a moderate effect on gambling behaviors and cognitions, using interventions which should be considered as highly cost-effective. The present meta-analysis is of importance, as it is the first meta-analyses on the efficacy of pop-up messages on gambling behaviors and cognitions [although narrative literature reviews have been previously conducted (34, 80, 81)]. As such, the meta-analysis contributes to the literature by filling an important gap in knowledge of the efficacy of pop-up messages as a tool to promote responsible gambling. The findings imply that there are benefits to using pop-up messages to promote responsible gambling, although caution should be exercised in terms of generalizability to real-world gambling settings, hence more studies in such contexts should be conducted.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

As only secondary data were analyzed the study was exempted from approval by Regional Committees for Medical and Health Research Ethics.

Author Contributions

SP assisted by BB, JS, and AB: conceptualization. BB, JS, and AB: literature search and study coding. BB, JS, and AB assisted by SP: data analysis. BB, JS, AB, and SP: drafting the manuscript. EE, TL, and MG: critically revising the original manuscript and the revised version. All authors contributed to finalizing the paper and approved the submitted version.

Funding

The present meta-analysis was funded by the Norwegian Competence Center for Gambling and Gaming Research at the University of Bergen, Norway.

Conflict of Interest

MG's university currently receives research funding from Norsk Tipping (the gambling operator owned by the Norwegian Government). MG has also received funding for a number of research projects in the area of gambling education for young people, social responsibility in gambling and gambling treatment from Gamble Aware (formerly the Responsible Gambling Trust), a charitable body which funds its research program based on donations from the gambling industry. MG regularly undertakes consultancy for various gaming companies in the area of social responsibility in gambling.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

1. Bolen DW, Boyd WH. Gambling and the gambler: a review and preliminary findings. Arch Gen Psychiatry. (1968) 18:617–30. doi: 10.1001/archpsyc.1968.01740050105018

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Calado F, Griffiths MD. Problem gambling worldwide: An update and systematic review of empirical research (2000-2015). J Behav Addict. (2016) 5:592–613. doi: 10.1556/2006.5.2016.073

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Blaszczynski A, Ladouceur R, Shaffer HJ. A science-based framework for responsible gambling: The Reno Model. J Gambling Studies. (2004) 20:301–17. doi: 10.1023/B:JOGS.0000040281.49444.e2

PubMed Abstract | CrossRef Full Text | Google Scholar

4. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (3rd ed.). Washington DC: American Psychiatric Association (1980).

Google Scholar

5. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (5th ed.). Washington DC: American Psychiatric Association (2013).

Google Scholar

6. World Health Organization. International Classification of Diseases 11th Revision. (2019). Available online at: https://icd.who.int/

Google Scholar

7. Ladouceur R, Boisvert JM, Pépin M, Loranger M, Sylvain C. Social cost of pathological gambling. J Gambling Studies. (1994) 10:399–409. doi: 10.1007/BF02104905

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Langham E, Thorne H, Browne M, Donaldson P, Rose J, Rockloff M. Understanding gambling related harm: a proposed definition, conceptual framework, and taxonomy of harms. Bmc Public Health. (2016) 16:80. doi: 10.1186/s12889-016-2747-0

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Li E, Browne M, Rawat V, Langham E, Rockloff M. Breaking bad: comparing gambling harms among gamblers and affected others. J Gambling Studies. (2017) 33:223–48. doi: 10.1007/s10899-016-9632-8

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Yakovenko I, Quigley L, Hemmelgarn BR, Hodgins DC, Ronksley P. The efficacy of motivational interviewing for disordered gambling: systematic review and meta-analysis. Addict Behav. (2015) 43:72–82. doi: 10.1016/j.addbeh.2014.12.011

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Maynard BR, Wilson AN, Labuzienski E, Whiting SW. Mindfulness-based approaches in the treatment of disordered gambling. Res Soc Work Pract. (2015) 28:348–62. doi: 10.1177/1049731515606977

CrossRef Full Text | Google Scholar

12. Goslar M, Leibetseder M, Muench HM, Hofmann SG, Laireiter AR. Pharmacological treatments for disordered gambling: a meta-analysis. J Gambling Studies. (2019) 35:415–45. doi: 10.1007/s10899-018-09815-y

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Pallesen S, Molde H, Arnestad HM, Laberg JC, Skutle A, Iversen E, et al. Outcome of pharmacological treatments of pathological gambling: a review and meta-analysis. J Clin Psychopharm. (2007) 27:357–64. doi: 10.1097/jcp.013e3180dcc304d

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Goslar M, Leibetseder M, Muench HM, Hofmann SG, Laireiter AR. Efficacy of face-to-face versus self-guided treatments for disordered gambling: a meta-analysis. J Behav Addict. (2017) 6:142–62. doi: 10.1556/2006.6.2017.034

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Suurvali H, Hodgins D, Toneatto T, Cunningham J. Treatment seeking among Ontario problem gamblers: results of a population survey. Psychiatr Serv. (2008) 59:1343–6. doi: 10.1176/ps.2008.59.11.1343

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Suurvali H, Hodgins DC, Toneatto T, Cunningham JA. Hesitation to seek gambling-related treatment among Ontario problem gamblers. J Addict Med. (2012) 6:39–49. doi: 10.1097/ADM.0b013e3182307dbb

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Shaffer HJ, Blaszczynski A, Ladouceur R. Considering the public health and Reno Models: strategic and tactical approaches for dealing with gambling-related harms. Int J Mental Health Addict. (2020) 18:808–18. doi: 10.1007/s11469-019-00149-3

CrossRef Full Text | Google Scholar

18. Hancock L, Smith G. Critiquing the Reno Model I-IV international influence on regulators and governments (2004–2015) — the distorted reality of “responsible gambling”. Int J Mental Health Addict. (2017) 15:1151–76. doi: 10.1007/s11469-017-9746-y

CrossRef Full Text | Google Scholar

19. Delfabbro P, King DL. Blame it on reno: a commentary on hancock and smith. Int J Mental Health Addict. (2017) 15:1203–8. doi: 10.1007/s11469-017-9777-4

CrossRef Full Text | Google Scholar

20. Hancock L, Smith G. Replacing the Reno Model with a robust public health approach to “responsible gambling”: hancock and Smith's response to commentaries on our original Reno Model critique. Int J Mental Health Addict. (2017) 15:1209–20. doi: 10.1007/s11469-017-9836-x

CrossRef Full Text | Google Scholar

21. Young M, Markham F. Rehabilitating reno: a commentary on hancock and smith. Int J Mental Health Addict. (2017) 15:1187–92. doi: 10.1007/s11469-017-9795-2

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Shaffer HJ, Ladouceur R, Blaszczynski A, Whyte K. Extending the Reno Model: clinical and ethical applications. Am J Orthopsych. (2016) 86:297–309. doi: 10.1037/ort0000123

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Korn DA, Shaffer HJ. Gambling and the health of the public: adopting a public health perspective. J Gambl Studies. (1999) 15:289–365. doi: 10.1023/A:1023005115932

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Engebø J, Torsheim T, Mentzoni RA, Molde H, Pallesen S. Predictors of gamblers beliefs about responsible gambling measures. J Gambl Studies. (2019) 35:1375–96. doi: 10.1007/s10899-019-09835-2

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Ladouceur R, Shaffer P, Blaszczynski A, Shaffer HJ. Responsible gambling: a synthesis of the empirical evidence. Addict Res Theory. (2017) 25:225–35. doi: 10.1080/16066359.2016.1245294

CrossRef Full Text | Google Scholar

26. Sagoe D, Pallesen S, Griffiths MD, Mentzoni RA, Leino T. Does individual gambling behavior vary across gambling venues with differing numbers of terminals? An empirical real-world study using player account data. Front Psychol. (2018) 9:158. doi: 10.3389/fpsyg.2018.00158

CrossRef Full Text | Google Scholar

27. Palmer du Preez K, Landon J, Bellringer M, Garrett N, Abbott M. The effects of pop-up harm minimisation messages on electronic gaming machine gambling behaviour in New Zealand. J Gambl Studies. (2016) 32:1115–26. doi: 10.1007/s10899-016-9603-0

CrossRef Full Text | Google Scholar

28. Blaszczynski A, Collins P, Fong D, Ladouceur R, Nower L, Shaffer HJ, et al. Responsible gambling: general principles and minimal requirements. J Gambl Studies. (2011) 27:565–73. doi: 10.1007/s10899-010-9214-0

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Peter SC, Brett EI, Suda MT, Leavens ELS, Miller MB, Leffingwell TR, et al. A meta-analysis of brief personalized feedback interventions for problematic gambling. J Gambl Studies. (2019) 35:447–64. doi: 10.1007/s10899-018-09818-9

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Tanner J, Drawson AS, Mushquash CJ, Mushquash AR, Mazmanian D. Harm reduction in gambling: a systematic review of industry strategies. Addict Res Theory. (2017) 25:485–94. doi: 10.1080/16066359.2017.1310204

CrossRef Full Text | Google Scholar

31. Ginley MK, Whelan JP, Pfund RA, Peter SC, Meyers AW. Warning messages for electronic gambling machines: evidence for regulatory policies. Addict Res Theory. (2017) 25:495–504. doi: 10.1080/16066359.2017.1321740

CrossRef Full Text | Google Scholar

32. ^*Harris A, Parke A. The interaction of gambling outcome and gambling harm-minimisation strategies for electronic gambling: The efficacy of computer generated self-appraisal messaging. Int J Mental Health Addict. (2015) 14:597–617. doi: 10.1007/s11469-015-9581-y

CrossRef Full Text | Google Scholar

33. ^*Monaghan S, Blaszczynski A. Impact of mode of display and message content of responsible gambling signs for electronic gaming machines on regular gamblers. J Gambl Studies. (2010) 26:67–88. doi: 10.1007/s10899-009-9150-z

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Monaghan S. Review of pop-up messages on electronic gaming machines as a proposed responsible gambling strategy. Int J Mental Health Addict. (2007) 6:214–22. doi: 10.1007/s11469-007-9133-1

CrossRef Full Text | Google Scholar

35. ^*Auer M, Malischnig D, Griffiths MD. Is “pop-up” messaging in online slot machine gambling effective as a responsible gambling strategy? J Gambl Issues. (2014) 29:1–10. doi: 10.4309/jgi.2014.29.3

CrossRef Full Text | Google Scholar

36. ^*Auer MM, Griffiths MD. Testing normative and self-appraisal feedback in an online slot-machine pop-up in a real-world setting. Front Psychol. (2015) 6:1–7. doi: 10.3389/fpsyg.2015.00339

PubMed Abstract | CrossRef Full Text | Google Scholar

37. ^*McGivern P, Hussain Z, Lipka S, Stupple E. The impact of pop-up warning messages of losses on expenditure in a simulated game of online roulette: A pilot study. BMC Public Health. (2019) 19:822. doi: 10.1186/s12889-019-7191-5

PubMed Abstract | CrossRef Full Text | Google Scholar

38. ^*Tabri N, Hollingshead SJ, Wohl MJA. A limit approaching pop-up message reduces gambling expenditures, except among players with a financially focused self-concept. Int Gambl Studies. (2019) 19:327–38. doi: 10.1080/14459795.2019.1567806

CrossRef Full Text | Google Scholar

39. ^*Monaghan S, Blaszczynski A. Recall of electronic gaming machine signs: a static versus a dynamic mode of presentation. J Gambl Issues. (2007) 20:253–68. doi: 10.4309/jgi.2007.20.8

CrossRef Full Text | Google Scholar

40. ^*Stewart MJ, Wohl MJA. Pop-up messages, dissociation, and craving: How monetary limit reminders facilitate adherence in a session of slot machine gambling. Psychol Addict Behav. (2013) 27:268–73. doi: 10.1002/central/CN-00861976/full

PubMed Abstract | CrossRef Full Text | Google Scholar

41. ^*Wohl MJ, Gainsbury S, Stewart MJ, Sztainert T. Facilitating responsible gambling: the relative effectiveness of education-based animation and monetary limit setting pop-up messages among electronic gaming machine players. J Gambl Studies. (2013) 29:703–17. doi: 10.1007/s10899-012-9340-y

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies that evaluate health care interventions: Explanation and elaboration. PLoS Med. (2009) 6:e1000100. doi: 10.1371/journal.pmed.1000100

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Moher D, Liberati A, Tetzlaff J, Altman DG, Group TP. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. (2009) 6:e1000097. doi: 10.1371/journal.pmed.1000097

CrossRef Full Text | Google Scholar

44. ^*Jardin BF, Wulfert E. The use of messages in altering risky gambling behavior in experienced gamblers. Psychol Addict Behav. (2012) 26:166–70. doi: 10.1037/a0026202

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Auer MM, Griffiths MD. Personalized behavioral feedback for online gamblers: A real world empirical study. Front Psychol. (2016) 7:1875. doi: 10.3389/fpsyg.2016.01875

PubMed Abstract | CrossRef Full Text | Google Scholar

46. ^*Byrne CA, Russell AMT. Making EGMs Accountable: Can an informative and dynamic interface help players self-regulate? J Gambl Studies. (2020) 36:1229–51. doi: 10.1007/s10899-019-09889-2

PubMed Abstract | CrossRef Full Text | Google Scholar

47. ^*Hollingshead SJ, Amar M, Santesso D, Wohl MJA. When should players be taught to gamble responsibly? Timing of educational information upregulates responsible gambling intentions. Addict Res Theory. (2019) 27:507–14. doi: 10.1080/16066359.2018.1555818

CrossRef Full Text | Google Scholar

48. ^*Rockloff MJ, Donaldson P, Browne M. Jackpot expiry: An experimental investigation of a new EGM player-protection feature. J Gambl Studies. (2015) 31:1505–14. doi: 10.1007/s10899-014-9472-3

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Wohl MJA, Davis CG, Hollingshead SJ. How much have you won or lost? Personalized behavioral feedback about gambling expenditures regulates play. Comput Hum Behav. (2017) 70:437–45. doi: 10.1016/j.chb.2017.01.025

CrossRef Full Text | Google Scholar

50. Armstrong T, Donaldson P, Langham E, Rockloff M, Browne M. Exploring the effectiveness of an intelligent messages framework for developing warning messages to reduce gambling intensity. J Gambl Issues. (2018) 38:67–84. doi: 10.4309/jgi.2018.38.4

CrossRef Full Text | Google Scholar

51. Benhsain K, Taillefer A, Ladouceur R. Awareness of independence of events and erroneous perceptions while gambling. Addict Behav. (2004) 29:399–404. doi: 10.1016/j.addbeh.2003.08.011

CrossRef Full Text | Google Scholar

52. Floyd K, Whelan JP, Meyers AW. Use of warning messages to modify gambling beliefs and behavior in a laboratory investigation. Psychol Addict Behav. (2006) 20:69–74. doi: 10.1037/0893-164X.20.1.69

CrossRef Full Text | Google Scholar

53. ^*Ginley MK, Whelan JP, Keating HA, Meyers AW. Gambling warning messages: The impact of winning and losing on message reception across a gambling session. Psychol Addict Behav. (2016) 30:931–38. doi: 10.1037/adb0000212

CrossRef Full Text | Google Scholar

54. Kennedy CE, Fonner VA, Armstrong KA, Denison JA, Yeh PT, O'Reilly KR, et al. The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies. System Rev. (2019) 8:3. doi: 10.1186/s13643-018-0925-0

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Borenstein M, Higgins JPT, Hedges LV, Rothstein HR. Basics of meta-analysis: I2 is not an absolute measure of heterogeneity. Res Synthesis Methods. (2017) 8:5–18. doi: 10.1002/jrsm.1230

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Statist Med. (2002) 21:1539–58. doi: 10.1002/sim.1186

PubMed Abstract | CrossRef Full Text | Google Scholar

57. Giang HTN, Ahmed AM, Fala RY, Khattab MM, Othman MHA, Abdelrahman SAM, et al. Methodological steps used by authors of systematic reviews and meta-analyses of clinical trials: A cross-sectional study. BMC Med Res Methodol. (2019) 19:164. doi: 10.1186/s12874-019-0780-2

PubMed Abstract | CrossRef Full Text | Google Scholar

58. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane handbook for systematic reviews of interventions (2nd ed.). Chichester (UK): John Wiley & Sons (2019).

PubMed Abstract | Google Scholar

59. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta-Analysis. Chichester (UK): John Wiley & Sons Inc (2009) doi: 10.1002/9780470743386

PubMed Abstract | CrossRef Full Text | Google Scholar

60. Fu R, Gartlehner G, Grant M, Shamliyan T, Sedrakyan A, Wilt TJ, et al. Conducting quantitative synthesis when comparing medical interventions: AHRQ and the effective health care program. J Clin Epidemiol. (2011) 64:1187–97. doi: 10.1016/j.jclinepi.2010.08.010

PubMed Abstract | CrossRef Full Text | Google Scholar

61. Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. (2000) 56:455–63. doi: 10.1111/j.0006-341x.2000.00455.x

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Orwin RG. A fail-safe N for effect size in meta-analysis. J Educat Stat. (1983) 8:157. doi: 10.2307/1164923

CrossRef Full Text | Google Scholar

63. Cohen J. Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Abingdon: Routledge Ltd. (2013).

PubMed Abstract | Google Scholar

64. ^*Cloutier M, Ladouceur R, Sevigny S. Responsible gambling tools: Pop-up messages and pauses on video lottery terminals. J Psychol. (2006) 140:434–8. doi: 10.3200/JRLP.140.5.434-438

PubMed Abstract | CrossRef Full Text | Google Scholar

65. ^*Harris A, Kuss D, Griffiths MD. Gambling, motor cautiousness, and choice impulsivity: an experimental study. J Behav Addict. (2018) 7:1030–43. doi: 10.1556/2006.7.2018.108

PubMed Abstract | CrossRef Full Text | Google Scholar

66. ^*Mizerski R, Lee A, Sadeque S, Jolley W, Wang S, Jiang J, et al. Boomerang Effects of Gambling Warnings Exposed to Non-problem Gamblers ANZMAC 2012: Proceedings of Australian & New Zealand Marketing Academy, Adelaide, S.Aust. (2012). Available online at: http://dro.deakin.edu.au/view/DU:30076071

Google Scholar

67. ^*Wohl MJA, Parush A, Kim HS, Warren K. Building it better: Applying human–computer interaction and persuasive system design principles to a monetary limit tool improves responsible gambling. Comput Hum Behav. (2014) 37:124–32. doi: 10.1016/j.chb.2014.04.045

CrossRef Full Text | Google Scholar

68. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. (1977) 33:159–74. doi: 10.2307/2529310

PubMed Abstract | CrossRef Full Text | Google Scholar

69. Gainsbury SM, Russell A, Blaszczynski A. Are psychology university student gamblers representative of non-university students and general gamblers? A comparative analysis. J Gambl Studies. (2014) 30:11–25. doi: 10.1007/s10899-012-9334-9

PubMed Abstract | CrossRef Full Text | Google Scholar

70. Holt DD, Green L, Myerson J. Is discounting impulsive?: Evidence from temporal and probability discounting in gambling and non-gambling college students. Behav Processes. (2003) 64:355–67. doi: 10.1016/S0376-6357(03)00141-4

PubMed Abstract | CrossRef Full Text | Google Scholar

71. Gainsbury S, Blaszczynski A. The appropriateness of using laboratories and student participants in gambling research. J Gambl Studies. (2011) 27:83–97. doi: 10.1007/s10899-010-9190-4

PubMed Abstract | CrossRef Full Text | Google Scholar

72. Ladouceur R, Mayrand M, Tourigny Y. Risk-taking behavior in gamblers and non-gamblers during prolonged exposure. J Gambl Behav. (1987) 3:115–22. doi: 10.1007/BF01043450

CrossRef Full Text | Google Scholar

73. Lyons CA. Methodological considerations in the experimental analysis of gambling. In: Gambling: Behavior Theory, Research, and Application. Reno, NV: Context Press. (2006). p. 91–104.

Google Scholar

74. Gainsbury SM, Aro D, Ball D, Tobar C, Russell A. Optimal content for warning messages to enhance consumer decision making and reduce problem gambling. J Business Res. (2015) 68:2093–101. doi: 10.1016/j.jbusres.2015.03.007

CrossRef Full Text | Google Scholar

75. Irwin JR, McClelland GH, Schulze WD. Hypothetical and real consequences in experimental auctions for insurance against low-probability risks. J Behav Decis Mak. (1992) 5:107–16. doi: 10.1002/bdm.3960050203

CrossRef Full Text | Google Scholar

76. Kühberger A, Schulte-Mecklenbeck M, Perner J. Framing decisions: hypothetical and real. Organiz Behav Hum Decis Processes. (2002) 89:1162–75. doi: 10.1016/S0749-5978(02)00021-3

CrossRef Full Text | Google Scholar

77. Madden GJ, Begotka AM, Raiff BR, Kastern LL. Delay discounting of real and hypothetical rewards. Exp Clin Psychopharm. (2003) 11:139–45. doi: 10.1037/1064-1297.11.2.139

PubMed Abstract | CrossRef Full Text | Google Scholar

78. Ladouceur R, Gaboury A, Bujold A, Lachance N, Tremblay S. Ecological validity of laboratory studies of videopoker gaming. J Gambl Studies. (1991) 7:109–16. doi: 10.1007/BF01014526

PubMed Abstract | CrossRef Full Text | Google Scholar

79. Weatherly JN, Phelps BJ. The pitfalls of studying gambling behavior in a laboratory situation. In: P. Ghezzi, C. A. Lyons, M. R. Dixon, G. R. Wilson, editors. Gambling: Behavior Theory, Research, and Application. Reno, NV: Context Press. (2006). p. 105–125.

Google Scholar

80. Forsström D, Spångberg J, Petterson A, Brolund A, Odeberg J. A systematic review of educational programs and consumer protection measures for gambling: an extension of previous reviews. Addict Res Theory. (2020) 33:1–15. doi: 10.1080/16066359.2020.1729753. [Epub ahead of print].

CrossRef Full Text | Google Scholar

81. Monaghan S. Responsible gambling strategies for Internet gambling: the theoretical and empirical base of using pop-up messages to encourage self-awareness. Comput Hum Behav. (2009) 25:202–7. doi: 10.1016/j.chb.2008.08.008

CrossRef Full Text | Google Scholar

*^References marked with an asterisk indicate studies included in the meta-analysis.