Negative decision outcomes are more common among people with lower decision-making competence: an item-level analysis of the Decision Outcome Inventory (DOI)

Most behavioral decision research takes place in carefully controlled laboratory settings, and examination of relationships between performance and specific real-world decision outcomes is rare. One prior study shows that people who perform better on hypothetical decision tasks, assessed using the Adult Decision-Making Competence (A-DMC) measure, also tend to experience better real-world decision outcomes, as reported on the Decision Outcomes Inventory (DOI). The DOI score reflects avoidance of outcomes that could result from poor decisions, ranging from serious (e.g., bankruptcy) to minor (e.g., blisters from sunburn). The present analyses go beyond the initial work, which focused on the overall DOI score, by analyzing the relationships between specific decision outcomes and A-DMC performance. Most outcomes are significantly more likely among people with lower A-DMC scores, even after taking into account two variables expected to produce worse real-world decision outcomes: younger age and lower socio-economic status. We discuss the usefulness of DOI as a measure of successful real-world decision-making.

Introduction

Throughout their lives, people face decisions that affect their finances, health, and overall quality of life. Behavioral decision research aims to understand how people make decisions, with the ultimate goal of improving experienced decision outcomes. However, examination of these real-world decision outcomes is rare. As an experimental science, behavioral decision research uses laboratory settings deliberately isolated from everyday decisions. The hope is that tasks designed to capture fundamental properties of those decisions will predict real-world decision performance and outcomes. In Parker and Fischhoff (2005), we examined whether adolescents who did better on commonly used laboratory decision tasks also did better on presumptive precursors and consequences of good decision-making. We found good construct validity, a pattern that has borne up in a unique longitudinal data collection effort that followed the sample for 11 years (Parker et al., 2014). Other research has also linked performance on decision-making competence tasks to various cognitive abilities (e.g., Stanovich and West, 2000, 2008; Peters et al., 2006; Finucane and Gullion, 2010 Del Missier et al., 2012, 2013), need for cognition and other cognitive styles (e.g., Carnevale et al., 2011; Smith and Levin, 1996), less regret (Parker et al., 2007), and fewer suspensions among students (Stanovich et al., 2003). Moreover, decision-making competence has been improved as a result of targeted decision education among high-school students (Jacobson et al., 2012).

In Bruine de Bruin et al. (2007), we adopted complementary strategies for assessing decision-making competence and outcomes with adults. We first developed a composite measure of Adult Decision-Making Competence (A-DMC), using a set of hypothetical decision tasks drawn from behavioral decision research. We then validated performance on A-DMC against a composite score of decision outcomes as self-reported on the Decision Outcomes Inventory (DOI), which can be accessed through the online Decision-Making Individual Differences Inventory¹.

The DOI elicits self-reports of outcomes that could result from poorly made decisions. They range from serious (e.g., declared bankruptcy) to minor (e.g., got blisters from sunburn). The DOI was patterned after “life events” scales, which assess individuals' overall life stress through their self-reports of life events found to have adverse consequences (Masten et al., 1994; Brady and Matthews, 2002). We used a 10-year time frame to allow time for severe, and relatively uncommon, events to happen. Most items were preceded by screening questions (have you ever had a credit card?), establishing the possibility of experiencing a bad outcome if a poor decision were made (have you ever had $5000 in credit card debt?). The overall DOI score reflects avoiding more negative decision outcomes, such that higher scores reflect better performance. In Bruine de Bruin et al. (2007), we found that people who did better on hypothetical decision tasks (as seen in high A-DMC scores) also experienced better real-world decision outcomes (as captured by the DOI score).

The approach to DOI scoring in Bruine de Bruin et al. (2007) assumed that, across time, people, and decisions, good decision processes would predict good decision outcomes on average. That said, even the soundest decision-making processes cannot guarantee good outcomes. In uncertain situations, some unhappy surprises are inevitable even to good decision makers. Moreover, disadvantaged socio-economic backgrounds may increase exposure to negative life events (Brady and Matthews, 2002). We found a significant relationship between decision-making competence (as measured on A-DMC) and better decision outcomes (as reported on the DOI), even after controlling for socio-economic status (Bruine de Bruin et al., 2007).

Because the DOI was designed to capture a range of decision outcomes varying in severity, the question arises how much each contributed to the relationship between A-DMC and DOI. There are several reasons to suspect that some of the negative outcomes listed on the DOI might not be correlated to decision-making competence (or A-DMC). Specifically, it is possible that some negative outcomes actually reflect the best decision that can be made in a bad situation. Some outcomes are bad in one respect, but good in another. Moreover, some outcomes may not reflect one's own decision making. For example, getting divorced may be the best outcome for a bad marriage, be instigated by one's spouse, and involve a mix of positive experiences (i.e., starting anew) and negative ones (i.e., regret about having chosen a spouse poorly). Answering these concerns bears both on the interpretation of the DOI—is it capturing a unified construct of poor decision outcomes—and on future development of the DOI—item-level analysis could help inform future refinement or reduction in length.

Here, we go beyond our previous analyses and examine the relationships between A-DMC scores and the individual outcomes listed on the DOI. We also control for age and socio-economic status (SES), both potentially correlated with individuals' exposure to the possibility of negative events and ability to keep them from happening. Younger individuals may make more poor decisions because they have had less opportunity to acquire experience and may have greater struggles with impulsivity and social pressures (Reyna and Farley, 2006). Lower SES could force individuals to confront harder decisions, hence creating more opportunities for negative outcomes, and to have fewer resources for mitigating their effects (Brady and Matthews, 2002). In Parker and Fischhoff (2005), we found evidence suggesting that young people growing up in lower SES environments may have less opportunity to observe sound decision making and receive less consistent positive reinforcement for their own attempts to master those skills.

Methods

We re-analyzed data collected from 360 participants, recruited from the Pittsburgh community through social-service and other community organizations (Bruine de Bruin et al., 2007). This strategy created a sample that varied widely in socio-economic status (which we assessed with the proxy measure of the social-service organizations through which we recruited participants). Respondents ranged in age from 18 to 88, 74% were women, 66% self-identified as White, 28% as African American, and 6% as other. Forty percent reported having at least a bachelor's degree.

In group-administered sessions, participants completed both the DOI and the A-DMC battery. The DOI asked them to self-report on the 41 negative decision outcomes in Table 1. Thirty-five of these outcomes were preceded by filter questions asking if participants had been in a position to experience the outcome. For example, the question on whether the participant had missed a flight (the outcome) was preceded by a question asking whether they had taken a plane trip (the filter). Overall DOI scores were computed as the percentage of negative outcomes experienced, among those that could have happened².

TABLE 1

Table 1. Outcomes captured by the Decision-Outcomes Inventory for low and high A-DMC subsamples (among those for whom each was possible).

A-DMC has six hypothetical decision tasks: Resistance to Framing, Recognizing Social Norms, Under/overconfidence, Applying Decision Rules, Consistency in Risk Perception, and Resistance to Sunk Costs (the full instrument can be found at the online Decision-Making Individual Differences Inventory³). Responses are scored in terms of either consistency with one another or correspondence to an external criterion. Here we use a summary A-DMC score, equal to an unweighted mean of the standardized task scores, standardized to have zero mean and standard deviation of one. We use a median split in Table 1 as an easy way to present the differences in performance among individuals with higher and lower scores.

This research was approved by the Carnegie Mellon University Institutional Review Board, and all data collection conformed to U.S. human subjects regulatory standards. Informed consent was obtained from all subjects.

Analysis Plan

We first summarize the rate of experiencing each DOI outcome for those with relatively low and high A-DMC (Table 1). We then (in Table 2) use bivariate and multivariate logistic regression to predict each DOI outcome and the overall DOI score with A-DMC (using the full range), age, and SES⁴.

TABLE 2

Table 2. Odds of outcomes on the Decision-Outcomes Inventory as a function of A-DMC, age, and SES.

This paper is accompanied by supplementary material that provides additional detail on the relationships between individual DOI outcomes and individual A-DMC subscales. The pattern of results broadly support the relationships reported below, as well as correlations of overall DOI with DMC subscales (Bruine de Bruin et al., 2007).

Results

How is A-DMC Related to Individual DOI Outcomes?

Table 1 shows the frequency of each negative DOI outcome, for participants with lower and higher DMC scores (among those for whom that outcome was possible), ordered by the difference in percentages. The first eight items are ones for which low A-DMC appears protective, in the sense of being reported less often by low A-DMC participants. The two statistically significant differences (p < 0.05) show that low-A-DMC participants are less likely to have more than $5000 credit card debt and to have gotten lost while driving. The other 33 outcomes are less likely for participants with high A-DMC, significantly so in most cases. On average, the outcomes are 7.5% less likely for high A-DMC participants. Thus, if A-DMC is accepted as a measure of decision-making ability, then most DOI outcomes are indicators of poor decision making. Nonetheless, the exceptions are puzzling and perhaps partially explained by the analyses that follow.

Do DOI Outcomes Reflect Factors other than Decision-Making Competence?

Table 2 considers the effects of two non-decision factors on DOI outcomes: age and SES. The results are odds ratios from logistic regression analyses, using the full range of the A-DMC scores (rather than a binary variable), normalized so that each unit increment represents one standard deviation. The bivariate odds express the difference in the prevalence of the DOI outcome in odds ratios terms, with ratios less than one meaning that the (bad) outcome is more likely for participants with lower A-DMC scores. As would be expected from Table 1, the first eight items have odds ratios greater than one⁵. The next column shows the odds ratio for the A-DMC scores, after controlling for age and SES. The overall pattern is largely unchanged. Three quarters of the odds ratios are less than one, and often significantly so. Nonetheless, most are closer to one than in the previous column, indicating that these controls are related to the decision outcomes.

The final two columns show odds ratios for age and SES, controlling for each other and A-DMC. Age is scaled continuously, with each unit equal to 10 years to show differences better⁶. The odds ratios for age are almost all less than one, indicating that the DOI outcomes are more likely for younger respondents (as mentioned, controlling for A-DMC and SES). For example, the younger respondents are 14% (1/0.88 – 1) more likely to report having bought and never worn clothing or shoes in the preceding 10 years (the reporting period for these items) and almost twice as likely to have been kicked out of a bar, restaurant, or hotel (the smallest ratio). In the final column (Low SES), about three-quarters of the odds ratios are greater than one, indicating that the outcome is more likely for lower-SES individuals. For example, they are 66% more likely not to have worn shoes or clothing that they bought (row 1) and almost 13 times more likely to report having been diagnosed with an STD (the highest ratio) (controlling for A-DMC and SES).

The final row of Table 2 presents standardized linear regression coefficients for predicting the overall percentage of negative outcomes that participants reported experiencing (among those possible for each individual) with A-DMC, age, and SES. The results show the strongest effect for age, with both A-DMC and SES showing significant marginal effects. However, controlling for SES has a much stronger impact on the A-DMC coefficient than does controlling for age, as A-DMC correlates strongly with SES (r = −0.52, p < 0.001) but not with age (r = 0.04, ns).

Discussion

Here, we set out to address concerns about whether or not individual items on the Decision Outcome Inventory (DOI) are indicative of decision-making competence. We found that most of the DOI outcomes are significantly more likely among participants with lower A-DMC scores, suggesting that these outcomes reflect, in part, poor decision-making competence. Many of these outcomes are also related to two variables that would be expected to produce worse outcomes: being young and being poor. Indeed, most DOI outcomes are more common among younger and lower SES participants. Young people have had less opportunity to learn how to make decisions. Lower SES individuals may face more perilous circumstances that expose them to higher chances of negative outcomes (e.g., taking risks to avoid crime or save money). Both may have fewer resources for limiting damage (e.g., legal representation). Parker and Fischhoff (2005) found that lower SES young adults had lower decision-making competence for reasons potentially related to their opportunities to acquire those skills (e.g., stable home environment). Thus, SES may affect both the decisions that individuals face and their ability to cope with them.

If age and SES are viewed as factors outside of individuals' control, then the correlation between DOI outcomes and A-DMC after controlling for them might indicate the degree to which those outcomes are under individuals' control. In that light, being evicted, quitting a job after a week, being suspended from school, getting five or more speeding tickets, and having a mortgage or loan foreclosed appear to be the bad outcomes that good decision making can do the most to avoid. These may also be the individual outcomes for which DOI has the clearest interpretation as reflecting poor decision-making competence.

Age has the strongest independent relationship to DOI, even after taking into account potential differences in A-DMC and SES. We found age differences for several outcomes: having a large credit card debt, having a check bounce, drinking enough alcohol to vomit, ruining clothes by not following laundry instructions, and being in a public fight or screaming argument Each of these outcomes may reflect the higher risks faced by younger people. Low SES has the strongest independent relationship to DOI (controlling for A-DMC and age) for outcomes involving economic problems potentially outside individuals' control, where decision-making competence cannot overcome circumstantial constraints: having a check bounce, declaring bankruptcy, and being behind in rent or mortgage payments. Those are followed by having a mortgage or loan foreclosed, being in jail overnight, being diagnosed with an STD, and having an unplanned pregnancy.

Suggestions for Future Research

The DOI was developed as an outcome assessment that (a) is convenient for administration in paper-and-pencil or web-based formats and (b) targets likely results of poor decision-making processes. The present analyses support its validity in two ways. One is that most of the individual outcomes are more common among individuals with lower scores on A-DMC. The second is that poor outcomes are more common among younger people and those from lower SES backgrounds, in predictable ways.

Although most outcomes show this pattern, some do not. One possible way to improve the measure is by reducing it to those items that performed best. A second is to refine the items that performed most poorly. For example, the 10-year reporting period may have taken younger respondents into time periods for which the outcome was irrelevant (e.g., accumulating $5000 in credit card debt while in high school). A related response is to create items targeting specific populations (e.g., fraud victimization for the elderly). Similarly, future work could examine domain-specific item sets (e.g., financial or health), similar to methods used in developing the Domain-Specific Risk-Taking (DOSPERT) scale (Blais and Weber, 2006). We would like to see the present analyses repeated with new samples, new items, and additional covariates having plausible relationships to decision outcomes. We would also like to see a complementary scale with good outcomes. Finally, DOI relies on the accuracy (and candor) of people's self-reports of negative outcomes⁷. We were surprised by how many poor outcomes participants reported. We speculate that having so many things that could go wrong made it easier to admit some as personal failings. Nonetheless, independent validation of some outcomes would be valuable, perhaps using existing panel studies that collect diverse data on health, finances, and other domains. Despite these limitations, the DOI, in whole or in part, may provide an efficient way to assess participants' success in decision making, given the constraints of their age and SES.

Author Contributions

All three authors contributed to the design, data acquisition, analysis, interpretation, and writing for the research described here.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

This research was supported in part by the National Science Foundation (SES 0213782), National Institute on Drug Abuse (P50 DA05605, R01 DA033322) and the European Union Seventh Framework Programme (Marie Curie Career Integration Grant 618522). The views expressed are those of the authors and do not necessarily represent the views of these funders.

Supplementary Material

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fpsyg.2015.00363/abstract

Footnotes

1. ^http://www.sjdm.org/dmidi/Decision_Outcome_Inventory.html

2. ^Here, we focus on the unweighted percentage, which is highly correlated with the weighted percentage (r = 0.92, p < 0.001) also reported by Bruine de Bruin et al. (2007), with which outcomes are weighted by the reciprocal of their frequency—assuming that rarer outcomes are worse. Hence, here overall DOI can be thought of as a simple summary of the 41 individual outcomes, rather than one emphasizing uncommon (and more severe) outcomes.

3. ^http://www.sjdm.org/dmidi/Adult_-_Decision_Making_Competence.html

4. ^We also considered gender, but chose to leave it out of the reported analyses, since we did not have strong a priori expectations for a direction of effect. That said, the results for gender were quite modest, especially considering the number of tests. In total, six outcomes (suspended from school, lost driver's license, kicked out of bar, drank so much you vomited, DUI, and public fight) were more likely among men (at p < 0.05) and only one outcome (bought new clothes or shoes you never wore) was more likely among women.

5. ^The tests result in slightly different significance levels than in Table 1, largely due to the use of the full range of the A-DMC variable, rather than a binarized version.

6. ^The marginal effect of a single year in age is quite small, hence we divided age by 10 to provide odds ratios that are easier to interpret.

7. ^In contrast, A-DMC is a performance metric, and is therefore less susceptible to self-report biases. Hence, the fact that the two correlate, without shared method variance, adds support to the validity of the DOI responses.

References

Blais, A. R., and Weber, E. U. (2006). A domain-specific risk-taking (DOSPERT) scale for adult populations. Judg. Dec. Making 1, 33–47.

Google Scholar

Brady, S. S., and Matthews, K. A. (2002). The influence of socioeconomic status and ethnicity on adolescents' exposure to stressful life events. J. Pediatr. Psychol. 27, 575–583. doi: 10.1093/jpepsy/27.7.575

PubMed Abstract | Full Text | CrossRef Full Text | Google Scholar

Bruine de Bruin, W., Parker, A. M., and Fischhoff, B. (2007). Individual differences in adult decision-making competence. J. Pers. Soc. Psychol. 92, 938–956. doi: 10.1037/0022-3514.92.5.938

PubMed Abstract | Full Text | CrossRef Full Text | Google Scholar

Carnevale, J. J., Inbar, Y., and Lerner, J. S. (2011). Individual differences in need for cognition and decision-making competence among leaders. Pers. Indiv. Diff. 51, 274–278. doi: 10.1016/j.paid.2010.07.002

CrossRef Full Text | Google Scholar

Del Missier, F., Mäntylä, T., and Bruine de Bruin, W. (2012). Decisionmaking competence, executive functioning, and general cognitive abilities. J. Behav. Decis. Making 25, 331–351. doi: 10.1002/bdm.731

PubMed Abstract | Full Text | CrossRef Full Text | Google Scholar

Del Missier, F., Mäntylä, T., Hansson, P., Bruine de Bruin, W., Parker, A., and Nilsson, L.-G. (2013). The manifold relationship between memory and decision making: an individual-differences study. J. Exp. Psychol. Learn. Mem. Cogn. 39, 1344–1364. doi: 10.1037/a0032379

PubMed Abstract | Full Text | CrossRef Full Text | Google Scholar

Finucane, M. L., and Gullion, C. M. (2010). Developing a tool for measuring the decision-making competence of older adults. Psychol. Aging 25, 271–288. doi: 10.1037/a0019106

PubMed Abstract | Full Text | CrossRef Full Text | Google Scholar

Jacobson, D., Parker, A. M., Spetzler, C., Bruine de Bruin, W., Hollenbeck, K., Heckerman, D., et al. (2012). Improved learning in U.S. history and decision competence with decision-focused curriculum. PLoS ONE 7:e45775. doi: 10.1371/journal.pone.0045775

PubMed Abstract | Full Text | CrossRef Full Text | Google Scholar

Masten, A. S., Neemann, J., and Andenas, S. (1994). Life events and adjustment in adolescents: the significance of event independence, desirability, and chronicity. J. Res. Adol. 4, 71–97. doi: 10.1207/s15327795jra0401_5

CrossRef Full Text | Google Scholar

Parker, A. M., and Fischhoff, B. (2005). Decision-making competence: external validation through an individual-differences approach. J. Behav. Decis. Making 18, 1–27. doi: 10.1002/bdm.481

CrossRef Full Text | Google Scholar

Parker, A. M., Bruine de Bruin, W., and Fischhoff, B. (2007). Maximizers vs. satisficers: Decision-making styles, competence, and outcomes. Judg. Decis. Making 2, 342–350.

Google Scholar

Parker, A. M., Bruine de Bruin, W., Fischhoff, B., and Weller, J. A. (2014). “Robustness of decision-making competence: evidence from an 11-year longitudinal study,” in Paper Presented to the Annual Meetings of the Society for Judgment and Decision Making, (Long Beach, CA).

Peters, E., Västfjäll, D., Slovic, P., Mertz, C. K., Mazzocco, K., and Dickert, S. (2006). Numeracy and decision making. Psychol. Sci. 17, 407–413. doi: 10.1111/j.1467-9280.2006.01720.x

PubMed Abstract | Full Text | CrossRef Full Text | Google Scholar

Reyna, V. F., and Farley, F. (2006). Risk and rationality in adolescent decision making implications for theory, practice, and public policy. Psychol. Sci. Pub. Int. 7, 1–44. doi: 10.1111/j.1529-1006.2006.00026.x

CrossRef Full Text | Google Scholar

Smith, S. M., and Levin, I. P. (1996). Need for cognition and choice framing effects. J. Behav. Decis. Making 9, 283–290.

PubMed Abstract | Full Text | Google Scholar

Stanovich, K. E., Grunewald, M., and West, R. F. (2003). Cost– benefit reasoning in students with multiple secondary school suspensions. Personal. Indiv. Diff. 35, 1061–1072. doi: 10.1016/S0191-8869(02)00318-5

CrossRef Full Text | Google Scholar

Stanovich, K. E., and West, R. F. (2000). Individual differences in reasoning: implications for the rationality debate? Behav. Brain Sci. 23, 645–726. doi: 10.1017/S0140525X00003435

PubMed Abstract | Full Text | CrossRef Full Text | Google Scholar

Stanovich, K. E., and West, R. F. (2008). On the relative independence of thinking biases and cognitive ability. J. Pers. Soc. Psychol. 94, 672–695. doi: 10.1037/0022-3514.94.4.672

PubMed Abstract | Full Text | CrossRef Full Text | Google Scholar