Narrative Case Notes Have the Potential to Predict Seclusion 3 Days in Advance: A Mixed-Method Analysis

Objectives: Current risk assessment tools can predict problematic behavior and the need for coercive measures, but only with a moderate level of accuracy. The aim of this study was to assess antecedents and triggers of seclusion.

Methods: Narrative notes of health care professionals on psychiatric inpatients were analyzed daily starting 3 days prior to seclusion in the case group (n = 26) and compared to a matched control group without seclusion (n = 26) by use of quantitative and qualitative research methods, based on qualitative content analysis.

Results: Quantitative measures showed more aggression in the case group with highly significant differences between the groups (p < 0.001) at all measurement times. Seclusion was significantly associated with the total word count of the narrative notes. Subjective emotional expressions by staff were more apparent before seclusion (p = 0.003). Most frequently, subjective expressions regarding “arduous/provocative” (p < 0.001) and “anxious” (p = 0.010) sentiments could be identified in the case group. Description of patients' behavior in the case group included more negatively assessed terms (p = 0.001). Moreover, sleep loss, refusing medication, high contact frequency, demanding behavior and denied requests were present in a significantly higher frequency before seclusion. Expressions like “threatening” (p = 0.001) were found only before seclusion and appeared to have the function of personal risk assessment. The expression “manageable” (p = 0.035) appeared often in difficult situations that could still be handled.

Conclusion: Several factors preceding seclusion could be identified. Narrative notes of staff already showed differences 3 days before the escalation. Particularly the word count, the analysis of terms describing patients' behavior, subjective expressions of staff, and terms used as a function of personal risk assessment could help to provide better predictions of aggressive incidents and to prevent coercive measures.

Introduction

Violence and aggression in mental health care units represent an imminent danger for patients and health care professionals (HCP). In high-income countries, almost one in five acutely admitted psychiatric inpatients commit an act of physical violence (1). Aggression defined as verbal or physical abuse and intimidation (2) should be identified early to prevent further escalation and coercion, which may cause severe psychological distress and is potentially traumatic for both patients (3–5) and staff (6–8).

A number of risk assessment tools to predict violence and reduce severe incidents or the need for coercion are available, but only reach a moderate level of predictive accuracy (9). Correspondingly, a variety of antecedents associated with aggressive behavior have been found in previous investigations (10), encompassing patient symptoms, behavioral and emotional cues, interactional factors (patient-patient or patient-staff interaction), external issues, and structural issues (e.g., environmental factors within wards). In a systematic review and meta-analysis, Dack et al. (11) summarized that the following patient related clinical factors were known to be associated with aggressive behavior: psychiatric in-patients being male, of young age, not married, involuntarily admitted, diagnosed with schizophrenia, having a greater number of previous admissions, a history of violence, self-destruction, and substance abuse. However, patient factors do not entirely explain the variation in the occurrence of dangerous behavior and its management, and a multifactorial cause should be considered instead, also—within others—including staff related factors (11, 12).

Indeed, staff-patient interactions can constitute an important precursor of aggressive incidents (13–15). Perceptions, attitudes, and emotional reactions of staff and patients are considered to affect interactional behavior preceding aggressive incidents (15–19). An example for problematic behavior given by Bowers et al. (20) describes health care professional frustration due to repetitive requests at inappropriate times concerning a trivial item. The type of management that is used to deal with problematic patient behavior influences the interactional response and may lead to an escalation (8, 21–23). Escalation can, e.g., be triggered by a controlling (15, 24, 25), over-confident, and punitive management approach by staff (26–29). In addition, restrictive regiments like limiting patients' freedom or denying patients' requests appear to be an essential antecedent of aggressive behavior (10, 14, 20, 30, 31).

Further, healthcare professionals' emotions and subjective perceptions can contribute to de-escalation and escalation of problematic behavior and could potentially be used to improve risk assessment and prediction. For example, emotions and exposure to patient aggression have been investigated in a cross-sectional questionnaire survey of nurses by Jalil et al. (32), who revealed a positive relationship between nurse-reported anger and exposure to—mostly verbal—patient aggression as well as endorsement of coercive violence interventions. Greater work experience was associated with more tolerance toward aggressive behavior (13), probably due to a better in-depth understanding (20, 33). Furthermore, a less tolerant attitude was related to a higher state of burnout traits like emotional exhaustion, depersonalization and personal accomplishment and an inappropriate management of aggressive incidents (13, 34). In contrast, staff on acute inpatient ward is frequently confronted with challenging patient behavior, which may among others result in a higher tolerance for mild-to-moderate aggression with the possible consequence of missing early warning signs. This might be accompanied by the tendency for underreporting of aggressive incidents (35). In summary, there is evidence for a possible role of staff emotions and subjective experience in the causation and prediction of problematic behavior and coercive measures, but this topic is currently under-researched.

The current study differs from preexisting studies by investigating precursors of seclusion in narrative case notes. Narrative notes contain a wide range of information and were used in previous studies as a source to capture specific information not otherwise available, e.g., patients' problem behavior (12, 19, 20, 36). Natural language processing has the potential power of identifying meaningful predictors in narrative notes (37, 38). In fact, there is evidence supporting this approach for the prediction of suicide from a comprehensive study by McCoy et al. (39). Using computerized natural language processing, the authors were able to investigate 845,417 discharges in a retrospective investigation and could show that cases with the expression of positive sentiments in discharge notes were less likely to attempt suicide.

The aim of the present study was to determine antecedents of seclusion by analyzing narrative notes of HCP in a group of patients with seclusion in comparison to a control group without seclusion in a quantitative and qualitative mixed-methods retrospective case-control study, examining the course of inpatient treatment and staff-patient interaction on a daily basis starting from 3 days prior to seclusion.

Methods

Inclusion and Exclusion Criteria

All inpatient cases admitted to the Department of Adult Psychiatry of the Psychiatric University Hospital Basel, Switzerland (Universitäre Psychiatrische Kliniken Basel (UPK), Klinik für Erwachsene, Universität Basel, Switzerland) between October 2014 and May 2016 were eligible for entry in the current study. To be included in the case group, persons had to have undergone seclusion during their inpatient treatment. Cases with < 3 days of inpatient treatment prior to seclusion, or without documentation on one or more of the 3 days prior to seclusion (e.g., because of absconding during this period) were excluded. For patients with repeated hospitalizations with seclusions during the observation period, only the first hospitalization with seclusion compatible with in- and exclusion criteria was entered. During the 20-months observation period, 26 inpatient cases were available for analysis.

For inclusion in the control group, 26 inpatient cases hospitalized during the observation period matched for approximate age (±6 years), gender, and main diagnosis according to ICD-10 chapter V main group were randomly selected. In total, 570 case notes from the documentation of 52 patients were included, M = 10.96, Min = 4, Max = 17, SD = 2.4.

Data Collection and Analysis

The following sociodemographic and clinical data of patients were collected from electronic health records (EHR): gender, age, nationality, main diagnosis according to ICD-10, comorbid substance use disorder (SUD), type of admission (compulsory or voluntary), number of previous hospitalizations, duration of inpatient treatment, marital status, highest educational attainment, and current employment status.

Narrative notes of staff consisting of nursing staff, physicians, psychologists, and other HCP were extracted from the EHR on a daily basis starting from 3 days prior to seclusion (day −3, day −2, and day −1) and at the day of seclusion before escalation (day 0). A day was defined as starting with the first entry of the staff morning shift and ending with the last entry from staff night shift. In cases with insomnia where escalation and seclusion occurred during the early morning hours of day 0, the night prior to seclusion was treated as beginning of day 0. For the control group, the treatment day on which seclusion was performed for the case was chosen as day 0 for the control. For example, if the case was subjected to seclusion on the seventh day of inpatient treatment (day 0 for the case), seventh day of the control was defined as day 0 for the control, and narrative notes of 4 days (days −3 to 0) were extracted. In cases where EHR notes were missing during this episode (e.g., because the patient had absconded), the closest possible day toward day 0 in the case group was chosen with a maximum tolerance of ±3 days.

Whole-text-analysis of narrative notes was repeated several times and performed systematically via qualitative content analysis according to Mayring (40) using a structural method approach and frequency analysis. Categories and codes were defined inductively and deductively applying the qualitative data analysis software MAXQDA12 (VERBI Software GmbH, Berlin, Germany). Qualitative coding methods as described by Saldaña (41) contributed to define categories and codes (e.g., magnitude coding, simultaneous coding, structural coding, descriptive coding, in-vivo coding, process coding, concept coding, emotion coding, evaluation coding, hypothesis coding, causation coding, values coding, pattern coding, theme coding). CS performed initial data analysis. A second experienced reviewer (physician), HCP of inpatient wards consisting of nurses, a psychologist, a physician of a psychiatric intensive care unit, and another health care professional experienced in qualitative research supplemented the coding process: comments were applied, codes and themes were discussed, and decision-making was made by consensus. Furthermore, CS and JS rated a subset of 30 narrative notes independently regarding the categories “staff subjectivity,” “sleep behavior,” “demanding behavior,” “compliance,” and “high contact frequency,” since these categories entail a higher risk for rater subjectivity. The average kappa corresponded to 0.86 ranging from substantial to almost perfect interrater reliability (κ = 0.63–1.00) (42) with 23/33 (70%) items having a Kappa > 0.80.

Quantitative Instruments

In addition to qualitative content analysis, the total number of words in staff documentation and the number of individual notes was calculated on a daily basis for days −3, −2, and −1. Word count and number of notes for day 0 were not assessed as seclusion occurred at different times, and thus the time period and amount of documentation on day 0 prior to seclusion showed considerable variation depending mainly on the time of seclusion, but not necessarily on differences in staff documentation. Moreover, the most frequently used terms were analyzed regarding the number of patients with at least one occurrence of the term and the number of days with at least one occurrence.

Furthermore, the following quantitative instruments were rated for each observed day: The Modified Overt Aggression Scale (MOAS) (43) comprising the four dimensions verbal aggression, aggression against property, auto-aggression, and physical aggression with five items per dimension; the Positive and Negative Syndrome Scale–Excited Component (PANSS-EC) (44) with the five items poor impulse control, tension, hostility, uncooperativeness, and excitement, rated on a 7-point Likert scale; and the Clinical Global Impressions–Severity of Aggression scale (CGI-A) (45–47), a rating for the global assessment of a patient's aggressiveness, ranging from no aggressive behavior, slight aggressive behavior, moderate aggressive behavior, and severe aggressive behavior to aggressive behavior present. Both the OAS (48) and the PANSS-EC have been validated for retrospective use (45). Concerning the psychometric properties of the instruments used in this study, an acceptable to good internal consistency could be achieved. In this study, ratings using the MOAS reached a Cronbach's alpha of 0.759 at Day −3, of 0.786 at Day −2, and of 0.801 at Day −1. Cronbach's alpha of the PANSS-EC reached 0.821 at Day −3, 0.822 at Day −2, and was 0.831 at Day −1.Cronbach's alpha was not determined for the CGI-A, as it is a single-item rating instrument.

Statistical Analysis

Descriptive statistics were performed for sociodemographic and clinical characteristics, quantitative measures of patient aggression, staff documentation word count, and frequently used terms. Shapiro-Wilk tests were used to test normality. χ²-tests were applied to test group differences for categorical data, t-tests were used for normally distributed continuous data differences, and Mann-Whitney-U-test was performed for not normally distributed continuous data.

In addition, logistic regression was performed to examine the predictive power of the variables “word count,” “positive subjectivity,” “unpredictable,” “sleep irregularities,” and “manageable” on membership in the case group. p < 0.05 was considered significant, and only two-tailed tests were used. Pearson's r was calculated as a measure of effect size according to Cohen (49). All calculations were performed using the software IBM SPSS Statistics version 24 (IBM Corporation, Armonk, NY, USA).

Results

Sample Description

Sociodemographic and clinical characteristics of the case group, the control group, and the total sample are listed in Table 1.

TABLE 1

Table 1. Clinical and sociodemographic characteristics.

Concerning the criteria used for matching cases and controls, included patients had a mean age of 39.7 (range: 19–64) years, 65.4% (34/52) of the patients were male, and 73.2% (38/52) had a main diagnosis of schizophrenia-spectrum disorder.

In the case group, compulsory admission was more frequent (61.5 vs. 26.9%, p = 0.012). Concerning comorbid SUD, more patients with a current addictive disorder (42.3 vs. 26.9%) and fewer patients without SUD (50.0 vs. 65.4%) were present in the case group, but these differences were not statistically significant. In addition, no significant differences emerged concerning the nationality, duration of inpatient treatment, marital status, number of previous hospitalizations, highest educational attainment, and current employment previous to hospitalization.

Measures of Aggressive Behavior

Patients in the case group showed more aggressive behavior than patients in the control group as assessed with the MOAS, PANSS-EC and CGI-A (cf. Table 2). Between-group differences were highly statistically significant (p < 0.001) for each observation day beginning from day −3, with an effect size of r = 0.4–0.75 (medium to high effect size). Furthermore, an increase of aggression scores from day −3 to day 0 became evident for the case group for all quantitative measurements.

TABLE 2

Table 2. Quantitative measures of aggression in the case and control groups over the course of the observation period (days −3 to day 0).

Qualitative and Quantitative Analysis of Narrative Notes

Qualitative analysis revealed 112 variables per observational day, and 400 variables in total. According to the topics covered by the extracted variables, the main focus of the analysis was placed on expressions of “staff subjectivity,” “terms describing patients' behavior,” terms associated with “risk assessment” and “sleep behavior,” “demanding behavior,” “requests,” “high contact frequency,” and “non-compliance.” Example quotes can be found in Table 3.

TABLE 3

Table 3. Example quotes.

Staff Subjectivity

Documentation of patients contains primarily descriptions of their behavior, but subjective statements by staff related to their own perception of patient behavior were also discernible. Staff statements that were accompanied with staff emotional involvement became apparent in narrative notes due to distinct subjective expressions and style of phrasing. Subjectivity was grouped into statements with positive valence containing “enthusiastic” and “compassionate” sentiments, or with negative valence containing “provoked/arduous,” “anxious,” and “pejorative” sentiments (cf. Table 4).

TABLE 4

Table 4. Staff subjectivity.

In total, subjective statements were three times more apparent in the case than in the control group, M (case group) = 1.8, SD = 1.6; M (control group) = 0.6, SD = 0.9; U₍₅₂₎ = 182.5, p = 0.003. Negative valence statements were only apparent in the case group, M (case group) = 1.3, SD = 1.5; M (control group) = 0, SD = 0; U₍₅₂₎ = 130.0, p < 0.001, but there were no significant differences concerning the frequency of positive valence statements that were present in both groups, M (case group) = 0.6, SD = 0.8; M (control group) = 0.6, SD = 0.9; U₍₅₂₎ = −0.162, p = 0.850.

Specifically, statements regarding “provoked/arduous” sentiments, M (case group) = 0.8, SD = 1.1; M (control group) = 0, SD = 0; U = 182.0, p < 0.001, and “anxious” sentiments, M (case group) = 0.3, SD = 0.6; M (control group) = 0, SD = 0; U₍₅₂₎ = 260.0, p = 0.010, were significantly more present in the case group. “Compassionate,” M (case group) = 0.5, SD = 0.7; M (control group) = 0.5, SD = 0.8; U₍₅₂₎ = 312.0, p = 0.564, “pejorative,” M (case group) = 0.2, SD = 0.5, M (control group) = 0, SD = 0; U₍₅₂₎ = 242, p = 0.057, and “enthusiastic” sentiments, M (case group) = 0.1, SD = 0.3; M (control group) = 0.2, SD = 0.5; U₍₅₂₎ = 269.0, p = 0.506, were documented in both groups with no significant differences in frequency.

Terms Describing Patients' Behavior

In both groups, staff used distinctive terms to describe a patient's current behavior. In total, 26 terms could be identified that appeared at least 8 times during the observation period. Behavioral terms were analyzed by their emotional value (positive, negative, or other) and assessed regarding their potential association with problematic behavior leading to escalation and ultimately seclusion as estimated by the context of their use (cf. Table 5).

TABLE 5

Table 5. Most frequently used behavioral terms.

Staff described patients with behavioral terms significantly more often in the case group than in the control group, M (case group) = 13.8, SD = 6.7; M (control group) = 8.2, SD = 4.3; t₍₅₂₎ = −3.5, p = 0.001. The description of patients' behavior in the case group included significantly more terms with negative valence, and these terms were potentially related to problematic behavior, M (case group) = 9.2, SD = 6.5; M (control group) = 1.5, SD = 2.4; U₍₅₂₎ = 80.0, p < 0.001.

More precisely, terms like “agitated,” “irritable,” “loud/screaming,” “obtrusive,” “restless,” “threatening,” “dysphoric,” “insulting/cursing,” “aggressive,” “bizarre/foolish,” and “provocative” emerged with a significantly higher frequency in the case group. The most frequently represented behavioral term in the case group was “agitated.”

Patients in the control group were significantly more often perceived to be “adequate,” “organized” and “unchanged,” but were described as “psychotic,” “volatile,” “distracted,” “reclusive,” and “sad” in their behavior in equal frequency as in the case group. Positively assessed terms like “friendly,” “calm,” “relaxed,” “reachable,” “good mood,” and “cooperative” appeared frequently in both groups. However, in total, terms with positive sentiments appeared significantly more often in the control group, M (case group) = 3.3, SD = 2.2; M (control group) = 5.0, SD = 3.4; t₍₅₂₎ = 2.1, p = 0.039. The term “friendly” was the most common behavioral term in the control group, but was not present with a significantly higher frequency compared to the case group.

Risk Assessment

Expressions like “threatening,” M (case group) = 0.5, SD = 0.7; M (control group) = 0, SD = 0; U₍₅₂₎ = 208.0, p = 0.001, or “unpredictable,” M (case group) = 0.1, SD = 0.3; M (control group) = 0, SD = 0; U₍₅₂₎ = 299.0, p = 0.077, were used more often before an aggressive event or escalation in the case group. They appeared to function as a surrogate of personal risk assessment, and can be considered as a personal assessment if seclusion will become necessary. The term “threatening” shows an increased use in the time preceding seclusion. These expressions were not used by staff documenting the course of treatment in the control group.

The term “manageable,” M (case group) = 0.8, SD = 1.1; M (control group) = 0.3, SD = 0.5; U₍₅₂₎ = 240.0, p = 0.035, shows a similar pattern; it is used more often in the case group, particularly in the context of problematic behavior that could still be handled.

Sleep Behavior

Sleep behavior was coded as sleep irregularities including “insomnia” defined as total loss of sleep, “late onset of sleep” if patients fell asleep later than 24:00, “early awakening” if patients awoke before 5:00, “sleep discontinuity,” and “no irregularities” (cf. Table 6). In the days before seclusion, patients in the case group showed significantly more sleep irregularities, M (case group) = 0.8, SD = 1.0; M (control group) = 0.3, SD = 0.5; U₍₅₂₎ = 240.5, p = 0.036. Moreover, insomnia increased in the case group toward the day of seclusion and was the type of sleep irregularity with the highest frequency in the case group, M (case group) = 0.9, SD = 1.0; M (control group) = 0.2, SD = 0.5; U₍₅₂₎ = 194.0, p = 0.004. In addition, greater latency to sleep onset, M (case group) = 0.7, SD = 0.7; M (control group) = 0.1, SD = 0.3; U₍₅₂₎ = 164.5, p < 0.001, and early awakening were observed and documented more often in the case group, M (case group) = 0.5, SD = 0.7; M (control group) = 0.2, SD = 0.4; U₍₅₂₎ = 254.0, p = 0.047.

TABLE 6

Table 6. Sleep behavior, high contact frequency, demanding behavior, requests, and non-compliance.

Sleep discontinuity showed the highest frequency within the control group, but no significant differences in frequency could be observed between the case and the control groups, M (case group) = 0.8, SD = 0.8; M (control group) = 0.9, SD = 1.2; U₍₅₂₎ = 319.0, p = 0.707.

Other Behavioral Precursors

High contact frequency of patients with staff was observed and documented more often in the case group, M (case group) = 2.0, SD = 1.7; M (control group) = 0.3, SD = 0.7; U₍₅₂₎ = 152.0, p < 0.001. Furthermore, patients were described as more demanding M (case group) = 1.5, SD = 1.1; M (control group) = 0.6, SD = 0.9; U₍₅₂₎ = 174.5, p = 0.002, e.g., by asking repeatedly for something like being allowed to temporarily leave the ward, receiving cigarettes or food at inappropriate times, inappropriate request like being supplied with alcohol or cannabis, or by ringing the bell repeatedly) and refusing medication more often, M (case group) = 1.7, SD = 1.3; M (control group) = 0.1, SD = 0.4; U₍₅₂₎ = 96.0, p < 0.001. Requests were denied more often in the case group, M (case group) = 1.1, SD = 1.1; M (control group) = 0.3, SD = 0.6; U₍₅₂₎ = 178.5, p = 0.001, but there were no significant differences regarding the frequency of requests that were fulfilled between both groups, M (case group) = 0.2, SD = 0.4; M (control group) = 0.3, SD = 0.7; U₍₅₂₎ = 333.0, p = 0.894.

Text Characteristics

The extracted narrative notes for the total sample of 52 patients had a mean total word count of 529.7 (SD = 296.6) words. Furthermore, narrative notes of the case group contained more words per treatment day than the control group, M (case group) = 639.8; SD = 322.5; M (control group) = 419.6; SD = 224.1; U₍₅₂₎ = 193.5, p = 0.008.

The mean number of individual entries of notes per day over all 52 patients was 11.0 (SD = 2.4) with no significant differences between the groups, M (case group) = 10.4, SD = 2.5; M (control group) = 11.6, SD = 2.2; U₍₅₂₎ = 241.5, p = 0.074.

Moreover, patients in the case group were cited more often via direct speech, M (case group) = 1.0, SD = 1.0; M (control group) = 0.3, SD = 0.7; U₍₅₂₎ = 198.5, p = 0.003, most commonly to document patient quotes deemed important as exactly as possible (for examples cf. Table 3).

Prediction of Seclusion

Logistic regression was performed with word count, positive subjectivity, “unpredictable,” sleep irregularities, and “manageable” as potential predictors of seclusion, yielding a highly significant model, χ²(5) = 18.340, p = 0.003, n = 52. Patients with a higher word count were at a 43.2% higher risk to get secluded, OR 1.432 per 100 words, p = 0.018, Nagelkerke R² = 0.396 (corresponding to a medium to high effect size). None of the other included variables emerged as significant predictors of seclusion in this multivariate analysis.

Discussion

The current case-control study used a mixed methodological approach in order to examine whether precursors of seclusion can be identified in the narrative notes about acutely ill psychiatric inpatients as written by HCP, particularly with regard to subjectivity and emotional involvement. Various significant differences in staff documentation were identified between the groups. Thus, the current study found evidence supporting the hypothesis that an upcoming escalation is preceded by specific characteristics of staff narrative notes in the days before seclusion.

First, seclusion was significantly associated with higher word count in the current study. Notes in the case group were longer and more substantial, while the frequency of staff documentation entries per day showed no significant difference. In particular, a greater extent of behavioral terms and the use of direct speech were prominent in the staff notes describing patients before seclusion. It seems that staff describes problematic behavior more extensively and with different terms, potentially to improve information transfer between staff members working different shifts, to improve justification for upcoming coercive measures, and to ensure legal protection.

Secondly, seclusion was observed to be associated strongly with staff emotional involvement as measured by use of subjective expressions. Results also indicate that emotional involvement in the shape of “provocative/arduous” and “anxious” terms precede impending seclusion. This is remarkable, as staff documentation tends to minimize emotional content to enhance standardization, objectivity, professionalism, and appropriateness for use in potential lawsuits (50). Thus, it can be assumed that only a fraction of what an individual staff member was actually feeling or intending is documented in the their notes. Nevertheless, relevant differences concerning emotional involvement and subjective expressions remain in the documentation on psychiatric intensive care units. It remains unclear if emotional involvement is only associated with problematic behavior or if it also contributes an escalation process leading to seclusion. Although there is a variety of possible underlying causes and triggers, there is evidence that staff emotions may contribute to seclusion. DeBenedictis et al. (51), e.g., identified a negative working climate, and especially anger and aggression among staff members, as resulting in a higher use of seclusion and restraint. In addition, the question rises if staff with a higher susceptibility for provocation or anxiety may be more likely to apply coercive measures. In-line with these considerations, paying greater attention to staff-patient interactions could help increasing awareness for unconscious and emotional processes (16) involved in escalation processes. Staff training in noticing their emotional state early and emphasizing patients' perception might constitute an important part of de-escalation trainings and help prevent coercive measures like seclusion (6, 14). In any case, emotional involvement in narrative case notes could serve as an early warning sign of an impending seclusion.

Altogether, subjective expressions were less prominent in narrative notes than expected. The style of describing patients was widely consistent and personal stylistic features in documentation were rarely notable. Based on our experience, attempts to be as objective as possible and to use a pragmatic language are widespread in hospital routine documentation in German speaking European countries. Sentiments and emotions are normally omitted when standardized documentation is disseminated due to legal and time management reasons. Hamilton et al. (50) described a patient's file as a “domain of management,” where HCP “experience some compelling pressure to adopt, audit and report on their risk management strategies” (p. 90). The presented data raise the question if—in contrary to the current standards—there should be a regular place for subjectivity in psychiatric routine documentation, as it may represent an essential contribution to clinical assessment and may be helpful for the early detection of seclusion.

Third, as assumed, patient behavior preceding seclusion is significantly more often associated with negatively assessed terms like “agitated,” “irritable,” and “loud/screaming.” Similar findings could be shown in a large study by Cullen et al. (36) were “restraint” and “shouting” showed the strongest likelihood of seclusion among all behavioral keywords. Although patients of the present study were equally perceived as “friendly” in both groups, there were—overall—more positively assessed terms in the control group. Patients in the control group were more often described as being “adequate” and “organized,” and their behavioral amplitude was perceived to be less dynamic, described by the term “unchanged.”

Fourth, patients before seclusion had a considerably higher frequency of insomnia. Sleep irregularities are a well-known comorbidity of patients with psychiatric diagnoses and amongst others associated with worsening of symptom severity in schizophrenia (52–54) and an increase in mania symptoms (55, 56). Consistent with previous studies, there is evidence for an association of poor sleep and aggressive behavior in psychiatric patients (57, 58). Despite this evidence for an association between sleep irregularities and coercive measures, which is corroborated by the current study's results, sleep disturbance has received little attention in previous studies analyzing antecedents of seclusion. In addition, patients in the case group were observed as more demanding and refused medication more often compared to patients in the control group. Staff denying patients' requests was present more often before seclusion, which is consistent with previous studies describing restrictions as a frequent antecedent of aggressive incidents (10, 20).

Fifth, specific terms appeared to serve as a personal evaluation of critical situations and personal risk assessment. “Manageable” appeared significantly more often in the case group, which may seem counterintuitive at first. However, “manageable” was applied in the context of problematic behavior that could still be handled, denoting that despite problematic behavior, staff did not deem coercive measures necessary at this time. Terms like “threatening” were exclusively used in the case group and in the context of seclusion. This is in line with the literature. In a study by Foster et al. (8), the main consequence of aggressive incidents was that staff members were feeling threatened. While not necessarily being subjected to actual threatening behavior, they seemed to feel a “threat of what might happen” (8).

The current study stands out due to the combination of quantitative and qualitative methods (59, 60). One of the strengths of a qualitative approach was the opportunity to determine unknown factors preceding seclusion. By including narrative notes not only of nursing staff but of all HCP, generalizability could be enhanced. Validity could be enhanced by independently performing assessments through two researchers with high interrater reliability, comparison with a control group, and inclusion of quantitative assessments. Matching the patients in a case-control study resulted in more homogeneous groups and improved comparability.

However, there are several notable limitations. Although interrater reliability has been assessed, the influence of subjectivity cannot be eliminated in qualitative analysis (61). Since the study evaluated notes written in German, it is unclear if the results can be generalized to settings with other languages or different cultural backgrounds. Furthermore, only patients with a minimum inpatient treatment duration of 3 days prior to seclusion could be considered, although seclusion already can occur in the first days of treatment with one third of aggressive incidents happening in the first seven days of admission (35). In addition, absconded patients could not be included in the analyses, as the study was focused on evaluating narrative notes in the days before seclusion, and assessments could not be performed when text entries were missing. Although the examined patient groups differ with regard to seclusion, they–indeed–also showed a different level of aggression. Therefore, the examined case note characteristics are not attributable to a development leading to seclusion independently of aggression, and the influence of seclusion and aggression cannot be disentangled with the current approach. In- and exclusion criteria therefore introduced a potential selection bias. Furthermore, the study only used retrospectively available data, leading to several limitations, e.g., the underreporting of aggressive events (35) and of subsequent interventions. In addition, although the current study's sample size is adequate for qualitative assessment (62), further replication studies with larger samples are needed.

Conclusions

Despite professionalism and self-imposed reduction of emotional and subjective content, narrative notes of healthcare professionals still contain normally unused information that is associated with coercive measures like seclusion as early as 3 days before these events. This information might improve risk assessment as well as the early prediction and intervention of aggressive incidents.

Furthermore, the present study indicates that narrative notes should not be completely superseded by fully standardized documentation. Moreover, integrating subjectivity and emotional content in a way compatible with current standards could improve the clinical usability of routine documentation. Integration of a subjective perspective could be technically implemented by introducing an extra text-field dedicated to subjective perception, countertransference, or more subjective personal remarks (e.g., “I was very scared, my colleague hid in the corner of the room.”) into routine electronic documentation. This would probably not lead to a relevant increase in documentation efforts, since the writing process is performed intuitively and without being censored. Text-analysis and evaluation of the risk level would be performed automatically after documentation is finalized. This is, e.g., achievable by generating computerized algorithms for subjectivity and using machine learning on the basis of natural language processing. This approach also allows to include data from structured risk-assessment instruments into the analysis. Apart from the acquisition costs of an appropriate software solution, no additional financial burden is to be expected, as narrative notes are immediately available for the software in most current documentation systems.

In the present study, factors not routinely used for risk assessment like subjective statements, sleep disturbance, and narrative note word count were associated with seclusion. If these results can be replicated and remain valid especially in a prospective setting, e.g., word count could become an economical and simple predictor for escalation. The implementation of automated text analysis would enable routine use of narrative notes in the early detection and prevention of violence and coercion, and this has to be examined in future studies.

Data Availability

The raw data supporting the conclusions of this manuscript will be made available by the authors, without undue reservation, to any qualified researcher.

Author Contributions

CS and CH designed the study, analyzed, and interpreted the data. CS and JS collected the data. CS wrote the initial draft of the manuscript. CS had full access to all the data in the study and takes responsibility for the integrity of the data, and the accuracy of data analysis. CS, JS, MW, SB, UL, and CH revised the article critically for important intellectual content and approved the final version of the manuscript.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We want to thank Lukas Imfeld, Julian Möller, Ph. D., Franziska Rabenschlag, Ph. D., Linda Weick, and Tatjana Zimmermann, MD, for their valuable advice during study design and data collection. Furthermore, we want to express our gratitude to the Margot und Erich Goldschmidt & Peter René Jacobson-Stiftung, Basel, Switzerland, for providing funding for this study. In addition, we would like to thank the get on track program of the University of Basel, which supported author CS with funding granting a temporary supplemental assistance for childcare. Parts of the current publication were presented at the 2016 research retreat of the Department of Adult Psychiatry, Psychiatric University Hospital Basel, and received the research prize, for which we would like to particularly thank the Head of the Department of Adult Psychiatry.

References

1. Iozzino L, Ferrari C, Large M, Nielssen O, de Girolamo G. Prevalence and risk factors of violence by psychiatric acute inpatients: a systematic review and meta-analysis. PLoS ONE. (2015) 10:e0128536. doi: 10.1371/journal.pone.0128536

PubMed Abstract | CrossRef Full Text | Google Scholar

2. O'Connell B, Young J, Brooks J, Hutchings J, Lofthouse J. Nurses' perceptions of the nature and frequency of aggression in general ward settings and high dependency areas. J Clin Nurs. (2000) 9:602–10. doi: 10.1046/j.1365-2702.2000.00401.x

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Steinert T, Birk M, Flammer E, Bergk J. Subjective distress after seclusion or mechanical restraint: one-year follow-up of a randomized controlled study. Psychiatr Serv. (2013) 64:1012–7. doi: 10.1176/appi.ps.201200315

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Olofsson B, Jacobsson L. A plea for respect: involuntarily hospitalized psychiatric patients' narratives about being subjected to coercion. J Psychiatr Ment Health Nurs. (2001) 8:357–66. doi: 10.1046/j.1365-2850.2001.00404.x

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Fugger G, Gleiss A, Baldinger P, Strnad A, Kasper S, Frey R. Psychiatric patients' perception of physical restraint. Acta Psychiatr Scand. (2016) 133:221–31. doi: 10.1111/acps.12501

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Najafi F, Fallahi-Khoshknab M, Ahmadi F, Dalvandi A, Rahgozar M. Antecedents and consequences of workplace violence against nurses: a qualitative study. J Clin Nurs. (2017) 27:e116–28.doi: 10.1111/jocn.13884

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Needham I, Abderhalden C, Halfens RJ, Fischer JE, Dassen T. Non-somatic effects of patient aggression on nurses: a systematic review. J Adv Nurs. (2005) 49:283–96. doi: 10.1111/j.1365-2648.2004.03286.x

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Foster C, Bowers L, Nijman H. Aggressive behaviour on acute psychiatric wards: prevalence, severity and management. J Adv Nurs. (2007) 58:140–9. doi: 10.1111/j.1365-2648.2007.04169.x

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Yang M, Wong SC, Coid J. The efficacy of violence prediction: a meta-analytic comparison of nine risk assessment tools. Psychol Bull. (2010) 136:740–67. doi: 10.1037/a0020473

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Papadopoulos C, Ross J, Stewart D, Dack C, James K, Bowers L. The antecedents of violence and aggression within psychiatric in-patient settings. Acta Psychiatr Scand. (2012) 125:425–39. doi: 10.1111/j.1600-0447.2012.01827.x

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Dack C, Ross J, Papadopoulos C, Stewart D, Bowers L. A review and meta-analysis of the patient factors associated with psychiatric in-patient aggression. Acta Psychiatr Scand. (2013) 127:255–68. doi: 10.1111/acps.12053

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Renwick L, Stewart D, Richardson M, Lavelle M, James K, Hardy C, et al. Aggression on inpatient units: clinical characteristics and consequences. Int J Ment Health Nurs. (2016) 25:308–18. doi: 10.1111/inm.12191

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Whittington R. Attitudes toward patient aggression amongst mental health nurses in the 'zero tolerance' era: associations with burnout and length of experience. J Clin Nurs. (2002) 11:819–25. doi: 10.1046/j.1365-2702.2002.00659.x

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Fagan-Pryor EC, Haber LC, Dunlap D, Nall JL, Stanley G, Wolpert R. Patients' views of causes of aggression by patients and effective interventions. Psychiatr Serv. (2003) 54:549–53. doi: 10.1176/appi.ps.54.4.549

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Duxbury J. An evaluation of staff and patient views of and strategies employed to manage inpatient aggression and violence on one mental health unit: a pluralistic design. J Psychiatr Ment Health Nurs. (2002) 9:325–37. doi: 10.1046/j.1365-2850.2002.00497.x

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Haugvaldstad MJ, Husum TL. Influence of staff's emotional reactions on the escalation of patient aggression in mental health care. Int J Law Psychiatry. (2016) 49:130–7. doi: 10.1016/j.ijlp.2016.09.001

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Pulsford D, Crumpton A, Baker A, Wilkins T, Wright K, Duxbury J. Aggression in a high secure hospital: staff and patient attitudes. J Psychiatr Ment Health Nurs. (2013) 20:296–304. doi: 10.1111/j.1365-2850.2012.01908.x

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Duxbury J, Hahn S, Needham I, Pulsford D. The management of aggression and violence attitude scale (MAVAS): a cross-national comparative study. J Adv Nurs. (2008) 62:596–606. doi: 10.1111/j.1365-2648.2008.04629.x

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Ross J, Bowers L, Stewart D. Conflict and containment events in inpatient psychiatric units. J Clin Nurs. (2012) 21:2306–15. doi: 10.1111/j.1365-2702.2012.04073.x

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Bowers L, Wright S, Stewart D. Patients subject to high levels of coercion: staff's understanding. Issues Ment Health Nurs. (2014) 35:364–71. doi: 10.3109/01612840.2013.871088

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Finnema EJ, Dassen T, Halfens R. Aggression in psychiatry: a qualitative study focusing on the characterization and perception of patient aggression by nurses working on psychiatric wards. J Adv Nurs. (1994) 19:1088–95.

PubMed Abstract | Google Scholar

22. Whittington R, Wykes T. An observational study of associations between nurse behaviour and violence in psychiatric hospitals. J Psychiatr Ment Health Nurs. (1994) 1:85–92.

PubMed Abstract | Google Scholar

23. Lamanna D, Ninkovic D, Vijayaratnam V, Balderson K, Spivak H, Brook S, et al. Aggression in psychiatric hospitalizations: a qualitative study of patient and provider perspectives. J Ment Health. (2016) 25:536–42. doi: 10.1080/09638237.2016.1207222

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Lo SB, Gaupp R, Huber C, Schneeberger A, Garic G, Voulgaris A, et al. Influence of an open door policy on ward climate: impact on treatment quality. Psychiatr Prax. (2018) 45:133–9. doi: 10.1055/s-0042-121784

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Blaesi S, Gairing SK, Walter M, Lang UE, Huber CG. Safety, therapeutic hold, and patient's cohesion on closed, recently opened, and open psychiatric wards. Psychiatr Prax. (2015) 42:76–81. doi: 10.1055/s-0033-1359871

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Baby M, Swain N, Gale C. Healthcare managers' perceptions of patient perpetrated aggression and prevention strategies: a cross sectional survey. Issues Ment Health Nurs. (2016) 37:507–16. doi: 10.3109/01612840.2016.1166300

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Hochstrasser L, Voulgaris A, Moller J, Zimmermann T, Steinauer R, Borgwardt S, et al. Reduced frequency of cases with seclusion is associated with opening the doors of a psychiatric intensive care unit. Front Psychiatry. (2018) 9:57. doi: 10.3389/fpsyt.2018.00057

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Hochstrasser L, Frohlich D, Schneeberger AR, Borgwardt S, Lang UE, Stieglitz RD, et al. Long-term reduction of seclusion and forced medication on a hospital-wide level: Implementation of an open-door policy over 6 years. Eur Psychiatry. (2018) 48:51–7. doi: 10.1016/j.eurpsy.2017.09.008

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Jungfer HA, Schneeberger AR, Borgwardt S, Walter M, Vogel M, Gairing SK, et al. Reduction of seclusion on a hospital-wide level: successful implementation of a less restrictive policy. J Psychiatr Res. (2014) 54:94–9. doi: 10.1016/j.jpsychires.2014.03.020

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Huber CG, Schneeberger AR, Kowalinski E, Frohlich D, von Felten S, Walter M, et al. Suicide risk and absconding in psychiatric hospitals with and without open door policies: a 15 year, observational study. Lancet Psychiatry. (2016) 3:842–9. doi: 10.1016/S2215-0366(16)30168-7

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Schneeberger AR, Kowalinski E, Frohlich D, Schroder K, von Felten S, Zinkler M, et al. Aggression and violence in psychiatric hospitals with and without open door policies: a 15-year naturalistic observational study. J Psychiatr Res. (2017) 95:189–95. doi: 10.1016/j.jpsychires.2017.08.017

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Jalil R, Huber JW, Sixsmith J, Dickens GL. Mental health nurses' emotions, exposure to patient aggression, attitudes to and use of coercive measures: cross sectional questionnaire survey. Int J Nurs Stud. (2017) 75:130–8. doi: 10.1016/j.ijnurstu.2017.07.018

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Frohlich D, Rabenschlag F, Schoppmann S, Borgwardt S, Lang UE, Huber CG. Positive effects of an anti-aggression and de-escalation training on ward atmosphere and subjective safety may depend on previous training experience. Front Psychiatry. (2018) 9:134. doi: 10.3389/fpsyt.2018.00134

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Katsuki F, Goto M, Someya T. A study of emotional attitude of psychiatric nurses: reliability and validity of the nurse attitude scale. Int J Ment Health Nurs. (2005) 14:265–70. doi: 10.1111/j.1440-0979.2005.00391.x

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Abderhalden C, Needham I, Dassen T, Halfens R, Fischer JE, Haug HJ. Frequency and severity of aggressive incidents in acute psychiatric wards in Switzerland. Clin Pract Epidemiol Ment Health. (2007) 3:30. doi: 10.1186/1745-0179-3-30

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Cullen AE, Bowers L, Khondoker M, Pettit S, Achilla E, Koeser L, et al. Factors associated with use of psychiatric intensive care and seclusion in adult inpatient mental health services. Epidemiol Psychiatr Sci. (2018) 27:51–61. doi: 10.1017/S2045796016000731

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Rumshisky A, Ghassemi M, Naumann T, Szolovits P, Castro VM, McCoy TH, et al. Predicting early psychiatric readmission with natural language processing of narrative discharge summaries. Transl Psychiatry. (2016) 6:e921. doi: 10.1038/tp.2015.182

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Stewart R, Davis K. 'Big data' in mental health research: current status and emerging possibilities. Soc Psychiatry Psychiatr Epidemiol. (2016) 51:1055–72. doi: 10.1007/s00127-016-1266-8

PubMed Abstract | CrossRef Full Text | Google Scholar

39. McCoy THJr, Castro VM, Roberson AM, Snapper LA, Perlis RH. Improving prediction of suicide and accidental death after discharge from general hospitals with natural language processing. JAMA Psychiatry. (2016) 73:1064–71. doi: 10.1001/jamapsychiatry.2016.2172

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Mayring P. Qualitative Inhaltsanalyse: Grundlagen und Techniken. Weilheim; Basel: Beltz (2010).

Google Scholar

41. Saldaña J. The Coding Manual for Qualitative Researchers. Vol. 3. Los Angeles, CA: SAGE Publications (2015).

Google Scholar

42. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med. (2012) 22:276–82. doi: 10.11613/BM.2012.031

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Knoedler DW. The modified overt aggression scale. Am J Psychiatry. (1989) 146:1081–2.

PubMed Abstract | Google Scholar

44. Montoya A, Valladares A, Lizan L, San L, Escobar R, Paz S. Validation of the excited component of the positive and negative syndrome scale (PANSS-EC) in a naturalistic sample of 278 patients with acute psychosis and agitation in a psychiatric emergency room. Health Qual Life Outcomes. (2011) 9:18. doi: 10.1186/1477-7525-9-18

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Huber CG, Lambert M, Naber D, Schacht A, Hundemer HP, Wagner TT, et al. Validation of a clinical global impression scale for aggression (CGI-A) in a sample of 558 psychiatric patients. Schizophr Res. (2008) 100:42–348. doi: 10.1016/j.schres.2007.12.480

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Huber CG, Schottle D, Lambert M, Hottenrott B, Agorastos A, Naber D, et al. Brief psychiatric rating scale - excited component (BPRS-EC) and neuropsychological dysfunction predict aggression, suicidality, and involuntary treatment in first-episode psychosis. Schizophr Res. (2012) 134:273–8. doi: 10.1016/j.schres.2011.12.002

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Huber CG, Schottle D, Lambert M, Moritz S, Naber D, Schroeder K. Applicability of the clinical global impressions-aggression (CGI-A) scale for use in file audit trials. Pharmacopsychiatry. (2011) 44:189–92. doi: 10.1055/s-0031-1280816

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Sorgi P, Ratey J, Knoedler DW, Markert RJ, Reichman M. Rating aggression in the clinical setting. A retrospective adaptation of the Overt Aggression Scale: preliminary results. J Neuropsychiatry Clin Neurosci. (1991) 3:S52–6.

PubMed Abstract | Google Scholar

49. Cohen J. A power primer. Psychol Bull. (1992) 112:155–9.

PubMed Abstract | Google Scholar

50. Hamilton B, Manias E. ‘She's manipulative and he's right off': a critical analysis of psychiatric nurses' oral and written language in the acute inpatient setting. Int J Ment Health Nurs. (2006) 15:84–92. doi: 10.1111/j.1447-0349.2006.00407.x

PubMed Abstract | CrossRef Full Text | Google Scholar

51. De Benedictis L, Dumais A, Sieu N, Mailhot MP, Letourneau G, Tran MA, et al. Staff perceptions and organizational factors as predictors of seclusion and restraint on psychiatric wards. Psychiatr Serv. (2011) 62:484–91. doi: 10.1176/appi.ps.62.5.484

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Afonso P, Brissos S, Figueira ML, Paiva T. Schizophrenia patients with predominantly positive symptoms have more disturbed sleep-wake cycles measured by actigraphy. Psychiatry Res. (2011) 189:62–6. doi: 10.1016/j.psychres.2010.12.031

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Mulligan LD, Haddock G, Emsley R, Neil ST, Kyle SD. High resolution examination of the role of sleep disturbance in predicting functioning and psychotic symptoms in schizophrenia: a novel experience sampling study. J Abnorm Psychol. (2016) 125:788–97. doi: 10.1037/abn0000180

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Cohrs S. Sleep disturbances in patients with schizophrenia: impact and effect of antipsychotics. CNS Drugs. (2008) 22:939–62. doi: 10.2165/00023210-200822110-00004

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Barbini B, Bertelli S, Colombo C, Smeraldi E. Sleep loss, a possible factor in augmenting manic episode. Psychiatry Res. (1996) 65:121–5.

PubMed Abstract | Google Scholar

56. Gruber J, Miklowitz DJ, Harvey AG, Frank E, Kupfer D, Thase ME, et al. Sleep matters: sleep functioning and course of illness in bipolar disorder. J Affect Disord. (2011) 134:416–20. doi: 10.1016/j.jad.2011.05.016

PubMed Abstract | CrossRef Full Text | Google Scholar

57. Kamphuis J, Meerlo P, Koolhaas JM, Lancel M. Poor sleep as a potential causal factor in aggression and violence. Sleep Med. (2012) 13:327–34. doi: 10.1016/j.sleep.2011.12.006

PubMed Abstract | CrossRef Full Text | Google Scholar

58. Langsrud K, Kallestad H, Vaaler A, Almvik R, Palmstierna T, Morken G. Sleep at night and association to aggressive behaviour; patients in a psychiatric intensive care unit. Psychiatry Res. (2018) 263:275–9. doi: 10.1016/j.psychres.2018.03.012

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Leech NL, Onwuegbuzie AJ. On becoming a pragmatic researcher: the importance of combining quantitative and qualitative research methodologies. Int J Soc Res Methodol. (2005) 8:375–87. doi: 10.1080/13645570500402447

CrossRef Full Text | Google Scholar

60. Ostlund U, Kidd L, Wengstrom Y, Rowa-Dewar N. Combining qualitative and quantitative research within mixed method research designs: a methodological review. Int J Nurs Stud. (2011) 48:369–83. doi: 10.1016/j.ijnurstu.2010.10.005

PubMed Abstract | CrossRef Full Text | Google Scholar

61. Qu S, Dumay J. The qualitative research interview. Qual Res Account Manag. (2011) 8:238–64. doi: 10.1108/11766091111162070

CrossRef Full Text | Google Scholar

62. Marshall B, Cardon P, Poddar A, Fontenot R. Does sample size matter in qualitative research?: a review of qualitative interviews in is research. J Comput Inform Syst. (2013) 54:11–22. doi: 10.1080/08874417.2013.11645667

CrossRef Full Text | Google Scholar