Addressing victimization to enable societal participation in flexible assertive community treatment: A process evaluation of the implementation of a new intervention

Background: Individuals with severe mental illness experience more victimization and discrimination than other persons in the community. Effective rehabilitation and recovery-oriented care interventions aimed at addressing this issue are lacking. We therefore developed a victimization-informed intervention (accompanied by a training module for professionals) called the Victoria intervention. The purpose of the present study was to understand the trial effects by examining the implementation process and the factors that influenced it.

Materials and methods: A process evaluation was conducted using a mixed-methods design. During the professionals’ intervision sessions, we used observations to understand the learning processes (n = 25). Subsequently, we studied the use of the intervention in practice through structured questionnaires (n = 215) and semi-structured interviews (n = 34) with clients and professionals. We used descriptive and inferential statistics for the quantitative data and the framework method for the analyses of the qualitative data.

Results: The observations showed that the trainings were well received. The professionals shared the urgency of paying attention to victimization and discrimination and its harmful effects on participation. They also found the intervention steps to be logical and the intervention protocol easy to use. Nevertheless, they mentioned in the interviews that they had experienced difficulties initiating a conversation about victimization, and if they started one, they did not always follow the steps of the intervention as intended. Few clients said that victimization was placed on the agenda, though those who had discussed victimization with their caregivers expressed their appreciation in the interviews; they felt acknowledged and supported.

Discussion: The findings indicate that the intervention was considered helpful in raising awareness and the acknowledgment of victimization. However, professionals remain reluctant to talk about the subject, and the results show they need more practical training in this regard. This process evaluation has an important added value in that it helps us to understand the results of the effect evaluation of the intervention. The findings will facilitate the development and implementation of interventions that address clients’ victimization experiences in community mental healthcare settings and subsequently enable their participation in society.

Introduction

Individuals with severe mental illness (SMI) are victims of crime and discrimination more often than other citizens (1–5). The consequences of victimization, discrimination, and stigmatization can worsen symptoms and additional mental health issues, re-victimization, perpetration, and social isolation (1, 6, 7), which are considered important barriers to societal participation and recovery (6, 8, 9). While evidence exists that understanding and addressing why a client is demoralized by previous victimization experiences are beneficial for rehabilitation and recovery (10–14), effective tools to recognize and address victimization experiences, including (anticipated) stigmatization and discrimination, are lacking. We developed and tested the Victoria intervention, a four-step intervention that aims to increase safe societal participation by increasing awareness and the acknowledgment of victimization (15). The effect study was published in a previous article (16).

Several characteristics of the Victoria intervention, the trial, and the results made it important to carry out a process evaluation. First, we wanted to understand the trial effects as we performed a cluster randomized controlled trial (RCT) with the intervention in flexible assertive community treatment (FACT) teams (17, 18). Although we found that the intervention successfully moderated experienced discrimination, it had no effects on participation and victimization or on other primary outcome measures, contrary to our expectations (16). Nevertheless, clients felt that their victimization experiences were acknowledged and that they were supported in the recovery process. In both the intervention and control groups, anticipated discrimination and self-efficacy increased slightly over time. However, even though previous studies have reported that participants feared relapse after discussing their victimization experiences (19, 20), psychosocial functioning, victimization, and other outcomes neither worsened nor increased in the intervention group. Conducting a process evaluation was warranted to help understand these mixed findings and move forward with the intervention in practice (21, 22).

Second, the design of the Victoria intervention makes it difficult to measure its use in daily practice. It trains professionals to discuss victimization experiences with their clients, and we expect clients to feel acknowledged and supported in their recovery process (i.e., for the intervention to have an indirect effect). What happens in the process is not easy to quantify in an effect evaluation. The degree of flexibility that is built into the intervention adds to this complexity. By executing a process evaluation, we can obtain knowledge about the extent to which professionals recognize and acknowledge victimization in their clients (22, 23).

Third, practical trials, such as the one in which the Victoria intervention was tested, are characterized by multiple interacting components, and the context, which may partially determine the outcome is difficult to control for (24). For example, we trained the professionals in the FACT teams and we measured the effects for 20 months. In this timeframe, we could not rule out any change of personnel or conversations between teams. Furthermore, the trial was performed in multiple sites, each of which had specific contexts that may have influenced outcomes (25). In these complex settings, whether the intervention works is naturally important, but contextual influences on implementation or outcomes also have to be examined to gain deeper insights (21, 25). Gathering the perspectives of all stakeholders can provide knowledge of these contextual effects. Moreover, process evaluations can help us understand why certain effects are obtained by measuring the response of the intervention among clients and professionals (23, 25).

Hence, by examining the implementation process (including training and use), the factors that influenced it, and the impact the intervention had on the clients, we aimed to understand the trial effects.

Materials and methods

Design

For this process evaluation, which was embedded in a previously published cluster RCT (26), we used a mixed-method design using both qualitative and quantitative data (see Table 1 for the research questions and accompanying data sources). First, as the intervention was interwoven in the rehabilitation work of the FACT team, treatment plans for clients from both the intervention and control groups were used to measure adherence to rehabilitation principles in general. Second, the intervision sessions (i.e., meetings coached by a trainer to discuss the conversations they have had in their daily work) following the training were audio-recorded so we could assess the extent to which the professionals used the intervention in daily practice, how and when they used it, and the factors that influenced its use. Third, for the quantitative part of this study, data from structured questionnaires among clients and professionals were analyzed to examine perceptions of clients’ conversations with their case managers and whether the professionals concerned were aware of their clients’ victimization experiences. Fourth, the professionals were asked to fill out a checklist during the intervision sessions to see how faithful they had been to the intervention itself. The checklist was designed in the development phase of the intervention. Finally, semi-structured interviews were conducted with professionals of the intervention teams, management, and trainers to gather information about their experiences of the training sessions, influences on the use of the intervention, how and when they used the intervention, and the impact of the intervention on the client from the perspective of the professional. Semi-structured interviews were held with clients to cross-check the use and impact of the intervention. The study protocol of the RCT was approved by the Medical Ethical Committee of the Elisabeth Hospital in Tilburg (NL53845.028.15) on November 18, 2015 for all participating sites. The study was registered with the Dutch Trial Register (NTR 5585).

TABLE 1

Table 1. Research questions and relevant data sources.

The intervention

The Victoria intervention was designed to be victimization informed and to serve as an add-on module so that professionals can be made aware of and sensitive to the topic while employing rehabilitation approaches. It supports professionals in starting and structuring conversations with their clients regarding their experience of victimization and how it impacts their societal participation. The intervention comprises four steps: exploration, analysis, clarifying the context, and future strategies (15). The first step involves exploring the client’s societal participation and their satisfaction with it. When a client is avoiding societal participation or if their desired progress is stagnating, continuing to the second step is indicated. During the second step, relevant negative experiences related to societal participation are discussed. A narrative approach is adopted in which the professional acknowledges the pain of the client’s victimization experiences (27) and tries to understand the avoidance or stagnation. The third step is to clarify the context of the experience, that is, why the client initially engaged with the situation in which they were victimized. The objective of this step is to switch from the negative experience to a more positive perspective as taking reasonable risks is necessary to progress in life. The fourth step is to conclude the conversation by determining future steps. This can vary from planning another conversation, reformulating a rehabilitation action plan, starting other interventions such as supported employment (e.g., IPS) or treatment for self-stigma (e.g., NECT), or, when there is underlying trauma, trauma-focused therapy.

Training consisted of a refresher on the rehabilitation method that had been used previously. This was followed by three sessions on the Victoria intervention for the entire team. The first session comprised securing the intervention on an individual and team level. The second and third sessions were provided as per the respective organizations. In the first session, ways were discussed to implement and secure the intervention as part of the participants’ daily routine. The second and third sessions comprised an explanation of the steps and role-playing practice. Finally, intervision meetings were held every 6–8 weeks. During these, the conversations the professionals had been having with their clients were discussed (with the assistance of a trainer). A short handout was designed to be used during conversations with clients.

Participants

The participants comprised mental health nurses, experts-by-experience, managers, and clients of eight FACT teams. The trainers were also invited to take part in qualitative interviews. For an overview of all participants and other data sources, see Table 2.

TABLE 2

Table 2. Data sources included in the analyses.

Table 3 lists the characteristics of the study participants in both the subsample for the qualitative interviews, as well as the full sample from the structured interviews from the RCT. In the subsample, half of the client participants were males, and the average age of the clients was 45. Most of the clients had a psychotic or mood disorder as a primary diagnosis. In the full sample, most had schizophrenia; 56.2% of the clients had been the victims of one or more incidents of crime in the previous year, and 46.1% of the clients in the RCT sample. Ten of the professionals were male and eight, female.

TABLE 3

Table 3. Characteristics of the interviewees at T2.

Procedure

All qualitative interviews were conducted between April and July 2018, during the 20-month follow-up period of the RCT. For the professionals, a purposive sample per team of three mental health professionals, including one expert-by-experience, was recruited to reflect a diverse range of opinions on the intervention. As the number of trainers and managers was limited, all were asked to participate, and all agreed.

From the clients who participated in the RCT and were exposed to the intervention at the request of their case manager, a random sample of 24 was contacted via telephone for a qualitative interview, and 16 agreed to participate. We based the number of clients needed to contact on several factors, such as the research questions, the nature of the interview topics (questions were rather specific), the use of other data (i.e., triangulation), and practicality (28, 29). Additionally, by taking into account a 15% attrition rate, we invited one extra client per intervention team. Non-participation had no consequences in terms of the care they received. The interviews were held at a location of the participant’s choosing (i.e., either home or FACT team office). Before the interviews took place, written consent was requested. The interviews with both the professionals and clients were led by an interview guide. Topics were based on the intervention components (15) and a framework for process evaluations for cluster RCTs (25). Since we wanted to elicit specific information from each of the groups, different interview guides were developed for each group. The guide for the professionals included their perspectives on the intervention, their ideas regarding its usability and degree of usage, and their ideas on clients’ experience of the intervention. For the clients, the guide included their experience of conversations (with their case manager) and their difficulties with societal participation and victimization.

To ensure their objectivity, the interviews were carried out by two research assistants who were not involved in the development of or training in the intervention. They had training in interviewing and discussed issues that emerged after the first interviews. The interviews were audio-recorded unless otherwise requested by the participant, and were pseudonymized. On average, the interviews with the professionals lasted 40 min, and with the clients, 28 min (see Table 1). The clients received financial remuneration of €10.

The structured questionnaires were collected during the effect evaluation within the RCT at T1 (10-month follow-up) and T2 (20-month follow-up). The professionals’ questions addressed the extent to which they had used the intervention in daily practice with each of the clients in the study and the insight gained into how victimization acted as a barrier to clients’ societal participation. The clients’ questions addressed the extent to which they felt they were being listened to and supported in their recovery process.

The intervention teams had on average 7–9 intervision meetings. Eighty percent (25 out of 31) of the intervision meetings were audio recorded (six were missing, for practical reasons unrelated to the study questions). A trainer, professionals from the intervention team, and a researcher participated in each meeting.

Pseudonymized treatment plans from a random sample of 15% of the clients participating in the RCT were collected at T0 (baseline) and T2, resulting in 63 treatment plans at T0 and 40 plans at T2. Adherence to rehabilitation principles was measured by the method developed by Wunderink et al. (30). This involved scoring the overall treatment plan (plan level) according to five criteria: whether goals were formulated; whether the treatment plan was written in the first person; whether there was space for clients’ consent; whether evaluation dates had been planned; and whether emergency agreements were available. Furthermore, the treatment goals (goal level) of the following nine life domains were scored: housing; work and occupational activities; education; recreation; social contacts; meaning in life; self-care; mental health; and physical health. Each goal area is scored according the choose-get-keep model of psychiatric rehabilitation (31): is the rehabilitation phase clear; is there a specified timeline; is there a clear task division; and is the client’s role clear. Finally, the overall quality score of the plan was calculated by adding up the items on plan level and the goal level.

Finally, the professionals in each team were asked to fill out a fidelity checklist including the objectives of each step of the intervention by using a four-point scale ranging from not at all to completely to measure the extent to which they followed each step and whether they met the accompanying objectives.

Analyses

All interviews and observations were audio-recorded and transcribed verbatim. Analyses were performed using the framework method (32), a form of thematic analysis consisting of five distinct phases: familiarization; identifying a thematic framework; coding and indexing; charting; mapping; and interpretation (33). The method is particularly suited for comparing and contrasting qualitative data, and due to the distinct steps, analyses are transparent and reproducible (33).

Atlas.ti was used for the initial open coding of a sample of the interviews, which were coded by the first author and discussed afterward with the second and third authors to ensure that all agreed with the codes. Categories and over-arching themes were identified; they formed the analytical framework on which the rest of the interviews were coded by the first author. Themes and subthemes were summarized in the form of charts and a framework matrix for interpretation and comparison. These were discussed with all authors. The results are reported according to the COREQ checklist (34). Data from the structured questionnaires for the clients and professionals in the intervention and control groups, the treatment plans, and the checklist were analyzed using descriptive statistics. All quantitative analyses were performed using SPSS Version 24.

Results

The adherence to rehabilitation principles is shown for the 125 clients in both the control and intervention condition at baseline and 20-month follow-up (Supplementary Table 1). In almost all treatment plans, one or more rehabilitation goals were formulated. The prevalent treatment goals were (from most to least mentioned) mental health; self-care; social contacts; work; meaning in life; physical health; housing; daytime activities; and learning. The overall quality of the treatment plans was 5.2 (on a scale from 1 to 10). Most of the plans revealed insufficient adherence to rehabilitation principles; in no case was there substantial or full adherence.

In the following sections, the major themes regarding the implementation process and its influencing factors identified in the interviews with professionals, managers, and trainers are discussed, together with additional information from the intervision meeting observations and the structured questionnaires. The themes were: (1) attitudes toward and reasons for discussing victimization; (2) the process of implementation; (3) factors affecting the use of the intervention; and (4) the perceived added value of the intervention. The results from the client interviews are also presented.

Attitudes toward and reasons for discussing victimization

Victimization was considered by all the participants to be an important issue for discussion. The professionals described three reasons for using the Victoria intervention. First, victimization is seen as something inherent in people with severe mental health problems, and therefore the topic should be embedded in one’s daily routine. Several of them argued that all of their clients experienced victimization on a regular basis, with some stating that victimization will become a greater problem in the future because of the changing care system and growing intolerance toward people with mental illness. One professional said that

Those people all have a history. A lot of have experiences and you’re not here for no reason. And um… so the chances are, if you’re at FACT and you have victimization experiences, those chances are pretty high (P15).

A second reason was that the intervention focuses on an undiscussed or oft-avoided topic. Victimization is not an easy subject to discuss and is sometimes avoided by both clients and professionals. One professional suggested that because victimization is inherent to the FACT population, many relate it to their psychopathology. Another professional confirmed that the topic often remains undiscussed: “In three-quarters of an hour you have to look at so many things. I notice a very conscious reflection on these things, that I think this happens far too little” (P12).

Yet another reason was that, according to the professionals, victimization has an enormous impact on participation. They recognized that clients sometimes refrained from undertaking activities because they want to avoid disappointment. Many clients “have become so familiar with disappointments that it is more or less an expectation that things would turn out that way” (P4). Several argued that supporting social recovery is what the job of a case manager is all about, so victimization should be given more priority.

However, a few professionals had some doubts about the intervention. One argued that due to high workloads and a lack of staff, the priority should be on treatment and not on barriers to rehabilitation in cases of addiction and personality disorders. Another expressed her concerns about focusing on victimization thus:

But with the emphasis you put on victimization um… you might also increase the likelihood that it’s precisely about that. You know what I mean, right? So people sometimes don’t move because they are victimized and so we have to pay attention to that. And then it’s a benefit. Um… but because you emphasize it so much, it can also be that… and that’s a bit… you know, social workers… social workers love trauma and victimization, huh? (P6).

Comparison with care as usual

All experts-by-experience indicated that the steps of the Victoria intervention were nothing new. They already focused on victimization in their conversations with clients, asking about previous (negative) experiences and what preceded them.

But I have a, an easier entrance, I think, because I get to talk about my own experiences as well… I’ve been through um a lot of traumas. And also psychoses from traumas. So yes, then…it’s already a bit natural that I’m really curious about how they…what they’ve been through, how their eh life situation is, eh…what they need, what they eh, eh…yes, I investigate that with the clients completely (P1).

Most of the case managers admitted they addressed victimization with their clients, but not on a regular basis or as extensively as the Victoria intervention. A commonly expressed view was “I did have these kinds of conversations, but never so precise.” One admitted the following:

I must say that I uhh, that I do find it a… in that sense a useful or a beautiful intervention, because it (victimization) uhh maybe by some people often underexposed, while I myself always uhh without knowing the intervention already had the idea that I focused on people’s recovery and why they do not come to recovery so to speak (P14).

Finally, the case managers all acknowledged that they often wanted change too fast too soon, or that they often started offering solutions.

Process of implementation

Extent of application

Results from the structured questionnaires (Supplementary Table 2) show that, according to the professionals, 55% and 60% of clients were exposed to the intervention at T1 and T2, respectively. The professionals stated that most of their clients had the occasional Victoria conversation (52% at T1 and 48% at T2). A small percentage had a Victoria conversation often (15% at T1 and 11% at T2). The extent to which professionals discussed Victoria conversations with colleagues increased over time, from 29% at T1 to 43% at T2.

The professionals reported in the fidelity checklist that they generally went through all the steps of the intervention, with a sum fidelity score of 3.06 (with scores ranging from 0 to 4; Table 4). The two lowest-scoring items were responses to “To what extent are you not going along with the avoidance?” and “To what extent can you say for yourself that you have not been too quick to think in terms of solutions?” This suggested that they had the most difficulty applying these components of the intervention.

TABLE 4

Table 4. Checklist of the steps of the intervention.

By contrast, in the qualitative interviews, the professionals indicated that they did not use the intervention extensively in their daily practice. While they valued it and thought that more attention should be paid to barriers to participation, they were unable to use the intervention as much as they would have liked to. The same can be stated for the teams. Most of the professionals explained that in their teams, neither the intervention nor victimization were discussed on a regular basis. Only in one team did clients who had meetings about their treatment plans the following week discuss the question of social functioning. The case manager of that team added that in other team meetings they also asked questions such as “Why is it that this client has issues with participating, why does he not go outside, or why did he quit his education” (P10) more often. According to the professionals, the main reason for not using the intervention in daily practice was because it was new and they were creatures of habit; they were only reminded of it in intervision meetings, after which it faded into the background again. Both trainers and case managers stated that intervision meetings consisted mainly of discussing potential clients rather than evaluating previous Victoria conversations.

Steps of the intervention in daily practice

The chief indicator of the Victoria intervention according to the protocol is when a client has issues with societal participation. According to the professionals, 34.4% of the clients were avoiding participation, and 33.6% stopped participating. The professionals confirmed that this was due to victimization in two-thirds of those cases (64% at T1 and 56% at T2; see Supplementary Table 2).

In the qualitative interviews, most of the professionals claimed that they applied the intervention flexibly, for example by starting to address victimization rather than the clients’ struggles with societal participation. Those who used the intervention more thoroughly noticed that it helped to address victimization in terms of participatory stagnation; it opened doors, and they also learned “new” things about their clients: “… while if you go more in the direction of, “Yes, but it’s terrible what happened to you,” that is easier for them to bear. Then it becomes an easier subject” (P16).

Most experts-by-experience indicated that they used the steps of the intervention as intended, and thought it was doable if you take your time. They also said they took a very flexible approach to the sequencing of the steps, as the outcomes of each were related to personal experiences and stories, so required a personalized approach.

They also pointed out that they often recognized certain feelings themselves (e.g., shame and discouragement), and this made it easier for them to relate to the feelings of the client; they therefore understood the importance of the second step in particular. In most case manager interviews, this step was not discussed, though two of them concluded that “What I’ve noticed in particular is that, when I’m working with Victoria … is that, when you ask about this (victimization), I’ve noticed that people are often open about it, if you really pay attention, if you really try to listen” (P10).

Most professionals argued that participation-related victimization was not a topic clients put on the agenda; indeed, they avoided it altogether. A potential issue was that the professionals might along with them. They recognized that they needed to be aware of this and take the initiative to make it a topic of conversation.

The professionals struggled with applying the intervention in daily practice partly because each step required a considerable amount of time. One of the trainers argued that most case managers had a tendency to focus on recognizing and acknowledging negative experiences and offering solutions, to the point where the conversation was no longer about social functioning.

The case managers argued that the tendency to offer solutions too soon was an issue. The necessary time had to be taken to complete the first two steps (i.e., exploration and analysis). One professional stated that “I had to sit on my hands quite often (P10).” This tendency was confirmed in the intervision meetings when cases were being discussed: “I wouldn’t come up with solutions myself, but I would ask “what have you tried?” … then you want to follow through on that as well” (W23).

Finally, beginning the intervention could also be a challenge. Many of the professionals started with a clear suspicion that the client had been victimized. Only a few case managers indicated that they introduced the intervention when a client was struggling with societal participation. They were also the ones who used the intervention quite often. Some experienced difficulties in choosing the right moment; a client had to be in the right recovery phase. This was confirmed in the intervision meetings.

Usability

In general, the intervention was perceived as clear, logical, and comprehensive. The professionals argued that the steps and the accompanying handouts were easy to use. However, the problem with using the intervention in daily practice lay in a certain reluctance stemming from a fear of inflicting more trauma. They felt that discussing victimization experiences could cause their clients distress, even though most of them agreed that in the long run, this would be beneficial. The case manager of a client who worked as a sex worker on the streets and was suffering from addiction said:

I assume that he has experienced a lot of traumas. But I… also one of those ladies once said to me: yes, I use because I experience so much on the streets. I use to cope. And so I can forget those bad experiences. When I ask someone in depth, Where was it? And how was it? I find that uhm, yes, this does not quite fit my role then. What if I’m rooting around in something that has bothered people for years? You have to know what you’re doing (P11).

Nevertheless, all the professionals said that they referred to a psychologist in their team if they suspected underlying trauma. They were also aware that this, in addition to using the trauma screener, is advised in the fourth step of the intervention. They argued that the intervention is not suitable for everyone. They agreed that for clients with psychosis or heavy addictions, the intervention should wait. Some were more hesitant than others: “If someone drinks 12–13 cans of beer a day, I don’t have to have this conversation. … Then I agree (with the client), if I come by uh, it is in the morning and (we agree) no alcohol” (P9).

Several preconditions for the application of the intervention emerged from the interviews. The most important one was that a narrative attitude requires time. Sitting down with the client and not being led by the delusion of the day creates a space for the client to be open. The downside of this is that professionals have to slow down and take more time with a client than is scheduled. Another precondition is having a connection or a good relationship with the client. This also creates a safe space in which they feel safe to open up about their victimization experiences, though having these kinds of conversations with new clients was not considered advisable.

Training

Attendance at the first training session was 97%, followed by 66% and 58% for the second and third sessions, respectively. In general, professionals perceived the training as “well put together.” On the role play and fictitious cases, they were divided. Some thought this was insightful and others found it difficult to practice the intervention in a created situation. Finally, they preferred to have the second and third part of the training per team, instead of per organization.

The intervision meetings were not a priority in most of the teams, which led to a lower attendance rate or even rescheduling meetings. Case managers were too busy and, for example, clients’ crises were of higher priority. This led to lesser continuity across meetings, because cases brought in at one meeting could not be followed up in the next. The input of cases was appreciated though and was perceived as practical and enlightening. However, it remained mostly talking about rather than evaluating actual conversations.

Factors affecting the use of the intervention

Three main factors were identified. Case managers have large caseloads that increasingly involve complex clients. This, coupled with the limited amount of time available, makes it difficult to sit down with a client and to take the time that is needed for the intervention. One of the professionals explained:

Often it’s really a matter of investing, of time… And that, from day-to-day you just lack time. That is what’s lacking. That you want to take the time, but that you still have to go like “oh, in half an hour I have to go to the next appointment.” So there is always pressure there, like, I have to… And my clients notice that too (P10).

The professionals argued that it was difficult to implement a new intervention and learn new skills because they had busy schedules, especially when other matters were more pressing (e.g., client crises). This issue was compounded by the high turnover in the participating teams during the 20-months of the study. This resulted in a greater focus on primary tasks and making sure clients were taken care of. The teams’ knowledge and understanding of the intervention were correspondingly impacted, and new case managers had to be trained up.

Second, Case managers and experts-by-experience felt that the importance of the intervention was not sufficiently recognized. Neither time nor space was available in their daily schedule, and there was a great deal of misalignment and poor communication. What would have helped, according to the professionals, was: (1) making the intervention part of formal processes, for example in electronic patient systems or in yearly team performance evaluations, and having it regulated and approved by management; and (2) creating the necessary room in people’s schedules where they could apply it:

It must also be supported by management and seen as important. So I think it’s from the bottom up, but also um… from the organization that’s behind it. You can’t expect people to have to do all sorts of things and then also have to implement… something themselves (P15).

The professionals indicated that, as case managers tend to work individually, it would have helped to integrate the intervention into the team setting, for example, in regular crisis meetings. Implementation would have also benefited from having a designated individual in each team who secured its place on the agenda. Additionally, some suggested that yearly booster training sessions would be helpful.

Perceived added value of the intervention

According to the professionals, the main added value of the intervention was that it forced them to sit back, listen, and take their time. This helped to create insight (for both themselves and their clients) and openings to discuss victimization experiences. As clients will usually only talk about negative experiences when the professional explicitly discusses them, the intervention made discussing what is a heavy and difficult topic more straightforward.

Because it’s always about vulnerable things. Things that people would rather not… That would rather not be there. And the more you have of those kinds of things the more you actually forget that they play a role. And sometimes it is also just eh for the client very enlightening to eh to get the insight. Why do I do what I do? What exactly is there then… Why do I always fall into the same trap or why do I never succeed or… What is it exactly? As far as I am concerned, this is a very nice instrument (P12).

What was also achieved was that “you are connecting (with the client), you are giving someone the idea of ‘hey, he/she is interested in what happened, why I am the way I am.”’ (P13).

The intervention provides a structured way of addressing the issue of caseloads and explicitly paying attention to clients who seem to be doing fine or are mentally stable but are not making much progress on societal participation. By using the intervention on a regular basis (and not only at intake), the professionals stated that they had gained insights into why their clients were struggling with societal participation or avoiding it altogether, what influenced certain choices regarding same, and learning about the clients’ wishes regarding rehabilitation.

The responses of the clients to the intervention, according to the professionals, were generally positive. However, at first, some clients resisted discussing victimization experiences because they were distressing. Returning to them in a subsequent appointment seemed to help resolve this sticking point. Professionals who initiated the Victoria conversation by starting with negative experiences rather than with participation struggles noticed that clients were reluctant to take part. On the other hand, the professionals who started by trying to connect with the client, took their time, acknowledged feelings associated with negative experiences, and related them to struggles with participation, received positive feedback.

The professionals said that the intervention influenced their daily practice. The most important effect was that their overall awareness of both victimization and participation increased. Several professionals acknowledged that beforehand, they were less conscious of the impact of victimization experiences, but subsequently, they noticed avoidance or stagnation with participation more. Additionally, they mentioned that they could no longer simply assume that their clients were satisfied with their social functioning, but checking-in frequently and “connecting with the client, and not so much having a biased, but an inquisitive attitude. Well, that works” (P4).

Clients’ conversations with professionals

Most of the clients discussed victimization or other setbacks related to societal participation felt that it was important to have such conversations (Table 5), though they did not think they had to discuss these topics more often.

TABLE 5

Table 5. Clients on social activities and victimization or setbacks at 10- and 20-month follow-up (N = 326 at T1; N = 315 at T2).

Many of the clients mentioned in their interviews that they had issues with participation (such as a lack of familial contact, a small social network, and wanting voluntary work); only a few mentioned discrimination or victimization (such as arguments with neighbors and traumatic encounters with the police). Even though victimization was rarely a topic of conversation with their case manager, many clients discussed it with their family, social contacts, and in relation to their needs regarding societal participation. Additionally, many of their conversations were about daily matters, such as how the week had been going, or medication and symptoms.

In general, the clients do not recognize that a new intervention was being used. They claimed that conversations on victimization were rarely followed up. They received different types of advice from their case manager, but few were rehabilitation-related and were often practical. However, most of the clients were satisfied with their case manager; they felt a connection with them and that they were interested in what they liked doing. As was the case with the structured interviews, what they gained most out of their conversations was that they could vent their feelings, and they felt listened to and supported.

It is a kind of handholding … if things were going a bit less or so, I can always go to her. And we can then, you know, she can always assess how things are going, and yes, that is also the reason that when I e-mailed so to speak. Sometimes I e-mailed because I eh… because I then, yes I did not really feel good, and worried much eh… or something. And often it’s better after I’ve written it off, so to speak, because then I’ve shared it and I know okay, you know, it’s known and that always gave me peace (C16).

Discussion

The present process evaluation aimed to understand the trial effects by examining the implementation process (including training and use of the intervention), the factors that influenced this process, and the impact the intervention had on the clients. The results show that the professionals shared the urgency of paying attention to victimization and discrimination and its harmful effects on societal participation. They also found the intervention steps to be logical and the intervention protocol easy to use. Even though they said they discussed victimization more often with their clients, they did not always follow the steps. Furthermore, they remained reluctant to initiate conversations about victimization, mainly due to a fear that their clients might relapse, become traumatized, or feel insecure or uncomfortable about bringing it up and talking about it. However, when the professionals began a conversation on victimization, their subsequent experiences were positive. The clients did not relapse, which confirmed the findings of previous studies (20). Additionally, they felt acknowledged. Using the intervention gave the professionals insights into their clients’ rehabilitation wishes and allowed them the opportunity to discuss victimization experiences.

The different means we used to assess the extent of use of the Victoria intervention showed some discrepancies. The professionals indicated in the checklist that they mostly followed the intervention steps, and similarly, in the structured questionnaires, they reported that they used the intervention on over half of the clients. However, in the interviews and intervision meetings, even though they indicated they were more aware of victimization and addressed the topic more often, they took a flexible approach to the application of the intervention. Furthermore, they stated in the interviews that they were reluctant to use it and did so only cursorily.

The findings indicate that the professionals need more training in how to address victimization and should sit on their hands more often when engaged in conversations on this difficult topic. Extending the current intervention with more in-depth and real-life role-plays using actors and peer workers is advised, as well as annual refreshers.

What was notable was that the RCT showed the positive effects of the intervention on the discrimination that the clients experienced and on the acknowledgment of this (16). It seemed that just addressing victimization helped societal participation. Higher fidelity to and the stricter use of the Victoria intervention might have enlarged the positive effects and improved other outcomes (including societal participation and victimization).

Several factors associated with victimization are often examined, including ethnicity. However, evidence is inconclusive; some studies found that ethnicity was a risk factor (35), and others established it as a (partially) protecting factor; when the majority in a (bad) neighborhood has the same ethnicity as the client, this appears to be protective (36). On the contrary, Dutch studies do not show a difference between victims and non-victims in relation to ethnicity (37, 38). In our effect study (16), we also found that it had no predictive value on the outcome measures. As it may still have an influence on the dynamic of the conversations on victimization between mental health staff and their clients, and this was not addressed in the semi-structured interviews, future studies should address this topic more directly.

Mental healthcare still focuses on the treatment of symptoms rather than rehabilitation (39). Concentrating on victimization experiences may lead to an overemphasis on the pain and emotions that clients experience. Several of the professionals used victimization experiences as the starting point of the intervention rather than participation issues, and this presented the possibility of the aforesaid. This accords with previous research (40). Traumatic experiences should not be dwelt upon, but they should be treated. Such an approach is incorporated in the Victoria intervention in several respects. Those implementing it are required to start by discussing struggles with participation. In the third step, the mindset shifts away from the victimization experience toward the wish to participate. Finally in the fourth step, one of the options is trauma-focused treatment. Even though we paid attention to the need not to dwell on recent victimization experiences, future implementation of the intervention (and the accompanying training) should concentrate on this more.

In the training, intervision sessions, and interviews, we found that experts-by-experience understood the essence of the intervention and that it required time, but questioned whether they had the time for it. They pointed out that they were already using large parts of it. These findings were in keeping with previous studies. First, experts-by-experience adopted a strengths-based approach more than the other professionals (41), which can empower and instill hope in clients. By disclosing their own stories, it is possible for them to regain control over their illness (41, 42). Second, because the nature of the contact in the intervention is more low-key than is usual, so is the distance between the professional and the client. Third, experts-by-experience can discuss day-to-day life, such as how to overcome stigma, more often than would be the case with, say, psychologists (42). This also encourages empathy, because experts-by-experience have similar experiences and can relate more to their clients (43). In future implementations of the intervention, case managers will have to undergo a greater attitude shift than experts-by-experience.

We discovered that there were significant barriers to the use of the intervention on a regular basis, for example, increasingly heavy caseloads. This was a function of the way FACT teams tend to work. The professionals stated that 80% of their days should be spent on clients, but they have busy schedules and so have only a limited amount of time. The role of management in successful implementation is well-known (44); we also found that a lack of support resulted in diminished use of the intervention. This needs to be addressed. The intervention should be made part of treatment plans and a component of team performance indicators.

Our intervention was interwoven into the Boston Psychiatric Rehabilitation method (BPR) (31). All the teams involved in the intervention were certified FACT teams; however, the literature shows that scores on rehabilitation items are lower than other scores that comprise the overall FACT fidelity score (45). Our findings confirmed this; while all the teams were trained in the BPR, adherence to rehabilitation principles was insufficient. We learned in the interviews that team meetings focused on crises and that the professionals worked mostly individually. A recent study of FACT teams (46) concluded that, although their fidelity scores were low, rehabilitation efforts increased over time. However, when rehabilitation principles had been more standard practice among the teams in the present study, the Victoria intervention would have offered even greater improvements.

In addition, the FACT teams’ mission was ambiguous. This is an important issue because mission determines successful implementation. In Netherlands, FACT teams provide care for relatively stable clients using rehabilitation methods and support them during psychiatric crises. Continuity of care provides stability for clients. However, due to recent policy changes (18), FACT teams have been forced to adopt a more treatment-oriented approach and focus less on care per se. Together with regional differences in client populations and organizational networks, FACT teams specialize in, for example, first psychosis episodes or addiction, or focus on treatment at the expense of rehabilitation. The Victoria intervention can help address this issue.

Strengths and limitations

The present study has yielded valuable information on the significance of recognizing and addressing victimization in clients and the difficulties professionals have discussing victimization with their clients. Other strengths of the study include its mixed-methods design and the large sample of participants and number of observations. The study has some limitations. First, we included clients who received the intervention according to their case managers, so we could not measure its true extent. Secondly, while we measured the extent of use by employing several methods, we did not design a bespoke fidelity scale that might have proven more accurate.

Conclusion

Results from this process evaluation indicate that the intervention increased awareness of victimization. Even though the steps of the intervention were not always followed as they should have, the professionals involved reported positive experiences. The intervention gave them greater insights into their clients’ rehabilitation wishes and allowed them to discuss victimization experiences. This process evaluation has an important added value in that it gave us a better understanding of the effect evaluation of the intervention. Our findings might facilitate the development and implementation of other interventions in community mental healthcare settings. In particular, it is hoped that the Victoria intervention might aid clients’ societal participation.

Data availability statement

The datasets presented in this article are not readily available because the data that are used in this study contain sensitive patient information, such as their diagnosis and information about criminal victimization. For this reason, there are restrictions on public data sharing. The funding body, NWO, imposed a contract with the data repository Data Archiving and Networked Services (DANS), to encourage long-term storage and availability for further research. After the entire study is finished, DANS will make the data available upon reasonable request via https://www.narcis.nl/. Requests to access the datasets should be directed to WA, w.m.m.albers@tilburguniversity.edu..

Ethics statement

The studies involving human participants were reviewed and approved by the Medical Ethical Committee of the Elisabeth Hospital in Tilburg (NL53845.028.15) on November 18, 2015, for all participating sites. The study was registered with the Dutch Trial Register (NL4172). The patients/participants provided their written informed consent to participate in this study.

Author contributions

WA recruited participants and research assistants conducted the interviews. WA, YN, and DR carried out the analyses. All authors were involved in the study’s conceptualization and design, contributed to the article, and approved the submitted version.

Funding

This study was part of the Violence Against Psychiatric Patients Program and was financed by the Netherlands Organization for Scientific Research (NWO) (https://www.nwo.nl/), grant number: 432-12-808.

Acknowledgments

We thank all study participants, participating organizations, and research assistants for making the study possible. We also thank Sjaak Roza, Ernst van ‘t Veer, and Els Luijten for their contribution to the Victoria intervention training course.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2022.956133/full#supplementary-material

References

1. Latalova K, Kamaradova D, Prasko J. Violent victimization of adult patients with severe mental illness: a systematic review. Neuropsychiatr Dis Treat. (2014) 10:1925–39. doi: 10.2147/NDT.S68321

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Thornicroft G, Brohan E, Rose D, Sartorius N, Leese M. Global pattern of experienced and anticipated discrimination against people with schizophrenia: a cross-sectional survey. Lancet. (2009) 373:408–15. doi: 10.1016/s0140-6736(08)61817-6

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Ye J, Chen TF, Paul D, McCahon R, Shankar S, Rosen A, et al. Stigma and discrimination experienced by people living with severe and persistent mental illness in assertive community treatment settings. Int J Soc Psychiatr. (2016) 62:532–41. doi: 10.1177/0020764016651459

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Kamperman A, Henrichs J, Bogaerts S, Lesaffre E, Wierdsma A, Ghauharali R, et al. Criminal victimisation in people with severe mental illness: a multi-site prevalence and incidence survey in the Netherlands. PLoS One. (2014) 9:e91029. doi: 10.1371/journal.pone.0091029

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Lasalvia A, Zoppei S, Van Bortel T, Bonetto C, Cristofalo D, Wahlbeck K, et al. Global pattern of experienced and anticipated discrimination reported by people with major depressive disorder: a cross-sectional survey. Lancet. (2013) 381:55–62. doi: 10.1016/s0140-6736(12)61379-8

CrossRef Full Text | Google Scholar

6. Perese EF. Stigma, poverty, and victimization: roadblocks to recovery for individuals with severe mental illness. J Am Psychiatr Nurses Assoc. (2007) 13:285–95. doi: 10.1177/1078390307307830

CrossRef Full Text | Google Scholar

7. Thornicroft G, Brohan E, Rose D, Sartorius N, Leese M, van Weeghel J, et al. Global pattern of experienced and anticipated discrimination gainst people with schizophrenia; a cross-sectional survey. Lancet. (2009) 373:408–15. doi: 10.1016/S0140-6736(08)61817-6

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Fitzgerald PB, Castella ARD, Filia KM, Filia SL, Benitez J, Kulkarni J. Victimization of patients with schizophrenia and related disorders. Aust N Zea J Psychiatr. (2005) 39:169–74. doi: 10.1080/j.1440-1614.2005.01539.x

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Silver E. Mental disorder and violent victimization: the mediating role of involvement in conflicted social relationships. Criminology. (2002) 40:191–212. doi: 10.1111/j.1745-9125.2002.tb00954.x

CrossRef Full Text | Google Scholar

10. Bertram G, Stickley T. Mental health nurses, promoters of inclusion or perpetuators of exclusion? J Psychiatr Mental Health Nurs. (2005) 12:387–95. doi: 10.1111/j.1365-2850.2005.00849.x

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Lynch T. Selfhood: A Key to the Recovery of Emotional Wellbeing, Mental Health and the Prevention of Mental Health Problems, or a Psychology Self Help Book for Effective Living and Handling Stress. Mental Health Publishing (2011).

Google Scholar

12. Stuart SR, Tansey L, Quayle E. What we talk about when we talk about recovery: a systematic review and best-fit framework synthesis of qualitative literature. J Mental Health. (2017) 26:291–304. doi: 10.1080/09638237.2016.1222056

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Kaliniecka H, Shawe-Taylor M. Promoting positive risk management: evaluation of a risk management panel. J. Psychiatr Mental Health Nurs. (2008) 15:654–61. doi: 10.1111/j.1365-2850.2008.01289.x

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Burns-Lynch B, Salzer MS, Baron R. Managing Risk in Community Integration: Promoting the Dignity of Risk and Supporting Personal Choice. Philadelphia: Temple University Collaborative on Community Inclusion of Individuals with Psychiatric Disabilities (2011).

Google Scholar

15. Albers WMM, Nijssen YAM, Roeg DPK, Bongers IMB, van Weeghel J. Development of an intervention aimed at increasing awareness and acknowledgement of victimisation and its consequences among people with severe mental illness. Commun Mental Health J. (2021) 57:1375–86. doi: 10.1007/s10597-021-00776-y

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Albers WMM, Roeg DP, Nijssen YA, Deen ML, Bongers I, van Weeghel J. Intervention to prevent and manage the effects of victimization related to social participation for people with severe mental illness: results from a cluster randomized controlled trial. Psychiatr Rehabilit J. (2021) 44:254–65. doi: 10.1037/prj0000449

PubMed Abstract | CrossRef Full Text | Google Scholar

17. van Veldhuizen JR. Fact: a dutch version of act. Commun Mental Health J. (2007) 43:421–33.

Google Scholar

18. Westen K, Boyle P, Kroon H. An observational comparison of fact and act in the Netherlands and the US. BMC Psychiatr. (2022) 22:311. doi: 10.1186/s12888-022-03927-x

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Holley J, Chambers M, Gillard S. The impact of risk management practice upon the implementation of recovery-oriented care in community mental health services: a qualitative investigation. J Mental Health. (2016) 25:315–22. doi: 10.3109/09638237.2015.1124402

PubMed Abstract | CrossRef Full Text | Google Scholar

20. van den Berg DP, de Bont PA, van der Vleugel BM, De Roos C, de Jongh A, van Minnen A, et al. Trauma-focused treatment in Ptsd patients with psychosis: symptom exacerbation, adverse events, and revictimization. Schizophr Bull. (2015) 42:693–702. doi: 10.1093/schbul/sbv172

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: medical research council guidance. BMJ. (2015) 350:h1258. doi: 10.1136/bmj.h1258

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Oakley A, Strange V, Bonell C, Allen E, Stephenson J. Process evaluation in randomised controlled trials of complex interventions. BMJ. (2006) 332:413–6. doi: 10.1136/bmj.332.7538.413

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Steckler AB, Linnan L, Israel B. Process Evaluation for Public Health Interventions and Research. San Francisco: Jossey-Bass (2002).

Google Scholar

24. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new medical research council guidance. Br Med J. (2008) 337:a1655. doi: 10.1136/bmj.a1655

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Grant A, Guthrie B, Foy R, Treweek S, Dreischulte T. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials. (2013) 14:15. doi: 10.1186/1745-6215-14-15

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Albers WMM, Roeg DPK, Nijssen Y, Bongers IMB, van Weeghel J. Effectiveness of an intervention for managing victimization risks related to societal participation for persons with severe mental illness: a cluster rct study protocol. BMC Psychiat. (2018) 18:247. doi: 10.1186/s12888-018-1831-7

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Rogers E, Anthony W, Lyass A, Penk WA. Randomized clinical trial of vocational rehabilitation for people with psychiatric disabilities. Rehabilit Counsel Bull. (2006) 49:143–56.

Google Scholar

28. Morse JM. Determining sample size. Qual Health Res. (2000) 10:3–5. doi: 10.1177/104973200129118183

CrossRef Full Text | Google Scholar

29. Boddy CR. Sample size for qualitative research. Qual Market Res. (2016) 19:426–32. doi: 10.1108/QMR-06-2016-0053

CrossRef Full Text | Google Scholar

30. de Heer-Wunderink C. Treatment plans in psychiatric community housing programs: do they reflect rehabilitation principles? Psychiatr Rehabilit J. (2012) 35:454–9. doi: 10.1007/BF00754453

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Anthony WA, Cohen M, Farkas M, Gagne C. Psychiatric Rehabilitation. 2nd ed. Boston: CPR Boston University (2002).

Google Scholar

32. Ritchie J, Lewis J, Nicholls CM, Ormston R. Qualitative Research Practice: A Guide for Social Science Students and Researchers. Thousand Oaks, CA: Sage (2013).

Google Scholar

33. Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. (2013) 13:117. doi: 10.1186/1471-2288-13-117

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (coreq): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. (2007) 19:349–57. doi: 10.1093/intqhc/mzm042

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Teplin LA, McClelland GM, Abram KM, Weiner DA. Crime victimization in adults with severe mental illness: comparison with the national crime victimization survey. Arch Gen Psychiatr. (2005) 62:911–21. doi: 10.1001/archpsyc.62.8.911

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Harris MN, Daigle LE. Resiliency from violent victimization for people with mental disorders: an examination using a community sample. Am J Crim Just (2021). doi: 10.1007/s12103-021-09663-4 [Epub ahead of print].

CrossRef Full Text | Google Scholar

37. de Vries S, Kamperman A, Lako IS, Sjoerd S, Bervoets P, van der Gaag M, et al. Geweld tegen psychiatrische patiënten in Nederland. Maandblad Geestelijke Volksgezondheid. (2011) 1:8–19.

Google Scholar

38. de Mooij LD, Kikkert M, Lommerse NM, Peen J, Meijwaard SC, Theunissen J, et al. Victimisation in adults with severe mental illness: prevalence and risk factors. Br J Psychiatr. (2015) 207:515–22. doi: 10.1192/bjp.bp.113.143370

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Llewellyn-Beardsley J, Rennick-Egglestone S, Callard F, Crawford P, Farkas M, Hui A, et al. Characteristics of mental health recovery narratives: systematic review and narrative synthesis. PLoS One. (2019) 14:e0214678. doi: 10.1371/journal.pone.0214678

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Rose SC, Bisson J, Churchill R, Wessely S. Psychological debriefing for preventing post traumatic stress disorder (PTSD). Cochrane Libr. (2002) doi: 10.1002/14651858.CD000560

CrossRef Full Text | Google Scholar

41. Lloyd-Evans B, Mayo-Wilson E, Harrison B, Istead H, Brown E, Pilling S, et al. A systematic review and meta-analysis of randomised controlled trials of peer support for people with severe mental illness. BMC Psychiat. (2014) 14:39. doi: 10.1186/1471-244X-14-39

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Davidson L, Bellamy C, Guy K, Miller R. Peer support among persons with severe mental illnesses: a review of evidence and experience. World Psychiatr. (2012) 11:123–8. doi: 10.1016/j.wpsyc.2012.05.009

PubMed Abstract | CrossRef Full Text | Google Scholar

43. van de Ven P. The Role Of Social Support In The Aftermath Of Victimization: Interpersonal Aspects of Coming to Terms with a Victimization Experience. Doctoral dissertation. Tilburg: Tilburg University (2022). https://research.tilburguniversity.edu/en/publications/the-role-of-social-support-in-the-aftermath-of-victimization-inte

Google Scholar

44. McGorry P. Prevention, innovation and implementation science in mental health: the next wave of reform. Br J Psychiatr. (2013) 202(Suppl.54.):s3–4. doi: 10.1192/bjp.bp.112.119222

PubMed Abstract | CrossRef Full Text | Google Scholar

45. van Vugt M, Mulder C, Bähler M, Delespaul P, Westen K, Kroon H. Modelgetrouwheid van flexible assertive community treatment (f-act)-teams: resultaten van vijf jaar auditeren. Tijdschrift Voor Psychiatr. (2018) 60:441–8.

PubMed Abstract | Google Scholar

46. Sanches SA, Swildens WE, Schaefer B, Moerbeek M, Feenstra TL, van Asselt AD, et al. Effectiveness of the Boston university approach to psychiatric rehabilitation in improving social participation in people with severe mental illnesses: a randomized controlled trial. Front Psychiatr. (2020) 11:571640. doi: 10.3389/fpsyt.2020.571640

PubMed Abstract | CrossRef Full Text | Google Scholar