Technology-assisted adaptive recruitment strategy for a large nation-wide COVID-19 vaccine immunogenicity study in Brunei

A national study was conducted in Brunei to assess and compare the immunogenicity of the various brands of COVID-19 vaccines administered to the population as part of the National COVID-19 Vaccination Programme. Most of the population have had received at least 2 doses of BBIBP-CorV, AZD1222 or MRNA-1273 vaccines. Neutralising antibodies against SARS-CoV-2 induced by these vaccines will be analysed to infer population-level immune protection against COVID-19. During the 5-week recruitment period, 24,260 eligible individuals were invited to the study via SMS, out of which 2,712 participants were enrolled into the study. This paper describes the novel adaptive strategy used to recruit the study participants. Digital technology was leveraged to perform targeted online recruitment to circumvent the limitations of traditional recruitment methods. Technology also enabled stratified random selection of these eligible individuals who were stratified based on age, gender and vaccine brand. Data was extracted from the electronic health records, the national mobile health application and a third-party survey platform and integrated into a dedicated research platform called EVYDResearch. The instant availability and access to up-to-date data on EVYDResearch enabled the study team to meet weekly and adopt an adaptive recruitment strategy informed by behavioural science, where interventions could be quickly implemented to improve response rates. Some examples of these include incorporating nudge messaging into SMS invitations, involving the Minister of Health to make press announcements on this study, media coverage, setting up an enquiries hotline and reaching out to foreign language speaking expatriates of a local multinational company to participate in this study. Data integration from various data sources, real time information sharing and a strong teamwork led to good outcomes adaptable to the progress of recruitment, compared to the more time-consuming and static traditional recruitment methods.

Introduction

Recruiting eligible participants into research studies is challenging (1, 2). Recruiting study participants using traditional methods is a labour and resource intensive process. There are high financial costs involved in obtaining sufficient resources to run recruitment sites and execute the logistics, such as conducting on-site recruitment and interviews, data entry and various administrative duties (3).

The biggest bottleneck prior to starting any health-related study is identifying eligible study participants (4). Electronic health record (EHR) systems in many modern healthcare institutions are rich sources of real-world data (RWD) and can be leveraged to automatically generate a list of patients who fulfil the inclusion and exclusion criteria (5, 6). Having access to data from EHR and other different sources required for clinical studies gives researchers a preliminary expectation of the recruitment rate, and facilitates planning for recruitment (7).

Another important consideration when selecting study participants is to ensure they are representative of the population. Probability sampling eliminates selection bias as any individual in the target population has an equal chance of being selected into the study (8). In clinical research, it may not always be feasible to conduct random sampling in practice due to the inability to identify eligible study participants beforehand, as well as resource and time limitations. Researchers often resort to convenience sampling due to cost or difficulties in recruitment at the expense of statistical bias and lack of representation (9). Stratified random sampling is a more rigorous type of probability sampling and allows for comparison between strata to draw meaningful conclusions. Access to data from different sources like EHR and census has enabled the use of stratified random sampling by identifying eligible sampling population efficiently beforehand (10).

Study teams need to successfully execute recruitment strategies. Success can be monitored by evaluating the recruitment process and performance (11). Technology can produce real-time data which gives prompt insights into the current recruitment progress. Data-driven insights must be communicated regularly with the study teams (12) so that feedback for improvements can be given (13), and prompt, targeted adjustments of recruitment strategies can be made to improve recruitment outcomes. Synchronising the recruitment planning reduces the burden of labour and resource, which leads to improved response and streamlined data collection (12).

The aim of this paper is to present the method and recruitment strategy of a technology assisted adaptive recruitment strategy for a national level study to infer the current level of immune protection against COVID-19 at the population level in Brunei Darussalam. Firstly, participants' data were integrated from the national EHR database and other sources and then scanned for eligibility. Eligible individuals were randomly sampled and invited to the study. The live response rates were shared with stakeholders weekly to enable adaptive recruitment. Data on the confirmed participants were imported into a secure and collaborative research platform for data collection and analysis.

Background

Vaccination is effective in preventing severe illnesses and death from COVID-19 (14–16). However, different vaccine brands can elicit varying degrees of immune responses (17–20). Their effectiveness can be assessed by measuring the levels of specific antibody in the serum which can be used as a correlate of COVID-19 protection post-vaccination (21). Anti-SARS-CoV-2 antibodies can persist for several months after vaccination, but have been shown to wane over time (22, 23). Particularly, high levels of blocking antibodies called neutralising antibodies (NAb) have been correlated with survival (24, 25) and appear to be the best measure of vaccine efficacy (26).

It is of great interest to monitor immune responses to the various vaccine preparations in populations to evaluate the effectiveness of these programmes (27). Limited data exist on the comparative immunogenicity of different COVID-19 vaccines in Southeast Asian populations. In Brunei, different vaccine types which were distributed for the primary immunisation series to adults included BBIBP-CorV (whole inactivated virus), AZD1222 (viral vector) and mRNA-1273 (mRNA) (28). This was followed by a booster dose of either mRNA-1273 or BNT1626b (mRNA) administered at least 3 months following completion of the primary series (29). The technology assisted adaptive recruitment strategy was implemented in a cross-sectional large nation-wide COVID-19 vaccine immunogenicity study in Brunei. This national study aims to obtain 3,000 blood samples from eligible individuals who received at least two doses of BBIBP-CorV, AZD1222 or mRNA-1273 COVID-19 vaccines (30, 31). Their level of NAb specific to SARS-CoV-2 induced by these vaccines will be analysed using a surrogate virus neutralisation test (32).

Materials and equipment

Data source

To identify the eligible population for sampling, data pertinent to study inclusion and exclusion criteria were obtained from different sources. These variables are described below and also listed in Supplementary Table 1 together with their sources. Refer to the Methods section for definitions of the inclusion and exclusion criteria.

Most patient data were obtained from the Brunei Darussalam Health Information System (Bru-HIMS) and BruHealth (Supplementary Table 1). Bru-HIMS is the national electronic health record (EHR) system used in Brunei which has more than 95% of the population's primary and secondary healthcare data across the entire government health network (33). In addition to this, a national mobile health application developed and managed by EVYD Technology Limited (EVYD) called BruHealth was launched by the Ministry of Health, Brunei Darussalam (MOH), in May 2020 to provide a virtual platform to control the spread of COVID-19 in Brunei via digital contact tracing and a QR code premise check-in (34, 35). The BruHealth backend framework integrates with Bru-HIMS, allowing for communication among the subsystems. This data aggregation allows BruHealth to be further developed into a population health management app for Bruneians with more extensive functionalities, including health self-assessment during the pandemic, self-quarantine management, access to personal health records, activity trace map of COVID-19 positive patients, map of medical resources, online appointments for COVID-19 vaccinations and medical consultations, plus health management plans like customised exercise programmes (35).

Patient information relating to personal COVID-19 vaccination records were obtained from BruHealth. Other data pertinent to the study inclusion and exclusion criteria, socio-demographics variables and co-morbidities (extracted based on the International Statistical Classification of Diseases and Related Health Problems 10th Revision [ICD-10]) were obtained from both Bru-HIMS and Qualtrics online survey software (36). Data from these datasets were then incorporated into a proprietary research platform developed by EVYD called EVYDResearch. District of residence and immunosuppressive drugs were collected at recruitment site and directly entered into EVYDResearch by data loggers.

Based on the predicted response rate and current capacity, a subset of identified eligible individuals from the above data sources were invited into the study by short messaging service (SMS). This included a link to a questionnaire to indicate their agreement to participate in the study. The questionnaire also contains a link to the Patient Information Sheet in English and Malay (37, 38). If they agree to participate, their data will be imported into the research platform. If they do not wish to participate, their data will not be included in the study. All participants were registered on arrival at the study site to verify details and ensure inclusion and exclusion criteria were met. Participants were briefed about the study and given opportunities to ask clarifying questions before physically signing a consent form before blood sampling. Once informed consent was given, the phlebotomist drew 5mL of blood samples from each participant via venepuncture (31).

Data management

EYVDResearch is engaged as the data management platform for this study (39). It is a web-based application on a secure cloud that allows patients' RWD ingestion from multiple sources, such as Bru-HIMS, BruHealth and Qualtrics survey platform. These datasets have undergone a standard data engineering process called extract, transform, load (ETL).

The aim of ETL is to obtain a complete view of participants' RWD which are normally disparate and housed in different sources which can have differing formats. The ETL process involves extraction of all data from the various sources, transforming the data into a cleaner and more readable format and finally hosting the clean data in a secure, on-premise centralised data operating platform called EVYDENCE. Data quality checks automatically occur after the data has loaded into the EVYDENCE database. These involve checking for empty fields, compliance to field types, field lengths and standardised key words, consistency of linked data and temporal logic. Any errors in the data will be flagged for manual checking. The data will then be processed into high-quality structured data. The necessary data from EVYDENCE can be extracted to EVYDResearch. The high-quality standardised and structuralised data facilitates effective search queries and statistical analyses.

Centralising all RWD onto EVYDResearch means better data management for collaborators to perform complex patient searches and track research milestones, including current recruitment progress. There are dashboards for data visualisation and modules to conduct customisable search queries and statistical analysis. The data can also be exported for in-depth analysis on dedicated statistical software. There is also a track data versioning for auditing purposes.

In this study, MOH retains data ownership of all patients' RWD and holds the rights to grant data access to authorised personnel for the purposes of conducting the research. EVYDResearch follows the data security and privacy requirements of MOH. There is a permission management platform for an independent administrator to distribute or withdraw the data permission for the necessary users via the individual metadata settings. Users can only read and/or amend the necessary fields as per the authorisation given by MOH depending on their roles.

Due to the sensitivity of the data, non-disclosure agreements were signed prior to commencing the research. As data on EVYDResearch is hosted on the MOH server, external users are required to obtain consent from MOH to access the virtual private network and EVYDResearch. Participants' identifiable details are confidential and were not disclosed to the study teams on EVYDResearch. Verification of participants' information was only conducted by data loggers at the recruitment site. Each study participant on EVYDResearch was deidentified and assigned a patient ID and a cPass ID which were used as their identifiers (Supplementary Table 1). The cPass ID would only be assigned if they have given their blood sample.

Ethics approval

This study is approved by the Medical and Ethical Research Committee (MHREC) of the Ministry of Health, Brunei Darussalam (Reference No: MHREC/MOH/2021/14(1) dated 18th November 2021.

Methods

Recruitment sampling

The sampling population included all eligible patients who meet the inclusion and exclusion criteria as defined in Table 1. Individuals who met the inclusion criteria were fully vaccinated with either BBIBP-CorV, AZD1222 or mRNA-1273 COVID-19 vaccines for at least 15 days but no more than 45 days (3 to 6 weeks' interval), and are at least 18 years of age. Exclusion criteria included a history of COVID-19 or overseas travel between receiving the first dose of vaccine and blood sample collection (Table 1).

TABLE 1

Table 1. Inclusion and exclusion criteria of study participants.

Socio-demographic variables including age, sex, ethnicity and district of residence are included in the study. Co-morbidities like diabetes mellitus type 2, chronic kidney diseases, cardiovascular diseases, ischaemic heart diseases, cancers and immunosuppressive drugs were also included in the study.

The sample size for each vaccination brand is 1,000. As individual immunity may vary between gender, age, vaccine brands and time elapsed after receipt of vaccine, the study participants were randomly selected and stratified by vaccine brands, gender, age groups and by number of weeks after receipt of the last vaccine dose. The targeted sample size would be the same and equally distributed across all strata (Supplementary Tables 2, 3). The stratified random sampling was performed using a pseudorandom number generator called Mersenne Twister that was run on Python version 2.7.15 (40). This involved assigning random numbers to all eligible individuals and letting the algorithm randomly select the required number of individuals to be invited from each stratum.

Recruitment workflow

The recruitment was staged into 5 weekly batches. Each batch required a fortnight of careful planning and prompt executions by different stakeholders as detailed below and in Supplementary Figure 1.

The recruitment methodology was designed based on the underlying principle of having equal sample sizes across all strata (as shown in Supplementary Tables 2, 3). Ideally, each batch would consist of 750 consenting participants from the same post-vaccination week (see Supplementary Table 4) and all strata would have equal representation (see Supplementary Table 5).

Bearing the above principle in mind, the recruitment strategy would be adaptive to the progress of recruitment, and oversample or undersample the relevant strata as appropriate. For this reason, the final batch would serve as a buffer in case of shortfalls in recruitment numbers. Details of the adaptations to the recruitment strategy are elaborated in sections batch 1 to batch 5. Individuals who agreed to participate would be sampled once only for this study. Individuals who did not consent to join or did not respond were not contacted again to join this study another time.

The maximum capacity of the recruitment centre for each study week (depending on manpower and venue availability) restricted the maximum number of blood samples which can be obtained in each batch. Up to seven hourly slots were available daily, and each phlebotomist was estimated to withdraw five blood samples per hour. The expected response rate was decided during weekly meetings guided by the previous batches' response rates to determine the maximum number of invitations which can be sent out, and was calculated as follows:

$\begin{array}{l}No.ofinvitationstosendout=\frac{Maximumcapacity}{Expectedresponserate}\end{array}$

In order to prepare the resources on-site, participants were asked to register their agreement to participate in the study on a survey form hosted on a third-party survey platform called Qualtrics. Invitation SMSs with the survey link were sent to selected potential study participants. A link to the patient information sheet was also provided in the survey link for information regarding the study. Refer to Table 2 for the SMS templates. Participants were given 3 days to respond to the Qualtrics survey, and a reminder SMS was sent the day before Qualtrics closure. Qualtrics responses were reviewed to estimate the probable attendance at the recruitment centre and optimised resource availability.

TABLE 2

Table 2. Template of the SMS invitations and reminders sent to invited eligible individuals to participate in the study. Nudge messaging was incorporated into SMSes from Batch 3 onwards.

A final SMS reminder was sent a day before the scheduled appointment for blood collection. A registration process was conducted at the study centre before blood collection to verify participants eligibility, explain study details and obtain written informed consent, and verify and update information collected on EVYDResearch. Blood collection was performed by trained phlebotomists. Walk in participants were accepted for blood collection as long as inclusion and exclusion criteria were fulfilled and written informed consent was obtained.

A dry run was conducted 2 weeks prior to the start of recruitment in order to test the SMS system, on-site logistics and coordination within the study team. In addition, the first batch was considered as a dress rehearsal of the workflow and response rate. Weekly meetings were held during the recruitment period to discuss the sampling population, Qualtrics response rate, number of blood samples collected and projections for response for different stratum in the following week, which provided the necessary and timely information to adapt the recruitment strategy for the next batch. Strata with low response rates were highlighted to support oversampling decisions to boost sample size for the strata. To fill up the invitation quota, priority strata were identified and allocated first. The number of invitations remaining after said allocations would be distributed equally amongst the remaining strata. Appointment days, time slots, age groups and gender strata for each vaccine brand were also balanced as much as possible. The national vaccination plans were also considered to prioritise recruitment for specific vaccine brands if required.

Serum titres for nAb (in international units per mL) were then analysed and calculated using a WHO International Standard conversion tool (30, 41). All participants NAb results were provided by SMS as positive or negative findings, and they were provided with an information sheet to help them interpret their results at the time of blood sampling (42).

Results of the recruitment

A total of 24,260 individuals were invited to the study. 2,712 blood samples were obtained, which equated to a response rate of 11.2%. Target numbers were met across all age and gender strata for MRNA-1273 but fell short for AZD1222 and BBIBP-CorV. Adjusting the invitation numbers and adjusting the recruitment strategy at each batch had helped to control the sample size of each stratum. Refer to Figures 1, 2, Supplementary Table 6, and Tables 3, 4 for further breakdown of the recruitment process. The recruitment strategy for each batch is described as below.

FIGURE 1

Figure 1. Flow diagram of recruitment numbers.

FIGURE 2

Figure 2. Breakdown of recruitment numbers and response rates by batch. Interventions taken and participants' behaviours in each batch are as listed and further described in sections “Results of the recruitment” and “Discussion”.

TABLE 3

Table 3. At the end of the recruitment, blood samples were collected from a total of 2,712 participants with the below strata distribution by vaccine brand, age and gender.

TABLE 4

Table 4. The total distribution of the blood samples collected according to the respective strata of the number of weeks post–vaccination.

Batch 1

Being the first live run of the recruitment, a conservative approach was taken to avoid overstretching resources. There was media coverage and ministerial support for the initial recruitment. 1,200 people were invited into the study, and it was predicted that 60% of the invitees would turn up. When the list of eligible individuals for the current and upcoming batches was evaluated, it was noted that there was a small and limited population of those who received the BBIBP-CorV vaccine. Hence, the priority was to invite all individuals who received the BBIBP-CorV vaccine and were at post vaccination weeks 5 and 6. The remaining quota was equally distributed between AZD1222 and MRNA-1273 sampling groups who were 6 weeks post-vaccination. Based on the final response rate of 10% in Batch 1, the response rate for Batch 2 was adjusted to 20%.

Batch 2

Like in Batch 1, all eligible individuals who received the BBIBP-CorV vaccine were prioritised. As there was a lack of BBIBP-CorV immunised population, individuals who had received a booster dose following a primary series of BBIBP-CorV at 5 to 6 weeks post-vaccination (defined as 2 BBIBP-CorV doses followed by 1 dose of MRNA-1273/BNT162b2) began to be included in the study. Due to insufficient sample size, the remaining quota was distributed evenly between AZD1222 and MRNA-1273 groups who were 6 weeks post-vaccination. The Minister of Health expressed his support for this study once again. Ultimately, we invited 4,723 subjects with a 13% response rate.

Batch 3

The study team continued to assume a response rate of 20%. As there were only 134 eligible BBIBP-CorV individuals, including the booster group, all were invited. The AZD1222 and MRNA-1273 groups who were 4 to 5 weeks post-vaccination were then invited into the study.

The nudge text was also introduced. In addition, there was also targeted recruitment of BBIBP-CorV vaccinated expatriates from a prominent multi-national company (MNC). The vaccination centre administered booster doses to this subgroup. As the expatriates' first language is Mandarin Chinese and they could not understand English or Malay, it was possible they were unable to participate in the study due to the language barrier. To avoid losing out on these MNC expatriates as potential participants, their HR department was contacted via the Ministry of Finance and Economy so that the study can be communicated to them in Mandarin Chinese.

Despite targeted recruitment and nudge messaging, overall response was poorer, with only 374 blood samples obtained from the invited 3,482 people, garnering a reduced response rate of 11%.

Batch 4

The expected response rate was revised down to 15%. As only one-third of the targeted sample size was reached, more resources were added to increase the recruitment effort. Eligible individuals who received the BBIBP-CorV primary series vaccines, and Week 5 to 6 BBIBP-CorV primary series plus booster vaccines were invited into the study. Following that, eligible individuals who received AZD1222 and MRNA-1273 primary series who were 3 to 5 weeks post-vaccination were invited. Due to smaller than expected sample size of AZD1222 and MRNA-1273, those who received either the BNT162b2 or MRNA-1273 booster doses were invited to fill up the remaining quota. Despite efforts to engage the public in the study via social media and nudge messaging, the response rate was the poorest at 7% with only 284 blood samples collected.

Batch 5

As this was the last week for recruitment, the maximum capacity was further increased in an attempt to reach the targeted sample size. The predicted response rate of 15% was maintained. The current recruitment progress was reviewed and it was noted that that the age groups of over 50 years were undersampled across all vaccine brands. There was interest to investigate how immunosenescence in individuals over 50 years of age affects nAb levels (43). The age groups of 50 to 60 years, and above 60 years were oversampled by inviting all eligible individuals over 50 years old into the study, independent of the vaccine brands they had received.

As the BBIBP-CorV vaccinated group remained undersampled, all eligible individuals who had received the BBIBP-CorV vaccines were recruited into the study. Similarly, all eligible individuals who received AZD1222 and MRNA-1273 vaccines respectively were invited into the study. Females who were given the MRNA-1273 vaccines were undersampled and allocated last as they were already overrepresented in the study compared to males. The response rate for Batch 5 is 12% with 1,304 blood samples obtained from the 10,913 people who were invited.

The nudge messaging was also amended to better highlight the personal benefits of joining the study. There was also media coverage of the study during the recruitment.

Discussion

Many studies have been abandoned because they failed to reach a sufficient sample size in a timely manner (44). Despite deploying resources across multiple centres for recruitment, it can still be difficult to achieve the desired sample size in a short amount of time (45). In multi-centred recruitment, information on the recruitment progress will need to be shared by different sites and aggregated together for discussion and alignment before implementing any changes to the recruitment strategy (46, 47). The recruitment workflow described here demonstrates a responsive and agile approach to large-scale recruitment with the use of technological solutions to successfully achieve the recruitment target in a short period of time. The below outlines and justifies the intervening strategies which improved the response rate of the study.

Data extraction

The access to Bru-HIMS and BruHealth have facilitated the auto-generation of the full sampling frame of eligible participants who meet the inclusion and exclusion criteria. This enabled us to carry out a more accurate and rigorous stratified sampling and avoids the issues of convenience sampling. Pre-assigning appointment slots to potential participants based on the expected response rate in the invitations is an effective crowd control measure and prevents overstretching of resources at the recruitment centre. Pre-planning the schedule ensured a pleasant experience for both participants and staff which adheres to COVID-19 social distancing and venue capacity guidelines.

EVYDResearch played a key role in providing up-to-date information which influenced the recruitment strategies. It has a graphical user interface which has a dashboard that shows the overall recruitment progress according to different stratification factors in the sampling scheme by showing the live sample size of each stratum, e.g., age, gender, vaccine brands, time from latest dose to blood collection etc., (Supplementary Figure 2).

Users can conduct conditional search queries visualised on tree diagrams in EVYDResearch via drag and drop and drop-down lists to filter the fields and conditions based on the inclusion and exclusion criteria (Supplementary Figure 3). Coding knowledge is not required to conduct the searches. Multiple cohort searches can be run. The search queries and results can be saved as projects and can be refreshed to reflect live data. These projects can also run simple statistical analyses such as descriptive statistics and univariate analysis.

During the weekly meeting, these statistics were generated quickly so that stakeholders could see the current recruitment progress by strata breakdown. Based on this information, sampling strategies had been proposed to try and improve response rates. Using Supplementary Figure 4 as an example, the univariate analysis revealed lower than targeted numbers of females above 60 years of age who received the AZD1222 vaccine. This would mean that more individuals from this stratum would need to be invited for the following batch in order to increase the sample size.

Recruitment by SMS

The eligible participants can be contacted directly as their contact details are available on Bru-HIMS and BruHealth. Contacting by phone can be a tedious, costly and time-consuming process, and does not guarantee a good response rate (48–51). Although emails are convenient and cost-effective, response rates are poor as they are usually ignored (52). Considering the large numbers of participants that had to be invited into the study and 88% smart phone ownership in Brunei as of 2018 (53), SMS was considered as the best communication method for recruitment especially when the messaging process could be automated. Furthermore, recruitment via SMS generated higher participation rates compared to emails (54). The study by Amorim et al. (55) showed the effectiveness of SMS for recruitment as well. Having MOH as the sender ID provided the government endorsement for the study and improved legitimacy. Like many providers around the world, SMS is also utilised by various governmental agencies, including MOH, to contact people for official matters like appointment confirmations. The use of SMS to send appointment reminders are effective in ensuring attendance (56).

Strategies to improve response rate

Nudge messaging

Incorporating nudges into text messages is a form of applied behavioural science to encourage participation in desirable positive behaviours at low costs, such as encouraging the uptake of influenza vaccines and implementing strategic policies (57, 58). Nudges are an interventional tool to influence choices without imposing mandates or bans (59). Initially, introducing the nudge messaging of “This will measure your antibody level after vaccination” in Batch 3 did not improve response rate in Batches 3 and 4. This could be because the cohorts could be less receptive to vaccinations. This is further discussed in section human behaviour. Adding another nudge messaging of “An antibody test is reserved for you next week” in Batch 5 improved response rate to 12%. It could be inferred that the content and style of the nudge messaging play a more important role in influencing behaviour than the act of nudge alone.

Implementation intention prompts

Appointment slots have been pre-assigned to each participant about 2 weeks in advance to better manage the flow of people at the recruitment centre and to comply with social distancing guidelines. The inclusion of a date and a time for the blood sampling appointment was not just a logistical convenience, it also acted as a nudge to encourage autonomous behaviour towards willing participation (60).

Giving people a clear and concrete plan encourages their cooperation to attend the desired future appointment (61). Imposing a short deadline to respond to the survey creates a sense of urgency that spurs participants into completing the survey and indicate their commitment to the study (62). Having the appointment dates set for the following week encourages participation as participants only need to evaluate if they can attend the appointment (63).

Governmental support

To establish legitimacy of the research, offline support was sought from governmental agencies such as MOH. The Minister of Health has made press announcements about the study and how it would inform future health policies, and urged invited citizens to participate (64, 65). Organisational and professional credentials give people the confidence to trust that the research is safe, ethical and benefiting the society (66). This is especially important in Brunei where the transparent leadership shown in the handling of the COVID-19 pandemic has already cemented public trust in the government (67). When the press releases were made, the response rates climbed to as high as 13%.

The study team noted that expatriates from the People's Republic of China working in a multi-national company (MNC) attended the vaccination centre en masse to obtain their BBIBP-CorV vaccine. Vaccination staff on-site noted they are not conversant in the working languages of Brunei which are Malay and English. There was the possibility that this BBIBP-CorV sub-group could have been missed out on the study due to their inability to comprehend the SMS invitation in English and Malay. A decision was made to reach out to this group directly. Assistance was sought from the Ministry of Finance and Economy and the Brunei Investment Agency, who engaged the Human Resources team of the MNC. Internal communication of the study and purpose of recruitment was conducted in Mandarin Chinese. This direct approach allowed us to recruit an eligible sub-group who was initially out of reach due to the language barrier. Language barriers could limit responses and considerations will need to be made to strategise the communication and outreach strategies (68).

Enquiries hotline

SMS, patient information sheets and survey questionnaires are convenient and contained all relevant details relating to the study. As the entire recruitment process was done online, there was no opportunity for individuals to clarify any questions. A nominated WhatsApp hotline was set up by the study team to attend to enquiries. Having extended discussions with the subject experts on the study would improve individuals' understanding and increased their likelihood of consenting to participate in the study (69). Getting assistance with technical difficulties could also overcome some barriers of recruitment and improve response rate, such as inability to navigate the survey form. More than 400 texts and calls were received over the recruitment period. Feedback can be grouped into clarifications on the study, logistics and research outcomes (Table 5).

TABLE 5

Table 5. Enquiries received by the project hotline.

Feedback received from participants who contacted the enquiries hotline has refined the communication of the recruitment. For example, the lengthy patient information sheet was hosted on a separate webpage instead of the Qualtrics survey questionnaire as it caused confusion and had led to incomplete questionnaires. There were also some suggestions for improvement which could not be feasibly implemented due to limitation in logistics, such as provision of incentives and phlebotomy services at other sites. Additionally, the feedback received had also given insights on participants' behaviour and receptivity to the study which can inform the planning of future recruitment. This is further elaborated in Section Human behaviour.

By deploying the handling of enquiries to an offsite location, the recruitment centre was able to focus solely on sampling blood from confirmed participants. This allowed for fast turnover of blood sampling with minimal interruptions to onsite logistics.

Media coverage

One of the barriers to recruitment is the lack of awareness of the studies being conducted (13, 70). Media campaigns could help to increase public awareness in research participation (71). To ensure the different age demographics were reached, the study was covered by a variety of media outlets including print media, television and social media. In Brunei, the younger population are more frequent users of social media while the older population tend to consume traditional media (72, 73). Traditional media also cross-posted their coverage on the study recruitment on their respective social media accounts. As the general population in Brunei have a high level of trust in the local state media (74), having media coverage of the study in multiple outlets could have had an influential role in study participation.

There was traditional media coverage during early stages of the recruitment to encourage participation, including a televised interview with the study team who explained the purpose of the study to increase public understanding (64, 65, 75).

In Brunei, there is a heavy usage of social media amongst most of the population that are smartphone dependent (72). Hence, social media was a suitable platform to spread awareness and understanding of the study to the masses especially the younger population, hence indirectly increasing response rates. MOH and the study team have published social media posts and videos on their Instagram accounts, with high public engagement by responding to comments (76–79). The higher response rates in Batch 1, 2 and 5 coincided with the publication of social media posts during the invitation week of said batches. The timing and consistency of the social media posting might have played a role in influencing response rates as response rates were lower in Batches 3 and 4 when there was lesser media coverage of the recruitment.

Limitations

Delays to recruitment

Time taken for funding and ethics approval would need to be considered and proactively buffered for in the planning stages as the start of recruitment could be delayed and even impact the sample size, especially when the inclusion criteria are time-sensitive (80–82). Although ample preparations have been made, the study was delayed by a few months as the sudden introduction of government-mandated restricted movements in Brunei had halted most research operations when the second wave of the COVID-19 pandemic hit Brunei in August 2021.

Based on the National COVID-19 Immunisation Programme, BBIBP-CorV, AZD1222 and MRNA-1273 COVID-19 vaccines were rolled out in the order of their procurement (83–86). When recruitment finally commenced, the population of BBIBP-CorV vaccinated individuals who were at 2–6 weeks after vaccination was small as most of the population received AZD1222 and MRNA-1273 vaccines by default (83). As part of the adaptive recruitment strategy to maintain adequate power, the inclusion criteria were modified to include booster populations into the study.

Human behaviour

The recruitment team noted a spectrum of education levels and health literacy amongst the participants. Based on some of the enquiries received in Table 5, some participants confused the study for a booster vaccination appointment or a blood donation drive. Others thought the SMS they received was a scam. Successful recruitment requires addressing and considering participants' level of understanding so that they are capable to give informed consent (69). The enquiries hotline, government endorsement and media coverage played important roles in educating, clarifying doubts and instilling confidence in participants to take part in this study. The linking of the purpose of the study to informing health policies probably motivated some to enrol into the study as an act of patriotism or altruism (87). The disclosure of participants' nAb levels via SMS, together with the support of medical doctors manning the enquiries hotline to explain what the results mean, promote transparency of the study with tangible results benefitting the participants.

Despite the various strategies to encourage recruitment, it seemed that participants' personal views on COVID-19 vaccinations influenced their receptivity to participate in the study. For context, Brunei entered the transition phase of the National COVID-19 Recovery Framework on 19 November 2021 when 70% of its population had received two doses of COVID-19 vaccine (88). All public-facing employees must have received their full vaccination, and patrons to public amenities must be fully vaccinated (89). The announcement of the framework could mean that many anti-vaccinationists might feel coerced to become vaccinated in November so that they can partake in their usual social activities during the transition phase. By the time they received their full vaccination in December, they could have been invited to participate in Batch 3 or 4 as they meet the eligibility criteria. They were likely to express similar reservations towards research studies on COVID-19 vaccines, and this was reflected by an increase in hostile comments about the study and the vaccine by the enquiries hotline. This could be the main reason for the lower-than-expected response rates in batches 3 and 4 despite the implementation of nudge messaging and earlier media coverage.

The behaviour of Batch 5 was a stark contrast to Batches 3 and 4. A greater proportion of Batch 5 participants were recipients of booster doses who received their vaccinations in mid-December when MOH expedited the administration of booster doses in light of the developing Omicron variant (90). Participants who received their booster shots demonstrated more cooperation and interest in the study as noted by the increased positive reception reported by the enquiries hotline. Based on this experience, there may be a need to consider participants' inherent beliefs towards aspects of the study when planning the recruitment strategy.

Future operational streamlining

This study serves as a proof-of-concept study that optimal efficiency in research studies can be achieved by adaptive study recruitment and real time information sharing via data intelligent technology, which allows data integration and governance from multiple sources including EMR and other user-friendly frontend applications.

EVYDENCE data management ecosystem

EVYDENCE is the underlying data operating platform from which EVYDResearch obtained participants' RWD from. At the time of this study, EVYDENCE was still in its final stages of development and could not be fully leveraged. In the future state, EVYDENCE will form the underlying engine on which the various technological solutions are built on, creating a highly interconnected data ecosystem. With information from BruHIMS as the base, data from datasets like Qualtrics and BruHealth can also be fed into EVYDENCE. EVYDResearch will be able to extract only the necessary data from EVYDENCE in order to generate new data insights which will feedback into EVYDENCE and other solutions to form a close loop. Strict data security protocols are applied universally across the entire ecosystem to protect data confidentiality.

Recruitment via BruHealth

BruHealth has a bespoke backend platform, content management system (CMS) module which can be used to conduct survey-based research activities. In January 2021, a COVID-19 vaccine intention survey was rolled out on BruHealth to the public which garnered 50,658 responses (12% contact rate) (91). The mandate by MOH to check in to premises using BruHealth combined with the high average daily usage of BruHealth mean that this app can be potentially used to send push notifications to identified users to participate in this study. The survey could also be housed on BruHealth instead of a third-party server. Unfortunately, the CMS module is still in the development stages and will only be available in the first half of 2022. If CMS were to be utilised, the app can identify non-responders and push notifications to remind them to complete the submission, resulting in a more targeted notification push. This feature, once fully ready, can be used as a formal recruitment platform instead of employing an external third-party survey application. This is possible as the BruHealth app has an average of more than 300,000 daily active users as they use it for COVID-19 surveillance purposes. Participants meeting the inclusion and exclusion criteria can be identified through a connected system, and then be pushed notifications to take part in the study. Reminder notifications can also be pushed to these patients at defined intervals if they still have not responded to take part in the study, and they can be customised by researchers and incorporate various nudge techniques to get them to participate. Push notifications to targeted groups at the right times could influence participation, similar to its application in influenza vaccine uptake (92).

Invited population can indicate their interest to participate in the research study through the app and fill in relevant screening questionnaires in BruHealth, and these data can be reflected in EVYDResearch via EVYDENCE for data analysis. Additionally, BruHealth could incorporate an option in the user's profile to indicate an interest to participate in any research study and to be notified when a study is open for recruitment. This information could further be used to target eligible participants, which may improve the participants' response rate to join the study. Participants involved in this study could even be tagged in BruHealth so that they could be contacted to join future studies.

Applications and future directions

This first-hand experience is proof of concept that large-scale recruitment can be done using technology and EHR. However, it is not just technology, it involves a very good planning of the workflow and understanding the requirements of the research, whilst also leveraging technology to be able to quickly plan recruitment strategies which are flexible and responsive to current recruitment progress. Being able to automate recruitment frees up manpower to focus on more vital tasks such as the data analysis and management of recruitment.

Although this proof of concept is applied to an epidemiological study, there is great potential for this recruitment methodology to be repurposed for research in other domains, especially clinical trials. With access to an EHR or a similar system, the recruitment strategy discussed can be replicated with ease. Following this methodology overcomes the limitations of traditional recruitment methods, especially by consolidating resources to one platform, automation and freeing up more bandwidth for researchers to focus on conducting the study on volunteers rather than burdening themselves with recruitment.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.

Ethics statement

The studies involving human participants were reviewed and approved by Medical and Ethical Research Committee (MHREC) of the Ministry of Health, Brunei Darussalam. The patients/participants provided their written informed consent to participate in this study.

Author contributions

CS, SC, YWe, HG, LA, HS, MA, SH, ST, CT, XO, L-FW, YWa, AL, HL, JW, LN, and AC have contributed to the study design, study conceptualisation, and data analyses. CS drafted the manuscript in full. YWe and AC provided critical revision of the manuscript. CS, SC, YWe, YWa, AL, and HL managed data integration from BruHIMS and BruHealth. UBD and EVYD colleagues, including CS, SC, YWe, HG, LA, HS, YWa, AL, LN, and AC managed recruitment of study participants, data coordination and blood withdrawal. MA, SH, ST, and JW managed press release and policy support for the study. CT, XO, and L-FW managed nAb measurement. All authors contributed to the article and approved the submitted version.

Funding

This research was funded by the Council for Research and the Advancement of Technology and Science (CREATES) (grant no: MITC/CREATES PROJECT UBD 001), Ministry of Transport and Infocommunications, Brunei Darussalam (Brunei Side) and Temasek Foundation (Singapore side) (grant no.: 2021-TF-0001).

Acknowledgments

The research team would like to acknowledge the hard work and enthusiastic support of the student volunteers and phlebotomists from the Ministry of Health, as well as the participants for their interests and donation of blood for this study. A poster of the manuscript's content was presented by CS virtually during the ISPOR 2022 conference (93).

Conflict of interest

Author CS is a part-time employee of EVYD Technology Limited which was the technology partner and research collaborator of this study. EVYD Technology Limited is the developer of EVYDResearch, BruHealth and EVYDENCE platforms. SC, YWe, YWa, AL, and HL are full-time employees of EVYD Technology Limited.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2022.983571/full#supplementary-material

References

1. Carlisle B, Kimmelman J, Ramsay T, MacKinnon N. Unsuccessful trial accrual and human subjects protections: an empirical analysis of recently closed trials. Clin Trials. (2015) 12:77–83. doi: 10.1177/1740774514558307

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Galea S, Tracy M. Participation rates in epidemiologic studies. Ann Epidemiol. (2007) 17:643–53. doi: 10.1016/j.annepidem.2007.03.013

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Kesse-Guyot E, Assmann K, Andreeva V, Castebon K, Méjean C, Touvier M, et al. Lessons learned From methodological validation research in e-epidemiology. JMIR Public Health Surveill. (2016) 2:e160. doi: 10.2196/publichealth.5880

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Mahon E, Roberts J, Furlong P, Uhlenbrauck G, Bull J. Barriers to trial recruitment and possible solutions. Appl Clin Trials. (2016) 25:20–5. doi: 10.1093/ageing/afab147

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Cuggia M, Besana P, Glasspool D. Comparing semi-automatic systems for recruitment of patients to clinical trials. Int J Med Inform. (2011) 80:371–88. doi: 10.1016/j.ijmedinf.2011.02.003

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Melzer G, Maiwald T, Prokosch H-U, Ganslandt T. Leveraging real-world data for the selection of relevant eligibility criteria for the implementation of electronic recruitment support in clinical trials. Appl Clin Inform. (2021) 12:017–26. doi: 10.1055/s-0040-1721010

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Dugas M, Amler S, Lange M, Gerß J, Breil B, Köpcke W. Estimation of patient accrual rates in clinical trials based on routine data from hospital information systems. Methods Inf Med. (2009) 48:263–6. doi: 10.3414/ME0582

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Shorten A, Moorley C. Selecting the sample. Evid Based Nurs. (2014) 17:32–3. doi: 10.1136/eb-2014-101747

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Stratton SJ. Population research: convenience sampling strategies. Prehosp Disaster Med. (2021) 36:373–4. doi: 10.1017/S1049023X21000649

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Mercaldo ND, Brothers KB, Carrell DS, Clayton EW, Connolly JJ, Holm IA, et al. Enrichment sampling for a multi-site patient survey using electronic health records and census data. J Am Med Inform Assoc. (2019) 26:219–27. doi: 10.1093/jamia/ocy164

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Clinical Trials Transformation Initiative,. CTTI Recommendations: Efficient Effective Clinical Trial Recruitment Planning. (2016). Available online at: https://ctti-clinicaltrials.org/wp-content/uploads/2021/06/CTTI_Recruitment_Recs_Recruitment_Planning.pdf (accessed March 3, 2022).

Google Scholar

12. Huang GD, Bull J, Johnston McKee K, Mahon E, Harper B, Roberts JN. Clinical trials recruitment planning: a proposed framework from the clinical trials transformation initiative. Contemp Clin Trials. (2018) 66:74–9. doi: 10.1016/j.cct.2018.01.003

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Newington L, Metcalfe A. Factors influencing recruitment to research: qualitative study of the experiences and perceptions of research teams. BMC Med Res Methodol. (2014) 14:10. doi: 10.1186/1471-2288-14-10

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Zheng C, Shao W, Chen X, Zhang B, Wang G, Zhang W. Real-world effectiveness of COVID-19 vaccines: a literature review and meta-analysis. Int J Infect Dis. (2022) 114:252–60. doi: 10.1016/j.ijid.2021.11.009

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Zinatizadeh MR, Zarandi PK, Zinatizadeh M, Yousefi MH, Amani J, Rezaei N. Efficacy of mRNA, adenoviral vector, and perfusion protein COVID-19 vaccines. Biomed Pharmacother. (2022) 146:112527. doi: 10.1016/j.biopha.2021.112527

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Marfe G, Perna S, Shukla A. Effectiveness of COVID-19 vaccines and their challenges (Review). Exp Ther Med. (2021) 22:1407. doi: 10.3892/etm.2021.10843

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Xia S, Zhang Y, Wang Y, Wang H, Yang Y, Gao GF, et al. Safety and immunogenicity of an inactivated SARS-CoV-2 vaccine, BBIBP-CorV: a randomised, double-blind, placebo-controlled, phase 1/2 trial. Lancet Infect Dis. (2021) 21:39–51. doi: 10.1016/S1473-3099(20)30831-8

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Thomas SJ, Moreira ED, Kitchin N, Absalon J, Gurtman A, Lockhart S, et al. Safety and efficacy of the BNT162b2 mRNA COVID-19 vaccine through 6 months. N Engl J Med. (2021) 385:1761–73. doi: 10.1056/NEJMoa2110345

PubMed Abstract | CrossRef Full Text | Google Scholar

19. el Sahly HM, Baden LR, Essink B, Doblecki-Lewis S, Martin JM, Anderson EJ, et al. Efficacy of the mRNA-1273 SARS-CoV-2 vaccine at completion of blinded phase. N Engl J Med. (2021) 385:1774–85. doi: 10.1056/NEJMoa2113017

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Ramasamy MN, Minassian AM, Ewer KJ, Flaxman AL, Folegatti PM, Owens DR, et al. Safety and immunogenicity of ChAdOx1 nCoV-19 vaccine administered in a prime-boost regimen in young and old adults (COV002): a single-blind, randomised, controlled, phase 2/3 trial. Lancet. (2020) 396:1979–93. doi: 10.1016/S0140-6736(20)32466-1

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Earle KA, Ambrosino DM, Fiore-Gartland A, Goldblatt D, Gilbert PB, Siber GR, et al. Evidence for antibody as a protective correlate for COVID-19 vaccines. Vaccine. (2021) 39:4423–8. doi: 10.1016/j.vaccine.2021.05.063

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Brisotto G, Muraro E, Montico M, Corso C, Evangelista C, Casarotto M, et al. IgG antibodies against SARS-CoV-2 decay but persist 4 months after vaccination in a cohort of healthcare workers. Clin Chim Acta. (2021) 523:476–82. doi: 10.1016/j.cca.2021.10.035

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Doria-Rose N, Suthar MS, Makowski M, O'Connell S, McDermott AB, Flach B, et al. Antibody persistence through 6 months after the second dose of mRNA-1273 vaccine for COVID-19. N Engl J Med. (2021) 384:2259–61. doi: 10.1056/NEJMc2103916

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Dispinseri S, Secchi M, Pirillo MF, Tolazzi M, Borghi M, Brigatti C, et al. Neutralizing antibody responses to SARS-CoV-2 in symptomatic COVID-19 is persistent and critical for survival. Nat Commun. (2021) 12:2670. doi: 10.1038/s41467-021-22958-8

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Garcia-Beltran WF, Lam EC, Astudillo MG, Yang D, Miller TE, Feldman J, et al. COVID-19-neutralizing antibodies predict disease severity and survival. Cell. (2021) 184:476–88.e11. doi: 10.1016/j.cell.2020.12.015

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Khoury DS, Cromer D, Reynaldi A, Schlub TE, Wheatley AK, Juno JA, et al. Neutralizing antibody levels are highly predictive of immune protection from symptomatic SARS-CoV-2 infection. Nat Med. (2021) 27:1205–11. doi: 10.1038/s41591-021-01377-8

PubMed Abstract | CrossRef Full Text | Google Scholar

27. dos Santos Ferreira CE, Gómez-Dantés H, Junqueira Bellei NC, López E, Nogales Crespo KA, O'Ryan M, et al. The role of serology testing in the context of immunization policies for COVID-19 in Latin American countries. Viruses. (2021) 13:2391. doi: 10.3390/v13122391

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Ministry of Health (Brunei Darussalam). COVID-19 Vaccines Authorized and Administered in Brunei Darussalam. Available online at: https://www.moh.gov.bn/Shared%20Documents/2019%20ncov/Details%20of%20COVID-19%20Vaccine%20in%20Brunei.pdf (accessed May 5, 2022).

Google Scholar

29. Ministry of Health (Brunei Darussalam). Media Statement on the Current Situation of COVID-19 in Brunei Darussalam - 16 Jamadilawal 1443 / 21 December 2021. (2021). Available online at: https://www.moh.gov.bn/Lists/Latest%20news/NewDispForm.aspx?ID=1138 (accessed May 5, 2022).

Google Scholar

30. Sharif H, Ghani H, Ahmad L, Tan CW, Wong J, Bagol S, et al. Heterologous prime-boost with mRNA-based COVID-19 booster vaccination elicits high levels of neutralizing antibodies.

Google Scholar

31. Ghani H, Ahmad L, Sharif H, Wong J, Bagol S, Alikhan MF, et al. Immunogenicity of COVID-19 vaccines and levels of SARS-CoV-2 neutralizing antibody in the Bruneian population: Protocol for a national study.

Google Scholar

32. Tan CW, Chia WN, Qin X, Liu P, Chen MI-C, Tiu C, et al. A SARS-CoV-2 surrogate virus neutralization test based on antibody-mediated blockage of ACE2–spike protein–protein interaction. Nat Biotechnol. (2020) 38:1073–8. doi: 10.1038/s41587-020-0631-z

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Ministry of Health (Brunei Darussalam). Bru-HIMS Introduction. Available online at: https://www.moh.gov.bn/SitePages/Bru-HIMS Introduction.aspx (accessed May 5, 2022).

Google Scholar

34. Ministry of Health (Brunei Darussalam). BruHealth. Available online at: https://www.moh.gov.bn/SitePages/bruhealth.aspx (accessed May 5, 2022).

Google Scholar

35. EVYD Technology Limited. Solutions - EVYD Technology. Available online at: https://www.evydtech.com/solutions/ (accessed May 5, 2022).

Google Scholar

36. Qualtrics. Online Survey Software - 1B+ Surveys Annually. (2022). Available online at: https://www.qualtrics.com/core-xm/survey-software/ (accessed May 5, 2022).

Google Scholar

37. UBD PAPRSB Institute of Health Sciences. Participant Information Sheet (Malay) for Brunei COVID-19 Antibody Study (Dec 2021-Jan 2022). (2022). Available online at: https://ihs.ubd.edu.bn/participant-information-sheet-malay-for-brunei-covid-19-antibody-study-dec-2021-jan-2022/ (accessed May 5, 2022).

Google Scholar

38. UBD PAPRSB Institute of Health Sciences. Participant Information Sheet (English) for Brunei COVID-19 Antibody Study (Dec 2021-Jan 2022). (2022). Available online at: https://ihs.ubd.edu.bn/participant-information-sheet-english-for-brunei-antibody-study-dec-2021-jan-2022/ (accessed May 5, 2022).

Google Scholar

39. EVYD Technology Limited. Research - EVYD Technology. Available online at: https://www.evydtech.com/solutions/research/ (accessed May 5, 2022).

Google Scholar

40. Matsumoto M, Nishimura T. Mersenne twister. ACM transactions on modeling and computer. Simulation. (1998) 8:3–30. doi: 10.1145/272991.272995

CrossRef Full Text | Google Scholar

41. Zhu F, Althaus T, Tan CW, Costantini A, Chia WN, van Vinh Chau N, et al. WHO international standard for SARS-CoV-2 antibodies to determine markers of protection. The Lancet Microbe. (2022) 3:e81–2. doi: 10.1016/S2666-5247(21)00307-4

PubMed Abstract | CrossRef Full Text | Google Scholar

42. UBD PAPRSB Institute of Health Sciences. MoH Info for cPass Test Recipients. (2022). Available online at: https://ihs.ubd.edu.bn/info-for-cpass-test-recipients/ (accessed May 5, 2022).

Google Scholar

43. Bates TA, Leier HC, Lyski ZL, Goodman JR, Curlin ME, Messer WB. Tafesse FG. Age-dependent neutralization of SARS-CoV-2 and P1 variant by vaccine immune serum samples. JAMA. (2021) 326:868–9. doi: 10.1001/jama.2021.11656

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Watson JM, Torgerson DJ. Increasing recruitment to randomised trials: a review of randomised controlled trials. BMC Med Res Methodol. (2006) 6:34. doi: 10.1186/1471-2288-6-34

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Monaghan H, Richens A, Colman S, Currie R, Girgis S, Jayne K, et al. Randomised trial of the effects of an additional communication strategy on recruitment into a large-scale, multi-centre trial. Contemp Clin Trials. (2007) 28:1–5. doi: 10.1016/j.cct.2006.06.004

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Mitchell EJ, Godolphin PJ, Meakin G, Sprange K. Do investigator meetings improve recruitment rates in clinical trials? a retrospective before-and-after study of data from nine multi-centre clinical trials. Trials. (2020) 21:514. doi: 10.1186/s13063-020-04465-1

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Myers BA, Pillay Y, Guyton Hornsby W, Shubrook J, Saha C, Mather KJ, et al. Recruitment effort and costs from a multi-center randomized controlled trial for treating depression in type 2 diabetes. Trials. (2019) 20:621. doi: 10.1186/s13063-019-3712-x

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Hu K, Tardif J, Huber M, Daly M, Langford AT, Kirby R, et al. Chasing the storm: recruiting non-hospitalized patients for a multi-site randomized controlled trial in the United States during the COVID-19 pandemic. Clin Transl Sci. (2021) 15:831–7. doi: 10.1111/cts.13211

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Bracken K, Keech A, Hague W, Kirby A, Robledo KP, Allan C, et al. Telephone call reminders did not increase screening uptake more than SMS reminders: a recruitment study within a trial. J Clin Epidemiol. (2019) 112:45–52. doi: 10.1016/j.jclinepi.2019.04.009

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Wacher NH, Reyes-Sánchez M, Vargas-Sánchez HR, Gamiochipi-Cano M, Rascón-Pacheco RA, Gómez-Díaz RA, et al. Stepwise strategies to successfully recruit diabetes patients in a large research study in Mexican population. Prim Care Diabetes. (2017) 11:297–304. doi: 10.1016/j.pcd.2017.02.005

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Treweek S, Pitkethly M, Cook J, Fraser C, Mitchell E, Sullivan F, et al. Strategies to improve recruitment to randomised trials. Cochrane Database Syst Rev. (2018) 2:MR000013. doi: 10.1002/14651858.MR000013.pub6

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Murphy CC, Craddock Lee SJ, Geiger AM, Cox J, Ahn C, Nair R, et al. A randomized trial of mail and email recruitment strategies for a physician survey on clinical trial accrual. BMC Med Res Methodol. (2020) 20:123. doi: 10.1186/s12874-020-01014-x

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Authority for Info-Communications Technology Industry of Brunei Darussalam (AITI). Brunei Darussalam Information Communications Technology (ICT) Household Report 2019. (2019). Available online at: https://www.aiti.gov.bn/Shared%20Documents/Final%20-%20ICT%20HOUSEHOLD%20REPORT.PDF (accessed March 4, 2022).

Google Scholar

54. Chow EPF, Fairley CK, Lee DM, Ong JJ, Rahman R, Afrizal A, et al. Is SMS text or email more effective for recruitment into sexual health research? Sex Transm Infect. (2022) 98:390–390. doi: 10.1136/sextrans-2022-055436

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Amorim AB, Coombs D, Richards B, Maher CG, Machado GC. Text messaging and web-based survey system to recruit patients with low back pain and collect outcomes in the emergency department: observational study. JMIR Mhealth Uhealth. (2021) 9:e22732. doi: 10.2196/22732

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Boksmati N, Butler-Henderson K, Anderson K, Sahama T. The effectiveness of SMS reminders on appointment attendance: a meta-analysis. J Med Syst. (2016) 40:90. doi: 10.1007/s10916-016-0452-2

PubMed Abstract | CrossRef Full Text | Google Scholar

57. Milkman KL, Patel MS, Gandhi L, Graci HN, Gromet DM, Ho H, et al. A megastudy of text-based nudges encouraging patients to get vaccinated at an upcoming doctor's appointment. Proc Natl Acad Sci U S A. (2021) 118:e2101165118. doi: 10.1073/pnas.2101165118

PubMed Abstract | CrossRef Full Text | Google Scholar

58. Benartzi S, Beshears J, Milkman KL, Sunstein CR, Thaler RH, Shankar M, et al. Should governments invest more in nudging? Psychol Sci. (2017) 28:1041–55. doi: 10.1177/0956797617702501

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Sunstein CR, Thaler RH. Nudge: Improving Decisions About Health, Wealth and Happiness. 1st ed London: Allen Lane. (2021).

PubMed Abstract | Google Scholar

60. Milkman KL, Beshears J, Choi JJ, Laibson D, Madrian BC. Using implementation intentions prompts to enhance influenza vaccination rates. Proc Natl Acad Sci U S A. (2011) 108:10415–20. doi: 10.1073/pnas.1103170108

PubMed Abstract | CrossRef Full Text | Google Scholar

61. Gollwitzer PM, Sheeran P. Implementation intentions and goal achievement: a meta-analysis of effects and processes. Adv Exp Soc Psychol. (2006) 38:69–119. doi: 10.1016/S0065-2601(06)38002-1

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Zhu M, Yang Y, Hsee CK. The mere urgency effect. J Consum Res. (2018) 45:673–90. doi: 10.1093/jcr/ucy008

CrossRef Full Text | Google Scholar

63. Liberman N, Trope Y. The role of feasibility and desirability considerations in near and distant future decisions: a test of temporal construal theory. J Pers Soc Psychol. (1998) 75:5–18. doi: 10.1037/0022-3514.75.1.5

CrossRef Full Text | Google Scholar

64. Hj Abu Bakar R. 3,000 people to participate in local study on protective immunity of COVID-19 vaccines. The Scoop. (2021). Available online at: https://thescoop.co/2021/11/28/3000-people-to-participate-in-local-study-on-protective-immunity-of-covid-19-vaccines/?fbclid=IwAR2NsEZrv4uQxGfRauekle08iVpvmKtVH0_k1DfJdVlo_zq2fHM9X4wG2eg (accessed May 7, 2022).

Google Scholar

65. Han S, Bandial A,. More Participants Sought to Study COVID Vaccine Efficacy—The Scoop. The Scoop. (2021). Available online at: https://thescoop.co/2021/12/07/more-participants-sought-to-study-covid-vaccine-efficacy/ (accessed May 7, 2022).

Google Scholar

66. Dixon-Woods M, Tarrant C. Why do people cooperate with medical research? findings from three studies. Soc Sci Med. (2009) 68:2215–22. doi: 10.1016/j.socscimed.2009.03.034

PubMed Abstract | CrossRef Full Text | Google Scholar

67. Wong J, Koh WC, Alikhan MF, Abdul Aziz ABZ, Naing L. Responding to COVID-19 in Brunei Darussalam: lessons for small countries. J Glob Health. (2020) 10:e10363. doi: 10.7189/jogh.10.010363

PubMed Abstract | CrossRef Full Text | Google Scholar

68. Thakur N, Lovinsky-Desir S, Appell D, Bime C, Castro L, Celedón JC, et al. Enhancing recruitment and retention of minority populations for clinical research in pulmonary, critical care, and sleep medicine: an official American Thoracic Society Research statement. Am J Respir Crit Care Med. (2021) 204:e26–50. doi: 10.1164/rccm.202105-1210ST

PubMed Abstract | CrossRef Full Text | Google Scholar

69. Nishimura A, Carey J, Erwin PJ, Tilburt JC, Murad MH, McCormick JB. Improving understanding in the research informed consent process: a systematic review of 54 interventions tested in randomized control trials. BMC Med Ethics. (2013) 14:28. doi: 10.1186/1472-6939-14-28

PubMed Abstract | CrossRef Full Text | Google Scholar

70. Reynolds T. Clinical trials: can technology solve the problem of low recruitment? BMJ. (2011) 342:d3662–d3662. doi: 10.1136/bmj.d3662

PubMed Abstract | CrossRef Full Text | Google Scholar

71. Mackenzie IS, Wei L, Rutherford D, Findlay EA, Saywood W, Campbell MK, et al. Promoting public awareness of randomised clinical trials using the media: the ‘Get Randomised' campaign. Br J Clin Pharmacol. (2010) 69:128–35. doi: 10.1111/j.1365-2125.2009.03561.x

PubMed Abstract | CrossRef Full Text | Google Scholar

72. Anshari M, Alas Y, Hardaker G, Jaidin JH, Smith M, Ahad AD. Smartphone habit and behavior in Brunei: personalization, gender, and generation gap. Comput Human Behav. (2016) 64:719–27. doi: 10.1016/j.chb.2016.07.063

CrossRef Full Text | Google Scholar

73. Abdullah Z. Accessing news among the elderly in Brunei Darussalam. Borneo Research Journal. (2019) 13:49–62. doi: 10.22452/brj.vol13no1.3

CrossRef Full Text | Google Scholar

74. Tokuda Y, Fujii S, Jimba M, Inoguchi T. The relationship between trust in mass media and the healthcare system and individual health: evidence from the AsiaBarometer Survey. BMC Med. (2009) 7:4. doi: 10.1186/1741-7015-7-4

PubMed Abstract | CrossRef Full Text | Google Scholar

75. Kon J. Large-scale study on COVID-19 vaccines' response underway. Borneo Bulletin. (2021). Available online at: https://borneobulletin.com.bn/large-scale-study-on-covid-19-vaccines-response-underway/ (accessed May 7, 2022).

Google Scholar

76. Universiti Brunei Darussalam,. Instagram Video Post on the Research Team Volunteers Behind the Ongoing Covid-19 Antibody Study. (2021). Available online at: https://www.instagram.com/tv/CX-UG_RKTxx/ (accessed May 7, 2022).

Google Scholar

77. UBD PAPRSB Institute of Health Sciences. Instagram Video Post on Study Team Interview. (2021). Available online at: https://www.instagram.com/tv/CYVQWJtpZ0-/ (accessed May 7, 2022).

Google Scholar

78. UBD PAPRSB Institute of Health Sciences. Instagram Video Post on the Tour of the On-Site Recruitment Process. (2021). Available online at: https://www.instagram.com/tv/CYSZu5bpSo_/ (accessed May 7, 2022).

Google Scholar

79. Ministry of Health (Brunei Darussalam). Instagram Post on “Immunogenecity of COVID-19 Vaccines and Levels of SARS-CoV-2 Neutralising Antibody in the Bruneian Population.” (2021). Available online at: https://www.instagram.com/p/CW4gonQrfzr/ (accessed May 7, 2022).

Google Scholar

80. Tikkinen KAO, Malekzadeh R, Schlegel M, Rutanen J, Glasziou P. COVID-19 clinical trials: learning from exceptions in the research chaos. Nat Med. (2020) 26:1671–2. doi: 10.1038/s41591-020-1077-z

PubMed Abstract | CrossRef Full Text | Google Scholar

81. Campbell M, Snowdon C, Francis D, Elbourne D, McDonald A, Knight R, et al. Recruitment to randomised trials: strategies for trial enrolment and participation study. The STEPS study. Health Technol Assess. (2007) 11:11480. doi: 10.3310/hta11480

PubMed Abstract | CrossRef Full Text | Google Scholar

82. Zahren C, Harvey S, Weekes L, Bradshaw C, Butala R, Andrews J, et al. Clinical trials site recruitment optimisation: guidance from clinical trials: impact and quality. Clin Trials. (2021) 18:594–605. doi: 10.1177/17407745211015924

PubMed Abstract | CrossRef Full Text | Google Scholar

83. Bandial A,. Astrazeneca is Now Brunei's Default Vaccine. The Scoop. (2021). Available online at: https://thescoop.co/2021/09/18/astrazeneca-will-be-default-vaccine-from-now-on/ (accessed May 8, 2022)

Google Scholar

84. The Scoop. Brunei, Singapore Agree to Vaccine Swap. The Scoop. (2021). Available online at: https://thescoop.co/2021/09/29/brunei-singapore-agree-to-vaccine-swap/ (accessed May 8, 2022).

Google Scholar

85. The Scoop. Australia donates 100,000 doses of AstraZeneca to Brunei. The Scoop. (2021). Available online at: https://thescoop.co/2021/09/18/australia-donates-100000-doses-of-astrazeneca-to-brunei/ (accessed May 8, 2022).

Google Scholar

86. Hj Abu Bakar R,. MoH to Restart First Dose Vaccinations on Thursday. The Scoop. (2021). Available online at: https://thescoop.co/2021/09/13/moh-to-restart-first-dose-vaccinations-on-thursday/ (accessed May 8, 2022).

Google Scholar

87. Williams B, Entwistle V, Haddow G, Wells M. Promoting research participation: Why not advertise altruism? Soc Sci Med. (2008) 66:1451–6. doi: 10.1016/j.socscimed.2007.12.013

PubMed Abstract | CrossRef Full Text | Google Scholar

88. Han S,. Brunei Ushers in COVID Transition Phase With 45 New Cases. The Scoop. (2021). Available online at: https://thescoop.co/2021/11/19/brunei-ushers-in-covid-transition-phase-with-45-new-cases/ (accessed May 7, 2022).

Google Scholar

89. Prime Minister's Office (Brunei Darussalam),. Announcement of the Transition Phase of the National COVID-19 recovery framework. (2021). Available online at: https://www.pmo.gov.bn/SiteCollectionDocuments/covid19/[English-FINAL-2] JPM - 091121 Announcement on Transition Phase.pdf (accessed May 5, 2022)

Google Scholar

90. Han S,. Brunei to Bring Forward COVID Booster Jabs in Wake of Omicron. The Scoop. (2021). Available online at: https://thescoop.co/2021/12/14/brunei-to-bring-forward-covid-booster-jabs-in-wake-of-omicron/ (accessed May 7, 2022).

Google Scholar

91. Bandial A,. 59% of Residents Say They are Prepared to Take COVID-19 Vaccine. The Scoop. (2021). Available online at: https://thescoop.co/2021/02/06/59-of-residents-say-they-are-prepared-to-take-covid-19-vaccine/ (accessed May 8, 2022).

Google Scholar

92. Dale LP, White L, Mitchell M, Faulkner G. Smartphone app uses loyalty point incentives and push notifications to encourage influenza vaccine uptake. Vaccine. (2019) 37:4594–600. doi: 10.1016/j.vaccine.2018.04.018

PubMed Abstract | CrossRef Full Text | Google Scholar

93. Shim C, Ghani H, Ahmad L, Sharif H, Alikhan M, Bagol S, et al. Technology-assisted adaptive recruitment strategy for a national study of immunogenicity of COVID-19 vaccines and levels of SARS-CoV-2 neutralising antibody in the Bruneian population. Value Health. (2022) 25:S1. doi: 10.1016/j.jval.2022.04.1682

CrossRef Full Text | Google Scholar