Skip to main content

REVIEW article

Front. Vet. Sci., 17 July 2024
Sec. Comparative and Clinical Medicine
This article is part of the Research Topic Best Practices in Clinical Research Conduct in Veterinary Medicine View all 4 articles

Best practices for recruitment in veterinary clinical trials

  • 1Veterinary Clinical Research Office, Virginia-Maryland College of Veterinary Medicine, Virginia Tech, Blacksburg, VA, United States
  • 2Clinical Studies, Department of Clinical Sciences, Ontario Veterinary College, University of Guelph, Guelph, ON, Canada
  • 3Department of Clinical Sciences, College of Veterinary Medicine and Biomedical Sciences, Colorado State University, Fort Collins, CO, United States

A successful clinical trial requires participants, but many factors can impede effective study recruitment. To better recruit for quality veterinary clinical trials in client-owned animals that lead to improved evidence-based patient care and outcomes, there is a collective need to share and implement current best practices for recruitment strategies. These strategies should utilize a holistic view of recruitment, encompassing study design and logistics, representative participation, incentives, personnel resources, advertising, and participant retention. Although human clinical trial data and resources can provide guidance, effort also needs to be put into evaluating current practices and opportunities for process improvement that are specific to the conduct of veterinary clinical trials. Considering the power of pets as naturally occurring models of disease and as sentinels, improved conduct of veterinary clinical research has the potential to inform human health outcomes. Continued development of collaborations surrounding best practices and training opportunities in veterinary clinical research will improve the impact of veterinary clinical trials teams, while also promoting workforce development and alternate career paths for veterinary professionals.

1 Introduction

Clinical trials in human and veterinary medicine aim to improve patient care and address unmet therapeutic needs. Not only can such research benefit patients directly, but the similarities between naturally occurring disease processes in companion animals and humans, along with our shared exposures, mean that new therapeutics and devices evaluated for one species may have benefits for others (1). While the potential advantages of such research are clear, how we conduct clinical trials is an area of continued discussion. The veterinary research community has made strides toward a more systematic and rigorous approach to best practices in companion animal clinical studies’ design, conduct, and ethical underpinnings (2). These best practice recommendations often mirror, to the extent practicable, those for human clinical research (3). However, the literature regarding one critical aspect, recruiting veterinary patients into trials, is sparse.

Experienced clinical investigators are often painfully aware that recruitment is more complex than putting up a flier and waiting for the phone to ring or an email to arrive. The facilitation of patient enrollment through outreach efforts such as advertisements, websites, print materials, and conversations between pet owners, referring veterinarians, in-hospital colleagues and coworkers, and study personnel is crucial. Additionally, a successful recruitment strategy must encompass all phases of the study, from design through data collection, and continue even after the study closes. This narrative review discusses current information and methods of optimizing patient enrollment, including designing recruitable studies, using incentives, managing a study communication plan, and building long-term, trusting relationships with pet owners, in-hospital colleagues and coworkers, and referring veterinarians. The current landscape regarding veterinary patient recruitment and the shortcomings of some common recruitment strategies are presented along with tools and additional considerations for clinical researchers as they plan and perform this crucial aspect of conducting quality clinical research.

2 Why is recruitment important?

In human clinical trials, enrolling eligible participants has been cited as the most substantial workload component (4, 5), with recruitment accounting for over 30% of study costs (6). Slow or ineffective study recruitment results in lost revenue, estimated at up to $8 million USD per day of delay in pharmaceutical trials (47). It has been reported that 79–90% of human clinical trials incur setbacks related to patient recruitment and fail to meet enrollment targets or timelines (79). Although similar data is unavailable in the veterinary clinical trials space, smaller study budgets and limited personnel in veterinary trials could be associated with an even higher failure percentage.

From a scientific standpoint, robust clinical trials require an adequately powered, representative group of participants from whom data can be collected within a defined time period. This limits variables and increases the applicability and reliability of study outcomes. While the veterinary profession has adopted Russell’s replace, reduce, and refine methodology for optimizing the numbers of patients needed in research studies, patient recruitment continues to be the lynchpin of successful trial outcomes (10). Successfully enrolling an appropriate sample size prevents animals, resources, and time from being used unnecessarily either due to overpowering, which can result in wastage, or underpowering, which can lead to results that must be discarded or heavily qualified (11). Most investigators conduct power analyses as part of their study design planning because underpowered studies are of limited value to inform clinical or scientific practice. However, problems with recruitment can lead to compromises or adjustments in patient numbers, resulting in underpowered studies and a higher incidence of type two errors (11, 12). Additionally, extending the study duration to allow additional time to enroll adequate patient numbers can lead to sample degradation, personnel changes, changes in resource availability, and other variations that decrease study quality. Thus, insufficient recruitment threatens the availability of quality, evidence-based medicine through impacts on study budget, study validity, knowledge transfer, and timelines (13).

Weak recruitment can also have direct effects on the research team. For paid research personnel, funding may run out prior to study completion. For students, residents, and interns, completion of projects is often linked to degree-granting or competitive advantage in the job market and ideally should be completed during the limited training program timeframe. Failure to complete meaningful studies not only delays improvements to clinical practice but can also undermine interest in continued participation in research and result in missed opportunities, such as critical grant cycles or promotion milestones. Challenges in recruitment for trainee projects can also threaten study quality and completion as the trainee looks to move forward with their career and another individual must be found to take over the partially-completed project.

Failure to complete trials and optimize trial outcomes also fails to honor the participating animals’ and pet owners’ time and effort, which can lead to a loss of trust. A survey of owner motivations for participating in clinical trials emphasized the primacy of establishing and maintaining a relationship of trust between research institutions and pet owners (14). As the concept of “animal wastage” has broadened to include the time, biological samples, and other contributions of companion animal trial participants, studies that end prematurely or fail to reach adequate statistical power erode public trust and undermine investigators’ ethical responsibilities to animal participants and funding agencies (3, 15). Accordingly, investigators must focus on case recruitment to ensure their study’s success and the sustainability and positive impact of research.

3 Overcoming barriers to recruitment

3.1 Study design

Serious recruitment issues may arise due to flawed or unrealistic study designs, such that recruitment goals may not be achievable or the reported results may not be meaningful. Investigators must carefully balance creating a well-designed, impactful study with ensuring they have the broadest possible enrollment criteria to facilitate rapid and inclusive recruitment. Lung cancer patient advocate Jill Feldman noted during a 2022 United States Department of Health and Human Services (HHS) Office of Human Research Protections Exploratory Workshop that investigators also need to consider what patients are giving up when they agree to participate in a clinical trial and what can be done to help them participate (16). A 2019 systematic review and meta-analysis found that structural and clinical barriers inhibited trial participation for over 75% of human cancer patients (17). Onerous recheck schedules, complex and burdensome requirements, need to travel, lack of transparency, insufficient financial incentives, lack of clinical benefit for enrolled pets, and overly-strict enrollment criteria can all contribute to suboptimal recruitment.

Randomization has long been considered a cornerstone of quality clinical trials; however, concerns over placebo group assignment may impact pet owners’ willingness to participate. A survey of cat owners by Gruen et al. (14) reported that 26% of respondents would be less or much less likely to have their cat participate in a study if there was a chance of them receiving a placebo. Interestingly, a similar survey of small animal practitioners noted that 74% believed that potential assignment to a placebo group was important or extremely important to clients’ decisions about clinical trial participation for their pet (18). The use of placebo groups in clinical patients can also raise ethical considerations (19, 20). Use of real-world data or study designs that allow all patients to receive treatment (e.g., cross-over studies, standard-of-care controls, adaptive design, interim assessments, and active-controls) can be considered to help overcome these issues. Each of these options has specific implications and must be evaluated critically during study design to ensure best patient and study outcomes. For example, although “standard-of-care” controls may be most familiar in the veterinary field and initially seem most medically ethical, using them inappropriately can undermine study outcomes through generation of misleading results that are difficult to interpret, compare, and implement and waste limited research resources (2123). A 2017 study found that 29% of human breast cancer randomized controlled trials may be using an inappropriate standard-of-care for control patients (22). Careful determination, delivery, and reporting of the “standard-of-care” used in a study, if a validated “standard-of-care” truly exists, as well as the appropriateness of this type of control in a specific study can help ensure the trial results will lead to improved patient care by guiding clinical practice recommendations and future clinical trial design (2124).

The scarcity of funding for veterinary clinical trials can limit investigators’ ability to perform well-designed studies. Limited budgets can lead to underpowered studies because of a lack of personnel, incentives, and failure to budget sufficient funds for drop-outs, complications, or unusable data. The need to alter study enrollment criteria or design due to recruitment failure increases study variables, which can decrease study validity, particularly given the small numbers of animals typically involved in veterinary clinical research.

Excessive optimism about case enrollment numbers and timeframes is another common study recruitment problem. “Lasagna’s Law” describes the perception that the incidence of patient availability sharply decreases when a study begins (9). This dip is due to the gap between the number of theoretically eligible study participants (as identified through prevalence measures, historical case data, etc.) and participants who actually enroll. In addition to searching epidemiological or historical medical record case data, it is essential to look at the overall disease incidence, thoroughly assess the enrollment criteria, engage with relevant colleagues to assess feasibility and applicability, and realistically gauge the willingness of owners to participate in the proposed study (9). Involving diverse, representative front-line veterinary professionals and pet-owning community members early in the research process through partnerships and participation on research review boards and clinical research teams can significantly enhance recruitment. This approach increases knowledge, awareness, and trust in clinical research for care providers and pet owners. Such engagement can also help identify community priorities and reduce barriers to communication and participation during the design phase of clinical trials.

Community engagement continues to expand in human clinical research as a means of addressing inequities in access to clinical trials and healthcare in diverse populations. Although the practice is less well-established and not currently required in veterinary medicine, the latest version of the American Veterinary Medical Association (AVMA) policy on the “Establishment and use of veterinary clinical studies committees” encourages the inclusion of “at least one person not affiliated with the entity performing the study” (25). Additionally, tracking the reasons for failure to enroll eligible animals can provide important insights to improve study design.

Recently, increased attention has been given to improving the scientific rigor of veterinary clinical study design as well as the reporting of clinical trial outcomes. Such initiatives aim to produce quality, transparent, and reproducible data that informs evidence-based clinical decision-making. Poor study design may result in a significant loss of time and resources and generate outcome assessments and noninformative comparisons that are clinically irrelevant, unnecessary, and potentially misleading (12). Resources such as the PetSORT Guidelines, a veterinary-focused adaptation of the Consolidated Standards of Reporting Trials (CONSORT), seek to create standard reporting guidelines that are harmonized and evidence- and consensus-based (3, 26, 27). The PetSORT checklist of items to address in a clinical trial report are useful for investigators to incorporate best practices in both study design and study reporting and for editors, reviewers, and content users to assess publications (3). With specific attention to recruitment, a simplified version of this checklist may be helpful to ensure that practical considerations are well-balanced with scientific rigor (Table 1).

Table 1
www.frontiersin.org

Table 1. Recruitment best practices checklist.

3.2 Logistics

Failure to comprehensively assess study logistics from the enrolled pet and owner’s perspective can limit recruitment. In human medicine, patient-centered clinical trials are becoming more common. When the public engages in medical research, it strengthens researcher accountability, increases communication and transparency, and may identify priorities and concerns not initially recognized by the researchers (28, 29). Stakeholder engagement has been incorporated into human clinical trials to address this issue. In 2018, the Clinical Trials Transformation Initiative (CTTI) proposed a framework for improving clinical trials recruitment emphasizing “human factors” as a key means of running more efficient trials. The framework encourages investigators to consider barriers such as physical access, time for appointments, and language difficulties, among others, during study design (4, 15, 30, 31).

Veterinary clinical trials require the pet owner to facilitate their pet’s participation, and therefore similar practices should be encouraged in the veterinary context. Practicalities should be weighed against the ideal study design. For example, when running a trial from a university research center, investigators’ recruitment efforts will likely benefit from the institution’s prestige. However, accessibility of the study site may influence enrollment and continued participation such that investigators may want to consider the possibility of having rechecks take place at the pet’s primary care practice. Although partnering with other institutions and hospitals comes with challenges, potential benefits can be seen in increased owner compliance, faster recruitment, and improved study power and reproducibility. Traditional offline recruitment strategies, such as soliciting in-clinic cases or clients within a short radius of the clinic, may be more convenient for the investigators. However, the limited scope of such strategies may create bias and limit the diversity of study participants and the generalizability of the study outcomes (7). Adapting clinical trial design to real-world circumstances may require larger participant numbers but has the potential to make study results more applicable to the intended population.

3.3 Representative participation

The socio-demographic characteristics of the pet owners should be considered when developing recruitment strategies to decrease bias and improve study outcomes and generalizability. Owners of sick pets incur increased costs associated with managing their pets’ health conditions. Some elements of a study’s design can inadvertently limit enrollment and fulfillment of study requirements by increasing the burden of pet owners. For example, if participation requires additional financial commitment in the form of missed work, travel, parking, pet care, etc., the subset of owners who are willing and able to participate shrinks – a demonstration of Lasagna’s Law in action (9). Investigators might end up only enrolling pets who benefit in other ways from their owners’ higher socio-demographic status, which could alter the study results by reducing the likelihood of enrolling a representative sample population.

The influence of lack of diversity and inclusion on clinical trial outcomes has been studied in human medicine and is an identified priority. Guidelines by CONSORT and PetSORT encourage the identification and tracking of reasons for refusals and loss to follow-up, which allows researchers to examine where skewed selection of participants or drop-outs due to socio-demographic concerns may arise (3, 4). Failure to include a diverse population can negatively impact the applicability of the research results. In human medicine, dose-finding studies often fail to include sufficient numbers of women despite their variable pharmacokinetics, body weight, and higher prevalence of medication use, and many post-approval drug withdrawals in the human market have been due to toxicity in women (32, 33). Biological sex is only one of many study participant variables (e.g., breed, species, age, comorbidities, exposures) that should be considered in optimal study design, particularly as knowledge advances in the area of precision medicine.

3.4 Incentives for participation

Clarity concerning study costs and obligations is critical to successful study recruitment. However, financial conversations are challenging and may be considered a lower priority by both the investigators and owners during complicated study enrollment discussions (16). Estimates of owners’ out-of-pocket costs may be missing from study descriptions or vague (e.g., “study-related costs are covered”).

Although more straightforward for a layperson to understand compared to medical terminology and procedures, the implications associated with incentives can be complex. Incentives can come in multiple forms, including reimbursement for costs incurred by study participants (i.e., parking, mileage), compensation for participants’ time, tokens of appreciation for study participation, as a part of the experimental intervention (i.e., studies investigating the influence of money on preferences and actions), and structured payments to encourage continued participation in the study. Studies might cover the costs of diagnostic testing, appointments, and other aspects of standard care, the cost of the study intervention, and allow access to novel treatments for participants.

Incentive payments involve many challenging issues, from ethical, theoretical, and psychological concepts, to increased financial, tax, and administrative burden. Research studies and scholarly discussion continue to question whether incentive payments undermine studies’ scientific validity and social value by biasing participants’ decision-making and compromising informed consent. An alternative view holds that incentives improve study conduct, inclusivity, and social value (34). Human clinical trial participants may find studies where compensation was not provided to be disrespectful of their time and effort, and some participants judge gift cards and other non-cash incentives to be patronizing and inadequate (16). However, incentives can also deter study recruitment if they are interpreted to undermine a participant’s contribution (e.g., participation is viewed as transactional instead of altruistic) or to be indicative of the risk of the study (3537). Careful consideration is needed to find the best ways to incorporate incentives into clinical trial design and conduct until additional evidence-based recommendations are available.

The literature about incentive payments in veterinary clinical trials is sparse. Gruen et al. (14, 18) reported that over 73% of surveyed cat owners and 87% of surveyed small animal practitioners ranked “free services” (e.g., laboratory tests, radiographs, examinations, etc.) as the “most encouraging” or “best” owner incentive to participate with their pet in a clinical trial. Current estimates in the United States and the United Kingdom put the percentage of insured pets between 1–22%, depending on location and species (3840). Given the lack of insurance coverage and the growing role of pets as companions and family members, the offer of free healthcare services may be especially powerful in the veterinary context. The tax implications of cash incentives may also play a role in these preferences.

In practice, robust financial incentives are likely uncommon in veterinary clinical trials. Evaluating 75 actively enrolling veterinary clinical studies listed on the American Veterinary Medical Association’s (AVMA) Animal Health Studies Database (AAHSD) in November and December of 2023, only 24% (18/75) of studies were listed as fully funded including initial screening (Figure 1) (41). Even when a trial is fully-funded, the process of obtaining a diagnosis is often lengthy and expensive, and owner out of pocket costs were listed as greater than $500 after enrollment in 31% (23/75) of the evaluated studies. Substantial out-of-pocket costs can limit study enrollment, compliance, and retention.

Figure 1
www.frontiersin.org

Figure 1. Financial incentive categorical choices in United States Dollars selected for veterinary clinical trials (n = 75 studies) posted on the American Veterinary Medical Association Animal Health Studies Database. Data collected between November 17 and December 17, 2023. Not specified, n = 1; unfunded, n = 4; partially funded, owner costs <$500, n = 7; partially funded, owner costs >$500, n = 23; fully funded after initial screening, n = 22; fully funded including initial screening, n = 18.

In recent years the discussion around payment for research participation in human clinical trials has incorporated the idea of justice. Clinical research requires participants, and therefore compensating participants, or in the case of veterinary medicine, their owners, as part of the research team for their time and contributions can be considered both ethical and necessary (16). Payment can remove the burden of participation and improve equity and representativeness in clinical trials. In addition, providing payment demonstrates respect for patient or pet owner time, expertise, risk, and value to the research process, and can remove power imbalances and improve commitment in study conduct. Halpern et al. (37) found that although higher payments may motivate research participation, commonly used payment levels were not considered bribes or unethical.

Researchers and funding agencies must carefully consider what incentive to provide for research to achieve successful patient recruitment numbers and best study outcomes, and protocol review boards must decide if study incentives are ethical. Although funding is often limited in veterinary research, providing compensation for research participation has the potential to improve recruitment and study outcomes and acknowledges the scientific and societal benefit provided by the owner and their pet, particularly considering the potential translational applications of many veterinary clinical trials. In all cases, compensation details, including what is being provided and by whom, must be transparent and consistent, and can thereby engender trust in the research process.

3.5 Personnel resources

A positive (or negative) experience with the primary study communicator can impact patient recruitment and retention. In human clinical trials, nurses and data managers contribute more than 60% of the workload in clinical research compared to 9% from physicians (42). In a human prostate cancer trial comparing the effectiveness of the primary communicator (nurses vs. surgeons), nurses tended to spend longer on each recruitment, were as effective in recruitment as surgeons, and, ultimately, were more cost-effective (43).

Information on the distribution of workload in recruitment for veterinary studies is sparse. However, the primary responsibility for contact and study conduct in veterinary studies often falls on the study clinician; this responsibility may shift to the research coordinator or technician if available. Dedicated clinical trial units in veterinary medicine are often small, if they exist at all. Recent survey data reported that just over half (59.1%) of the 22 responding veterinary academic institutions across the United States and Canada had a centralized veterinary clinical research unit, which employed a median of approximately 4 full time equivalent personnel (44).

When available, clinical trials/research teams may vary by study type, study purpose, and between study sites. Stable, permanently-funded positions allow flexibility to adapt to the changing needs of the clinical trials program. All study personnel should be trained in their respective duties (i.e., consenting, study procedures, adverse event reporting, etc.). Although formal certification in veterinary clinical trials is not currently available, resources are expanding and training should be encouraged where available (Supplementary Table). Use of human clinical trials training resources is an option, although there are limitations in the applicability of some information, such as regulatory requirements. Expanding training for areas such as recruitment, consenting, and adverse event reporting to all hospital staff, students, and trainees can help ensure all individuals are supporting quality clinical research and consistent communication with pet owners. Specialty training in sensitive situations, such as trials involving emergency medicine cases, may also be beneficial for successful recruitment.

With small teams and limited resources, the future of veterinary clinical trials may involve a shift to expanded online recruitment strategies. In a systematic review and meta-analysis of human clinical trials, online recruitment has been reported to offer superior time efficiency and cost-effectiveness compared to offline (in-person) recruitment (7). There is a fine balance, however, as this study also reported that in-person recruitment had a higher conversion rate when compared to online (7).

3.6 Advertising

Another major impediment to successful recruitment is study advertising. When possible, the clinical trials team can create recruitment materials. However, teams may have limited capacity due to personnel resources and a lack of training in effective advertising and marketing strategies. Including individuals with expertise in communication and with the target communities in the clinical trials team can help optimize advertising materials and strategies. If such expertise is unavailable, developing a knowledge bank of available, successful advertising and marketing strategies can be useful. Strategies can be mixed and matched to suit specific project needs and resources (Figure 2).

Figure 2
www.frontiersin.org

Figure 2. Recruitment strategies for veterinary clinical trials. ICU, Intensive Care Unit; PR, Public Relations; DVM, Doctor of Veterinary Medicine.

Even among veterinarians, there is a lack of awareness of the availability and impact of clinical trials in veterinary medicine. A 2017 survey (18) found that 28% of veterinarians did not usually learn about clinical trials in their area. Even recent DVM graduates from academic institutions were often unaware of clinical research studies, and only 55% of recent graduates had investigated clinical trial participation for their patients (18). Veterinarians were more likely to recommend clinical trial participation if a respected investigator was conducting the study, an academic institution sponsored the study, and the study results were disseminated back to the veterinarian and scientific community (18). A 2014 survey of cat owners reported that the majority (75–89%) viewed veterinarian recommendation as a significant factor in study participation (14). When asked which method of recruitment communication is the most effective, referring veterinarians preferred email communication to the practice (70%) followed by clinical trials websites (60%), printed materials sent to the practice (54%), and visits from the study investigator (43%) (18).

Although referring veterinarians play an important role in clinical trials recruitment, successful advertising strategies should also target pet owners. Data from The Ohio State University’s Clinical Trials Office show that the vast majority (96%; 1,226 of 1,278 total forms) of initial screening forms completed for ongoing veterinary clinical trials from 2018–2021 were by pet owners (personal communication). Similarly, efforts in recruitment to human clinical trials have shown that directing outreach toward potential study participants and their caregivers was more successful and cost-effective than targeting primary care health professionals (45).

Ultimately, recruitment materials should empower pet owners to “read, understand, and act” on their pet’s behalf by providing informed consent for study participation (46). Previous studies have explored issues around the readability of study materials (2, 46, 47). Most study-related materials far exceed the average American’s 8th-grade reading level and the recommended 6th-grade reading level for medical information (2, 46, 47). Failure to provide clear materials can cause issues with both recruitment and compliance by compromising the ability of pet owners to make informed medical decisions and potentially eroding public trust in veterinary professionals (46, 48). Simple remedies, such as readability calculators (e.g., Microsoft Word and the free, web-based readability tool Automatic Readability Checker) (46) and artificial intelligence (AI)-based writing tools (e.g., ChatGPT and others), are available to help improve readability. Employing the assistance of a patient/pet owner advocate or a lay reader are alternative strategies to ensure that written materials are understandable and appealing to the target audience. Additionally, using graphics and other visuals (e.g., bullet points, videos, study visit calendars) with easy-to-read study information, including study personnel contact information, can help overcome literacy and numeracy barriers to study recruitment (16, 46). In all cases, and particularly with the use of new AI tools, users must critically review the information to ensure accuracy and that ethical issues such as potential bias, privacy, and authorship are addressed (49).

Efforts have been made to create a centralized, easily-searchable registry of clinical trials to increase awareness of ongoing veterinary clinical trials. The American Veterinary Medicine Association’s (AVMA) Animal Health Studies Database (AAHSD) was launched in 2016 to help connect pet owners and referring veterinarians with reliable and comprehensive study listings (41, 50). The available development and implementation resources had limited the utility of the database, however, the AVMA launched an updated database, the AVMA Veterinary Clinical Trials Registry, in 2024 intended to address previous limitations.

3.7 Retention

Study attrition refers to the failure to retain participants after enrollment, which can occur at any time during the study and for various reasons depending on the population, study duration, condition, intervention, and outcome measures used (51). Attrition rates of 26% for the primary endpoint and 44% for the end of the study have been reported in human supportive and palliative oncology clinical trials (52). Some studies have shown attrition rates up to 67% (53). Loss of this number of patients, particularly from one patient group, can lead to sampling bias and/or reduction in sample size and affect the internal and external validity of studies. Reasons for attrition in the above-mentioned studies included patient characteristics, such as high baseline symptom burden, and study characteristics including placement in a placebo group, longer study duration, and outpatient studies (52, 53). Difficulty in understanding the study requirements and consent form, failure to obtain timely responses from study personnel, communication failures, socioeconomic conditions, and experiencing stress during study visits are other commonly-cited causes of attrition (54, 55). Participant retention is important for all studies, especially those with longitudinal designs, and recording reasons for patient dropout can provide information to improve future study recruitment and conduct (55, 56).

Retention in veterinary clinical trials relies on a solid and consistent veterinary client-patient relationship (VCPR) centered around communication from the first contact (i.e., study recruitment), and throughout the study, up to and following dissemination of study results. Clinical trials are often conducted at a specialty referral hospital; therefore, this referral can be viewed as an extension of a primary veterinarian’s care (57). The VCPR extends between study veterinarians, their clients (and patients), collaborating colleagues, and the veterinary profession. Continuation of care is an important concept in study design and recruitment, both to ensure patient support continues after a study ends, as well as to overcome a potential barrier to referral—the fear of losing patients through clinical trial participation (18, 45).

For clinical trials, streamlined communications throughout the study process facilitate recruitment, repeated visits, and adherence to strict study protocols. In an Australian longitudinal water quality study, families who continued to participate for the 68-week study duration cited being kept well-informed via a monthly newsletter as the strongest determining factor of their long-term compliance (58). Similarly, pet owners are 40% more likely to comply with veterinary recommendations when communication is clear, thorough, and trustworthy (59).

Having a single contact person or designated care team can help create continuity and trusted relationships. In addition, simple gestures can help to maintain this relationship, such as sharing study results once published or larger community outreach events such as study days or celebrations (45). Exposure to branded promotional items may also increase awareness of and affinity for the clinical research enterprise, especially when the “brand” is unfamiliar or for longitudinal studies where retention can present special challenges (60). Chhatre et al. (61) demonstrated success in recruiting and retaining participants in a multi-site human prostate cancer study by using a multi-faceted, patient-centered strategy that included personalized thank-you notes, contact via preferred communication methods, and providing study progress reports to enrolled patients. Such efforts can help create a community of individuals supportive of veterinary clinical research that can facilitate future recruitment and retention efforts across the profession. With the rise of social media, increased owner access to electronic veterinary medical records, and other digital tools, there has been an increase in direct outreach to potential study participants via digital strategies (46, 51). While social media has the potential for unwanted attention on research if there are adverse outcomes, an owner’s high investment in studies of specific diseases, breeds, and/or treatment(s) can further increase awareness and advertisement of the study. A positive clinical trial experience can also influence pet owners’ future study participation and support.

4 Conclusions and future directions

Clinical trials are performed to improve patient health outcomes, and a successful clinical trial requires adequate numbers of participants. However, many factors impede effective study recruitment, and there is a collective need to share and implement current best practices for recruitment, including study design and logistics, encouraging representative participation, use of incentives and personnel resources, and strategies for advertising and participant retention. Although the use of human clinical trial data and resources can provide some guidance, evaluation of current practices and opportunities for process improvement in the conduct of veterinary clinical trials is needed.

Greater awareness of veterinary clinical trials and standardization of study conduct can create opportunities for expanded and effective collaborations. According to a 2021 survey of North American veterinary schools, 80% of veterinary clinical studies were conducted at a single site (44). While multi-center studies require a larger up-front time investment, human multi-center clinical trials have demonstrated the benefits of a larger and more equitable recruitment pool, and increased study efficiency may outweigh the costs. Efforts are underway to address challenges with multi-center trials in veterinary medicine. For example, the development of a veterinary Streamlined, Multisite, Accelerated Resources for Trials (SMART) Institutional Animal Care and Use Committee (IACUC) reliance platform similar to the human SMART Institutional Review Board (IRB) may help facilitate larger collaborative studies (62, 63). Ultimately, collaboration can reduce redundancy and support more robust and consistent study design, efficient ethical review, study advertising and recruitment, good clinical practices, and pooling of resources to run studies that lead to improved clinical decision making for veterinary patients.

Clear communication between all stakeholders throughout the research process is key to successful and meaningful clinical research, and training in science communication can be beneficial to all members of a research team. Increased community engagement in veterinary clinical research, including pet owners, advocates, and primary care veterinarians and staff, is another area of opportunity to foster support for research and improve the research process. Investment in trained, dedicated clinical trial personnel and infrastructure can support successful communication and collaboration and drive improvements in veterinary clinical medicine by providing the time and expertise to effectively complete impactful clinical trials. Continued development of collaborations surrounding best practices and training opportunities in veterinary clinical research will improve the impact of veterinary clinical trials teams, while also promoting workforce development and alternate career paths for veterinary professionals. Including plans for widespread dissemination and implementation of veterinary clinical research results will validate the research process, honor the participants, maximize the positive impact on clinical practice, and support future successful research efforts.

Ultimately, putting recruitment at the forefront of clinical trials can minimize patient and pet owner burden and maximize patient and pet owner appreciation, all while achieving sound scientific and clinically relevant results. Given the diversity of veterinary clinical trials, a one-size-fits-all approach to study recruitment will never exist. However, researchers can thoughtfully draw from an expanding variety of evidence-based, patient-centered approaches to improve their odds of successful study recruitment and completion.

Author contributions

MQ: Writing – original draft, Writing – review & editing. CM: Writing – original draft, Writing – review & editing. TW: Conceptualization, Funding acquisition, Writing – original draft, Writing – review & editing.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. Publication fees were covered by Colorado State University agreement.

Acknowledgments

The authors wish to thank Dr. Ed Murphey for providing information about the AVMA AAHSD. The authors also wish to thank Ashley Smith from The Ohio State University for providing information on study recruitment. Michael J. Stamper at Virginia Tech contributed to the design of Figure 2. Katie Sample and Tess Fowler at Virginia Tech provided data collection for the section on incentive payments.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fvets.2024.1418747/full#supplementary-material

References

1. Kol, A, Arzi, B, Athanasiou, KA, Farmer, DL, Nolta, JA, Rebhun, RB, et al. Companion animals: translational scientist’s new best friends. Sci Transl Med. (2015) 7:308ps21. doi: 10.1126/scitranslmed.aaa9116

Crossref Full Text | Google Scholar

2. Page, R, Baneux, P, Vail, D, Duda, L, Olson, P, Anestidou, L, et al. Conduct, oversight, and ethical considerations of clinical trials in companion animals with cancer: report of a workshop on best practice recommendations. J Vet Intern Med. (2016) 30:527–35. doi: 10.1111/jvim.13916

Crossref Full Text | Google Scholar

3. Sargeant, JM, Ruple, A, Selmic, LE, and O’Connor, AM. The standards of reporting trials in pets (PetSORT): explanation and elaboration. Front Vet Sci. (2023) 10:1137781. doi: 10.3389/fvets.2023.1137781

Crossref Full Text | Google Scholar

4. Aitken, L, Gallagher, R, and Madronio, C. Principles of recruitment and retention in clinical trials. Int J Nurs Pract. (2003) 9:338–46. doi: 10.1046/j.1440-172X.2003.00449.x

Crossref Full Text | Google Scholar

5. Chaudhari, N, Ravi, R, Gogtay, NJ, and Thatte, UM. Recruitment and retention of the participants in clinical trials: challenges and solutions. Perspect Clin Res. (2020) 11:64–9. doi: 10.4103/picr.PICR_206_19

Crossref Full Text | Google Scholar

6. Deloitte University EMEA CVBA. (2020). Deloitte insights, intelligent clinical trials. Available at: https://www2.deloitte.com/content/dam/insights/us/articles/22934_intelligent-clinical-trials/DI_Intelligent-clinical-trials.pdf. (Accessed May 24, 2024)]

Google Scholar

7. Brøgger-Mikkelsen, M, Ali, Z, Zibert, JR, Andersen, AD, and Thomsen, SF. Online patient recruitment in clinical trials: systematic review and meta-analysis. J Med Internet Res. (2020) 22:e22179. doi: 10.2196/22179

Crossref Full Text | Google Scholar

8. Bower, P, Wallace, P, Ward, E, Graffy, J, Miller, J, Delaney, B, et al. Improving recruitment to health research in primary care. Fam Pract. (2009) 26:391–7. doi: 10.1093/fampra/cmp037

Crossref Full Text | Google Scholar

9. Bogin, V . Lasagna’s law: A dish best served early. Contemp Clin Trials Commun. (2022) 26:100900. doi: 10.1016/j.conctc.2022.100900

Crossref Full Text | Google Scholar

10. Russell, WMS, and Burch, RL. The principles of experimental technique. London: Methuen and Co (1959) Available at: http://altweb.jhsph.edu/pubs/books/humane_exp/het-toc.

Google Scholar

11. Kilkenny, C, Parsons, N, Kadyszewski, E, Festing, MFW, Cuthill, IC, Fry, D, et al. Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS One. (2009) 4:e7824. doi: 10.1371/journal.pone.0007824

Crossref Full Text | Google Scholar

12. Giuffrida, MA . Defining the primary research question in veterinary clinical studies. J Am Vet Med Assoc. (2016) 249:547–51. doi: 10.2460/javma.249.5.547

Crossref Full Text | Google Scholar

13. Fritz, MC, and Stanley, BJ. Feline clinical trials: considerations for improving recruitment. J Feline Med Surg. (2014) 16:725–6. doi: 10.1177/1098612X14545272

Crossref Full Text | Google Scholar

14. Gruen, ME, Jiamachello, KN, Thomson, A, and Lascelles, BDX. Clinical trials involving cats: what factors affect owner participation? J Feline Med Surg. (2014) 16:727–35. doi: 10.1177/1098612X14539499

Crossref Full Text | Google Scholar

15. Fogel, DB . Factors associated with clinical trials that fail and opportunities for improving the likelihood of success: A review. Contemp Clin Trials Commun. (2018) 11:156–64. doi: 10.1016/j.conctc.2018.08.001

Crossref Full Text | Google Scholar

16. OASH. Beyond altruism—exploring payment for research participation program book 2. Available at: https://www.hhs.gov/sites/default/files/oew-program-book-2022.pdf

Google Scholar

17. Unger, JM, Vaidya, R, Hershman, DL, Minasian, LM, and Fleury, ME. Systematic review and meta-analysis of the magnitude of structural, clinical, and physician and patient barriers to cancer clinical trial participation. J Natl Cancer Inst. (2019) 111:245–55. doi: 10.1093/jnci/djy221

Crossref Full Text | Google Scholar

18. Gruen, ME, Griffith, EH, Caney, SMA, Rishniw, M, and Lascelles, BDX. Attitudes of small animal practitioners toward participation in veterinary clinical trials. J Am Vet Med Assoc. (2017) 250:86–97. doi: 10.2460/javma.250.1.86

Crossref Full Text | Google Scholar

19. Millum, J, and Grady, C. The ethics of placebo-controlled trials: methodological justifications. Contemp Clin Trials. (2013) 36:510–4. doi: 10.1016/j.cct.2013.09.003

Crossref Full Text | Google Scholar

20. Krol, FJ, Hagin, M, Vieta, E, Harazi, R, Lotan, A, Strous, RD, et al. Placebo—to be or not to be? Are there really alternatives to placebo-controlled trials? Eur Neuropsychopharmacol. (2020) 32:1–11. doi: 10.1016/j.euroneuro.2019.12.117

Crossref Full Text | Google Scholar

21. Miller, FG, and Silverman, HJ. The ethical relevance of the standard of care in the design of clinical trials. Am J Respir Crit Care Med. (2003) 169:562–4. doi: 10.1164/rccm.200311-1577CP

Crossref Full Text | Google Scholar

22. Dear, RF, McGeechan, K, Barnet, MB, Barratt, AL, and Tattersall, MHN. “Standard care” in cancer clinical trials: an analysis of care provided to women in the control arms of breast cancer clinical trials. J Natl Compr Cancer Netw. (2017) 15:1131–9. doi: 10.6004/jnccn.2017.7012

Crossref Full Text | Google Scholar

23. Ayling, K, Brierley, S, Johnson, B, Heller, S, and Eiser, C. How standard is standard care? Exploring control group outcomes in behaviour change interventions for young people with type 1 diabetes. Psychol Health. (2015) 30:85–103. doi: 10.1080/08870446.2014.953528

Crossref Full Text | Google Scholar

24. Yu, AM, Balasubramanaiam, B, Offringa, M, and Kelly, LE. Reporting of interventions and “standard of care” control arms in pediatric clinical trials: a quantitative analysis. Pediatr Res. (2028) 84:393–8. doi: 10.1038/s41390-018-0019-7

Crossref Full Text | Google Scholar

25. American Veterinary Medical Association. (2024) Establishment and use of veterinary clinical studies committees. Available at: https://www.avma.org/resources-tools/avma-policies/establishment-and-use-veterinary-clinical-studies-committees. (Accessed May 18, 2024)

Google Scholar

26. Sargeant, JM, Plishka, M, Ruple, A, Selmic, LE, Totton, SC, and Vriezen, ER. Quality of reporting of clinical trials in dogs and cats: an update. J Vet Intern Med. (2021) 35:1957–71. doi: 10.1111/jvim.16204

Crossref Full Text | Google Scholar

27. Butcher, NJ, Monsour, A, Mew, EJ, Chan, A-W, Moher, D, Mayo-Wilson, E, et al. Guidelines for reporting outcomes in trial reports: The CONSORT-outcomes 2022 extension. JAMA. (2022) 328:2252. doi: 10.1001/jama.2022.21022

Crossref Full Text | Google Scholar

28. Ahmed, SM, and Palermo, A-GS. Community engagement in research: frameworks for education and peer review. Am J Public Health. (2010) 100:1380–7. doi: 10.2105/AJPH.2009.178137

PubMed Abstract | Crossref Full Text | Google Scholar

29. Holzer, JK, Ellis, L, and Merritt, MW. Why we need community engagement in medical research. J Investig Med. (2014) 62:851–5. doi: 10.1097/JIM.0000000000000097

Crossref Full Text | Google Scholar

30. Huang, GD, Bull, J, Johnston McKee, K, Mahon, E, Harper, B, and Roberts, JN. Clinical trials recruitment planning: A proposed framework from the clinical trials transformation initiative. Contemp Clin Trials. (2018) 66:74–9. doi: 10.1016/j.cct.2018.01.003

Crossref Full Text | Google Scholar

31. Sheridan, S, Schrandt, S, Forsythe, L, Hilliard, TS, and Paez, KAAdvisory Panel on Patient Engagement (2013 Inaugural Panel). The PCORI engagement rubric: promising practices for partnering in research. Ann Fam Med. (2017) 15:165–70. doi: 10.1370/afm.2042

PubMed Abstract | Crossref Full Text | Google Scholar

32. Zucker, I, and Prendergast, BJ. Sex differences in pharmacokinetics predict adverse drug reactions in women. Biol Sex Differ. (2020) 11:32. doi: 10.1186/s13293-020-00308-5

Crossref Full Text | Google Scholar

33. Carey, JL, Nader, N, Chai, PR, Carreiro, S, Griswold, MK, and Boyle, KL. Drugs and medical devices: adverse events and the impact on women’s health. Clin Ther. (2017) 39:10–22. doi: 10.1016/j.clinthera.2016.12.009

Crossref Full Text | Google Scholar

34. Bierer, BE, White, SA, Gelinas, L, and Strauss, DH. Fair payment and just benefits to enhance diversity in clinical research. J Clin Transl Sci. (2021) 5:e159. doi: 10.1017/cts.2021.816

Crossref Full Text | Google Scholar

35. Gneezy, U, Meier, S, and Rey-Biel, P. When and why incentives (don’t) work to modify behavior. J Econ Perspect. (2011) 25:191–210. doi: 10.1257/jep.25.4.191

Crossref Full Text | Google Scholar

36. Hoffart, J, and Scheibehenne, B. Pill or bill? Influence of monetary incentives on the perceived riskiness and the ethical approval of clinical trials. Judgm Decis Mak. (2019) 14:130–4. doi: 10.1017/S1930297500003363

Crossref Full Text | Google Scholar

37. Halpern, SD, Karlawish, JHT, Casarett, D, Berlin, JA, and Asch, DA. Empirical assessment of whether moderate payments are undue or unjust inducements for participation in clinical trials. Arch Intern Med. (2004) 164:801. doi: 10.1001/archinte.164.7.801

Crossref Full Text | Google Scholar

38. Hawn, R . (2021). Pet insurance: cases and clarity. Available at: https://www.aaha.org/trends-magazine/publications/pet-insurance/

Google Scholar

39. Metz, J . (2024). Pet insurance statistics 2024. Available at: https://www.forbes.com/advisor/pet-insurance/pet-insurance-statistics/. (Accessed May 18, 2024)

Google Scholar

40. Grey, E . (2024). 15 UK pet insurance statistics & facts to know in 2024. Available at: https://www.dogster.com/lifestyle/pet-insurance-statistics-uk#:~:text=Top%2015%20UK%20Pet%20Insurance%20Statistics&text=3.,of%20the%20total%20written%20premiums. (Accessed May 18, 2024)

Google Scholar

41. AAHSD. The AVMA Animal Health Studies Database (AAHSD). Available at: http://www.avma.org/findvetstudies

Google Scholar

42. Baer, AR, Zon, R, Devine, S, and Lyss, AP. The clinical research team. J Oncol Pract. (2011) 7:188–92. doi: 10.1200/JOP.2011.000276

Crossref Full Text | Google Scholar

43. Donovan, JL, Peters, TJ, Noble, S, Powell, P, Gillatt, D, Oliver, SE, et al. Who can best recruit to randomized trials? J Clin Epidemiol. (2003) 56:605–9. doi: 10.1016/S0895-4356(03)00083-0

Crossref Full Text | Google Scholar

44. Moore, SA, McCleary-Wheeler, A, Coates, JR, Olby, N, and London, C. A CTSA One Health Alliance (COHA) survey of clinical trial infrastructure in North American veterinary institutions. BMC Vet Res. (2021) 17:90. doi: 10.1186/s12917-021-02795-z

Crossref Full Text | Google Scholar

45. Carr, SA, Davis, R, Spencer, D, Smart, M, Hudson, J, Freeman, S, et al. Comparison of recruitment efforts targeted at primary care physicians versus the community at large for participation in Alzheimer disease clinical trials. Alzheimer Dis Assoc Disord. (2010) 24:165–70. doi: 10.1097/WAD.0b013e3181aba927

Crossref Full Text | Google Scholar

46. McKenna, C, Quigley, M, and Webb, TL. Evaluating the readability of recruitment materials in veterinary clinical research. J Vet Intern Med. (2023) 37:2125–30. doi: 10.1111/jvim.16857

Crossref Full Text | Google Scholar

47. Sobolewski, J, Bryan, JN, Duval, D, O’Kell, A, Tate, DJ, Webb, T, et al. Readability of consent forms in veterinary clinical research. J Vet Intern Med. (2019) 33:350–5. doi: 10.1111/jvim.15462

Crossref Full Text | Google Scholar

48. Sheats, MK, Royal, K, and Kedrowicz, A. Using readability software to enhance the health literacy of equine veterinary clients: an analysis of 17 American Association of Equine Practitioners’ newsletter and website articles. Equine Vet J. (2019) 51:552–5. doi: 10.1111/evj.13042

Crossref Full Text | Google Scholar

49. Doyal, AS, Sender, D, Nanda, M, and Serrano, RA. ChatGPT and artificial intelligence in medical writing: concerns and ethical considerations. Cureus. (2023) 15:e43292. doi: 10.7759/cureus.43292

Crossref Full Text | Google Scholar

50. Murphey, D, and The, E. AVMA animal health studies database. Top Companion Anim Med. (2019) 37:100361. doi: 10.1016/j.tcam.2019.100361

Crossref Full Text | Google Scholar

51. Frampton, GK, Shepherd, J, Pickett, K, Griffiths, G, and Wyatt, JC. Digital tools for the recruitment and retention of participants in randomised controlled trials: a systematic map. Trials. (2020) 21:478. doi: 10.1186/s13063-020-04358-3

Crossref Full Text | Google Scholar

52. Hui, D, Glitza, I, Chisholm, G, Yennu, S, and Bruera, E. Attrition rates, reasons, and predictive factors in supportive care and palliative oncology clinical trials. Cancer. (2013) 119:1098–105. doi: 10.1002/cncr.27854

Crossref Full Text | Google Scholar

53. Khan, A, Khan, SR, Leventhal, RM, and Brown, WA. Symptom reduction and suicide risk among patients treated with placebo in antipsychotic clinical trials: an analysis of the food and drug administration database. Am J Psychiatry. (2001) 158:1449–54. doi: 10.1176/appi.ajp.158.9.1449

Crossref Full Text | Google Scholar

54. Gabales, S . (2021). Retention in clinical trials: keeping patients on protocols. Available at: https://www.advarra.com/resource-library/retention-in-clinical-trials-keeping-patients-on-protocols/

Google Scholar

55. Abshire, M, Dinglas, VD, Cajita, MIA, Eakin, MN, Needham, DM, and Himmelfarb, CD. Participant retention practices in longitudinal clinical research studies with high retention rates. BMC Med Res Methodol. (2017) 17:30. doi: 10.1186/s12874-017-0310-z

Crossref Full Text | Google Scholar

56. Hutchison, E, Zhang, Y, Nampally, S, Neelufer, IK, Malkov, V, Weatherall, J, et al. (2021). Modeling clinical trial attrition using machine intelligence: a driver analytics case study using 1,325 trials representing one million patients medRxiv. Available at: https://doi.org/10.1101/2021.11.12.21266277

Google Scholar

57. Mulherin, BL, and Bannon, KM. Communication, veterinary-client-patient relationship, and teledentistry. Vet Clin North Am Small Anim Pract. (2022) 52:25–47. doi: 10.1016/j.cvsm.2021.08.002

Crossref Full Text | Google Scholar

58. Hellard, ME . Methods used to maintain a high level of participant involvement in a clinical trial. J Epidemiol Community Health. (2001) 55:348–51. doi: 10.1136/jech.55.5.348

Crossref Full Text | Google Scholar

59. Lue, TW, Pantenburg, DP, and Crawford, PM. Impact of the owner-pet and client-veterinarian bond on the care that pets receive. J Am Vet Med Assoc. (2008) 232:531–40. doi: 10.2460/javma.232.4.531

Crossref Full Text | Google Scholar

60. Kamleitner, B, and Marckhgott, E. Silent persuasion. Incidental use of promotional merchandise benefits unfamiliar brands. Int J Advert. (2021) 40:1135–59. doi: 10.1080/02650487.2020.1855823

Crossref Full Text | Google Scholar

61. Chhatre, S, Jefferson, A, Cook, R, Meeker, CR, Kim, JH, Hartz, KM, et al. Patient-centered recruitment and retention for a randomized controlled study. Trials. (2018) 19:205. doi: 10.1186/s13063-018-2578-7

Crossref Full Text | Google Scholar

62. SMART IRB. Supporting single IRB review. Advancing collaborative research. Available at: https://smartirb.org

Google Scholar

63. COHA. SMART IACUC Network. Available at: https://ctsaonehealthalliance.org/2023/09/27/smart-iacuc-network/. (Accessed May 26, 2024)

Google Scholar

Keywords: study recruitment, clinical trial, veterinary, incentive, retention, advertising, study design

Citation: Quigley M, McKenna C and Webb TL (2024) Best practices for recruitment in veterinary clinical trials. Front. Vet. Sci. 11:1418747. doi: 10.3389/fvets.2024.1418747

Received: 17 April 2024; Accepted: 11 June 2024;
Published: 17 July 2024.

Edited by:

Claire Rebecca Sharp, Murdoch University, Australia

Reviewed by:

Lisa Smart, Small Animal Specialist Hospital (SASH), Australia
Maryanne Murphy, The University of Tennessee, Knoxville, United States

Copyright © 2024 Quigley, McKenna and Webb. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Tracy L. Webb, tracy.webb@colostate.edu

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.