Effects of Probiotic Supplementation on the Gut Microbiota and Antibiotic Resistome Development in Preterm Infants

Objectives: In 2014 probiotic supplementation (Lactobacillus acidophilus and Bifidobacterium longum subspecies infantis; Infloran®) was introduced as standard of care to prevent necrotizing enterocolitis (NEC) in extremely preterm infants in Norway. We aimed to evaluate the influence of probiotics and antibiotic therapy on the developing gut microbiota and antibiotic resistome in extremely preterm infants, and to compare with very preterm infants and term infants not given probiotics.

Study design: A prospective, observational multicenter study in six tertiary-care neonatal units. We enrolled 76 infants; 31 probiotic-supplemented extremely preterm infants <28 weeks gestation, 35 very preterm infants 28–31 weeks gestation not given probiotics and 10 healthy full-term control infants. Taxonomic composition and collection of antibiotic resistance genes (resistome) in fecal samples, collected at 7 and 28 days and 4 months age, were analyzed using shotgun-metagenome sequencing.

Results: Median (IQR) birth weight was 835 (680–945) g and 1,290 (1,150–1,445) g in preterm infants exposed and not exposed to probiotics, respectively. Two extremely preterm infants receiving probiotic developed NEC requiring surgery. At 7 days of age we found higher median relative abundance of Bifidobacterium in probiotic supplemented infants (64.7%) compared to non-supplemented preterm infants (0.0%) and term control infants (43.9%). Lactobacillus was only detected in small amounts in all groups, but the relative abundance increased up to 4 months. Extremely preterm infants receiving probiotics had also much higher antibiotic exposure, still overall microbial diversity and resistome was not different than in more mature infants at 4 weeks and 4 months.

Conclusion: Probiotic supplementation may induce colonization resistance and alleviate harmful effects of antibiotics on the gut microbiota and antibiotic resistome.

Clinical Trial Registration: Clinicaltrials.gov: NCT02197468. https://clinicaltrials.gov/ct2/show/NCT02197468

Introduction

Preterm infants experience unique challenges in establishing their gut microbiota. Cesarean deliveries, extensive antenatal, and neonatal antibiotic exposure, parenteral nutrition and residing for long periods in a neonatal intensive care unit (NICU), may cause unpredictable perturbations of the gut microbiota development (1). Gut microbiota dysbiosis in the first weeks of life is associated with perturbations of the developing immune system (2), and an increased risk of necrotizing enterocolitis (NEC) (3). Probiotic supplementation aims to restore the gut microbiota, and thereby preventing NEC and other complications (4–6). Meta-analyses of randomized and observational trials show that probiotic supplementation, mainly with bifidobacteria and/or lactobacilli, reduce rates of NEC (4, 5, 7, 8). The effects seem to be strain-specific (5) and not all products are efficacious (9). Still, based on recent evidence (4, 10) and expert opinion (11), many NICUs in Europe, Australia, and Canada have implemented routine probiotic-supplementation to preterm infants. Probiotics are infrequently used in preterm infants in the USA (12). Risks of probiotic sepsis and contaminations of probiotic products may explain skepticism (13–16). Some experts recommend waiting for additional studies to confirm the safety and efficacy of an available and reliable product (17). Moreover, there is a paucity of in-depth knowledge on microbiological effects and effective dose of probiotic therapy.

Antibiotics are the most commonly prescribed medications in the NICU (18), and prolonged therapy increases the risk of NEC (19, 20). Antibiotics may influence both the physiological gut microbiota composition and the collection of antibiotic resistance genes (ARGs) in the gut, defined as the gut antibiotic resistome (21, 22). However, there is limited knowledge on how probiotic supplementation and antibiotic therapy influence the gut antibiotic resistome in extremely preterm infants.

In Norway probiotic supplementation was implemented as standard of care for extremely preterm infants in 2014. In a longitudinal multi-center study, using shotgun-metagenomic sequencing, we set out to evaluate the influence of probiotics and antibiotic therapy on the developing gut microbiota and antibiotic resistome in extremely preterm infants supplemented with probiotics. We also compared these results to very preterm infants not supplemented with probiotics and a group of healthy, full-term infants.

Materials and Methods

Study Patients and Sampling Procedure

We prospectively planned to include two convenient groups of preterm infants from six Norwegian NICUs; one group of extremely preterm infants (gestational age 25–27 weeks) supplemented with probiotics, and one group of very preterm infants (gestational age 28–31 weeks) not supplemented with probiotics. Exclusion criteria were gestation below 25 weeks and/or an early, life threatening condition leading to high risk of not surviving the first weeks of life. We included a control group of 10 healthy, vaginally delivered full-term control (FTC) infants born at the University Hospital of Northern Norway. Sample size calculation for studies assessing gut microbiota taxonomic composition can be performed by assessing matrices of pairwise distances between groups (23). We expected that around 30 infants in each group of preterm infants would afford 90% statistical power to detect differences in gut microbiota composition that were smaller than effects previously observed in microbiota studies of antibiotic exposure (23). The sample size was also adapted to cover the high expenses for shotgun-metagenome sequencing. The original protocol (24) focused on taxonomic composition. We decided post hoc to add a resistome analysis.

After careful instructions, fecal samples were collected by a nurse in the NICU at around seven and 28 days of age, and by the parents at home at around 4 months of age. We used a commercially available sampling kit (OMNIgen GUT kit, DNA Genotek, Ottawa, Canada) allowing storage of samples at ambient temperatures for up to 14 days before DNA extraction (25). We obtained routine clinical data including details on antibiotic exposure. NEC was defined as Bell's stage 2–3 (26).

DNA Extraction, Library Preparation, and Sequencing

Total metagenomic DNA was extracted using the NorDiag Arrow Stool DNA Extraction kit (NorDiag, Oslo, Norway). An extra beadbeating step was added to facilitate cell lysis as studies have shown that this can increase extraction of DNA from Gram-positive bacteria. DNA was quantified using the Nanodrop 1000 and Qubit® 2.0 Fluorometer (Invitrogen, Carlsbad, CA, USA) along with the Qubit® dsDNA HR assay kit (Thermo Fisher Scientific, Waltham, MA, USA). DNA was then stored at −70°C. The indexed paired-end libraries were prepared for whole genome sequencing using the Nextera XT Kit (Illumina, San Diego, CA, USA), according to the manufacturer's instructions. Fifty nanograms of genomic DNA was tagmented at 55°C for 10 min. The tagmented DNA was amplified with two primers from Nextera DNA sample preparation Index Kit. PCR products were cleaned using Agencourt AMPure XP beads (Beckman Coulter, Indiana, USA). Purified PCR products were quantified using the Qubit® 2.0 (Invitrogen, Carlsbad, CA, USA), along with the Qubit® dsDNA HS assay kit (Thermo Fisher Scientific, Waltham, MA, USA). The fragment size distribution (500–1,000 bp) was analyzed using the Agilent 2100 Bioanalyzer System (Agilent Technologies, Waldbronn, Germany). The samples were pooled at concentration of 4 nM per sample. Eight to twelve samples were pooled per each sequencing run. Pooled samples was denatured with 0.2 N NaOH, then diluted to 10 pM with hybridization buffer. Subsequently, samples were submitted for v3 reagents with 2 × 300 cycles paired-end sequencing using the Illumina Miseq platform, according to the manufacturer's instructions. In total, 184 samples were sequenced to an average (range) sequence depth of 4.8 (1.8–12.6) million reads per sample for microbiota and functional analysis. Prior to all downstream data analysis, the sequence quality was calculated using FastQC (v0.11.3). All samples were screened for human contamination using Deconseq with default parameters and build up 38 of the human genome as reference. Quality filtering of the read was performed using Trimmomatic v0.36 with LEADING:3, TRAILING:3, MINLEN:75 as parameter settings. Assemblies were performed on the trimmed reads using MEGAHIT. Functional annotation was added using an in-house genome annotation pipeline, the META-pipe (Department of Chemistry, University of Tromsø, Norway [https://arxiv.org/abs/1604.04103]). The sequences are deposited in the European Nucleotide Archive (www.ebi.ac.uk/ena); study accession nr. PRJEB29052.

Taxonomic Profiling

The relative abundance of bacteria at genus level was calculated from the trimmed reads using MetaPhlAn 2.0 (27). Relative abundance tables for each individual sample were merged. To calculate longitudinal changes, sequences were reconstructed using the Lowest Common Ancestor (LCA) classifier.

The Gut Antibiotic Resistome

The prediction of genes presumed to confer antibiotic resistance was performed on the assembled metagenomes using Abricate [https://github.com/tseemann/abricate] against the resistance gene identifier in the Comprehensive Antibiotic Resistance Database (CARD; version 1.1.1; Dept. of Biochemistry and Biomedical Science, McMaster University, Canada, https://card.mcmaster.ca/home]) (25–28) with the minimum identity threshold set to 75% (28). Because of the fragmented nature of the metagenome assemblies, and therefore presence of fragmented genes, multiple hits against the same antibiotic resistance gene (ARG) were regarded as one hit. Data are presented as distribution of ARG classes among the three different groups of infants at three time points. Classes of antibiotic resistance genes in the CARD database and the specific genes included in each class are listed below

• Beta lactamase: blaMIR, blaZ, blaACT, blaTEM, blaCMY, blaLEN, blaADC, blaACI, blaOXA, blaOXY, blaSHV, blaDHA, blaOKP, blaACC, blaSED, blaMOR, blaCMG, blaCFE, cfiA, cepA, cfxA

• Methicillin resistance: mecA

• Aminoglycosides: aac(6′)-aph(2), aac(6′)-Ic, aac(6′)-Im, aadA, aadB, aadD, aadE, ant(6)-Ia, aph(2)-Ib, aph(3)-Ia, aph(3)-III, spc, str, strA,strB

• Tetracyclines: tet(A), tet(B), tet(M), tet(K), tet(X), tet(O), tet(L), tet(U), tet(Q), tet(W), tet(S), tet(32), tet(34), tet(35), tet(37), tet(40), tet(41), Otr(A)

• Fluoroquinolones: QnrB, QnrD

• MLS: Macrolide: erm(A), erm(B), erm(C), erm(F),erm(G), erm(T), erm(X), mph(A), mph(C); Lincosamide: lnu(B), lnu(C); Streptogranin: vat(B), vat(F)

• ABC efflux: lsa(A),lsa(B), lsa(C), msr(A), mrs(C), msr(D), ole(B), car(A)

• RND efflux pumps: oqxA

• Efflux pumps: vga(A), mef(A)

• Multidrug efflux pumps: norA

• Chloramphenicol: cat, catA, catB, catS, cmlA, cml

• Fosfomycin: fos(A)

• Sulfonamides: sul1, sul2

• Antibiotic target: dfrA, dfrG

• Vancomycin: VanC, VanS, VanT, VanR, VanY

• Metronidazole: nimB

In order to obtain quantitative measures of the putative ARGs in each sample, the quality trimmed reads were analyzed using Short, Better Representative Extract Dataset (ShortBRED) (29) against a formatted CARD database and normalized per total reads in each sample. Data are presented as abundance of ARGs among the three different groups of infants at three time points. Using ShortBRED we identified the antibiotic resistance gene classes and genes listed below:

• Class A Beta lactamase

• Class C Beta lactamase

• Aminoglycoside acetyltransferase

• Aminoglycoside phosphotransferase

• Aminoglycoside nucleotidyltransferase

• Tetracycline efflux

• Tetracycline ribosomal protection

• Quinolone resistance

• Macrolide/MLS resistance

• Adenosine triphosphate (ATP)-binding cassette (ABC) efflux pump

• Resistance/nodulation/division (RND) antibiotic efflux

• Major facilitator superfamily (MFS) antibiotic efflux

• Multidrug efflux pump activity

• Multidrug resistance efflux pump

• Genes modulating antibiotic efflux: norA, baeR, marA, phoQ, ramA, soxR

• Small multidrug resistance (SMR) antibiotic efflux

• Chloramphenicol acetyltransferase

• Antibiotic target

• Genes modulating resistance: WblE, WhiB

• rRNA methyltransferase

• Other ARG: bacA

Probiotic Supplementation

A consensus-based protocol for probiotic supplementation was implemented in Norway in 2014 (30). Extremely preterm infants, contributing to around 90% of NEC cases in Norway, were considered as the target group for probiotic prophylaxis. At this time, probiotics was not used routinely for more mature preterm infants (≥28 weeks gestation) in any Norwegian neonatal unit. After considering the safety profile, a widely used probiotic combination product was selected (Infloran®) (31). One capsule Infloran contained 10⁹ colony forming units (CFU) Lactobacillus acidophilus (ATCC 4356) and 10⁹ CFU B. longum subspecies infantis (ATCC 15697). One-half capsule once daily was initiated on day 3–4 and increased to one capsule daily after 4–7 days. One capsule was opened and the content was diluted in 2 ml of breast milk, or formula. It was thereafter administered enteral via a nasogastric tube, either 1 ml (1/2 capsule) or 2 ml (one capsule).

Influence of Antibiotic Therapy

To quantify changes in the gut microbiota composition and resistome after antibiotic exposure, we stratified four different categories of antibiotic exposure: (i) antenatal exposure, (ii) short (≤72 h) vs. prolonged (>72 h) exposure in the first week of life (19, 22), (iii) any exposure after first week of life (yes/no), and (iv) narrow- vs. broad-spectrum exposure after first week of life. Potential effects of antenatal exposure and short vs. prolonged therapy after birth were only investigated at 7 days of age. We defined regimens including third-generation cephalosporins or carbapenems as a broad-spectrum regimens when compared to regimens containing aminoglycosides for coverage against Gram-negative bacteria. This definition was based on the fact that neonatal empiric treatment using a third-generation cephalosporin for Gram-negative coverage induce significantly higher antibiotic resistance rates among colonizing bacteria than a regimen containing an aminoglycoside (32).

Ethics, Trial Registration, and Statistical Analysis

The study was approved by the Norwegian Regional Ethical Committee (2014/930/REK nord) and registered in Clinicaltrials.gov (https://clinicaltrials.gov/ct2/show/NCT02197468). Informed written consent was obtained from all parents.

Data were analyzed using IBM-SPSS version 22 (IBM, Armonk NY, USA) statistical software, the R statistical framework (version 3.2.4; http://www.r-project.org/), and Statistical Analysis of Metagenomic Profiles (STAMP) software package (33). We used Mann–Whitney U-test or a Kruskal–Wallis test for comparisons between two or multiple independent groups. We used a Poisson generalized linear model to calculate trends in the relative abundance of genera and ARGs in the gut microbiota. Corrections based on multiple comparisons were performed by the Benjamini–Hochberg false discovery rate (FDR) (34). A FDR Q ≤ 0.10 was considered significant for any analyses with multiple comparisons. A standard P ≤ 0.05 was considered significant for all other analyses.

Alpha diversity was assessed by calculating the Shannon Diversity index (MEGAN, v5.10.6) (35). To detect changes in alpha diversity over time, we first performed a normality test and found that the residuals were normally distributed. Therefore, differences in alpha diversity over time between the three different groups were calculated using linear mixed models. The same model was used to calculate the influence of antibiotic exposure on alpha diversity. Multiple beta diversity metrics of samples was performed using non-metrical multidimensional scaling (NMDS) based on a matrix of Bray-Curtis distances calculated using the vegan R package. Differences between groups were tested using permutational multivariate analysis on beta diversity matrices.

Results

Study Population and Antibiotic Exposure

Figure 1 shows study flow. We enrolled 66 preterm infants and 10 healthy full-term control (FTC) infants between February and October 2015. The six study sites had different admission numbers, and recruited each between 7 and 24 preterm infants (Figure 1). Clinical characteristics, antibiotic and probiotic exposure, duration of parenteral nutrition and enteral nutrition data are reported in Table 1. The “probiotic extremely preterm (PEP)” infants received much more antibiotics than the “non-probiotic very preterm (NPVP)” infants after first week of life. Two infants in the PEP-group were operated for NEC, both survived.

FIGURE 1

Figure 1. CONSORT study flow diagram. PEP, probiotic extremely preterm; NPVP, non-probiotic very preterm; FTC, full term control; NICU, Neonatal Intensive care Unit.

TABLE 1

Table 1. Clinical background data.

Taxonomic Composition

On day 7, we found higher relative abundance of Bifidobacterium and Lactobacillus in PEP-infants compared to NPVP-infants (Figure 2A, Table 2). FTC infants had higher abundance of some genera (Streptococcus, Veilonella, and Haemophilus) that were only sparsely present in the two preterm infant groups (Figure 2A). Mode of delivery did not lead to detectable differences in the microbiota composition within the preterm groups on day 7 (data not shown).

FIGURE 2

Figure 2. (A–C) Median relative abundance of dominant taxa at genus level. Box plot diagram where the inside bar represent median, the outer horizontal line of the box represents the 25th and the 75th percentile. (A) Median relative abundance at 7 days. (B) Median relative abundance at 28 days. (C) Median relative abundance at 4 months.

TABLE 2

Table 2. Median relative abundance (%) of dominant genera in infant gut microbiota at 7, 28 days, and 4 months of age.

On day 28, there was a striking increase in relative abundance of Escherichia in the PEP-infants and a similar striking increase in relative abundance of Bifidobacterium in NPVP-infants. FTC infants had significantly higher relative abundance of Lactobacillus than NPVP-infants. Overall, at 28 days of age the FTC- and NPVP-infants had higher abundance of Veilonella and Streptococcus than PEP-infants, while both preterm groups had higher relative abundance of Staphylococcus and Enterococcus than FTC-infants (Figure 2B).

By 4 months of age, there were no significant differences in taxonomic profile between PEP- and FTC-infants. The NPVP-infants had more Prevotella than PEP-infants, but otherwise all three groups were similar (Figure 2C). Duration of parenteral nutrition did not lead to detectable differences in the microbial composition between the preterm group(s) on 28 days and at 4 months of age (data not shown). We found no differences in abundance of bifidobacteria and or lactobacilli between hospitals at any time point.

Influence of Antibiotic Exposure on Taxonomic Composition

We found no significant influence of antenatal antibiotic exposure on the gut microbiota composition on day 7. However, 57/66 (86%) preterm infants also received antibiotic therapy (ampicillin or penicillin + gentamicin) during the first week of life (Table 1), limiting the possibility to detect isolated effects of antenatal exposure. There was no difference in the gut microbiota between those exposed to a short (≤72 h) compared to a prolonged (>72 h) course during first week of life. Broad-spectrum antibiotic therapy after the first week of life was mainly given to PEP-infants. Only one child in the NPVP-group received third generation cephalosporins after first week of life. At 4 months of age there was reduced relative abundance of Lactobacillus and Veilonella in those exposed to broad-spectrum antibiotics compared to infants exposed to narrow-spectrum therapy (Tables 3, 4). Moreover, there was a non-significant trend toward reduced relative abundance of Bifidobacterium and increased relative abundance of Escherichia among all preterm infants exposed to broad-spectrum antibiotics at both 28 days and 4 months of age (Tables 3, 4).

TABLE 3

Table 3. Influence of antibiotic exposure (broad* vs. narrow) on taxonomic composition in all preterm infants (both PEP- and NPVP-infants) with fecal samples and who received antibiotics after first week of life.

TABLE 4

Table 4. Influence of antibiotic exposure (broad* vs. narrow) on taxonomic composition in only the PEP-infants with fecal samples and who received antibiotics after first week of life.

Diversity of the Gut Microbiota and Influence of Antibiotic Exposure

We found large intra-individual differences in the gut microbiota composition, in particular at 7 and 28 days of age (Figures 2A–C). The alpha diversity increased significantly with age in both preterm infant groups, but not in FTC-infants (Figure 3A). FTC-infants had significant higher diversity compared to PEP infants at 7 days of age. On day 28 and at 4 months of age, there were no significant differences in alpha diversity between any groups. Significant overall community (beta diversity) differences using Bray-Curtis dissimilarity were detected comparing the three groups on infants (PEP, NPVP, and FTC) at 7 days of age (P = 0.001) and 28 days of age (P = 0.003) (Figures 3B–D). However, we found no difference in alpha or beta diversity between different categories of antibiotic exposure at the three sampling time points.

FIGURE 3

Figure 3. (A–D) Alpha diversity calculated by Shannon diversity index and beta diversity between PEP, NPVP, and FTC infants calculated by non-metrical multidimensional scaling (NMDS). Box plot diagram where the inside bar represent median, the outer horizontal line of the box represents the 25th and the 75th percentile. Error bars represent the standard error. Differences between groups at a given time point and at different time points were tested with linear mixed model. (A) Shannon diversity index of three groups of infants at three sampling points. (B) Beta diversity (NMDS) at 7 days. (C) Beta diversity (NMDS) at 28 days. (D) Beta diversity (NMDS) at 4 months.

Antibiotic Resistome–Distribution of ARG Classes and Abundance of ARGs

In all three groups, we identified putative ARGs conferring resistance to nine different classes of antibiotics, including beta lactams, aminoglycosides, tetracyclines, fosfomycine, sulphonamides, vancomycin, and the macrolide-lincosamide-streptogramin B group. Genes conferring resistance to fluoroquinolones and chloramphenicol were only detected in PEP- and NPVP-infants. Several genes encoding efflux pumps were also identified at all three sampling time points. In total 99 unique ARGs were identified, of which 28 (28%) were located on mobile genetic elements, and these latter were found in more than 80% of all infants (Table 5).

TABLE 5

Table 5. Distribution of classes of antibiotic resistance genes among infants in each group.

We found 21 different genes encoding beta-lactamases, including broad-spectrum and extended-spectrum beta lactamases (ESBLs). ESBL-genes were represented at all three time points in NPVP- and FTC-infants, but not detected in PEP-infants. The methicillin resistance gene (mecA) was identified at 7 and 28 days of age in 11/35 NPVP-infants and 13/31 PEP-infants, but not at 4 months of age. Only one PEP-infant and four NPVP-infants were persistent fecal carriers of mecA at days 7 and 28. Vancomycin ARGs were identified at 4 months of age in 16 infants, but only four of these had received vancomycin. Many of the ARGs identified, encoded resistance to other antibiotics than those used in the NICUs.

On day 7 NPVP-infants had higher abundance of ARGs from four different ARG classes and PEP-infants higher abundance of ARGs from two other ARG classes (Table 6). Only 24% of ARG-classes changed significantly their abundance during the three sampling points (P < 0.05) (Table 6).

TABLE 6

Table 6. Median abundance of antibiotic resistance genes among infants in each group.

On day 7 and at 4 months of age, different antibiotic exposure did not result in significant difference in total abundance of ARGs. However, on day 28, we detected significantly higher abundances of four classes of ARGs, including genes encoding beta-lactam and aminoglycoside resistance, in preterm infants exposed to broad-spectrum antibiotics compared to infants treated with narrow-spectrum regimens (Table 7). For the subset of preterm infants given probiotics there were no significant differences in abundance of ARGs at 4 weeks and 4 months (Table 8).

TABLE 7

Table 7. Influence of antibiotic exposure (broad vs. narrow spectrum regimen after first week of life) on abundance of antibiotic resistance genes (ARGs) in all preterm infants.

TABLE 8

Table 8. Influence of antibiotic exposure (broad vs. narrow after first week of life) on abundance of antibiotic resistance genes (ARGs) in probiotic supplemented extremely preterm (PEP) infants.

Discussion

The main aim of this explorative, observational multi-center study was to obtain in-depth knowledge on how probiotics and antibiotic therapy influenced the developing gut microbiota and antibiotic resistome of preterm infants. Previous studies have shown that the gut microbiota in preterm infants differs from term infants with limited diversity and delayed acquisition of a stable profile (36–38). However, most studies have assessed the gut microbiota composition collapsed at higher taxonomic rank levels (above species-genera level) by sequencing of the 16S ribosomal RNA gene (31, 39). There is limited data (21) on the association between use of probiotics, antibiotics and gut resistome development using shotgun-metagenomic sequencing.

Bifidobacteria strongly dominated the gut microbiota in extremely preterm infants only few days after commencing probiotic supplementation, in sharp contrast to very preterm infants not receiving probiotics who predominantly had Escherichia. High levels of probiotic bacteria are not necessarily indicative of colonization, but may represent the passage of DNA from the administered probiotic species through the host (40). Still, early dominance of bifidobacteria may theoretically enhance the risk of translocation to the blood stream, in particular during first weeks of life in extremely preterm infants when enteral nutrition with “fuel for bifidobacteria” is not yet fully established (13, 14). However, bifidobacterial infections are usually mild (14, 41), in contrast to sepsis caused by Gram-negative bacteria (Proteobacteria), which in preterm infants are the first colonizers of the intestinal tract. Previous studies have shown that the gut microbiota of preterm infants shortly after birth have a high proportion of Proteobacteria and that a bloom of Bifidobacterium occurs first around 33 weeks of age, in line with our findings in NPVP-infants at 7 and 28 days of age (42, 43).

Lactobacillus was only detected in small amounts in all groups, but relative abundance increased up to 4 months of age in all three groups. High levels of Bifidobacterium and barely detectable levels of Lactobacillus have been reported earlier in infants supplemented with equal doses of a probiotic combination of bifidobacteria and lactobacilli (31). A possible explanation for this observation is the spatial organization of intestinal bacteria, where lactobacilli are found in intestinal crypts, thus less accessible when collecting luminal contents (44). Indeed, a recent study in adults showed marked differences between the small intestine microbiota compared to the colonic microbiota (45), indicating the scientific limitations with fecal samples when aiming to understand the entire human intestinal ecosystem.

There is no consensus on the optimal dose of probiotics. One study from India compared standard and high-dose probiotic regimens and found no difference in proportion of infants colonized or quantitative colonization rates with probiotic species (46). Most large randomized trial have used daily doses of 1 × 10⁸-10⁹ CFU (40, 47, 48). Some authors suggest that at least 1 × 10⁹ CFU is required to achieve a beneficial effect, in line with the doses used in our study (49). We observed an early and high relative abundance of Bifidobacterium in PEP-infants. However, we did not use traditional microbiological methods to assess the overall bacterial abundance in the gut. Some authors have suggested that a gradual increase in probiotic supplementation concomitantly with increased enteral nutrition may replicate the physiological gut microbiota development, and secure gut growth, digestive maturation and an appropriate response to bacterial colonization (50, 51). Our study does not allow us to draw any conclusions on dosing. A recent study reported that a daily dose of the same probiotic used in our study (Infloran®) leads to significantly higher levels of Bifidobacterium when compared to dosing bi-weekly or weekly (52).

A lower relative abundance of Bifidobacterium, Lactobacillus, and Veilonella, and a higher relative abundance of Escherichia, were observed at day 28 and 4 months of age among infants treated with broad-spectrum compared to narrow-spectrum antibiotic regimens. Reduced abundance of protective anaerobe commensals and higher abundance of Enterobacteriaceae after antibiotic exposure has also previously been reported (53, 54). When comparing presence and absence of antibiotic exposure after the first week of life, no differences in diversity or taxonomic composition were found. Previous studies on alpha diversity and influence of antibiotic treatment have shown inconsistent results (55). However, infants who were most heavily exposed to antibiotic treatment in our study were also supplemented with probiotics. In animals, probiotics may alleviate the potential loss of microbial diversity created by antibiotic treatment (56). This may explain why PEP-infants, exposed to massive antibiotic pressure, did not have reduced microbial gut diversity compared to other groups. Thus, probiotic supplementation may offer a protective effect partly compensating harmful effects of antibiotics in preterm infants. However, the early low number of taxa in preterm infant stools places constraints on interpreting diversity changes as diversity in a non-complex population may reflect changes in only one taxon.

In line with others, we found that the gut antibiotic resistome of preterm and term infants is established early, independent of antibiotic exposure (21, 57–59). We detected significant higher abundance of ARGs in infants receiving broad-spectrum antibiotics compared to narrow-spectrum regimens. Gibson and co-workers also showed that broad-spectrum antibiotic therapy in preterm infants, was associated with enrichment of specific ARGs (21). We aimed to investigate how probiotic supplementation can influence the gut antibiotic resistome. Overall, there were no differences in distribution of ARG-classes or abundance of ARGs at 28 days and 4 months of age between PEP-infants, exposed to massive antibiotic therapy, and the two other groups with limited or no antibiotic exposure. One possible mechanism for this finding is that probiotic bacteria can produce bacteriocins that improve mucosal integrity and thereby reduces the pathogenic bacterial population and antibiotic resistance (60).

Strengths and Limitations

At the time of this study, probiotic supplementation to extremely preterm infants was considered “standard of care” in Norway. We were therefore beyond equipoise to perform a randomized study comparing probiotic to no probiotic supplementation in this population. The NPVP-infant group has limitations as a control group due to maturational differences and the difference in antibiotic exposure compared to the PEP-infants. However, more antibiotic exposure in the PEP-infants would most likely have led to less diversity and higher abundance of ARGs. Still, we found few differences between the two preterm groups at 28 days and 4 months of age, suggesting a protective effect of probiotics in the PEP-infant group. The gut microbiota composition of preterm infants may differ between hospitals (61), but our multi-center approach intended to average local differences and strengthen generalizability. Infants harbor a much lower gut microbial diversity compared to adults. Any variation in the gut microbiota composition caused by storage may thus theoretically have a proportionally greater effect on the composition (25). We chose a standardized sampling technique in order to avoid potential biases due to freezing of samples at different time points and temperature variation during transport to the laboratory. However, in the most immature infants the DNA content in the early fecal samples was very low, and we were only able to obtain sequence data from 20/31 samples at 1 week of age.

Conclusion

Probiotic-supplemented extremely preterm (PEP) infants had a high relative abundance of Bifidobacterium at 1 week of age, only few days after start of probiotic supplementation. PEP-infants were also exposed to much more antibiotics, but overall microbial diversity and resistome was not different than in more mature infants at 4 weeks and 4 months. We speculate that probiotic supplementation may induce colonization resistance and thereby partly alleviate harmful effects of antibiotics on gut microbiota composition and the antibiotic resistome development.

Data Availability

The raw data supporting the conclusion of this manuscript will be made available by the authors, without undue reservation, to any qualified researcher.

Author Contributions

EE organized all phases of the study, analyzed data, wrote the first version of the manuscript, and revised the manuscript. She had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. TP, JA, SR, RS, and BN were responsible for inclusion of patients at participating centers, data retrieval, and revised the manuscript. JC, EH, and NW took part in study design, were responsible for microbiological (JC) and bioinformatic (EH, NW) analyses and revised the manuscript. CK conceptualized and designed the study, directed all phases of the study, and revised the final manuscript. He had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

Funding

Northern Norway Regional Health Authority, The Dept. of Clinical medicine, UiT The Arctic University of Norway and Odd Berg Group, Research fund.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We thank the parents of the infants participating in this study and the nurses at the participating centers for helping collect fecal samples.

Abbreviations

ARG, Antibiotic resistance genes; CARD, Comprehensive antibiotic resistance database; CFU, Colony forming units; FDR, False discovery rate; FTC, Full-term control; NEC, Necrotizing enterocolitis; NICU, Neonatal intensive care unit; NMDS, Non-metrical multidimensional scaling; NVPVP, Non-probiotic very preterm; PEP, Probiotic extremely preterm.

References

1. Groer MW, Luciano AA, Dishaw LJ, Ashmeade TL, Miller E, Gilbert JA. Development of the preterm infant gut microbiome: a research priority. Microbiome (2014) 2:38. doi: 10.1186/2049-2618-2-38

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Olin A, Henckel E, Chen Y, Lakshmikanth T, Pou C, Mikes J, et al. Stereotypic immune system development in newborn children. Cell (2018) 5:1277–92.e14. doi: 10.1016/j.cell.2018.06.045

CrossRef Full Text | Google Scholar

3. Pammi M, Cope J, Tarr PI, Warner BB, Morrow AL, Mai V, et al. Intestinal dysbiosis in preterm infants preceding necrotizing enterocolitis: a systematic review and meta-analysis. Microbiome (2017) 1:31. doi: 10.1186/s40168-017-0248-8

CrossRef Full Text

4. AlFaleh K, Anabrees J. Probiotics for prevention of necrotizing enterocolitis in preterm infants. Cochrane Database Syst Rev. (2014) 10:CD005496. doi: 10.1002/14651858.CD005496.pub4

CrossRef Full Text | Google Scholar

5. Aceti A, Gori D, Barone G, Callegari ML, Di Mauro A, Fantini MP, et al. Probiotics for prevention of necrotizing enterocolitis in preterm infants: systematic review and meta-analysis. Ital J Pediatr. (2015) 41:89. doi: 10.1186/s13052-015-0199-2

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Sawh SC, Deshpande S, Jansen S, Reynaert CJ, Jones PM. Prevention of necrotizing enterocolitis with probiotics: a systematic review and meta-analysis. PeerJ. (2016) 4:e2429. doi: 10.7717/peerj.2429

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Lau CS, Chamberlain RS. Probiotic administration can prevent necrotizing enterocolitis in preterm infants: a meta-analysis. J Pediatr Surg. (2015) 8:1405–12. doi: 10.1016/j.jpedsurg.2015.05.008

CrossRef Full Text | Google Scholar

8. Olsen R, Greisen G, Schroder M, Brok J. Prophylactic probiotics for preterm infants: a systematic review and meta-analysis of observational studies. Neonatology (2016) 2:105–12. doi: 10.1159/000441274

CrossRef Full Text | Google Scholar

9. Costeloe K, Bowler U, Brocklehurst P, Hardy P, Heal P, Juszczak E, et al. A randomised controlled trial of the probiotic Bifidobacterium breve BBG-001 in preterm babies to prevent sepsis, necrotising enterocolitis and death: the Probiotics in Preterm infantS (PiPS) trial. Health Technol Assess. (2016) 66:1–194. doi: 10.3310/hta20660.

CrossRef Full Text | Google Scholar

10. Denkel LA, Schwab F, Garten L, Geffers C, Gastmeier P, Piening B. Protective effect of dual-strain probiotics in preterm infants: a multi-center time series analysis. PLoS ONE (2016) 6:e0158136. doi: 10.1371/journal.pone.0158136

CrossRef Full Text | Google Scholar

11. Tarnow-Mordi W, Soll RF. Probiotic supplementation in preterm infants: it is time to change practice. J Pediatr. (2014) 5:959–60. doi: 10.1016/j.jpeds.2013.12.050

CrossRef Full Text | Google Scholar

12. Viswanathan S, Lau C, Akbari H, Hoyen C, Walsh MC. Survey and evidence based review of probiotics used in very low birth weight preterm infants within the United States. J Perinatol. (2016) 12:1106–11. doi: 10.1038/jp.2016.144.

CrossRef Full Text | Google Scholar

13. Esaiassen E, Cavanagh P, Hjerde E, Simonsen GS, Stoen R, Klingenberg C. Bifidobacterium longum subspecies infantis bacteremia in 3 extremely preterm infants receiving probiotics. Emerg Infect Dis. (2016) 9:1664–6. doi: 10.3201/eid2209.160033

CrossRef Full Text | Google Scholar

14. Zbinden A, Zbinden R, Berger C, Arlettaz R. Case series of Bifidobacterium longum bacteremia in three preterm infants on probiotic therapy. Neonatology (2015) 1:56–9. doi: 10.1159/000367985

CrossRef Full Text | Google Scholar

15. Kolacek S, Hojsak I, Berni Canani R, Guarino A, Indrio F, et al. Commercial probiotic products: a call for improved quality control. a position paper by the ESPGHAN working group for probiotics and prebiotics. J Pediatr Gastroenterol Nutr. (2017) 1:117–24. doi: 10.1097/mpg.0000000000001603

CrossRef Full Text | Google Scholar

16. Vallabhaneni S, Walker TA, Lockhart SR, Ng D, Chiller T, Melchreit R, et al. Notes from the field: fatal gastrointestinal mucormycosis in a premature infant associated with a contaminated dietary supplement–Connecticut, 2014. MMWR Morb Mortal Wkly Rep. (2015) 6:155–6.

Google Scholar

17. Frost BL, Modi BP, Jaksic T, Caplan MS. New medical and surgical insights into neonatal necrotizing enterocolitis: a review. JAMA Pediatr. (2017) 1:83–8. doi: 10.1001/jamapediatrics.2016.2708

CrossRef Full Text | Google Scholar

18. Hsieh EM, Hornik CP, Clark RH, Laughon MM, Benjamin DK Jr, Smith PB. Medication use in the neonatal intensive care unit. Am J Perinatol. (2014) 9:811–21. doi: 10.1055/s-0033-1361933.

CrossRef Full Text | Google Scholar

19. Esaiassen E, Fjalstad JW, Juvet LK, van den Anker JN, Klingenberg C. Antibiotic exposure in neonates and early adverse outcomes: a systematic review and meta-analysis. J Antimicrob Chemother. (2017) 72:1858–70. doi: 10.1093/jac/dkx088

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Cotten CM, Taylor S, Stoll B, Goldberg RN, Hansen NI, Sanchez PJ, et al. Prolonged duration of initial empirical antibiotic treatment is associated with increased rates of necrotizing enterocolitis and death for extremely low birth weight infants. Pediatrics (2009) 1:58–66. doi: 10.1542/peds.2007-3423

CrossRef Full Text | Google Scholar

21. Gibson MK, Wang B, Ahmadi S, Burnham CA, Tarr PI, Warner BB, et al. Developmental dynamics of the preterm infant gut microbiota and antibiotic resistome. Nat Microbiol. (2016) 1:16024. doi: 10.1038/nmicrobiol.2016.24

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Fjalstad JW, Esaiassen E, Juvet LK, van den Anker JN, Klingenberg C. Antibiotic therapy in neonates and impact on gut microbiota and antibiotic resistance development: a systematic review. J Antimicrob Chemother. (2017) 73:569–80. doi: 10.1093/jac/dkx426

CrossRef Full Text | Google Scholar

23. Kelly BJ, Gross R, Bittinger K, Sherrill-Mix S, Lewis JD, Collman RG, et al. Power and sample-size estimation for microbiome studies using pairwise distances and PERMANOV. Bioinformatics (2015) 15:2461–8. doi: 10.1093/bioinformatics/btv183

CrossRef Full Text | Google Scholar

24. Klingenberg C. Preterm Infant Gut (PINGU) - a Norwegian Multi Centre Study (PINGU). Available online at: https://clinicaltrials.gov/ct2/show/NCT02197468

25. Hill CJ, Brown JR, Lynch DB, Jeffery IB, Ryan CA, Ross RP, et al. Effect of room temperature transport vials on DNA quality and phylogenetic composition of faecal microbiota of elderly adults and infants. Microbiome (2016) 1:19. doi: 10.1186/s40168-016-0164-3

CrossRef Full Text | Google Scholar

26. Neu J, Walker WA. Necrotizing enterocolitis. N Engl J Med. (2011) 3:255–64. doi: 10.1056/NEJMra1005408

CrossRef Full Text | Google Scholar

27. Truong DT, Franzosa EA, Tickle TL, Scholz M, Weingart G, Pasolli E, et al. MetaPhlAn2 for enhanced metagenomic taxonomic profiling. Nat Methods (2015) 10:902–3. doi: 10.1038/nmeth.3589

CrossRef Full Text | Google Scholar

28. McArthur AG, Waglechner N, Nizam F, Yan A, Azad MA, Baylay AJ, et al. The comprehensive antibiotic resistance database. Antimicrob Agents Chemother. (2013) 7:3348–57. doi: 10.1128/aac.00419-13

CrossRef Full Text | Google Scholar

29. Kaminski J, Gibson MK, Franzosa EA, Segata N, Dantas G, Huttenhower C. High-specificity targeted functional profiling in microbial communities with shortBRED. PLoS Comput Biol. (2015) 12:e1004557. doi: 10.1371/journal.pcbi.1004557

CrossRef Full Text | Google Scholar

30. Klingenberg C. Manual of Neonatal Medicine. 5th Edn. Tromsø: Paediatric Deptartment, University Hospital of Northern Norway (2017).

31. Abdulkadir B, Nelson A, Skeath T, Marrs EC, Perry JD, Cummings SP, et al. Routine use of probiotics in preterm infants: longitudinal impact on the microbiome and metabolome. Neonatology (2016) 4:239–47. doi: 10.1159/000442936

CrossRef Full Text | Google Scholar

32. de Man P, Verhoeven BA, Verbrugh HA, Vos MC, van den Anker JN. An antibiotic policy to prevent emergence of resistant bacilli. Lancet (2000) 9208:973–8. doi: 10.1016/S0140-6736(00)90015-1

CrossRef Full Text | Google Scholar

33. Parks DH, Tyson GW, Hugenholtz P, Beiko RG. STAMP: statistical analysis of taxonomic and functional profiles. Bioinformatics (2014) 21:3123–4. doi: 10.1093/bioinformatics/btu494

CrossRef Full Text

34. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Series B (1995) 57:289–300.

Google Scholar

35. Huson DH, Auch AF, Qi J, Schuster SC. MEGAN analysis of metagenomic data. Genome Res. (2007) 3:377–86. doi: 10.1101/gr.5969107

CrossRef Full Text | Google Scholar

36. Weng M, Walker WA. The role of gut microbiota in programming the immune phenotype. J Dev Orig Health Dis. (2013) 3:203–14. doi: 10.1017/s2040174412000712

CrossRef Full Text | Google Scholar

37. Hill CJ, Lynch DB, Murphy K, Ulaszewska M, Jeffery IB, O'Shea CA, et al. Evolution of gut microbiota composition from birth to 24 weeks in the INFANTMET Cohort. Microbiome (2017) 1:4. doi: 10.1186/s40168-016-0213-y

CrossRef Full Text

38. Arboleya S, Binetti A, Salazar N, Fernandez N, Solis G, Hernandez-Barranco A, et al. Establishment and development of intestinal microbiota in preterm neonates. FEMS Microbiol Ecol. (2012) 3:763–72. doi: 10.1111/j.1574-6941.2011.01261.x

CrossRef Full Text | Google Scholar

39. Stewart CJ, Embleton ND, Marrs EC, Smith DP, Nelson A, Abdulkadir B, et al. Temporal bacterial and metabolic development of the preterm gut reveals specific signatures in health and disease. Microbiome (2016) 1:67. doi: 10.1186/s40168-016-0216-8

CrossRef Full Text

40. Jacobs SE, Tobin JM, Opie GF, Donath S, Tabrizi SN, Pirotta M, et al. Probiotic effects on late-onset sepsis in very preterm infants: a randomized controlled trial. Pediatrics (2013) 6:1055–62. doi: 10.1542/peds.2013-1339

CrossRef Full Text | Google Scholar

41. Bertelli C, Pillonel T, Torregrossa A, Prod'hom G, Fischer CJ, Greub G, et al. Bifidobacterium longum bacteremia in preterm infants receiving probiotics. Clin Infect Dis. (2015) 6:924–7. doi: 10.1093/cid/ciu946

CrossRef Full Text | Google Scholar

42. Wang Y, Hoenig JD, Malin KJ, Qamar S, Petrof EO, Sun J, et al. 16S rRNA gene-based analysis of fecal microbiota from preterm infants with and without necrotizing enterocolitis. ISME J. (2009) 8:944–54. doi: 10.1038/ismej.2009.37

CrossRef Full Text

43. Butel MJ, Suau A, Campeotto F, Magne F, Aires J, Ferraris L, et al. Conditions of bifidobacterial colonization in preterm infants: a prospective analysis. J Pediatr Gastroenterol Nutr. (2007) 5:577–82. doi: 10.1097/MPG.0b013e3180406b20

CrossRef Full Text | Google Scholar

44. Swidsinski A, Loening-Baucke V, Lochs H, Hale LP. Spatial organization of bacterial flora in normal and inflamed intestine: a fluorescence in situ hybridization study in mice. World J Gastroenterol. (2005) 8:1131–40. doi: 10.3748/wjg.v11.i8.1131

CrossRef Full Text | Google Scholar

45. Villmones HC, Haug ES, Ulvestad E, Grude N, Stenstad T, Halland A, et al. Species level description of the human ileal bacterial microbiota. Sci Rep. (2018) 1:4736. doi: 10.1038/s41598-018-23198-5

CrossRef Full Text | Google Scholar

46. Dutta S, Ray P, Narang A. Comparison of stool colonization in premature infants by three dose regimes of a probiotic combination: a randomized controlled trial. Am J Perinatol. (2015) 8:733–40. doi: 10.1055/s-0034-1395473

CrossRef Full Text | Google Scholar

47. Costeloe K, Hardy P, Juszczak E, Wilks M, Millar MR. Bifidobacterium breve BBG-001 in very preterm infants: a randomised controlled phase 3 trial. Lancet (2016) 10019:649–60. doi: 10.1016/s0140-6736(15)01027-2

CrossRef Full Text | Google Scholar

48. Gritz EC, Bhandari V. The human neonatal gut microbiome: a brief review. Front Pediatr. (2015) 3:17. doi: 10.3389/fped.2015.00017

CrossRef Full Text | Google Scholar

49. Deshpande GC, Rao SC, Keil AD, Patole SK. Evidence-based guidelines for use of probiotics in preterm neonates. BMC Med. (2011) 9:92. doi: 10.1186/1741-7015-9-92

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Siggers RH, Siggers J, Thymann T, Boye M, Sangild PT. Nutritional modulation of the gut microbiota and immune system in preterm neonates susceptible to necrotizing enterocolitis. J Nutr Biochem. (2011) 6:511–21. doi: 10.1016/j.jnutbio.2010.08.002

CrossRef Full Text | Google Scholar

51. Cilieborg MS, Boye M, Sangild PT. Bacterial colonization and gut development in preterm neonates. Early Hum Dev. (2012) 88:S41–9. doi: 10.1016/j.earlhumdev.2011.12.027

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Watkins C, Murphy K, Dempsey EM, O'Shea CA, Murphy BP, O'Toole PW, et al. Dose-interval study of a dual probiotic in preterm infants. Arch Dis Child Fetal Neonatal Ed. (2018) 20:313468. doi: 10.1136/archdischild-2017-313468

CrossRef Full Text | Google Scholar

53. Hall MA, Cole CB, Smith SL, Fuller Rolles CJ. Factors influencing the presence of faecal lactobacilli in early infancy. Arch Dis Childhood (1990) 2:185–88.

Google Scholar

54. Fouhy F, Guinane CM, Hussey S, Wall R, Ryan CA, Dempsey EM, et al. High-throughput sequencing reveals the incomplete, short-term recovery of infant gut microbiota following parenteral antibiotic treatment with ampicillin and gentamicin. Antimicrob Agents Chemother. (2012) 11:5811–20. doi: 10.1128/aac.00789-12

CrossRef Full Text

55. Rose G, Shaw AG, Sim K, Wooldridge DJ, Li MS, Gharbia S, et al. Antibiotic resistance potential of the healthy preterm infant gut microbiome. PeerJ. (2017) 5:e2928. doi: 10.7717/peerj.2928

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Grazul H, Kanda LL, Gondek D. Impact of probiotic supplements on microbiome diversity following antibiotic treatment of mice. Gut Microbes (2016) 2:101–14. doi: 10.1080/19490976.2016.1138197

CrossRef Full Text | Google Scholar

57. Fouhy F, Ogilvie LA, Jones B, Ross RP, Ryan AC, Dempsey EM, et al. Identification of aminoglycoside and beta-lactam resistance genes from within an infant gut functional metagenomic library. PLoS ONE (2014) 9:e108016. doi: 10.1371/journal.pone.0108016

PubMed Abstract | CrossRef Full Text | Google Scholar

58. von Wintersdorff CJ, Wolffs PF, Savelkoul PH, Nijsen RR, Lau S, Gerhold K, et al. The gut resistome is highly dynamic during the first months of life. Future Microbiol. (2016) 4:501–10. doi: 10.2217/fmb.15.154

CrossRef Full Text | Google Scholar

59. Moore AM, Patel S, Forsberg KJ, Wang B, Bentley G, Razia Y, et al. Pediatric fecal microbiota harbor diverse and novel antibiotic resistance genes. PLoS ONE (2013) 11:e78822. doi: 10.1371/journal.pone.0078822

CrossRef Full Text | Google Scholar

60. Pamer EG. Resurrecting the intestinal microbiota to combat antibiotic-resistant pathogens. Science (2016) 6285:535–8. doi: 10.1126/science.aad9382

CrossRef Full Text | Google Scholar

61. Ravi A, Estensmo ELF, Abee-Lund TML, Foley SL, Allgaier B, Martin CR, et al. Association of the gut microbiota mobilome with hospital location and birth weight in preterm infants. Pediatr Res. (2017) 82:829–838. doi: 10.1038/pr.2017.146

PubMed Abstract | CrossRef Full Text | Google Scholar