Candida spp. Determination and Th1/Th2 Mixed Cytokine Profile in Oral Samples From HIV+ Patients With Chronic Periodontitis

Background: Chronic periodontitis (CP), caused by bacteria and fungi, appears in up to 66% of HIV-patients. The impact and association of HIV-treatment (HAART) and Candida itself has not been properly evaluated in the development and progression of CP. The immunopathogenesis is characterized by CD4⁺ T-cells activation and the balance between the T-helper 1 (Th1) and T-helper 2 (Th2) or a mixed cytokine profile. Currently, the associated causes of an immune response in HIV-patients with CP is controversial. Our aims were the determination of Candida spp. and cytokine profile in oral samples from HIV-positive patients with CP, considering the CD4⁺ T cells levels and HAART use.

Methods: From 500 HIV-positive patients evaluated, 228 patients were enrolled. Patients were separated in groups: (A) n = 53 (≤200 CD4⁺ T-cells on HAART); (B) n = 57 (≤200 CD4⁺ T-cells without HAART); (C) n = 50 (>200 CD4⁺ T-cells without HAART); (D) n = 68 (>200 CD4⁺ T-cells on HAART). Candida spp. were isolated from the oral biofilm and crevicular fluid in CHROMagar and confirmed by endpoint PCR. Cytokine levels were measured by beads-based immunoassay in saliva by flow cytometry.

Results: 147 patients (64.5%) were positive to Candida spp. and 204 strains were isolated; 138 (67.6%) were C. albicans and the remaining C. non-albicans species (C. glabrata>C. tropicalis>C. krusei>C. dubliniensis). In this study, CHROMagar showed good sensitivity (95%) but poor specificity (68%); since of the 152 samples identified as C. albicans, only 131 were confirmed by PCR; from the 10 samples identified as C. glabrata, only six were confirmed. Finally, of the 42 samples detected as C. tropicalis, only five were confirmed. When evaluating Candida spp. presence, group A and D had higher isolation, while group B had the highest species diversity. Whereas, group C had a significant reduction of Candida spp. Despite the presence of Candida and HAART, we found a Th1/Th2 hybrid profile in the saliva of patients with low CD4⁺ T-cell count (group A).

Conclusion: Abundance and diversity of the Candida spp. detected in HIV-patients with CP could be related to HAART and low CD4⁺ T-cells levels. Also, the immunosuppression might promote a local Th1/Th2 hybrid cytokine profile.

Introduction

In HIV positive (HIV+) patients, periodontitis represents one of the first opportunistic pathologies that manifest in the oral cavity, appearing in up to 66% of these patients (1). Periodontitis is a disease caused by both Gram-positive and negative bacteria and oral fungi. It is characterized by inflammation in the gum and adjacent tissues, which causes the destruction of teeth structural supports (2, 3). The mechanisms of damage include both the direct tissue damage caused by the bacterial products of the dental biofilm and the indirect damage produced by the bacterial induction of host inflammatory and immune response (2–5). The immunopathogenesis of periodontitis is orchestrated by the innate and adaptive immune response, mainly through the activation of CD4⁺ T cells and the balance between the T helper 1 (Th1) and T helper 2 (Th2) subtypes or a mixed cytokine profile (5, 6). Th1 signature cytokines are interleukin-2 (IL-2) and Interferon-gamma (IFN-γ), while Th2 cells secrete their signature cytokine interleukin 4 (IL-4) plus interleukin 5 (IL-5) and interleukin 10 (IL-10). It is well-known, that oral response against fungi in HIV negative (HIV-) subjects is orchestrated by a Th17 profile, this cytokine profile promote inflammation through the induction of a Th1 cytokine profile (inflammatory), recruitment of neutrophils, and production of reactive oxygen species such as nitrogen oxide (NO). This oral microenvironment is produced by infiltrating immune cells that are stimulated by the presence of opportunistic microorganisms and commensal bacteria that are part of the subgingival biofilm. This proinflammatory microenvironment (IL-1β, IFN- γ, IL-6, NO) is associated to bone loss, and the induction of a Th2 profile is a compensatory mechanism that controls the inflammation and promotes immune homeostasis (7).

Since HIV infection is characterized by the depletion of CD4⁺ T cells and several changes in the whole cellular and humoral immune response that could lead to an immunodeficient state, which may allow subgingival colonization by different pathogens, HIV+ patients exhibit more oral manifestations that HIV- subjects (8). One of such pathogens is Candida, which can aggregate jointly with other bacteria to the subgingival biofilm and attach to the epithelial cells of patients with periodontal disease (8–10). C. albicans is the most frequent species identified in patients with periodontal disease. However, other species have also been found like C. tropicalis, C. glabrata, C. krusei, and C. guilliermondii (8, 10, 11). Also, it has been suggested that Candida spp. play a role in both the pathogenesis and the severity of periodontal disease (11–13). These last aspects gather importance in HIV+ patients. Both the local and systemic effects of the periodontal disease and the chronic immune activation associated with a co-infection are crucial factors in AIDS severity and progression. Furthermore, periodontal bacteria favor Epstein-Barr virus and Kaposi's sarcoma-associated herpesvirus reactivation (14).

It has been reported that CD4⁺ T cells count >200 cells/μL increase the probability to acquire oral manifestations such as oropharyngeal candidiasis (OPC) (15, 16). A possible explanation to this susceptibility is by the depletion of Th17 (particularly in the gut) and an increase of a Th2 profile in the mucosa of HIV+ patients, as both factors are associated with susceptibility to mucosal candidiasis (7, 17, 18). In OPC, CD8⁺ T cells are recruited to the mucosa in response to Candida infection, promoting a proinflammatory microenvironment characterized by a Th1 cytokine profile (19). This compensatory response in OPC is reflected in a Th1/Th2 cytokine profile, however, the establishment of a similar immune profile has not been described in CP.

In our knowledge, HIV+ patients with CP have been studied without considering Candida infection, nor the clinical conditions of the patients, such as CD4⁺ T cell count and HAART. In addition, they do not evaluate if CD4⁺ T cell count or HAART could modify the cytokine environment or promote atypical Candida spp. colonization in the mouth. Several studies analyzed the role of Candida spp. in mucosal infection in HIV+ patients, however, most of them are focused in OPC, while this comorbidity has been studied microbiologically and immunologically, other infections driven by Candida spp. such as CP, remain poorly known. We hypothesized that Candida spp. colonization in HIV+ patients with CP is associated with a Th1/Th2 profile independently of the immune state and the presence of HAART, however, these two last factors are related to the diversity of Candida spp.

Our aims were the determination of Candida spp. and cytokine profile in oral samples from HIV+ patients with CP, considering the CD4⁺ T cells levels and HAART use.

Materials and Methods

Study Population

We performed a cross-sectional (observational) study including 500 HIV+ patients with untreated CP in the last 3 months. HIV patients were recruited from the HIV clinic of the tertiary care university hospital “Antiguo Hospital Civil de Guadalajara—Fray Antonio Alcalde” (a 1,000-bed teaching hospital in Western Mexico) from February to September 2017. Non-inclusion criteria were pregnancy, breastfeeding, no sign of CP, oral candidiasis, and use of dental prostheses. This study was carried out in accordance with the recommendations of the Declaration of Helsinki, World Medical Association with written informed consent from all subjects. The protocol was approved by the ethics committee in research of the “Antiguo Hospital Civil de Guadalajara—Fray Antonio Alcalde” with registry number 020/16 (20). All subjects gave written informed consent in accordance with the Declaration of Helsinki.

The sample size was calculated using a finite population proportion formula based on the prevalence previously reported (21). Detailed sociodemographic, immunological, and health-related information were obtained during the scheduled medical visit.

Selected patients were grouped as follows: Group (A) patients with CD4⁺ ≤200 cells/μL on HAART; Group (B) patients with CD4⁺ ≤200 cells/μL without HAART; Group (C) patients with CD4⁺ >200 cells/μL without HAART; and Group (D) patients with CD4⁺ >200 cells/μL on HAART.

Periodontal Evaluation

The periodontal evaluation was performed by a periodontist with a North Carolina periodontal probe number 12 (Hu-Friedy). This analysis evaluated the mesiobuccal, buccal, distobuccal, mesiolingual, lingual, and distolingual site of each tooth (except the third molars). In each site: sounding depth, Clinical Attachment Level (CAL) and bleeding on probing were evaluated. Periodontitis diagnosis was established according to the Disease Control/American Association of Periodontology Classification (22), which considers the CAL scale.

Sample Collection

Crevicular fluid and subgingival biofilm were obtained from the most severe site in each oral quadrant at the base of the periodontal pocket. The crevicular fluid was obtained by introducing filter paper for 60 s. Subgingival biofilm was obtained with a sterile Gracey curette (Hu-Friedy). For Candida spp. identification, crevicular fluid, and subgingival biofilm were collected in thioglycolate medium (BD) tubes. Non-stimulated saliva samples were collected in a clean 2 mL amber tubes and stored at −70°C. All samples were collected before noon after periodontal evaluation.

Candida spp. Detection by CHROMagar^®

Crevicular fluid and subgingival biofilm were streak-plated in CHROMagar Candida^® medium, incubated at 36°C for 48 h. Identification was carried out by colorimetric changes as specified by the manufacturer's instructions. All CHROMagar were analyzed by a certified microbiologist.

Endpoint PCR From Yeast Colonies

Endpoint PCR was performed in yeast colonies as previously described (https://openwetware.org/wiki/Smolke:Protocols/Yeast_Colony_PCR). Briefly, primary Candida spp. acquired from the CHROMagar were re-cultured overnight in yeast 1% extract, 2% peptone, and 2% dextrose (YPD) medium. Yeast colonies were harvested with a sterile pipette tip and resuspended in 3 μL of freshly made NaOH (20 mM). Cells were lysed and incubated at 95°C for 10 min. The 3 μL were used as molds for the PCR reaction, which was performed in a total volume of 25 μL. PCR mix was made with 5 mM dNTPs, 15 mM MgCl₂, and Taq polymerase following the manufacturer's instruction (MyTaq^TM DNA Polymerase Kit, Bioline). Three percent of DMSO was added to improve reaction as previously described (23) and 10 mM of primers specific for every species (Table 1).

TABLE 1

Table 1. Primer sets used for Candida spp. PCR detection.

Endpoint PCR was performed in a MiniCycler MJ Research thermal cycler. PCR program was as follows: 5 min of denaturalization at 95°C, 30 cycles of 30 s at 94°C (denaturalization), 30 s at 50°C (hybridization), and 30 s at 72°C (elongation). Afterward, a final elongation at 72°C for 10 min was performed (24).

Electrophoresis

DNA fragments from the PCR reaction were separated in 2% agarose gel (CONDA Micro and Molecular Biology^®) prepared with TAE buffer (Tris-acetate 40 mM, pH 8.5; EDTA 0.05 mM). Ten microliter of Red Safe^® (iNtRON) were added. Photographs were taken with a UVITEC transilluminator.

Cytokine Determination

Saliva samples were centrifuged at 1,000 g for 10 min, the supernatant was recollected and used for the assay. LEGENDplex Human T Helper Cytokine Th1/Th2 Panel kit (BioLegend) was used for cytokines detection. The evaluated cytokines were: IL-2, IL-4, IL-5, IL-6, IL-10, IL-13, IFN-γ, and Tumor Necrosis Factor-alpha (TNF-α). The assay was performed following manufacturer recommendations. Samples were read using the NxT Attune flow cytometer (Applied Bioscience) and analyzed with the LEGENDplex v8 Software (BioLegend).

Statistical Analysis

Data were analyzed using Statistical Package for the Social Scientist (SPSS) version 23 (IBM) and GraphPad Prism 6 software. Medians and interquartile ranges (IQR) or means and standard deviations (SD) were reported depending on data distribution. Sociodemographic and Candida spp. presence data were compared with either Student's t-test, Fisher exact, or Chi-square, according to the variable analyzed; binary linear regression analysis was performed to estimate the association of candida presence, considering the following variables: use of HAART, age, gender, level of CD4⁺ T cells (> or ≤200 cells/μL), tobacco, alcohol, and drug use; and Cohen's kappa coefficient (κ) was reported. The cytokines differences between the different groups were estimated using the Kruskal-Wallis test with Dunn's Multiple Comparison test. Shapiro-Wilks normality test was significant, so the data distribution was considered non-Gaussian. p-values < 0.05 were considered significant.

Results

Group Study Characteristics

Periodontal evaluation was performed in 500 HIV+ volunteer patients. After assessing the selection criteria, 228 patients were included in the study. Next, patients were divided into different groups: 53 patients were assigned to Group A, 57 patients to Group B, 50 patients to Group C, and 68 patients to Group D (Figure 1).

FIGURE 1

Figure 1. Flowchart of patient recruitment and distribution. Two hundred twenty-eight patients were selected after exclusion criteria and separated into four groups. Due to low quality in saliva samples, we were not able to determine the cytokines profile in the samples from all patients included in this study.

Cytokine determination in saliva was performed only in 34 from Group A, 47 from Group B, 47 from Group C, and 53 from Group D; this due to lack of enough sample or samples that were blood-stained. It was observed a significant difference among age, CD4⁺ T cells count, CD4/CD8 ratio, viral load, duration of HAART, and grades of severity according to the CAL scale. Regarding age, the group with the older patients was group D (43 ± 11 years; p = 0.05). Patients with ≤200 CD4⁺ T cells had a median of 17 months under HAART compared to the 38 months among patients with >200 CD4⁺ T cells (p = 0.0001). Furthermore, this last group also had an undetectable viral load (p = 0.0001). In addition, group B—with ≤200 CD4 ⁺ T cells- presented less “mild” cases and more “severe” cases, according to the CAL classification, when compared against group C—with >200 CD4⁺ T cells-, both groups without HAART (p = 0.05).

In respect to gender, the majority of our patients were men (4:1). Because in group B there was only one woman, a significant difference was found but this statistic finding should be considered with caution. We did not find differences among the four groups in terms of the habits of smoking, drinking or drug use, but is important to emphasize that more than 62% of patients used tobacco, that is already considered as a risk factor for CP. Besides, diabetes was absent in more than 93% in all groups, that is relevant considering that it is another known risk factor for CP and was absent in the majority of our patients (25). The treatment scheme in the groups under HAART was based mainly in non-nucleoside reverse transcriptase inhibitors (NNRT), followed by protease inhibitors (PI), and integrase inhibitors; nonetheless, there were no differences statistically significant between the groups. Complete sociodemographic data is described in Table 2.

TABLE 2

Table 2. Sociodemographic characteristics including all patients.

After the logistic regression analysis, we found a significantly association with the presence of Candida spp. with levels ≤200 cells/μL of CD4⁺ T cells (OR: 3.59 CI 95% 1.89–6.80; p=0.0001), the use of PIs (OR: 2.67 CI 95% 1.13–6.29; p = 0.025) and NNTRIs (OR: 2.99 CI 95% 1.37–6.49; p = 0.006; κ = 0.408).

Candida spp. Varies Between Groups and Identification Methods

HIV infection is associated with an increased colonization rate by Candida spp. as well as the development of other diseases, such as periodontal disease. Our first goal was to identify the Candida spp. present in the different groups of HIV+ patients with periodontal disease.

From the 228 patients included in the study, 147 (64.5%) were positive to Candida spp. In total 204 strains were isolated, of which 138 (67.6%) were C. albicans, and the remaining 66 (32.4%) had C. non-albicans species. The identified non-albicans species were: 48 (23.6%) C. glabrata, 10 (4.9%) C. tropicalis, 7 (3.4%) C. krusei, and 1 (0.5%) C. dubliniensis.

In 145 (71.1%) of the total isolated strains, only one type of Candida was isolated, while the remaining 59 isolated strains (28.9%) more than one type of Candida were isolated. The most common combinations were: C. albicans + C. glabrata (19.6%), C. albicans + C. krusei (4.9%), C. glabrata + C. tropicalis (2%), and C. albicans + C. glabrata + C. tropicalis (1.5%). When comparing all combinations between the study groups, Group C had significantly fewer combinations in comparison with Group D (p = 0.011). The distribution of Candida spp. strains separated by groups are shown in the Supplementary Table 1.

Identification of Candida Species With Different Methods

When carrying out this study, it was necessary to use different methods of identification for Candida spp., including conventional methods as well as molecular techniques. At the beginning, with streak-plating technique by CHROMagar, C. albicans, C. glabrata, and C. tropicalis were identified in 152, 10, and 42 samples; respectively. However, of the 152 C. albicans identified, molecular confirmation was positive in 131 of these samples. Therefore, PCR for the other Candida species was performed in the remaining 21 samples; 14 of these samples were positive for C. glabrata, and 3 for C. tropicalis. The last four samples were not identified by PCR. From the 10 samples identified as C. glabrata, six were confirmed by molecular techniques. The remaining four samples were tested by PCR for the other Candida species, two were identified as C. albicans and the others as C. tropicalis. The last 42 samples were detected as C. tropicalis, but only five were confirmed by PCR. In the other 37 samples, 28 were identified as C. glabrata, and five as C. albicans. Again, four samples were not identified to any Candida spp. by PCR.

The eight samples that were negative in all PCRs were sent to “La Fe” Hospital in Valencia, Spain. Candida spp. was detected through matrix-assisted laser desorption/ionization time-of-flight mass spectrometry (MALDI-TOF MS). From the four samples detected as C. albicans by streak-plating technique, one was identified as C. dubliniensis and the remaining as C. krusei. The four samples detected as C. tropicalis by plating were confirmed as C. krusei by MALDI-TOF MS. A summary is provided in Figure 2.

FIGURE 2

Figure 2. Candida spp. identification by methods. All samples were cultivated first in CHROMagar for initial identification. Afterward, all identified Candida spp. by traditional microbiologic methods were evaluated by PCR technique to confirm the species. If the PCR for the specific species found in CHROMagar was negative, further PCRs were made for other Candida spp. Any Candida spp. that was not confirmed by PCR, due to lack of the primers specific for that species, were identified through MALDI-TOF technique.

With this information, we compared the accuracy of CHROMagar Candida medium to identify Candida spp. with streak-plating technique against PCR; results are shown in Table 3.

TABLE 3

Table 3. Capacity of Candida spp. identification with CHROMagar Candida Medium.

Candida spp. and Colonization Varies Depending on CD4⁺ T Cells Count and HAART

In the course of HIV infection, the rate of Candida infection is inversely related to the CD4⁺ T cells count. When evaluating Candida spp. between the groups, C. albicans and C. glabrata were identified in all of them; C. krusei was detected in Groups A, B, and D; C. tropicalis in Group B and D, while C. dubliniensis was present exclusively in Group B.

When analyzing the number of patients with Candida in each group, we found a significant reduction of Candida spp. in the Group C in comparison with the other groups (p ≤ 0.001 vs. A, B, D). As well, C. albicans was significantly diminished in Group C (p < 0.01 vs. A and B; p < 0.05 vs. D). For patients with C. glabrata, Group C had also a significant decrease in comparison with B and D (p < 0.05 and p < 0.01, respectively). Data are shown in Table 4.

TABLE 4

Table 4. Patients with presence of Candida spp. in each group.

Th1/Th2 Mixed Cytokines Profile in the Saliva Is Mainly Promoted by Low CD4⁺ T Cell Count

Saliva samples from HIV+ patients with CP and Candida spp. growth were analyzed to evaluate if CD4⁺ T cell count or HAART modifies the local Th1 and Th2 related cytokine expression. With respect to Th1-related cytokines, IFN-γ was significantly higher in group A and B in relation to group D; the levels of TNF-α in group A were significantly higher against group D, and significantly higher to group B for IL-2. Finally, group B had significantly more IL-6 in comparison with group C and D.

Regarding Th2 signature cytokines, group A had a significant increase of IL-5 in comparison with group B and D; whilst both groups with low CD4⁺ T cell count (A and B) had a significant increase of IL-10 and IL-4 respect to group D. Group A was also significant with group C for IL-4. When comparing cytokine expression in each group, there was no difference for IL-13 (Table 5; Figure S1). We were able to observe a mixed increase of some Th1 and Th2 cytokines, although a major predisposition for a Th2 profile was found. Interestingly, the groups with more levels of cytokines overall are the ones with low CD4⁺ T cell count.

TABLE 5

Table 5. Cytokine levels in saliva samples from HIV+ patients per group.

Discussion

This study is one of the few studies that measure CP in different groups considering the CD4⁺ T cell count and HAART. Because other studies divided their groups differently to ours, we separate and regroup our data as required to better compare our results with the literature.

In this study, patients with CP and CD4⁺ T cells ≤200 cells/μL had a major prevalence of Candida infection (73.7%) than the one reported in previous studies (11–13, 26). In addition, the number of patients, in this study, colonized with C. albicans (40.8%), C. tropicalis (3.9%), and C. krusei (2.2%) is less than the ones found previously (11, 26). However, similar results to ours were found by another group (27). Furthermore, we found a higher amount of C. glabrata colonization (16.2%) in comparison with the literature (11, 26–28).

Regarding the distribution of Candida spp. isolation, C. albicans (67.7%), C. glabrata (23.5%), and C. krusei (3.4%) had a major distribution than the one reported (13, 29, 30). On the other hand, the distribution that we detected for C. tropicalis (4.9%) was lower than previous papers (13, 29, 30). Like previous studies, we identified C. dubliniensis in one of the patients (0.5%) (26, 29, 30). This is the first study in Mexico that identifies Candida dubliniensis in HIV+ patients with CP; it was previously reported in USA (31) and this could increase the relevance of screening this species in Mexican populations with CP.

In this study, we required to use several techniques for Candida spp. identification. Currently, several authors have considered the use of CHROMagar Candida medium as a diagnostic test directly from the oral cavity (27, 28, 32). Taking advantage that we used this technique, we evaluated the CHROMagar effectiveness as a diagnosis technique. Since CHROMagar is a presumptive test, we confirmed our results with endpoint PCR and found a sensibility of 95%, a specificity of 68%, and an accuracy of 86% for CHROMagar. These results differ from the studies that suggest a sensibility and a specificity of 99% (33, 34). These discrepancies could be explained by the different saliva sampling techniques used: isotopes method (28, 30, 35), mouthwash method (11), filter papers method (12, 36); or the identification technique performed: biochemistry tests (11, 28, 30), chromogenic medium (12, 28, 36) PCR (29) or MALDI-TOF MS (27). Therefore, an acquisition technique should be standardized when CHROMagar is used. Nonetheless, considering our results we suggest that CHROMagar Candida Medium requires a confirmatory technique, such as PCR, to ensure the results, especially for Candida non-albicans species, as recommended by other authors (24, 29, 32).

It is known, that the interaction of various Candida spp. with bacteria from the dental plaque promotes bacterial colonization and proliferation in the periodontal pockets. This contributes to the progression of periodontal disease (8). We isolated in 71.1% of the patients a single Candida spp.; and two or more Candida spp. in 28.9% of our patients. During the periodontal evaluation, the most severe patients were the ones with two or more Candida spp. (CAL ≥5 mm). Also, we detected an increase distribution of the C. albicans + C. glabrata coinfection (19.6%) in comparison with other studies (26, 29). Our results suggest that a transition to coinfection with multiple Candida spp. is a factor that facilitates CP and increases severity. Clark et al. (29), did a study with HIV+ Mexican patients without a periodontal evaluation and through a different acquisition technique (mouthwash) and they found in 94.1% of their patients a single Candida spp. and two or more species in 6.5% of them. Furthermore, the most frequent Candida non-albicans species that we found was C. glabrata (23.6%) in comparison with Clark et al. (29), which found C. tropicalis. In this sense, the differences found between our study and Clark et al. (29) study could be related to potential geographical environmental factors; it has already been proposed that environmental factors could have importance in the type of Candida spp. in oral colonization, factors that could be relevant when we compared our results with the literature (37).

The use of HAART and the immune reconstitution have been associated with a significant reduction of viral load and opportunistic infections, including oral manifestations like oral hairy leukoplakia, Kaposi's sarcoma and oral candidiasis (10, 21). However, it remains controversial the effect of HAART in incidence and severity of the atypical periodontal diseases, such as linear gingival erythema, necrotizing periodontitis and CP (10, 21, 38, 39); the latter, probably caused by the known effect of HAART over the salivary flow (hyposalivation, xerostomia, or dysgeusia), which modifies the oral microbiota and facilitates Candida spp. colonization (40, 41); yet other studies could not find a correlation between Candida colonization and HAART usage (29, 30, 42, 43). In this sense, we evaluated if the presence of HAART affected the amount of Candida colonization in CP. We detected that groups with HAART, especially with the use of PIs and NNRTIs, had a significant increase of Candida colonization. Interestingly, we also found an increase in diversity. Therefore, we suggest that a secondary effect of HAART is the promotion of oral dysbiosis.

Concerning CD4⁺ T cell count, the groups with CD4⁺ T cells count ≤200 cells/μL had a significant increase of Candida spp. colonization in comparison with the groups with CD4⁺ T cell count >200 cells/μL (OR: 3.59; p = 0.0001). This finding is consistent with the literature (11, 44). In summary, CD4⁺ T cell count in HIV+ patients play a major role in the subgingival colonization of Candida spp. in CP. It has been reported that during periodontal disease the expression of CD4⁺, CXCR5⁺, and CCR5⁺ is increased (45, 46); the latter has also been reported increased in smokers (47). This overexpression facilitates the entry of HIV virions through dendritic cells (DC) and macrophages, which are the prevalent cells in the oral mucosa; this could cause major cell death, local inflammation, and promotion for pathogen colonization like Candida. On average, 66% of HIV+ patients in our study are current smokers, it seems that this continues to be an important factor for the development of periodontal disease, but in our study, it was not shown that tobacco influenced Candida colonization.

Lastly, we assessed the local cytokine environment occurring in HIV+ patients with CP. For this, we analyzed saliva samples and found that the patients had a mixed cytokine profile. Similar results have been found previously in the literature, however, the cytokine diversity has varied within the studies (17, 18, 48, 49). This could be attributed to discrepancies in selection criteria within the studies and the period of the disease -acute or chronic-. In addition, we saw that the most increase of cytokines occurred in the groups with low CD4⁺ T cell count. A plausible theory for this finding is that cytokine production could be innate-mediated. DC and macrophages could be the main cells producing cytokine as they have been observed to remain prevalent in oral mucosa during periodontal disease (45, 46). Although, other types of cells could be involved in the cytokine environment, such as innate lymphoid cells (ILC) and natural killer (NK) cells. Li and Reeves (50) showed that ILC are increased and activated in oral mucosa during simian immunodeficiency viral infection, these cells can produce several cytokines and we consider that the innate immune system should be studied further in CP, as well as evaluating the role of ILC and NK cells in human's oral cavity. As we previously mentioned, HIV+ patients have a dominant Th2 cytokine environment, irrespective of periodontal disease status (18), since the immune response against Candida spp. requires a plethora of immune cells (mainly CD8⁺ T cells); the presence of a Th1 cytokine profile seems to be essential in the development of a resolutive immune response in CP. It is unknown, whether the Th2 cytokines present in the Th2/Th1 profile remains as a consequence in the oral immune microenvironment by the HIV infection, or if they are a secondary immune reactivation in order to establish a homeostatic mechanism. Something that we did observe is that the Th1/Th2 profile is present in all groups colonized with one or more Candida spp., despite their clinical, and demographic differences.

Though it has been reported that smoking promotes a Th1/Th2 cytokine profile in the saliva of HIV+ patients with OPC (51), in our study this factor did not have a major impact in the establishment of the mixed cytokine profile, since the majority of our study population presented a Th1/Th2 cytokine profile yet only 40% of them reported smoking habits. We considered that the shift in the local immune response from a Th2 to a Th1/Th2 profile is linked directly to the presence of Candida spp. (regardless of colonization), this data is consistent with the previous cytokine profile reported in OPC (17). In a non-HIV context, the progression of CP evolves from Th1/Th2 to Th1/Th17 profile (52, 53). Unfortunately, in our study we did not evaluate the Th17 profile. Whether these results are an early phase, which will further develop into a Th1/Th17 cytokine profile remains unknown and an interesting question to be answered in future studies.

With these findings and what it has been reported in the literature (7), we proposed a potential diagram of the immune physiopathology of CP in patients with HIV (Figure 3). Briefly, the loss of Th17 cytokines and increased presence of the Th2 profile, facilitate the oral colonization with pathogens such as Candida spp. This colonization activates local innate immune cells -DC, macrophages, ILC, NK cells, and others- which secrete Th1 cytokines, promoting the presence of a Th1/Th2 profile, and also secreting chemoattractant (TNF-α, CXCL1, and GM-CSF), that allows neutrophils recruitment and increase their survivability (54). At the same time, the release of ROS elements promotes bone resorption by activated osteoclasts (55). This local immune behavior is strengthened by smoking habit that increases the CCR5 expression, which facilitates local HIV entrance to DC and macrophages, promoting an onset of a “trojan horse” effect and a major local inflammation. Altogether, this damage promotes the destruction of the teeth architecture and facilitates the translocation of microbial products that could be enhancing a systemic inflammation.

FIGURE 3

Figure 3. Proposal of the immune response in HIV patients with CP. (A) Suggests the progression to CP in the oral cavity while (B) shows a proposal for the involvement of CP in the systemic inflammation environment of HIV patients. Created with BioRender.

Data Availability

All datasets generated for this study are included in the manuscript and/or the Supplementary Files.

Ethics Statement

This study was carried out in accordance with the recommendations of the Declaration of Helsinki, World Medical Association with written informed consent from all subjects. The protocol was approved by the ethics committee in research of the Antiguo Hospital Civil de Guadalajara—Fray Antonio Alcalde with registry number 020/16. All subjects gave written informed consent in accordance with the Declaration of Helsinki.

Author Contributions

SL-M performed the recruitment of patients, their periodontal evaluations, saliva acquisition, and original draft. SL-M, LG-H, VR-A, and ML-M helped with the diagnosis, evaluation, and assessment for inclusion or exclusion of patients for this paper. SL-M, EV-G, and JV-H performed PCRs and Candida identification. MA-Z, KS-R, MR-S, and RC-S performed cytokine determination and analysis. SL-M, JV-H, EV-G, RC-S, MA-Z, KS-R, MR-S, LG-H, and JA-V literature search, writing, review, and editing. SL-M, RC-S, MA-Z, KS-R, and MR-S developed the tables and figures for this manuscript. EV-G, SM-S, LG-H, and JA-V acquired the funds for this project.

Funding

This work was partially supported by grant number PI12/01797 from Spanish MINECO integrated in the Fondo Europeo de Desarrollo Regional (FEDER) and cofinanced by Instituto de Salud Carlos III (ISCIII) and the 2016 and 2017 Programa de Movilidad para Formación, Investigación y Docencia (ProMoFID).

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We thank GMCA Research Unit (University of Valencia, Spain) and Severe Infection Group (Health Research Institute La Fe, Valencia, Spain) for their technical support in Candida identification.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fimmu.2019.01465/full#supplementary-material

References

1. Groenewegen H, Bierman WFW, Delli K, Dijkstra PU, Nesse W, Vissink A, et al. Severe periodontitis is more common in HIV- infected patients. J Infect. (2018) 78:171–7. doi: 10.1016/j.jinf.2018.11.008

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Artega Chirinos F, Quiñónez B, Prado J, Florido R. Enfermedades periodontales asociadas a la infección del virus de inmunodeficiencia adquirida, reporte de lesiones orales y corporales. Rev ADM. (2008) LXV:322–6. Available online at: https://www.medigraphic.com/pdfs/adm/od-2008/od086h.pdf

Google Scholar

3. Gonçalves LS, Gonçalves BML, Fontes TV. Periodontal disease in HIV-infected adults in the HAART era: clinical, immunological, and microbiological aspects. Arch Oral Biol. (2013) 58:1385–96. doi: 10.1016/j.archoralbio.2013.05.002

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Agbelusi GA, Eweka OM, Umeizudike KA, Okoh M. Oral manifestations of HIV. In: Current Perspectives in HIV Infection. InTech (2013). Available oline at: http://www.intechopen.com/books/current-perspectives-in-hiv-infection/oral-manifestations-of-hiv doi: 10.5772/52941

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Silva N, Abusleme L, Bravo D, Dutzan N, Garcia-Sesnich J, Vernal R, et al. Host response mechanisms in periodontal diseases. J Appl Oral Sci. (2015) 23:329–55. doi: 10.1590/1678-775720140259

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Ohlrich E, Cullinan M, Seymour G. The immunopathogenesis of periodontal disease. Aust Dent J. (2009) 54:S2–10. doi: 10.1111/j.1834-7819.2009.01139.x

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Heron SE, Elahi S. HIV infection and compromised mucosal immunity: oral manifestations and systemic inflammation. Front Immunol. (2017) 8:241. doi: 10.3389/fimmu.2017.00241

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Sardi JCO, Duque C, Mariano FS, Peixoto ITA, Höfling JF, Gonçalves RB. Candida spp. in periodontal disease: a brief review. J Oral Sci. (2010) 52:177–85. doi: 10.2334/josnusd.52.177

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Murray PA. Periodontal diseases in patients infected by human immunodeficiency virus. Periodontol 2000. (1994) 6:50–67. doi: 10.1111/j.1600-0757.1994.tb00026.x

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Mataftsi M, Skoura L, Sakellari D. HIV infection and periodontal diseases: an overview of the post-HAART era. Oral Dis. (2011) 17:13–25. doi: 10.1111/j.1601-0825.2010.01727.x

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Lourenço AG, Ribeiro AERA, Nakao C, Motta ACF, Antonio LGL, Machado AA, et al. Oral Candida spp carriage and periodontal diseases in HIV-infected patients in Ribeirão Preto, Brazil. Rev Inst Med Trop Sao Paulo. (2017) 59:e29. doi: 10.1590/s1678-9946201759029

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Gaetti-Jardim Júnior E, Nakano V, Wahasugui TC, Cabral FC, Gamba R, Avila-Campos MJ. Occurrence of yeasts, enterococci and other enteric bacteria in subgingival biofilm of HIV-positive patients with chronic gingivitis and necrotizing periodontitis. Braz J Microbiol. (2008) 39:257–61. doi: 10.1590/S1517-83822008000200011

CrossRef Full Text | Google Scholar

13. Junqueira JC, Vilela SFG, Rossoni RD, Barbosa JO, Costa ACBP, Rasteiro VMC, et al. Oral colonization by yeasts in HIV-positive patients in Brazil. Rev Inst Med Trop Sao Paulo. (2012) 54:17–24. doi: 10.1590/S0036-46652012000100004

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Imai K, Ochiai K. Role of histone modification on transcriptional regulation and HIV-1 gene expression: possible mechanisms of periodontal diseases in AIDS progression. J Oral Sci. (2011) 53:1–13. doi: 10.2334/josnusd.53.1

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Adeshnee MNHW. HIV-Associated oral lesions in HIV-seropositive patients at an HIV treatment clinic in South Africa. J AIDS Clin Res. (2015) 6:422. doi: 10.4172/2155-6113.1000422

CrossRef Full Text | Google Scholar

16. Bodhade AS, Ganvir SM, Hazarey VK. Oral manifestations of HIV infection and their correlation with CD4 count. J Oral Sci. (2011) 53:203–11. doi: 10.2334/josnusd.53.203

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Leigh JE, Steele C, Wormley FL, Luo W, Clark RA, Gallaher W, et al. Th1/Th2 cytokine expression in saliva of HIV-positive and HIV-negative individuals: a pilot study in HIV-positive individuals with oropharyngeal candidiasis. J Acquir Immune Defic Syndr Hum Retrovirol. (1998) 19:373–80. doi: 10.1097/00042560-199812010-00008

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Vastardis S, Leigh JE, Wozniak K, Yukna R, Fidel PL. Influence of periodontal disease on Th1/Th2-type cytokines in saliva of HIV-positive individuals. Oral Microbiol Immunol. (2003) 18:88–91. doi: 10.1034/j.1399-302X.2003.00045.x

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Leigh JE, McNulty KM, Fidel PL. Characterization of the immune status of CD8+ T cells in oral lesions of human immunodeficiency virus-infected persons with oropharyngeal Candidiasis. Clin Vaccine Immunol. (2006) 13:678–83. doi: 10.1128/CVI.00015-06

PubMed Abstract | CrossRef Full Text | Google Scholar

20. World Medical Association. World medical association declaration of Helsinki. JAMA. (2013) 310:2191–4. doi: 10.1001/jama.2013.281053

CrossRef Full Text | Google Scholar

21. Kroidl A, Schaeben A, Oette M, Wettstein M, Herfordt A, Häussinger D. Prevalence of oral lesions and periodontal diseases in HIV-infected patients on antiretroviral therapy. Eur J Med Res. (2005) 10:448–53. Available online at: https://daignet.de/site-content/die-daig/fachorgan/archiv/2005/ejomr-2005-vol.10/448.pdf

PubMed Abstract | Google Scholar

22. Armitage GC. Development of a classification system for periodontal diseases and conditions. Ann Periodontol. (1999) 4:1–6. doi: 10.1902/annals.1999.4.1.1

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Chakrabarti R, Schutt CE. The enhancement of PCR amplification by low molecular weight amides. Nucleic Acids Res. (2001) 29:2377–81. doi: 10.1093/nar/29.11.2377

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Martínez JMG, Gómez EV, Pemán J, Cantón E, García MG, del Castillo Agudo L. Identification of pathogenic yeast species by polymerase chain reaction amplification of the RPS0 gene intron fragment. J Appl Microbiol. (2010) 108:1917–27. doi: 10.1111/j.1365-2672.2009.04595.x

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Gupta N, Gupta ND, Garg S, Goyal L, Gupta A, Khan S, et al. The effect of type 2 diabetes mellitus and smoking on periodontal parameters and salivary matrix metalloproteinase-8 levels. J Oral Sci. (2016) 58:156–63. doi: 10.2334/josnusd.58.1

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Campisi G, Pizzo G, Milici ME, Mancuso S, Margiotta V. Candidal carriage in the oral cavity of human immunodeficiency virus–infected subjects. Oral Surg Oral Med Oral Pathol Oral Radiol. (2002) 93:281–6. doi: 10.1067/moe.2002.120804

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Mushi MF, Mtemisika CI, Bader O, Bii C, Mirambo MM, Groß U, et al. High oral carriage of non- albicans Candida spp. among HIV-infected individuals. Int J Infect Dis. (2016) 49:185–8. doi: 10.1016/j.ijid.2016.07.001

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Jabra-Rizk MA, Ferreira SMS, Sabet M, Falkler WA, Merz WG, Meiller TF. Recovery of Candida dubliniensis and other yeasts from human immunodeficiency virus-associated periodontal lesions. J Clin Microbiol. (2001) 39:4520–2. doi: 10.1128/JCM.39.12.4520-4522.2001

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Clark-Ordóñez I, Callejas-Negrete OA, Aréchiga-Carvajal ET, Mouriño-Pérez RR. Candida species diversity and antifungal susceptibility patterns in oral samples of HIV/AIDS patients in Baja California, Mexico. Med Mycol. (2017) 55:myw069. doi: 10.1093/mmy/myw069

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Costa CR, Cohen AJ, Fernandes OFL, Miranda KC, Passos XS, Souza LKH, et al. Asymptomatic oral carriage of Candida species in HIV-infected patients in the highly active antiretroviral therapy era. Rev Inst Med Trop Sao Paulo. (2006) 48:257–61. doi: 10.1590/S0036-46652006000500004

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Kirkpatrick WR, Revankar SG, Mcatee RK, Lopez-Ribot JL, Fothergill AW, McCarthy DI, et al. Detection of Candida dubliniensis in oropharyngeal samples from human immunodeficiency virus-infected patients in North America by primary CHROMagar candida screening and susceptibility testing of isolates. J Clin Microbiol. (1998) 36:3007–12.

PubMed Abstract | Google Scholar

32. Hulimane S, Maluvadi-Krishnappa R, Mulki S, Rai H, Dayakar A, Kabbinahalli M. Speciation of Candida using CHROMagar in cases with oral epithelial dysplasia and squamous cell carcinoma. J Clin Exp Dent. (2018) 10:e657–e660. doi: 10.4317/jced.54737

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Nadeem SG, Hakim ST, Kazmi SU. Use of CHROMagar Candida for the presumptive identification of Candida species directly from clinical specimens in resource-limited settings. Libyan J Med. (2010) 5:2144. doi: 10.3402/ljm.v5i0.2144

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Nayak S, Kavitha B, Sriram G, Saraswathi T, Sivapathasundharam B, Dorothy A. Comparative study of Candida by conventional and CHROMagar method in non-denture and denture wearers by oral rinse technique. Indian J Dent Res. (2012) 23:490–7. doi: 10.4103/0970-9290.104956

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Ribeiro Ribeiro AL, de Alencar Menezes TO, de Melo Alves-Junior S, de Menezes SAF, Marques-da-Silva SH, Rosário Vallinoto AC. Oral carriage of Candida species in HIV-infected patients during highly active antiretroviral therapy (HAART) in Belém, Brazil. Oral Surg Oral Med Oral Pathol Oral Radiol. (2015) 120:29–33. doi: 10.1016/j.oooo.2015.03.008

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Canabarro A, Valle C, Farias MR, Santos FB, Lazera M, Wanke B. Association of subgingival colonization of Candida albicans and other yeasts with severity of chronic periodontitis. J Periodontal Res. (2013) 48:428–32. doi: 10.1111/jre.12022

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Hannula J, Dogan B, Slots J, Okte E, Asikainen S. Subgingival strains of Candida albicans in relation to geographical origin and occurrence of periodontal pathogenic bacteria. Oral Microbiol Immunol. (2001) 16:113–8. doi: 10.1034/j.1399-302x.2001.016002113.x

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Yin MT, Dobkin JF, Grbic JT. Epidemiology, pathogenesis, and management of human immunodeficiency virus infection in patients with periodontal disease. Periodontol. (2007) 44:55–81. doi: 10.1111/j.1600-0757.2007.00205.x

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Ryder MI. An update on HIV and periodontal disease. J Periodontol. (2002) 73:1071–8. doi: 10.1902/jop.2002.73.9.1071

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Shinozaki S, Moriyama M, Hayashida J-N, Tanaka A, Maehara T, Ieda S, et al. Close association between oral Candida species and oral mucosal disorders in patients with xerostomia. Oral Dis. (2012) 18:667–72. doi: 10.1111/j.1601-0825.2012.01923.x

PubMed Abstract | CrossRef Full Text | Google Scholar

41. López-Verdín S, Andrade-Villanueva J, Zamora-Perez AL, Bologna-Molina R, Cervantes-Cabrera JJ, Molina-Frechero N. Differences in salivary flow level, xerostomia, and flavor alteration in mexican HIV patients who did or did not receive antiretroviral therapy. AIDS Res Treat. (2013) 2013:613278. doi: 10.1155/2013/613278

CrossRef Full Text | Google Scholar

42. Cannon RD, Chaffin WL. Oral colonization by Candida albicans. Crit Rev Oral Biol Med. (1999) 10:359–83. doi: 10.1177/10454411990100030701

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Margiotta V, Campisi G, Mancuso S. Plasma HIV-1 RNA and route of transmission in oral candidiasis and oral hairy leukoplakia. Oral Dis. (2008) 6:194–5. doi: 10.1111/j.1601-0825.2000.tb00333.x

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Delgado ACD, de Jesus Pedro R, Aoki FH, Resende MR, Trabasso P, Colombo AL, et al. Clinical and microbiological assessment of patients with a long-term diagnosis of human immunodeficiency virus infection and Candida oral colonization. Clin Microbiol Infect. (2009) 15:364–71. doi: 10.1111/j.1469-0691.2009.02707.x

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Cutler CW, Jotwani R. Oral mucosal expression of HIV-1 receptors, co-receptors, and α-defensins: tableau of resistance or susceptibility to HIV infection? Adv Dent Res. (2006) 19:49–51. doi: 10.1177/154407370601900110

CrossRef Full Text | Google Scholar

46. Kweon M-N. Sublingual mucosa: a new vaccination route for systemic and mucosal immunity. Cytokine. (2011) 54:1–5. doi: 10.1016/j.cyto.2010.12.014

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Cornwell WD, Kim V, Fan X, Vega ME, Ramsey FV, Criner GJ, et al. Activation and polarization of circulating monocytes in severe chronic obstructive pulmonary disease. BMC Pulm Med. (2018) 18:101. doi: 10.1186/s12890-018-0664-y

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Takeichi O, Haber J, Kawai T, Smith DJ, Moro I, Taubman MA. Cytokine Profiles of T-lymphocytes from gingival tissues with pathological pocketing. J Dent Res. (2000) 79:1548–55. doi: 10.1177/00220345000790080401

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Berglundh T, Liljenberg B, Lindhe J. Some cytokine profiles of T-helper cells in lesions of advanced periodontitis. J Clin Periodontol. (2002) 29:705–9. doi: 10.1034/j.1600-051X.2002.290807.x

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Li H, Reeves RK. Functional perturbation of classical natural killer and innate lymphoid cells in the oral mucosa during SIV infection. Front Immunol. (2013) 3:417. doi: 10.3389/fimmu.2012.00417

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Slavinsky J, Myers T, Swoboda RK, Leigh JE, Hager S, Fidel PL. Th1/Th2 cytokine profiles in saliva of HIV-positive smokers with oropharyngeal candidiasis. Oral Microbiol Immunol. (2002) 17:38–43. doi: 10.1046/j.0902-0055.2001.00080.x

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Dezerega A, Maggiolo S, Garrido M, Dutzan N. The TH17 vs. TREG imbalance in the pathogenesis of periodontitis: new approach for dichotomy TH1 vs. TH2. Rev Clínica Periodoncia, Implantol y Rehabil Oral. (2008) 1:70–2. doi: 10.1016/S0718-5391(08)70012-0

CrossRef Full Text | Google Scholar

53. Gaffen SL, Hajishengallis G. A new inflammatory cytokine on the block: re-thinking periodontal disease and the Th1/Th2 paradigm in the context of Th17 cells and IL-17. J Dent Res. (2008) 87:817–28. doi: 10.1177/154405910808700908

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Prame Kumar K, Nicholls AJ, Wong CHY. Partners in crime: neutrophils and monocytes/macrophages in inflammation and disease. Cell Tissue Res. (2018) 371:551–65. doi: 10.1007/s00441-017-2753-2

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Herrera BS, Martins-Porto R, Maia-Dantas A, Campi P, Spolidorio LC, Costa SKP, et al. iNOS-derived nitric oxide stimulates osteoclast activity and alveolar bone loss in ligature-induced periodontitis in rats. J Periodontol. (2011) 82:1608–15. doi: 10.1902/jop.2011.100768

PubMed Abstract | CrossRef Full Text | Google Scholar